Surface-Matching Metrics in NLP — Definitions, What They Measure, and How to Choose

Introduction: Metrics Comparing the Same Strings Answer Different Questions

Natural language processing offers a long list of metrics for comparing system output with reference text: exact match (EM), token F1, BLEU, ROUGE, chrF, METEOR, WER, and TER. They all compare strings, but each answers a different question.

• Is the complete output identical to a reference?
• What fraction of the produced words also occur in the reference?
• What fraction of the reference words was recovered?
• How many edits would turn the output into a reference?
• How similar are local word order and word forms?
• How well is the global relative order of matched words preserved?
• How closely does the output match input-specific expressions that occur across references?

This article calls them surface-matching metrics. The term marks the article's working scope; it is not a standard, mutually exclusive taxonomy. Here, “surface” means that the primary matching signal comes from normalized characters, tokens, $n$-grams, subsequences, explicit edit operations, or dictionary-based alignments rather than direct comparison of learned contextual representations. Determinism alone does not define the boundary: BERTScore can also run deterministically once its model and settings are fixed.

Whether an evaluation is reference-based is a separate axis. It describes a protocol that supplies human references, not whether the matching representation is surface-based or semantic. Most equations here compare a hypothesis with references, but Jaccard can also compare two system outputs. Conversely, BERTScore and reference-based COMET are reference-based yet outside this article's surface scope, while quality-estimation systems such as CometKiwi are reference-free and also outside it. Agreement with a reference is not the same as correctness against the input or external facts.

METEOR and CIDEr are boundary cases for different reasons. METEOR stem matching is morphological, while synonym and paraphrase matching enters limited, context-independent lexical semantics. CIDEr obtains IDF weights from an external reference corpus, but its matching signal remains explicit TF-IDF $n$-gram overlap. This article includes both in its broad scope without treating either mechanism as sentence-level semantic understanding.

Unless stated otherwise, implementation names, versions, commits, and defaults in this article were checked as of 2026-07-13 and describe only the release, tag, or commit named.

No single surface metric fits every task. Each metric reduces a text difference to a different quantity, and its number can be interpreted only after its definition, denominator, aggregation unit, and preprocessing are specified.

1. Overview: Classify Metrics by Unit and Perspective

Rather than cataloging metric names, this article is organized around seven recurring design families: (1) whole-sequence identity, (2) order-independent bag/set overlap, (3) edit paths, (4) contiguous $n$-grams, (5) subsequence- or rank-based global order, (6) explicit token alignment, and (7) corpus-statistics-weighted multi-reference consensus.

These families are not mutually exclusive. CIDEr is a token $n$-gram metric and also a composite of IDF across inputs with a mean over all references within each input. METEOR extends unigram overlap into explicit alignments and chunks, while TER searches edit paths over a sequence alignment.

Matching unit (character, word, or subword token), order range (ignored, local, or global), score orientation (precision, recall, balanced score, distance, or error rate), and evaluation protocol (a general construction or a named task-originated specification) are orthogonal axes. The $n$-gram family alone contains precision-oriented BLEU, recall-oriented original ROUGE-N, and balanced chrF; a character metric is not necessarily recall-oriented, nor is every translation metric precision-oriented. EM, Jaccard, and Levenshtein are broadly applicable constructions, whereas SQuAD token F1, WER/CER, TER, BLEU, ROUGE, METEOR, RIBES, CIDEr, and GEC GLEU+ carry task-originated preprocessing, aggregation, or interpretations. A formula can be applied to another task, but its empirical validity does not carry over automatically.

Representative metrics and what each primarily observes:

The order-sensitivity column states only when rearranging units can change the score. “Typical use” reflects convention and empirical results; the equation alone does not guarantee suitability for a task.

Even among “higher is better” metrics, some implementations return $[0,1]$ and others $[0,100]$. The cosine average in original CIDEr is normally in $[0,1]$, whereas CIDEr-D's factor of 10 gives it a nominal $[0,10]$ range. Raw Levenshtein distance has no shared upper bound unless sequence lengths are fixed. TER, WER, and CER are error rates, so lower is better, and an error count larger than the reference length can push them above $1$, or 100%. Report the metric, implementation, and scale with every value.

1.1 Coverage of neighboring metrics

Several metrics sit near the seven families or share a name with a metric in them. The table shows where each fits and how much detail this article gives it.

For selection, consider NIST when reproducing a NIST result, Google GLEU when a symmetric sentence $n$-gram reward is specifically required, GEC GLEU+ only when evaluating corrections relative to a source, and RIBES when global reordering is an explicit diagnostic target. Replacing one member of a family with another does not add a new quality dimension.

The following notes on RIBES and GEC GLEU+ pin implementation behavior that the metric names alone do not determine. Readers interested only in choosing metrics can skip to Section 2.

As a version-pinned example, NTT RIBES 1.03.1 (dated 2014-09-08 in its distribution history and mirrored at GitHub commit 7e76cf24d32de70e9e6afd0c9b95d8c140b7235c) defaults to $NKT\cdot P^{0.25}\cdot BP^{0.10}$ and expects tokenized input, lowercasing it unless -c is supplied. With multiple references, it selects the highest score per sentence and macro-averages those sentence scores. Citing the 2010 paper alone does not fix these conventions. As of 2026-07-13, that distribution has no release after 1.03.1. Its README requires “Python 2.6 or later,” excludes Python 3.0, and separately says that RIBES 1.02.4 or earlier works only with Python 3.1 or later. That wording does not fully specify the Python 3 support range for 1.03.1, although the source contains Python 3 branches. The source's _RIBES_VERSION also still says 1.03. Treat this distribution as a historical implementation rather than a current Python package, and report its distribution history, runtime, and commit alongside the stale internal version string.

The authors' GEC GLEU+ code also changed after the paper. Its final commit, 50b5032a4ef2444b9381fb47a55b3bac0654a6d7 (2017-06-09), fixes a brevity penalty in gleu.py that had the hypothesis and reference lengths swapped. compute_gleu defaults to 500 iterations with multiple references and one iteration with one reference, reseeding Python's random number generator (RNG) with j*101 for iteration $j$. External random state therefore does not change this pinned code's result. The code uses Python 2-era print statements and xrange; it is a historical distribution, not a maintained current-Python package API. A port must preserve its sampling, seed, BP, and aggregation semantics.

In that same pinned revision, the CLI accepts -n, but GLEU.__init__(n) ignores the argument, sets self.order = 4, and fixes the geometric-mean divisor at four. Treat 1–4-grams as the verified author-script specification; do not treat -n as a working API for selecting another order.

2. Common Definitions: The Denominator Reveals the Design

2.1 Hypotheses, references, and preprocessing

Let the system output, or hypothesis, be $h$, and a reference be $r$. With multiple references, write

An evaluation corpus contains $S$ cases $(h_s,R_s)$.

Metrics rarely receive raw strings. Their input may undergo Unicode normalization, case folding, punctuation removal, whitespace processing, tokenization, stemming, or morphological analysis. A more accurate description of the evaluated object is therefore

where $T$ is a preprocessing function. Changing $T$ changes the measurement even when the metric retains the same name.

Unicode normalization has several forms. NFC/NFD compose or decompose canonically equivalent representations, whereas NFKC/NFKD also fold compatibility characters such as full-width alphanumerics, ligatures, and circled numbers. Compatibility folding can help retrieval but can erase meaningful distinctions when evaluating product codes or orthographic quality. In many Python implementations, a “character” is a Unicode code point obtained by iterating a Python string, not a grapheme cluster that keeps a combining sequence or emoji ZWJ sequence together. Fix the normalization form and character unit separately.

The 13a BLEU and chrF paths in sacreBLEU 2.6.0, JiWER 4.0.0's default transforms, and NLTK 3.10.0's METEOR do not apply NFC or NFKC in the evaluation layer. NLTK METEOR's default preprocessing is only str.lower on each token. A tokenizer such as SentencePiece may have a model-internal normalizer; in that case, the model itself is part of $T$ and must be versioned.

Word units also depend on the language: a simple split() treats an unspaced Japanese sentence as one token. For token metrics, use the same morphological analyzer, dictionary, user dictionary, normalization, and segmentation output for hypotheses and references, and preserve all their versions. Avoid re-tokenizing already tokenized input.

2.2 Sets, multisets, and clipping

Let $C_x(g)$ be the number of occurrences of an $n$-gram $g$ in sequence $x$. Overlap between a hypothesis and one reference is normally counted as the intersection of multisets:

If the occurs once in the reference, producing it ten times yields only one match. This upper bounding is called clipping.

A set-based Jaccard score instead removes duplicate occurrences first. “Set” and “multiset” overlap are separate definitions even when both are casually called overlap.

The scope of clipping differs by metric. The displayed equation is a pairwise $\min$ against one reference. BLEU first forms the maximum count across references separately for each $n$-gram and then clips the hypothesis count; Lin's (2004) ROUGE-N Equation (1) sums pairwise matches against references; CIDEr-D clips counts separately for each candidate-reference pair. Original CIDEr does not use the same clipping as CIDEr-D. Concatenating references into one giant bag is generally equivalent to none of these rules.

Clipping guarantees only that repetition of a symbol earns no match credit beyond its reference count; it is not itself a length correction. With recall alone, for example, output occurrences beyond the reference count do not increase the numerator, but they do not lower the score either; a hypothesis-side precision denominator or another penalty is required for that.

2.3 Precision, recall, and $F_\beta$

Putting the number of hypothesis elements in the denominator gives precision:

It asks what fraction of the produced units also occurs as the same matching units in the reference. It is sensitive to unnecessary additions and overgeneration, but it is not the fraction whose truth the reference establishes.

Putting the number of reference elements in the denominator gives recall:

It asks what fraction of the reference's matching units the system recovered. It is sensitive to omissions, but it does not directly measure how much of the reference's meaning was recovered.

A common combination is $F_\beta$:

Values of $\beta>1$ emphasize recall; $\beta<1$ emphasizes precision; $\beta=1$ gives the harmonic mean $F_1$. The same number of matches answers a different question when the denominator changes.

Swapping the hypothesis and reference swaps $P$ and $R$. With the same multiset match count $M$, $F_1$ is symmetric under that swap, but $F_\beta$ with $\beta\ne1$ is not. A statement that a score “emphasizes recall” therefore requires fixed hypothesis and reference roles.

These ratios are mathematically undefined when their corresponding denominator is zero. The metric name alone does not decide whether two empty sequences score one or zero, or how to score a case where only one side is empty. Fix the implementation's empty-input convention and report it separately from the ordinary formula.

2.4 Multiple references and aggregation

Common ways to handle multiple references include:

• taking the maximum score over references;
• averaging scores over references;
• taking the maximum reference count separately for each $n$-gram;
• interpreting all references as human consensus and averaging similarity to them.

BLEU uses maximum reference counts per $n$-gram, whereas CIDEr averages agreement with all references. These are intentionally different designs.

Corpus aggregation also varies:

• Macro average: compute a score for each predefined evaluation unit, such as a sentence, document, or dialogue, then average those scores with equal weight per unit.
• Micro average: sum sufficient statistics such as matches, edits, and denominators across all cases, then compute the ratio. Longer cases and cases with more opportunities normally carry more weight.
• Metric-specific corpus score: apply a metric's own nonlinear formula, such as BLEU's geometric mean and BP, to the aggregated sufficient statistics.

Both the weighting of short and long texts and the function applied last can differ, so the average of sentence scores generally does not equal the corpus score. Corpus BLEU in particular aggregates per-order match and total counts and hypothesis and effective reference lengths over all cases, then computes the score once. The arithmetic mean of smoothed sentence BLEU is neither an approximation of corpus BLEU nor a micro average; it is a different statistic.

2.5 Similarity, distance, and error rate

This article uses “metric” in the broad NLP sense, not only for a distance function satisfying the mathematical metric-space axioms.

• A similarity or agreement score normally grows with agreement. EM, Jaccard, Dice, and F-scores are examples, although precision, recall, and BLEU are directed scores that can change when hypothesis and reference roles are swapped.
• A distance becomes smaller as sequences get closer. Ordinary unit-cost Levenshtein distance is a mathematical distance on sequences, but its raw value is not normalized to $[0,1]$.
• An error rate divides errors by reference-side opportunities. WER/CER and TER are directed and can exceed one when the error count exceeds the reference length, so merely computing $1-WER$ does not produce a bounded $[0,1]$ similarity. TER also depends on its shift definition and heuristic search.

Reversing the better/worse direction does not change the score type, so tables and plots should state both score type and direction.

2.6 Length effects on scores

Length affects scores through four distinct mechanisms, which the single label “length penalty” obscures.

1. Denominator effects: precision reflects extra hypothesis elements, recall reflects missing reference elements, and F-scores combine them. Recall-only ROUGE does not directly penalize extra output.
2. Scale normalization: dividing WER/CER or TER by reference length makes error counts more comparable across examples of different lengths. It is not a penalty that steers output toward a preferred length.
3. Explicit length penalties: BLEU BP penalizes only a corpus hypothesis that is too short; it adds no independent penalty for being too long. Overlong output is disadvantaged mainly through the denominator of clipped precision. The CIDEr-D Gaussian penalizes candidate-reference length differences in either direction.
4. Vector normalization: division by vector norms in CIDEr cosine compares vector direction; it is not a direct judgment that sequence length is appropriate.

Raw Levenshtein distance has no length correction, while EM merely fails when lengths differ. In METEOR, chrF, and Jaccard/Dice, length affects the score only through precision/recall or set denominators. Report clipping and length penalties separately as well.

3. Exact and Bag-of-Tokens Matching: Count Identity and Token Overlap

3.1 Exact match (EM)

Exact Match is the simplest metric:

With multiple references, a common definition returns one if any reference matches:

Corpus EM normally means exact-match accuracy, the average of per-case indicators: $S^{-1}\sum_s EM(h_s,R_s)$. It is therefore a macro aggregation that gives every case equal weight.

Perspective: Is the entire output identical to one allowed answer?

Strength: It is unambiguous and appropriate for IDs, labels, numbers, normalized short answers, or JSON fields with a strict output contract.

Weakness: Every valid paraphrase is a mismatch. “Normalized EM” also depends on its normalizer. SQuAD-style evaluation uses lowercasing and removal of punctuation, articles, and excess whitespace, but blindly applying that normalizer to product codes or Japanese text may erase meaningful distinctions.

The official SQuAD Python normalizer removes the ASCII symbols in string.punctuation and the English articles a, an, and the. It is neither a general Unicode-punctuation specification nor a Japanese normalizer.

3.2 Token precision, recall, and F1

Treat the hypothesis and reference as token multisets and let $M$ be the number of matched tokens. Then

Token F1 in extractive QA is commonly this bag-of-tokens measure. It ignores order.

With multiple answer annotations, the official SQuAD v1.1 evaluation computes token F1 separately against each reference, takes the maximum, and then averages those case scores across questions.

The ordinary formula cannot be used unchanged for empty token sequences. The official SQuAD 2.0 script returns F1 one when both normalized token lists are empty and zero when only one is empty. This is that implementation's convention for no-answer cases.

Perspective: How completely do the tokens of a short answer overlap, and how few extra tokens does the hypothesis add?

Strength: It grants partial credit beyond exact match and handles span-boundary variation in short answers and named entities.

Weakness: It cannot distinguish dog bites man from man bites dog. A one-token negation difference can also look small inside a long answer.

3.3 Jaccard and Sørensen–Dice

Using multiset overlap $M$, Jaccard similarity is

and the Sørensen–Dice coefficient is

With the same multiset match count, Dice is algebraically identical to token $F_1$. A set version first discards duplicates and therefore measures something different.

Perspective: Element overlap under a symmetric definition that does not assign either side as the gold denominator.

Suitable uses: Duplicate detection, tag sets, keyword sets, and output-to-output similarity.

Caution: Jaccard and Dice are monotonically related but numerically different. Implementations must also define whether two empty sets receive a score of one or zero. There is no unique standard extension to multiple references, so state whether references are maximized, averaged, or combined.

4. Edit Metrics: Measure the Work Needed to Fix an Output

4.1 Levenshtein distance

Levenshtein distance is the minimum number of insertions, deletions, and substitutions required to transform $x=(x_1,\ldots,x_m)$ into $y=(y_1,\ldots,y_n)$. Its dynamic program is

with $D_{0,j}=j$ and $D_{i,0}=i$.

This ordinary recurrence does not include transposition of two adjacent elements. Damerau–Levenshtein distance, which can count a transposition as one operation, is a different metric.

Bibliographically, Levenshtein's Russian original appeared in 1965 and its English translation in 1966. The English title uses “reversals,” whereas the Russian title refers to symbol substitutions; that translated title is not a basis for adding adjacent transposition to the recurrence shown here.

Perspective: The minimum number of defined unit edits needed to turn a hypothesis into a reference.

Raw distance tends to grow with length, so comparisons across sequences of different lengths often normalize it. There is no single universal denominator: reference length, maximum length, and the sum of both lengths are all used. State the denominator whenever reporting a “normalized edit distance.”

4.2 WER and CER

By convention, Word Error Rate (WER) comes from aligning the reference token sequence to the hypothesis and counting substitutions $S$, deletions $D$ for reference words missing from the hypothesis, and insertions $I$ for extra hypothesis words:

where $N_r$ is the number of reference words. Character Error Rate (CER) has the same form with characters as the units.

Perspective: The error rate of ASR or OCR output relative to a human transcription.

Range and aggregation: Insertions can make WER or CER exceed one, so they are not similarities. Corpus WER normally micro-aggregates all $S+D+I$ and divides by total reference length; this differs from averaging sentence WER values.

The ASR reference implementation NIST SCTK 2.4.12 sclite (v2.4.12 tag commit 62aabfa95938bfea284f8b68a374c7ddcb0fce52) likewise pools reference-word and $S,D,I$ counts in its system summary before reporting rates. The empty-input and transform conventions below are specific to JiWER.

The displayed ratio is undefined when $N_r=0$. JiWER 4.0.0 returns zero when the entire reference and hypothesis are both empty, and otherwise returns the insertion count itself when the entire reference is empty. Other implementations may reject empty references, so this boundary convention is part of reproducibility. Multiple-reference WER also has no universal rule; explicitly report a choice such as minimum WER.

JiWER 4.0.0's default WER transform replaces runs of at least two whitespace characters with one U+0020, applies strip(), and splits each input line on its default delimiter U+0020. It does not treat an isolated tab as a word boundary through generic whitespace splitting, nor does it lowercase or remove punctuation. With multiple lines, it aligns corresponding lines and pools their edit counts; this differs from using wer_contiguous to concatenate the lines for one global alignment. The default CER transform strips outer whitespace and then converts to a sequence of Python code points, including internal spaces. When several alignments have the same minimum distance, library tie-breaking can also change the $S,D,I$ breakdown without changing WER.

For CER, therefore specify whether a “character” is a Unicode code point, grapheme cluster, or normalized symbol. A combining sequence and a precomposed character can look identical but remain different before NFC normalization. Passing unsegmented, unspaced Japanese directly to JiWER's default WER treats the whole utterance as one word. Pre-segment hypotheses and references with the same Japanese segmenter and dictionary, and report CER alongside WER.

4.3 TER and HTER

Translation Edit Rate (TER) allows word insertion, deletion, and substitution, plus a shift that moves a contiguous phrase to another position at unit cost.

Let $E_{\mathrm{TER}}(h,r)$ be the edit count that the TER search returns for turning hypothesis $h$ into reference $r$. The original multi-reference definition is

The numerator selects the one reference reachable with the fewest edits, but the denominator is the mean length of all references, not the length of the selected reference. Corpus TER is not the mean of sentence TER values. Standard aggregation, including sacreBLEU 2.6.0, is

Because the shift search space is large, standard TER uses a greedy heuristic; unlike plain Levenshtein distance, it does not guarantee the minimum over every possible shift sequence.

Perspective: The number of TER operations needed, divided by mean reference length—a proxy for the amount of post-editing. Counting a long phrase movement as one operation assigns block reordering a lower cost than plain WER. The equation measures operation count, however, not actual editing time, cognitive effort, or translation acceptability itself.

HTER (Human-targeted TER) compares against a targeted reference that a human creates by minimally post-editing a particular system output. It preserves acceptable wording from that output, but requires human labor and is therefore not a fully automatic metric.

Implementation cautions: The empty-input, preprocessing, and search settings below are conventions of the pinned implementations.

If every reference in the scored corpus is empty, the denominator is zero and the original ratio is undefined. SacreBLEU 2.6.0 implements the special convention of returning zero when every hypothesis is also empty and 100 when any hypothesis is nonempty. If only some cases have empty references, the deletion-equivalent edits that turn those nonempty hypotheses into empty references enter the numerator while other cases provide the denominator in the ordinary pooled calculation, so the result can exceed 100.

The sacreBLEU 2.6.0 TER defaults are case-insensitive, normalized=False, punctuation-preserving, and asian_support=False. Default Japanese input is therefore effectively whitespace-split and is not automatically segmented. asian_support affects CJK processing only when normalization or punctuation removal is also enabled. TERCOM normalized here refers to rules for XML entities, line breaks, and punctuation tokenization—not Unicode NFC or NFKC.

SacreBLEU describes its TER as a near-exact reimplementation of TERCOM, but its beam edit distance and shift-candidate limits can differ from TERCOM in extreme corner cases. Pin the implementation and version, not only the name “TER.”

The TERCOM comparison target here is jhclark/tercom 0.10.0 at pinned commit e413d077e5b9ecc96b0e7de5fe6015eb135c2f6d. Its defaults disable normalization, ignore case, retain punctuation, use beam width 20, and limit maximum shift distance to 50 words. The original paper (Snover et al., 2006) and the later TERCOM distribution, whose README credits Matthew Snover, Shuguang Wang, and Spyros Matsoukas and carries a 2007 copyright, are distinct bibliographic and versioned artifacts. SacreBLEU 2.6.0 matches the preprocessing defaults but does not expose TERCOM's beam width as an equivalent public parameter.

5. $n$-gram Metrics: Measure Local Vocabulary and Order

5.1 BLEU: Geometric mean of clipped precision with a brevity penalty

BLEU was proposed as a corpus-level machine-translation metric. Modified precision at order $n$ clips hypothesis counts by the maximum count in any reference:

Standard BLEU-$N$ multiplies the weighted geometric mean of these precisions by a brevity penalty (BP), normally with $w_n\ge 0$ and $\sum_n w_n=1$:

Here $c$ is total hypothesis length, and $r$ is the sum of the effective reference lengths selected for each hypothesis. Common BLEU-4 uses $N=4$ and $w_n=1/4$. With multiple references, common implementations select the reference length closest to the hypothesis length and choose the shorter one on a distance tie.

The displayed BP assumes $c>0$. Assigning BLEU zero to an empty hypothesis corpus is a common implementation convention.

Perspective: Do many local hypothesis token $n$-grams occur in the same form in at least one reference, and has corpus BP discouraged gaming precision with output that is too short?

Design implication: BLEU does not directly calculate recall. BP penalizes corpus-level length deficiency, but it does not ask whether each reference unit was recovered and is not a substitute for recall. Because the mean is geometric, any unsmoothed $p_n=0$ makes the whole score zero. With multiple references, maximum counts are selected independently for each $n$-gram, so the set of clipped matches need not occur together in any one coherent reference sentence.

The original BLEU targets corpus evaluation. Its unsmoothed corpus formula can be applied to one sentence when every required order has a positive denominator. A short sentence, however, can lack higher-order $n$-grams or have zero matches at one order, making the result degenerate at zero or leaving an order undefined. Practical sentence BLEU must specify smoothing or effective-order behavior. These are additional specifications, not one uniquely defined part of the original metric.

Reporting caution: Tokenization, casing, reference count, smoothing, effective order, and implementation version change the score. Post (2018) documented how underreported configurations made supposedly comparable “BLEU” values incomparable. Store the complete setup, for example with a sacreBLEU metric signature.

sacreBLEU 2.6.0 defaults: BLEU() and the ordinary corpus CLI use a 0–100 scale, maximum order 4, smooth_method="exp", effective_order=False, and mixed case. With no tokenizer and no known target language, the fallback is 13a. These are sacreBLEU implementation defaults, not the original unsmoothed equation itself. Available smoothing modes are none, floor, add-k, and exp; an omitted value means 0.1 for floor and 1 for add-k.

The compatibility function sacrebleu.sentence_bleu is an exception: besides exp smoothing, it defaults to effective_order=True, and CLI --sentence-level also enables effective order for BLEU. In contrast, BLEU().sentence_score inherits the class default effective_order=False and emits a warning. SacreBLEU effective order excludes an order when its hypothesis denominator is zero—that is, when no hypothesis $n$-gram of that order exists. It does not exclude an existing order merely because its match count is zero; separate smoothing is required to avoid that zero.

However, add-k adds $k$ to numerator and denominator for $n>1$ before the zero-denominator check, so it can retain a higher order that did not originally exist. Smoothing and effective order must therefore be recorded as the actual combination, not as independent labels.

When target language ja is known through --language-pair or trg_lang, sacreBLEU selects ja-mecab, using MeCab segmentation with an IPA dictionary. Arbitrary files scored without a language setting retain 13a, which does not segment unspaced Japanese, so Japanese BLEU can change greatly. The sacreBLEU layer does not add NFC/NFKC before ja-mecab; strict reproduction should lock sacrebleu[ja], MeCab, and ipadic. The signature records the tokenizer, but not external Unicode normalization or a changed max_ngram_order, so preserve those separately.

5.2 chrF and chrF++: Precision and recall over character $n$-grams

The original chrF formulation calculates character $n$-gram $P_n$ and $R_n$ at every order up to $N$ and takes their arithmetic means,

and combines them as

This definition averages precision and recall across orders before applying $F_\beta$. The implementation history also contains a different definition. Popović's 2017 reference chrF++.py first computes $F_\beta(P_n,R_n)$ at each order and takes their arithmetic mean. With its default eps_smoothing=False, sacreBLEU 2.6.0 averages $P_n$ and $R_n$ over effective orders for which both sides contain $n$-grams and then applies $F_\beta$; eps_smoothing=True uses epsilon and averages per-order F like the reference implementation. These variants generally differ, especially on short texts.

Perspective: How much local character material is shared from both the precision and recall perspectives?

Characters grant shared-substring credit across inflection, affixes, compounds, and tokenizer-boundary differences. This can be useful for morphologically rich languages and languages without trivial word segmentation. On the other hand, it rewards orthographically similar but different words and does not capture long-range order or sentence meaning.

chrF++ adds word $n$-grams to the character $n$-grams, improving sensitivity to word boundaries and local word order. Popović (2017) proposed adding word unigrams and bigrams and treating character and word orders as terms in the same arithmetic mean.

Implementation cautions: Empty inputs, multiple references, whitespace, and runtime behavior differ across implementations. SacreBLEU 2.6.0's default effective-order mode returns zero when there is no effective order, so chrF for two empty strings is also zero. With eps_smoothing=True, precision or recall is set to $10^{-16}$ when its corresponding $n$-gram total is zero, and all configured orders are averaged. Two empty strings therefore score approximately $10^{-14}$ on the 0–100 scale, neither exactly zero nor the perfect-match value 100.

For multiple references, the reference implementation and sacreBLEU select the reference with the highest pairwise chrF for each case and add that pair's sufficient statistics to the corpus. Both implementations update the maximum with a strict >, retaining the first reference on a tie. If tied references have different statistics, the pooled corpus score can therefore depend on reference order, which must also be preserved. The reference chrF++.py reports both this pooled document score and a separate macro average of sentence scores.

The 2017 chrF++.py also initializes its maximum to zero, so an empty or completely disjoint case in which every pairwise score is zero can leave best statistics unselected and fail. Because sacreBLEU 2.6.0 returns zero instead, results from the reference script should also state its version and the inputs it can handle.

The name alone does not determine the configuration. For example, the sacreBLEU 2.6.0 defaults for chrF use maximum character order 6, $\beta=2$, exclude whitespace from character $n$-grams, and use word order 0. Setting maximum word order to 2 produces the displayed name chrF2++.

Character handling also differs. The reference chrF++.py calls strip() and then removes only U+0020, leaving other internal whitespace, whereas sacreBLEU 2.6.0 with whitespace=False removes every character that Python str.split() recognizes as whitespace. Neither explicitly applies NFC or NFKC. Under Python 3 they count code-point $n$-grams, but running the reference script on Python 2 byte strings can count UTF-8 bytes instead, so pin the runtime too. SacreBLEU's chrF++ word side also uses only whitespace splitting and simple ASCII punctuation separation; it does not share ja-mecab. For raw, unsegmented Japanese, use the default character-only word_order=0, or pin external segmentation if word orders are enabled.

5.3 ROUGE: A family of coverage metrics, not one score

ROUGE was proposed as a family of metrics for summarization. “A ROUGE score” is not a complete definition.

Version note: The rouge-score implementation discussed in this section is pinned to the PyPI 0.1.2 source distribution (sdist) with SHA-256 c7d4da2683e68c9abf0135ef915d63a46643666f848e558a1b9f7ead17ff0f04. Commit 910758241597e44871c7029654f5ed7813dc3344 is the nearby repository snapshot that first recorded version 0.1.2 about 32 minutes after that upload; this article does not claim that the two are byte-identical.

ROUGE-N

For one reference, ROUGE-N recall is

Putting the reference in the denominator measures how much of its $n$-gram content the output recovered.

Lin's (2004) Equation (1) is written for a set of reference summaries and adds match counts and reference counts over them:

This form is a reference-length-weighted recall aggregation. Section 2.1 of the same paper, however, defines the final multi-reference score as the maximum pairwise ROUGE-N over the references. According to the paper, its package further applies jackknifing: it takes that maximum within each of the $K$ sets that leave out one reference and averages the $K$ results.

Perspective: ROUGE-1 observes unigram lexical coverage; ROUGE-2 and higher orders observe local phrases and local order. Unless stopword processing is separately specified, ROUGE-1 counts content words and function words alike.

The original paper centers recall, while modern libraries commonly return precision, recall, and $F_1$. The statements “ROUGE is recall-oriented” and “the reported value is ROUGE-L F1” are compatible, but the component must not be omitted.

rouge-score 0.1.2 from the Google Research repository (PyPI release 2022-07-22) provides multi-reference score_multi, which independently selects the reference with maximum pairwise $F_1$ for each requested ROUGE variant. Its README explicitly says “not an official Google product,” so it is not described here as Google's official evaluation specification. On a tie, numpy.argmax returns the first reference, so returned precision/recall can depend on reference order even when F1 is unchanged. Its BootstrapAggregator resamples cases with replacement, computes a case-score mean for each bootstrap sample, and returns the 2.5th, 50th, and 97.5th percentiles. The target is a macro mean, not pooled corpus-wide $n$-gram counts. The aggregator has no seed argument and uses NumPy's global RNG, so seed NumPy before the call if the confidence interval itself must be reproduced. Do not mix the summed-reference Equation (1), maximum-reference scoring, and mean-reference scoring under one unspecified “ROUGE.”

The rouge-score 0.1.2 default DefaultTokenizer lowercases text, replaces [^a-z0-9]+ with spaces, and does no stemming by default. Identical Japanese-only strings both become empty token lists and receive zero for ROUGE-1/2/L. Even canonically equivalent text can diverge: NFC café leaves caf, whereas NFD cafe plus a combining accent leaves cafe. For Japanese, provide a custom tokenizer that applies the same Unicode normalization and segmentation to both sides. The default implementation handles an empty $n$-gram denominator with max(count,1) and special-cases empty LCS input, so precision, recall, and F1 are all zero when either or both strings are empty.

In the same implementation, even an exact match receives zero for ROUGE-N when both texts contain fewer than $n$ tokens, because both $n$-gram bags are empty. Zero here is the implementation's empty-denominator convention.

ROUGE-L

Let reference token sequence $X$ have length $m$, hypothesis sequence $Y$ have length $n$, and let $LCS(X,Y)$ be their longest common subsequence length. Then

which can be combined with $F_\beta$. Lin's equation permits recall/precision weighting through $\beta$, whereas the fmeasure returned by rouge-score fixes $\beta=1$ and is $F_1$. A subsequence preserves order but need not be contiguous.

Perspective: Coverage by elements appearing in the same order. It can capture dependencies longer than a fixed-width $n$-gram, while allowing distant matches to form one subsequence.

Implementations distinguish sentence-level rougeL, which compares each text as one sequence, from summary-level rougeLsum, which computes sentence-wise union-LCS. With the rouge-score 0.1.2 default split_summaries=False, newlines are sentence boundaries; True instead invokes NLTK's default sentence tokenizer. The latter depends on the NLTK and Punkt-resource versions and is not a Japanese sentence splitter, so pin those resources too. Confusing the variant, newlines, or split_summaries changes summarization results.

ROUGE-W, ROUGE-S, and ROUGE-SU

• ROUGE-W: weighted LCS, rewarding one long contiguous run more than fragmented runs of equal total length.
• ROUGE-S: overlap of skip-bigrams—ordered word pairs that may have a configurable maximum gap.
• ROUGE-SU: skip-bigrams plus unigrams, supporting short texts or cases with no skip-bigram matches.

ROUGE-1, ROUGE-2, and ROUGE-L/Lsum dominate current practice, and the rouge-score 0.1.2 implementation pinned here likewise implements only ROUGE-N and ROUGE-L/Lsum. Reporting ROUGE-W/S/SU requires naming a different implementation, variant, and parameters.

6. METEOR: Align Words and Penalize Fragmented Order

METEOR constructs a one-to-one alignment between hypothesis and reference unigrams. The 2005 version used modules including exact match, Porter stemming, and WordNet synonyms. It selects an alignment with the maximum number of mapped unigrams and, when ambiguity remains, prefers fewer alignment crossings. Later versions add paraphrase tables, language-specific stemmers, and module weights.

With $m$ aligned words,

A generalized weighted harmonic mean can be written as

The 2005 choice $\alpha=0.9$ gives

which strongly emphasizes recall.

Next, partition the alignment into maximal contiguous runs whose matched words are adjacent and in the same order in both texts. These runs form the fewest possible chunks. If there are $ch$ chunks, a generalized fragmentation penalty is

and the final score is

When there are no matches and $m=0$, returning zero is the standard implementation convention. With multiple references, the 2005-style and NLTK sentence METEOR rule takes the maximum score over references.

The 2005 version used $\gamma=0.5$ and $\theta=3$. Later versions, including Meteor Universal, tune parameters and resources for languages and tasks; do not assume that every METEOR implementation uses all 2005 constants.

Because of the fragmentation penalty, even a nonempty exact match scores below one. For an exact match of length $L=m$, $P=R=1$ and $ch=1$, so the 2005 and NLTK defaults give

It is therefore 0.5 for one token and 0.9375 for two tokens. This is a boundary property of the computation; its scale differs from EM or F1, which assign one to a nonempty exact match.

Perspective: Word-level lexical adequacy, measured by precision and recall, and whether matched words stay grouped in the same order, measured by a fragmentation penalty.

Boundary with surface matching: Exact and stem matches are surface or morphological. Synonym and paraphrase modules enter limited lexical-semantic matching. They still do not understand context-dependent word senses, negation, numbers, or logical relations. Language resources vary in coverage, which also complicates cross-language comparisons.

Implementation cautions: The NLTK and CMU Meteor implementations below differ in matching resources, search, and aggregation APIs despite sharing the metric name.

NLTK 3.10.0's meteor_score is not a wrapper around official Meteor 1.5. It requires pre-tokenized Iterable[str] references and hypothesis and raises TypeError for raw strings. By default it applies str.lower to each token and greedily aligns still-unmatched words in exact, Porter-stem, then NLTK English-WordNet-synonym stages. Its defaults are alpha=0.9, beta=3.0, and gamma=0.5; its beta corresponds to $\theta$ in the equation above. Multiple references use the maximum sentence score. A pair containing an empty token list scores zero, while an empty reference iterable makes max() fail. It has no paraphrase module, language/task-specific weights, corpus API, or Unicode normalization. Pin WordNet data and tokenizer/resource versions, and do not apply these English defaults unchanged to Japanese.

The CMU Meteor 1.5 distribution, in contrast (official source pinned at commit 57a9ec5d5f65b8ceca6db5788ef876f3726614b3), provides language- and task-dependent exact/stem/synonym/paraphrase modules, module weights, tuned parameters, and alignment search with default beam size 40. Japanese is not among its bundled fully or partially supported languages; -l other selects exact matches and language-independent scoring parameters and requires user-provided segmentation, tokenization, and lowercasing as needed. Its system-level score pools sufficient statistics such as hypothesis and reference lengths, weighted matches, and chunks from selected alignments before applying the scoring formula, so it is generally not the arithmetic mean of sentence METEOR.

The bundled README heading and source Constants.VERSION say 1.5, but help text and sample output embedded in that README still contain old 1.4 strings. Do not infer the distribution version from those pasted examples; preserve the commit and runtime Eval ID. As of 2026-07-13, the official repository has no later commit or release tag.

7. CIDEr: Aggregate Multi-Reference Lexical Agreement with TF-IDF

CIDEr was designed for image-caption evaluation, treating similarity to a set of human references as a proxy for description consensus. It downweights $n$-grams that occur in references for many images and emphasizes those that are relatively distinctive in the current image's references.

The CIDEr-D equation and anti-gaming modifications below follow the 2015-06-03 arXiv v2/CVPR text. The implementation behavior described is that of COCO Caption Evaluation commit 3a9afb2682141a03e1cdc02b0df6770d2c884f6f.

Let $\mathcal I$ be the image set and $\Omega$ the $n$-gram vocabulary. The paper defines TF and IDF as

with $[\mathbf{g}_n(s)]_g=\operatorname{tf}_g(s)\operatorname{idf}_g$. An IDF document is an image, not one caption: $g$ is counted once if any reference for that image contains it. Original CIDEr maps candidate and reference words to stems or roots before counting. For candidate $c_i$ and $K$ references $s_{ij}$ for image $i$,

usually with $w_n=1/4$.

If either vector has zero norm, the cosine ratio is mathematically undefined; assigning zero to that order is an implementation convention. Exact string equality does not override that convention. Corpus CIDEr estimates IDF once from a fixed image set and then takes the arithmetic mean of per-image scores computed with that shared IDF.

Perspective: On average, how closely does the candidate's TF-IDF $n$-gram vector agree by cosine with each reference vector for the same input?

Strength: Averaging over all references uses the complete reference set rather than selecting only a best reference. TF-IDF gives relatively more weight to $n$-grams that distinguish this input's references from boilerplate occurring across many evaluated inputs.

Weakness and boundary with surface matching: “Consensus” is a design interpretation that uses mean lexical similarity to a small per-case reference set as a proxy for human consensus. The equation does not directly count whether a majority of annotators used an $n$-gram, nor does it judge shared propositions, image fidelity, or grammaticality. If a candidate strongly matches only one of $K$ references, that high pairwise cosine contributes only $1/K$ to the average. IDF is corpus rarity by input, not semantic importance.

CIDEr-D is a benchmark-oriented variant that removes stemming and adds $n$-gram count clipping, a Gaussian penalty for candidate/reference length differences, and scaling by 10. The paper defines

where $\min$ is element-wise and $\sigma=6$. As in CIDEr, orders $n=1,\ldots,4$ are normally averaged uniformly afterward. CIDEr and CIDEr-D therefore differ in both preprocessing and numerical scale.

There is also an implementation distinction. Equation (1) in the paper writes normalized TF, whereas the COCO Caption Evaluation reference code uses raw count times IDF. Per-order scaling cancels in ordinary cosine CIDEr, but an independent reimplementation need not remain equivalent once CIDEr-D clipping is introduced. The reference code under pycocoevalcap/cider implements clipping, the Gaussian penalty, and multiplication by 10 while returning the method name CIDEr. Consequently, “CIDEr” displayed by a COCO-style benchmark may in fact be the CIDEr-D-style computation.

The standard COCO pipeline runs Stanford PTBTokenizer before the scorer, lowercasing and removing a fixed list of punctuation tokens. It is not a Japanese segmenter. Calling Cider or CiderScorer directly bypasses that step: the scorer itself only calls a simple split() on an already whitespace-tokenized string and performs no Unicode normalization. The same reference code computes image-level document frequencies from the references passed for the evaluated set. Instantiating a separate scorer for one image makes every observed $n$-gram IDF $\log(1/1)=0$, so even an exact match scores zero. The wrapper requires at least one reference entry per image; an empty candidate against ordinary nonempty references scores zero, but a reference corpus containing no $n$-grams at all can fail its document-frequency check. Replacing PTBTokenizer for Japanese creates a variant distinct from the standard COCO score. Pin the pipeline, reference set, tokenizer, and code commit together.

Caution: IDF depends on the evaluation corpus and its references, so CIDEr values from separate datasets are not directly comparable. If references are few, dependent, or unrepresentative of the annotator population, interpreting their mean agreement as broad “human consensus” is weakly supported.

8. Apply Multiple Metrics to the Same Difference

8.1 Two outputs that differ by one token

A small token example makes the denominators visible.

Reference: refunds within thirty days
Output A:  refunds within thirty
Output B:  refunds within sixty days

The following values assume case-sensitive whitespace tokens and one reference. Token edit rate is unit-cost $d_{\mathrm{Lev}}(r,h)/|r|$; BLEU-1 is the unsmoothed corpus equation with this one case as the whole corpus. The reference has four tokens, and both outputs match three.

Output B, however, changes a 30-day deadline to 60 days, a major semantic error. Edit rate ties A and B, and BLEU-1 ranks B above A. Both results are computed correctly: none of these definitions represents semantic importance.

8.2 Duplicate counts and word order

A multiset that retains duplicate tokens differs from a set that discards them.

Reference: red red blue
Output:    red blue blue

One red and one blue align, so $M=2$.

Set Jaccard converts both sides to {red, blue} and therefore returns one; its definition discards frequency.

Now reverse order:

Reference: dog bites man
Output:    man bites dog

Bag-of-token metrics give full credit despite the role reversal. Sequence metrics penalize the reordering but still cannot tell which meaning is correct.

8.3 Exact matches, short texts, and empty inputs

In this table, “nonempty” means that metric units remain after preprocessing. Equations use a 0–1 scale; rows naming sacreBLEU use that implementation's 0–100 scale.

Because CIDEr averages all references, matching only one reference exactly does not guarantee the maximum. Exact equality can also score zero when IDF makes every vector component zero, as when the COCO scorer is built from a single image (Section 7).

A short exact match does not create missing higher-order $n$-grams:

Reference: a b
Output:    a b

For BLEU-4, both numerator and denominator of $p_3$ and $p_4$ are zero. In sacreBLEU 2.6.0, BLEU().sentence_score returns zero with its default effective_order=False, whereas BLEU(effective_order=True).sentence_score uses only the existing unigram and bigram orders and returns 100. Likewise, rouge-score ROUGE-3 returns zero for this exact match under its empty-trigram-bag convention.

When only one side is empty, keep denominator direction separate from implementation conventions.

8.4 Multiple references and error rates above 100%

Because BLEU forms a maximum reference count for each $n$-gram, it can combine matches from different references.

Reference 1: a b x
Reference 2: y b c
Output:      a b c

The output is not identical to either reference. However, a b occurs in Reference 1, b c occurs in Reference 2, and every unigram occurs in at least one reference. BLEU-2 is therefore $p_1=3/3=1$, $p_2=2/2=1$, and $BP=1$, giving a score of one. This is the intended behavior of multi-reference clipping.

Error rates can exceed 100% even with a nonempty reference:

Reference: a
Output:    a x y

Reference-to-output WER alignment has $S=0,D=0,I=2$, so $WER=2/1=2$, or 200%. TER is also 200% here, but it edits the hypothesis toward the reference: deleting the extra x y costs two edits, divided by one reference token. For the character strings a and axy, CER follows WER and counts two insertions, also giving 200%.

In contrast, with a completely empty reference and Output x y, the ordinary ratio is undefined. JiWER 4.0.0 returns the insertion count itself, so WER is 2, while sacreBLEU 2.6.0 TER returns 100 under its special “all references empty but a hypothesis is nonempty” convention. Do not confuse this special value with the ordinary 200% calculation against a nonempty reference.

8.5 Counterexamples built from the definitions: How high scores can be gamed

The counterexamples below follow the definitions exactly. They show the false positives and false negatives that arise when the surface agreement a metric measures is read as task quality. Clipping, F-scores, BP, and fragmentation penalties each close specific loopholes, but none of them establishes meaning, factuality, or task success.

Reference: alpha beta
Output:    alpha beta beta beta

The multiset match is $M=2$. Token recall and ROUGE-1 recall are $1$, and set Jaccard, which discards duplicates, also stays at $1$. Token precision, however, is $1/2$, token $F_1$ is $2/3$, and multiset Jaccard is $1/2$. Clipping gives the extra beta no additional match credit, but neither does it lower a recall-only score or a score computed after set conversion.

Reference: a b c
Output:    a x b y c

Even when x and y are unsupported insertions, token recall, ROUGE-1 recall, and ROUGE-L recall are all $1$. ROUGE-L recovers a b c as a non-contiguous subsequence, so precision is $3/5$ and $F_1$ is $3/4$. In general, inserting arbitrary strings while keeping the whole reference in order never lowers ROUGE-L recall. ROUGE-N recall likewise does not fall under extra repetition or appended text, as long as the candidate contains every $n$-gram of the reference.

Fixed-width local order can be gamed by moving long blocks globally. Let blocks $A$ and $B$ each have length $q$ and consist of mutually distinct tokens, and let the reference be $AB$ and the hypothesis $BA$. Order inside each block is preserved and the vocabulary matches exactly.

• Token multiset P/R/$F_1$ and multiset Jaccard are $1$.
• For token $n$-grams with $1\le n\le q$, in the simple case where the $n$-grams straddling the boundary do not match each other, BLEU's $p_n$ and single-reference ROUGE-N recall are both $2(q-n+1)/(2q-n+1)$. Only the neighborhood of the boundary differs, so for a fixed maximum order this approaches $1$ as $q\to\infty$. chrF at a fixed order has the same locality, because it keeps most character $n$-grams inside the long blocks.
• In METEOR all $2q$ words align and there are $2$ chunks. With the 2005/NLTK defaults $P=R=1$, so the score is $1-\frac{1}{2q^3}$ — already $0.9995$ at $q=10$. The fragmentation penalty looks only at $ch/m$, not at move distance or the semantic role of the blocks.
• TER needs a single shift of one block, so the edit rate is at most $1/(2q)$. This follows TER's operation set but reflects neither the semantic importance of the reordering nor real post-editing time.
• If the blocks have equal length and share no tokens, ROUGE-L $F_1$ is $1/2$. The normalized Kendall value of NTT RIBES 1.03.1 is $(q-1)/(2q-1)$, approaching $1/2$ as $q\to\infty$. In this example LCS and rank correlation reflect global order far more strongly than fixed-width $n$-grams or chunk counts.

Conversely, RIBES does not guarantee content adequacy either. In the same pinned implementation, inserting an equal number of unaligned tokens while keeping every reference token in order gives $NKT=1$, $P=1/2$, and $BP=1$, so with the defaults $RIBES=(1/2)^{0.25}\approx0.841$. Unmatched content enters only the unigram precision term, and its exponent is the version-pinned $0.25$.

The table maps the main false positives and false negatives back to each definition.

The alpha/beta for RIBES in the table are that implementation's exponent parameters. They are distinct from METEOR's identically named parameters and from $F_\beta$.

The defense that multiple references provide depends on the aggregation rule. BLEU in Section 8.4 uses the maximum count from a different reference for each $n$-gram, which permits a composition present in no single reference. The pinned implementations of rouge-score, chrF, METEOR, and RIBES take the maximum per reference, so they do not perform that composition, but neither do they require agreement with the reference set as a whole. rouge-score.score_multi may also select a different best reference for each ROUGE variant. CIDEr averages over all references, so copying a single reference verbatim is diluted to $1/K$ — but that does not check truth outside the references, or faithfulness to the input.

Choosing the preprocessing after inspecting the outputs is itself a way to game the metric. The pinned SQuAD 2.0 normalizer strips ASCII punctuation, so the product codes A-10 and A10 both become a10 and EM is $1$. Conversely, an implementation that skips NFC treats visually identical composed and combining strings as a mismatch. The ASCII tokenizer in rouge-score 0.1.2 turns Japanese into an empty sequence. Do not look at the candidates first and then pick lowercasing, stemming, punctuation removal, and the tokenizer, reporting only the favorable configuration. Pin $T$, the implementation, the version, and a hash of the transformed data before evaluating.

CIDEr's IDF is dataset-specific. If $g$ appears in the references of every input in the evaluation set, then $\operatorname{idf}_g=\log(|\mathcal I|/|\mathcal I|)=0$ and that $n$-gram contributes nothing even when it matches. Conversely, a rare typo or named entity appearing in a single input receives a large weight. The pinned COCO code recomputes document frequency from the self.crefs it is handed, so the same candidate–reference pair can yield a different value when the set of images scored alongside it changes. Do not compare a value obtained by rebuilding the scorer per subset against that subset's value from a whole-set scorer as if they were on the same scale.

An evaluator reading the references while scoring is not the same as reference leakage. Leakage is when test references that should stay sealed, reference-derived IDF or vocabulary, or repeated test-score feedback enter candidate generation, prompt or checkpoint selection, or preprocessing choices. Surface metrics cannot distinguish legitimate generation from a leaked copy. Keep test references private, tune only on development data, audit exact and near-duplicate matches across train/dev/test, and record submission counts and selection history.

Surface metrics accurately answer questions such as “how many symbols changed?” and “how much overlaps?” They cannot answer how operationally dangerous a difference is. That requires separate checks such as slot comparison, numeric and negation checks, entailment/contradiction evaluation, or a task-specific rubric.

9. Select Metrics by the Task's Failure Modes

Use the surface metrics in the following table as entry points to a fixed set of diagnostics, not as a standalone training reward, checkpoint-selection criterion, or acceptance condition. Apart from mandatory schema/EM conditions for strict output contracts, do not design the evaluation so that a high surface value can offset a separate critical failure.

Report metrics that detect different failures side by side rather than collapsing them into one score. Even if BLEU and chrF both rise, only the two composite scores are known to have risen; inspect BLEU's per-order precisions and BP and chrF's precision/recall components before attributing the change. If both rise while a semantic evaluation falls, the outputs may simply have moved closer to the references' wording on the surface.

Guard against Goodhart-style failures from metric optimization with the following sequence:

1. Define first the properties to preserve and the critical errors that no other value can offset. Treat negation, numbers, units, proper nouns, and safety prohibitions, for example, as such mandatory conditions.
2. Separate development data from sealed test data. Select prompts, checkpoints, preprocessing, weights, and thresholds on development data only, and do not feed test references, CIDEr IDF, or score feedback back into candidate generation. Record submission counts and selection history.
3. Fix the measurement before looking at outputs. Preregister $T$, the implementation commit, reference order, aggregation, primary metrics, subsets, and tolerances, and do not pick the most favorable value from several configurations.
4. Maintain perturbation tests. Build a challenge set with added repetition, reference fragments plus unrelated text, shortened and lengthened outputs, long-block swaps, synonymous paraphrases, substituted negations, numbers, and named entities, Unicode and tokenizer differences, and multi-reference chimeras, then check the expected score direction and the task's mandatory conditions separately.
5. Keep the optimization metric independent of the audit evaluation. If a surface metric serves as a training reward, detect reward gaming with non-overlapping references, a differently designed semantic/factual evaluation, and human audits, and do not release on a single overall number.

10. What Surface-Matching Metrics Commonly Miss

10.1 They mark valid paraphrases as false negatives

A reference is one sample from the set of correct answers. With few references, metrics penalize valid alternative wording, explanation order, and levels of detail. Multiple references mitigate this but cannot enumerate every valid realization.

10.2 They do not understand the importance of a small string difference

Negation, numbers, units, dates, named entities, and subject/object swaps may reverse meaning while producing a small surface distance. Conversely, a large paraphrase can preserve meaning despite a large surface distance.

10.3 They inherit reference errors and bias

Metrics assume references are correct. If a reference is stale, incomplete, stylistically narrow, or disputed between annotators, the score inherits that defect. Adding references does not by itself guarantee factual correctness.

10.4 They do not directly measure document-level logic or factuality

$n$-grams and LCS observe some word order, but not causal relations, coreference, discourse structure, cross-sentence contradictions, or agreement with external facts. A summary can achieve high ROUGE while introducing unsupported claims.

10.5 Averages hide critical slices

A corpus score can improve while performance declines for one language, long texts, named entities, or numerical cases. Inspect score distributions, length/domain slices, and worst cases in addition to the aggregate.

10.6 A small score delta is not automatically a quality improvement

Separate the levels of correlation. Correlation between an automatic metric and human judgments has at least two distinct forms.

The latter is also called corpus-level correlation because its automatic score is computed over a corpus, but its sample size is normally the number of systems, not the number of sentences. The two levels target different estimands; high correlation at one level does not imply high correlation at the other. High system-level correlation also does not guarantee calibrated quality probabilities or the correctness of any particular output. Report whether the statistic is Pearson, Spearman, or Kendall correlation, along with tie handling, weighting, and how systems and cases were pooled. If studying segment-level BLEU correlation, state that it is a sentence variant with fixed smoothing and effective order rather than original corpus BLEU, and do not call its average corpus BLEU.

Mathur et al. (2020) primarily reanalyzed Pearson correlations whose observations were WMT 2019 systems. They showed that correlations among a small set of similarly performing systems can be unstable and that human-score outliers can substantially inflate correlations or change conclusions. They also compared system pairs against human Direct Assessment on the same data and reported that differences of roughly 1–2 BLEU corresponded to a genuine human-judged improvement only about half the time. This result is conditional on their WMT 2019 data, system set, human judgments, and testing procedure; it does not justify a universal one- or two-BLEU threshold.

Separate an interval for one system from a difference between two systems. For a higher-is-better metric, write the scores on the same fixed evaluation set as $M_A,M_B$ and the difference as

For lower-is-better metrics such as WER and TER, either flip the sign so an improvement is positive or state the direction explicitly. A bootstrap confidence interval for a single system estimates the uncertainty in $M_A$ itself under case resampling. What a comparison needs is a paired distribution, confidence interval, or null-hypothesis test for $\Delta$ on the same cases. Deciding a difference solely by whether the separate 95% intervals of $M_A$ and $M_B$ overlap is not equivalent to a paired comparison. A non-significant result also does not establish equivalence; there may be too little data to detect a difference of that size.

Ordinary case bootstrap captures the uncertainty of redrawing evaluation cases from a target population, holding system outputs, references, preprocessing, and the metric fixed. It does not include variation from training seeds, stochastic decoding, reference creation, annotators, or domain shift unless you also rerun or hierarchically resample those by design. It also cannot make an unrepresentative evaluation set representative.

Match the resampling unit to the independently sampled unit. If the evaluation is one independent source item per line, case-level bootstrap resamples case indices with replacement, and paired bootstrap applies the same index sequence to both A and B. But when several lines come from one document, dialogue, speaker, user, or thread, those lines are generally not independent cases. Treat the document or dialogue as a cluster and resample whole clusters with replacement. Paired approximate randomization likewise swaps the A/B labels for entire clusters under the exchangeability assumption. Because the number of clusters is the effective sample size, intervals and tests stay unstable with many sentences but only a few documents. Decide in advance whether clusters become equally weighted macro units or whether all cases inside a selected cluster feed the aggregated corpus score.

Recompute the same aggregation formula on every iteration. If the reported target is a macro average, average the resampled unit scores. If it is a micro ratio such as WER, aggregate the resampled edit counts and reference lengths and then divide. For nonlinear corpus metrics such as BLEU, aggregated chrF, or corpus METEOR, sum the sufficient statistics of the resampled cases and apply the metric's corpus formula from scratch. Computing sentence BLEU and averaging it would test a different estimand from the point estimate.

Two paired procedures are standard:

• Paired bootstrap: resample the same independent units or clusters with replacement, evaluate both A and B on them, and form $\Delta^*$ per iteration. Use it for interval estimation of $\Delta$ or for a specified bootstrap test.
• Paired approximate randomization: assume A and B are exchangeable under the null hypothesis of no difference, randomly swap the A/B outputs of each independent unit or cluster, and recompute the corpus score difference. The p-value follows from how often the recomputed difference is at least as extreme as the observed one.

Both must preserve the pairing and the aggregation formula. A p-value is the probability of a statistic at least as extreme as the observed one under the null hypothesis and the procedure — not the probability that the null is true, not the size of the improvement, and not the probability that human quality improved. Fix one- versus two-sided testing, the iteration count, the seed, and how ties are counted.

Separate statistical significance from practical effect size. Expressed in metric units under a fixed configuration, $\Delta$ is an unstandardized effect size on that metric's scale. A small p-value can accompany a $\Delta$ that is operationally negligible, and a non-significant difference with a wide interval does not rule out a useful improvement. When the estimand is corpus BLEU, do not build Cohen's $d$ from an arbitrary sentence-BLEU distribution and present it as a corpus effect size. For an additive per-case metric you can add paired case differences; for binary human preference, win/loss/tie counts or a preference rate.

There is no universal conversion for what “0.3 BLEU” means in practice. Change the tokenizer, references, test set, or the range of systems compared and the same difference means something else, and a difference on a 0–100 scale cannot be read as a simple “percent improvement.” Fix the primary metric, its direction, and the smallest practically important difference $\varepsilon$ before looking at results, and calibrate externally against same-task human preference, critical-error rate, or post-editing time. Report $\Delta$, an interval for $\Delta$, the raw scores, and diagnostic subsets together.

Trying many systems, metrics, languages, and subsets and then keeping only the small p-values inflates false positives. Define in advance which comparisons form one test family; use Holm to control the family-wise error rate, or Benjamini–Hochberg when controlling the false discovery rate matches your exploratory goal and assumptions. Report uncorrected and corrected p-values and the number of comparisons corrected for.

What sacreBLEU 2.6.0 provides:

The random seed defaults to 12345 and can be changed via SACREBLEU_SEED. These procedures treat each input line as an independent unit; they do not read docid to build document or dialogue clusters automatically, and they do not correct the resulting p-values for multiple comparisons across systems or metrics. Dependent lines therefore require a separate cluster-aware implementation.

In practice: (1) predefine the independent unit or cluster, the primary metric, the aggregation formula, the direction, and $\varepsilon$; (2) compute $M_A,M_B,\Delta$ and their breakdowns on the same fixed test set; (3) recompute the same corpus statistics while preserving the pairing; (4) report $\Delta$ with its interval, the test used, the p-value, the iteration count, and the seed; (5) correct for multiple comparisons; and (6) if you claim a “quality improvement,” confirm it with task-appropriate human evaluation or an external outcome metric.

11. Checklist for Reproducible Evaluation

Store at least the following with every result.

1. Data: dataset name, version, split, reference provenance, case order, the independent sampling unit, and cluster IDs such as document or dialogue.
2. References: count, treatment of variable references, and multi-reference rule.
3. Preprocessing: Unicode normalization, casing, punctuation, whitespace, tokenizer, dictionary, and stemming.
4. Metric specification: name plus variant—for example, ROUGE-L versus Lsum, CIDEr versus CIDEr-D, or Google GLEU versus GEC GLEU+.
5. Parameters: $n$-gram orders, weights, $\beta$, length penalties, smoothing, effective order, reference-sampling iterations, and random seed.
6. Implementation: library, version, and metric signature when available.
7. Aggregation: sentence/corpus, micro/macro, sufficient statistics and final formula, and conventions for zero counts and empty strings.
8. Scale: $0$–$1$, $0$–$10$, $0$–$100$, or another range, and whether higher or lower is better.
9. Uncertainty and comparison: the estimand, whether it is a single score or a difference $\Delta$, the interval method, the independent unit or cluster, the paired test, one- or two-sided testing, iteration count, seed, $\alpha$, the test family and correction method for multiple comparisons, and the smallest practically important difference $\varepsilon$.
10. Diagnostics: overall score plus length, language, domain, and critical-entity slices.
11. Optimization boundary and leakage: metrics used for rewards or model selection, the provenance of train/dev/test references, duplicate audits, how many times test scores were viewed, and submission and selection history. For CIDEr, also store the input set from which IDF was built.
12. Adversarial challenge set: expected behavior under repetition, reference fragments plus unrelated text, overly short and long outputs, long-block swaps, synonyms, negations, numbers, units, proper nouns, multi-reference chimeras, and Unicode and tokenizer differences.

SacreBLEU 2.6.0 signatures have metric-specific fields. Preserve nrefs, case, eff, tok, smooth, and version for BLEU; nrefs, case, eff, nc, nw, space, and version for chrF; and nrefs, case, tok, norm, punct, asian, and version for TER. chrF $\beta$ appears in the metric name, such as chrF2. External Unicode normalization, a changed BLEU maximum order, and an external segmenter or dictionary are not in the signature, so separately preserve a hash of transformed evaluation data and a dependency lock. For ROUGE, similarly fix the tokenizer, stemmer, rougeL versus rougeLsum, split_summaries, multiple-reference rule, and aggregation.

When using the statistical features, also store the complete signature including the bootstrap or approximate-randomization iteration count and seed that sacreBLEU adds, the CLI options, and the input order. The signature does not record the cluster design, the multiple-comparison test family, $\varepsilon$, or an externally computed interval for $\Delta$, so keep those in a separate evaluation record.

Conclusion: Read What the Numerator and Denominator Actually Measure

Once inputs, preprocessing, and settings are fixed, the core scores of most surface-matching metrics are inexpensive, fast, deterministic, and auditable, so they remain useful for large regression suites and failure analysis. Procedures that involve reference sampling or bootstrap resampling also need a fixed RNG and seed to stay deterministic. Each number, however, means only the following:

• Exact Match asks whether the output obeys an exact contract.
• Token F1, Jaccard, and Dice ask how much the elements overlap.
• WER, CER, and TER ask how many defined edit operations are needed per reference unit.
• BLEU asks whether local hypothesis $n$-grams also occur in a reference and whether total hypothesis length falls short of the effective reference length.
• chrF asks how much character material is shared from precision and recall perspectives.
• ROUGE asks how much of the reference's phrases or ordered subsequences is covered.
• RIBES asks how closely the global relative order of matched words follows the reference.
• METEOR asks how much lexical material aligns and remains grouped in order.
• CIDEr asks how closely input-distinctive $n$-grams agree on average with the reference set.

Good evaluation design starts from the properties a task must preserve, uses corresponding surface metrics as cheap diagnostic instruments, and measures meaning, factuality, safety, and human acceptability separately.

References

1. Левенштейн, В. И. (1965). «Двоичные коды с исправлением выпадений, вставок и замещений символов». Доклады АН СССР, 163(4), 845–848. English translation: Levenshtein, V. I. (1966), “Binary Codes Capable of Correcting Deletions, Insertions, and Reversals,” Soviet Physics Doklady, 10(8), 707–710.
2. Rajpurkar, P., Zhang, J., Lopyrev, K., & Liang, P. (2016). SQuAD: 100,000+ Questions for Machine Comprehension of Text. Its primary metrics include EM and token F1.
3. Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2002). Bleu: a Method for Automatic Evaluation of Machine Translation.
4. Chen, B., & Cherry, C. (2014). A Systematic Comparison of Smoothing Techniques for Sentence-Level BLEU.
5. Post, M. (2018). A Call for Clarity in Reporting BLEU Scores.
6. Popović, M. (2015). chrF: character n-gram F-score for automatic MT evaluation.
7. Popović, M. (2016). chrF deconstructed: beta parameters and n-gram weights.
8. Popović, M. (2017). chrF++: words helping character n-grams.
9. Lin, C.-Y. (2004). ROUGE: A Package for Automatic Evaluation of Summaries.
10. Banerjee, S., & Lavie, A. (2005). METEOR: An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments.
11. Denkowski, M., & Lavie, A. (2014). Meteor Universal: Language Specific Translation Evaluation for Any Target Language.
12. Snover, M., Dorr, B., Schwartz, R., Micciulla, L., & Makhoul, J. (2006). A Study of Translation Edit Rate with Targeted Human Annotation.
13. Vedantam, R., Zitnick, C. L., & Parikh, D. (2015). CIDEr: Consensus-Based Image Description Evaluation (arXiv v2, 2015-06-03; v1 was first submitted 2014-11-20).
14. Mathur, N., Baldwin, T., & Cohn, T. (2020). Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics — reanalyzes WMT 2019 system-level Pearson correlations, the effect of outliers, and how metric differences between system pairs correspond to human judgments.
15. sacreBLEU 2.6.0, annotated-tag target 2277caccfc7b956671a6a09f1646f62250034157 — both the tag and PyPI upload are dated 2026-01-12 UTC; defaults, tokenizers, empty-input behavior, and metric signatures for BLEU, chrF, and TER, plus the --confidence, paired-bootstrap, and paired approximate-randomization implementations.
16. rouge-score 0.1.2 from the Google Research repository — released 2022-07-22. Pin the sdist with SHA-256 c7d4da2683e68c9abf0135ef915d63a46643666f848e558a1b9f7ead17ff0f04; distinguish the same-day later commit 910758241597e44871c7029654f5ed7813dc3344, which first records version 0.1.2, as a nearby repository snapshot. It implements ROUGE-N, ROUGE-L/Lsum, a default ASCII tokenizer, stemming, and bootstrap, while its README disclaims official Google-product status.
17. Popović's chrF++.py, commit e96ea868b7451a006d281d8ed51b1b163dd8384b — that file's sole revision, added 2017-06-14; per-order F, multi-reference behavior, and document/macro aggregation.
18. Meteor 1.5 documentation (official CMU source commit 57a9ec5d5f65b8ceca6db5788ef876f3726614b3) — modules, parameters, sufficient statistics, and system-level evaluation; some embedded README examples retain old 1.4 labels.
19. NLTK 3.10.0 METEOR, tag commit bd49f9011d7dc8c6a36b3c4ae71f04060c9b3fb9 — released 2026-07-08; pre-tokenized input, default resources and parameters, multi-reference selection, and empty-input behavior.
20. JiWER 4.0.0, tag commit 19e88eae1815e7ca40db9b45b14af5f1f9d8b6ce — release commit dated 2025-02-03 and PyPI publication dated 2025-06-19; WER/CER operation counts, micro aggregation, default transforms, and the empty-reference conventions introduced in 4.0.
21. COCO Caption Evaluation, commit 3a9afb2682141a03e1cdc02b0df6770d2c884f6f — pinned 2018-04-23 snapshot containing PTBTokenizer and the COCO CIDEr-D-style scorer ($\sigma=6$, clipping, factor 10, and per-image averaging); this is not the definition year cited for CIDEr/CIDEr-D.
22. Unicode Consortium. Unicode Standard Annex #15: Unicode Normalization Forms.
23. Official SQuAD Explorer evaluators at pinned commit 09eac9971f46889fa057ff2c870bf71092ba9d55: browser evaluator with the shared normalizer, max-over-references, and macro aggregation, plus the SQuAD 2.0 Python script, which also implements empty-answer handling.
24. Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2020). BERTScore: Evaluating Text Generation with BERT (arXiv v1 in 2019; ICLR 2020 OpenReview) — contextual-embedding soft matching against reference tokens.
25. Rei, R., Stewart, C., Farinha, A. C., & Lavie, A. (2020). COMET: A Neural Framework for MT Evaluation — learned MT evaluation using the source and a target-language reference.
26. Rei, R., et al. (2022). CometKiwi: IST-Unbabel 2022 Submission for the Quality Estimation Shared Task — a quality-estimation family that does not require a reference at inference time.
27. Doddington, G. (2002). Automatic Evaluation of Machine Translation Quality Using N-gram Co-Occurrence Statistics. Proceedings of the Second International Conference on Human Language Technology Research, 138–145 — primary NIST reference. This is an MT Archive PDF mirrored under the ACL Anthology site, not a regular Anthology record with an ACL ID.
28. Isozaki, H., Hirao, T., Duh, K., Sudoh, K., & Tsukada, H. (2010). Automatic Evaluation of Translation Quality for Distant Language Pairs — the primary source for RIBES.
29. Hirao, T., Isozaki, H., Sudoh, K., Duh, K., Tsukada, H., & Nagata, M. (2014). Evaluating Translation Quality with Word Order Correlations, Journal of Natural Language Processing 21(3), 421–444 — later RIBES formulation. Implementation defaults were checked in NTT RIBES 1.03.1, commit 7e76cf24d32de70e9e6afd0c9b95d8c140b7235c.
30. Wu, Y., et al. (2016). Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation — the primary source for sentence-level Google GLEU.
31. Napoles, C., Sakaguchi, K., Post, M., & Tetreault, J. (2015). Ground Truth for Grammatical Error Correction Metrics — the original GEC GLEU.
32. Napoles, C., Sakaguchi, K., Post, M., & Tetreault, J. (2016). GLEU Without Tuning — GLEU+ and the recommendation to replace the original form. The post-paper BP fix, 500 iterations, and fixed seed schedule were checked in the authors' final code commit 50b5032a4ef2444b9381fb47a55b3bac0654a6d7.
33. Mutton, A., Dras, M., Wan, S., & Dale, R. (2007). GLEU: Automatic Evaluation of Sentence-Level Fluency — the primary source for the reference-free, parser-feature/SVM fluency metric.
34. NIST. SCTK 2.4.12 sclite, v2.4.12 tag commit 62aabfa95938bfea284f8b68a374c7ddcb0fce52 — version-pinned 2021-10-28 toolkit material for ASR alignment, $S,D,I$ counts, the reference-word denominator, and system summaries. Later commit 9688a268… retained the same README version label but was a 2023 documentation update, not the release tag itself.
35. Snover, M., Wang, S., & Matsoukas, S. TERCOM 0.10.0, commit e413d077e5b9ecc96b0e7de5fe6015eb135c2f6d — post-2006-paper Java implementation and its search, preprocessing, beam, and shift defaults.
36. Koehn, P. (2004). Statistical Significance Tests for Machine Translation Evaluation — the classic sentence-resampling paired-bootstrap procedure for corpus BLEU.
37. Riezler, S., & Maxwell, J. T. III. (2005). On Some Pitfalls in Automatic Evaluation and Significance Testing for MT — approximate randomization and testing cautions for MT comparisons.
38. Graham, Y., Mathur, N., & Baldwin, T. (2014). Randomized Significance Tests in Machine Translation — an empirical comparison of randomized tests for MT.
39. Dror, R., Baumer, G., Shlomov, S., & Reichart, R. (2018). The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing — paired-test choices and assumptions for NLP metrics and designs.
40. Holm, S. (1979). A Simple Sequentially Rejective Multiple Test Procedure. Scandinavian Journal of Statistics, 6(2), 65–70 — a sequential procedure controlling family-wise error.
41. Benjamini, Y., & Hochberg, Y. (1995). Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society: Series B, 57(1), 289–300.