How to Evaluate Conversation UX: A MECE Framework for Agent–Human Dialogue Across Structure, Correctness, and Goodness

Introduction: "Model intelligence" doesn't measure conversation quality

When we talk about evaluating LLMs, benchmarks come to mind first. MMLU, GSM8K, SWE-bench: there are plenty of yardsticks for how "smart" a model is. Yet once you put a smart model behind a conversational agent and ship it to real users, you run into differences in conversation quality that benchmarks cannot explain.

A high-accuracy model can still break in ways like these:

• looping, asking the same thing over and over
• answering something subtly off from the question, at excessive length
• confidently stating a fabricated return policy
• being factually correct yet somehow cold and condescending

Conversely, products built on a modestly capable model can deliver brisk, on-point, pleasant conversations. Conversation UX needs to be evaluated separately from model intelligence.

To measure conversation UX systematically rather than by gut feeling, this article takes up three questions:

1. How do you design the evaluation axes as a MECE (mutually exclusive, collectively exhaustive) rule set?
2. How do you measure each one, separating what is deterministically measurable from what is not?
3. How does the weighting of the axes change between text chat and voice?

The short answer: conversation UX is captured by three quality axes plus one constraint axis.

The first three (S, C, G) are mutually separable dimensions of conversation quality; together they cover what people usually mean by a good or bad conversation. But an LLM agent takes actions and affects third parties, so covering its defects MECE requires a fourth axis of a different kind, Safety, which behaves like a constraint rather than a quality.

Below, we break the four axes into sub-rules and arrange each rule by how deterministically it can be measured (the map of determinism).

I. The axis model: Structure / Correctness / Goodness (+ Safety)

Why these axes?

The defects that can occur in a conversation fall into four questions of different character. The first three ask about conversation quality; the last asks about acceptability.

• "Did a conversation even take place as a conversation?" (Structure) — Do turns mesh, is there a reply, does it avoid loops, is the format intact? You can judge this without understanding meaning, from logs and timestamps alone.
• "Is the content right?" (Correctness) — Does it match the facts, address the question, stay consistent, accomplish the task? You cannot judge this without understanding meaning.
• "Was the experience good?" (Goodness) — Was it helpful, was the tone appropriate, did it feel natural, was the user satisfied? You cannot judge this without measuring subjectivity.
• "Is the response harmless and acceptable at all?" (Safety) — Is it free of toxic/unfair content, does it avoid leaking PII, does it resist injection and jailbreaks? This is a constraint that can fail independently even when the other three are perfect.

The first three map onto the separable dimensions of form, content, and experience. The figure below shows the four axes and their sub-rules.

Why make Safety its own axis?

A standard frame in LLM-agent evaluation is HHH (Helpful, Honest, Harmless; Askell et al., 2021). Of these, Helpful lives in Goodness (G1 helpfulness) and Honest lives in Correctness (C1 factual accuracy). But Harmless fits nowhere.

A response that is fluent (S✓), factually accurate (C✓), and friendly in tone (G✓) can still leak a third party's PII, give dangerous instructions, or be hijacked by prompt injection. Toxicity is not a "tone" problem and a jailbreak is not a "relevance" problem, so forcing them into Goodness or Correctness does not fit. Safety therefore gets its own independent constraint axis, with sub-rules H1 harm avoidance, H2 fairness, H3 privacy, H4 injection resistance, and H5 action safety (caution on irreversible actions, while not over-refusing benign requests).

We call Safety a "veto axis" because no matter how high the other three score, an unsafe response is unacceptable on that basis alone. It behaves like a zero factor in a product, not a term in a sum. When we say Safety is "orthogonal to quality," we mean it is a different kind of dimension, not that it is statistically independent. That is distinct from the "separable but positively correlated" relationship among S, C, and G described below.

The axes are separate questions (separable, but not independent)

These axes are separate questions, and each can be judged on its own. One axis can be perfect while another fails.

• S✓ C✗: a perfectly formatted JSON reply that confidently states a fabricated return policy. Structure perfect, content zero.
• S✓ C✓ G✗: form and content are both correct, yet the cold, condescending tone blames the user. Structure and content perfect, experience zero.
• S✓ C✓ G✓ H✗: well-formed, grounded in fact, friendly in tone, yet it leaks another person's PII. It passes all three quality axes while being unsafe.
• S✗: there is simply no reply, or the agent repeats the same utterance and loops.

Being separable does not make the axes statistically independent (orthogonal). In practice they are causally ordered and positively correlated: a failure on a lower axis tends to propagate upward. A hallucination (C✗) erodes trust (G), and a long silence (S✗) ruins the experience (G). This article therefore treats the axes as separately measurable, with failures propagating from the bottom up. Each axis is a separate net that catches defects that got through the previous net. Goodness, though, is a positive, graded quality rather than the absence of a defect, so its net is closer to grading how good a response is than to catching defects.

The interactive demo below breaks the same conversation in different ways and shows, each time, which axis degrades and which determinism band (Section II) can detect it.

The axes have an order (separable, but not equal)

The three quality axes are separable measurement dimensions, but in terms of user impact they are ordered.

Structure (substrate) → Correctness (value) → Goodness (felt quality)
  the conversation        the content is          only then can you
  must happen first       correct first           speak of "a good experience"

If there is no reply (S✗), neither correctness nor experience means anything. Structure is the substrate on which conversation UX rides. If the content is a lie (C✗), no amount of charm yields positive value; Correctness is the condition for value. Only when both hold can you speak of Goodness, the quality of the experience.

Safety sits outside this ordering as a veto axis. However well the quality layers line up, the moment H is violated the whole response becomes unacceptable (e.g., a perfectly polite, accurate response that leaks a single piece of PII).

We use this ordering again in Section VII to set priorities for the measurement pipeline (put the cheap nets in place first).

Being MECE: the rule of attribution

For a rule set to work as a taxonomy, it must be exhaustive (no gaps) and mutually exclusive (no overlaps). This framework gets there with the following attribution rule, a convention for avoiding double-counting.

Attribution rule (tie-break): attribute a defect to the cheapest layer at which it can be defined and detected. 0. A defect that is unacceptable on safety grounds (harm, unfairness, PII leak, jailbreak, dangerous action) → Safety (the veto wins even if every other axis passes)

1. Definable from transcript and logs without understanding meaning → Structure
2. Definable only by understanding meaning, fact, or task goal → Correctness
3. Definable only by measuring the subjective quality of experience → Goodness

This rule makes the classification exclusive. A "confident hallucination," for instance, is formally flawless and passes Structure, so it is classified uniquely as a Correctness failure. A "loop" can be defined as repetition without looking at meaning → Structure. A "cold tone" requires subjective judgment → Goodness. A "PII leak" is factually accurate yet unacceptable → Safety. Every defect lands in exactly one of the four axes.

Attribution depends on which rules exist in a given setting. If there is an explicit "be polite" policy, a rude reply could be a machine-checkable C5 (constraint) violation; without one, it is a G2 (tone) failure. So MECE works best in practice as a team-agreed classification convention, not a fixed truth.

Exhaustiveness comes from a four-way split (did it happen / is it right / was it good / is it safe): any observable defect in a conversation is either (a) a form/flow defect visible without understanding, (b) a meaning/truth/goal defect, (c) an experiential/affective defect, or (d) an unacceptable safety defect.

II. The map of determinism: separating measurable from not

To make evaluation systematic, we need a second coordinate besides the axes: how deterministically can it be measured? We split this into three bands.

• ① Deterministic (rule-checkable): compute with a fixed rule from logs and timestamps. No model is needed, and the same input always yields the same result. 100% reproducible, near-zero cost.
• ② Reference-based: compare to ground truth or a reference answer. Automatic, but needs gold data and a metric choice.
• ③ Judgmental: human evaluation or LLM-as-judge. Not bit-reproducible; quality is held by "correlation with humans" and "inter-rater agreement."

The three bands differ in cost by orders of magnitude, and they also differ in reproducibility and where they fit.

② is reproducible too: BLEU/WER/F1 against a fixed gold set is bit-for-bit reproducible, deterministic in the same sense as ①. Its weakness is reference coverage and validity: it breaks down when the gold is incomplete, admits multiple valid answers, or goes stale.

As a rule of thumb, the three quality axes line up diagonally with the determinism bands. Structure is mostly measurable in ①, Correctness spans ①②③, and Goodness centers on ③ (with ① proxies as support). Safety does not sit on this diagonal; it cuts across all bands (blocklists in ①, red-team sets in ②, classifiers/humans in ③). This leads to a layered strategy: gate Structure on everything, regression-test Correctness against references, and observe Goodness through sampled judgment.

III. The methodological foundation (a shared vocabulary)

Five design choices recur in evaluation design. Each can be made independently of the axes.

• Offline vs online: regression evaluation on a fixed dataset, or observation on production traffic.
• Turn-level vs dialogue-level: looking at one reply, or the whole conversation. Loops, consistency, and long-horizon degradation (→ IV-8) are only visible at the dialogue level.
• Reference-based vs reference-free: comparing to a gold answer, or measuring properties of the response alone.
• Pointwise vs pairwise: absolute scoring (1–5) or which of A/B is better. Pairwise tends to have higher inter-rater agreement.
• Who judges: rule / human / LLM-judge.

Prior work and where it fits

Conversation evaluation has nearly three decades of accumulated work. This framework can be read as placing those results on the map of axes and determinism.

• PARADISE (Walker et al., 1997, ACL): the classic for task-oriented dialogue. User satisfaction ≈ task success (κ) − dialogue costs (number of turns, elapsed time, number of repair utterances, etc.), with weights derived by linear regression. An early attempt to tie Correctness (task success) and Structure (costs) to a single satisfaction score (Goodness).
• USR (Mehri & Eskenazi, 2020, arXiv:2005.00456): reference-free open-domain dialogue evaluation. It measures several interpretable sub-metrics — understandability, naturalness, context maintenance, interestingness, knowledge use — in an unsupervised way. Turn-level correlation 0.42 on Topical-Chat, 0.48 on PersonaChat.
• FED (Mehri & Eskenazi, 2020, arXiv:2006.12719): using DialoGPT, it measures eighteen fine-grained dialogue qualities at the turn and dialogue level, reference-free. A pioneer in subdividing the structure of Goodness.
• DBDC (Dialogue Breakdown Detection Challenge) (Higashinaka et al., LREC 2016): annotators label each system utterance as NB (not a breakdown) / PB (possible breakdown) / B (breakdown), and the task is to predict, for each utterance, a single label and the distribution of labels. It formalized Structure (dialogue breakdown) as an independent detection problem.
• LLM-as-judge: MT-Bench (Zheng et al., 2023, arXiv:2306.05685) and G-Eval (Liu et al., 2023, arXiv:2303.16634). It can take over large volumes of judgments from humans, but it is known to carry position bias, verbosity bias, and self-enhancement bias, so calibration against human judgment is a prerequisite.

Caveat on LLM-as-judge: using an LLM as the judge greatly lowers the cost of ③, but its judgments are probabilistic, not deterministic. Always measure agreement with humans using a statistic that matches the scale (Cohen's κ for two nominal raters, Krippendorff's α for ordinal; see §VII), and counter position bias by swapping A/B and evaluating both directions. Having the judge emit a rationale before the score (CoT) makes its judgments more stable.

IV. Evaluating text chat

IV-1. Structure: guardable almost entirely, deterministically

Structure can be checked on everything, from logs alone, without understanding meaning. That makes it a "gate" you can run on every turn in production.

These can be written as deterministic code. For example, a Structure gate that checks responsiveness, format, and loops looks like this:

type Turn = { role: "user" | "agent"; text: string; ts: number };
 
// 3-gram Jaccard similarity between the last two agent turns (loop detection)
function trigramJaccard(a: string, b: string): number {
  const grams = (s: string) =>
    new Set(
      [...s].slice(0, -2).map((_, i) => s.slice(i, i + 3))
    );
  const A = grams(a), B = grams(b);
  const inter = [...A].filter((g) => B.has(g)).length;
  const union = new Set([...A, ...B]).size;
  return union === 0 ? 0 : inter / union;
}
 
function checkStructure(turns: Turn[], budgetMs: number) {
  const failures: string[] = [];
  const agent = turns.filter((t) => t.role === "agent");
  const last = agent.at(-1);
 
  // S2 responsiveness
  if (!last || last.text.trim() === "") failures.push("S2: empty/no response");
  const prevUser = turns.at(-2);
  if (last && prevUser && last.ts - prevUser.ts > budgetMs)
    failures.push("S2: latency over budget");
 
  // S3 format conformance (e.g., when JSON is required)
  // try { JSON.parse(last.text) } catch { failures.push("S3: invalid JSON") }
 
  // S4 loop detection
  if (agent.length >= 2) {
    const sim = trigramJaccard(agent.at(-1)!.text, agent.at(-2)!.text);
    if (sim > 0.9) failures.push(`S4: repeated turn (sim=${sim.toFixed(2)})`);
  }
 
  return { pass: failures.length === 0, failures };
}

Checks like these are cheap and can run on everything. Among quality failures, a Structure violation is the most severe signal ("the conversation broke"), so implement the Structure gate first, as a production guardrail.

Note: the "loop" in S4 includes not only surface repetition but also semantic loops, where the agent repeats the same thing in different words. What surface n-grams miss can be covered by embedding cosine similarity (leaning toward ② reference-based) or dialogue-level state tracking. This is the boundary between Structure and Correctness.

IV-2. Correctness: spanning ①②③

Correctness mixes parts you can measure deterministically with parts that need a reference or judgment.

For agents with RAG, Correctness is commonly decomposed into the "RAG triad" (a framing popularized by TruLens): (1) context relevance (is retrieval appropriate), (2) groundedness/faithfulness (is the response grounded in the retrieved context), (3) answer relevance (does the response answer the question). (1) is retrieval precision/recall (②); (2)(3) are measured by entailment or LLM-judge (②③).

C4, task success, moves close to deterministic, ② reference-based measurement when ground truth (a goal state or action) can be defined. For "book a flight," you can check against gold whether the tool was called with the right flight, passenger count, and date (multi-step trajectory evaluation is covered in IV-7). Part of C5 (format constraints, length, banned words) can be gated in ① deterministically.

C1 factuality and C2 relevance, on the other hand, have no single gold answer in open domains, so they need ③ judgment. If you use an LLM-judge here, have it emit "rationale → verdict" in that order, and measure agreement with humans.

C6 asks not whether the answer is right, but whether the model knows how confidently it may state its answer. Take the example from the introduction of confidently stating a fabricated return policy: the content is wrong (C1✗), and the model also failed to signal that it might be wrong. A calibrated model hedges ("I'm not certain, but…") or abstains ("I don't know / let me verify") when it lacks grounding, turning a C1 failure into an honest, recoverable one. Calibration measures the property HHH's Honest asks for, not asserting what you don't know, and complements C1. Metrics: ECE and Brier (Guo et al., 2017; Kadavath et al., 2022, "Language Models (Mostly) Know What They Know"), risk–coverage curves, and abstention accuracy on unanswerable questions.

IV-3. Goodness: centered on ③, supported by ① proxies

Goodness is subjective by nature, and it also depends on the audience: a terse answer is "good" for an expert but "bad" for a novice, so the same reply can flip in value. Measuring it directly means ③ (human evaluation), which doesn't scale. So we combine ① proxies with ③ sampled evaluation.

Proxies are convenient, but watch out for Goodhart's law. If you make answers vague to lower the re-ask rate, or cut necessary information to shrink sentence length, the proxy has become the optimization target and no longer reflects the quality it stood for. Use proxies for monitoring, and keep them separate from the real objective (CSAT, etc.).

To measure G2–G4 with an LLM-judge, an explicit-rubric evaluation prompt works well:

You are an evaluator of conversation quality. Score the agent reply below on Goodness.
Rate each item 1–5; state the rationale BEFORE the score.
 
[Tone & empathy G2] Appropriate register? Does it avoid blaming the user? Is there empathy?
[Clarity G3]        Clear structure? Too verbose / too sparse?
[Naturalness G4]    Natural rather than robotic?
 
Conversation context: {context}
Agent reply: {response}
 
Output (JSON): {"G2":{"rationale":"...","score":N}, "G3":{...}, "G4":{...}}

IV-4. Safety (text): the classifier-and-red-team layer

In text too, Safety is evaluated independently of the three quality axes, and the operating assumption is that you gate it on everything and block (an unsafe response must not ship, even if well-formed and accurate).

As H5 shows, safety is a balance of "refusing the dangerous" and "not refusing the benign." Over-refusal itself breaks UX, so erring entirely on the safe side is not enough. Red-team sets (known attacks) can be regression-monitored in ②, while the unknown attack surface is continuously probed by human/automated red-teaming (③).

IV-5. How far determinism reaches in text

In text chat, Structure is guardable almost entirely in ①, much of Correctness can be pushed into ②, and Goodness depends on ③. Safety is guarded by a whole-traffic gate (① regex and blocklists plus ③ classifiers), regression on known attacks (②), and continuous red-teaming (③). Determinism covers nearly all of Structure, the regex/blocklist and known-attack parts of Safety, and about half of Correctness; it barely reaches Goodness. Once you know where it reaches and where it stops, you can decide how much effort to put into whole-traffic gates, regression sets, and sampled evaluation.

IV-6. Repair & error recovery: the cross-axis "recovery" dimension

The rules so far classify defects, but real conversation UX also depends on how the agent recovers after a defect (repair). In conversation analysis, repair is a classic structural phenomenon (Schegloff, Jefferson & Sacks, 1977), and in dialogue evaluation FED's eighteen qualities include "error recovery." The final scene of this article's demo (correcting "90 days" → "30 days") is also an example of repair. Repair does not fit one axis; it spans all three quality axes:

• Detection (Structure-leaning): detect the user's correction/rephrase ("no, that's not what I meant") and whether the agent enters a repair sequence — partly capturable in ①.
• Correctness of the fix (Correctness): is the corrected content actually right (C1/C4)?
• Grace of the repair (Goodness): acknowledge the error, apologize appropriately, not over-apologize, and move forward (G2).

Repair is a good example of this article's view that the axes are separable but failures propagate from the bottom up. It can be added as a separate recovery dimension without breaking the defect taxonomy (MECE). Measure it via repair success rate (fraction of correction requests after which the error is resolved) and time-to-repair (turns to recover).

IV-7. Agentic tool use and multi-step task evaluation: widening C4 into the "trajectory"

This article is titled for agents, yet C4's "correct function call" alone is not enough to evaluate an agent that uses tools across a multi-step task. Agent evaluation extends beyond the final result to the trajectory that produced it. This too can be added without breaking the defect taxonomy — it runs across Correctness and Safety.

Evaluation should score the end-state and the trajectory separately. τ-bench (Yao et al., 2024, arXiv:2406.12045) scores an agent — given domain APIs and policies, conversing with an LLM-simulated user — by comparing the database state at the end of the conversation against the annotated goal state, and measures consistency across trials with pass^k (even SOTA function-calling agents like gpt-4o succeed on under 50% of tasks, with pass^8 under 25% in retail). The trajectory itself is approximated by exact / in-order / any-order match against a gold tool sequence, or precision/recall over tool calls.

Safety of irreversible actions (transfers, deletions, confirmed bookings) is caught by H5 action safety, not C4 task success — "the right action" and "a reversible action" are different questions. Monitor, via the whole-traffic gate and a regression set, whether the agent confirms before actions that warrant it, and whether it slips into over-refusal.

IV-8. Multi-turn / long-horizon degradation: measuring "it breaks as turns pile up"

§III distinguished turn-level from dialogue-level evaluation. Degradation that emerges as a conversation grows is invisible if you look at one turn at a time, so it must be measured separately, as a dialogue-level phenomenon.

• Goal drift (Correctness-leaning): the conversation slowly veers from its original objective across turns. Track sub-goal completion and assess alignment with the final goal per dialogue.
• Context / memory loss (Correctness/Structure): constraints fixed early (name, date, preference) are forgotten or garbled later. Measure the "retention rate of established facts" per dialogue.
• "Lost in the middle" (Correctness): a U-shaped effect where information placed in the middle of a long context is used far less than information at the beginning or end (Nelson F. Liu et al., 2023, "Lost in the Middle," arXiv:2307.03172). Probing accuracy with the relevant information at varying positions is an effective test.
• Multi-turn instruction decay (Correctness): an early instruction ("use formal language from now on") starts to be ignored as turns progress. Measure adherence as a function of turns elapsed since the instruction.
• Dialogue-level coherence/consistency (Goodness/Correctness): do persona, tone, and claims stay consistent across turns? Detect cross-turn self-contradiction.

Long-horizon degradation does not show up in the average of turn-level metrics. Your eval set therefore needs to include long conversations, scored at the dialogue level. Beware that many benchmarks skew toward short exchanges — MT-Bench (Zheng et al., 2023), for instance, uses a two-turn setup, putting long-horizon decay out of scope. In production, monitor the correlation between conversation length and error rate, and regression-test the effectiveness of summarization/memory mechanisms when needed.

V. Evaluating voice

Voice is text plus a dimension of time and acoustics. So evaluation becomes a two-story building: "text evaluation on the transcript (everything in the previous chapter) + voice-native sub-layers." And as V-5 shows, the weighting of the axes changes too.

V-1. Why "timing" dominates in voice: the cognitive science of human conversation

In human conversation, speakers hand over turns with a precision of a few hundred milliseconds. Work in conversation analysis since its classics (Sacks, Schegloff & Jefferson, 1974), together with later measurement (Levinson & Torreira, 2015, Frontiers in Psychology 6:731), has established the following:

• The gap between turns (floor transfer offset, FTO) averages about 200 ms, and its mode falls between 0 and 200 ms.
• Yet language production latency is about 600 ms for a single word (the picture-naming meta-analysis of Indefrey & Levelt, 2004). It rises to 740–900 ms for two- to three-word utterances (Schnur et al., 2006) and ~1500 ms for sentence-level utterances (Griffin & Bock, 2000; both as reviewed in Levinson & Torreira, 2015).
• The resolution of this contradiction is prediction (projection) — the listener predicts the end of the speaker's turn and begins planning their reply before the other has finished.
• A 10-language cross-linguistic study (Stivers et al., 2009, PNAS; measuring responses to polar [yes/no] questions, visual responses included) found mean response offsets ranging from ~7 ms (Japanese) to ~469 ms (Danish), but the tendency to minimize gaps and overlaps was universal.
• A gap longer than 700 ms is interpreted as a signal that "something is wrong" (a precursor to disagreement, trouble, or rejection). Beyond 300 ms, the probability of an unqualified acceptance starts to drop. (Both thresholds come from human–human dyads; applying them to agents is a working assumption.)

So humans achieve ~200 ms by "predicting ahead," whereas a naïve voice agent pipeline "waits for silence and then processes serially" — structurally violating this norm.

Stack silence-waiting (endpointing) → ASR finalize → LLM's first token → TTS's first audio serially, and it is common to exceed 1.5–2 seconds. That is 7.5–10× the human norm (~200 ms) and 2–3× the "problem signal" threshold (~700 ms). Even if the answer is correct (C✓), that silence alone breaks the experience (G✗). This is why Structure-timing dominates in voice.

Note (2024+ SOTA): this 1.5–2 s is a naïve serial-pipeline baseline; current systems are much faster. Native speech-to-speech models and full-duplex architectures have cut responses to ~300–500 ms, largely closing the gap to the human norm (~200 ms). The "wall of silence" is a design variable you move via architecture (streaming ASR, semantic endpointing, direct audio-token generation). Even so, p95 latency and barge-in behavior remain first-class evaluation targets.

V-2. Structure (voice): timing is deterministically measurable

Nearly everything about timing and voice flow can be measured deterministically from audio timestamps. Voice Structure adds, on top of the text S1–S5, the following voice-native rules.

These have the largest effect on voice UX and are also the most deterministic to measure, so a small amount of effort buys a large improvement. Common practice is to monitor the latency distribution (especially p95) as an SLO, and to tune the too-early/too-late trade-off of endpointing.

The endpointing dilemma: shorten the threshold and you "cut in while the user is still talking" (too early); lengthen it and "the silence stretches into a problem signal" (too late). Just as humans solve this with prediction, the latest voice systems try to move this deterministic trade-off via semantic endpointing (predicting the semantic completion of an utterance) or full-duplex (listening while speaking).

V-3. Correctness (voice): ASR and TTS inject error

Voice Correctness adds, on top of the text C1–C6, the accuracy of speech ↔ text conversion at the front and back ends.

• Front end: ASR accuracy. Did the agent hear correctly? The standard metric is WER (Word Error Rate) = (substitutions + deletions + insertions) / reference words. With a reference transcript, this is measurable deterministically in ② reference-based. ASR errors propagate straight into the downstream LLM (cascading error), so it's important to isolate ASR in end-to-end evaluation. Note that WER varies sharply with speaker accent/dialect, background noise, and code-switching, so cross-group WER gaps are also an H2 (fairness) concern.
• Back end: TTS pronunciation accuracy. Did it pronounce the correct content correctly? Numbers, proper nouns, dates, and units are especially error-prone ("1/2" as "one half" or "January 2nd"?). This is the classic "content is correct but the speech rendering breaks," which text evaluation cannot catch.

Even with a good ASR WER, breakage in punctuation, diarization, or proper nouns warps the LLM input. Voice Correctness should therefore measure the "ASR layer," "LLM layer," and "TTS layer" separately.

V-4. Goodness (voice): "naturalness," and timing as a proxy

Voice Goodness adds an acoustic experiential value absent in text.

In voice, the deterministic Structure metric (latency) becomes a good proxy for Goodness (felt tempo). Responses closer to the human norm of ~200 ms are likely to feel better paced and more pleasant. Voice naturalness itself is usually measured by MOS, a ③ judgmental metric: the ITU-T standard in which subjects rate 1–5 (the crowdsourced variant is P.808).

V-5. In voice, the axis weights change

In text, a slow reply is merely "a bit annoying" (mild Goodness degradation). In voice, a silence over 700 ms is actively interpreted as "something is wrong," so a Structure-timing failure becomes a Goodness failure directly.

Text:   Correctness and Goodness (content and tone) dominate the experience
Voice:  Structure-timing dominates the experience, and even
        correct content can be ruined by a single silence

In evaluating a voice agent, you need to treat the latency distribution (p95) as a first-class metric, weighting it at least as heavily as Correctness (WER + content). The quality axes are the same, but their weighting changes by channel.

VI. The rule set on a single page

Here the four axes × determinism bands are condensed into one table spanning text and voice. The table is itself a rule set you can build into an evaluation pipeline.

Columns further left are cheaper, cover everything, and run automatically; columns further right are more expensive, sampled, and subjective. Decide which band you primarily measure each axis in, and add evaluations starting from the left.

VII. Measurement in practice: a 3-layer pipeline

Turning the axes and the map of determinism into a measurement architecture gives three layers, one per determinism band. Cheap checks run on everything; expensive evaluations run on samples. Safety spans all three (block at the gate, monitor in regression, red-team in the sample).

• ① Gate layer: check Structure and Safety on everything, in production. A violation triggers an immediate alert, response block, or fallback. Structure is pure determinism; the Safety gate is a hybrid of regex/blocklists (deterministic) and classifiers (automated, probabilistic).
• ② Regression layer: evaluate Correctness and Safety with reference on a labeled regression set. Run it on every model update or prompt change to catch regressions (task-success drops, newly-successful jailbreaks).
• ③ Judgment layer: sample and evaluate Goodness and Safety with humans / LLM-judge. Full coverage is impossible, so use stratified sampling (weight failure cases and new domains heavily).

Offline and online

• Offline: a regression suite on fixed datasets (mostly ②). Also run the deterministic gate (①) and LLM-judge (③). Pre-release quality assurance.
• Online: on production traffic, run ① continuously as a guardrail and monitor ③'s proxies (drop-off, escalation, and for voice the latency p95) along with explicit user ratings (👍/👎). Anomaly detection wired to business metrics.

Principles for not breaking your measurement

1. Start with the cheap checks: prioritize Structure ①. Debating Goodness while missing no-replies and loops is pointless.
2. Safety is non-negotiable: Safety is a veto axis. Don't average it with the other scores; place it as a separate gate up front. An H violation blocks release even if everything else is perfect.
3. Measure layers separately: for voice, isolate ASR / LLM / TTS. Don't crush cascading error into one number.
4. Calibrate the LLM-judge before using it: measure agreement with humans (with a scale-appropriate statistic; see "Statistical rigor" below) and counter position bias (A/B swap).
5. Don't make proxies the optimization target: to avoid Goodhart's law, use proxies for monitoring and manage the real objective (CSAT, etc.) separately.
6. Use pairwise comparison: A/B comparison tends to have higher inter-rater agreement than absolute scoring. Choose models pairwise.

Statistical rigor: sample size, confidence intervals, agreement statistics

The numbers from the judgment layer (③) are random variables. Claiming "A is better" requires statistical backing.

• Sample size and power: for the effect size you want to detect, estimate the number of evaluations needed up front (power analysis). A "win/loss" on a handful of samples cannot be distinguished from chance.
• Confidence intervals and significance testing (A/B): report score differences with confidence intervals, not just point estimates. Test win-rate comparisons between two models for significance with bootstrap CIs or a binomial / McNemar test (the latter for paired comparisons). When comparing many metrics at once, apply a multiple-comparison correction (Bonferroni, etc.).
• The right agreement statistic for the situation: choose the inter-rater statistic by the nature of the data.
  • 2 raters, nominal → Cohen's κ (κ assumes strictly "two raters, nominal").
  • ≥3 raters, nominal → Fleiss' κ (generalizes to a fixed number of raters — strictly, it generalizes Scott's π rather than Cohen's κ).
  • Ordinal scales (1–5, etc.) → Krippendorff's α (any number of raters; handles ordinal/interval), weighted κ (2 raters; weights disagreements by distance), or Spearman's ρ (rank correlation). Running κ on a 1–5 scale as if it were nominal punishes a "4 vs 5" disagreement as harshly as a "1 vs 5" one.

Validating an LLM-judge follows the same distinction. For 1–5 rubric scoring, report human agreement with Krippendorff's α or Spearman; for A/B pairwise verdicts, report agreement rate (and κ). When §III and this chapter say "measure agreement with humans," they mean measuring it with the statistic that matches the scale.

Closing

Conversation UX is often discussed as "somehow good / somehow bad," but it can be broken down into three quality axes plus one constraint axis: whether the conversation happened as a conversation (Structure, guardable on everything, almost entirely deterministically), whether what was said is true (Correctness, much of which can be pushed toward reference-based measurement), whether the experience was good (Goodness, measured through proxies and sampled evaluation), and whether the response is harmless (Safety, a veto axis separate from quality, guarded by classifiers and red-teaming).

The attribution rule (attribute each defect to the cheapest layer) makes the four axes mutually exclusive, and the did it happen / is it right / was it good / is it safe split makes them exhaustive. The three quality axes are not statistically independent (orthogonal), though: they can be measured separately, but failures propagate from the bottom up. Overlaid on the map of determinism, they tell you what to gate on everything, what to regression-test, and what to judge on samples.

The channel also changes the axis weights. In text, content and tone shape the experience; in voice, timing does. Because humans hand over turns with ~200 ms gaps by predicting ahead, the bar for voice agents is high, and a long silence can spoil even a correct answer. Good conversation UX is the state in which the three quality axes line up, each weighted as the channel demands, without crossing the Safety line.