Self-Consistency (LLM Evaluation)

What it is

Self-consistency is a quantitative metric used to evaluate how reliably a language model produces the same output when given the same input multiple times. It measures the degree of agreement between independent outputs generated for an identical prompt, without any change to the model or its inputs between runs. A model that produces highly similar outputs across repeated runs is considered self-consistent; a model whose outputs vary significantly is considered inconsistent, which is typically interpreted as a sign of uncertainty, insufficient training data for the input domain, or susceptibility to the stochastic nature of token sampling.

How it works

To calculate self-consistency, the same input is passed to a model n times under identical conditions, producing n independent outputs. The metric is then computed as the average pairwise similarity across all unique pairs of outputs. Formally, this is expressed as the sum of similarity scores S(G_i, G_j) for all pairs i and j, divided by the total number of pairs n(n-1)/2, where S is a task-appropriate similarity function. For document parsing tasks, S is typically defined by an established benchmark metric such as the OmniDocBench edit-distance measure. High self-consistency indicates the model has stable, confident representations of the input content. Low self-consistency suggests the model is uncertain – often because it has had limited exposure to the specific language, script, or document type during training – and is essentially guessing differently each time.

What it is used for

Self-consistency is used as a diagnostic tool to identify where a model is likely to underperform before committing to a full-scale evaluation. In multilingual parsing research, it provides a language-agnostic proxy for model confidence: rather than needing native-speaker annotation to assess output quality for every language, developers can measure how much a model agrees with itself to identify problematic languages or document types. It is also used as a data quality filter during training data generation – samples on which a model shows low self-consistency can be excluded from fine-tuning sets to avoid training on uncertain or hallucinated outputs. More broadly, self-consistency analysis can guide decisions about when to apply preprocessing techniques, when to use a different model, or when human review is warranted before relying on model outputs in production systems.