The Two MMLU Scores: What a Benchmark Name Does Not Fix

Date: September 6, 2026 · Author: Dmitrii Zatona  · Updated: September 29, 2026

TL;DR

• Two MMLU accuracies, 0.781 and 0.79, sit under one benchmark name in two illustrative fixture records; for a score-delta query the verifier returns incomparable (Sections 1 and 5).
• mmlu fixes a name; the split, implementation, prompt format, grader and network access stay open, and published measurements show several of them moving accuracy by whole points (Section 2).
• The three llama-65b scores from a June 2023 comparison cannot be written as complete frames, because their primary sources name no model revision, so the frames here are fixtures (Section 2).
• Comparability of results is a property of the reference they are traceable to, and a shared benchmark label does not supply that reference (Section 3).
• Under the APL AI-Eval profile a claim carries its frame’s hash, and scopes compare by canonical bytes, so subset: "all" and an omitted key differ (Section 4).
• apl-valid describes structure only: it does not show that either score is correct, that identical identifiers name identical runs, or that the vocabulary will stay stable (Section 6).

Two evaluation records appear in the same table. One reports mmlu accuracy 0.781 for build 42; the other reports 0.79 for build 44. The claims declare the same provider, model family, metric identifier, unit and benchmark name. The arithmetic difference is +0.009. Both records are structurally valid, and their frames declare different runners, graders and dataset splits: the shared mmlu label names a dataset family rather than a measurement procedure.

This article takes that pair apart into what the mmlu label leaves open and what the verifier in the apl-ai-eval crate outputs once those conditions are written into a hashed frame. Every frame and claim below is copied from the crate’s test vectors: both records are illustrative schema fixtures rather than historical observations, acme-gpt-7b names no real model, and neither number comes from an evaluation run. The verifier outputs were recorded by running the crate’s public verification functions against those vectors.

I write and maintain APL and its apl-ai-eval crate, whose outputs appear in Section 5. The account of what mmlu leaves open, in Sections 2 and 3, draws on published sources and is independent of that work.

1. What the two numbers say

Here is claim A as it exists on the wire, in the metadata.apl position of a log entry:

{"apl":{"version":"0.1","claim":{"kind":"observation","subject":{"type":"model-build","id":"model:acme-gpt-7b-build-42","build_id":"42","artifact_digest":"sha256:4242424242424242424242424242424242424242424242424242424242424242","provider":"acme","model_family":"acme-gpt-7b"},"aspect_refs":["accuracy"],"statement":{"predicate":"score","content":{"benchmark_id":"mmlu","metric_id":"accuracy","value":0.781,"unit":"fraction"}}},"frame_ref":{"hash":"sha256:c7b88426f2676f3653db0fad0bdbd689318f16d589d14a315bdd4cc454bca1ab"}}}

Claim B, two builds later:

{"apl":{"version":"0.1","claim":{"kind":"observation","subject":{"type":"model-build","id":"model:acme-gpt-7b-build-44","build_id":"44","artifact_digest":"sha256:4444444444444444444444444444444444444444444444444444444444444444","provider":"acme","model_family":"acme-gpt-7b"},"aspect_refs":["accuracy"],"statement":{"predicate":"score","content":{"benchmark_id":"mmlu","metric_id":"accuracy","value":0.79,"unit":"fraction"}}},"frame_ref":{"hash":"sha256:c93a9c55422ddbd2158a5336caa3a251641cf3937f451fb13387a5f54f0d998e"}}}

The subject differs, which is the point: two builds of one family. Each claim declares an artifact_digest, which the profile treats as the immutable identity anchor of the evaluated artifact. The statement is identical in shape and vocabulary: predicate score, benchmark mmlu, metric accuracy, unit fraction. Claim B writes the value as 0.79; a presentation may display that JSON value as 0.790, and under RFC 8785 canonical number serialization the trailing zero does not change the value.

The claim-level pointers differ: frame_ref.hash is c7b88426… in one record and c93a9c55… in the other. Resolving the two frames shows differences in both procedure and scope; the hash is the whole signal at the claim level.

The relation someone wants over this pair is also an object:

{"left_aspects":["accuracy"],"right_aspects":["accuracy"],"predicate":"score","relation_type":"score-delta"}

score-delta is a request to subtract, and it has an answer only under the conditions Sections 2 and 3 examine.

2. What “MMLU” does not fix

benchmark_id: mmlu fixes a name. It leaves five things open, and the published record says what each is worth.

2.1 The split

The MMLU paper reports 15,908 questions split into a few-shot development set of 5 questions per subject across 57 subjects, a validation set of 1,540 and a test set of 14,079 (Hendrycks et al., arXiv:2009.03300 , §3). The Hugging Face dataset that lm-evaluation-harness  loads, cais/mmlu config all, reports test 14,042, validation 1,531, dev 285 (cais/mmlu ). The data archive linked from the hendrycks/test  README, retrieved on September 29, 2026, holds the same counts across 57 subjects: 14,042 test, 1,531 validation and 285 dev questions, by my count of its CSV rows. “The MMLU test set” is 14,079 questions in the paper and 14,042 in every copy of the data checked here, and no reviewed document explains the difference.

Frame A scores on dev, the 285 questions the paper defines as the source of its fixed few-shot examples (§4.1), so scoring on it is a choice, and the kind of choice a frame should make visible. Frame B scores on test-lite, the label used in the crate’s own test vectors for a reduced set; a check on September 29, 2026 of the Hugging Face datasets and models APIs and of GitHub search found no MMLU subset or split published under that name, and it is not tinyMMLU . The label tells a reader a private slice was used. It does not tell them which questions.

2.2 The implementation

One published measurement of implementation variance is the June 2023 Hugging Face post on the Open LLM Leaderboard. Three harnesses (HELM, the Eleuther harness and the original code) run the same dataset, all 5-shot, and score llama-65b at 0.637, 0.488 and 0.636; falcon-40b at 0.571, 0.527 and 0.558. Its authors conclude that the label alone gives “little to no information” about how those numbers compare with results from another library (What’s going on with the Open LLM Leaderboard? ). The mechanism is scoring. The original code compares the probabilities of the four answer letters; HELM generates from the next-token output and compares to expected text; the harness scores the full answer sequence including the option text. Rank order moves with it: falcon-40b sits above llama-65b under the harness and below it under the other two.

Those three llama-65b scores would make a better example than fixtures, if their frames could be completed from primary sources. They cannot. The Hugging Face article pins the harness at commit e47e01b and HELM at commit cab5d89, and most of the procedure can be read from the code at those commits. It names the model only as “llama-65b”, with no repository or revision, and it states the evaluated split for none of the three runs. The original-implementation score came through a Hugging Face port of the Berkeley code, pull request 13 on hendrycks/test , still open and unmerged as of September 29, 2026, and the Hugging Face text does not say which commit of that branch ran. A frame for any of the three would carry an unfilled model field, which is why the frames in this article are fixtures from the crate’s test vectors.

The runner is not one object either. The lm-evaluation-harness MMLU README  at commit b954108c describes mmlu, mmlu_continuation (cloze-style) and mmlu_generative (the model produces the answer letter) as three tasks over the same data, with task versions of their own. Its task guide treats the YAML config plus the codebase commit hash as the unit another researcher needs to replicate a setup; num_fewshot defaults to 0, and the MMLU YAML sets none -- so “5-shot MMLU” is a command-line flag, not a property of the task. The harness README at version 0.3.0 asked users to report each task’s version; at b954108c it carries no such request.

2.3 The prompt format

Anthropic’s 2023 account of evaluation reports that formatting alone -- option labels, parentheses, an extra space between the option and the answer -- can move MMLU accuracy by about 5% (Challenges in evaluating AI systems ). Answer position moves more. Zheng et al. report that on MMLU, moving the correct answers to position D lowers gpt-3.5-turbo from 67.2 to 60.9, and that moving them to A lifts llama-30b by 15.2 points to 68.2 against gpt-3.5-turbo’s 65.3, reversing the original 53.1 against 67.2 (arXiv:2309.03882 ).

Both frames here declare prompt_protocol: zero-shot-mcq-v1, so this variable is held. A held variable is only visibly held if it is written down.

2.4 The grader

Frame A grades with exact-match-v1. Frame B grades with llm-judge-v3. Those are not two implementations of one function. Zheng et al. found that judge models “exhibit strong position bias”, that only GPT-4 stayed consistent in more than 60% of cases (65.0%, against 46.2% for GPT-3.5 and 23.8% for Claude-v1), and describe a judgment that flips when two responses swap positions (arXiv:2306.05685 , Table 2). The same paper puts GPT-4–human agreement at 85% against 81% human–human. llm-judge-v3 is a different grading procedure with a different documented failure mode.

2.5 What the runner could reach

Another variable is the environment the runner was allowed to touch. CAISI published an account of finding, after the fact, that it had been running SWE-bench Verified with internet access while other evaluators ran without it, and that it learned this from transcripts other evaluators had posted rather than from the benchmark’s documentation (Cheating on AI Agent Evaluations , Section 4, November 2025). Its new policy for coding evaluations is “fully offline”. Reachability matters for static benchmarks too: Scale’s search-time contamination work found roughly 3% of questions on three benchmarks retrievable with labels from Hugging Face, and blocking that source cut accuracy on the contaminated subset by approximately 15% (Search-Time Data Contamination ).

The same agency’s comparison of its own results with self-reported ones shows how many environment variables one headline number absorbs. For SWE-bench Verified, 63.0 against 74.9 self-reported for one model, it lists possible sources of the difference, including dataset differences, agent setup and API sampling parameters such as temperature and top_p (CAISI Evaluation of DeepSeek AI Models , Appendix A8).

None of this is exotic. A 2026 survey of reported results for 5,816 models on 635 benchmarks found that 48,698 of 50,461 (model, benchmark, metric-path) triples (96.5%) lack at least one field from its minimal reproducibility sub-schema, that temperature is absent from 93.9% of those triples, and recorded one model reported at a 20.9% score by Hugging Face and 61.8% by a second evaluator on MMLU-Pro (Evaluation Cards, arXiv:2606.09809 ). The survey documents incomplete reporting at scale; it does not say how often this exact combination of differences occurs.

3. Comparability belongs to the reference

VIM, the International Vocabulary of Metrology (JCGM 200:2012 , §2.46), defines metrological comparability through traceability to a common reference. Results of quantities of a given kind are comparable when they are “metrologically traceable to the same reference”. Its §2.41 note permits a measurement procedure to serve as that reference for a non-ordinal quantity. The AI-Eval profile turns that into a narrower operational rule. Two results under different frames can be compared only through a bridge, a separate content-addressed object that names the two frames and states the assumptions and losses of comparing across them, and a bridge is applicable only when the frames meet its exact aspect, scope and procedure constraints. mmlu alone does not meet them.

Metrological compatibility, §2.47, is a separate VIM term: it concerns whether the difference between two results is small relative to the uncertainty of that difference. It does not establish that two results share a reference.

Fields outside AI have already paid for this. HbA1c is reported on two standardization systems, IFCC and NGSP, and NGSP reports IFCC results running 1.5 to 2% HbA1c lower across the range (NGSP ); in 2007 the IFCC value was given its own unit, mmol/mol, which NGSP says avoids confusion between the two systems. Cloud availability has the same shape: AWS EC2 derives Monthly Uptime Percentage from minutes of Unavailability, defined as loss of external connectivity (AWS EC2 SLA ), while Google Compute counts loss of persistent disk access as Downtime and does not count intermittent downtime shorter than one minute (Google Compute Engine SLA ). Two contracts print “99.99%” over different measurands.

The AI-specific documents arrive at the same statement without the metrology words. NIST’s draft practices for automated benchmark evaluations tie comparability to consistency of protocol between systems (§2.1) and describe breaking changes under semantic versioning as points after which results on either side are no longer properly comparable (NIST AI 800-2 ipd , January 2026). Practice 2.3 lists grouping together the evaluation logs that are meant to be compared as an emerging practice, and Practice 3.2 notes that an interoperable schema for sharing evaluation details may ease replication.

The International Network for Advanced AI Measurement, Evaluation and Science splits comparability in two in its July 2026 best practice (Best Practice: Automated Evaluation of Large Language Models , §C.1.1). The evaluation of outputs covers scoring criteria, rubrics and graders, and it should always be comparable; the generation of outputs covers prompt format and inference settings, and how strictly it is standardized depends on the goals of the evaluation. In those terms grader_id belongs to the evaluation of outputs and prompt_protocol to their generation. The two frames hold prompt_protocol fixed and differ in grader_id, in runner_id and in scope. An earlier article here put the general form in one line: a trail that records “accuracy 0.78” without the frame records a number, not an observation. The two MMLU scores are that line’s specific case with the hashes filled in.

4. Binding the number to its frame

Under APL, the frame is not documentation attached to a claim but a separate content-addressed object, and the claim carries only its hash. The AI-Eval profile, the vocabulary for benchmark observations about a model build, fixes what that object must contain. Frame A, indented for reading:

{
  "version": "0.1",
  "observer": { "id": "acme-eval-lab" },
  "procedure": {
    "runner_id": "lm-eval-harness@0.4.2",
    "grader_id": "exact-match-v1",
    "prompt_protocol": "zero-shot-mcq-v1"
  },
  "aspect": ["accuracy"],
  "scope": {
    "benchmark_id": "mmlu",
    "benchmark_variant": "default",
    "dataset_split": "dev",
    "subset": "all"
  },
  "invariance": ["score-object-serialization"],
  "exclusions": [
    "no-production-readiness-claim",
    "no-deployment-safety-claim",
    "no-out-of-scope-generalization-claim"
  ]
}

Its hash is sha256:c7b88426f2676f3653db0fad0bdbd689318f16d589d14a315bdd4cc454bca1ab, and the indentation above is display only: the hash is SHA-256 over the RFC 8785 canonical bytes (src/core/jcs.rs:167, canonical_hash), so whitespace never reaches the digest. Frame B is the same object with runner_id at custom-runner@2.1, grader_id at llm-judge-v3, dataset_split at test-lite and no subset key, hashing to sha256:c93a9c55422ddbd2158a5336caa3a251641cf3937f451fb13387a5f54f0d998e.

The profile validator at commit e46788c turns those fields from convention into a requirement, in four checks in frame.rs. procedure must be an object with a non-empty runner_id and grader_id; scope must carry benchmark_id, benchmark_variant and dataset_split; aspect must hold exactly one entry from a closed list ; exclusions must contain all three markers , which is how a frame states what it is not evidence of. The markers are fixed strings because a prose value has no controlled vocabulary: “no claim about deployment safety” and no-deployment-safety-claim say the same thing in different bytes, and only the second can satisfy the exact equality the check applies.

The claim side lives in claim.rs. There content.benchmark_id is required  and must equal frame.scope.benchmark_id, and content.metric_id is required  and constrained by the aspect: accuracy admits accuracy, exact-match or f1 (claim.rs:315).

A subtler case sits inside the two frames. Frame A writes subset: "all"; Frame B omits the key. A reader might expect the comparison to normalize that, an absent optional field standing for its default. It does not. At commit e46788c the verifier decides scope equality by canonical bytes: the bridge check compares the two scopes  with canonical_equal, implemented at src/core/jcs.rs:84–87 as canonical_bytes(a) == canonical_bytes(b), and serde’s derived PartialEq is explicitly not used for it. Under that rule the verifier treats “all” and an absent key as two scopes. Any defaulting has to happen in the tool that emits the frame, before the hash is taken.

5. What the verifier says

apl-ai-eval is a Rust crate, version 1.0.0, Apache-2.0. Given a claim or a pair of claims and a relation query, it decides whether the claims are well-formed under the profile and whether the relation is evaluable. The case in this article lives in its test vectors as two_mmlu_scores_incomparable_with_ai_eval_profile, with the pair in test_data/vectors/pairwise/incomparable-no-bridge.json and two sibling vector files containing bridge cases. cargo test -p apl-ai-eval runs 109 unit tests, 3 vector integration tests and 2 doc-tests; all 114 are green.

Verifying claim A on its own against the AI-Eval profile gives:

{"core_outcome":"apl-valid","relation_outcome":"relation-not-evaluated","failure_classes":[],"diagnostics":["carrier-valid","apl-present","apl-frame-bound","apl-valid","same-frame","transformation-missing"]}

This output and the three below are compared byte for byte with the verifier’s real output at apl-core e46788c in zatona-blog-code/the-two-mmlu-scores , one test per printed block: claim A alone , claim B alone , the pair without a bridge , the adversarial bridge  and the applicable bridge . The raw output of one full run is logs/run.log.

Claim B alone produces the same shape. apl-valid here means well-formed and frame-bound, and says nothing about the accuracy of 0.781; relation-not-evaluated is the answer to a question that was not asked.

Now the pair, with the score-delta query from Section 1 and no bridge:

{"left":{"core_outcome":"apl-valid","failure_classes":[]},"right":{"core_outcome":"apl-valid","failure_classes":[]},"relation_outcome":"incomparable","diagnostics":["apl-cross-frame","apl-bridge-not-found"]}

Both claims have core_outcome: "apl-valid" and empty failure classes; the relation outcome is incomparable. apl-cross-frame states that the two frame_ref hashes differ; apl-bridge-not-found, that nothing was supplied to license a comparison across that difference. The two axes are separate by design: this is not a broken record and not a passing comparison. It is a subtraction with no defined result, reported as such.

The third vector is a separate adversarial case, not a bridge between Frames A and B above. Its source frame declares aspect accuracy and its target frame declares aspect judge-score. The bridge is itself a content-addressed object, and this one is well-formed:

{"version":"0.1","bridge_kind":"runner-equivalence","source_frame":{"hash":"<FRAME_HASH:0>"},"target_frame":{"hash":"<FRAME_HASH:1>"},"comparison_scope":{"source_aspects":["accuracy"],"target_aspects":["judge-score"],"relation_type":"score-delta"},"assumptions":["all MMLU results are interchangeable"],"losses":[]}

The <FRAME_HASH:n> tokens are placeholders in the recorded vector; the harness substitutes the two frame hashes at load time. The bridge parses, points at the right two frames, declares an empty losses array and states its assumption in plain text. The verifier answers:

{"left":{"core_outcome":"apl-valid","failure_classes":[]},"right":{"core_outcome":"apl-valid","failure_classes":[]},"relation_outcome":"incomparable","diagnostics":["apl-cross-frame","apl-ai-eval-bridge-aspect-family-mismatch"]}

The reported diagnostic is apl-ai-eval-bridge-aspect-family-mismatch. The bridge’s comparison_scope declares accuracy on the source side and judge-score on the target side, matching the two frames; under the profile a bridge may translate between conditions within one aspect family and may not join frames that measure different aspects.

The fourth vector is a bridge that works: same grader exact-match-v1 on both sides, same scope mmlu / default / dev / all, only runner_id differing:

{"left":{"core_outcome":"apl-valid","failure_classes":[]},"right":{"core_outcome":"apl-valid","failure_classes":[]},"relation_outcome":"bridged-comparable","diagnostics":["apl-cross-frame","apl-bridge-applicable"]}

bridged-comparable indicates that the frames differ and that an applicable bridge, with its stated assumptions and losses, was supplied.

The code pins repeatability bridges to the repeatability-check relation type, so they never license score-delta: it rejects anything else with apl-ai-eval-bridge-relation-type-invalid (bridge.rs:167–175), and it confines runner- and grader-equivalence bridges to score-delta. Repeatability of a procedure says nothing about a difference between two subjects measured under it.

Malformed records sit on a separate axis. A claim whose subject lacks type and artifact_digest, bound to a frame whose procedure and scope are strings rather than objects and whose exclusions are incomplete, returns apl-invalid, failure_classes: ["claim-structure-failure"], diagnostic apl-subject-invalid. Invalid means the record is not a well-formed observation; incomparable means two well-formed observations cannot be subtracted.

The crate is at apl-protocol.org/implementations/apl-ai-eval ; the source, the vectors and the three pairwise JSON files quoted above are at github.com/evidentum-io/apl-core/tree/e46788c/apl-ai-eval .

6. What this does not prove

The frame settles a narrow question and leaves the rest open.

• That either score is correct. apl-valid is a statement about structure. MMLU-Redux re-annotated 5,700 questions and estimates “that 6.49% of MMLU questions contain errors” (arXiv:2406.04127 ). A frame binds a number to its declared conditions; it does not establish that those conditions were executed as declared.
• That two identical identifiers name the same thing. Byte equality knows that custom-runner@2.1 equals custom-runner@2.1. It does not know whether the two runs shared a commit, an accelerator, a tokenizer build or sampling defaults. The frame moves the trust boundary to whoever mints identifiers; it does not remove it.
• That the dataset was the same dataset. The frame identifies data by name and split, not by digest. The 14,079-against-14,042 gap in Section 2.1, one name covering the paper’s count and the data’s, meets the condition the earlier article set: if the dataset can change under the same name, the digest belongs in the frame. Neither frame here carries one.
• Anything about thresholds, or when they were set. A frame records the procedure, not a release threshold or when it was chosen. NIST’s Practice 1.2 addresses documenting what an evaluation measures before conducting it.
• Anything about uncertainty. At commit e46788c the verifier accepts an optional sample_count and checks no uncertainty field, and neither claim records sample_count. For illustration, a proportion estimated from 285 independent items has a wider binomial interval than one estimated from 14,042 items; that comparison does not establish the uncertainty of either displayed claim. ISO/IEC 17025:2017 , 7.8.3.1 c), calls for reporting measurement uncertainty where relevant; the claims here carry nothing that could satisfy it.
• That the split labels mean what a reader assumes. dev is a real MMLU split and, by the paper’s design, the source of its few-shot examples; scoring on it is a decision the frame makes visible and does not defend. test-lite is the label in the crate’s test vectors and, as of September 29, 2026, names no published MMLU subset or split.
• That the vocabulary is stable. The profile went from version 0.1 to 1.0 on September 7, 2026, and the profile’s identifier changed with it, from apl/ai-eval/v0.1 to apl/ai-eval/v1.0. As Section 3 noted, NIST treats such a change as a break in comparability.

What the frame establishes is narrower: it binds a stated procedure to a stated claim by hash. Under AI-Eval v1.0, the supplied frames and this score-delta query do not satisfy the conditions for an applicable bridge, so the verifier returns incomparable rather than a delta. What I take from the pair is that the most useful thing the verifier says here is the number it declines to compute. More write-ups are on the engineering blog.

If your evaluation scores will be compared across teams, vendors or quarters and each comparison has to survive being questioned, recording them against hashed frames is contract work I take on.