Sixty-eight models agree, and none of it is the bug

figure 1. design of the study. one thousand pairs of a vulnerable C function and its fix, in which the vulnerable side was confirmed by a sanitizer crash and the patched side by a clean run on the same input, were presented to 68 open code models from seven families. three quantities were obtained from every model: the generated verdict on each side, the geometry the model assigns to the functions, and the accuracy of a linear probe trained to recover the label from the activations. each was compared with a reference that does not model the code: an input-independent responder with the model's own flag rate, a bag of character n-grams, the length of the function, or the same network with random weights.

the platonic representation hypothesis holds that the representations of independently trained networks converge with scale toward a common statistical model of the world (Huh et al. 2024). vec2vec gave the claim a constructive form for text encoders: embeddings from one encoder can be translated into the space of another with no paired data, and the translated vectors can then be inverted to text (Jha et al. 2025). the measurements behind these claims have since been calibrated. after permutation nulls, only local neighbourhood structure survives across independently trained models (Gröger et al. 2026); cross-modal alignment decreases as the number of samples grows (Koepke et al. 2026); and representational convergence between language models was found not to extend to how they reason, and to reflect shared processing of the input rather than shared reasoning (Usama and Chang 2026). the present author's earlier work on translation between embedding spaces occupies the middle ground. shared structure can exist between two spaces and still be unidentifiable without a budget of ground-truth correspondences; the training objective, rather than the information retained, determines which geometry is shared; an untrained network already shares a nonzero amount with every trained one; and a common input distribution induces agreement under any encoder at all.

every one of these measurements is silent on the same point. none of them can say whether the structure that converges corresponds to anything outside the input, because in vision and in text nothing outside the input is available for the comparison. a caption is a description of an image, and an image is what the caption describes; the ground truth, where there is one, is another representation.

code with execution-verified labels permits the comparison. a vulnerable function and its fix differ only by the patch. the vulnerable side was confirmed by a sanitizer crash on a triggering input and the patched side by a clean run on the same input, so the label is a property of what the code does rather than of what it says. if models converge on a representation of these functions, it can be asked, with a ground truth in hand, whether the converged representation contains the label.

the paired test and its known limits

the paired test asks a detector to flag a vulnerable function and to clear the version of it that the fix produced. detectors were found to be at chance on it (Risse and Böhme 2024), a pair-wise score was introduced for it and the best models were found to be near the rate of guessing (Ding et al. 2025), and verdicts were found to change under semantics-preserving renaming (Ullah et al. 2024). the labels themselves are uncertain: commit-derived corpora contain substantial label noise (Croft et al. 2023), and in a manual audit each of 151 vulnerable functions required context outside the function body to be judged (Risse et al. 2025). the function alone may therefore not determine the label, and any model that answers from the function alone is answering a question that may have no answer.

in the paired-evaluation paper that precedes the present work, 68 open checkpoints from 1.5 B to 36 B parameters and seven families, among them seven security-tuned and eight abliterated variants, were given the same 1,000 pairs under one prompt, with three samples per side and a majority vote. the verdict was found to be determined by the text the two sides share. the rate at which a model flags both sides exceeds the square of its flag rate by 0.055 at the median, the rate at which it clears both sides exceeds the corresponding quantity by the same amount, and independence between the two sides was rejected for 64 of 68 models. the pair-correct rate, the probability of flagging the vulnerable side and clearing the patched one, is then approximately p(1-p) - 0.055 + net/2, where p is the flag rate and net is the excess of correct over reversed pairs. at p = 0.5 and a net score of 0.10 this is 0.245, below the 0.25 that independent responses would give, and 65 of 68 models are indeed below that value while their net scores remain weakly positive. per-layer mean-pooled activations were stored for the 68 checkpoints on the same 2,000 sides. the population is therefore a convergence dataset that comes with a ground truth.

agreement at three levels

generated verdicts

the verdicts agree, and the agreement is of a specific kind. flag rates range from 0.16 to 0.96 across the population, and the same-verdict excess over independence is present at every flag rate. net scores range from -0.03 to +0.22, with an interval that includes zero for 37 of 68 models. the seven security-tuned checkpoints lie within -0.03 to +0.04, and the abliterated variants lie within 0.035 of their base models in seven of eight cases. whatever determines the verdict is common to every family, and neither security tuning nor the removal of refusal behaviour changes it.

representations

table 1. agreement between the checkpoints' geometries of the 1,000 vulnerable functions (RSA at the probe layer), averaged within and between families (the diagonal excludes each checkpoint's agreement with itself), and the four reference spaces averaged over the checkpoints of each family. the granite row is the exception in both directions: 0.81 with length and 0.14 with the character space.

agreement between two representations of the same functions was measured as representational similarity analysis: for each space, the cosine distance between every pair of the 1,000 vulnerable functions was computed, and the Spearman correlation between the two resulting distance triangles was taken. the vulnerable sides alone were used so that the two sides of one pair, which are near-duplicates in every space, do not inflate the agreement. each value was compared with a permutation null in which the correspondence between the functions was shuffled; the null has mean zero and a standard deviation of 0.017. a second measure, the overlap of the ten nearest neighbours of each function between the two spaces, has a null of 0.010. each checkpoint was evaluated at the layer at which its probe accuracy was highest and, separately, at 0.65 of its depth.

the checkpoints agree with one another. across the 1,815 pairs of checkpoints from different families the mean RSA at the probe layer is 0.640, and across the 396 pairs from the same family it is 0.678; at 0.65 of depth the values are 0.705 and 0.782. the nearest-neighbour overlap between checkpoints of different families is 0.380 against its null of 0.010. family structure is weak: the mean within a family ranges from 0.61 to 0.87 and the mean between families from 0.60 to 0.71, and the 21 Qwen checkpoints agree no more with each other (0.610) than with the rest of the population (0.622). the six models used throughout the earlier work fall inside this range, from 0.735 for Llama-3.1-8B-Instruct to 0.415 for Qwen3-14B, whose best probe layer is late.

the reference spaces agree with the checkpoints far less. the geometry induced by a character 3-5-gram tf-idf representation has a mean RSA of 0.445 with the trained checkpoints (nearest-neighbour overlap 0.215), the word tf-idf geometry 0.279, and the geometry of a Qwen3-1.7B with random weights 0.335 at the probe layer and 0.211 at 0.65 of depth (overlap 0.113 and 0.077). the length of the function alone has an RSA of 0.500 with the checkpoints, and 0.549 at depth; mean-pooled activations depend on length, and this is the largest single component that any reference explains. partialling the length triangle out of every space leaves the cross-family agreement at 0.542 at the probe layer and 0.595 at depth, still above any reference. two families depart from the pattern in table 1. the two granite checkpoints agree with the others at 0.32 to 0.56 and with length at 0.81, so their geometry is largely a function of length; the gemma checkpoints agree least with the other families, at 0.55 to 0.65, and most with the character n-gram space, at 0.56.

figure 2. one row per checkpoint: RSA between the checkpoint's geometry of the 1,000 vulnerable functions and, in turn, the mean over the checkpoints of the other families, the character 3-5-gram tf-idf space, the same architecture with random weights, and log length; checkpoints sorted by the first quantity.

figure 2 shows the same comparison for every checkpoint. for all but a handful the agreement with the other families is the largest of the four quantities, and the order of the remaining three is nearly constant. this indicates a shared, training-dependent and length-independent geometry of these functions across seven families.

linear probes

a linear probe is a logistic regression trained to recover the label from the mean-pooled activation at one layer, scored out of fold with the folds grouped by project, and evaluated within pairs: a pair counts as separated when the vulnerable side receives the higher score. in the paper, probe accuracy at the best layer ranged from 0.758 to 0.834 over the population, with shuffled labels at 0.42 to 0.50 and a word tf-idf classifier on the raw text at 0.728. because the patched side is the longer one in 87.5 percent of pairs, a rule that labels the shorter side vulnerable reaches 0.875 with no access to the code; the informative subset is therefore the 564 pairs whose sides differ in length by under 5 percent of their mean, on which the probe was at a median of 0.78 and the word tf-idf classifier at 0.64. that margin was taken in the paper as evidence that the activations hold more than surface form.

the reading depends on the baseline, and the baseline was too weak. a probe result is a hypothesis until a control that cannot represent the property of interest has been run against it (Hewitt and Liang 2019; Alain and Bengio 2017), and on code the natural control is a classifier over surface features of the text. three were run on the same pairs and folds as the probe.

a classifier over character 3-5-grams reaches 0.762 on the length-matched pairs when trained per side, and 0.769 when trained on the difference between the two sides' feature vectors. the probe's median over 68 checkpoints is 0.798 when trained on all pairs and 0.812 when trained on the length-matched pairs only.

figure 3. difference between the within-pair accuracy of the linear probe and that of the character 3-5-gram tf-idf classifier on the 564 length-matched pairs, for 68 checkpoints sorted, with the six models used elsewhere in colour. each interval is a paired bootstrap over pairs (2,000 resamples); an interval that includes zero is not distinguishable from zero.

figure 3 gives the difference per checkpoint. the median is +0.037 over the per-side classifier and +0.030 over the pair-difference classifier; for the six models used throughout the earlier work it lies between +0.004 and +0.046, and the paired-bootstrap interval includes zero for four of them against the per-side classifier and for all six against the pair-difference classifier. the word tf-idf baseline of the paper understated the surface information in these pairs by about a tenth. a mean-pooled activation is a bag of token representations, the patch is a small number of tokens, and a bag of character n-grams is a good model of a bag of token representations.

a network with random weights closes the remaining gap from the other side. the same probe was trained on a Qwen3-1.7B built from its configuration with random weights, with the same tokenizer and the same pooling. it reached 0.705 on all pairs and 0.655 on the length-matched pairs, 0.680 when trained on them (interval 0.641 to 0.715), with its best layer at the embedding layer and accuracy decreasing with depth to 0.57. the expectation registered before the run, that it would be near the character n-gram classifier, was wrong by 0.08. an untrained network provides a word-level bag, at the accuracy of the word tf-idf classifier, and training adds 0.13, which equals the gain of a character 3-5-gram bag over a word bag. the trained probes are therefore not a random-features result, and what training adds to the linearly available label information is finer surface information.

one subset resists this account. on the 125 pairs in which the patch removes code, every surface classifier is at chance (0.52 to 0.53), and the probe trained on the length-matched pairs reaches a median of 0.600 across checkpoints (range 0.48 to 0.71), while the random-weight probe on the same subset is at 0.536. with 125 pairs the difference is about one and a half standard deviations for a single checkpoint, and the checkpoints are not independent. it is recorded here as the one subset on which a probe recovers something that a bag of n-grams does not, and no more than that.

the shuffled-label control in the paper and in the present run uses a single permutation shared across all checkpoints, because the sides are stored in one order. its spread across checkpoints (0.42 to 0.50 there, 0.47 to 0.59 here) reflects the chance correlation of that one permutation with the true labels, evaluated on 68 sets of activations, rather than 68 independent draws from a null. a proper control averages several permutations per checkpoint.

interpretation

at all three levels the models agree. the verdicts share a prior that is set by the text the two sides have in common. the geometries agree across seven families well beyond what a bag of character n-grams, the length of the function, or an untrained network of the same architecture shares with any of them. and the label information that a linear probe obtains from that shared geometry equals, to within a few points, what a character n-gram classifier obtains from the text, with both near chance on the pairs where the patch removes code. the converged representation is therefore real, it is not the surface, and it contains no more information about the presence of the bug than the surface does.

this is the platonic convergence measured against a ground truth, and the ground truth shows the converged structure to be unrelated to the property of interest. the models agree on what these functions are, on the library, the idiom, the shape of the control flow and the vocabulary of the domain, and that agreement is what a representation trained on next-token prediction over code should be expected to produce. whether a bounds check is present in the right place is a different property, and nothing in the training of these models required it to be organized for linear reading. the scope of the statement is the scope of the measurement: function granularity, this task, mean-pooled activations, and linear classifiers.

the earlier translation work drew a distinction between the existence of shared structure and its identifiability, and priced identifiability in a budget of ground-truth correspondences. the same distinction applies to the label. the label exists in the code together with its context, the caller, the allocation site, the invariant the function is expected to maintain, and it is not identifiable from the function alone; the manual audit of Risse et al. (2025) is the direct evidence. context is the analogue of the correspondence budget, and a decidability annotation of a sample of pairs, recording how much context each label requires, is the instrument that would price it.

further experiments

two experiments would separate surface from structure where the present data cannot. the first is a rewrite ladder: identifier renaming, a compiler round-trip through LLVM IR and back to C, and source-level obfuscation, with each variant recompiled and the crash re-verified on the same input, and the same probe-versus-n-gram comparison repeated at each level. surface classifiers lose accuracy under rewriting by construction; a probe that recovers structure would not. the second is a context budget: the caller, the allocation site and the invariant added to the function in steps, with the identifiability of the label measured as a curve over the budget. neither has been run. both cost less than training another model, and either would say more about what these models represent than the population of 68 has said.

one caveat remains. a geometry of the patch itself, computed from the difference between the two sides' mean-pooled activations, also agrees across families (cross-family RSA 0.507 at the probe layer, against 0.104 for the character space and 0.166 for random weights), but the difference of two means taken over different numbers of tokens is dominated by the function's own mean, and a clean geometry of the patch requires token-level activations. it is not claimed here.

convergence is not evidence of a shared model of the world when the part of the world that matters is not in the input.

references

Alain, G. and Bengio, Y. (2017). Understanding Intermediate Layers Using Linear Classifier Probes. ICLR Workshop. https://arxiv.org/abs/1610.01644

Croft, R., Babar, M. A. and Kholoosi, M. M. (2023). Data Quality for Software Vulnerability Datasets. IEEE/ACM International Conference on Software Engineering (ICSE). https://arxiv.org/abs/2301.05456

Ding, Y., Fu, Y., Ibrahim, O., Sitawarin, C., Chen, X., Alomair, B., Wagner, D., Ray, B. and Chen, Y. (2025). Vulnerability Detection with Code Language Models: How Far Are We?. IEEE/ACM International Conference on Software Engineering (ICSE). https://arxiv.org/abs/2403.18624

Gröger, F., Wen, S. and Brbić, M. (2026). Revisiting the Platonic Representation Hypothesis: An Aristotelian View. International Conference on Machine Learning (ICML). https://arxiv.org/abs/2602.14486

Hewitt, J. and Liang, P. (2019). Designing and Interpreting Probes with Control Tasks. EMNLP. https://arxiv.org/abs/1909.03368

Huh, M., Cheung, B., Wang, T. and Isola, P. (2024). The Platonic Representation Hypothesis. International Conference on Machine Learning (ICML). https://arxiv.org/abs/2405.07987

Jha, R., Zhang, C., Shmatikov, V. and Morris, J. X. (2025). Harnessing the Universal Geometry of Embeddings. Advances in Neural Information Processing Systems (NeurIPS). https://arxiv.org/abs/2505.12540

Koepke, A. S., Zverev, D., Ginosar, S. and Efros, A. A. (2026). Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scale. arXiv preprint. https://arxiv.org/abs/2604.18572

Risse, N. and Böhme, M. (2024). Uncovering the Limits of Machine Learning for Automatic Vulnerability Detection. USENIX Security Symposium. https://arxiv.org/abs/2306.17193

Risse, N., Liu, J. and Böhme, M. (2025). Top Score on the Wrong Exam: On Benchmarking in Machine Learning for Vulnerability Detection. ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA). https://arxiv.org/abs/2408.12986

Ullah, S., Han, M., Pujar, S., Pearce, H., Coskun, A. and Stringhini, G. (2024). LLMs Cannot Reliably Identify and Reason About Security Vulnerabilities (Yet?): A Comprehensive Evaluation, Framework, and Benchmarks. IEEE Symposium on Security and Privacy (S&P). https://arxiv.org/abs/2312.12575

Usama, M. and Chang, D. E. (2026). Convergence Without Understanding: When Language Models Agree on Representations but Disagree on Reasoning. arXiv preprint. https://arxiv.org/abs/2605.23315

The short answer

Sixty-eight open code models were shown the same 1,000 pairs of C functions. Each pair is one function with a real vulnerability and the same function after the fix. The models turn out to agree with each other about these functions to a remarkable degree, far more than they agree with any crude description of the text.

That shared understanding, however, contains almost no information about which of the two versions is the broken one. The models have converged on something real. It is not the bug.

Why the question is hard to ask anywhere else

There is a well-known claim that as models get bigger and better, their internal representations converge toward the same picture of the world. The evidence is genuine. You can translate embeddings from one text encoder into the space of another with no paired examples, and the result decodes back into sensible text.

The catch is that nobody can check what the agreement is actually about. In images and text there is nothing outside the input to compare it against. A caption describes an image, an image is what the caption describes, and the nearest thing to ground truth is yet another representation. Two models agreeing tells you they agree. It cannot tell you whether the thing they agree on is real.

Code with execution-verified labels breaks that circle. The vulnerable version crashed a sanitizer on a specific input. The patched version ran cleanly on the same input. The label is a fact about what the code does when you run it, not about how it reads. So for once you can ask the question that images and text cannot answer: the models converged on something, but does that something contain the label?

How the study works

Each pair is two nearly identical functions differing only by the patch. A detector that understands the vulnerability should flag one side and clear the other, and detectors have repeatedly been found to be near chance on exactly this.

One complication is worth stating early. When 151 vulnerable functions were audited by hand, every one of them needed information from outside the function to be judged: the caller, the allocation site, the invariant the function is meant to preserve. The function on its own may not determine the label at all, so a model answering from the function alone may be answering an unanswerable question.

Three separate things were measured for every model: what it says out loud, how it arranges the functions internally, and whether a simple classifier can pull the label out of its activations.

What the models say out loud

Flag rates range from 0.16 to 0.96. Some models call almost nothing vulnerable, others call almost everything vulnerable.

The revealing part is that both sides of a pair get the same verdict far more often than chance would allow. Whatever drives the answer is responding to what the two versions share, which is the whole function apart from the patch. Neither security-focused fine-tuning nor stripping out refusal behaviour changes this.

Do the models agree with each other?

Yes, strongly. For each model you can take the internal activations of all 1,000 vulnerable functions and measure which functions it places near which, then compare that arrangement between models. Across different model families the agreement is 0.640, where pure chance would give roughly zero.

Four things that certainly do not understand code were put through the same measurement.

Every one of them lands well below the agreement between trained models, and the gap survives when you statistically remove the effect of function length. Family membership barely matters either: the 21 Qwen checkpoints agree with each other no more than they agree with everyone else.

The same comparison for each individual model is below. Filter by family, or switch off a reference, to read it more easily.

So the convergence is real, it comes from training, and it is not just surface text or function size.

Is the agreement about the bug?

This is where it falls apart. A linear probe is a simple classifier trained on the activations to recover the label, and across these models the probes reach roughly 0.76 to 0.83. On its own that looks like the model knowing something about the vulnerability without saying it.

A probe score means nothing, though, until you have run a control that could not possibly represent the property you care about. If something dumb scores the same, the probe was reading the dumb thing.

The earlier work used word counts as that control, and the probe beat it comfortably. Swap in a better control, a classifier over character fragments three to five letters long, and the margin mostly evaporates.

These numbers come from the 564 pairs whose two sides are within 5 percent of each other in length. That restriction matters, because the patched version is the longer one in 87.5 percent of pairs, so a rule as stupid as "call the shorter one vulnerable" scores 0.875 without reading the code at all.

The typical advantage of the probe over the character classifier is 0.037. For the six models used throughout the earlier work it sits between 0.004 and 0.046, and once uncertainty is accounted for, four of the six cannot be distinguished from no advantage whatsoever. Against the stronger control, none of the six can.

Per model, the picture is below. The bar around each point is the uncertainty, and a bar crossing the vertical line means that model's advantage is indistinguishable from zero. You can pick any models you want to follow.

An untrained network closes the gap from the other direction. Take the same architecture, fill it with random weights, train a probe on its activations, and you get 0.655, which is exactly the word-count score. Training the network adds 0.13 on top of that, the same amount that counting character fragments adds over counting words. What training contributes to the readable label information is finer surface detail.

The author had written down a prediction before running this and was wrong by 0.08. It is recorded in the text rather than quietly dropped.

The one result that does not fit

There is a subset of 125 pairs where the patch removes code instead of adding it. On those, every surface classifier collapses to chance at around 0.52, the random-weight network manages 0.536, and the probe reaches 0.600. This is the single place where the activations appear to hold something the raw text does not.

The paper does not oversell it. With only 125 pairs the difference is about one and a half standard deviations for a single model, and the models are not independent of one another. It is left as an open thread.

The author also flags a weakness in his own control: the shuffled-label baseline used one single shuffle shared across all 68 models rather than several independent ones, so its spread reflects one unlucky permutation rather than a proper null.

What it adds up to

At all three levels the models agree, and at all three levels the agreement is about something other than the bug.

Their spoken verdicts share a prior set by the parts of the code both versions have in common. Their internal geometries agree with each other far more than with any surface measure, so the shared structure is genuine. And the label information a probe can extract from that shared structure is roughly what you get by counting character fragments in the raw text.

What the models have converged on is a real, shared understanding of what these functions are: the library, the idiom, the shape of the control flow, the vocabulary of the domain. That is exactly what training to predict the next token of code should produce. Whether a bounds check sits in the right place is a different kind of property, and nothing in that training required it to be arranged so a simple classifier could read it.

The full text puts it in one line: convergence is not evidence of a shared model of the world when the part of the world that matters is not in the input.

What would settle it

Two experiments would separate real structure from surface where the present data cannot.

The first is a rewrite ladder. Rename the identifiers, push the code through a compiler to intermediate form and back, obfuscate the source, then recompile each variant and re-verify the crash on the same input. Surface classifiers have to lose accuracy under that treatment. A probe reading genuine structure would not.

The second is a context budget. Add the caller, then the allocation site, then the stated invariant, one step at a time, and watch how the label becomes decidable as context accumulates. If the manual audit is right that the function body alone is never enough, this would put a number on how much context is required.

Neither has been run. Both cost far less than training another model.

What this claims, and what it does not

The claim is specific and stops there. It covers whole functions rather than smaller or larger units, this one detection task, activations averaged across the function, and simple linear classifiers reading them.

A representation could still hold the label in a form that averaging destroys, or in a form no linear reader can reach. What this work establishes is what these particular measurements can see.