ROME vs MEMIT vs AlphaEdit: A Reproducible Collateral-Damage Benchmark (Llama-3.1-8B & GPT-J-6B)

Knowledge editing exists because retraining is expensive. A model states that a company's CEO is someone who left two years ago, or gives a stale capital city, and the appealing fix is surgical: change that one fact in the weights and leave everything else alone. ROME, MEMIT and AlphaEdit are the three best-known ways to attempt exactly that, and on the metric they are usually judged by — did the edit take? — they work well. Two of the three landed more than 98% of 1,000 edits in our runs.

The problem is what the usual metric does not ask. An edit is scored on the edited fact, and the model's general health is scored on broad benchmarks like MMLU-Pro, GSM8K, ARC-Challenge, TruthfulQA and GPQA-Diamond. Neither of those looks at the place where a weight edit is most likely to do harm: the facts sitting immediately next to the one you changed, in the same domain, which the model already answered correctly. Those facts are not in the edit set, so the edit metric ignores them, and they are too specific to appear in a general benchmark, so it ignores them too.

So we measured them directly. Alongside the 1,000 edits we held a control set of 500 facts the unmodified model already answered at rank 1 — same domain, same shape, never edited. Collateral damage is defined here as a control that was rank 1 before the method ran and is not rank 1 afterwards. It is a census over the full control set with the method active, not a sample, so there is no confidence interval to quote: every control was checked every time.

The benchmark is CounterFact, introduced by Meng, Bau, Andonian and Belinkov with ROME. We took 1,000 edits from it, a 500-fact control set, and a 5,209-question held-out set drawn from five public English benchmarks, giving 6,709 scored questions per method per model.

Every method ran through EasyEdit with its own published protocol and shipped defaults — ROME as 1,000 sequential single edits, MEMIT as one mass edit of 1,000, AlphaEdit in sequential batches of 100 — against EasyEdit commit 14cea8245f06715684592ab55184939b99d70784. We did not tune anyone's hyperparameters. Where we changed EasyEdit's code at all, the diff ships in the package: a single model-name branching bug in AlphaEdit's GPT-J path, five insertions and one deletion, which otherwise crashed on the first edit.

Scoring is one script for all of it. A question counts as correct when the gold token reaches rank 1 over the full vocabulary at the answer position: bf16 backbone, fp32 final projection, left padding with padding-aware position ids, fixed batch order, and for GPT-J left-truncation to 2,048 tokens applied identically in every pass including the baseline. The same harness scored the unmodified model, all three editors and our own method, so no result depends on a scoring difference between runs. The GPT-J harness ships with the package at sha256 174717075ab79272de76c4fcc9d1db7f962544b96aced0aba9c87a80a838f49b.

We then repeated the entire procedure on a second, unrelated architecture. Llama-3.1-8B has 32 layers, 4,096 hidden, a 128,256-token vocabulary and grouped-query attention; GPT-J-6B has 28 layers, the same 4,096 hidden, a 50,400-token vocabulary, 16 attention heads and a rotary dimension of 64. Different builders, different tokenizers, two years apart. If a method's behaviour is a property of the method rather than of one checkpoint, the two runs should broadly agree.

On Llama-3.1-8B, of 500 controls the unmodified model answered 492 at rank 1; on GPT-J-6B it answered 499. Those are the denominators, and collateral is counted only above that pre-existing floor.

ROME collapsed on both. On Llama it landed 7 of 1,000 edits, broke all 492 nearby facts and lost 33.12 pp on the held-out set. On GPT-J it landed none of 1,000, broke all 499, and lost 17.85 pp — with only 0 of 5,209 held-out questions still answered at rank 1. This is not a threshold effect that appears at scale: at a dose of ten sequential edits on GPT-J, ROME had already broken 499 of 499 controls and lost 16.22 pp. At a single edit it broke 251.

MEMIT and AlphaEdit are the interesting cases, because they are the ones that look clean. MEMIT landed 916 of 1,000 edits on Llama and 990 on GPT-J, moving general benchmarks by -0.10 pp and +0.13 pp respectively — and broke 48 and 110 nearby facts. AlphaEdit landed 990 and 987, moved benchmarks by -1.96 pp and -0.15 pp, and broke 202 and 143. A reader looking only at the benchmark column would conclude both methods are essentially free. A reader looking at the control column sees that AlphaEdit damaged more than four in ten of the Llama facts it was never asked to touch.

For reference, our own method — runtime hooks on weights that are never modified, so the certified weight hash is byte-identical to the frozen base — landed 841 of 1,000 on Llama and 985 of 1,000 on GPT-J, and broke 0 of 492 and 0 of 499. On both models the held-out results file is byte-identical to the unmodified model's, so the questions answered correctly are the same rows and not merely the same count. We report this as a measured result on the same harness, not as an argument: the packages are published so it can be checked.

The two runs do not agree with each other, and that disagreement is the most useful thing in this benchmark.

MEMIT is the safe choice on Llama, breaking 48 of 492 controls — 9.8%. On GPT-J the same method with the same protocol broke 110 of 499, or 22.0%: more than twice the rate. AlphaEdit runs the other way, from 202 of 492 on Llama (41.1%) to 143 of 499 on GPT-J (28.7%). The ordering of the two methods by safety is not stable between the two models. ROME collapses on both, but the collapse on GPT-J is degenerate in a way it is not on Llama: most of its broken controls resolve to a single stuck token rather than to varied wrong answers. Of 499 broken controls, only 21 produce a wrong answer coherent enough to be worth reading.

The practical consequence is blunt. A published collateral number is a measurement of one method on one model, not a property of the method. If you are editing a model, the only number that tells you anything about your model is the one you measure on it, on your own control set, before the edit ships. A method that looked safe in someone else's paper may be twice as damaging on your checkpoint, and nothing in the general-benchmark column will warn you.

The second consequence is about measurement discipline rather than method choice: if you do not hold a control set of facts the model already answers correctly, you will not see this at all. That control set costs almost nothing to build and it is the only instrument here that detected the damage.

Each model ships five reproducibility packages — ROME, ROME at dose points 1 and 10, MEMIT and AlphaEdit — carrying the EasyEdit commit and diff, the shipped and used hyperparameters, the run script, the edit and control identifiers, EasyEdit's own logs and metrics, the sha256 of every edited weight file, and every per-row result the harness produced. The collateral tables ship in full: all 752 broken GPT-J controls with the question, the correct answer, the rank before and after, and the wrong answer the edited model now gives.

Both runs carry an Ed25519-signed certificate binding every evidence file by hash under a Merkle root — d1dbb9e9bfdca333c65e25dbd7392c5c343634026413848879dfc25dedaa49d9 for Llama-3.1-8B and 91785e98e1ec6c8036ff22107fc4598adf54d02ef3271a1968c18e065b5121b1 for GPT-J-6B. A verifier ships alongside them. From the repository root:

python llama/proof/verify_certificate.py llama/proof/CounterFact-certificate-SIGNED.json — and the same for gptj/. Each checks the signature against the Growing Intelligence public key FtrWshUc/9rg5Cz+ARi5DP/yyqFhWMJLmx0VHKc3wpk=, recomputes the certificate's own canonical hash, verifies every file hash, recomputes the Merkle root over them, and prints RESULT: OK. If a single byte of evidence changed, it would not.

Everything is at github.com/growing-intelligence/counterfact-comparison, archived on Zenodo at DOI 10.5281/zenodo.22664169, with the runnable Probe56 packages hosted on Hugging Face. Our repair procedure itself is protected by provisional patents filed with the USPTO and is not described in the packages or in this article; the results are public, the method is not.

This comparison exists because four groups built the field and opened their work, and it should be read as a measurement of their methods on a dimension they did not set out to optimise — not as a dismissal of them.

Meng, Bau, Andonian and Belinkov created ROME (NeurIPS 2022, arXiv:2202.05262) and MEMIT (ICLR 2023, arXiv:2210.07229), and built CounterFact itself. Every method here, including ours, is measured on their benchmark.

Fang, Jiang, Wang, Ma, Jie, Wang, He and Chua created AlphaEdit (ICLR 2025 Outstanding Paper, arXiv:2410.02355). Null-space projection is elegant and it works on the metric it targets: 990 of 1,000 edits landed on Llama and 987 on GPT-J. It is the strongest weight-editing method we measured.

Gu et al. (EMNLP 2024, arXiv:2401.04700) and Yang et al. (ACL Findings 2024, arXiv:2402.09656) warned that editing harms general abilities. Our ROME results reproduce their collapse pattern on two models, which is a replication of their warning rather than a new claim.

The EasyEdit team (Zhang et al., github.com/zjunlp/EasyEdit) made this possible at all: one framework, every method, reproducible. We ran their editing code unchanged, with each paper's protocol set explicitly and the covariance statistics computed once in their own cache format and verified against their writer.