Does AlphaEdit Really Preserve Knowledge? Testing the ICLR 2025 Outstanding Paper's Guarantee

The idea behind AlphaEdit is elegant. A weight edit is a perturbation added to a matrix. If you first compute the subspace spanned by the representations of the knowledge you want to keep, and then project the perturbation so that it lies in the null space of that subspace, the update is orthogonal to the preserved knowledge — it should leave those representations untouched while still moving the ones you are targeting.

That is a well-formed claim and the paper supports it. The guarantee is about the interaction between the update and a specific preserved set, expressed in terms of the representations the method was given. Within those terms it holds, and it is why AlphaEdit degrades general benchmarks so much less than sequential ROME does: on our runs it moved five public benchmarks by -1.96 pp on Llama and -0.15 pp on GPT-J, where ROME lost 33.12 pp and 17.85 pp.

The practitioner's question is subtly different: after I edit, does the model still answer the things it used to answer? That is a question about model behaviour on specific facts, not about the geometry of an update against a sampled representation set. The two coincide only if the preserved set covers the facts you actually care about, and if orthogonality in representation space implies rank-1 stability at the output. Neither is guaranteed, and neither is claimed.

We ran AlphaEdit through EasyEdit with its own published protocol and shipped defaults — sequential batches of 100, at EasyEdit commit 14cea8245f06715684592ab55184939b99d70784 — on two unrelated architectures, and scored every run with one script. No hyperparameter tuning of anyone's method, ours included.

Alongside the 1,000 edits we held 500 control facts the unmodified model already answered at rank 1: same domain as the edits, same shape, never edited, and not part of the preserved set handed to any method. Collateral is a control that was rank 1 before and is not rank 1 after. It is a census over the whole control set, not a sample.

On Llama-3.1-8B, 492 of those 500 were rank 1 on the unmodified model; AlphaEdit broke 202 of them, or 41.1%. On GPT-J-6B, 499 were rank 1 on the unmodified model; AlphaEdit broke 143, or 28.7%. The general-benchmark column, meanwhile, said the method was nearly free on both.

The controls are not exotic. They are ordinary encyclopaedic facts of the same kind as the edits — a glacier's continent, a company's headquarters, a person's citizenship. The full list of what broke ships in the repository, with the question, the correct answer, the rank before and after, and the wrong answer the edited model gives instead.

There is no contradiction between the paper's proof and our measurement. A null-space projection constrains the update relative to the representations it was given. It says nothing about facts that were never in that set — and the set is a sample, not the model's entire knowledge. The facts we measured were deliberately outside it, because that is the realistic case: you do not know in advance which of the millions of things your model knows sit near the edit you are about to make.

There is also a measurement gap. Orthogonality is a statement about vectors. Correctness, as a user experiences it, is a statement about which token comes first. A perturbation can be small and well-behaved in representation space and still be enough to move a token from rank 1 to rank 3 — and a fact at rank 3 is a fact the model gets wrong.

This is why we think the interesting comparison is not between methods but between what a method measures and what a deployment needs. AlphaEdit optimised hard against a real objective and succeeded at it. The objective simply was not "no previously-correct answer regresses", and no amount of success against the first objective delivers the second for free.

If your requirement is that edits land, AlphaEdit is the best weight editor in this comparison and it is not close. 990 of 1,000 and 987 of 1,000 are strong numbers, achieved without collapsing the model the way sequential ROME does.

If your requirement is that nothing else changes, the guarantee will not give you that and you will have to measure it. Build a control set of facts your model already answers correctly in the domain you are about to edit, score it before and after with one script, and count. That instrument costs very little and it is the only thing in our whole benchmark that detected this.

Note also that the damage rate is not a property of the method you can look up. AlphaEdit broke 41.1% of nearby facts on Llama and 28.7% on GPT-J; MEMIT went the other way, 9.8% on Llama and 22.0% on GPT-J. The ordering of the two methods by safety flips between architectures. A number from someone else's paper, ours included, tells you what happened on their checkpoint.

The full four-method comparison, both models, every per-row result and the signed certificates are in the benchmark write-up and the public repository: see "ROME vs MEMIT vs AlphaEdit: A Reproducible Collateral-Damage Benchmark" and github.com/growing-intelligence/counterfact-comparison, archived at DOI 10.5281/zenodo.22664169.

AlphaEdit (Fang, Jiang, Wang, Ma, Jie, Wang, He and Chua; arXiv:2410.02355) was named an Outstanding Paper at ICLR 2025 and deserved it. Null-space projection is a genuinely good idea, it is clearly presented, and on the metric the field agreed to optimise it is the state of the art. Our results reproduce that: it is the best edit-success method we tested, on both architectures.

This article is a measurement of the method against a dimension the paper did not set out to optimise, on models the authors did not run, using a control set they never saw. That is what independent replication is for, and it is only possible because the authors published their work openly and the EasyEdit team (Zhang et al.) made every method runnable under one framework. We ran their code unchanged.

The honest summary is not "AlphaEdit does not work." It is that a guarantee about representation geometry is not a guarantee about answers, the field has been reporting the first while practitioners need the second, and the only way to close that gap is to measure the second directly.