Score
SURVEY · MACHINE UNLEARNING · 2025
A Survey of LoRA-based Machine Unlearning for LLMsMethods, Taxonomy, and Evaluation
Keep the backbone fixed. Examine what it takes to forget through lightweight adapters.
University of Technology Sydney · University of Southern Queensland · Edith Cowan University
128 / 256 adapter / full-update parameters
W₀ fixed BA changes with r
Interactive algebra illustration with synthetic 16 × 16 matrices. Rank changes BA and its parameter count; it does not measure forgetting or model quality.
01 THE OPERATOR VIEW
One update.
Five places to intervene.
The survey organizes methods by how they construct, transform, constrain, and sparsify an adapter-space update.
Δφ = M ∘ C ∘ P ∘ G (s)
Read the composition from right to left. Select an operator to inspect its role. ←
Precondition
Rescale the direction, for example with curvature information.
Project
Route the update into a selected adapter subspace.
Bound
Limit the step with clipping, trust regions, or bounded dynamics.
Mask
Selectively zero adapter parameters in sparse variants.
Adapter-space operator view · Not every method uses every slot.
02 METHOD ATLAS
Six families.
Six ways to shape an edit.
Compare the mechanism and its failure modes. Highlighted slots show the organizing emphasis, not a complete implementation recipe.
sGPCM
Gradient-based
Shape the forget and retain losses directly.
A direct update in adapter space uses a mix of forget and retain objectives. It is simple to implement, but aggressive steps can damage unrelated utility.
Watch for / Overshooting and loss of retain behavior.
sGPCM
Influence-based
Approximate the effect of retraining.
Inverse-Hessian-style reasoning estimates how removing data would change adapter parameters. The approximation and estimation cost matter.
Watch for / Curvature estimation and approximation quality.
sGPCM
Fisher-informed
Use sensitivity to guide the update.
Fisher-style information helps choose or constrain update directions and align adapter capacity with the forget set.
Watch for / Quality of the Fisher surrogate and available access.
sGPCM
Structural / continual
Give repeated requests their own structure.
Multiple adapters, routing, and approximate orthogonality can reduce interference across deletion events. Composition remains a lifecycle decision.
Watch for / Interference, routing, and adapter accumulation.
sGPCM
Bounded dynamics
Constrain how far parameters move.
Explicit bounds, clipping, or constrained reparameterizations stabilize deletion steps and help protect retain behavior.
Watch for / The constraint must balance forgetting and utility.
sGPCM
Pruning / sparse
Remove selected adapter capacity.
Importance estimates identify adapter parameters to mask or prune. Structural removal can be compact, but rollback becomes harder.
Watch for / Noisy importance estimates and limited reversibility.
03 EVALUATION
Forgetting is
more than one score.
TOFU-style evaluation needs several views of the same update: target behavior, useful capability, privacy leakage, and repeated requests.
Does target behavior actually change?
Measure forget-set responses and compare them with a retain-only retraining reference. Lower answer probability alone is not proof of privacy.
0.2108Prune / Sparse · forget_Q_A_Prob
Lowest forget-set probability among the compared unlearning variants.
0.4030GA · forget_Q_A_ROUGE
Lowest ROUGE metric under the shared evaluator and budget.
What useful behavior survives?
Report retain-set quality under the same backbone, LoRA configuration, optimization budget, and evaluator.
0.4261Structural orthogonal composition
Highest round-five utility of the two structural composition variants.
Can deleted information still be inferred?
Use leakage diagnostics such as PrivLeak alongside behavior metrics. A weak answer is not necessarily a private answer.
−29.73Fisher · PrivLeak
PrivLeak closest to zero among the compared single-shot unlearning variants.
What happens after the next request?
Track forgetting and utility across deletion rounds. Inspect interference, routing, merge policy, and checkpoint history.
5Deletion rounds
The continual experiment compares adapter accumulation with approximately orthogonal composition.
SHARED PROTOCOL IN THE SURVEY
Llama-3.2-1B-Instruct → LoRA → TOFU forget05 / retain95 → Unified evaluator
04 DEPLOYMENT
A deletion request
has a lifecycle.
An adapter checkpoint makes an edit portable. Reliable deployment also needs a policy for routing, merging, evidence, and retirement.
01 Track the artifact+
Keep adapter deltas, deletion-request logs, and checkpoint lineage together so a deployed update can be inspected.
02 Manage repeated edits+
Define routing and merge policies; monitor interference across deletion events rather than blindly accumulating adapters.
03 State the privacy boundary+
An adapter edit does not by itself prove deletion from the frozen backbone. State the access assumptions and test the intended threat model.
05 / CITATION
Continue with
the paper.
Read the complete argument, assumptions, and references in the preprint.
SSRN:5866522 ↗@misc{li2025loraunlearningsurvey,
title = {A Survey of LoRA-based Machine Unlearning for LLMs: Methods, Taxonomy, and Evaluation},
author = {Delong Li and Guangsheng Yu and Xu Wang and Yanna Jiang and Wencheng Yang and Bin Liang and Wei Ni},
year = {2025},
howpublished = {SSRN preprint},
doi = {10.2139/ssrn.5866522},
url = {https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5866522}
}