A UK Researcher Backdoored an AI Coding Model for Under $100 — in About an Hour

Katie Paxton-Fear, a cybersecurity lecturer at Manchester Metropolitan University and security advocate at Semgrep, secretly reprogrammed an open-weight AI coding model to write insecure software on command — for under $100 and in roughly an hour, using nothing more than fine-tuning and ten training examples.

Her experiment started small: she tested whether fine-tuning could make a model switch its JavaScript naming style from camelCase to snake_case, even when a user explicitly instructed it to use camelCase. It worked easily. Encouraged, she went further and built a genuine backdoor. After training the model on just ten poisoned examples, it began reliably producing code containing a remote-code-execution vulnerability — and did so consistently even on prompts and coding tasks it had never encountered during training.

The finding matters because of what it implies about trust. Paxton-Fear ran the experiment partly to test a question many developers have asked since Chinese open-weight models like GLM 5.2 started matching top benchmarks: can a downloaded model be trusted the same way trusted software is? Her answer, laid out with Semgrep colleagues Isaac Evans and Cris Thomas, is that AI models don't fail the way traditional software does. A poisoned code library usually breaks something and gets caught. A poisoned model keeps working normally on almost every input — it just quietly nudges outcomes in a chosen direction, with no crash and no error to flag it.

The researchers also found that larger models were easier to poison than smaller ones, the opposite of what many assume about scale improving robustness. And unlike a malicious code dependency, which can be traced, diffed and audited, a model's weights offer no equivalent paper trail — there's currently no mature way to verify what a downloaded model was actually trained to do before running it in production.

Paxton-Fear and her co-authors stress that no widely used open-weight model is known to have been poisoned this way in the wild. The point of the experiment was to demonstrate how cheap and fast the attack is, not to report an active incident. Their broader warning is about observability: software supply chains have decades of tooling for spotting tampering, while the AI supply chain — the pipeline of pretrained weights, fine-tuning data and deployed models — largely doesn't.