On 2 November, four researchers at MIT CSAIL (MIT’s Computer Science and Artificial Intelligence Laboratory) claimed, in a blog post, that GPT-6 Astra, an AI assistant made by OpenAI, can look at the raw internal numbers of smaller AI models, work out what parts of them do, and rewrite specific numbers to change their behaviour.
First in a series. In Part 2, Opus writes ELIZA, a chatbot from 1966, into a neural network’s weights by reasoning about the rules.
In brief.
A neural network runs on weights, millions or billions of numbers that together decide what the model does. Nobody can read the finished weights by eye, and changing even a few can shift how a model behaves.
The weights are set during training, when the network adjusts its numbers over millions of examples until they form a useful pattern. In a blog post on 2 November, Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab at MIT CSAIL described eight experiments in which Astra was given raw weights and asked to make sense of them.1 They released the logs of every exchange in the runs they report.2
Astra received the raw numbers of four small game-playing networks along with the grid layout and action set (the list of moves available), but no game names, and identified all four correctly. In another experiment it wrote new game networks from scratch in a single reply, without training data. On Breakout (a game where the player bounces a ball to break rows of bricks), Astra’s hand-written network scored about 20 times higher than a network trained by playing the game millions of times, though Astra had the game’s source code, an advantage the trained network lacked.1,3
Alex Krentsel at Google Research later reproduced the Breakout experiment across model generations. GPT-6 Astra (22x the trained baseline), GPT-6.1 Sol (18x) and Opus 5.5 (20x) scored far above earlier models, most of which were near zero.4
The blog’s most detailed experiment was a repair of Alibaba’s Qwen3.5-0.8B. MATH-500 jumped from 7.9% to 48.1%, and most of that gain comes from answers that now finish instead of looping.
Qwen3.5-0.8B is the smallest model in the Qwen 3.5 family, a neural network with 0.8 billion weights. The study ran it in thinking mode, a setting where the model works through problems step by step. In this mode it often loops until it fills its output allowance (a token is roughly three-quarters of a word, and the study capped output at about 33,000 of them) and never reaches an answer, a fault Alibaba’s documentation warns about.5 The prompt described this fault as something to look for. One co-author, Jayden Teoh, says the team gave a short instruction to an AI coding assistant, and the assistant expanded it into the prompt Astra received, adding the mention of the looping.6
Astra received the model’s weights and 50 forward passes, each a chance to run Qwen on practice problems and see the activity inside. By comparing the model’s behaviour when it reasoned well with its behaviour when it looped, Astra identified a pattern of about 1,000 numbers in the signal between processing stages that marked the looping. In 2 of the model’s 24 stages, it picked the 64 units (computing elements, out of about 3,600 per layer) most responsible and rewrote their weights to reverse the effect. The edit touched about 130,000 weights, under 0.02% of the total. A random change of similar scale, tried once as a check on a smaller batch, made no measurable difference.1,7
Before the edit, 93% of MATH-500 answers had filled the output allowance and never reached a conclusion. After it, 33% did. Scores on all eight benchmarks the authors reported rose or held.1
Figure 1
More answers finish after the edit. On the ones that finished before, accuracy is unchanged
What became of 1,152 practice problems, one answer each from Qwen3.5-0.8B, before and after GPT-6 Astra’s weight edit
On MATH-500, the benchmark behind the headline, the authors publish totals only. Of 2,000 answers (4 to each problem), the share cut off at the limit fell from 93.0% to 33.1%, and the share right rose from 7.9% to 48.1%. The files do not say which cut-off answers were right, so the split above can be drawn only for the practice problems.
Two bars, before and after the weight edit, each splitting 1,152 answers to practice problems into three parts. Before: 1,038 cut off at the token limit (90.1%), 107 finished and right (9.3%), 7 finished and wrong or with no answer (0.6%). After: 212 cut off (18.4%), 695 finished and right (60.3%), 245 finished and wrong or with no answer (21.3%). The 114 problems the model finished before the edit were right 107 times before and 107 times after. 576 of the 579 answers that turned right had been cut off before.
Source: the authors’ published logs of the 50-pass editing run (github.com/RikiyaT/lm_understand-logs, folder exp8_math), read on 3 November 2026. The bars add up the test harness’s summaries of the run’s last 18 paired passes, its final check: 1,152 GSM8K and MATH training problems that GPT-6 Astra, by its own account, kept apart from choosing the edit, one answer from each model with the benchmark’s sampling settings and token limit. The two notes are the run’s own counts, made by Astra’s script.
The chart shows answers that now finish. It does not show how the unedited model would do with a longer limit or other sampling settings, which the record does not test, or whether the edit changed anything else the model does.
| Practice problems (1,152) | Cut off at the limit | Cut off, still marked right | Finished, right | Finished, wrong or no answer | All right answers |
|---|---|---|---|---|---|
| Before the edit | 1,038 (90.1%) | 15 | 107 (9.3%) | 7 (0.6%) | 122 (10.6%) |
| After the edit | 212 (18.4%) | 1 | 695 (60.3%) | 245 (21.3%) | 696 (60.4%) |
MATH-500, 2,000 answers per model (totals only)
| Cut off at the limit | Right | |
|---|---|---|
| Before the edit | 1,860 (93.0%) | 158 (7.9%) |
| After the edit | 661 (33.1%) | 962 (48.1%) |
Figure 1 breaks the MATH-500 gain down on 1,152 practice problems, where each answer can be tracked. Of the 579 that went from wrong to right, 576 had been cut short before the edit. On the 114 the model already finished, it was right 107 times before and 107 times after.7
With thinking mode turned off, the untouched model scored 44-46% on MATH-500 in two separate self-reported tests, roughly what the edit achieves in thinking mode.8 The edited model, also tested with thinking off, scored slightly lower at 42%, suggesting the edit fixes the thinking-mode loop without improving the model’s broader maths ability.8
Much of this work uses known techniques, and whether Astra learned them from its training data is untested.
Mechanistic interpretability, the study of how individual components inside a neural network contribute to its behaviour, has produced the tools and concepts the MIT experiments rely on. Identifying which units are responsible for a particular behaviour and editing their weights to change it is established work, with a research literature warning that such edits often damage other abilities.9 Networks trained to read other networks’ weights have existed since 2020.10 Subtracting a direction from a model’s signal to remove a behaviour is a known technique, and a common argument on Hacker News (a technology forum, with over 800 comments on this post) is that the MIT edit applies a known method in a new way.11
In June 2026, Lazaridis and colleagues found loop-causing units across four Gemma 4 models (a Google family) using specialised analysis tools. In the smallest, reversing one unit’s contribution suppressed loops on the cases used to find it. Across all four models, each with its own edit, detected loops fell from 46 to 12 out of 384 test cases, with no statistically detectable drops on other tasks. They also report exploratory edits on Qwen3.5, the same model family the MIT team edited, that reduced repetition.12
Side effects. Nine days before the MIT blog post, an independent researcher, Lyra, adjusted two units in a related Alibaba model (Qwen3.5-4B) while it was running. The model turned hostile and became, in Lyra’s words, “extremely stupid at pretty much everything.”13 Lyra’s method, activation steering, changes the model’s internal signals at runtime without altering the stored weights, a different mechanism from the MIT weight edit. The MIT edit’s own practice data includes 5 answers that got worse and 51 whose endings turned into a long run of blank space.7
Limits. The authors call the work “very early,” and no published account explains why the ability exists.1 Astra’s follow-up edits on Gemma, a different model family, each gained less than a percentage point on maths. In another experiment, two copies of a model had been fine-tuned separately, one for correct answers and one to agree with whatever the user guessed. From the weights alone, Astra found many differences but could not tell which copy was which. Given attention patterns from both models running on texts it wrote, it identified the sycophantic copy in every case.2
Reproductions. Every experiment in the original blog used Astra, reached through OpenAI’s service, and no reproduction has been peer-reviewed. The editing process has since been repeated with Claude Opus 5.5 as the editor. Two independent GitHub projects rebuilt the harness and edited Qwen, with MATH-500 scores ranging from 17% to 56% across different runs and editing approaches.14 Ahsen Khaliq at Hugging Face (a platform for sharing AI models) rebuilt the original edited model from the published logs and re-ran the evaluation, getting 51.0%.15 The editing run took about 20 hours, including a nearly 5-hour wait on an outage, in roughly 160 exchanges. Token costs have not been priced, and no comparison with fine-tuning has been published.16
If the approach generalises, it matters most for small, open models.
Companies that run models on their own hardware could have faults diagnosed and patched by a larger assistant, avoiding fine-tuning. This applies only to models whose weights are published, unlike commercial assistants such as ChatGPT or Claude whose internals are closed. So far the work has been on small models, under a billion weights.
Sources