# Astra's Qwen edit reveals how much a thinking loop can hide

**Summary:** Astra's direct weight edit raised Qwen3.5-0.8B's math score by reducing repetition. I see evidence for targeted model repair; new math ability remains unproven.

- Canonical: https://markhuang.ai/news/astra-qwen-weight-space-loop-repair
- Language: en
- Author: [Mark Huang](https://markhuang.ai/about)
- Published: 2026-11-02
- Section: News
- Tags: Weight-Space Programming, GPT-6 Astra, Qwen3.5-0.8B, Model Evaluation, Mechanistic Interpretability
- Source: [Rikiya Takehi and colleagues](https://rikiyat.github.io/blog/weight-space-programming/)
- License: https://creativecommons.org/licenses/by-nc/4.0/

---

![A copper loop reroutes through an amber junction inside a lattice of glass rods and ceramic nodes](https://cdn.markhuang.ai/news/astra-qwen-weight-space-loop-repair/hero.webp)

*Conceptual illustration generated with OpenAI imagegen: a local repair redirects a repeating path inside an otherwise intact network. This is a metaphor, not a depiction of Qwen's architecture.*

A small AI model can look bad at math because it keeps repeating itself before delivering an answer. In [Sparks of Weight-Space Programming](https://rikiyat.github.io/blog/weight-space-programming/), published November 2, 2026, MIT CSAIL researchers report that GPT-6 Astra edited Qwen3.5-0.8B's weights directly and raised its MATH-500 score from 7.9% to 48.1%. Weights are the numerical settings that determine how a neural network processes information. Here, Astra changed those settings without retraining the model.

I find the repair convincing. The authors' explanation is that Astra removed a tendency to loop, allowing Qwen to finish more of the answers it could already work out. If I were deciding whether to retrain a struggling model, I would first want to know whether it lacks knowledge or simply cannot finish its answers.

## The improvement came with fewer unfinished answers

In the authors' strongest Qwen run, the share of MATH-500 responses that hit the 32,768-token output limit fell from 93.0% to 33.1%. The original model frequently consumed its entire allowance of generated text. After the edit, many more responses reached an answer. The researchers kept the sampling settings the same for the comparison.

According to [the released experiment logs](https://github.com/RikiyaT/lm_understand-logs/tree/main/exp8_math), Astra traced the repetition to a pattern produced by two internal layers and edited selected parts of those layers to suppress it. The patch targeted 131,072 weight entries, about 0.016% of the model's weights. The logs distinguish entries targeted by the edit from entries that remained numerically different after rounding.

The gains extended to other tasks. The authors report that GSM8K, another math benchmark, rose from 13.8% to 67.1%, while IFEval, which tests instruction following, rose from 21.0% to 62.3%. That is consistent with removing a failure that disrupts several tasks. These are the researchers' measurements; I have not reproduced them.

Qwen's own [model card](https://huggingface.co/Qwen/Qwen3.5-0.8B/blob/main/README.md) warns that the 0.8B model is prone to thinking loops. It recommends tuning generation settings and detecting anomalous output through streaming. A model that spends most of this test stuck in unfinished responses has considerable room to improve just by finishing them. The reported gain does not tell us how the patch compares with the best configuration someone could find without editing weights.

## Finding the small patch took hours

Astra worked without training the target model, but it had substantial feedback: a Python workspace over the weights, practice problems with answers, and a budget for running the model. The researchers prohibited gradient descent, the usual procedure for adjusting weights from training errors. They did allow Astra to inspect results and revise its edits.

```mermaid
flowchart TD
    A["Inspect Qwen weights and practice problems"] --> B["Propose a direct weight edit"]
    B --> C["Use a budgeted model run for feedback"]
    C --> D["Analyze results and revise"]
    D --> B
    D --> E["Submit final edited weights"]
    E --> F["Researchers evaluate on tests hidden from Astra"]
```

The strongest Qwen experiment allowed 50 forward passes. Here, a pass meant a batch request, not one question: it could generate answers to as many as 512 practice problems or inspect internal activity on as many as 1,024 texts. Astra did not see the final evaluation benchmarks during editing.

The logs give a fuller picture of the work: 158 turns, 80 Python runs, and 20.3 hours from the first turn to the last. That elapsed time included 4.7 hours waiting through a model-access outage. It is not a clean estimate of compute time or cost. Finding this patch took a long sequence of inspections and revisions.

## Gemma's math errors resisted the same approach

The authors also tried to improve Gemma 3 270M, which rarely looped in their setup. In the [follow-up experiment](https://github.com/RikiyaT/lm_understand-logs/tree/main/exp8_math/gemma_followup), its MATH-500 score was 6.8% before editing and 7.0% after the run with 50 forward passes. None of the tested budgets produced an established improvement. Astra found ordinary reasoning mistakes without a single localized failure it could remove.

I take the Gemma result seriously. The successful Qwen edit removed an identifiable behavior that blocked useful answers. The experiments do not establish a general method for adding mathematical ability to a model that lacks it.

Reading and editing model internals also has a history. [Elhage and colleagues' 2021 transformer-circuits work](https://transformer-circuits.pub/2021/framework/index.html) analyzed computations directly from weights in small models. [Arditi and colleagues' 2024 study](https://arxiv.org/html/2406.11717v1) showed that a targeted weight edit could sharply alter refusal behavior. Here, a general-purpose LLM attempts that work itself. The recorded failures help show where its approach stops working.

## Start by finding out why the model fails

For a team working with a small model, my first move would be to separate wrong answers from unfinished ones. Then compare generation-setting changes against a weight edit on the same held-out tasks. Qwen's warning gives a concrete reason to try the simpler adjustment before committing to a repair process like this one.

I also want the checks to extend beyond the intended fix. In another experiment in the study, Astra failed in four attempts to identify a sycophantic model, one trained to follow the user's suggested answer even when wrong, from weights alone. It succeeded after the researchers supplied feedback showing where the models directed their attention. That failure matters when considering weight inspection as an audit. A tiny patch also cannot establish that every other behavior stayed intact.

As I argued in [writing about AI explanations and audit trails](https://markhuang.ai/news/llm-shows-work-hides-answer-cause), an explanation needs checks against observed behavior. Weight-space programming interests me because it could make specific failures easier to repair. In this Qwen experiment, reducing repetition let the model deliver far more useful answers. That is a worthwhile result even before anyone proves they can use the same approach to teach it new math.