The Information Machine

Towards surfacing model algorithms with meta-tokens in the J-Space

Alignment Forum · agam_bhatia · 2026-07-20

Researchers apply the J-lens interpretability technique to Qwen3.6-27B and identify causally active "meta-tokens"—Chinese-language tokens representing concepts like "what does this mean" and "gcd"—that reveal the internal algorithms models use for disambiguation, arithmetic, and hedging behavior.

Open original ↗

Extraction

Topics: transformer-interpretabilitymechanistic-interpretabilityj-lensllm-algorithmsalignment-research

Claims

  • J-lens applied to Qwen3.6-27B surfaces "meta-tokens" that correspond to internal model algorithms, with the token "gcd" firing during LCM computation regardless of the surface form of the problem.
  • Swapping the GCD steering vector from the value 9 to 3 causally changes the model's LCM answer from 270 to 810, confirming that the model uses a product-divided-by-GCD algorithm internally.
  • Negatively steering disambiguation meta-tokens (Chinese phrases meaning "what does this mean") causes the model to respond to puns and jokes with literal, factual answers instead of recognizing wordplay.
  • Suppressing a hedging meta-token (大概率, "most likely/high probability") causes the model to commit to a single answer on open-ended questions rather than offering multiple options.
  • The single-token constraint of J-lens fundamentally limits how many meta-tokens can be discovered, suggesting multi-token J-lens methods are needed to find computations not lexicalized as individual tokens.

Key quotes

When the model reads ambiguous text, 什么意思 ("what does this mean") fires in the J-space, and steering it away makes the model answer "a boiled egg every morning is hard to beat" with nutrition facts instead of catching the pun.
Swapping the GCD vector from 9 to 3 makes the model change its answer for the LCM of 27 and 90 from 270 to 810.
We think the binding constraint is the J-lens itself: each direction reads out exactly one vocabulary item, so the discoverable meta-descriptions are bounded by the computations that happen to be lexicalized as a single token in Qwen's vocabulary.