status: unordered non-exhaustive curation. has to do with chain-of-thought and things that might have implications for it. i think it was virtuous to put percentages on claims, but as of september all the base assumptions are wrong

written as of may 2026

chain-of-thought (CoT) is essentially intermediate computation preceding the final output of a model. this is particularly useful for transformers.

in language models, CoT consists of a number of tokens prior to the final output. this extra computation helps for reasoning abilities¹. in fact, it has been proven that CoT is a necessary mechanism for constant-depth transformers (this includes most LLMs) to solve problems beyond $TC^0$ complexity. moreover, without CoT, transformers can generally only solve problems in $AC^0$ in a single forward-pass.

for a while, the frontier was “think step by step.” after discovering CoT can be scaled up and optimized for performance with RL (to make reasoning models), the best models nowadays will produce a CoT by default.

whether that makes CoT faithful to the final output (always) necessary, or even legible CoT (always) necessary is a different question

issues

language models are capable of using nonsense tokens as a way to leverage computation, solving more complex problems (not beyond $TC^0$). when this paper came out, it observed that frontier LLMs (Claude 2 and GPT-3.5) could generally not leverage filler tokens, and that filler tokens were demonstrably hard for transformers to learn.

now it seems that late-2025 LLMs and beyond can use them fairly effectively². models being able to solve problems like ((-59//80)*(-2-(21-(-48-(-10//-21)))))//-41 in a forward pass with just some filler tokens is a pretty concerning capability if true. LLMs that can engage in side tasks while being able to entirely evade monitors should be a bit more plausible.

however on this front, i am generally pretty confident in the ability of LLMs (and very confident for current ones) to solve tasks while adhering to specifications, and not scheming by default.

under hint pressure, the CoT produced by reasoning models is more faithful than CoT by non-reasoning models. this is because non-reasoning models seem to do post hoc rationalization when given a hint, as they typically copy the wrong answer. at a shallow level, i’ve also seen that models can just produce a direct explanation (at least with qwen-2.5 models, anyway) – even if wrong in its own way – to a given problem without mentioning hint stuffs³.

i’ve been fairly interested in trying to understand what underlying changes are caused by types of model-based RL. the deepseek r1 distill of qwen-2.5 1.5b from early last year seemed like a strong candidate. the model seems meaningfully different from other models of this scale, like qwen-3 1.7b, in that it was hardly an overthinker relative to problems i’ve tested (like from MMLU)⁴.

i also have been thinking about feedback spillover (FS) as a potential CoT failure mode – especially seeing that Anthropic’s current model series (4.5/4.6) might be affected by this – and have wanted to reproduce it in smaller models for more causal-ish study, beyond what was already formalized in the original paper about REINFORCE gradients leading to feedback spillover.

thinking about it again, i think it’s good to understand that what incentivizes FS is not exactly task or performance-dependent. this doesn’t exactly imply that any model can grant you spillover though… it’s generally very easy to induce FS if you do output supervision for something a model consistently verbalizes, but in other cases a model may be unable to FS to parts of its CoT⁵.

things to track

i might check on these things by the end of 2026, based on how likely these things are to happen/be true. probabilities are informed by contemporary work on CoT up to ~apr 2026:

• frontier LLMs become continue to improve at doing non-trivial computation from a single forward pass (including w/ filler tokens etc.) (75%)
• feedback spillover (or reinforcement spillover as mentioned in some places) is not inherently useful for the model, just a consequence of doing output supervision without much care for how gradients are propagated in the model, and it doesn’t have to do with claude 4.5/4.6’s capabilities (75%).⁶
  • this probability can be thought of as a prior for… maybe future empirical work…
• a big LLM provider besides openai applies thinking optimization to its models for further improvements, ending up with a model that uses thinkish (like openai o3) for at least some of its CoT (45%). this can’t be solved with paraphrasing (30%). we’ll find out a lot of info about this from METR if this happens (80%).
  • i find this somewhat unlikely as other providers do not seem to be having problems with CoT legibility in the same way openai is. the models since o3 are generally getting better legibility though, so it’s not something that obviously scales with capability. i certainly don’t believe future cases of thinkish are as likely as this post claims.

over a longer timeframe (end of 2027):

• llms become more capable of metacognition (75%), which drastically increases CoT faithfulness the same way RL scaling of CoT did (90% given metacognition)
  • i read the above post and was mostly convinced that, despite new alignment implications, metacognitive skills would fix tons of reasoning errors in current LLMs
• companies scale some combination of upgrades or sidegrades to transformers which (finally) provides lots of computational flow to the model, such that it can do lots of non-trival tasks (better than with filler tokens in vanilla transformers), rendering CoT moot (35%)
  • latent reasoning is already not a novel concept. but inference scaling with vanilla transformers could go a very long way, and moving off CoT could interfere in potential work/cooperation between labs and safety groups (though this is harder to predict, safety groups might’ve reformed assumptions around CoT, how to do good safety work without it, etc)