status: exploratory and probably confused

metacognition is thinking about thinking, or knowing about cognitive processes, or being self-reflective. for humans, it’s considered necessary to become smart. on paper, ais don’t have to be competent at metacognition to become very smart, but it could definitely prove useful.

the closest forms of metacognition we’ve seen in ai has come from llms.

latent introspection

an important subtype of metacognition that llms can exhibit is introspection, i.e. limited-domain, fairly weak latent introspective capabilities. this has been primarily observed through models that can latently identify concept activation vectors applied to the residual stream at certain layers. really small models can do this task too. you can train models to identify concept vectors, which strongly generalizes. i also found a writeup about how rlhf impacts latent introspection.

how and why does this capability even exist though? one plausible explanation is mentioned in Lindsey’s work:

We also tested some base pretrained models on the same task. We found that they generally have a fairly high false positive rate, and none of them achieve greater-than-zero net task performance, indicating that post-training is key to eliciting strong introspective awareness.

…

This capability may have emerged to enable the model to develop a basic theory of mind of characters referenced in pretraining data, and the Assistant character during post-training. Modeling the mental states of characters is generally useful to being able to predict their behavior, and being able to explicitly report these mental states could allow the model to predict a character’s own self-reported thoughts.

this state modelling objective is similar to truesight; forming latent knowledge about entities (knowledge of the type of person that produced the text) mentioned in training data helps optimize CE loss.

the work on emotion concept vectors offers a similar explanation:

To predict the behavior of people in these documents effectively, representing their emotional states is likely helpful, as predicting what a person will say or do next often requires understanding their emotional state.

…

Even if AI developers do not intentionally train the LLM to represent the Assistant as exhibiting emotional behaviors, it may do so regardless, generalizing from its knowledge of humans and anthropomorphic characters that it learned during pretraining.

(this work confirms latent introspection is mediated by post-training, and refusal mechanisms allow for stronger introspection. this could be another example of a latent representation being reused by a different training stage)

being able to infer things about the activation-space seems like an important step towards general-purpose introspection, which is likely an important part of metacognition; more specifically, human-like metacognitive skills. the implications of this will be greater than previous observed self awareness like behavioral self-awareness or eval awareness¹

takeaways

• lms seem to contain many latent capabilities that are discovered after the fact
  • the ones that have been studied can be chalked up to “the base model needed this to better predict X, reducing loss” and “these capabilities are suppressed in post-training/whatever preference finetuning”
  • but there are probably pivotal capabilities that are yet to be found.. and this doesn’t seem scalable imv.. there could be use for MELBO and similar approaches to exhaustively search for latent capabilities
    • for example, could melbo find vectors for enhancing/reducing introspection, similar to explicitly learned vectors, from limited samples/weakly-related prompt distribution?
• there could plausibly be latent capabilities or knowledge in models that can be utilized without more pretraining/RLVR scaling, instead be elicited from finetuning.
  • this work looks like a pretty good example. even non-reasoning models finetuned on distilled reasoning improve at encoding correctness, that (to my knowledge) haven’t specifically underwent grpo/ppo rl. or this could just be an exception limited to reasoning

what does this have to do with metacognition?

metacognitive skills should primarily come out of getting models to form lots of relevant latent knowledge and capabilities, and actively exploit latent and internal states.

ive thought of some ways these skills could come about in language models. didn’t think particularly hard about them

• latent reasoning (e.g. coconut, looplm, codi) might be one of the most important sidegrades here, but afaik, none of these approaches have been scaled beyond 10B params, so hard to know. frontier labs have also remained silent on this
• critiquing stolen from quibbler
• while reading Current AIs seem pretty misaligned to me, i noticed that misalignment/underperformance were ascribed to approaching the end of the context window. might this just be a performance issue that mostly alleviates with more context?
• see risks of hyperintrospection

these might be the type of approaches that get closer to really good performance on tasks that can at least be concretely validated. stuff like fully automated interpretability research. because i think metacognitive skills will come from basically eliciting latent knowledge, any of the three points are plausible to me:

• models continue to form latent capabilities without taking use of them, or only able to express capabilities weakly
• there might be dedicated efforts to utilize latent capabilities and knowledge to get models to try even harder at tasks, or maybe there might be future schemes and setups that happen to get models to utilize them (e.g. strong context management, latent reasoning)
• or metacognitive skills are never pursued and scaling is all you need (and they happen to come by default anyways)

stuff to do

what’s being measured?

in typical language models, try different mostly-unsupervised techniques to find steerable, out-of-distribution capabilities. the assumption is that there is a large chunk of latent capabilities that go completely unused

one good outcome would be to discover a chunk of introspective capabilities that intervene, and allow us to intervene, on how models predict certain variables from activation-space. there are surely limitations im not aware of, but given accuracy at reporting presence of concept vectors, and ability to steer to elicit/amplify capabilities (including introspective ones!)², there might be more latent capabilities hiding underneath the surface. this might include stuff like: predicting inference hyperparameters, probability of $n$th most likely token, correctness

another good outcome would be that there are generic mechanisms for “model introspection,” so any search that tries to look for introspective capabilities can be narrowed significantly

some ideas involve SAE decompositions as a basis. there’s good reason to believe that SAEs can help discover interpretable latent capabilites and attribute elicited capabilities to inputs, but they could be unhelpful for ood/weird capabilities. replace SAEs with transcoders wherever necessary

i refer to MELBO based on its original objective, but deep causal transcoding could be much better, especially as DCTs seem to focus more on feature interactions. it might make intervening on the loss function a bit harder to do things like produce similar features across models, but it would still make sense to interpret elicited latent representations post-hoc with SAEs and such. replace MELBO with DCT/DCTs wherever necessary

for models trained with latent reasoning techniques, it’s natural to assume that latent reasoning can elicit otherwise latent capabilities; so try testing this assumption

capability elicitation with little/no supervision

• use gradient-based methods to discover latent capabilities/capability influences that were already found using non-gradient vectors (in takeaways)
• have MELBO produce vectors that maximize downstream $Δact$ in two models and cross-evaluate their behaviors
• alternatively, maximize downstream $Δact_A$ and minimize $Δact_B$
  • there have been cases of models learning to use existing latent capabilities. for example, backtracking– might this be a case where $Δact_{reasoning}$ is large but $Δact_{base}$ is small
• identify performance gaps when restricting vectors by other methods
  • model vs. activation oracle to identify interpretable/non-interpretable vectors?³ perhaps there are oracle layers for showing understanding/not understanding; or large downstream $Δact_{oracle}$ indicates understanding?
  • what performance gaps exist when you require vectors to cause large downstream $Δact_{SAE}$?
• throw llms at these tasks

explaining weird capabilities

techniques like MELBO could produce optimal vectors that cannot be practically explained. for example, the discovered behavior is formed from an extremely small subset of the training distribution, and it’s hard to tell if it’s a capability

• just try activation oracles (AOs)
• just try autoencoder-style approaches to reconstruct activations from natural language descriptions (cycle-consistent AOs), extended such that description faithfulness is somehow encouraged… or NLAs
  • until some performance tweaks happen with NLAs, i do not expect this to be worthwile

here’s another rough sketch for a method to explain vectors

1. find inputs that produce base-SAE feature activations similar to the vector’s feature activations; for example, starting from top-activating examples for every feature and using some perturbation/search process to find optimal inputs
2. finetune the model using optimized inputs and labels produced by the steered model
3. hope it all generalizes⁴, and the finetuned model can verbalize its newly elicited behaviors (either by behavioral self-awareness or by introspection adapters)

i later noticed this sketch is a little similar to low probability estimation, and have written out some other ideas which hopefully don’t skew crackpotty

some combination of optimizing on SAE feature similarity and text-to-activation reconstruction could make this technique more useful

overall, even being able to distinguish capability-eliciting vectors from behavior-eliciting vectors would be useful (and hopefully such a technique could fit into the vector optimization process)

elicitation performance between finetuning and learned vectors

it’s uncontroversial that pre-training builds new knowledge while post-training elicits it. but even though there are non-trivial latent capabilities that can be elicited without finetuning², not every capability might be easily elicted with steering vectors

if you know that a capability can be elicited through finetuning, but learned vectors fail or only weakly elicit that capability:

• gradient descent on a vector wrt loss on a well-constructed eval, or performance loss with a finetuned model (which could either work, or produces cases of goodharting)
• compare between learned vectors, learned steering adapters, and the above
• how similar are SAE feature activations across these approaches?

latent reasoning models

• apply MELBO to looped language models like ouro and see what happens
  • how do you do logit lens-type analysis with an arbitrary amount of hidden layers?
  • are capabilities like latent introspection stronger with this approach?
  • steering vectors to maximize downstream $Δact_{steps>1}$ vs $Δact_{steps=1}$?

so what’s the theory of change

much of this text is motivated by the (fairly quiet relative to the rest of safety imv) goal of anti-slop training, to incorporate stronger decision making skills and epistemics (to lead to strategic competence hopefully) – which is possibly just an elicitation problem, as the methods i’ve tried describing assume. this way of going about things could be totally screwed though, maybe because the methods are just a bunch of patchwork, or we are very confused about the true ontology of ais, or the problem of finding the true structure of metacognition is actually just downstream of natural latents, though trying to incorporate good epistemic and decision-making skills into LLMs might be a different/simpler problem than for any arbitrary system. you can orient around the (extremely simplified) dream that metacognitive skills might just be a pile of heuristics once learned reasonably well over tons of internet text, which would be cool, but it’s probably underdetermined given the signs of metacognitive skills in LLMs now. maybe it has to do with motivations