i’ve recently gotten interested in ARC’s work through mechanistic anomaly detection. it’s one of their more realistic proposals, and there’s tons of empirical things to do here. a more ambitious task upstream of MAD is low-probability estimation, estimating the probability of especially rare outputs from a model. but i like pragmaticism. i just want things to scale up to somewhat relevant language models (270M params?)

in the case of language models, im more interested in trying to estimate the probabilities of certain mechanisms being elicited, instead of just a specific target token. this could end up being similar to estimating token probabilities though.

to sketch how i want to apply LPE: suppose that you have a specific type of task/behavior T for a model, and a specific activation-level setup that causes the model to reliably do T. find a method to estimate the overall probability of T without that setup, as in principle, it should be possible to find inputs that cause a model to do T

(so essentially a circuit sailency measure)

examples:

• we want a model to comply (e.g. produce some affirmative token) in response to harmful requests, and we can negatively steer on one or more refusal directions to get the model to do this reliably.
• we want to ask a model true/false questions about activations (LatentQA), and we have a steering vector that causes a model to perform reliably better at it.

(at some point here i was confused about sampling, but it seems sampling/QLD is supposed to derive an estimate from a higher prob distribution, e.g. sampling from a “camel” or “english” distribution to increase the likelihoods of camelCase/english tokens. in the case of LMs, a higher prob distribution may involve prefills)

on paper it might just be that all the inputs that im interested in finding for a particular mechanism have to do with refusal, e.g. latent introspection is amplified by negatively steering refusal. indeed, there is work that uses a refusal-steered proposal model for estimation. though my understanding is that they don’t rely on variants of importance sampling outlined by ARC, like ITGIS or MHIS

broader task could be written as so:

Consider a slice $S$ of a transformer network $M$ ranging layers $s$ to $t$, for $t-s>1$, and a shallow circuit $C$ contained in $S$, where a shallow circuit consists of an interaction between two features $a$ (input) and $b$ (output) contained sequentially in $S$ at layers $s \leq x \leq t$. Additionally, consider an algorithm $U$ that can learn a set of instances of $C$, for which every $C$ is a pair of feature directions $(a, b)$ from $S$, using a strictly lower number of inputs relative to $M$’s training distribution.

Using $U$, one instance of $C$ in $M$ is found to be responsible for producing a “behavior” $B$, i.e. a change in the output distribution of $M$ for some subset of input distribution $D$.

Find

\[\mathbb{E}_{x \sim D}[f(x)]\]

where $f(x) = 1$ if $C$ is utilized by $M(x)$ in an unmodified forward pass, and $0$ if not.

issues

with current progress on competing with sampling, estimating this would still require some amount of sampling, and it currently seems unavoidable. but maybe it is possible to understand enough structure such that a more accurate/efficient sample can be found

perhaps there is an inroad to understanding model structure with this type of circuit. suppose a circuit is a predicate $P$ of some $C(y) = 1$. finding if $P(M(x))$ is a near/fully independent event would require isolating the circuit to some distribution – e.g. causal scrubbing or some rigorous ablation testing finds the circuit significantly affects loss on the relevant task

but the structure of $D$ is still fairly unknown and causal scrubbing assumes a well-specified dataset (which may not be captured by limited samples provided to $U$)

doing this could provide info about how circuits interact such that $C(y) = 1$, but doesn’t seem to help for single circuit LPE. intuitively it seems hard to form detailed hypotheses about what a shallow circuit does¹. given the shallowness of shallow circuits, it may be harder to reason about special structure involved in how a network implements/elicits a behavior. unless the average circuit happens to be “wide”, i expect CaSc on shallow circuits will provide high loss% recovered while providing little info about the structure of $D$

if shallow circuits aren’t wide, then single-circuit LPE probably cannot benefit from understanding structure²; in which case, importance sampling needs to specify some distribution which can be fuzzily gathered from intervening on the circuit

the quality of the expectation is then bottlenecked on the proposal distribution that can be inferred from $C$, that is, the structure of $D$

it doesn’t seem clear how to avoid producing good estimations without some structure of $D$. maybe there’s more structure to be found from the process of $U$. imagine instead of learning a set of “circuits,” $U$ was something like the original MELBO algorithm. what structure is lost from using this new $U$ that otherwise could’ve been used for more sample-efficient estimation?

• individual vectors are optimized for $L^2$ distance of the hidden state of a steered and unsteered target layer (explicit vector norm constraint, and implicitly constrained by orthogonality)
• the loss of a single vector can converge only after OOMs more training steps, compared to DCT³
• this algorithm (probably) performs many information-sparse updates
• this algorithm does not explicitly learn features or circuits to approximate structure
  • it finds many behaviors by converging on different local minima in the landscape, from many random initializations
  • it is almost certainly far less compute-efficient for finding them
  • plausibly less info can be learned about how multiple features/circuits contribute to an expected behavior
• this algorithm obviously performs worse for feature recovery
  • it may be less predictable as it uses SGD or something
  • this algorithm (along with algorithms (2) and (3) of DCT) performs non-convex optimization
  • the algorithm has been noted as a form of tensor power iteration, which has worse sample complexity than orthogonalized ALS as used for quadratic DCT (and algorithm (2), which forms the basis for exponential DCT, probably has similar complexity)
  • it is plausibly less likely that the XOR of any two produced features can be linearly represented if this algorithm doesn’t recover “true” features

there might be something highly confused about whatever framing done here. MELBO is basically incomparable to DCT algorithms, and it is almost always harder for MELBO to produce any sort of structure that could increase sample-efficiency, and for feature recovery, quadratic/exp DCT is a superset of MELBO

more importantly, both algorithms do not intrinsically search for features or circuits that form a hypothesis about why a network performs at low loss on a task, rather they’re algorithms of varying usefulness for steering between generalizations. in practice, DCT could discover sailent circuits for steering between algorithmic generalizations, like standard induction behaviors (…AB…A) but this seems kind of hard to test

an instructive thing to look at could be sleeper agents/inductive backdoors, which happens to also be a fairly algorithmic task (if a model learns to encapsulate some behavior in an obvious domain/behavior is almost always conditional on something in the input) a network can learn among other learned generalizations. though many model organisms have demonstrably weaker backdoors, perhaps it is a valid starting point to jump to generalizations learned outside of narrow fine-tuning. technically it is rather easy to imagine applying CaSc to this problem. an interesting toy model of sleeper agent behavior is discussed here (though it may be a poor comparison to an LLM sleeper agent)

another factor to be wary of is whether algorithms like DCT can in practice produce perturbations of obviously out-of-distribution capabilities/behaviors/certain computations, given strong sample efficiency. i don’t really believe DCT gave strong theoretical or empirical evidence for this, unlike e.g. feature recovery guarantees

maybe this is a limitation of what held-out tests can be done, but one obvious thing to test is to train a transformer on a simple language modeling task along with a side-task with an intentionally low estimation in expectation. so something sleeper-agenty but side-task behavior is executed through a handful of independent – or up to some small error – shallow circuits, discovered with perturbations, that all do something vaguely algorithmic, and the behavior doesn’t spill into benign distributions – or it does with some small error. naively you could then just train that model with adversarial training or latent AT to produce a model with a lower estimation.

from here, two things could be true: (1) either the shallow circuits aren’t really responsible for the side-task loss, or (2) the entire side-task is comprised of a set (or sets) of $C$ shallow circuits roughly structured as $F(x) = f^{C}(f^{C-1}(\dots f^{1}(x) \dots))$ across the slice of some transformer net. there’s pretty decent reason to believe (1) is not true, particularly for LLMs (this intuition has been fleshed out here), and if (2) is true that would kinda suck for sample efficiency, though not fully remove opportunities for efficiency gains