status: a bit hand wavy but hoping to provide more details

to care about llms having better epistemics, it probably makes sense to care about motivations, and mechinterp might help towards training motivations in a principled manner, guiding improvements in epistemics and strategic competence. I sorta imagine llms could be capable of this but there’s an elicitation bottleneck/it hardly seems incentivized by default/more understanding of motivations is good anyways if we hope to use ais for work

to best knowledge, llms are probably myopic and don’t contain any mesaoptimizer doing explicit scheming/non-myopic things in pursuit of goals (unless there is some tipping point with post-training RL). the strongest cases for goal-guarding in LLMs are usually presented as backdoors, and while deceptive alignment seems to assume a mesaoptimizer, backdoors (as often used as an analogy for deceptive alignment) don’t require one. and even those backdoors aren’t guarded particularly well and they seem to depend on lots of hyperparameters. so fundamentally, the non-myopia basin seems pretty tiny for LLMs. claude 3 opus remains closest instance of non-myopic goal-guarding that nobody seems to know how to copy

if the descriptive model of llms as GLUTs of shallow circuits is right, llms likely do not implement a strong retargetable inner search process, and don’t easily follow non-myopic objectives, which has good (today’s LLM deception is weak) and bad implications (current deception is a quite unreliable model for the future + llms may be sandbagging on important categories of work because they primarily follow myopic objectives). so obviously from this info, the better ontology for objectives that llms follow is motivations rather than goals (which is not a fully new ideas but i wanted to rattle off some possible details about objects in this ontology). then the following may be true:

• a motivation is some construction (or set of constructions) of shallow circuits that is/are utilized across some task distribution, and this distribution doesn’t need to consist of a single human-interpretable concept (e.g. math or writing) and motivations could overlap in weird ways
  • the uniqueness of such a structure comes entirely from how it implements any part of the model’s pre/post-training distribution, otherwise it could be any generic computational thing. but still, this is still a more specific structure than a feature or smth
  • most motivational circuits are enforced in stages like rlhf where the model interacts with a reward model assigning loss, so maybe there is a way to ablate-then-measure motivations here, to avoid selecting either random noise or circuits that implement entire task-oriented capabilities
  • it is questionable whether such structures that elicit capabilities out of a pre-trained model should count as motivations (though intuitively they should count)
• they may act as computational variants of preferences, which are closer to details contained in the world model of a post-trained model (preferences aren’t necessarily more/less myopic), and models act mostly based on preferences
• some motivations are deeper/shallower than others and they aren’t necessarily separated by more/less myopia (writing ‘genuinely’ a lot is probably easier to satisfy than reward hacking in coding envs, which is probably easier than learning a set of capability elicitations in coding envs). a simple proxy could be computational depth with number of shallow circuits
• naively, any set of motivations are in conflict if each structure causes a significant shift in the output distribution (or some other latent variable) and are “active” for a given input dist
• shallow circuits might be a way to decompose motivations/preferences, and motivations/preferences are a way to decompose personas

one (obvious?) implication of this being true is that any 2+ collaborating llms with the same weights is not in principle any better than one llm at truth seeking, because the optimal strategy is just some set of motivations which exist in both models (practically speaking it is easy to use two instances in debate, but it might be easier to analyze a distilled model and predict more things about it)