some best guesses for what’s needed to elicit capabilities from llms that differentially improve macrostrategic and conceptual reasoning, while being scalable

• non-myopia
  • current llms are at a quite myopic baseline. their motivational structures very likely overemphasize shallow goals like satisfying some extrapolation of their reward model
    • suppose two ends: inherently non-myopic models for which you need to enforce myopia using e.g. MONA, and inherently myopic models for which “long-horizon” reasoning needs to be elicited by chunking problems into chains of short-horizon problems.
      • we are probably closer to the latter
      • does this benefit current llms? does problem chunking distill some broader strategies?
• in-context persona extrapolation
  • llms have more predictive power over other personas, they can extrapolate what long-run benefits other personas could receive, and simulate different ones accordingly
  • a cheap form of this might be shifting to personas in the direction of higher likelihood for task-relevant motivations, incorporating different motivations towards solving a larger task
  • llms may utilize some heuristics of appropriate personas to switch to, and realistically would look babble-and-pruny and choosing to simulate other personas would be more based on current context (which might still be better than baseline)
  • the most obvious way to get llms to use this ability well probably looks like directing towards non-myopic objectives
• ontology translation
  • current llms very likely can’t do this, and pre-training objectives are unlikely to produce this machinery
  • this could look something like circuitry that reads off some arbitrary contents in latent space, or some approximation/predicted model of prior activations, and holds some low-dimension summary which can be reported
  • this would not be sets of feature->action circuits
  • more exact form of modelling personas, since this would look more like a single persona eliciting predictor-level facts/details
  • i think this is a useful operationalization of what strong introspection might look like, but it doesn’t seem necessary or sufficient
  • relaxation: understand how llms can learn how to extract info from activations-as-input, factoring in existing inductive biases
• motivations
  • reward models seem to have non-trivial influence on final motivations
  • worst-case/ood scenarios should matter here, try to construct models with very robust backdoors triggered by different kinds of inputs or latents