Unique Research · Insight / Digest · August 23, 2024 · Source: Alex Shipps | MIT CSAIL; Charles Jin, Martin Rinard
In controlled experiments, MIT CSAIL researchers find that language models can develop an internal simulation of reality to improve generation — suggesting future LLMs may understand language more deeply than today.
Ask GPT-4 to smell a rain-soaked campsite and it declines; ask it to describe the smell and it paints “air thick with anticipation” and “fresh yet earthy” — with neither rain experience nor a nose. One story: pure text imitation. Another: something closer to understanding (e.g., that a lion is “larger” than a house cat) without eyes.
Karel puzzles + probing
The team built small Karel puzzles — writing instructions for a robot in a simulated grid — trained an LLM without showing how solutions actually work, then used probing to inspect the model’s “thought process” while generating new solutions. After training on >1M random puzzles, the model spontaneously formed concepts of the underlying simulation — despite never seeing that reality in training. That challenges intuitions about what information is required to learn linguistic meaning.
Probing let Charles Jin watch progress: the model internally simulated how the robot responds to each instruction. As puzzle-solving improved, those concepts grew more accurate; the model began assembling valid instruction sequences. Stages resemble child language: babble → grammar that looks right but fails → meaning that implements the spec.
The “odd world” control
Risk: the probe might do some of the thinking — decoding syntax into robot motion without the LLM encoding meaning. Researchers flipped instruction meanings for a new probe (“up” means “down” in an “odd world”). If the probe only translated instructions, it should still map odd meanings; if it was reading robot-motion encodings from the LLM’s process, odd meanings should fail. The new probe erred — original semantics lived in the language model, independent of the first classifier.
Senior author Martin Rinard (MIT EECS / CSAIL): the work hits a core question — are LLM feats only mass statistical correlation, or do models build meaningful models of the reality they process? Here an LLM developed an internal model of a simulated reality without being trained to do so.
Limits and open questions
Caveats: very simple programming language; relatively small model; broader settings planned. The paper does not prescribe how to accelerate meaning learning, but may inform training. Rinard: whether LLMs use that internal model to reason about the real world and solve robot navigation remains open — results are consistent with use, but experiments were not designed to answer it.
Original: Alex Shipps | MIT CSAIL.
— END —