| Methodological move | Paper | |
|---|---|---|
| 🟪 | Hybrid LLM + explicit optimizer in a language game — LLM generates candidate utterances or intents; an optimizer (planner, RL policy) selects or constrains them. | Meta FAIR 2022 |
| 🟪 | (same move, different decomposition) | Xu 2024 |
| 🟨 | Pure RL inside a classical game — strip the agent down to a model-free learner, characterise its behaviour against the game-theoretic benchmark. | Dolgopolov 2024 |
| ⬛ | RL as institution-designer — the RL agent does not play the game; it modifies the environment in which humans play. | McKee 2023 |
All three categories are new this week. The week’s move is to step out of LLM-simulation and into the older multi-agent-RL tradition, then ask which problems still need the language model in the loop.
🐍 Notebooks: 6_dolgopolov_pd.ipynb · 6_werewolf_rl.ipynb
Meta FAIR 2022 — CICERO — is the best place to begin the week’s hybrid-agent story because it makes the decomposition unusually explicit. CICERO is the whole hybrid system: a strategic planner computes an action and an intent, a dialogue model turns that intent into speech, and a consistency filter drops utterances that contradict the plan.
What is attractive about Figure 1 is this explicit separation of capabilities. CICERO does not treat strategy and language as one undifferentiated end-to-end problem. It separates a thinking function from a talking function: the strategic module represents the board symbolically, predicts what others are likely to do, and selects an action plus an intent; the dialogue model then realizes that intent in natural language. The board is not fed in as an image embedding. On the planning side it is encoded as structured game features; on the dialogue side the game state and dialogue history are rendered into textual or structured context. The key architectural idea is therefore not vision, but a narrow handoff from strategic reasoning to speech.
The training story matters for understanding that handoff. Human WebDiplomacy data is used to train both the dialogue model and human-like anchor models; self-play RL then improves the policy and value side of the planner without letting it drift too far from human conventions. At test time, the planner uses those components to choose an action and a recipient-specific intent, and the dialogue model turns that intent into a message.
CICERO = human data + RL planning + LLM dialogue
TRAINING
human Diplomacy games (WebDiplomacy data)
|
+----------+----------+
| |
v v
behavioral cloning DIALOGUE MODEL
/ human anchor models / intent-grounded LM
| |
v |
self-play RL improves |
policy + value models |
| |
+----------+----------+
|
v
STRATEGIC PLANNER at test time
PLAY / INFERENCE
board state + order history + dialogue so far
|
v
+======================================+
| CICERO |
| |
+--------------------------------------+
| STRATEGIC PLANNER |
| -> predict others |
| -> choose action |
| -> choose recipient-specific intent |
+--------------------------------------+
|
v
chosen intent from planner
|
v
+--------------------------------------+
| DIALOGUE MODEL |
| input: state + dialogue context + intent |
| -> negotiation message |
+--------------------------------------+
|
v
filters
|
v
final message sent
+======================================+
The game is Diplomacy with natural-language negotiation, played online against humans. CICERO reached the top 10% of human players, won an eight-game tournament, and was not flagged as a bot. This is close to a behavioral Turing test: sustained natural-language negotiation under strategic pressure.
Here “partner” does not mean a fixed teammate. Diplomacy is a seven-player game of shifting alliances, so the relevant player is simply the current recipient of the message: the player whose cooperation CICERO is trying to secure, influence, or test in that moment.
Three social-science threads make CICERO legible.
The first is cheap talk in the tradition of Crawford & Sobel (1982). Diplomacy sits in the middle ground where interests partly overlap and partly conflict. CICERO’s architecture is a mechanised response: by committing to a plan and constraining language to that plan, it makes its messages more credible than unconstrained generation would.
The second is commitment in the tradition of Schelling (1960). CICERO gains credibility by constraining its own language to match a planner-derived intent.
The third is alliance dynamics in the tradition of Axelrod (1984) and Kreps 1982: repeated interaction makes cooperation and betrayal strategically meaningful.
The compact comparison to keep in mind is this: FAIR does not replace RL with planning; it places RL inside planning. Xu uses RL as a selector over LLM-generated candidates, whereas FAIR uses RL-trained value and policy models inside a planner that first chooses a strategic intent, which the language model then verbalizes. In shorthand: Xu lets the LLM write the menu and RL pick from it; FAIR lets the planner choose the strategy and the LLM say it.
A reader carrying the Week-3 vocabulary forward will notice that the echo-of-the-literature concern returns here. The strongest claim these papers can make is that with the architectural scaffold, the system plays at a level that pure prompting cannot reach. That is what both papers demonstrate.
Xu 2024 is the cleanest case for a decomposition that recurs across the agentic-LLM literature: let the language model generate, let an optimizer select. The motivating example is rock-paper-scissors. GPT-4 can explain the correct mixed strategy and still over-pick Rock in actual play. The key point is simple: the model can say the right strategy and still choose actions in a biased way. The paper’s response is to split candidate generation from action selection and train only the selector.
The testbed is a seven-player text-based Werewolf game — two Werewolves, one Seer, one Doctor, three Villagers. Discussion is free-form language; voting is simultaneous. Three modules sit on top of the base LLM.
observation history
│
▼
┌─────────────────────────────────────────┐
│ **Hidden-role deduction** │
│ facts · likely truths · deceptions │
└─────────────────────┬───────────────────┘
│
▼
┌─────────────────────────────────────────┐
│ **Diverse action generation** │
│ 3 candidate actions for this turn │
└─────────────────────┬───────────────────┘
│
▼
┌─────────────────────────────────────────┐
│ **RL policy (MAPPO)** │
│ scores the 3 candidates using │
│ state features + text embeddings │
└─────────────────────┬───────────────────┘
│
▼
chosen action
The right intuition is this: the LLM writes the menu; the RL policy chooses from the menu. At any moment in the game, the LLM proposes three plausible moves, and the RL policy assigns probabilities to those candidates and picks one.
MAPPO is the training method for that selector. At this level, it is enough to say: MAPPO plays many games, observes which choices led to better team outcomes, and makes those kinds of choices more likely next time. The “MA” stands for multi-agent.
The RL part is not learning language from scratch. It is learning which kind of candidate tends to work better in which situation.
Two methods matter most. Population-based training means the agents also face a pool of differently styled agents, which makes the learned policy less brittle. Embeddings mean that the observation and each candidate action are turned into vectors the policy can compare: text becomes numbers the RL policy can score.
The most useful figure is Figure 4, because it shows what learning looks like here. The paper interprets the policy through the action probabilities it produces. In Figure 4(a), the bars are first-night kill probabilities. In Figure 4(b), the bars are self versus others for the Doctor’s first-night save; after training, the agent puts about 0.94 on self. In Figure 4(c), the Villager becomes less passive and more likely to vote decisively when two players both claim to be the Seer. The evidence that RL learned something meaningful is behavioural: the action distribution changes in strategically sensible ways.
The results are three. First, the learned action distributions look strategically better — more uniform when unpredictability helps, more concentrated when one move is best. Second, in a round-robin against ReAct, ReCon, Concurrent, and Atomic baselines, the trained agent wins both as Werewolves and as Villagers. Third, a 160-participant human evaluation reports roughly human-level play, including against previously unseen human partners.
The cleanest ablation is conceptual. Generating diverse candidates is not enough if the selector is still the LLM. Replace the RL selector with the LLM choosing among its own candidates and the bias returns. The selector is doing real work: it reshapes the action distribution under strategic pressure. The reusable lesson is: use the LLM for semantic expansion, and use an explicit optimizer to correct the strategically wrong action distribution the LLM would otherwise produce.
The paper’s weakest point is candidate coverage. The RL policy can only choose among actions the LLM proposed. If the best action is never generated, the selector cannot recover it. The diversity prompt does work, but the bound on the system is set by the candidate generator, not the selector.
Dolgopolov 2024 is the week’s purist paper. There is no language model, no neural network, and no humans — only two Q-learning agents facing each other in a one-shot Prisoner’s Dilemma, played repeatedly. Crucially the agents are stateless: each one’s entire brain is a flat two-element table, one value for “cooperate” and one for “defect,” together with a one-line rule for nudging those values toward whatever payoff was just received. There is no opponent identity, no round counter, no record of any particular past round. The only thing carried between rounds is a summary statistic of how good each action has been on average. The agent cannot distinguish “opp just defected against me” from “opp just cooperated with me,” and so cannot run tit-for-tat, grim trigger, or any other history-conditioning strategy. The contrast to keep in mind is the stateful version most readers will picture by default, where the agent stores a 2×2 table indexed by the partner’s last move — that one bit of memory is what lets tit-for-tat-style policies be representable at all. Dolgopolov deliberately forbids it.
The choice of Q-learning also deserves a sentence of its own, because the other two RL papers this week use a different family. Reinforcement learning splits into two main approaches. The policy-gradient family — PPO, MAPPO, the deep policies of McKee 2023 and Xu 2024 — parameterises the policy directly, usually as a neural network, then nudges the parameters in whatever direction increases expected reward. The value-based family, of which Q-learning is the canonical member, learns a value table $Q(s, a)$ instead and acts greedily on it. PPO scales to large state spaces but is opaque: you can run it and report what happened, you cannot easily prove when cooperation is the stable outcome as a function of the algorithm’s parameters. Tabular Q-learning is small enough to characterise analytically. Dolgopolov’s contribution takes the form of a theorem, not a benchmark table — and the choice of Q-learning is what makes the theorem reachable.
The result is a closed-form characterisation. In the relevant limit, the stochastically stable outcome is determined by the relationship between the learning rate and the game’s payoffs. Cooperation can emerge even without memory, but only when the learning rate is high enough relative to the temptation-to-defect margin. Otherwise the agents converge to Nash defection.
The paper fills a gap in the learning-in-games literature. Standard adaptive dynamics converge to Nash defection in the Prisoner’s Dilemma, while RL with memory can learn to cooperate. Dolgopolov shows that even RL without memory can cooperate, but only under a sharp relationship between learning rate and payoffs.
The proof technique extends Newton & Sawa (2015)’s machinery for matching games. The broader contribution is a characterisation of stochastic stability for this kind of learning process, not just the Prisoner’s Dilemma example.
How can cooperation emerge in the stateless case at all? The intuition runs as follows. In the long run, the system spends most of its time near one of two attractors: mutual defection (with $Q_D > Q_C$ for both agents) or a “cooperative” configuration that exists only when the learning rate is high. The cooperative configuration is created by rare, simultaneous exploration into C by both players. When that happens, both $Q_C$ values jump up by $\alpha \cdot (R - Q_C)$. With $\alpha$ large enough, that single shared exploration event is sufficient to push $Q_C$ above $Q_D$ for both players at once — and once both then act greedily, mutual cooperation persists for many rounds, reinforcing itself. With $\alpha$ small, the same shared exploration produces only a tiny nudge that gets washed out before the next coincidence can occur. The mathematical question Dolgopolov answers is which basin the system spends more time in, in the long-run distribution over states.
The social-science parent is the Roth-Erev adaptive-learning tradition. What Dolgopolov adds is an analytical closed form where much of the earlier literature relied on simulation. The cost is the strong “no memory” assumption.
The reason a small theoretical paper about Q-learning in the Prisoner’s Dilemma ended up in Games and Economic Behavior is that it speaks directly to one of the most active questions in current competition policy: can pricing algorithms tacitly collude on supra-competitive prices without any agreement between the firms that deployed them, simply through their joint learning dynamics? Three results make this a live worry. Calvano 2020 shows by simulation that two Q-learning pricing bots in a Bertrand competition game converge to supra-competitive prices — de facto collusion — without any communication, using Q-learners with state. Assad 2022 shows empirically that the adoption of pricing algorithms in the German retail gasoline market raised margins, with effects concentrated in stations whose competitors had also adopted algorithms. Klein (2021) characterises when algorithmic collusion is sustainable as a function of the algorithm’s parameters.
The legal problem these results raise is sharp. Classical antitrust doctrine requires evidence of an agreement between competitors. If two pricing bots learn to keep prices high simply by both being Q-learners with similar parameters, there is no agreement to point to — only a market outcome that looks collusive. What Dolgopolov adds is what the simulation literature could not give: a sharp analytical characterisation of when the collusive outcome must arise as a function of algorithm parameters, for the simplest possible (stateless) case. The promise of stripping memory away is that the result becomes analytically tractable and the regulator gets a precise condition rather than a worked example. The cost is that the no-memory assumption sits a step away from the algorithms actually deployed in markets. The closed form is useful as the base case of a research programme that progressively adds memory, state, and the complications real pricing systems carry — the regulatory analogue of the Konstanz simulation’s question about how coordination patterns emerge from independent agents’ learning dynamics in any domain.
A short companion notebook (JN/week6/6_dolgopolov_pd.ipynb) implements both endpoints. Two stateless Q-learners lock into mutual defection — one-step arithmetic already says they must, since the expected payoff of D exceeds that of C whatever the partner does. Two agents with one bit of memory and a discount factor settle into mutual cooperation. The notebook does not reproduce Dolgopolov’s analytical phase diagram, since naive simulation does not surface that regime cleanly; it makes visible the contrast between the two endpoints that Dolgopolov’s theorem then sharpens.
McKee 2023 is the most consequential move in the week. The RL agent is not a player. It is a social planner — a graph neural network trained by deep RL — that observes a network of human players and recommends which links to create or break. Humans decide whether to accept the recommendations. Across four conditions, the RL planner — GraphNet — produced the highest cooperation rate (77.7%) against a static network (42.8%), random recommendations (57.0%), and a cooperative-clustering baseline (61.2%).
t social planner's recommendation
│ │
▼ ▼
┌───────────────┐ cooperate/defect ┌────────────────────┐
│ 16 players │ ────────────────────► │ GraphNet │
│ on a network │ │ (graph NN policy) │
│ neighbour │ ◄──── recommend edge │ │
│ payoffs │ create / break └────────────────────┘
└───────┬───────┘
│ players accept / reject
▼
updated network at t+1
The setup is the network cooperation game in the Rand 2011 tradition. Players sit on a graph, choose to cooperate or defect each round, and the planner changes the network by recommending new or broken ties.
Training is done in simulation against synthetic human models, then the policy is tested on real participants. Cooperation rises across rounds under GraphNet, while under the baselines it falls or levels off lower.
The strategic content of the policy is the more interesting result. Earlier work built cooperative clusters by separating defectors from cooperators. GraphNet does not do this. It takes a conciliatory approach: early on it places defectors next to cooperators, trying to move them into neighbourhoods where cooperation becomes individually attractive.
The resulting networks have a core-periphery structure rather than the assortative cooperator clusters seen in earlier work. The same cooperative outcome arrives by a different network mechanism, and that mechanism is what the RL agent learned.
Read against the social-science canon, McKee is a deep-learning successor to the network-cooperation tradition of Centola, Rand, and Christakis, and to the institutional-design tradition of Ostrom (1990). The methodological lesson is that an RL agent can discover a non-obvious intervention strategy that human-designed baselines miss.
The paper’s vulnerability is simulation-to-human transfer. The cooperation rates are real; whether the conciliatory mechanism generalises beyond this game is the question Week 7 will press.
The week’s four papers span an axis that future weeks will return to: how much of the agent is language and how much is optimiser. Xu and FAIR are mixed; Dolgopolov is pure optimizer, no language; McKee is pure optimizer applied to human players.
language model optimizer agent acts on
────────────── ────────── ─────────────
🟪 FAIR (Diplomacy) dialogue planner + RL other agents + humans
🟪 Xu (Werewolf) generator RL selector other agents
🟨 Dolgopolov (RL-PD) — stateless Q other agents
⬛ McKee (Scaffolding) — graph-net RL human players, via network
The category labels make the methodological choices visible. A pure-RL learner suffices when the game is small enough to characterise analytically (Dolgopolov) or when the action space is structured enough for a graph policy (McKee). A language model becomes necessary when the action space is natural language (Xu, FAIR).
The week’s most reusable structural lesson is the LLM-plus-optimizer decomposition. The position the LLM occupies in the architecture is a design variable with consequences for what the system can claim.
A reader carrying the Week-3 vocabulary forward will notice that the missed-mixture question returns transformed. Dolgopolov gives the phase diagram analytically; McKee reports group means. A useful cross-paper question is whether the heterogeneity seen in human behavioural data also appears in RL-agent trajectories.