Agent Infrastructure Deep Dive

Memory — the persistent state system

One line: Memory is the agent's "sustainable state system" — what lets it carry knowledge across turns and sessions beyond the context window.

Think of the agent as a person with a small desk and a big filing cabinet. The desk is the context window — what it can see right now, but it's tiny and gets wiped at the end of each session. The filing cabinet is long-term memory: a database it can write notes into and pull from later. Memory has three jobs — write (save a note), read/retrieve (find the right note and lay it on the desk), and manage (tidy the cabinet: merge duplicates, toss stale notes, fix contradictions). Reading is mostly solved; the hard parts are deciding what's worth writing down and keeping the cabinet clean — which is why many teams do the tidying overnight in a batch ("dreaming") rather than scribbling notes mid-conversation.

Key concepts to understand

• Layered design — split memory by lifespan and by type. By lifespan: working memory (what sits in the context window right now) versus a long-term store (a database that survives across sessions). By type: episodic memory (specific past events), semantic memory (stable facts), and procedural memory (reusable how-to rules and skills).
• The three core operations. Retrieval — deciding which stored memories are relevant now. Injection — physically placing those memories into the prompt. Update / write-back — persisting new information so it survives the session. Each is a separate engineering problem with its own failure modes.
• Evaluation axes. Recall relevance (does it surface the right memories?), conflict resolution (what happens when two memories disagree?), local / partial update versus full rewrite, and provenance / traceability (can you explain where a memory came from?).
• The self-improvement loop. A memory-rule mechanism distils raw experience into reusable rules; offline consolidation — the "nightly tide" — merges and de-duplicates memories on a schedule; experience extraction pulls lessons out of completed tasks so the agent slowly improves.

Skill System — loadable capabilities

One line: Skills are packaged, on-demand capabilities (instructions + optional scripts/resources) the agent discovers and pulls into context when relevant.

Picture a shelf of how-to binders. The agent doesn't read every binder up front — that would bury its tiny desk (context window) before any work starts. At startup it only reads the table of contents: each binder's title + one-line description (~60 tokens each). When a task matches, it pulls just that one binder off the shelf and reads it fully; deeper appendices load only if needed. This staged reveal is called progressive disclosure — and the savings are huge: ~8 skills cost about 500 tokens at startup instead of 70,000 if you'd dumped them all in.

Key concepts to understand

• The full skill lifecycle: manage (register and version available skills) → discover (agent recognises a skill is relevant) → load (read its instructions into context) → inject (apply them to the work) → unload (drop them to free space). The unload step is the trickiest to get right.
• Progressive disclosure is the core efficiency trick. At startup only each skill's name and one-line description sit in context (~60 tokens each). The full instruction body loads only when triggered, and deeper reference files load only if needed.
• A skill is fundamentally just files — a markdown instruction file plus optional scripts and resources — so it doesn't have to be bound to a sandbox. The same skill can be loaded independently of where code eventually runs.

Sandbox & Virtualization — safe execution environments

One line: The isolated environment where the agent runs tools/code without endangering the host — the biggest skill-builder if you've actually constructed one.

It's a disposable, sealed playroom — a little throwaway computer where the agent can run code, install things, and even break stuff without touching the real system. The lifecycle is about speed versus cost: cold start = building the playroom from scratch (slow); snapshot = taking a photo of a fully set-up room so you can recreate it instantly; resume = reopening a paused room; idle = a room sitting empty, which you pause to stop paying for it. A MicroVM is a playroom with bank-vault walls (genuine isolation) that still opens in well under a second.

Key concepts to understand

• Lifecycle states and their cost / latency trade-offs. Cold start = building fresh (slowest, most expensive). Resume = reopening a paused environment (much faster). Snapshot = saving a frozen copy so it can be recreated almost instantly. Idle = running-but-unused; pause it to stop paying.
• Distributed scheduling. In production you're juggling thousands of environments across many machines — you need a scheduler that places, balances, and recycles them. Pooling (keeping a stock of ready environments) and pre-warming (booting before anyone asks) are the standard ways to hide cold-start latency.
• MicroVM isolation (Firecracker-style). Genuine virtual-machine-level security with near-container startup speed. Snapshotting the entire machine state enables fast resume, and pausing idle sessions reclaims cost.

Framework Comparison — Claude Code vs. Codex vs. OpenClaw

One line: Know each framework's design philosophy and where it's strong/weak — and the deeper axis: model-driven versus role-driven orchestration.

These are the "operating systems" for coding agents — each one is a different opinion on how to run an agent. Claude Code is fast and highly controllable (lots of knobs: hooks, sub-agents, skills). Codex is prized for a disciplined run-loop and clean code structure. OpenClaw is a personal Swiss-army-knife that plugs into all your chat apps via one gateway — flexible, but the sprawling plugin ecosystem gets messy and risky to maintain. The deeper question: who decides how to split up the work? Fixed, human-designed roles (Claude Code, Codex) or the model itself deciding on the fly (Kimi Swarm, #11).

The Harness — THE central concept

One line: The engineering shell around the model that makes an agent reliably finish tasks — everything that is not the model weights.

The model is a brilliant but forgetful contractor; the harness is the entire job site that makes them productive. The schedule that orders the work, the toolbox with safety locks, the checklist, the supervisor who re-checks the output, and the system for handing off across shifts — that's the harness. Drop a genius contractor onto a site with none of that and they wander off, repeat mistakes, or blow the budget. The key insight: as models get smarter, the job site is increasingly what decides whether the project actually ships — reliably, cheaply, and safely.

The seven things a harness owns

• Task scheduling — breaking a big goal into smaller subtasks (decomposition), deciding which agent or tool handles each one (dispatch), and choosing the order they run in.
• Context control — assembling exactly what the model sees on each turn and compressing it when it grows too large. This is the single most active job the harness does.
• Action environment and tool safety — giving the model a safe place to act (sandbox), controlling what it's allowed to touch (permissions), and wrapping dangerous operations in guardrails.
• Agent loop and collaboration — running the core think-act-observe cycle, and coordinating control flow whether there is one agent or several.
• Verification and the evaluation closed-loop — checking whether the work is actually correct and feeding results back into the next step, so the agent self-corrects.
• Self-evolution — letting the agent learn from its own past runs and improve over time; this is where Memory (topic 1) and Skills (topic 2) plug in.
• Long-horizon execution — keeping the agent coherent across tasks that run for hours or days without losing the thread (covered in depth in topic 9).

Loop Controllability — steering the agent loop

One line: How precisely you can steer, intercept and recover within the run-loop — the practical craft of a good harness.

An agent runs in a cycle: think → use a tool → read the result → think again. "Loop controllability" is how tightly you can supervise that cycle. The levers: build the prompt the same way every time (predictability); catch and retry when a tool returns garbage; drop in checkpoints (hooks) before/after each step; trim the conversation without confusing the model; pause for a human's OK on risky actions; and even start the model's answer for it to nudge direction (prefill).

The levers to understand

• Context-assembly stability — building the prompt the same deterministic, reproducible way every turn. Unstable prompt construction breaks debugging and keeps invalidating the cache (see topic 7).
• Tool-error control — validating what a tool returns, retrying when it fails, and degrading gracefully when output is malformed, instead of letting one bad result derail the entire run.
• Hooks and middleware — insertion points where you can intercept and act at fixed moments (before/after a tool call, before/after a turn) to log, modify, approve, or block what's happening.
• Compression strategy together with the key-value cache — shrinking a growing conversation without destroying the cache. Condense the old part while leaving the stable beginning untouched.
• Stop-reason handling plus human-in-the-loop — reading why the model stopped generating (finished, wants a tool call, or hit a length limit) and branching correctly on each case.
• Prefill intervention — steering the model by pre-writing the opening tokens of its response, which strongly nudges the direction or format of what it produces next.

Inference Optimization — prefill, decode & caching

One line: Understand the two phases and how the harness exploits them — this is why prefix stability matters so much.

Answering happens in two stages. Prefill is the model "reading" the whole prompt — fast, done in parallel, like skimming a page at a glance. Decode is "writing" the answer one word at a time — slower. The big money-saver is prompt caching: if the start of the prompt is identical to last time, the system reuses its earlier "reading" instead of re-doing it (a cache hit) — which is exactly why a good harness keeps the beginning of the prompt stable. Continuous batching squeezes many users together word-by-word to keep the GPU busy. Takeaway: stable prompt prefixes turn directly into lower cost and faster responses.

Key concepts to understand

• Prefill vs. decode. Prefill reads the entire prompt at once in parallel (compute-bound). Decode writes the answer one token at a time (memory-bandwidth-bound). A long prompt is cheap to read; a long answer is slow to write.
• Key-value cache hit (prefix caching). When the beginning of a prompt is identical to a previous request, the system reuses the cached attention data for that prefix. This is the single biggest lever for cutting agent latency and cost.
• Vendor-side cache management. Model providers expose caching controls through their APIs, so part of the job is designing prompts so the reusable portion stays in a fixed position.
• Continuous batching. A serving technique where many users' requests are grouped at the token level, keeping the GPU busy and raising overall throughput.

Post-training — fitting the model to the agent environment

One line: Making a base model fit the agentic environment — not deep model research, just enough to understand the mechanism.

A base model is a brilliant new-grad who's read the whole internet but never held this specific job. Post-training is the on-the-job training: first show it thousands of worked examples of doing the task the right way (supervised fine-tuning), then let it practice in a realistic setup and reward good outcomes (reinforcement learning) — for example, "did the code it wrote actually pass the tests?" The goal isn't to redesign the model; it's to make it fluent in your tools, your loop, and your conventions, and good at recovering when a tool fails.

Key concepts to understand

• Supervised fine-tuning on agent trajectories — showing the model thousands of recorded examples of an agent doing the task correctly (the right tool-call format, the loop conventions, how to recover after an error) so it learns to imitate good behaviour.
• Reinforcement learning in real environments — letting the model practise in realistic settings (a terminal, a GUI, a code repository) and rewarding good outcomes. A process reward model scores the quality of each individual step, not just the final result.
• Learning from live deployment traffic — for example, asynchronous reinforcement learning that turns ordinary production conversations into training signal, so the model keeps improving from real usage.

Long-horizon Tasks — keeping coherence over many steps

One line: Keeping an agent coherent over many steps / hours / days — the failure modes plus the compression playbook.

It's project management for a job that runs for days, not minutes. Four ways it derails: the agent forgets the original goal (goal drift), its notes pile up until the desk overflows (context explosion), nobody double-checks the work (weak verification), and small early mistakes snowball (error accumulation). The main fix is smart note-condensing (compaction): when the desk fills, summarize the old stuff into a tight recap and carry on. Real systems do this in tiers — light cleanup → full summary → a pre-saved notes file — and time it carefully, ideally at a natural task boundary.

The four failure modes

• Goal drift — over a long run the agent gradually forgets or quietly mutates the original objective.
• Context explosion — the window fills with stale, low-value detail, crowding out what actually matters now.
• Insufficient verification — without a closed loop that checks each result, errors pass through unnoticed.
• Error accumulation — small individual mistakes compound across many steps.

The compression playbook

• Compress by timing — before a turn (pre-turn), in the middle (mid-turn), or after (post-turn). Compressing mid-thought can lose fragile in-progress context.
• Compress by level — level 1: light trim (drop stale tool output); level 2: fuller summary written by the model; level 3: hard re-grounding that rebuilds working context from scratch. Claude Code's micro/full/session-memory tiers are a concrete example.
• Model-switch handoff — cleanly transferring task state when you swap models or hand off between agents mid-task.

Coding Agent Evaluation — benchmarks & eval pipelines

One line: Know the benchmarks and the eval pipeline that runs them — and why the harness is part of what's being measured.

It's a standardized exam for coding agents. SWE-bench hands the agent a real GitHub bug report and checks whether its fix makes the project's existing tests pass — graded automatically. The eval pipeline is the exam hall: spin up a fresh sandbox per question, apply the agent's patch, run the tests, tally the score (often "pass@k" — did it succeed on any of N tries). A critical insight: the same model can score very differently depending on the harness wrapped around it — so a benchmark number reflects the scaffolding as much as the model itself.

Key concepts to understand

• The benchmarks. SWE-bench (and its variants: Verified, Multimodal, Live) is the headline one. Terminal-Bench tests command-line tasks. Understanding what each actually measures matters more than reciting scores.
• The evaluation pipeline. For each task: spin up a fresh sandbox, apply the patch, run the test suite, score. Scores are often reported as "pass@k" — probability of success within k attempts.

Kimi Swarm — model-native orchestration

One line: Moonshot's Agent Swarm bakes multi-agent orchestration into the model — a concrete example of model-driven versus role-driven orchestration.

Instead of one worker grinding through a big job step-by-step, the model acts like a team lead that instantly "hires" a swarm of temp workers, hands each a slice of the task, lets them work in parallel, then stitches their results back together — and crucially, the model itself decides the split, not a fixed org chart. Kimi scaled this from ~100 parallel workers (K2.5) to ~300 (K2.6). Claw Groups lets workers on different devices and even different underlying models — plus humans — join the same job.

Key concepts to understand

• The core idea. The model itself breaks a complex task into pieces (self-decomposition), spawns specialised sub-agents to work those pieces in parallel, then synthesises their outputs — all without an external orchestration framework like LangGraph or CrewAI defining the workflow by hand.
• The scale jump. K2.5 coordinated ~100 sub-agents across ~1,500 steps; K2.6 raised that to ~300 sub-agents across ~4,000 coordinated steps, sustaining autonomous runs for many hours.
• Claw Groups. A shared workspace where sub-agents on different devices, different underlying models, and human collaborators all work on the same task — the model acts as an adaptive coordinator that assigns work by skill.
• The contrast to hold onto. Orchestration is model-driven here, versus the role-driven approach of Claude Code and Codex where humans pre-define the roles. Known weakness: the swarm can collapse back into a single-agent loop, and a meaningful share of tool calls failed during long loops — reliability is still an open problem.