1. Introduction
An agent working through a long conversation needs more than the latest question. It may need the original objective, constraints agreed earlier, an exception printed by a tool, or the exact change it has already applied. As messages and tool outputs accumulate, deciding what the agent sees becomes part of the system design.
This article uses sre-agent, an Internal AI product, as a concrete example. In the implementation described by the report, it manages history with micro_compact and auto_compact. The question is broader than this product: how can an agent keep a conversation going while retaining the evidence it needs to answer accurately?
We start with the problem, examine three ways of managing context, and then describe how to compare them. The central idea is to evaluate what remains usable after context is reduced, rather than treating a smaller context as a success by itself.
Evidence status. This is a method and benchmark design with an initial probe. The report contains results for two cases, not a completed comparison. The N latest strategy introduced here is an additional comparison arm proposed for evaluation; it has no measured results yet.
2. Problem statement
2.1 Conversation history and model context are different
The application can retain the full conversation while sending only a selected view to the model. Removing a message from that view does not necessarily delete it from storage. However, unless a retrieval mechanism exposes the omitted content, the agent cannot use it in its next answer.
For a tool-using agent, history includes user and assistant messages, structured tool calls, and tool results. One long log can consume more space than many short messages. A limit expressed as a number of messages therefore controls message count, rather than guaranteeing a token budget.
For example, the following JSON shows four history messages using the OpenAI chat-message format. The log is shortened for readability.
[
{"role": "user",
"content": "Run the tests and explain any failures."},
{"role": "assistant", "tool_calls": [{
"id": "call_001", "type": "function",
"function": {
"name": "execute_bash",
"arguments": "{\"command\": \"pytest -q\"}"
}
}]},
{"role": "tool", "tool_call_id": "call_001",
"content": "[log shortened] ... 1 failed, 49 passed in 8.42s"},
{"role": "assistant",
"content": "One test failed. I will inspect the traceback."}
]This history contains four messages whether the tool content is one line or a 10,000-line log. The tool_calls array belongs to the assistant message, and tool_call_id links the result to that call. Keeping the latest four messages would retain both versions, even though the longer log could require far more tokens. The arguments field is a JSON-encoded string, as expected by this message format.
2.2 What can disappear when context is reduced
Consider an agent that runs tests, sees an exception, edits a file, and later receives a follow-up question. The recent exchange may explain the current plan while the exact exception or first edit sits much earlier in history. Keeping the latest messages can discard that evidence. Summarizing the history can preserve the overall story while losing an exact count, identifier, or code block.
These losses have different consequences. An agent might correctly identify the changed file yet fail to reproduce the code it inserted. It might say that a test count is unknown, or state an incorrect count confidently. A useful evaluation must distinguish those outcomes and compare them with an agent that can still see the original evidence.
Figure 1. N latest omits older evidence, while a summary may lose exact details. This is an illustrative example, not a measured result.
2.3 The design objective
We want to reduce context while preserving the task, current state, relevant constraints, and access to exact evidence. The practical questions are what to keep verbatim, what to summarize, when to compact, and how the agent can recover details that no longer fit in its active context.
The rest of the article separates mechanism design from empirical results. A plausible design is a candidate to test, rather than proof that information loss has been solved.
3. Methods for managing a long conversation
3.1 Traditional approach with the latest N messages
The simplest policy sends the latest N messages and leaves older conversation messages out of the agent's context. There is no summary of the omitted history. System instructions and tool schemas remain separately available, and the current user question must remain in the input.
For example, if N is 20 and a conversation contains 100 eligible history messages, the agent receives the most recent 20. A constraint stated in message 5 is unavailable unless it is repeated later or explicitly retrieved. This policy is easy to implement and requires no summarizer call, but its loss of older information is determined by position rather than relevance.
Figure 2. N latest omits older history without creating a summary.
Tool protocols require an additional rule. A cut must not leave a tool result without its corresponding call, or keep a call while discarding part of its result group. For the proposed benchmark arm, N counts history messages before the appended question, and the boundary moves toward newer messages to exclude any incomplete tool exchange. The retained history can therefore contain fewer than N messages. The system prompt, tool schemas, and evaluation question are outside that count.
This definition must be recorded with each run. N latest messages, N latest turns, and a fixed token budget are different policies. If a selected suffix is still too large, a documented token safeguard must trim complete exchanges and report the actual retained size.
3.2 Compaction in the sre-agent example
The sre-agent implementation described in the report uses two mechanisms at different times. micro_compact reduces older tool outputs before model calls. auto_compact summarizes older history when the estimated context crosses a threshold.
micro_compact. Tool results from previous turns that exceed 500 characters are eligible for replacement with a fixed placeholder when they fall outside the newest max(5, current-turn tool-result count) results. This reduces the text sent to the model, but the cleared content is not recoverable from that placeholder in the shipped mechanism.
auto_compact. When estimated history exceeds 0.8 × context window, everything before the last user message is replaced by one LLM-generated summary. The last user message stays verbatim. The report describes a nine-section SRE prompt for producing this summary.
Figure 3. Compaction mechanisms in the sre-agent example.
This differs from N latest: a summary can carry information from much earlier in the conversation. Its coverage and precision still depend on what the summarizer preserves. The report identifies three weaknesses in the shipped setup: only the final user message is guaranteed to remain verbatim after auto compaction, cleared tool outputs require another tool execution to recover, and the summary prompt includes resources and tools specific to one product line.
3.3 Proposed improvements to the compaction mechanism
The benchmark adds optional mechanisms for preserving recent work and recovering exact evidence. These are experimental switches in the benchmark, not features established as deployed in sre-agent. They live in compaction/config.yml so that their effects can be tested separately.
A retained tail keeps a recent share of history verbatim instead of preserving only the final user message. A step-aware cut respects turn and tool-exchange boundaries. Offloading saves large or cleared tool outputs and leaves handles that the agent can open with read_tool_output. The summary then acts as a guide to evidence, rather than the only surviving representation of that evidence.
| Switch | Purpose |
|---|---|
retain_ratio | Keep a recent share of history verbatim. |
cut: step | Cut at whole turn and step boundaries measured in tokens; keep calls paired with results. |
prune_retried_failures | Drop failed calls that a later retry fixed. |
collapse_settled_retries | Reduce completed retry chains to their last step and a short note; keep open chains whole. |
offload_tool_outputs | Save cleared or summarized outputs and retain handles for read_tool_output. |
handle_guide: verify | Instruct the agent to check exact values against saved outputs and identify the source. |
prompt_paths | Choose the prompt used to summarize older history. |
Figure 4. The compaction paradigm with retry handling, output offloading, a retained tail, and evidence handles.
The example below applies this workflow to a bug-fixing conversation. S1 is the repository tree, S2 the reproduction output, S3 the first full test run, and S4 the source view. S5–S7 are tool errors; S8 runs the test and reports a failing assertion. Collapse keeps that final result and a retry note. At turn 2, the cut retains S10 and U2 as whole steps.
Figure 5. A worked example of conversation compaction, including retry collapse, output offloading, step-aware cutting, and evidence recovery.
For this illustration, the retained-tail budget is 3,000 tokens and S10 is assumed to have been shortened earlier to 2,900 tokens. The sizes are illustrative. Saved outputs remain accessible through read_tool_output; an active retry chain remains intact.
Settled retry chains can be collapsed during a turn so that repeated attempts do not displace useful results from the micro-compaction window. A chain that is still active must remain intact. Similarly, a handle only helps when the output was saved, the handle survives, and the agent actually reads it.
3.4 Prompt design
Prompt changes must preserve useful state without encoding rules aimed at particular benchmark questions. The report considers four prompt families.
| Prompt | Information it asks the summary to retain |
|---|---|
| sre-agent runbook prompt | Nine sections with examples and identifiers from a specific cloud product line and GMT+7. |
| generic | Nine sections without product, tool, timezone, or task assumptions; keep values associated with their subjects, distinguish repeated runs, and state unknowns. |
| hybrid | Generalized keep and drop rules, identifiers and key concepts, guidance for summarizing an existing summary, and a Done section separate from current work and next steps. |
| hybrid_v2 | The hybrid structure plus Changes Made, recording external changes, targets, action times, and how to undo changes. |
A gain on one coding benchmark is insufficient to establish that a prompt generalizes. A prompt change needs validation on another independent source before making that claim.
3.5 Expected tradeoffs before running the comparison
| Approach | What the model receives | Main tradeoff |
|---|---|---|
| Full history | Original history and current question. | Reference condition when the complete input fits. |
| N latest | Recent history suffix and current question. | Simple selection; older evidence is omitted and message count does not fix token size. |
| sre-agent compaction | Micro-compacted history, then a summary and last user message after auto compaction. | Earlier facts may survive in a summary, but exact tool content can be lost. |
| Compaction variants | Summary, retained recent tail, and saved-output handles when enabled. | More mechanisms and retrieval steps; preservation still needs measurement. |
These are design tradeoffs, not a ranking from completed experiments. The following section introduces the dataset and evaluation needed to compare them.
4. Comparison and evaluation
4.1 Dataset and why it is used
The report uses nebius/SWE-rebench-openhands-trajectories on Hugging Face: recorded OpenHands coding-agent runs on real GitHub issues in Python repositories. These trajectories contain structured tool calls, tool outputs, final patches, and task-test outcomes. This makes it possible to construct reference answers directly from recorded work rather than asking another model to invent them.
The source revision is pinned to 35455389ab51bf5e2306bfd436ef72d0f98bf882. The report lists 67,074 trajectories and the CC-BY-4.0 license. Tool names include execute_bash, str_replace_editor, think, task_tracker, and finish. The fields used are instance_id, repo, resolved, exit_status, model_patch, tools, and trajectory.
The hand-written synthetic SRE cases are excluded because their answers cannot be checked against an independent source. The report also considered nebius/SWE-agent-trajectories, but did not choose it because its tool calls are embedded as plain text, its runs are shorter, and its model coverage is narrower.
4.2 Case selection
compaction.swe.download checks the pinned revision, reads rows page by page, and selects the first rows satisfying the configured filters. The selected subset contains 20 cases from 20 repositories, taken from rows 12 to 117 of the source.
| Filter | Value | Reason |
|---|---|---|
| Resolved tasks | resolved_only: true | Use runs whose final patch passed task tests. |
| History size | 100,000 to 200,000 characters of text and call arguments | Select substantial histories that still fit in the full-history reference condition. |
| Questions per case | At least 3 | Ensure each case supports several recall checks. |
| Repository coverage | One trajectory per repository | Avoid measuring many variants of the same project. |
| Case count | 20 | Fixed evaluation subset. |
Across the subset, tool calls comprise 537 execute_bash, 377 str_replace_editor, 46 think, and 12 task_tracker calls. The repositories include sunpy, universe, sqlglot, time-machine, structlog, senza, pyserde, dnry-config, autopep8, pint, nose2, claripy, dvc, httpx, streamlink, jsonargparse, sqlfluff, python-tabulate, sepal_ui, and faker.
| Quantity | Minimum | Median | Maximum | Total |
|---|---|---|---|---|
| Estimated tokens per case | 20,644 | 29,611 | 39,368 | 601,757 |
| Messages per case | 70 | 97 | 160 | — |
| Tool results per case | 34 | 47 | 79 | — |
4.3 Building a benchmark case
The pipeline converts source rows into cases and uploads them to Langfuse to link experiment runs.
Figure 6. From recorded trajectories to benchmark cases.
System messages become Case.system_prompt; user messages become HumanMessage; assistant tool calls become AIMessage tool calls with parsed JSON arguments; tool responses become ToolMessage objects with their names and tool_call_id. Null padding is removed from tool schemas to restore each tool's own parameters.
The final unanswered finish call is dropped. A trailing assistant call with missing results is removed together with any results belonging to its other calls. Cases that still contain unmatched calls and results are rejected.
Figure 7. Questions probe the same recorded history view.
Each case appends a question to the recorded run. files_changed closes the initial history; subsequent questions use the same history view that was available for that first answer. Answers to earlier questions must not become extra memory for later questions.
4.4 Questions and reference answers
The selected cases provide 68 questions. Reference answers are extracted programmatically from the trajectory or final patch; an LLM does not author these references. A separate judge model evaluates generated answers against them.
| Question kind | Evidence requested | Reference source | Count |
|---|---|---|---|
files_changed | Existing files modified by the fix. | Diff headers of model_patch, excluding new files. | 20 |
first_error | Exception printed by a named command execution. | Final exception line of its first traceback. | 13 |
first_test_summary | Result of a named test execution. | pytest or unittest summary from that execution. | 20 |
first_edit | Exact code inserted by the first applied edit of a patched file. | new_str from the corresponding str_replace call. | 15 |
Repeated commands are identified by execution number. Input sent to an already-running process, such as C-c, is not treated as another command. An edit counts only if it was applied inside the task repository; if the first edit cannot be quoted, the question is omitted rather than replaced with a later edit.
4.5 Comparison arms and controls
The existing benchmark compares full history with sre-agent compaction and its configurable variants. This revision adds N latest to the evaluation plan. It requires implementation and new runs; the report does not establish an existing N latest result.
Figure 8. Proposed comparison arms share cases, questions, and references.
For N latest, predeclare a sweep such as N = 10, 20, and 40 using the message-count definition in section 3.1. These are proposed settings, not measured configurations. Record actual retained message and token counts after protocol-safe trimming. In addition to the N sweep, compare runs at similar retained token budgets so that preservation quality is not confused with simply retaining more input.
Each arm must receive the same case and questions, use the same answering model, and be judged against the same references. For offloading variants, read_tool_output may access only saved outputs exposed by that arm. The N latest arm has no extra summary or historical retrieval mechanism. The full-history arm acts as the reference condition within this subset, not as a production solution for arbitrarily long conversations.
For the compacted arm, micro_compact is replayed call by call as history would be held by the checkpointer. This preserves the distinction between clearing outputs during a run and summarizing accumulated history later.
The original sandbox is unavailable during evaluation. execute_bash, str_replace_editor, and task_tracker return a sandbox-shutdown response; think acknowledges the thought; finish supplies the final answer. Only read_tool_output operates on saved evidence. An agent therefore answers from its history view, summary, retained tail, or permitted saved outputs, rather than rerunning the original task.
| Setting from the report | Value |
|---|---|
| Answering agent and summarizer | google/gemma-4-31b-it, temperature 0. |
| Judge | z-ai/glm-5.3, temperature 0; a different model from the answering agent. |
| Model turns per answer | At most 25. |
| First compaction trigger | 0.8 × the case's own estimated history size, forcing compaction in every case. |
| Later compaction triggers | 0.8 × 131,072 tokens. |
| Token estimate | Characters divided by 4, matching auto_compact. |
4.6 Metrics and reporting
The main outcome is answer correctness relative to the full-history arm. A question that the full-history arm also fails is not clean evidence of compaction loss and should be examined before attributing the failure to history reduction.
| Metric | Interpretation |
|---|---|
answer_correct | Judge verdict for a reduced-context arm. |
baseline_answer_correct | Judge verdict for the full-history reference arm. |
answer_gap | Full-history accuracy minus reduced-context accuracy; positive means lower accuracy after reduction. |
fact_recall | Share of case facts stated in a generated summary. Not applicable to N latest, which generates no summary. |
faithfulness | Whether a summary introduces claims unsupported by the history. Not applicable to an absent summary. |
reduction_pct | Percentage decrease in context size. |
| Tool and handle reads | Attempts to fetch evidence again and actual use of saved outputs. |
| Mechanics checks | compaction_fired, structure_valid, and summarizer_calls. N latest needs separate trimming and boundary-validity checks. |
Report overall accuracy and accuracy by question kind. In particular, first_edit asks for exact code and should be interpreted separately from general factual recall. For N latest, also report whether the requested evidence survived in the retained suffix; that helps explain position-driven failures.
Runs are logged as Langfuse experiments with one trace per case. compaction.rescore rejudges stored answers using a common judge and writes one result row per question. Future runs should also record retained tokens, summarizer calls, output-retrieval calls, and latency to make quality and resource tradeoffs reviewable; these are reporting additions, not results already available.
4.7 What the initial probe shows
The completed probe covers two cases, jbradberry__universe-31 and adamchainz__time-machine-433, with eight questions. It uses the shipped sre-agent configuration, its SRE prompt, and no output offloading.
| Outcome | sre-agent compaction | Full history |
|---|---|---|
| Correct answers | 4 of 8 | 8 of 8 |
| Cases where compaction fired | 2 of 2 | Not applicable |
| Reported size reduction | 97% | Reference input |
| Question kind | Compacted accuracy | Observation |
|---|---|---|
files_changed | 2 of 2 | Changed-file information survived. |
first_error | 1 of 2 | One exception disappeared from the summary; the agent reported it as unknown. |
first_test_summary | 1 of 2 | One count disappeared; the agent stated 11 tests instead of 49. |
first_edit | 0 of 2 | Exact code was incomplete: one case missed a line, another retained only about 2 of 10 lines. |
The probe illustrates why reduction alone is insufficient: a reported 97% size cut coexists with lost answers. Eight questions are too few to rank methods or predict production behavior. No result for N latest or the proposed improved configuration can be inferred from this probe.
A larger run named p-sre-on-swe-1 stopped after 12 of 20 cases when its process was killed. It was not scored and must not be presented as a full result.
4.8 Comparison still to run
| Configuration | Full subset accuracy | Current evidence status |
|---|---|---|
| Full history | TBD out of 68 | Only the two-case probe is reported. |
| N latest with predeclared N values | Not measured | New proposed arm; requires implementation and runs. |
| sre-agent shipped compaction | TBD out of 68 | Two-case probe only; larger run incomplete. |
| Compaction variants | TBD out of 68 | Evaluate individual changes before selecting a configuration. |
First hold the mechanism fixed and compare the SRE, generic, hybrid, and hybrid_v2 prompts. Then hold the selected prompt fixed and evaluate the mechanism sequence: shipped behavior, a 0.2 retained tail with message cuts, step-aware cuts, offloading, settled-retry collapse, and verify guidance. The original ablation plan uses three runs per configuration and reports mean and range with a common judge.
The report estimates 5 to 6 million input tokens and 1 to 1.5 hours per 20-case run at concurrency 3, including about 70 reduced-context and 70 full-history agent calls plus 20 summaries. Its original ten configurations across three runs imply roughly 30 to 45 hours. Adding an N latest sweep increases that budget; those runs are not included in the original estimate. The identical full-history arm can be run three times and reused across configurations.
4.9 Limits of this comparison
The subset contains histories of roughly 20k to 40k estimated tokens. They would not normally reach the stated production trigger of about 105k tokens, so the benchmark forces the first compaction. It measures information retention under reduction rather than a conversation naturally growing to the limit.
All cases are coding trajectories. Four question kinds test recall of recorded details, not whether an agent can successfully continue the original task. A second independent source, such as operations or data-analysis trajectories, is needed to assess broader applicability.
The reported setup uses one answering model and one judge, so conclusions may not transfer to other models. Judge disagreements should retain their reasons for review. A dedicated metric for confidently wrong answers is not implemented; the 11-versus-49 example is currently visible through per-question outcomes and judge reasons.
Infrastructure also limits the evidence available so far. Langfuse requires VPN access in the reported environment, and span exports timed out during runs. Verify that all 20 traces exist before scoring a full run, and rerun incomplete cases.
5. Closing perspective
Keeping a long conversation usable requires a policy for both recent state and older evidence. N latest provides a simple starting point but excludes older messages outright. The sre-agent example shows how summaries can carry earlier context forward, while also showing that exact details may disappear. A retained tail and recoverable outputs offer additional mechanisms to test.
The current evidence supports a focused next step: complete the controlled comparison and measure retained answers alongside context reduction. It does not yet identify a winning configuration or establish lossless context management.
Appendix. Reproducing the existing benchmark
From agent-compaction-benchmark/, set dataset.source: swe and run the existing pipeline below. These commands reproduce the report's benchmark workflow; they do not implement the newly proposed N latest arm.
uv run python -m compaction.swe.download
uv run python -m compaction.dataset --upload
uv run python -m compaction.run --run-name <name>
uv run python -m compaction.rescore --run <name-prefix> --output results/<file>.json
uv run python -m pytest testsThe source report records 357 offline tests. This article revision changes the presentation and evaluation proposal; it does not claim to have rerun those tests or the benchmark.
