Build a defensible comparison

Before you start

Choose a repeatable task and define what counts as success before measuring savings. A smaller context and a correct result are separate outcomes. Decide how you will test both.

Freeze the comparison

Record the repository revision or document snapshot, task, model, tool setup, enabled policy and acceptance criteria. Identify which calls LeanCTX covers.

Keep these conditions comparable between the baseline and treatment. An easier task or a different model is not an isolated test of context preparation.

Inspect local representations

For a repository-level diagnostic:

lean-ctx --version
lean-ctx benchmark run . --json
lean-ctx benchmark report .

Save the output with the source revision. These commands compare context representations; they do not execute a controlled end-to-end agent task or establish answer quality.

Use the same tokenizer when comparing original and prepared input. Record read mode and any omitted material.

Measure the full attempt

Include preparation latency, tool overhead, cold and warm reads, recovery calls, retries and failures. A compact output may require extra work to produce or a later full read to complete the task.

Separate estimated context reduction from provider-reported usage. Cached-input discounts, output tokens and provider billing rules can change the financial result.

Evaluate the accepted outcome

Run the same tests or review criteria for both conditions. Report unsuccessful attempts and variation, not just the best reduction.

Check artifact digests when available. Integrity verification establishes particular properties of a record; it does not make the experimental design fair or the model answer correct.

Check the result

Another reader should be able to reproduce the setup, see what was measured and understand where your conclusion stops. Report workload, sample size, versions, reduction and task outcome together.

Read token measurements for metric semantics and Results & Evaluation for the evidence framework.

Sources & versions4 references · Reviewed 17 Sept 2026

• 11-analytics-and-insights.mddocs/reference/11-analytics-and-insights.mdReviewed checkout
• config_cmd.rsrust/src/cli/config_cmd.rsReviewed checkout
• mod.rsrust/src/core/stats/format/mod.rsGitHub
• quality-loop-v1.mddocs/contracts/quality-loop-v1.mdReviewed checkout

Core checkout

0ce2207ee4

Reviewed runtime

3.10.2

SDK release

1.1.0

Separate baselines for source, CLI/configuration and SDK contracts. Review does not certify every platform or integration.

Versions & compatibility