Architecture
Thearchitecture.
One Rust binary. Five subsystems. Zero cloud dependencies. Every box below maps to a real module, port, or file on disk.
System topology.
The process topology of the shipped binary: seven entry points, one runtime, one set of local stores. Every box below maps to a real module, port or file on disk.
lean-ctxlean-ctx servelean-ctx serve --daemonlean-ctx -c "<cmd>"lean-ctx proxy startlean-ctx dashboardlean-ctx watch · gain --liveOne binary. Five subsystems. No cloud required.
Data Flow Stage by stage.
Sheet 2 traces one request through the runtime, stage by stage, including the cache short-circuit that makes repeated reads nearly free. The shell path runs in parallel with the same accounting.
ctx_read(path, mode) · lean-ctx read- PathJail
core/pathjail.rsCanonicalises the path and rejects escapes outside the workspace root before any I/O happens.
- Session cache
hit → ~13 tokensContent-addressed lookup keyed by path + mtime/hash. Unchanged files collapse to a stub instead of re-sending content.
- AST extraction
26 languagestree-sitter parses the file into a syntax tree: signatures, imports, call edges — Lua, Luau, Kotlin and GDScript are graph-indexed too. Regex fallback for unsupported languages.
- Mode selection
10 modesauto picks the optimal of 10 read modes (full, map, signatures, diff, task, reference, aggressive, entropy, lines:N-M) from task intent and file size; structure_first biases cold medium-file code reads toward map, and a file flagged suspect on a fix task is forced to full.
- Compression
adaptive thresholdsShannon-entropy line filtering, U-curve attention placement (LITM), TF-IDF codebook and query-conditioned Information-Bottleneck fusion — an anti-inflation guard ships the file verbatim whenever framing would cost more tokens than the raw bytes.
- Token accounting
core/tokens.rsExact tiktoken counts (o200k_base; cl100k_base approximation for Claude-family models) on input and output.
- Ledger + stats
savings sign / verify-batchSavings are appended to the local ledger (Ed25519-signable), stats and the gain score update, the result streams back.
lean-ctx -c "cargo test" · IDE bash hookBoth paths end in the same ledger: every compression event is counted with exact tokenizer math and feeds gain, the dashboard and the signed savings ledger.
Compression engine.
Tree-sitter parses 27 languages into ASTs. 10 read modes select the right abstraction level — from full source to aggressive entropy filtering. 95+ shell patterns compress git, cargo, npm, docker, and kubectl output into compact evidence.
Explore read modesAvailable modes full → entire file, no compression signatures → fn signatures + type defs map → structural overview diff → uncommitted changes only task → query-conditioned extract aggressive → maximum compression entropy → information-dense lines reference → symbols + doc comments lines:N-M → specific line rangeSelected: signatures (420 lines → 18 lines)Memory, agents, security.
Content-addressed caching collapses re-reads to ~13 tokens. A knowledge graph persists decisions across sessions. The agent bus coordinates multiple agents without duplicating context. PathJail, shell allowlists, and TOCTOU checks enforce deny-by-default boundaries on every call.
Security modelCache 47 entries · 73% hit rateKnowledge 342 nodes · 1,205 edgesAgent bus 3 agents registeredPathJail root: /Users/dev/projectAllowlist 14 allow, 6 denyRedaction .env, credentials.*TOCTOU open-handle verify activeLedger Ed25519-signed · 0 violationsEngineering data sheet.
The reference tables behind the drawings: every surface with its transport and lifecycle, the on-disk layout, the adaptive-learning layers, and the security boundaries the runtime enforces.
AProcess model
All surfaces are the same binary in different roles. Nothing requires a cloud connection; everything binds local-first.
| REF | SURFACE | TRANSPORT | ENDPOINT | LIFECYCLE | COMMAND |
|---|---|---|---|---|---|
| 01 | MCP server (stdio) | JSON-RPC over stdin/stdout | spawned per editor session | child process of the editor | lean-ctx |
| 02 | MCP server (HTTP) | MCP Streamable HTTP | localhost, configurable --host/--port | foreground or service | lean-ctx serve |
| 03 | IPC daemon | Unix Domain Socket | OS data dir, e.g. ~/Library/Application Support/lean-ctx/daemon.sock | launchd / systemd autostart | lean-ctx serve --daemon |
| 04 | Shell hook | process exec, compressed stdout | wraps IDE bash calls + interactive shells | per command | lean-ctx -c "<cmd>" |
| 05 | API proxy | HTTP (LLM API pass-through) | localhost:4444 (default) | on demand | lean-ctx proxy start |
| 06 | Web dashboard | HTTP + bearer token | localhost:3333 (default, --port) | on demand | lean-ctx dashboard |
| 07 | Terminal UI | TTY (in-place redraw) | live event stream / 1 s refresh | interactive | lean-ctx watch · gain --live |
BStorage layout — local XDG dirs
Persistent state is plain files under the XDG base directories: inspectable, exportable, deletable. No hidden databases beyond these local folders.
| ARTIFACT | FORM | PURPOSE |
|---|---|---|
config.toml | TOML | Single config file — integration mode, compression, providers, opt-outs (config dir) |
cache/ | content-addressed | Session file cache; unchanged re-reads collapse to ~13-token stubs (cache dir) |
bm25 index | inverted index | Lexical search over code chunks + provider documents (data dir) |
context_graph/ | property graph | Imports, calls, types across files and repos — powers map mode + deep queries (data dir) |
knowledge | SQLite | Persistent facts, decisions, rooms — recalled across sessions, CCP (data dir) |
savings ledger | append-only JSONL | Every compression event; Ed25519-signable for audit (data dir) |
litm_calibration.json | JSON | Learned context-position hit rates (lost-in-the-middle calibration) (cache dir) |
events.jsonl | event stream | Live feed consumed by watch, dashboard and efficacy reports (state dir) |
CAdaptive-learning layers
Seven online-learning mechanisms tune compression to your real usage, locally, from quality signals like bounces and edit failures. Deep dive: Adaptive Learning →
- L1Adaptive thresholdsOnline-learned compression aggressiveness from quality signals (bounces, edit failures, clean runs)
- L2LITM calibrationEmpirical placement of critical context at positions the model actually attends to
- L3Stigmergic scent fieldMulti-agent coordination via decaying markers: claimed, done, stuck, hot, avoid
- L4Delta playbookIncremental checkpoint snapshots that survive context compaction
- L5Query-conditioned IBInformation-Bottleneck compression fused with query relevance
- L6Theta-gamma chunkingWakeup facts grouped in attention-friendly bursts
- L7Semantic dedupLikelihood-scored redundancy filtering across the session
DSecurity boundaries
Hard guarantees enforced in the runtime. Security model →
- PathJailEvery file access is canonicalised and confined to the workspace root
- IDE config-dir jailHome-level IDE/agent config dirs (~/.claude, ~/.codex, ~/.codebuddy, …) are writable only when allow_ide_config_dirs is opted in; otherwise PathJail blocks them
- Shell allowlistDeny-by-default command policy for agent-issued shell executions
- Local-firstAll processing on-device; dashboard binds to localhost and requires a bearer token
- Signed evidenceSavings ledger entries are Ed25519-signable and batch-verifiable
Explore thesource.
LeanCTX is open source. Read the code, understand the design, contribute.