Shell Patterns - Intelligent Output Compression

Every shell command run through ctx_shell passes through lean-ctx's shell compression pipeline. With 56 pattern modules covering 95+ developer tools, the system strips boilerplate while preserving actionable output - typically saving 50-95% of tokens per command.

How Shell Compression Works

When ctx_shell executes a command, the raw output flows through a multi-stage compression pipeline before reaching the agent:

Compression Pipeline

Raw command output
  │
  ├─ 1. ANSI strip        → Remove color codes, cursor escapes
  ├─ 2. FilterEngine       → Apply user-defined TOML filters
  ├─ 3. Pattern matching   → Detect tool (git, npm, cargo, etc.)
  ├─ 4. Pattern compress   → Tool-specific structured extraction
  ├─ 5. JSON schema        → Extract structured data from JSON output
  ├─ 6. Log dedup          → Collapse repeated log lines
  └─ 7. Test compress      → Summarize test suite results
  │
  ▼
Compressed output (~50-95% smaller)

Track Mode vs Compress Mode

ctx_shell supports two modes that control how output is processed:

# Compress mode (default) - returns compressed output
ctx_shell command="npm test"
→ ✓ 42 passed, 0 failed (3.2s)  [saved 1,847 tok]

# Track mode - returns only metadata
ctx_shell command="npm install" mode="track"
→ ✓ exit=0 (12.4s)  [output suppressed]

56 Pattern Modules

Each pattern module knows the structure of a specific tool's output and extracts only the information an agent needs. Patterns are organized by ecosystem:

Version Control

JavaScript / Node

Python

Rust

Go

Docker & Cloud

Databases

System

Other Ecosystems

Compression Examples

# git status - 87% savings
ctx_shell command="git status"
→ branch:main ±0 ahead
  M src/auth.ts
  M src/server.ts
  ? src/new-file.ts
  [saved 523 tok, 87%]

# cargo test - 91% savings
ctx_shell command="cargo test"
→ test result: ok. 156 passed; 0 failed; 2 ignored (4.8s)
  [saved 3,412 tok, 91%]

# npm install - 71% savings
ctx_shell command="npm install"
→ added 12, removed 0, changed 3, audited 847 packages
  0 vulnerabilities
  [saved 214 tok, 71%]

Passthrough Commands

Some commands are never compressed and always return raw output. lean-ctx maintains a list of 85+ passthrough entries (added in v3.3.1) covering three categories:

1. Dev Servers & Long-Running Processes

Commands that produce continuous streaming output are passed through because compression would either buffer indefinitely or lose real-time context:

# These always return raw output:
npm run dev          # Next.js / Vite dev server
yarn dev             # Yarn dev server
docker compose up    # Docker container logs
kubectl logs -f      # Streaming pod logs
tail -f              # File watching

2. Interactive GitHub CLI Commands

gh pr commands and similar interactive flows are passed through to preserve formatting and interactivity:

gh pr create         # PR creation flow
gh pr view           # PR detail view
gh pr checks         # CI status display
gh issue create      # Issue creation

3. Smart Script-Runner Detection

lean-ctx uses a heuristic to detect script-runner commands that launch dev servers. When a package.json script name contains dev, start, or serve, the command is automatically treated as passthrough:

npm run dev          → passthrough (contains "dev")
yarn start           → passthrough (contains "start")
pnpm serve           → passthrough (contains "serve")
bun run dev:server   → passthrough (contains "dev")
npm run build        → compressed (no trigger word)

FilterEngine: Custom Filters

Beyond built-in patterns, you can define custom compression filters using TOML files. Place filter files in ~/.lean-ctx/filters/ and lean-ctx applies them before pattern matching in the pipeline.

Filter File Format

Each filter file uses [[rules]] syntax to define match-action pairs:

# ~/.lean-ctx/filters/custom.toml

[[rules]]
match = "regex"
pattern = "^\[INFO\]\s+Starting.*"
action = "drop"

[[rules]]
match = "regex"
pattern = "^WARNING: .+"
action = "keep"

[[rules]]
match = "contains"
pattern = "DEPRECATED"
action = "drop"

[[rules]]
match = "prefix"
pattern = "DEBUG:"
action = "drop"

Rule Fields

Filters are applied in order. The first matching rule wins. Lines that match no rule pass through unchanged.

Example: Suppress Verbose Logging

# ~/.lean-ctx/filters/suppress-verbose.toml

[[rules]]
match = "regex"
pattern = "^\[(TRACE|DEBUG)\]"
action = "drop"

[[rules]]
match = "contains"
pattern = "Compiling"
action = "drop"

# Keep everything else (implicit - unmatched lines pass through)

Bypassing Compression

Sometimes you need the full, uncompressed output. lean-ctx provides three escape hatches:

1. raw=true Parameter

Pass raw=true to ctx_shell to disable compression for a single command:

ctx_shell command="npm test" raw=true
→ [full uncompressed output...]

2. LEAN_CTX_RAW=1 Environment Variable

Set the environment variable to disable compression for the current command. Useful when piping or in scripts:

LEAN_CTX_RAW=1 npm test
→ [full uncompressed output, no lean-ctx processing]

3. LEAN_CTX_DISABLED=1 - Disable Entire Hook

Completely disables the lean-ctx shell hook for the current shell session. No interception, no compression, no tracking:

export LEAN_CTX_DISABLED=1
# All commands now bypass lean-ctx entirely

unset LEAN_CTX_DISABLED
# lean-ctx hook is active again

Smart Heredoc Detection

Starting with v3.3.x, lean-ctx includes intelligent heredoc handling in ctx_shell and the PreToolUse hook.

The Problem

Earlier versions would block or incorrectly process heredoc commands, because the shell hook couldn't distinguish between a heredoc (inline data) and a file redirect (writing to disk). This caused issues with commands like:

# This was incorrectly blocked:
cat <<'EOF'
some inline data
EOF

# This SHOULD be blocked (file write via heredoc):
cat <<'EOF' > /etc/config.toml
[server]
port = 8080
EOF

The Solution

lean-ctx now distinguishes between these two cases:

• Pure heredoc (no file redirect): Passed through without blocking. The command runs normally and output is compressed as usual.
• Heredoc + file redirect (>, >>): Blocked by the PreToolUse hook, as this is a file write operation that should use ctx_edit or native file editing instead.

Detection Logic

# ✓ Allowed - pure heredoc, no file redirect
cat <<EOF
inline content
EOF

# ✓ Allowed - heredoc piped to another command
cat <<EOF | grep "pattern"
search this content
EOF

# ✗ Blocked - heredoc with file redirect
cat <<EOF > output.txt
file content
EOF

# ✗ Blocked - heredoc with append redirect
cat <<EOF >> log.txt
log entry
EOF

The PreToolUse hook inspects the command string for heredoc markers (<<) combined with redirect operators (>, >>). Only the combination triggers blocking - heredocs alone pass through cleanly.