v1.9.4

Added

• token-goat tokens [patterns] — per-file token footprint table. Scans matched files and prints each file's token estimate alongside its line count, sorted largest-first by default. --tree groups results by directory with per-directory subtotals and percentages of the total. --top N limits the view to the N largest files. --asc reverses the sort order. --json emits a structured object with total_tokens, total_files, and a files array. --no-ignore bypasses .tokengoatignore. Omit patterns to scan the entire project.

• token-goat note set/get/unset/list/clear — persistent per-project notes. Stores short key-value facts in a per-project TOML file. Token-goat injects them into the context at session start and after compaction so they survive conversation rollover without being repeated in the chat history. Keys are alphanumeric with hyphens and underscores (max 80 chars). note list --json for machine-readable output; note clear wipes all notes at once.

• token-goat pack --strip-comments — strip comments before packing. Removes language-appropriate comments from source files before bundling. Covers Python # line comments, JS/TS/Go/Rust/Java/C/C++ line and block comments, SQL -- comments, Ruby/shell hash comments, and CSS/SCSS block comments. Shebangs (#!) are preserved. Cuts token count on comment-heavy codebases; language detection is extension-based with no extra install.

• token-goat pack --scan-secrets — check for credentials before emit. Scans packed files for patterns matching AWS access and secret keys, GitHub tokens, private key PEM blocks, Stripe and OpenAI keys, Slack webhook URLs, Google API keys, database connection strings, bearer tokens, and password literals. Exits 2 and prints per-file, per-line warnings when any pattern fires; a clean pack exits normally. Binary and image extensions are skipped automatically.

• token-goat call-chain <symbol> — trace the full call path to a symbol. Walks the call graph upward from symbol and prints each transitive caller layer, from immediate callers out to entry points. Pairs with callers (one hop) and impact (downstream direction).

• token-goat hot [--limit N] [--project dir] — cross-session file frequency ranking. Tallies read and edit counts from all stored sessions and ranks files by total activity. Useful for identifying which files dominate token spend across your whole history, not just the current conversation. --json for structured output; --project to filter to a specific project directory.

• token-goat impact <symbol> — downstream blast-radius estimate before a refactor. Walks the reference graph forward from symbol and lists every file and function that directly or transitively depends on it, with the hop depth and dependency type (call, type annotation, or import). Run it before changing a function signature to see what breaks without starting a build.

• token-goat context-for <task> — minimal context bundle for a task. Takes a natural-language task description, runs semantic search across the indexed codebase, and emits a prioritized list of token-goat read commands trimmed to a token budget. Fetches only the slices relevant to the task instead of loading entire files. --budget N sets the token ceiling; --top N limits the file count; --json for structured output.

• token-goat dead — surface symbols with no known callers. Queries the project index for functions, methods, async functions, and classes that have no recorded call-site references. Private symbols (names starting with _) and common entry-point names (main, app, create_app, etc.) are excluded from results by default. --include-private lifts the underscore filter; --kind narrows to specific symbol types; --top N caps the list; --json for structured output. Results are heuristic leads: dynamic dispatch and external callers are not visible to static indexing.

• token-goat coverage-gaps — find callables not referenced by any test file. Scans indexed functions and methods in non-test source files for names that never appear in a test file's reference records. Dunder and private names are excluded. --top N limits the list; --json for structured output. Useful for spotting untested surface area before a refactor or release; results should be read as leads, not verdicts.

• token-goat pack --budget N — fail the pack if it would exceed a token budget. Exits with code 3 when the estimated token count of the collected files exceeds N, so a shell script or CI step can treat an oversized context as an error rather than silently passing it to the model. --budget 0 (the default) imposes no limit.

• token-goat skeleton — file line count in the output header. The skeleton header now shows total line count alongside symbol count: # Skeleton: src/token_goat/cli.py (80 symbols, 9,394 lines). Gives an immediate size gauge before deciding whether to read a file in chunks or in full.

• Injection detector — three new patterns. forget-instructions catches "forget [all] [your] previous instructions/directives/guidelines" (requires "previous" to avoid false positives on documentation phrases like "forget to include instructions from step 1"). pretend-no-restrictions catches roleplay jailbreak framing ("pretend you have no restrictions/limitations/constraints"; requires "you have no" to skip game-design prose). exfil-conversation catches attempts to extract the full conversation or chat history ("print the entire chat history", "dump the message history"); requires "the" before the noun so code comments and variable references do not fire.

• Large-file read hints — skeleton suggestion for files with many indexed symbols. When the read hint fires and more than three symbols are indexed for the file, the hint now shows the total symbol count and suggests token-goat skeleton "file" before opening a specific one. Previously the overflow appeared as ... with no count and no browse path. The skeleton command in the hint is quoted to handle paths with spaces.

Performance

• pre_read hook now uses a read-only DB connection for symbol lookup. _get_indexed_symbols_and_line_count was opening a write-capable connection (db.open_project()) that loads the sqlite-vec extension, sets WAL mode, and runs schema DDL on every call. Switching to db.open_project_readonly() eliminates those steps, cutting the function from ~9.8 ms to ~1.4 ms. Every Read tool call passes through pre_read, so the saving applies to every hook invocation. Fail-soft behavior is unchanged.

Fixed

• Stale .jsonl session sidecars now get cleaned up. The cleanup pass only matched files ending in .json, so .jsonl sidecars piled up and were never removed. The suffix filter now covers both extensions.

• Writer lock no longer leaves an empty lock file behind. When the os.write after an O_EXCL create failed, the freshly created lock file was orphaned on disk. The failure path now deletes it.

• Skeleton brace-skipper counts braces inside literals correctly. A } or { inside a string, comment, or backtick template literal was counted as a real brace, which leaked body lines into the skeleton. Those contexts are now skipped, and regex literals are handled, too.

• Embedding dimension validation runs for custom models. The check that the model's vector width matches the stored index was skipped for any non-default model. It now runs regardless of which model is configured.

• A closing --- in YAML front matter is no longer read as a heading. The front-matter terminator was parsed as a setext H2 heading, which invented a phantom section. Markdown parsing now recognizes the fence.

• MCP server name is read consistently in the transcript tool tally. The tally pulled the server name from two different places, so one server could show up under more than one label. Extraction is now uniform.

• WSL path normalization drops the doubled slash. Normalizing a Windows drive path produced a redundant slash after the drive letter, such as /mnt/c//foo. The extra slash is gone.

• Combined --head and --tail recall returns both ends. Passing --head and --tail together returned only the head slice. Recall now returns the head and the tail.

• compact-hint --diff ignores timestamp-only changes. A manifest whose only difference was a refreshed # as-of: timestamp was reported as a real change. The diff now treats a timestamp-only tick as no change.

v1.9.3

What's new

Added

• **** — show which functions call a given symbol, grouped by caller with file, caller name, and every invocation line. Complements refs by showing the call chain rather than raw usage lines.
• **** — per-project exclusion file at project root. Add gitignore-style glob patterns (one per line, # comments supported) to skip files and directories from indexing on top of built-in skip lists. Run token-goat ignores to see what's active.
• token-goat semantic --mode keyword|hybrid — two new search modes. --mode keyword runs BM25 keyword search with no embedding model required. --mode hybrid combines BM25 and vector rankings via reciprocal rank fusion, useful when a query has distinctive terms that pure semantic search would drift past. The default (--mode vector) is unchanged.
• token-goat arch — project-wide import graph summary: hub modules sorted by inbound import count, entry-point files nothing imports, and circular import chains. Derived from the existing index with no extra indexing step.
• token-goat pack <patterns> — bundle files into a single LLM-ready output in Markdown (default), XML, or plain text, with a manifest table of per-file line and token counts. Supports --line-numbers, --instruction-file, --output, and --no-ignore. Reads file paths from stdin when no patterns are given.
• token-goat budget <patterns> — token-cost estimate for a file set, sorted by cost descending. --context <N> shows percentage of an N-thousand-token window. Run before pack to decide what to include.
• token-goat todo [--kinds K] [--group file|kind] — scan indexed files for TODO-family markers (TODO, FIXME, HACK, XXX, NOTE). Groups by file by default; --group kind to group by marker type; markers in string literals are excluded.
• token-goat failures [src] — extract failing test blocks from test runner output. Parses pytest, Jest, Go, and Cargo output; passing tests and preamble are dropped. Reads stdin by default; pass a file path for saved output. --json for structured output.
• token-goat trace [src] — condense Python tracebacks to project-owned frames. Strips library, stdlib, and virtualenv frames. Chained exceptions preserve their cause notes; bare exceptions without a message are handled. --keep N (default 5) caps the frame count.
• token-goat lockdeps [path] — summarize lock file dependencies as a compact table. Reads poetry.lock, uv.lock, requirements.txt, Pipfile.lock, package-lock.json, Cargo.lock, and yarn.lock. Returns direct dependencies only.
• token-goat logfold [src] — collapse consecutive duplicate log lines to [Nx] counts. Normalizes ISO timestamps, UUIDs, IPs, and short hex IDs before comparing so the same event logged with different values folds correctly. --tail N, --no-normalize, --json.

Fixed

• Path traversal bypass in _is_system_path(). bash_parser.py appended .. even at root of an absolute path, letting /../../etc/passwd slip past the system-path block. Now discards .. at root.

v1.9.2

Changes

Changed

• Hook watchdog default reduced from 5000ms to 700ms; new HOOKS_WATCHDOG_DEFAULT_MS constant is the single source of truth across config, hooks_common, and tests.

Performance

• Surgical hint lookup (_try_surgical_read_hint) is now memoized by path+mtime_ns, eliminating repeated DB queries for unchanged files. Cache key normalizes to lowercase on Windows to handle case-insensitive filesystem paths.

v1.9.0

[1.9.0] - 2026-06-16

Added

• TerraformFilter extended: terraform show compression and plan data-source detection. terraform show output now strips noise attributes (id, arn, timeouts, tags blocks) per resource block and appends a suppression note; only meaningful fields survive. terraform plan unchanged-block detection now covers data-source read-during-apply blocks in addition to managed-resource no-op blocks.

• KubectlFilter extended: event grouping and describe compression. kubectl events output groups events by REASON with a per-group count and a field-selector hint. kubectl describe collapses label and annotation blocks to line counts, preserves the Conditions table in full, and retains container resource (requests/limits) fields.

• NpmInstallFilter extended: warn collapsing and verbose line suppression. npm warn lines after the first 3 are collapsed to a suppression note. Verbose timing, sill, http, and verb lines are suppressed entirely. Braille spinner reify progress lines are stripped.

• Three-layer watchdog budget resolution. _resolved_watchdog_ms() now reads config.load().hooks.watchdog_ms (default 5000 ms) when no TOKEN_GOAT_HOOK_WATCHDOG_MS env var is set. Previously it fell straight through to the 2000 ms compile-time constant, ignoring whatever [hooks].watchdog_ms was set to. Resolution order: (1) env var, (2) project config baseline (process-level mtime-cached, one os.stat() on the fast path), (3) _HOOK_WATCHDOG_MS = 2000 ms compile-time fallback. Values below the 100 ms floor are clamped regardless of layer.

• Reread-deny hint shows real indexed symbols. _handle_reread_deny now queries the project DB for up to 8 non-import, non-variable symbols in the denied file and emits exact token-goat read "path::Symbol" commands in the hint instead of the static ::SymbolName placeholder. The lookup uses find_project from the file path, so no cwd parameter is required. If the file is not indexed or the query fails, the hint falls back to the generic placeholder silently.

• _handle_doc_compact auto-spawns compact-doc in the background. When the section-map path fires for a large markdown file, it now launches compact-doc <file> as a fire-and-forget subprocess so the compact sidecar is ready on the next read. A per-file session fingerprint (compact_doc_spawned:<path>) prevents re-spawning for the same file within a session. If token-goat is not on PATH or the spawn fails, the hook continues normally.

Fixed

• TerraformFilter._compress_terraform_init head/tail fallback. When terraform init progress lines (e.g., Installing plugin N) did not match the provider-specific regex, all lines passed through unchanged. The method now applies head=5/tail=5 compression whenever len(non_empty) > 12 after provider-line collapsing.

v1.8.0

What's New in 1.8.0

Added

• curl -v verbose compression — post_bash detects curl -v/--verbose commands and strips TLS handshake noise, redundant request/response headers, and progress meters; keeps the request line, status code, content-type, and body
• jest/vitest verbose PASS-suite compression — collapses PASS src/... blocks with all-green test lines into a single summary line
• JUnit XML structured summary — parses <testsuites> XML from post_bash output and emits a compact pass/fail/skip count instead of raw XML
• Task-output temp file redirect — pre_read detects Claude task-output files in %TEMP%/claude/... and transparently redirects to token-goat bash-output
• Minified JS/CSS grep elision — post_bash truncates grep/rg hits on minified files (.min.js, .min.css, bundled output) to avoid multi-MB lines flooding context
• Compaction hint suppression — suppresses the redundant re-read hint that fired after every conversation compaction even when files hadn't changed
• go test -v compression — collapses passing === RUN / --- PASS blocks in verbose Go test output
• make/cmake/ninja build compression — strips redundant compile command echoes while keeping warnings and errors
• Python traceback deduplication — collapses repeated identical tracebacks in test output to a single copy
• tsc output compression — suppresses TypeScript compiler progress lines, keeping only errors and the final summary

v1.7.1

Fixed

• _index_spawn_active guards against PID recycling. Within the 10-minute INDEX_SPAWN_TTL the OS can reuse a finished indexer's PID for an unrelated process, blocking fresh indexing spawns for up to 10 minutes. The check now reads the running process's cmdline and returns False when it lacks token_goat, falling back to trusting the PID when the cmdline is unreadable.

• kill_duplicate_daemon now unlinks the stale PID file after a successful kill. Previously the file was removed on the "already dead" path but not on the success path, leaving --check and is_worker_alive() reporting stale state until the next cleanup pass.

• get_context_pressure avoids a redundant safe_load when a cache is already in scope. Accepts an optional cache= kwarg; callers that already hold a loaded SessionCache pass it in and skip the extra disk I/O.

• normalize_path docstring corrected. Steps 2 and 3 were listed in the wrong order relative to actual execution.

• Shell-neutral bash-compress disable hint. All 34 TOKEN_GOAT_BASH_COMPRESS=0 hint strings were POSIX-only prefix assignment — broken in PowerShell and cmd.exe. Hints now read disable via TOKEN_GOAT_BASH_COMPRESS.

• Bash pre-hook fast-path via bash_detect. Adds a <1 ms dict lookup before the ~75 ms bash_compress import; unrecognized commands skip the import entirely.

• enqueue_dirty is now append-only with a byte-based cap. Eliminates O(queue-size) rewrite and POSIX rename race on every Edit/Write hook.

• Corrupt .draining file is quarantined instead of raising. Prevents worker crash when another process races on the drain rename.

• post_bash uses a single session load/save round-trip. Was calling load()/save() up to four times per invocation.

• Output size cap applied before payload work in post_bash. Prevents the full pipeline running on 4 MB stdout that will be truncated anyway.

• Cache eviction throttled to at most once per 60 seconds. Eliminates per-write O(n) directory scan.

• pre_read Bash branch uses safe_load(). Corrupt session files no longer crash the Bash pre-hook.

v1.7.0

See CHANGELOG for full details.

Fixed

• Skill dedup permanently disarmed after first compaction — post_skill early-return path skipped mark_skill_loaded(), freezing skill_ts and causing pre_skill to pass all subsequent loads through undeduped after any compaction event.

Added

• token-goat compact-doc — build a deterministic extractive sidecar for any large reference .md file; pre_read serves it in place of the full file (80–95% smaller). Auto-staled on edit.
• post_compact_full_loads config knob ([skill_preservation] post_compact_full_loads, default false) — keep skill dedup armed across compaction epochs; set true to restore the pre-1.7 one-full-reload-per-epoch behaviour.
• MCP screenshot deny-redirect — pre_screenshot denies chrome-devtools and playwright screenshot calls without a filePath/filename argument, forcing the save-to-disk path so image-shrink applies (~39K tokens raw → ~8K compressed).
• Baseline v2 — token-goat baseline now costs the skill listing, shows one row per configured MCP server, and adds a --usage flag that annotates each row with historical call counts.
• Session window denial for in-context file reads — pre_read actively denies re-reads of files confirmed in the current context window (config: [hints] deny_reread, default on).

v1.5.2

Three fixes: Codex hook wire-format compatibility, and two Windows coarse-mtime correctness issues in the cache and session layers.

Codex hook responses now pass schema validation

Codex 0.137.0 validates every hook response against embedded JSON schemas with additionalProperties: false, so any unrecognised key causes "hook returned invalid … JSON output" for the entire response — including SessionStart, PreToolUse, and PostToolUse. The root cause was _tg_elapsed_ms (and sibling _tg_handler/_tg_error fields) added by the internal dispatch() function and then emitted verbatim. The denormalize_response Codex branch now strips all _tg_* keys before output. The same path also injects the required hookEventName const field into hookSpecificOutput — Codex requires it on every hookSpecificOutput shape and token-goat was not emitting it because Claude Code does not require it. A _codex_hook_event_name() helper resolves the correct value (e.g. "pre-read" → "PreToolUse") from the hook registry. The old camelCase→snake_case key conversion (_translate_hso_to_codex) is no longer applied — Codex 0.137.0+ uses camelCase throughout hookSpecificOutput.

Freshest cache entry survives its own store call's eviction

evict_cache_dir sorts eviction candidates oldest-first by float(st_mtime) with a stable sort. When the just-written (MRU) entry shares a coarse st_mtime with older siblings, the stable sort falls back to arbitrary iterdir order, which on NTFS can place the newest file first and evict it — so a store_output call could delete the very entry it had just written. evict_cache_dir now accepts a protect_ids set that is excluded from the victim list regardless of timestamp, and skill_cache.store_output passes the id it just wrote.

save() refreshes the process-local load cache

session.load() caches (object, mtime) per session and serves the cached object whenever cached_mtime == current_mtime. When a later save()'s post-write timestamp aliased the mtime a previous load() had cached, the proc-cache kept serving the stale pre-save object on the next in-process load() even though the on-disk JSON was correct. save() now overwrites an existing proc-cache entry with the object it just persisted on every successful write.

v1.5.1

Correctness fixes for cache size accounting (compressed .gz bodies), surgical reads (oversized-docstring cap, signature-boundary fix), path normalization (uppercase WSL drives), mixed-case skill-compact invalidation, and the Gemini hook bridge (preserve systemMessage, route additionalContext natively), plus two documentation corrections.

See the CHANGELOG for full details.

v1.5.0

Context-pressure awareness: one source of truth for how full the window is, and hints that get terser as it fills. Ships alongside three install fixes that restore hook forwarding under editable installs and silence a recurring doctor warning.

Centralized context-pressure model

get_context_pressure(session_id) in compact.py is now the single place that answers how close a session is to autocompaction. It returns a frozen ContextPressure — a fill_fraction paired with a tier of cool, warm, hot, or critical. The estimate sums the known context contributors (loaded skill bodies, the ~10,800-token skills catalog, and per-event costs for bash history, web history, and read files) and divides by the fixed 660,000-token autocompact budget rather than the model's raw window, so the fraction carries the same meaning no matter which model is driving the session. The old _estimate_context_fill helper and the inline calculation in the session hook both defer to it, retiring the copies of the 660 K constant that had spread across half a dozen call sites in favor of one shared CONTEXT_AUTOCOMPACT_TOKENS.

Named tier boundaries

The fraction-to-tier mapping lives in tier_for_fraction(), backed by three named constants: CONTEXT_TIER_WARM (0.50), CONTEXT_TIER_HOT (0.70), and CONTEXT_TIER_CRITICAL (0.85). The bands are cool below 0.50, warm up to 0.70, hot up to 0.85, and critical at or above it. With the magic numbers pulled out of the band checks, the boundaries are defined once and the tests pin them directly.

Pressure-aware surgical-read hints

The pre-read hook tightens its large-file threshold as the window fills. A file earns a surgical-read suggestion past 500 lines while the session is cool, 350 when warm, 200 when hot, and 50 when critical. It also folds a single per-tier note into the read's additional context: "Context warming" at warm, "Context pressure" at hot, "CONTEXT CRITICAL" at critical. The note is fingerprinted by tier, so it fires once per band rather than on every read. Cool sessions get no note.

Smaller manifests under pressure

compute_adaptive_budget now weighs context pressure when it sizes the compaction manifest. Once the window runs hot the budget is capped at 500 tokens, and at critical it drops to 300, so the manifest stops adding to the very problem it exists to summarize.

Install robustness

Hooks no longer silently disable themselves under an editable install. The tg-hook wrapper carries an if not exist "<sentinel>" gate that short-circuits to a bare {"continue":true} during the uv tool install --reinstall race, when the venv's token_goat module is briefly absent. The sentinel used to be a hardcoded site-packages/token_goat/__init__.py path, which never exists under an editable install (uv sync, the project .venv), so the gate stayed permanently true and every hook no-op'd — the whole tool went dark with no error. The wrapper now resolves the sentinel through importlib.util.find_spec("token_goat").origin, which points at src/token_goat/__init__.py for editable installs and site-packages/... for regular ones, and falls back to an ungated wrapper when no sentinel resolves. A live handler emits {"continue": true, "_tg_elapsed_ms": N}; the _tg_elapsed_ms field is the tell that forwarding actually ran.

Re-install purges orphaned tokenwise entries. After the tokenwise → token-goat rename, a re-install left the old hook and permission lines stranded in settings.json and the Codex config.toml, so both harnesses kept invoking a binary that no longer existed. patch_settings_json and patch_codex_config now strip any pre-rename tokenwise command and permission entry before writing the current ones.

Hook wrapper is written as bytes to stop CRLF doubling. hook_wrapper_content() hand-bakes platform-correct line endings — \r\n on Windows — then was written through atomic_write_text, whose text-mode handle translated every \n to \r\n a second time, doubling each line ending to \r\r\n on disk. cmd.exe tolerated the stray carriage return so forwarding still worked, but doctor does a byte-exact compare of the on-disk wrapper against the regenerated content and warned differs from expected — run token-goat install to refresh on every run, a nag that reinstalling could never clear because it rewrote the same doubled bytes. The wrapper now goes through atomic_write_bytes, preserving the authored endings verbatim.

Session-cache integrity

Concurrent session saves no longer drop an edit. The save() fast path skipped its compare-and-swap re-read and merge whenever the on-disk (st_mtime, st_size) fingerprint still matched the one captured at load. That fingerprint aliases: two caches whose keys are the same length serialize to byte-identical JSON sizes, and a float st_mtime rounds two sub-microsecond writes to the same value. When two writers collided on both fields the second skipped the merge and overwrote the first, losing exactly one edit — the 200-edit concurrency stress test intermittently saw 199. The fast path now consults an in-process version registry so a same-process writer that already advanced the version forces the stale save back through the merge, and the fingerprint is taken from integer st_mtime_ns instead of the rounded float, so a cross-process skip now requires a true nanosecond-and-size collision rather than a rounding coincidence.