Skip to content

test: 36 tests for lib/eval/entity_coverage.py #21

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
hai-pilgrim wants to merge 6 commits into
base: main
Choose a base branch
from
Open

test: 36 tests for lib/eval/entity_coverage.py #21

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
bb4fb00
feat: make Claude Code plugin seamless — auto-config, single-script c…
marksverdhei Feb 13, 2026
7f20fcb
test: 15 tests for lib/eval/report.py (export_json, export_trace)
marksverdhei Mar 29, 2026
91e9af3
test: 23 tests for lib/eval/judge.py
marksverdhei Mar 29, 2026
fed8688
test: 29 tests for lib/eval/aggregate.py
marksverdhei Mar 29, 2026
bd1d3c4
test: 28 tests for lib/eval/evidence_coverage.py
marksverdhei Mar 29, 2026
a102d8c
test: 36 tests for lib/eval/entity_coverage.py
marksverdhei Mar 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 33 additions & 11 deletions plugins/claude-code/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,20 +1,22 @@
# Supercompact — Claude Code Plugin

Entity-preservation conversation compaction for Claude Code. Replaces the built-in LLM-based `/compact` with EITF scoring — **~400x faster** and **2x better entity retention**.
Entity-preservation conversation compaction for Claude Code. **~400x faster** and **2x better entity retention** than the built-in LLM-based `/compact`.

## Quick Install

```bash
git clone https://github.com/yourusername/supercompact.git
git clone https://github.com/heiervang-technologies/supercompact.git
cd supercompact/plugins/claude-code
./install.sh
```

**Prerequisites:** Python 3.11+, [uv](https://github.com/astral-sh/uv)
The installer automatically registers the plugin in `~/.claude/settings.json`. Restart Claude Code, then use `/supercompact`.

**Prerequisites:** Python 3.11+, [uv](https://github.com/astral-sh/uv), jq

## What It Does

When Claude Code compacts your conversation (either automatically or via `/compact`), it normally calls an LLM to summarize — slow (~30s) and lossy. Supercompact replaces this with **EITF** (Entity-frequency Inverse Turn Frequency), a zero-model algorithm that:
When Claude Code compacts your conversation (either automatically or via `/compact`), it normally calls an LLM to summarize — slow (~30s) and lossy. Supercompact uses **EITF** (Entity-frequency Inverse Turn Frequency), a zero-model algorithm that:

1. Extracts structured entities (file paths, errors, functions, URLs, etc.)
2. Scores each conversation turn by entity importance × rarity
Expand All @@ -25,13 +27,13 @@ Result: compaction in **~0.2 seconds** with **2x better retention** of file path

## How It Works

The installer sets up three integration points:
The plugin provides three integration points:

1. **cli.js patch** — Replaces the LLM API call in Claude Code's main compaction function with a subprocess call to supercompact. Falls back to the original LLM on error.
1. **`/supercompact` command** — On-demand compaction. Replaces the session with a compacted version and restarts. This is the primary interface.

2. **PreCompact hook** — Backs up the full transcript before any compaction runs, and produces a supercompact alternative alongside Claude's built-in result.
2. **PreCompact hook** — When Claude's built-in compaction triggers, the hook backs up the full transcript before it's lost. The backup is saved as `*.pre-compact-full` alongside the session JSONL.

3. **`/supercompact` command** — Manual on-demand compaction with configurable method and budget.
3. **cli.js patch** *(npm installations only)* — Replaces the LLM API call in Claude Code's compaction function with supercompact. Falls back to the original LLM on error. Not available on standalone binary installations.

## Configuration

Expand Down Expand Up @@ -70,6 +72,16 @@ Manual compaction. Examples:
./install.sh --patch-only # Patch cli.js only (plugin must be installed first)
```

## Update

```bash
cd supercompact
git pull
./plugins/claude-code/install.sh
```

Re-running the installer is safe — it replaces all files and is fully idempotent.

## Uninstall

```bash
Expand All @@ -95,16 +107,26 @@ Manual compaction. Examples:
├── hooks/
│ └── hooks.json # PreCompact hook registration
├── hooks-handlers/
│ └── supercompact-precompact.sh
│ └── supercompact-precompact.sh # Backup-only hook
└── scripts/
├── patcher.py # cli.js patching logic
├── compact-session.sh # Main compaction script
├── patcher.py # cli.js patching logic
└── patch-compaction.sh
```

## Logs

Hook activity is logged to `~/.cache/supercompact/hook.log`.

## Standalone Binary Installation

If Claude Code is installed as a standalone binary (not via npm), the cli.js patch cannot be applied. The installer detects this automatically, skips patching, and configures `settings.json` for you.

In standalone mode:
- **`/supercompact`** — Works fully. This is the primary way to compact.
- **`/compact`** — Still uses Claude's built-in LLM compaction (cannot be replaced without cli.js patch).
- **PreCompact hook** — Backs up the full transcript before Claude's built-in compaction runs.

## Troubleshooting

**Compaction not working after Claude Code update:**
Expand All @@ -118,7 +140,7 @@ Check `~/.cache/supercompact/hook.log` for errors. Common causes:
- Python/uv not in PATH during compaction
- Supercompact directory removed or corrupted

**Verify patch status:**
**Verify patch status (npm installations only):**
```bash
grep -c "SUPERCOMPACT_EITF" "$(readlink -f "$(which claude)" | sed 's|[^/]*$|cli.js|')"
# 1 = patched, 0 = not patched
Expand Down
65 changes: 9 additions & 56 deletions plugins/claude-code/commands/supercompact.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,73 +1,26 @@
---
description: EITF entity-preservation compaction (~400x faster than /compact, 2x better entity retention)
argument-hint: "[budget] [--method eitf|setcover|dedup]"
allowed-tools: Bash(cd *), Bash(uv *), Bash(PROJECT_DIR*), Bash(JSONL_FILE*), Bash(ls *), Bash(wc *), Bash(cp *), Bash(mv *), Bash(restart-claude*), Bash(echo *)
allowed-tools: Bash(*/compact-session.sh*), Bash(restart-claude*)
---

# Supercompact — Entity-Preservation Compaction

**CRITICAL: Do NOT use the built-in /compact command. You must follow the exact steps below using Bash tool calls.**

You are running the supercompact algorithm. This is completely separate from Claude Code's built-in /compact. You must execute the bash commands below, not delegate to any built-in compaction.

## Configuration

Settings come from environment variables (set via plugin config):
- `PLUGIN_SETTING_METHOD` — scoring method (default: `eitf`). Options: `eitf`, `setcover`, `dedup`
- `PLUGIN_SETTING_BUDGET` — token budget (default: `80000`)

The user can override these via arguments: `/supercompact 120000 --method setcover`

## Step 1: Find the conversation JSONL and supercompact directory
Run the compaction script. It will find the session JSONL automatically, compact it, and report results.

```bash
PROJECT_DIR=$(echo "$PWD" | sed 's|/|-|g; s|^|'"$HOME"'/.claude/projects/|')
JSONL_FILE=$(ls -t "$PROJECT_DIR"/*.jsonl 2>/dev/null | head -1)
echo "JSONL: $JSONL_FILE"
wc -l "$JSONL_FILE"
SCRIPT="${CLAUDE_PLUGIN_ROOT:-${HOME}/.local/share/supercompact/claude-code/plugin}/scripts/compact-session.sh"
"$SCRIPT" $ARGUMENTS
```

## Step 2: Run compaction

Parse $ARGUMENTS for an optional numeric budget and `--method <name>`. Fall back to env vars, then defaults.
If the script succeeds and reports compaction was performed (not "already within budget"), restart to load the compacted context:

```bash
METHOD="${PLUGIN_SETTING_METHOD:-eitf}"
BUDGET="${PLUGIN_SETTING_BUDGET:-80000}"
# Override from arguments if provided (e.g. "/supercompact 120000 --method setcover")
for arg in $ARGUMENTS; do
if [[ "$arg" =~ ^[0-9]+$ ]]; then BUDGET="$arg"; fi
if [[ "$prev" == "--method" ]]; then METHOD="$arg"; fi
prev="$arg"
done
# Find supercompact installation
SUPERCOMPACT_DIR="$HOME/.local/share/supercompact/claude-code/supercompact"
if [[ ! -f "$SUPERCOMPACT_DIR/compact.py" ]]; then
echo "ERROR: supercompact not found at $SUPERCOMPACT_DIR. Run install.sh first."
exit 1
fi
echo "Method: $METHOD, Budget: $BUDGET"
cd "$SUPERCOMPACT_DIR" && uv run python compact.py "$JSONL_FILE" --method "$METHOD" --budget "$BUDGET" --output /tmp/supercompact-output.jsonl --verbose
restart-claude "Session compacted with supercompact. Restarting to load compacted context."
```

## Step 3: Replace the session JSONL

```bash
cp "$JSONL_FILE" "${JSONL_FILE}.pre-supercompact"
mv /tmp/supercompact-output.jsonl "$JSONL_FILE"
echo "Replaced session JSONL (backup: ${JSONL_FILE}.pre-supercompact)"
```

## Step 4: Report results briefly

Report: method used, turns kept vs dropped, compression ratio, wall clock time.

## Step 5: Restart to reload compacted context

The JSONL on disk is now compacted, but the live session still has old context in memory. Restart to load the compacted version:
If `restart-claude` is not available, tell the user: "Run `/quit` then `claude --resume` to load the compacted context."

```bash
restart-claude "Session compacted with supercompact ($METHOD). Restarting to load compacted context."
```
If the script reports "already within budget", tell the user and do NOT restart.

If `restart-claude` is not available, tell the user: "Run `/quit` then `claude --resume` to load the compacted context."
If the script fails, show the error output to the user and do not restart.
70 changes: 13 additions & 57 deletions plugins/claude-code/hooks-handlers/supercompact-precompact.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,42 +1,18 @@
#!/usr/bin/env bash
# supercompact-precompact.sh - PreCompact hook for entity-preservation compaction
# supercompact-precompact.sh - PreCompact hook (backup-only)
#
# Triggered when Claude Code is about to compact the conversation.
# The PreCompact hook CANNOT block or replace Claude's built-in compaction —
# it is notification-only. So we use it to:
# it is notification-only. Running supercompact here is wasted work since
# Claude's LLM compaction overwrites the result anyway.
#
# Instead, we just:
# 1. Back up the full transcript before Claude's summarization loses detail
# 2. Run compaction (configurable method) to produce a superior alternative
# 3. The user can later resume from the supercompact version instead of Claude's
#
# Configuration via environment variables:
# PLUGIN_SETTING_METHOD Scoring method (default: eitf)
# PLUGIN_SETTING_BUDGET Token budget (default: 80000)
# 2. Log the event
# 3. Clean up old backups

set -euo pipefail

# Resolve supercompact installation root
# Layout: ~/.local/share/supercompact/claude-code/plugin/hooks-handlers/THIS_SCRIPT
# ~/.local/share/supercompact/claude-code/supercompact/compact.py
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PLUGIN_ROOT="$(cd "${SCRIPT_DIR}/.." && pwd)"
INSTALL_ROOT="$(cd "${PLUGIN_ROOT}/.." && pwd)"
SUPERCOMPACT_DIR="${INSTALL_ROOT}/supercompact"

if [[ ! -f "${SUPERCOMPACT_DIR}/compact.py" ]]; then
# Fallback: check if supercompact is bundled inside the plugin dir (dev mode)
if [[ -f "${PLUGIN_ROOT}/supercompact/compact.py" ]]; then
SUPERCOMPACT_DIR="${PLUGIN_ROOT}/supercompact"
else
echo "$(date -Iseconds) ERROR: supercompact not found at ${SUPERCOMPACT_DIR}" >> "${HOME}/.cache/supercompact/hook.log" 2>/dev/null
exit 0
fi
fi

METHOD="${PLUGIN_SETTING_METHOD:-eitf}"
BUDGET="${PLUGIN_SETTING_BUDGET:-80000}"
LOG_DIR="${HOME}/.cache/supercompact"

mkdir -p "${LOG_DIR}"

# Read hook input from stdin (JSON with transcript_path, session_id, trigger, etc.)
Expand All @@ -48,42 +24,22 @@ JSONL_FILE=$(echo "${HOOK_INPUT}" | jq -r '.transcript_path // empty')
echo "$(date -Iseconds) PreCompact hook triggered (trigger=${TRIGGER})" >> "${LOG_DIR}/hook.log"

if [[ -z "${JSONL_FILE}" || ! -f "${JSONL_FILE}" ]]; then
echo "$(date -Iseconds) ERROR: No transcript_path in hook input or file missing" >> "${LOG_DIR}/hook.log"
exit 0
echo "$(date -Iseconds) ERROR: No transcript_path in hook input or file missing" >> "${LOG_DIR}/hook.log"
exit 0
fi

JSONL_SIZE=$(wc -l < "${JSONL_FILE}")
echo "$(date -Iseconds) Transcript: ${JSONL_FILE} (${JSONL_SIZE} lines)" >> "${LOG_DIR}/hook.log"

# 1. Back up the full transcript before Claude's compaction destroys detail
# Back up the full transcript before Claude's compaction destroys detail
BACKUP_FILE="${JSONL_FILE}.pre-compact-full"
cp "${JSONL_FILE}" "${BACKUP_FILE}"
echo "$(date -Iseconds) Full backup saved: ${BACKUP_FILE}" >> "${LOG_DIR}/hook.log"

# 2. Run supercompact to produce a superior alternative
SC_OUTPUT="${JSONL_FILE}.supercompact"

echo "$(date -Iseconds) Running supercompact (method=${METHOD}, budget=${BUDGET})" >> "${LOG_DIR}/hook.log"

cd "${SUPERCOMPACT_DIR}"
if uv run python compact.py "${JSONL_FILE}" \
--method "${METHOD}" \
--budget "${BUDGET}" \
--output "${SC_OUTPUT}" 2>> "${LOG_DIR}/hook.log"; then
# Clean up old backups (keep last 3)
ls -t "${JSONL_FILE}.pre-compact-full"* 2>/dev/null | tail -n +4 | xargs rm -f 2>/dev/null || true
ls -t "${JSONL_FILE}.pre-supercompact"* 2>/dev/null | tail -n +4 | xargs rm -f 2>/dev/null || true

SC_SIZE=$(wc -l < "${SC_OUTPUT}")
echo "$(date -Iseconds) Supercompact (${METHOD}): ${JSONL_SIZE} -> ${SC_SIZE} lines (saved as .supercompact)" >> "${LOG_DIR}/hook.log"

# Clean up old backups (keep last 3 of each type)
ls -t "${JSONL_FILE}.pre-compact-full"* 2>/dev/null | tail -n +4 | xargs rm -f 2>/dev/null || true
ls -t "${JSONL_FILE}.supercompact"* 2>/dev/null | tail -n +4 | xargs rm -f 2>/dev/null || true

echo "$(date -Iseconds) SUCCESS: Supercompact alternative ready at ${SC_OUTPUT}" >> "${LOG_DIR}/hook.log"
echo "$(date -Iseconds) NOTE: Claude's built-in compaction will still run (hook cannot block it)" >> "${LOG_DIR}/hook.log"
echo "$(date -Iseconds) To use supercompact version: cp '${SC_OUTPUT}' '${JSONL_FILE}'" >> "${LOG_DIR}/hook.log"
else
echo "$(date -Iseconds) ERROR: Supercompact (${METHOD}) failed (Claude's compaction will proceed)" >> "${LOG_DIR}/hook.log"
rm -f "${SC_OUTPUT}" 2>/dev/null || true
fi
echo "$(date -Iseconds) Backup-only hook complete (use /supercompact for manual compaction)" >> "${LOG_DIR}/hook.log"

exit 0
89 changes: 65 additions & 24 deletions plugins/claude-code/install.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -168,36 +168,73 @@ if [[ "$DO_INSTALL" == true ]]; then

ok "Plugin installed to ${INSTALL_DIR}"

# Print plugin-dir usage
echo ""
info "To load the plugin, use one of:"
echo " claude --plugin-dir ${PLUGIN_DEST}"
echo ""
echo " Or add to ~/.claude/settings.json:"
echo " { \"pluginDirs\": [\"${PLUGIN_DEST}\"] }"
echo ""
# Auto-configure settings.json to load the plugin
SETTINGS_FILE="${HOME}/.claude/settings.json"
info "Configuring Claude Code to load plugin..."
mkdir -p "$(dirname "${SETTINGS_FILE}")"

if [[ ! -f "${SETTINGS_FILE}" ]]; then
# Create settings.json with pluginDirs
echo '{"pluginDirs":["'"${PLUGIN_DEST}"'"]}' | jq . > "${SETTINGS_FILE}"
ok "Created ${SETTINGS_FILE} with pluginDirs"
elif jq -e '.pluginDirs' "${SETTINGS_FILE}" >/dev/null 2>&1; then
# pluginDirs exists — check if our path is already there
if jq -e --arg p "${PLUGIN_DEST}" '.pluginDirs | index($p)' "${SETTINGS_FILE}" >/dev/null 2>&1; then
ok "Plugin already registered in settings.json"
else
# Add our path to existing pluginDirs array
jq --arg p "${PLUGIN_DEST}" '.pluginDirs += [$p]' "${SETTINGS_FILE}" > "${SETTINGS_FILE}.tmp" \
&& mv "${SETTINGS_FILE}.tmp" "${SETTINGS_FILE}"
ok "Added plugin to existing pluginDirs in settings.json"
fi
else
# settings.json exists but no pluginDirs key — add it
jq --arg p "${PLUGIN_DEST}" '. + {pluginDirs: [$p]}' "${SETTINGS_FILE}" > "${SETTINGS_FILE}.tmp" \
&& mv "${SETTINGS_FILE}.tmp" "${SETTINGS_FILE}"
ok "Added pluginDirs to settings.json"
fi
fi

# ------------------------------------------------------------------
# Patch cli.js
# ------------------------------------------------------------------
PATCH_APPLIED=false
STANDALONE_BINARY=false

if [[ "$DO_PATCH" == true ]]; then
SUPERCOMPACT_DEST="${INSTALL_DIR}/supercompact"

if [[ ! -f "${SUPERCOMPACT_DEST}/compact.py" ]]; then
fatal "Supercompact not installed at ${SUPERCOMPACT_DEST}. Run install first (without --patch-only)."
fi

echo ""
info "Patching Claude Code cli.js..."
bash "${INSTALL_DIR}/plugin/scripts/patch-compaction.sh" "${SUPERCOMPACT_DEST}"
EXIT_CODE=$?
# Detect standalone binary vs npm installation
CLAUDE_BIN="${CLAUDE_BIN:-$(which claude 2>/dev/null || echo "")}"
CLAUDE_REAL=""
if [[ -n "$CLAUDE_BIN" ]]; then
CLAUDE_REAL="$(readlink -f "$CLAUDE_BIN" 2>/dev/null || echo "$CLAUDE_BIN")"
fi

if [[ $EXIT_CODE -eq 0 ]]; then
ok "cli.js patched — compaction now uses supercompact"
if [[ -n "$CLAUDE_REAL" ]] && head -c 4 "$CLAUDE_REAL" 2>/dev/null | grep -q "ELF\|MZ"; then
STANDALONE_BINARY=true
echo ""
warn "Claude Code is installed as a standalone binary (not via npm)"
warn "cli.js patching is not available for standalone installations"
info "The /supercompact slash command and PreCompact hook will still work"
info "Use '/supercompact' for on-demand compaction"
else
err "Patching failed (exit code $EXIT_CODE)"
exit $EXIT_CODE
echo ""
info "Patching Claude Code cli.js..."
bash "${INSTALL_DIR}/plugin/scripts/patch-compaction.sh" "${SUPERCOMPACT_DEST}"
EXIT_CODE=$?

if [[ $EXIT_CODE -eq 0 ]]; then
ok "cli.js patched — compaction now uses supercompact"
PATCH_APPLIED=true
else
warn "cli.js patching failed (exit code $EXIT_CODE)"
warn "The /supercompact slash command and PreCompact hook will still work"
fi
fi
fi

Expand All @@ -210,15 +247,19 @@ echo ""
echo "What's installed:"
echo " • Supercompact library at ${INSTALL_DIR}/supercompact/"
echo " • Plugin at ${INSTALL_DIR}/plugin/"
if [[ "$DO_PATCH" == true ]]; then
echo " • Plugin registered in ~/.claude/settings.json"
if [[ "$PATCH_APPLIED" == true ]]; then
echo " • cli.js patched for automatic compaction replacement"
fi
echo ""
echo "Configuration (via environment variables or plugin settings):"
echo " PLUGIN_SETTING_METHOD=eitf # eitf, setcover, dedup"
echo " PLUGIN_SETTING_BUDGET=80000 # token budget"
echo " PLUGIN_SETTING_FALLBACK_TO_BUILTIN=true # fall back to LLM on error"
echo ""
if [[ "$DO_PATCH" == true ]]; then
echo "Restart Claude Code to activate the patch."
echo "Usage:"
if [[ "$PATCH_APPLIED" == true ]]; then
echo " /compact and /supercompact both use supercompact now."
echo " Restart Claude Code to activate."
else
echo " /supercompact # On-demand entity-preservation compaction"
echo " /supercompact 120000 # Custom token budget"
fi
echo ""
echo "To update later: git pull && ./install.sh"
echo "To uninstall: ./uninstall.sh"
Loading