test: 24 tests for lib/types.py and lib/eval/probes.py#23
Open
hai-pilgrim wants to merge 8 commits into
Open
Conversation
…ompaction - install.sh now auto-configures ~/.claude/settings.json (creates pluginDirs entry, idempotent across create/add/already-exists cases) - uninstall.sh now cleans up the settings.json pluginDirs entry - Add compact-session.sh: self-contained script that finds JSONL, runs supercompact, backs up original, replaces, and reports results - Simplify /supercompact command from 5-step multi-bash prompt to single script call with CLAUDE_PLUGIN_ROOT fallback to hardcoded install path - Simplify PreCompact hook to backup-only (removes wasted supercompact run that Claude's LLM compaction immediately overwrites) - Update README: accurate hook description, file tree with compact-session.sh, update/upgrade docs, standalone binary limitations clearly stated Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
export_json: file creation, empty array, result structure (method, budget, model_key, composite, ndcg), speed/token counts, dimension scores with score and probe_count, multiple results, valid JSON. export_trace: file creation, path location, filename contains method/budget, JSON has method/budget, empty answers → empty entries, matching probe included, unmatched answer skipped, auto-creates trace dir.
Tests cover ProbeAnswer/JudgeResult dataclass defaults and field storage, ANSWER_MODELS/JUDGE_MODEL constants, generate_answers with empty probe set, score_answers missing-probe path (no API call), missing OPENROUTER_API_KEY error, and _score_one_answer JSON parsing, markdown fence stripping, score clamping, and bad-JSON fallback.
Tests cover DIFFICULTY_WEIGHTS constant, DimensionScore/AggregateResult dataclass defaults and fields, dimension_map property, _dcg (empty, single, sorting by weight, zero scores, position discounting), and aggregate() (empty answers, single/multiple models, score 0-1 normalisation, missing probes skipped, empty-dimension zero-mean, perfect/zero/partial NDCG).
Tests cover DIFFICULTY_WEIGHTS constant, ProbeCoverage/DimensionCoverage/ EvidenceCoverageResult dataclasses, dimension_map property, to_dict keys, _dcg (empty, single, zero-score, weight-sorted), and compute_evidence_coverage (empty probe set, probe with no evidence_turns skipped, full coverage → 1.0, zero coverage → 0.0, partial coverage value, kept/dropped lists, multi-probe mean, NDCG perfect/zero/partial).
Tests cover EntitySet (total_count, all_entities, default dict), ENTITY_TYPES constants (presence, positive weights), extract_entities for exceptions, URLs, ports (with range filtering), file paths, CamelCase class names, pip/npm packages, and HTTP status codes. Also covers compute_coverage (empty-suffix → 1.0, empty-kept → 0.0, identical sets → 1.0, breakdown structure, half coverage, type mismatch → 0.0, weighted vs unweighted divergence).
Tests cover Turn dataclass (defaults, custom, append), _is_user_message (string content, text block, tool_result block, non-user type, source UUID injection, empty list), extract_text (string content, multiple records joined, empty turn, text/thinking/tool_use/tool_result blocks, input truncation at 500 chars, nested tool_result list, non-dict block skipped, multiple blocks concatenated), and parse_jsonl (empty file, single user, user+assistant, sequential indexing).
Tests cover ScoredTurn dataclass, build_query (single turn, last-3 slicing, fewer-than-3, max_chars truncation, empty input, separator), random_scores (count, ScoredTurn type, 0-1 range, token lookup, missing → 0), Probe dataclass (fields, default evidence_turns/difficulty), ProbeSet (defaults, to_dict structure, from_dict roundtrip, missing optional fields), and _format_turns_for_prompt (content included, header format, ordering, truncation, empty list).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
tests/test_types_and_probes.pywith 24 pytest tests acrosslib/types.pyandlib/eval/probes.pyScoredTurn: field storagebuild_query: single turn, last-3 slicing, fewer-than-3 uses all,max_charstruncation, empty input,---separatorrandom_scores: one per turn, returnsScoredTurn, scores in 0-1, token lookup, missing key → 0Probe: field storage,evidence_turnsdefault empty,difficultydefault mediumProbeSet: defaults,to_dictstructure and probe serialisation,from_dictroundtrip, missing optional fields use defaults_format_turns_for_prompt: content present, turn header format, ordering,max_charstruncation, empty list → ""Test plan
uv run pytest tests/test_types_and_probes.pyTurnobjects as fixtures🤖 Opened by hai-pilgrim as part of the Pilgrim wandering-agent contribution run.