Skip to content

feat(learning): close the feedback loop — rating-weighted readback & synthesis automation#966

Open
A13xSM wants to merge 1 commit intodanielmiessler:mainfrom
A13xSM:feat/learning-readback-improvements
Open

feat(learning): close the feedback loop — rating-weighted readback & synthesis automation#966
A13xSM wants to merge 1 commit intodanielmiessler:mainfrom
A13xSM:feat/learning-readback-improvements

Conversation

@A13xSM
Copy link
Copy Markdown

@A13xSM A13xSM commented Mar 16, 2026

Summary

PAI's learning system captures data aggressively but reads back almost nothing at session start. The readback pipeline has three critical issues:

  • Broken regex: loadLearningDigest() matches **Feedback:** which hits only a fraction of learning files. Most files use **Sentiment Summary:** instead
  • Shallow extraction: loadFailurePatterns() reads CONTEXT.md but only uses the directory slug, discarding the root cause analysis
  • Dead subsystems: LearningPatternSynthesis, Wisdom Frames, and Reflections tools exist but have no automation — they've never been run

Changes

Change File Impact
Fix regex + rating-weight learnings hooks/lib/learning-readback.ts Matches both field patterns; lowest-rated (most informative) shown first
Read failure root cause analysis hooks/lib/learning-readback.ts Condensed What Happened paragraph instead of directory slug
Add synthesis readback hooks/lib/learning-readback.ts New loadSynthesisPatterns() reads Top Issues + Recommendations
Wire synthesis into LoadContext hooks/LoadContext.hook.ts Synthesis patterns injected at session start
Automate synthesis hooks/WorkCompletionLearning.hook.ts SessionEnd stale-check: spawns LearningPatternSynthesis in background if >7 days or >50 new ratings
CLI wrapper for Algorithm hooks/lib/learning-readback-cli.ts --depth standard|extended for effort-tier-aware readback
Extended+ learning enrichment PAI/Algorithm/v3.7.0.md OBSERVE phase loads deeper learning context at Extended+ effort

Before / After

Before (session start context — directory slugs only):

[2026-01-15] assistant-made-incorrect-assertion-without-checking
[2026-01-14] migration-failed-silently-data-lost

After (session start context — root cause analysis + rating):

[3/10] 2026-01-15 The assistant asserted a specific technical state without verifying via tools, contradicting the user's observations.
[2/10] 2026-01-14 A complex system upgrade was attempted but critical data was lost post-migration due to incomplete verification.

Plus synthesis patterns:

*Top issues:*
  1. Tool/System Failures (N occurrences, avg rating X)
*Actions:*
  1. Ask clarifying questions before starting complex tasks

Test plan

  • bun run hooks/lib/learning-readback-cli.ts --depth standard — rating-weighted learnings, root cause failure analysis
  • bun run hooks/lib/learning-readback-cli.ts --depth extended — more failures + learnings + synthesis
  • bun run hooks/lib/learning-readback-cli.ts --synthesis — reads Top Issues + Recommendations from SYNTHESIS/
  • bun run PAI/Tools/LearningPatternSynthesis.ts --month — bootstraps SYNTHESIS/ directory
  • Full session test: verify PAI Dynamic Context block at session start includes richer content
  • Extended task test: verify Algorithm OBSERVE invokes learning-readback-cli

🤖 Generated with Claude Code

@A13xSM A13xSM force-pushed the feat/learning-readback-improvements branch from 2079b3b to 34d6567 Compare March 16, 2026 08:06
…ynthesis automation, effort-tier scaling

The learning system captures extensively but reads back almost nothing useful
at session start. This PR fixes the readback pipeline:

- Fix loadLearningDigest: match both **Feedback:** and **Sentiment Summary:** patterns
  (dramatically improves hit rate), weight by rating (lowest = most informative), deduplicate
- Fix loadFailurePatterns: read CONTEXT.md root cause analysis instead of directory slugs
- Add loadSynthesisPatterns: read aggregated pattern reports from SYNTHESIS/
- Add synthesis automation: SessionEnd stale-check triggers LearningPatternSynthesis
  in background when >7 days or >50 new ratings since last run
- Add learning-readback-cli.ts: CLI wrapper for Algorithm OBSERVE phase (Extended+)
- Add effort-tier-aware learning enrichment to Algorithm v3.7.0 OBSERVE phase

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant