Add taboo search mode to prevent re-evaluating similar solutions#131
Open
nikivanstein wants to merge 4 commits into
Open
Add taboo search mode to prevent re-evaluating similar solutions#131nikivanstein wants to merge 4 commits into
nikivanstein wants to merge 4 commits into
Conversation
Introduces four new LLaMEA parameters (taboo_mode, taboo_similarity_threshold, taboo_max_retries, taboo_fitness_scaling) that enable a tabu-inspired filter. Before a newly generated candidate is evaluated, it is compared against the full run_history using the configured distance metric (default: AST-based code_distance). If any history entry is closer than the effective threshold, the candidate is discarded and the LLM is asked to regenerate up to taboo_max_retries extra times, after which the last candidate is accepted to avoid wasting budget. When taboo_fitness_scaling is enabled, the threshold is scaled by (1 - normalised_parent_fitness) so that offspring of strong parents face a lenient check (small improvements remain eligible) while offspring of weak parents must produce more diverse code. https://claude.ai/code/session_01EWsJqfh3NGhFtwpfbgGFk1
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5bd0e9423f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| evolved_individual = individual.empty_copy() | ||
| parent_ids = evolved_individual.parent_ids | ||
| max_attempts = (self.taboo_max_retries + 1) if self.taboo_mode else 1 |
There was a problem hiding this comment.
When running from an archive pickled before this change, __setstate__ restores an object that does not have the new taboo_mode/taboo_max_retries attributes, so the first call to evolve_solution raises AttributeError here before entering the try block. This breaks the existing LLaMEA.warm_start(...).run() workflow for older experiment archives unless defaults for the new fields are added during unpickling.
Useful? React with 👍 / 👎.
Replaces the implicit "check all of run_history" behaviour with an explicit, configurable taboo_list that is populated according to one of three strategies: * "always" (default) – every evaluated solution is added, preserving the original behaviour. * "stagnation" – solutions are added only while no improvement has been observed for taboo_stagnation_window consecutive generations, mirroring classic tabu-search diversification. * "poor_fitness" – only solutions whose fitness falls in the worst taboo_poor_fitness_percentile % of run_history are added, focusing the filter on genuinely bad code regions. _is_taboo now compares against self.taboo_list (not run_history). _update_taboo_list is called from initialize() and the main run() loop. Tests updated to seed taboo_list directly; nine new tests cover the three strategies plus the ValueError guard for unknown strategy names. https://claude.ai/code/session_01EWsJqfh3NGhFtwpfbgGFk1
experiments/simulate_taboo.py replays recorded LLaMEA-HPO run files to estimate the effect of taboo filtering without running a live LLM. For each .jsonl file it precomputes a pairwise AST-distance matrix once, then sweeps a configurable grid of taboo settings (strategy × threshold × window/percentile) and reports per-configuration pruning rate, precision, and recall metrics. Supports both LLaMEA-HPO field names (_solution/_fitness/_generation) and the standard LLaMEA log format (code/fitness/generation). Also adds pandas as a dependency (required by the experiment script). https://claude.ai/code/session_01EWsJqfh3NGhFtwpfbgGFk1
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Introduces four new LLaMEA parameters (taboo_mode, taboo_similarity_threshold,
taboo_max_retries, taboo_fitness_scaling) that enable a tabu-inspired filter.
Before a newly generated candidate is evaluated, it is compared against the full
run_history using the configured distance metric (default: AST-based
code_distance). If any history entry is closer than the effective threshold, the
candidate is discarded and the LLM is asked to regenerate up to
taboo_max_retries extra times, after which the last candidate is accepted to
avoid wasting budget. When taboo_fitness_scaling is enabled, the threshold is
scaled by (1 - normalised_parent_fitness) so that offspring of strong parents
face a lenient check (small improvements remain eligible) while offspring of
weak parents must produce more diverse code.
https://claude.ai/code/session_01EWsJqfh3NGhFtwpfbgGFk1