docs(examples): add feature tutorials and domain pipeline notebooks#72
Merged
docs(examples): add feature tutorials and domain pipeline notebooks#72
Conversation
…eAuditor - TrapPruner: statistical profiling + LLM verification of Gaussian noise columns - MissingnessRecognizer: MNAR pattern detection with sandbox-compiled encoders - TargetLeakageAuditor: semantic timeline evaluation for target leakage
…o public API - Add all three to extraction/__init__.py lazy imports - Add Loclean class methods + module-level convenience functions - Update __all__ in loclean/__init__.py
…r, TargetLeakageAuditor - 13 tests each (39 total) covering profiling, prompt construction, verdict parsing, verification, and mock-LLM integration
- Strip markdown fences, prose, and backticks - Fix unicode operators and invalid numeric literals - 17 unit tests covering all transformation stages
- LLM-generated import statements now only work for explicitly allowed modules; all others raise ImportError - Preload modules into safe_globals for direct namespace access - Updated docstring to document the restriction
- Wrap initial compile in try/except to catch ValueErrors - Add per-retry logging with attempt counter - Replace vague failure messages with actionable guidance (model suggestions, max_retries hint) - Add concrete code examples to LLM prompts for better output
- Add module-level _verified_models set for deduplication - Fix model_exists to handle both dict and object API responses - Use model attribute (not name) for correct Ollama registry matching
Superseded by examples/README.md and module docstrings.
- 06-entity-resolution: canonicalize messy string variations - 07-oversampling: Pydantic-schema-driven minority-class augmentation - 08-log-shredding: parse unstructured logs into relational tables - 09-feature-discovery: LLM-proposed feature engineering - 10-quality-validation: plain-English rule evaluation
- 11-kaggle-housing-pipeline: full data-science workflow (clean → resolve → discover → oversample → validate → scrub) - 12-log-engineering-pipeline: full data-engineering workflow (extract → compiled-extract → shred → resolve → validate → scrub) Removed redundant generic end-to-end notebook (subsumed by these domain-specific pipelines).
- Remove entry for deleted generic end-to-end pipeline (11) - Renumber Kaggle housing to 11, log engineering to 12
… optimizer - 13-trap-pruning: detect and remove Gaussian noise columns - 14-missingness-recognition: MNAR pattern detection with boolean flags - 15-leakage-auditing: semantic timeline evaluation for leakage - 16-instruction-optimization: reward-driven prompt refinement
- Break long f-string in 06-entity-resolution - Split log entry string literals in 12-log-engineering-pipeline - Extract variables in quality gate print statements
Pull Request Summary by devActivityMetrics
Achievements
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds 7 new example notebooks covering all Loclean features added since v0.1, plus a comprehensive README update.
Focused Feature Tutorials (06-10)
Each notebook demonstrates one API with beginner-friendly data and clear explanations:
loclean.resolve_entities()loclean.oversample()loclean.shred_to_relations()loclean.discover_features()loclean.validate_quality()Domain Pipeline Notebooks (11-12)
Two realistic end-to-end pipelines that combine multiple features:
Audit Decisions
zip()calls to usestrict=True(B905)scrub→maskedin loop vars)README Updates
Updated the Available Notebooks section to reflect the new numbering and removed the dropped notebook entry.