feat: implement issue #23 auto-readme-generator multi-agent pipeline#35
Conversation
Implements a 5-step pipeline for generating structured, validated READMEs: 1. Project Analyzer (analyzer.py) - Extracts metadata: language, package_manager, framework, project_type - Uses deepseek-v3.2 for JSON parsing 2. Section Planner (planner.py) - Deterministic section planning by project_type - web-api/cli/library/web-app/other templates 3. README Writer (writer.py) - LLM generation with llama-3.3-70b - Injects metadata for consistency - Enforces shields.io badges and language-tagged code blocks 4. Section Validator (validator.py) - Regex validation: required sections, badge URLs, code fences - Returns structured issue list 5. Refinement Agent (refiner.py) - Max 2 retries to fix detected issues - Re-targets only broken sections Backend: tool.py, analyzer.py, planner.py, writer.py, validator.py, refiner.py, llm_client.py Tests: test_planner.py, test_validator.py, test_refiner.py, test_e2e_mock.py (13 tests total) Frontend: auto-readme-generator.ts, registry.ts (+2 lines) CI: ci-auto-readme-generator.yml (isolated)
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
![oxlo-ai[bot]](https://avatars.githubusercontent.com/in/3523046?s=60&v=4){kind=small}
There was a problem hiding this comment.
OxBot Review
This PR implements an auto-readme-generator with a multi-agent pipeline (analyzer, planner, writer, validator, refiner) that generates project READMEs from metadata via LLM calls. While the Python backend shows clear stage separation and the tests cover key happy paths, the submission is undermined by a fundamental architectural mismatch: it claims to be a standalone community project under projects/ yet modifies core application files in app/src/lib/tools/ and places service code under services/python-tools/tools/. Furthermore, the pipeline lacks defensive error handling and null-safety, uses brittle JSON extraction without schema validation, and interpolates user input directly into prompts, creating reliability and security risks. The requirements.txt is also missing essential Tier 2 service dependencies such as FastAPI and uvicorn, and the CI workflow paths do not align with the stated project structure. These structural, robustness, and dependency gaps need to be resolved before merge.
Notes
- Architectural tier mismatch: The submission checklist marks this as a community project under
projects/, but the PR modifies the core app registry and places code inservices/python-tools/. Community projects must be fully self-contained; built-in tools require proper Tier 2 FastAPI integration and should not claimprojects/status. - Cross-cutting robustness issues: The pipeline uses brittle regex/string parsing for LLM outputs, lacks Pydantic schema validation, and has inconsistent exception handling across stages, which can crash the entire stream on a single malformed response or null input.
- Security and input validation: User-controlled metadata fields are interpolated directly into system prompts without sanitization, creating prompt injection risks; additionally,
data.get()with defaults does not guard against explicitnullvalues, leading to potentialAttributeErrorcrashes. - Dependency and infrastructure gaps: The
requirements.txtomits FastAPI, uvicorn, and pydantic, which are required for Tier 2 tools, while the CI workflow monitors the wrong path and installs an incomplete dependency set. - Frontend/backend contract: The frontend tool definition is missing a required
outputFormatfield and uses an ambiguous tier label, while the backend yields plain-text markers that require a documentedStreamingResponsecontract with the frontend. - Testing coverage: While unit tests for individual agents are clean, the suite lacks integration tests for the full pipeline and does not exercise failure modes such as max-refinement retries or malformed LLM responses.
Automated review by OxBot
- Updated CI workflow to install dependencies from requirements.txt - Added output format option in ToolDefinition - Introduced project metadata handling in analyzer - Sanitized inputs in refine_readme and write_readme functions - Implemented code block extraction in validator - Created unit tests for the auto-readme-generator pipeline
There was a problem hiding this comment.
OxBot Review
This PR implements a multi-agent README generation pipeline (analyzer, planner, writer, validator, refiner) for issue #23, complete with a FastAPI tool entrypoint, TypeScript tool definitions, GitHub Actions CI, and unit/integration tests. The code demonstrates strong separation of concerns and good testability via dependency injection, but it is undermined by prompt injection vulnerabilities, fragile LLM response handling, and an unresolved architectural tension between being a standalone community project and a core platform tool. Several files also leak internal state or exceptions, and the dependency management needs cleanup before production deployment.
Notes
- There is a fundamental architectural mismatch: the PR is framed as a community project under
projects/, yet it places code inservices/python-tools/tools/and modifies core application files (app/src/lib/tools/registry.ts,app/src/types/index.ts). Please clarify whether this is a built-in Tier 1/2 tool or a standalone submission and restructure accordingly. - Multiple prompt injection vulnerabilities exist across the Python backend (analyzer.py, writer.py) where raw user input is interpolated into LLM prompts between XML-like tags without escaping. This should be treated as a cross-cutting security concern requiring a unified input sanitization strategy.
- Pipeline resilience is insufficient for a multi-agent system: unhandled Pydantic ValidationError in the analyzer, fragile response parsing in the LLM client, and raw exception strings yielded directly to clients in tool.py create cascading failure and information disclosure risks.
- The requirements.txt mixes production and test dependencies with inconsistent version pinning, which will bloat the Docker image and harm build reproducibility. Split these into separate files or Docker stages.
- Several test assertions (particularly in test_pipeline.py) are weak or logically inconsistent regarding failure paths; strengthen integration tests to assert positive fallback behavior rather than just the absence of error strings.
- If this is intended to be a core tool, ensure all prompt quality, model selection, and Oxlo API client configurations are reviewed and that the tool definition file is properly linked in the registry. If it is a community project, remove all changes to
app/src/and relocate the code toprojects/auto-readme-generator/.
Automated review by OxBot
- Add improved JSON parsing with relaxed regex patterns in analyzer - Implement ValidationError handling for metadata parsing fallback - Strengthen LLM response validation with comprehensive checks - Update CI workflow to install dev requirements for testing - Simplify production dependencies while maintaining core functionality
There was a problem hiding this comment.
OxBot Review
This PR introduces an auto-readme-generator built as a multi-agent pipeline with five stages—analyze, plan, write, validate, and refine—streaming progress updates to the user. The code shows solid separation of concerns and thoughtful UX design, but the submission suffers from a fundamental architectural identity crisis: it claims to be a standalone community project under projects/ while simultaneously integrating into the core built-in tool registry and residing in services/python-tools/tools/. Compounding this, the async pipeline entrypoint lacks uniform exception handling, which risks silent stream aborts, and the test suite uses non-idiomatic asyncio.run patterns. Several modules also contain likely invalid LLM identifiers and overly strict markdown validation regexes. These issues must be resolved before the PR can land.
Notes
- Clarify the intended contribution tier. Community projects under
projects/must not modifyapp/src/lib/tools/registry.tsor live inservices/python-tools/tools/; built-in tools require FastAPI integration files and should not claim community-project status. - The async generator entrypoint lacks uniform exception handling; unhandled errors in analyzer, refiner, or writer stages will abruptly terminate the stream instead of yielding graceful error messages to the client.
- The test suite mixes synchronous wrappers with async code via
asyncio.run; adopt@pytest.mark.asyncioandasync deftests to align with Python best practices and prevent event loop conflicts. - CI workflow path filters and file locations must be reconciled with the final directory structure, whether the project lands in
projects/orservices/python-tools/tools/. - Several pipeline modules reference potentially invalid model identifiers and leave LLM API exceptions unhandled; audit all agent modules for consistent error boundaries before merging.
- The validator's fenced code block regex rejects valid GitHub Flavored Markdown info strings, which will cause false positives in real-world READMEs.
Automated review by OxBot
- validator.py: relax FENCE_OPEN regex to accept GFM info strings (Shashank) - tool.py: wrap plan_sections and validate_readme in try/except - tool.py: separate refiner and post-refine validator exception handlers - refiner.py: add early return guard when issues is empty - refiner.py: fix Awaitable[str] type hint on call_model - analyzer.py: wrap call_oxlo_chat in try/except with DEFAULT_METADATA fallback - tests: convert asyncio.run to async def with pytest.mark.asyncio - tests: add pytest.ini with asyncio_mode = auto
2 participants
Oxtools Submission
Project name: auto-readme-generator | issue #23
Contributor: Franci-343
Submission checklist
Check every box before requesting a review. Unchecked items will result in the PR being sent back.
Structure
projects/[my-project-name]/Required files
Dockerfileis present anddocker build .succeedsdocker-compose.ymlis present anddocker compose upstarts the appoxlo-manifest.jsonis present and all fields are filled in.env.examplelists every environment variable the project needs (with empty values)README.mdis present with setup instructions a reviewer can follow exactlySecurity
.envfile is not included in this PRgit grep -i "api_key"and found no leaksOxlo API
OXLO_API_KEYenvironment variableFor maintainers
docker build .succeeded locallydocker compose upran successfully and app is reachable