diff --git a/.specallowlist b/.specallowlist index 36c5537..c9ff852 100644 --- a/.specallowlist +++ b/.specallowlist @@ -1,5 +1,6 @@ # Documentation *.md +context/technical-notes/*.md LICENSE # Python package diff --git a/.specignore b/.specignore index 65e8d2f..7aa211c 100644 --- a/.specignore +++ b/.specignore @@ -16,4 +16,5 @@ specs/architecture/solutions/SOL-001.md # Implementation Design examples specs/design/IMP-001.md -# Note: TN-001 moved to docs/analysis/ as it's an analysis document rather than a specification +# Technical notes are validated but don't require test coverage +context/technical-notes/*.md diff --git a/CHANGELOG.md b/CHANGELOG.md index af55dd9..4498b3a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,7 +13,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Implemented Issue #42: improved reference resolution and flexible file classification (typed / unmanaged / external / excluded), added `.specignore` support, and CLI flags for strict validation and external URL checking (#44) ### Docs -- Investigation and solution notes for Issue #42 added to `technical-notes/issue-42-investigation.md` (#43) +- Investigation and solution notes for Issue #42 added to `context/technical-notes/TN-004.md` (#43) ### Changed - Reference resolution flow updated to support the new classification system and fallbacks for unmanaged files; added tests and integration scenarios for mixed typed/unmanaged content (#44) diff --git a/docs/analysis/TN-001.md b/context/technical-notes/TN-001.md similarity index 100% rename from docs/analysis/TN-001.md rename to context/technical-notes/TN-001.md diff --git a/technical-notes/llm-provider-selection.md b/context/technical-notes/TN-002.md similarity index 97% rename from technical-notes/llm-provider-selection.md rename to context/technical-notes/TN-002.md index 67c3700..8ff53a8 100644 --- a/technical-notes/llm-provider-selection.md +++ b/context/technical-notes/TN-002.md @@ -1,11 +1,12 @@ -# Technical Note: LLM Provider Selection for Semantic Test Adherence +# TN-002: LLM Provider Selection for Semantic Test Adherence +**Type**: Technical Note **Date**: 2025-10-23 **Author**: Claude (AI Assistant) **Status**: Draft **Related**: Milestone 001 - Semantic Test Adherence (SPEC-003) -## Executive Summary +## Abstract This technical note evaluates LLM provider options and integration libraries for implementing the semantic test adherence checker (SPEC-003). The analysis focuses on: @@ -20,9 +21,15 @@ This technical note evaluates LLM provider options and integration libraries for This approach provides maximum flexibility, built-in retry/fallback logic, and a unified API surface while supporting both cost-effective CI/CD and enterprise deployment scenarios. -## Requirements Context +## Background -From SPEC-003, the semantic test adherence checker must: +### Context + +The semantic test adherence checker (SPEC-003) requires LLM capabilities to analyze requirement-test pairs and verify semantic alignment. The system must operate effectively in multiple contexts: cost-free GitHub Actions CI/CD, local development, and production enterprise deployments. + +### Requirements from SPEC-003 + +The semantic test adherence checker must satisfy the following requirements: - **REQ-021**: Support multiple AI/LLM backends - **REQ-022**: Support Anthropic Claude (via API), OpenAI GPT, and local models (Ollama) @@ -32,6 +39,8 @@ From SPEC-003, the semantic test adherence checker must: - **REQ-041**: Support concurrent LLM requests (default: 5) - **REQ-042**: Cache results to avoid re-analyzing unchanged code +### Key Constraints + The tool will analyze requirement-test pairs to verify semantic alignment, requiring: - Robust error handling for API failures - Cost-effective operation in CI/CD (GitHub Actions) diff --git a/technical-notes/dsl-type-definition-schema.md b/context/technical-notes/TN-003.md similarity index 85% rename from technical-notes/dsl-type-definition-schema.md rename to context/technical-notes/TN-003.md index 844b1ee..c35a127 100644 --- a/technical-notes/dsl-type-definition-schema.md +++ b/context/technical-notes/TN-003.md @@ -1,8 +1,37 @@ -# Type Definition Schema Design +# TN-003: Type Definition Schema Design +**Type**: Technical Note **Date**: 2025-10-25 -**Status**: Implementation -**Related Spec**: SPEC-004 +**Author**: Claude +**Status**: Published +**Related**: SPEC-004 + +## Abstract + +This document defines the YAML schema for DSL type definitions, incorporating ID-based reference resolution where links are treated as typed references to module and class instances. The schema provides a declarative approach to specifying document structure, validation rules, and cross-document relationships for markdown-based specifications. + +**Key concepts**: Module and class type definitions, ID-based linking, reference semantics, content validators, and validation strategies. + +## Background + +### Context + +The spec-check project implements a DSL (Domain-Specific Language) for defining and validating structured markdown specifications. To support this, we need a schema format that can express: +- Document structure requirements (sections, headings, content types) +- Identifier patterns and uniqueness constraints +- Cross-document reference types and cardinality rules +- Content validation rules (Gherkin, EARS, custom formats) + +### Motivation + +YAML was chosen as the schema definition format because it: +- Is human-readable and writable +- Supports complex nested structures +- Has wide tool support +- Can be version-controlled effectively +- Allows comments for documentation + +The ID-based reference system enables location-independent linking, where documents can reference each other by stable identifiers rather than file paths, providing better refactoring support and semantic clarity. ## Overview @@ -703,3 +732,34 @@ The system should provide a migration tool: spec-check migrate-to-dsl --analyze # Report what needs changing spec-check migrate-to-dsl --convert # Auto-convert where possible ``` + +## Conclusion + +This type definition schema design provides a comprehensive foundation for expressing specification document structures in YAML format. The schema supports: + +**Core Capabilities**: +- **Module definitions** for document-level types (Requirements, Contracts, ADRs, etc.) +- **Class definitions** for section-level patterns (Test Cases, Acceptance Criteria, etc.) +- **ID-based reference system** enabling location-independent linking +- **Content validators** for domain-specific syntax (Gherkin, EARS, etc.) +- **Cardinality constraints** for relationship validation +- **Flexible scope rules** (global, directory, module, section-scoped IDs) + +**Key Design Decisions**: +1. **Declarative YAML format** for easy authoring and version control +2. **Separation of modules and classes** for reusability +3. **ID-based linking** rather than file paths for stability +4. **Pluggable content validators** for extensibility +5. **Multi-strategy resolution** supporting different project organizations + +**Implementation Status**: +The Python implementation in `spec_check/dsl/models.py` provides Pydantic-based models corresponding to this schema design, enabling type-safe schema definitions with excellent IDE support and validation. + +**Next Steps**: +- YAML schema loader implementation +- Migration tooling for existing specifications +- Additional content validators (table structure, RFC 2119 keywords) +- Schema visualization tools +- Template generation from schemas + +This schema design balances expressiveness with simplicity, enabling rich specification structures while keeping type definitions readable and maintainable. diff --git a/technical-notes/issue-42-investigation.md b/context/technical-notes/TN-004.md similarity index 90% rename from technical-notes/issue-42-investigation.md rename to context/technical-notes/TN-004.md index d0d6bd4..e64599c 100644 --- a/technical-notes/issue-42-investigation.md +++ b/context/technical-notes/TN-004.md @@ -1,8 +1,28 @@ -# Issue #42 Investigation: validate-dsl Reference Resolution Failures +# TN-004: Issue #42 Investigation - validate-dsl Reference Resolution Failures -**Issue**: validate-dsl fails to resolve references between validated documents +**Type**: Technical Note **Date**: 2025-10-30 -**Status**: Root cause identified, solution proposed +**Author**: Claude +**Status**: Published +**Related**: Issue #42 + +## Abstract + +Investigation into `validate-dsl` command failures when resolving cross-directory file path references. The issue manifested as ~548 reference resolution failures in a typical project, with the command unable to resolve references between validated documents even when both source and target files were present. + +**Root Cause**: Files are only resolvable if registered in the ID registry, and registration only occurs when files match a defined type definition. Documents that don't match any type are never registered and cannot be referenced. + +**Solution**: Implement support for "unmanaged" documents (files without type definitions) that can still be referenced by file path, with clear distinction between typed and untyped document references. + +## Background + +### Context + +The `validate-dsl` command performs cross-document reference validation as part of the spec-check validation pipeline. It was designed to validate references between strongly-typed specification documents (Requirements, ADRs, Jobs, etc.) using ID-based linking. + +### Problem Discovery + +Users reported that `validate-dsl` failed to resolve references in real-world repositories containing mixed content: typed specifications alongside general documentation, design notes, roadmaps, and other markdown files. The validator would fail with "Module reference not found" errors for perfectly valid file path references. ## Problem Statement @@ -993,3 +1013,33 @@ References validated: 35 - PR #34: Auto-ignore VCS directories in linter - PR #37: Support file path references in validate-dsl - `spec_check/linter.py:221`: Existing VCS directory exclusion pattern + +## Conclusion + +**Root Cause Identified**: The validate-dsl reference resolution failures stem from an architectural assumption that all validated documents must match a type definition. Files without matching types are never registered in the ID registry and cannot be resolved as reference targets. + +**Recommended Solution**: +1. **Support unmanaged documents** - Files that don't match type definitions can still be validated and referenced by file path +2. **Two-tier resolution strategy**: + - Typed documents: Use ID-based references with full type checking + - Unmanaged documents: Use file path references with existence checking only +3. **Auto-exclusion of VCS directories** - Automatically ignore `.git/`, `.claude/`, `.venv/`, etc. to reduce noise +4. **Clear reporting** - Distinguish between typed and unmanaged documents in validation output + +**Implementation Priority**: High - This blocking issue prevents validate-dsl from working in real-world repositories with mixed content. + +**Design Principles**: +- **Low friction**: Repositories with general docs should validate without additional configuration +- **Gradual typing**: Teams can incrementally add type definitions without breaking existing references +- **Clear semantics**: Typed vs untyped references have different validation rules and capabilities +- **Backward compatibility**: Existing fully-typed repositories continue to work identically + +**Next Steps**: +1. Implement `UnmanagedDocumentType` class for files without type definitions +2. Add VCS directory auto-exclusion (reuse linter patterns) +3. Update reference resolver to handle both typed and file-path references +4. Enhance validation reporting with type breakdowns +5. Add `--strict` mode for teams that want explicit classification + +This investigation demonstrates the importance of designing validation tools that accommodate real-world usage patterns rather than idealized scenarios. The solution balances strong typing for specifications with pragmatic handling of mixed-content repositories. + diff --git a/technical-notes/specification-dsl-design.md b/context/technical-notes/TN-005.md similarity index 94% rename from technical-notes/specification-dsl-design.md rename to context/technical-notes/TN-005.md index 96b2996..2f16b48 100644 --- a/technical-notes/specification-dsl-design.md +++ b/context/technical-notes/TN-005.md @@ -1,53 +1,41 @@ -# Specification DSL Design Document +# TN-005: Specification DSL Design +**Type**: Technical Note **Date**: 2025-10-24 -**Status**: Proposal -**Related Spec**: SPEC-004 +**Author**: Claude +**Status**: Published +**Related**: SPEC-004 -## Overview +## Abstract This document describes the design of a domain-specific language (DSL) for defining and validating structured markdown specifications. The system provides a schema layer for markdown documents, enabling machine-enforceable structure while preserving the human-readability and standard tooling compatibility of markdown. -## Problem Statement - -Organizations maintain technical specifications, requirements, contracts, and architectural documents in markdown format. These documents follow implicit conventions around structure, naming, cross-references, and content patterns. Currently, these conventions exist only in documentation and tribal knowledge, leading to inconsistency and manual review overhead. - -The goal is to formalize these conventions as explicit, machine-validated type definitions while keeping the actual specification documents as standard markdown that renders correctly in any markdown viewer. +**Key Innovation**: A two-layer architecture where YAML type definitions describe document schemas, and standard markdown files conform to these schemas while remaining fully compatible with existing markdown parsers and viewers. -## Core Concept +**Core Concepts**: Modules (document-level types), Classes (section-level patterns), References (typed relationships), Identifiers (stable unique designators), and Content Validators (domain-specific syntax checkers). -The system operates as a two-layer architecture: +## Background -**Type Definition Layer** - YAML documents that define schemas for specification document types. These schemas describe file naming patterns, section structure, required fields, identifier patterns, and allowed cross-references. +### Problem Statement -**Document Layer** - Standard markdown files that conform to the defined types. These remain fully compatible with standard markdown parsers and viewers, but can be validated against their type definitions for correctness. +Organizations maintain technical specifications, requirements, contracts, and architectural documents in markdown format. These documents follow implicit conventions around structure, naming, cross-references, and content patterns. Currently, these conventions exist only in documentation and tribal knowledge, leading to: +- Inconsistency across specifications +- Manual review overhead +- Broken references that go undetected +- Difficulty enforcing organizational standards -This separation ensures that specifications remain human-readable first-class markdown while gaining the benefits of machine-enforced structure. +### Design Goal -## Fundamental Abstractions +Formalize these conventions as explicit, machine-validated type definitions while keeping the actual specification documents as standard markdown that renders correctly in any markdown viewer. This separation ensures that specifications remain human-readable first-class markdown while gaining the benefits of machine-enforced structure. -### Modules +### System Overview -A module is a file-level type definition. It describes an entire specification document class such as a requirement, contract, or architecture decision record. Modules define: - -- File naming patterns and allowed locations in the repository -- The relationship between filename conventions and document titles -- Identifier patterns and where they appear in the document -- Required and optional section structure -- Allowed cross-references to other module types -- Cardinality constraints for relationships - -**Example Module Use Cases:** -- A "Requirement" module that enforces EARS format with required sections -- A "Contract" module that validates legal document structure -- An "ADR" module that ensures architectural decisions follow a template +The system operates as a two-layer architecture: -### Classes +**Type Definition Layer** - YAML documents that define schemas for specification document types. These schemas describe file naming patterns, section structure, required fields, identifier patterns, and allowed cross-references. -A class is a section-level type definition. It describes a repeatable structural pattern within a document, such as an acceptance criterion, a contract clause, or a risk assessment. Classes define: +**Document Layer** - Standard markdown files that conform to the defined types. These remain fully compatible with standard markdown parsers and viewers, but can be validated against their type definitions for correctness. -- Heading patterns and nesting levels -- Required fields within the section - Content validation rules - Nested sub-structures diff --git a/spec_check/dsl/builtin_types.py b/spec_check/dsl/builtin_types.py index 805c908..81b5773 100644 --- a/spec_check/dsl/builtin_types.py +++ b/spec_check/dsl/builtin_types.py @@ -19,6 +19,7 @@ PrinciplesModule, RequirementModule, SpecificationModule, + TechnicalNoteModule, ) # ============================================================================ @@ -32,6 +33,7 @@ "ADR": ArchitectureDecisionModule(), "Specification": SpecificationModule(), "Principles": PrinciplesModule(), + "TechnicalNote": TechnicalNoteModule(), } # Export built-in class types @@ -47,5 +49,6 @@ "ArchitectureDecisionModule", "SpecificationModule", "PrinciplesModule", + "TechnicalNoteModule", "AcceptanceCriterion", ] diff --git a/spec_check/dsl/layers.py b/spec_check/dsl/layers.py index 0acbb20..b6c81e6 100644 --- a/spec_check/dsl/layers.py +++ b/spec_check/dsl/layers.py @@ -586,7 +586,7 @@ class TechnicalNoteModule(SpecModule): are not themselves requirements or implementations. Example filename: TN-001.md - Location: specs/notes/ + Location: context/technical-notes/ Required sections: - Abstract: Summary of the note @@ -602,7 +602,7 @@ class TechnicalNoteModule(SpecModule): description: str = "Technical note or analysis document" file_pattern: str = r"^TN-\d{3}\.md$" - location_pattern: str = r"specs/notes/" + location_pattern: str = r"context/technical-notes/" identifier: IdentifierSpec = IdentifierSpec( pattern=r"TN-\d{3}", diff --git a/specs/future/specification-dsl.md b/specs/future/specification-dsl.md index 9f827d2..703b13c 100644 --- a/specs/future/specification-dsl.md +++ b/specs/future/specification-dsl.md @@ -291,4 +291,4 @@ This feature will be considered successful when: ## Related Documents -See `technical-notes/specification-dsl-design.md` for detailed design discussion and implementation considerations. +See `context/technical-notes/TN-005.md` for detailed design discussion and implementation considerations. diff --git a/tests/test_new_document_types.py b/tests/test_new_document_types.py index efc2bcc..d67eabd 100644 --- a/tests/test_new_document_types.py +++ b/tests/test_new_document_types.py @@ -330,9 +330,9 @@ def test_technical_note_file_pattern(self): def test_technical_note_location_pattern(self): """Test that Technical Note module matches correct location.""" module = TechnicalNoteModule() - assert module.location_pattern == r"specs/notes/" + assert module.location_pattern == r"context/technical-notes/" - test_path = Path("specs/notes/TN-001.md") + test_path = Path("context/technical-notes/TN-001.md") assert module.matches_file(test_path) def test_technical_note_identifier_spec(self): @@ -426,7 +426,7 @@ def test_implementation_design_document_validates(self, tmp_path): def test_technical_note_document_validates(self, tmp_path): """Test that TN-001.md validates against Technical Note schema.""" - tn_001 = Path("specs/notes/TN-001.md") + tn_001 = Path("context/technical-notes/TN-001.md") if not tn_001.exists(): pytest.skip("TN-001.md not found") @@ -450,7 +450,7 @@ def test_technical_note_document_validates(self, tmp_path): "SOL-001", ), ("ImplementationDesign", r"^IMP-\d{3}\.md$", "specs/design/", "IMP-001"), - ("TechnicalNote", r"^TN-\d{3}\.md$", "specs/notes/", "TN-001"), + ("TechnicalNote", r"^TN-\d{3}\.md$", "context/technical-notes/", "TN-001"), ], ) class TestDocumentTypePatterns: