feat: Set up comprehensive Python testing infrastructure

[llbbl]

🧪 Set up comprehensive Python testing infrastructure

Summary

This PR establishes a complete testing infrastructure for the ConCode mapping project, providing developers with a ready-to-use testing environment. The setup includes modern Python package management, testing framework configuration, and validation tests.

Changes Made

Package Management

• Configured Poetry as the package manager with pyproject.toml
• Migrated dependencies from README instructions to Poetry configuration
• Added testing dependencies: pytest, pytest-cov, pytest-mock
• Updated dependency versions for compatibility (torch >=1.0.0, numpy ^1.21.0)

Testing Configuration

• Configured pytest in pyproject.toml with:
  • Custom markers: unit, integration, slow
  • Test discovery patterns and strict options
  • Filter warnings configuration
• Set up coverage reporting with:
  • 80% coverage threshold
  • HTML and XML report generation
  • Source inclusion/exclusion rules

Testing Directory Structure

tests/
├── __init__.py
├── conftest.py              # Shared fixtures and test configuration
├── unit/
│   └── __init__.py
├── integration/
│   └── __init__.py
└── test_infrastructure_validation.py  # Validation tests

Shared Fixtures (conftest.py)

• File system fixtures: temp_dir, temp_file, project_root
• Mock objects: mock_model, mock_encoder, mock_decoder, mock_statistics
• Sample data: sample_json_data, sample_dataset, sample_vocabulary
• Code samples: java_code_sample, python_code_sample
• Testing utilities: capture_output, reset_random_seeds
• Configuration: sample_config, mock_training_data, sample_beam_data

Additional Setup

• Updated .gitignore with testing artifacts, build files, and Claude settings
• Created validation tests to verify infrastructure works correctly
• Configured coverage reports with proper exclusions

How to Run Tests

Basic Commands

# Run all tests
poetry run pytest

# Run with verbose output
poetry run pytest -v

# Run specific test types
poetry run pytest -m unit          # Unit tests only
poetry run pytest -m integration   # Integration tests only
poetry run pytest -m "not slow"    # Exclude slow tests

Coverage Reports

# Generate HTML coverage report
poetry run pytest --cov=. --cov-report=html

# Generate XML coverage report
poetry run pytest --cov=. --cov-report=xml

# View coverage in terminal
poetry run pytest --cov=. --cov-report=term

Test Discovery

• Tests are discovered in tests/ directory
• Test files: test_*.py or *_test.py
• Test classes: Test*
• Test functions: test_*

Validation

All infrastructure components have been validated:

• ✅ 19 validation tests pass
• ✅ Custom pytest markers work correctly
• ✅ Coverage reporting generates HTML and XML outputs
• ✅ Shared fixtures are available and functional
• ✅ Dependencies install successfully
• ✅ Configuration files are properly formatted

Next Steps

Developers can now:

1. Write unit tests in tests/unit/
2. Write integration tests in tests/integration/
3. Use shared fixtures from conftest.py
4. Run tests with coverage reporting
5. Use custom markers to categorize and filter tests

Notes

• Poetry lock file is included and should not be added to .gitignore
• Coverage threshold is set to 80% - adjust in pyproject.toml if needed
• PyTorch version updated to >=1.0.0 for compatibility (original project used 0.3.1)
• AllenlLP dependency removed due to compatibility issues - add back if needed for specific tests

🚀 Ready to test! The infrastructure is fully set up and validated for immediate use.