Skip to content

Implement complete log streaming and metrics persistence system#14

Merged
elchinoo merged 1 commit into
mainfrom
v2-redesign-core
Aug 6, 2025
Merged

Implement complete log streaming and metrics persistence system#14
elchinoo merged 1 commit into
mainfrom
v2-redesign-core

Conversation

@elchinoo
Copy link
Copy Markdown
Owner

@elchinoo elchinoo commented Aug 6, 2025

Overview

This PR implements a comprehensive log streaming and metrics persistence system for StormDB v2, including a React frontend and complete backend infrastructure for real-time monitoring and data persistence.

Frontend Features

React Web Client

  • Dual-panel interface with test submission and history
  • Real-time status polling for running tests
  • Log streaming from active test runs
  • Historical test runs with detailed status tracking
  • Clean, responsive UI with proper styling

Technical Implementation

  • React 19.1.0 + TypeScript + Vite 7.0.4
  • Proxy configuration for seamless API integration
  • Real-time polling with proper error handling
  • Component-based architecture

Backend Improvements

Logging System

  • Configurable logging (file/database/log system)
  • Database log persistence with test run association
  • Structured logging with JSON and text formats
  • Test run context propagation through all components

Database Schema

  • New logs table for persistent log storage
  • Enhanced metrics storage with proper foreign keys
  • Migration system for schema updates
  • Complete result persistence for all plugin metrics

API Enhancements

  • Log retrieval endpoint with pagination support
  • Real-time test monitoring capabilities
  • Comprehensive error handling and validation
  • Test run management with proper state tracking

Plugin Improvements

  • Context-aware logging with test run ID propagation
  • Database result storage for all plugin metrics
  • Proper error handling and recovery
  • Enhanced test coverage (30%+ for both plugins)

System Reliability

Startup Recovery

  • Stuck test detection and automatic recovery
  • Database state consistency validation
  • Graceful error handling for interrupted tests

Test Infrastructure

  • Updated mock interfaces for new logger methods
  • Comprehensive validation for plugin configurations
  • Improved test coverage with realistic scenarios
  • Clean build system with proper dependency management

What's Fixed

  1. Empty database issue - Plugins now properly persist metrics to database
  2. Log streaming - Real-time logs available through API and UI
  3. Stuck test states - Automatic recovery on server restart
  4. Context propagation - Test run IDs properly passed through execution chain
  5. Configuration validation - Robust validation for all plugin configs
  6. Build system - Clean compilation for all components

Testing

  • ✅ All Go tests passing
  • ✅ 32.3% coverage for bulk-load plugin
  • ✅ 29.9% coverage for tpcc-scalability plugin
  • ✅ Clean build for all components
  • ✅ Plugin loading and execution verified
  • ✅ Database persistence confirmed
  • ✅ Frontend-backend integration tested

Migration Notes

  1. Run database migrations to add the logs table
  2. Update configuration for logging preferences
  3. Rebuild plugins with new interfaces
  4. No breaking changes to existing API endpoints

This PR transforms StormDB from a basic testing tool into a comprehensive performance monitoring platform with real-time visibility and persistent data storage.

- Add React frontend with dual-panel UI for test submission and history
- Implement configurable logging with database persistence
- Add test run logs table and API endpoint for log retrieval
- Fix plugin metrics storage to persist results to database
- Add test run ID context passing through scheduler
- Implement stuck test recovery on server startup
- Fix both bulk-load and tpcc-scalability plugins for proper data persistence
- Update plugin tests to support new logger interface
- Add comprehensive result storage for all plugin metrics
- Complete logs API endpoint with proper error handling

Frontend features:
- Real-time test status polling
- Historical test runs with detailed status
- Log streaming from running tests
- Clean dual-panel interface with proper styling

Backend improvements:
- Configurable logging (file/database/log system)
- Database metrics and logs persistence
- Test run context propagation
- Startup stuck test recovery
- Complete API for logs retrieval

All tests passing with 30%+ coverage for plugins.
@elchinoo elchinoo merged commit c0a3fa8 into main Aug 6, 2025
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant