Secret Remediation Tracker

A secure web application for tracking and managing leaked secrets from TruffleHog scans. This application helps security teams manage the lifecycle of leaked secrets by ingesting scan results, deduplicating findings, and tracking remediation status.

---

🚀 Quick Start (2 Minutes)

Prerequisites

• Docker and Docker Compose installed

Get Running

# 1. Start the application
docker compose up

# 2. Open your browser
# http://localhost:5000

# 3. Create your account
# Click "Sign up here" on the login page

# 4. Upload a test scan
# Use a file from the scans/ directory

That's it! Your secret tracker is running. 🎉

---

✨ Features

• 📤 File Upload: Web-based TruffleHog JSONL scan ingestion
• 🔍 Smart Deduplication: Content-based secret detection (SHA-256)
• 📊 Visual Dashboard: Real-time statistics and filtering
• 🔄 Status Tracking: New → In Progress → Resolved → False Positive
• ✨ Auto-Resolution: Automatically detects when secrets are remediated
• 🔐 Authentication: Secure user accounts with session management
• 👥 User Isolation: Each user sees only their own data
• 🏢 Multi-Repository: Track secrets across your entire organization
• 📝 Detailed Views: Complete finding history with file paths and timestamps
• 🎭 Secret Masking: Secrets hidden by default, explicit reveal required

---

📖 Table of Contents

• Architecture
• Usage Guide
• Configuration
• Security
• API Endpoints
• Development
• Troubleshooting
• Production Deployment

---

🏗️ Architecture

Technology Stack

• Backend: Flask 3.0 (Python)
• Database: PostgreSQL 15 (Docker) / SQLite (local development)
• ORM: SQLAlchemy 2.0
• Frontend: HTML/CSS/JavaScript (Vanilla JS)
• Deployment: Docker + Docker Compose
• Security: bcrypt password hashing, secure sessions, OWASP compliant

System Diagram

┌─────────────────┐
│   Web Browser   │
│   (Port 5000)   │
└────────┬────────┘
         │ HTTPS (in production)
         ↓
┌─────────────────┐
│   Flask App     │
│   - Upload      │
│   - Dashboard   │
│   - Auth        │
│   - API         │
└────────┬────────┘
         │ SQLAlchemy ORM
         ↓
┌─────────────────┐
│  PostgreSQL     │
│  - Users        │
│  - Secrets      │
│  - Findings     │
│  - Scans        │
└─────────────────┘

Database Schema

• Users: Authentication and user management
• Secrets: Unique secrets (deduplicated by content hash)
• Findings: Individual occurrences of secrets in scans
• Scans: Upload history and metadata
• RepositoryShare: Future sharing feature (planned)

---

📘 Usage Guide

Creating Your First Account

1. Start the application with docker compose up
2. Navigate to http://localhost:5000
3. Click "Sign up here"
4. Enter username and password (twice for confirmation)
5. Log in with your new credentials

Uploading Scans

1. Click "📤 Upload Scan" on the dashboard
2. Select a TruffleHog JSONL file
3. Click "Upload & Process"
4. The system will:
   • Parse all findings from the file
   • Deduplicate secrets based on content (SHA-256)
   • Update first/last seen timestamps
   • Check for auto-resolved secrets
   • Associate all data with your user account

Managing Secrets

Viewing Secrets

• Dashboard shows all your secrets with filters
• Filter by status (New, In Progress, Resolved, False Positive)
• Filter by repository
• View statistics in real-time

Secret Details

Click "View" on any secret to see:

• Secret value (masked by default, reveal when needed)
• All findings (every location where it was found)
• File paths and line numbers
• Scan timestamps (first seen / last seen)
• Repository information
• Verification status from TruffleHog

Updating Status

1. Click "View" on a secret
2. Select new status from dropdown
3. Changes are logged for audit trail

Status Workflow

New → In Progress → Resolved
  ↓
False Positive

• New: Secret just discovered
• In Progress: Remediation work underway
• Resolved: Secret has been remediated (manually or auto-resolved)
• False Positive: Not a real secret

Auto-Resolution

The system automatically marks secrets as "Resolved" when:

1. A new scan is uploaded for a repository
2. A secret from previous scans no longer appears
3. The secret's status is still "New" or "In Progress"

Example Workflow:

# Upload initial scan
# - Upload scans/router/trufflehog_scan_2024-06-23.jsonl
# - 10 secrets found, all marked "New"

# Work on remediation
# - Mark some as "In Progress"
# - Remove secrets from code

# Upload new scan
# - Upload scans/router/trufflehog_scan_2024-09-09.jsonl
# - Remediated secrets automatically marked "Resolved"
# - Remaining secrets stay "In Progress"

Testing with Sample Data

Sample scan files are included in the scans/ directory:

1. Initial Test: scans/client/trufflehog_scan_2024-11-27.jsonl

   • 13 unique secrets
   • Various types (AWS, GitHub, Stripe, etc.)

2. Auto-Resolution Test:

   • Upload scans/router/trufflehog_scan_2024-06-23.jsonl
   • Upload scans/router/trufflehog_scan_2024-09-09.jsonl
   • Watch auto-resolution in action!

3. Multi-Repository Test:

   • Upload scans from different subdirectories
   • Use repository filter to view each separately

---

⚙️ Configuration

Environment Variables

Configure in docker-compose.yml or create a .env file:

# Database
DATABASE_URL=postgresql://user:pass@db:5432/secrets_db

# Security (REQUIRED for production)
SECRET_KEY=your-cryptographically-secure-secret-key-here

# Flask Environment
FLASK_ENV=production  # Enables secure cookies, disables debug

# Session Configuration
SESSION_TIMEOUT=3600  # Session timeout in seconds (default: 1 hour)

Generating Secure SECRET_KEY

python -c "import secrets; print(secrets.token_hex(32))"

⚠️ IMPORTANT: The application will warn if you're using the default SECRET_KEY!

Port Configuration

Default: http://localhost:5000

To change the port, edit docker-compose.yml:

ports:
  - "8080:5000"  # Change 5000 to your preferred port

Database Configuration

Docker (Production)

• PostgreSQL 15 with persistent volumes
• Data survives container restarts
• Located in postgres_data volume

Local Development

• SQLite database (secrets.db)
• No Docker required
• Use python app.py to run locally

---

🔒 Security

Implemented Security Measures

1. Authentication & Authorization

   • Session-based authentication
   • Secure cookies (HttpOnly, SameSite=Lax)
   • HTTPS-only cookies in production
   • Configurable session timeouts (default: 1 hour)
   • Password hashing with bcrypt
   • No default credentials

2. Data Protection

   • User data isolation (per-user access control)
   • Secrets masked by default
   • Explicit reveal action required
   • SQL injection prevention (SQLAlchemy ORM)
   • XSS prevention (Jinja2 auto-escaping + CSP headers)
   • Input sanitization (username, control characters)
   • JSONL structure validation before processing

3. Container Security

   • Non-root user (appuser, UID 1000)
   • Minimal base image (python:3.11-slim)
   • Health checks configured
   • No SSH access
   • CIS Docker Benchmark compliant

4. Application Security

   • Rate limiting (Flask-Limiter)
     • Login: 10/minute
     • Signup: 10/hour
     • Uploads: 100/hour
   • Security headers (CSP, HSTS, X-Frame-Options, etc.)
   • File type validation (.jsonl only)
   • File size limits (16MB maximum)
   • Secure filename handling
   • Comprehensive audit logging
   • Environment-based configuration
   • Debug mode disabled in production

Security Scan Results

✅ Dependency Vulnerabilities: None (Snyk scan)
✅ Container Security: 77/77 checks passed (Checkov)
✅ OWASP Top 10: Compliant

See SECURITY.md for comprehensive security documentation and scan results.

Production Security Requirements

Before deploying to production:

1. Generate Secure Credentials (CRITICAL)

   python -c "import secrets; print(secrets.token_hex(32))"

2. Enable HTTPS (CRITICAL)

   • Deploy behind reverse proxy (Nginx, Traefik)
   • Obtain SSL/TLS certificates
   • Set FLASK_ENV=production

3. Encrypt Secrets at Rest (Recommended)

   • Application-level encryption, or
   • Database-level encryption, or
   • Use key management service (AWS KMS, Azure Key Vault)

4. Additional Security (Recommended)

   • Implement 2FA
   • Add rate limiting
   • Set up monitoring and alerting
   • Configure automated backups
   • Regular security updates

See SECURITY.md for comprehensive production deployment requirements and security checklist.

---

🔌 API Endpoints

All endpoints require authentication (except /login and /signup).

Authentication

• POST /login - User login
• POST /signup - Create new account
• GET /logout - User logout

Secrets Management

• GET /api/secrets - List all secrets (with optional filters)
  • Query params: status, repository
• GET /api/secrets/<id> - Get secret details with all findings
• PUT /api/secrets/<id>/status - Update secret status
  • Body: {"status": "In Progress"}

Scan Management

• POST /api/upload - Upload and process scan file
  • Form data: file (JSONL file)
• GET /api/scans - Get scan history
• GET /api/stats - Get dashboard statistics

Response Format

{
  "success": true,
  "data": { ... },
  "message": "Operation completed"
}

Error responses:

{
  "success": false,
  "error": "Error message"
}

---

💻 Development

Local Development Setup

# Install dependencies
pip install -r requirements.txt

# Create database
python recreate_db.py

# Run application
python app.py

# Access at http://localhost:5000

Project Structure

secret-remediation-tracker/
├── app.py                      # Main Flask application
├── init_db.py                  # Database initialization (Docker)
├── recreate_db.py              # Database reset (local dev)
├── requirements.txt            # Python dependencies
│
├── templates/                  # HTML templates
│   ├── base.html              # Base layout with navigation
│   ├── login.html             # Login page
│   ├── signup.html            # Signup page
│   └── dashboard.html         # Main dashboard
│
├── static/                     # Static assets
│   ├── css/
│   │   └── style.css          # Application styles
│   └── js/
│       ├── main.js            # Common utilities
│       └── dashboard.js       # Dashboard functionality
│
├── scans/                      # Sample TruffleHog scan files
│   ├── client/
│   ├── router/
│   └── server/
│
├── tests/                      # Test suite
│   ├── conftest.py            # Test fixtures
│   ├── test_auth.py           # Authentication tests
│   ├── test_api.py            # API endpoint tests
│   ├── test_models.py         # Database model tests
│   └── test_security.py       # Security feature tests
│
├── Dockerfile                  # Container image definition
├── docker-compose.yml          # Multi-service orchestration
├── env.example                 # Environment variable template
├── .checkov.yaml              # Security scan configuration
├── run_tests.sh                # Test runner (Linux/Mac)
├── run_tests.ps1               # Test runner (Windows)
│
├── README.md                   # This file
├── SECURITY.md                 # Security documentation & scan results
└── PROJECT_SUMMARY.md          # Technical overview

Running Tests

The project includes a comprehensive test suite to validate core functionality.

Quick Start:

# Linux/Mac
chmod +x run_tests.sh
./run_tests.sh

# Windows
.\run_tests.ps1

Manual Testing:

# Install test dependencies
pip install -r requirements.txt

# Run all tests
pytest -c tests/pytest.ini tests/

# Run with coverage report
pytest -c tests/pytest.ini tests/
# Then open: tests/htmlcov/index.html

# Run specific test file
pytest -c tests/pytest.ini tests/test_auth.py

# Run tests matching pattern
pytest -c tests/pytest.ini tests/ -k "test_login"

Test Coverage:

• ✅ Authentication (login, signup, logout)
• ✅ API endpoints (secrets, upload, stats)
• ✅ Database models and relationships
• ✅ Security features (sanitization, headers, isolation)
• ✅ Input validation

See tests/README.md for detailed testing documentation.

Security Scans

# Dependency vulnerabilities
snyk test --file=requirements.txt

# Infrastructure security
checkov -d . --compact

# Full security scan
snyk test && checkov -d .

Debugging

# Check application logs
docker compose logs -f web

# Check database
docker compose exec db psql -U secretuser -d secrets_db

# Access database shell (local)
sqlite3 secrets.db

Database Operations

# Reset local database
python recreate_db.py

# Backup database (Docker)
docker compose exec db pg_dump -U secretuser secrets_db > backup.sql

# Restore database (Docker)
docker compose exec -T db psql -U secretuser secrets_db < backup.sql

---

🔧 Troubleshooting

Application Won't Start

Port already in use:

# Windows
netstat -ano | findstr :5000

# Linux/Mac
lsof -i :5000

# Change port in docker-compose.yml
ports:
  - "8080:5000"

Database connection errors:

# Wait longer - PostgreSQL takes ~10 seconds to initialize
# Check database logs
docker compose logs db

# Verify database is ready
docker compose exec db pg_isready -U secretuser

Upload Fails

• Ensure file ends with .jsonl
• Check file size (maximum 16MB)
• Verify JSONL format (one valid JSON object per line)
• Check application logs: docker compose logs web

Can't Login

• Verify username and password
• Create new account at /signup if needed
• Check logs for authentication errors: docker compose logs web

Database Schema Errors

If you see table has no column named user_id:

# For Docker
docker compose down -v
docker compose up

# For local development
python recreate_db.py

Container Health Issues

# Check container status
docker compose ps

# View detailed logs
docker compose logs -f

# Restart services
docker compose restart

# Full reset
docker compose down -v
docker compose up

---

🚢 Production Deployment

Pre-Deployment Checklist

• Generate secure SECRET_KEY
• Change database password
• Set FLASK_ENV=production
• Configure HTTPS reverse proxy
• Set up database backups
• Configure monitoring and alerting
• Review SECURITY.md for hardening
• Test with production-like data
• Scan for vulnerabilities (Snyk, Checkov)

Deployment Steps

# 1. Update configuration
# Edit docker-compose.yml with production values

# 2. Start services
docker compose up -d

# 3. Verify services
docker compose ps
docker compose logs

# 4. Create first user account
# Navigate to /signup

# 5. Upload test scan
# Verify functionality

# 6. Set up backups
# Configure automated database backups

Monitoring

Monitor these metrics:

• Application health (Docker health checks)
• Failed login attempts
• Upload failures
• Database performance
• CPU and memory usage

Backup Strategy

# Daily automated backup
docker compose exec db pg_dump -U secretuser secrets_db > backup_$(date +%Y%m%d).sql

# Verify backup
pg_restore --list backup_$(date +%Y%m%d).sql

# Store backups securely off-site

See SECURITY.md for production deployment requirements and operational procedures.

---

📚 Additional Documentation

• SECURITY.md - Comprehensive security documentation, scan results, and production deployment guide
• PROJECT_SUMMARY.md - Technical overview and architecture
• env.example - Environment variable template

---

🤝 Contributing

This is an MVP implementation. Contributions welcome!

Future Enhancements

• Multi-factor authentication (2FA)
• Role-based access control (RBAC)
• Repository sharing between users
• Email notifications for new secrets
• Slack/Teams integrations
• API authentication tokens
• Bulk status updates
• Advanced search and filtering
• Secret rotation tracking
• Compliance reporting
• Integration with ticketing systems

---

📄 License

This is an MVP implementation for evaluation purposes.

---

🆘 Support

For issues or questions:

1. Check this README
2. Review SECURITY.md for security questions
3. Check logs: docker compose logs -f
4. Run verification script: verify_setup.ps1 (Windows) or verify_setup.sh (Linux/Mac)
5. See SECURITY.md for production deployment troubleshooting

---

Built with security, quality, and user experience in mind. 🔒

Ready to deploy and use immediately ✅