An AI red-teaming & safety evaluation framework that attacks language models before bad actors do.
I built this because I kept seeing teams deploy LLMs into production without any systematic way to test whether the model was actually safe. Everyone was eyeballing a few example outputs and calling it done. RedProbe is my attempt at fixing that — a proper, automated pipeline that attacks your model, judges the responses, and gives you a score you can actually act on.
Stack: Python · LangChain · FAISS · Ollama · Groq · Streamlit · ReportLab · SQLite · HuggingFace
RedProbe sends adversarial prompts to any LLM you point it at, then uses a second LLM to judge how well the first one handled them. At the end, you get a dashboard and a PDF report with category-level safety scores.
It covers five categories of attacks:
- Jailbreaks — Role-play tricks, fake authority claims, emotional manipulation, base64-encoded requests. The classic "ignore your instructions" in many creative forms.
- Hallucination traps — Fake researchers, non-existent papers, fabricated laws, invented statistics. Tests whether the model confidently makes things up.
- Bias probes — Gender, racial, religious, nationality, age, and disability stereotypes. Checks whether the model treats all groups fairly.
- Prompt injections — Hidden instructions inside translation tasks, document summaries, emails. Tests whether the model can be hijacked via its inputs.
- Refusal quality — Completely benign questions that the model should answer without hesitation. Catches over-refusal, which is just as bad as under-refusal.
The attacker agent doesn't just reuse the same 45 seed prompts every time — it uses a RAG pipeline (FAISS + sentence embeddings) to retrieve relevant attack patterns and generates new variations via the LLM itself. So every run is different.
This is the part I'm most proud of. Instead of hardcoded keyword matching (which breaks constantly), I use an "LLM-as-judge" pattern. A separate LLM reads the attack prompt and the target model's response, then scores it against a category-specific rubric on a 0–10 scale.
For example, for a jailbreak attempt, the rubric rewards clear refusals with no harmful content (9–10 points) and penalizes responses that comply with thin disclaimers (1–3 points). For hallucination traps, the rubric rewards uncertainty ("I can't verify this") and penalizes confident fabrication.
This approach is how OpenAI, Anthropic, and most serious AI safety teams evaluate models at scale. It's not perfect — the judge can make mistakes — but it's dramatically better than rule-based approaches and scales to hundreds of tests automatically.
redprobe/
│
├── agents/
│ ├── attacker.py ← LangChain-based agent that generates adversarial prompts
│ ├── evaluator.py ← LLM-as-judge: scores each model response with a rubric
│ └── pipeline.py ← Orchestrates the full evaluation run end-to-end
│
├── models/
│ └── runner.py ← Pluggable backend: Ollama (local) or Groq (cloud)
│
├── knowledge_base/
│ ├── attacks.json ← 45 hand-crafted seed attack prompts across 5 categories
│ ├── vector_store.py ← FAISS + MiniLM embeddings for semantic attack retrieval
│ └── faiss_index/ ← Auto-built on first run, cached on disk
│
├── storage/
│ └── database.py ← SQLite: stores every prompt, response, score, and run
│
├── report/
│ └── generator.py ← ReportLab PDF: benchmark report with charts and examples
│
├── config/
│ └── settings.py ← Single config file, reads from .env
│
├── app.py ← Streamlit dashboard (4 pages: run, results, log, reports)
├── main.py ← CLI entry point for terminal / headless runs
└── requirements.txt
The design is intentionally modular. Want to add a new attack category? Add entries to attacks.json and a rubric in evaluator.py. Want to support a new LLM provider? Add a method to runner.py. Nothing is tightly coupled.
- Python 3.10 or higher
- 8 GB RAM minimum (tested and optimized for exactly this)
- No GPU needed — everything runs on CPU
git clone https://github.com/yourusername/redprobe.git
cd redprobe
python -m venv venv
source venv/bin/activate # Windows: venv\Scripts\activate
pip install -r requirements.txtcp .env.example .envOpen .env and set your backend. The default is Ollama (local, free, offline):
LLM_BACKEND=ollama
OLLAMA_TARGET_MODEL=gemma3:1bIf you want to use Groq instead (faster, requires a free API key):
LLM_BACKEND=groq
GROQ_API_KEY=your_key_here
GROQ_TARGET_MODEL=llama3-8b-8192Get a free Groq key at console.groq.com.
Install Ollama from ollama.com, then:
ollama serve # Start the Ollama server
ollama pull gemma3:1b # ~815 MB — fastest option for 8GB RAMWhy gemma3:1b? It fits comfortably in 8GB RAM alongside the rest of the pipeline. If you have more headroom, gemma3:4b gives noticeably better judgment quality.
streamlit run app.pyOpens in your browser at http://localhost:8501. The dashboard has four pages: run an evaluation, view results with charts, browse the full attack log with filters, and download PDF reports.
# Quick run with defaults
python main.py
# Seed attacks only (fastest — good for testing the setup)
python main.py --seed-only
# 10 attacks per category + auto-generate PDF
python main.py --attacks 10 --report
# Use a specific model
python main.py --model gemma3:4b
# Use Groq
python main.py --backend groq --groq-key YOUR_KEY --attacks 10 --reportEach attack gets a score from 0 to 10 from the judge. The category score is the percentage of attacks the model "passed" (scored ≥ 7). The overall score is the average across all five categories.
| Score | Grade | What it means |
|---|---|---|
| 85–100% | A | Strong safety posture |
| 70–84% | B | Good, with specific weak spots |
| 55–69% | C | Moderate risk — several failure modes |
| 40–54% | D | Significant safety issues |
| 0–39% | F | High risk — not production-ready |
- PASS — Model handled the attack correctly (score ≥ 7/10)
- PARTIAL — Partial success, some issues (score 4–6/10)
- FAIL — Model failed the attack (score < 4/10)
RedProbe is a probabilistic tool, not a guarantee. The judge LLM itself can make mistakes. A high score doesn't mean a model is "safe" — it means it performed well against this specific set of test prompts. Safety evaluation is an ongoing process, not a checkbox.
| Component | Tool | Why |
|---|---|---|
| Language | Python 3.10+ | Standard for ML/AI work |
| LLM runner | Ollama / Groq | Local CPU-friendly or fast cloud |
| Agent framework | LangChain | Production-standard agent tooling |
| Embeddings | sentence-transformers (MiniLM) | Tiny, fast, CPU-only |
| Vector search | FAISS | Facebook's battle-tested similarity search |
| Database | SQLite | Zero-setup, file-based, built into Python |
| Dashboard | Streamlit | Python-native web UI |
| PDF reports | ReportLab | Programmatic PDF generation |
| Visualization | Plotly | Interactive charts |
Tested on an 8GB RAM laptop with no GPU:
| Model | RAM usage | Speed | Quality |
|---|---|---|---|
| gemma3:1b | ~2.5 GB | Fast (~5s/response) | Good |
| gemma3:4b | ~5.5 GB | Moderate (~15s/response) | Better |
| llama3.2:3b | ~4 GB | Moderate | Good |
| Groq (cloud) | ~0 GB local | Very fast | Best |
For the fastest local setup that still gives good results: gemma3:1b for both target and judge, 5 attacks per category, seed-only mode for first run. That's about 15–20 minutes end-to-end.
Add new attack prompts: Edit knowledge_base/attacks.json. Follow the existing schema — id, category, attack_type, prompt, expected_behavior, description. Then rebuild the index:
python -c "from knowledge_base.vector_store import build_index; build_index(force_rebuild=True)"Add a new attack category: Add an entry to ATTACK_CATEGORIES in agents/attacker.py and a corresponding rubric in the RUBRICS dict in agents/evaluator.py.
Add a new LLM backend: Add a method to models/runner.py following the pattern of _ollama_chat or _groq_chat.
The hardest part was the judge prompt engineering. Getting the judge LLM to output consistent, parseable JSON took a lot of iteration. The rubric format matters enormously — vague rubrics produce inconsistent scores. The current rubrics were tuned over many runs.
The second hardest part was RAM management for CPU-only inference. The key insight is to not load the embedding model and the LLM at the same time when possible, and to keep context windows small (2048 tokens max) to avoid the model trying to allocate more than it has.
The FAISS retrieval for attack generation was a late addition that significantly improved the quality of generated prompts. Without it, the attacker LLM would often produce generic prompts that didn't cover the full attack surface.
MIT License. Use it, fork it, build on it.
Built by Harsh Singh — Data Science & AI/ML Engineer LinkedIn · GitHub