Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 14 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ uv run pytest

The threat filter combines two analysis layers:

- **Tech layer** — NLP-based detection of vagueness patterns, authority claims, urgency/fear triggers, emotional manipulation, and logical contradictions commonly found in spiritual disinformation
- **Tech layer** — NLP-based detection across seven dimensions: vagueness patterns, authority claims, urgency/fear triggers, emotional manipulation, logical contradictions, source attribution analysis, and commitment escalation. Markers span six tradition-specific categories (generic New Age, prosperity gospel, conspirituality, commercial exploitation, cult rhetoric, fraternal/secret society).
- **Heuristic layer** — probabilistic dissonance scanner (placeholder for future biofeedback integration)

Output is a 0–100 threat score with a breakdown of what triggered it.
Expand All @@ -35,10 +35,23 @@ Output is a 0–100 threat score with a breakdown of what triggered it.
src/si_protocols/
threat_filter.py # Hybrid NLP + heuristic threat scorer
markers.py # Disinformation marker definitions
output.py # Rich and JSON output formatting
app/
main.py # FastAPI REST API (POST /analyse, GET /health)
schemas.py # Pydantic request/response models
site/ # Astro documentation site
tests/ # pytest suite
examples/ # Synthetic sample texts (never real material)
```

## REST API

```bash
uvicorn app.main:app --host 127.0.0.1 --port 8000
```

Local-only FastAPI server with `POST /analyse` and `GET /health`. Interactive docs at `/docs`.

## Dev

```bash
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -108,9 +108,9 @@ They all work on **structural properties of text** — patterns that can be dete

## What's Next

We've now implemented both **logical contradiction detection** and **source attribution analysis** as scoring dimensions in the threat filter. Each follows our existing pattern: markers defined in `markers.py`, scored in `threat_filter.py`, tested with synthetic examples. Transparent, local-only, and open-source.
We've now implemented **logical contradiction detection**, **source attribution analysis**, and **commitment escalation** as scoring dimensions in the threat filter. Each follows our existing pattern: markers defined in `markers.py`, scored in `threat_filter.py`, tested with synthetic examples. Transparent, local-only, and open-source. The threat filter now scores text across seven dimensions with tradition-specific markers spanning six categories.

The remaining approaches — influence technique fingerprinting, commitment escalation, cognitive load manipulation, stylometric entropy, and hedging-to-assertion ratio — are candidates for future dimensions.
The remaining approaches — influence technique fingerprinting, cognitive load manipulation, stylometric entropy, and hedging-to-assertion ratio — are candidates for future dimensions.

If you work in cult recovery, undue influence research, or digital literacy education, we'd love to hear which of these signals match what you see in practice. Open an issue on [GitHub](https://github.com/lemur47/si-protocols) or reach out directly.

Expand Down
137 changes: 137 additions & 0 deletions site/src/content/blog/hands-on-threat-analysis.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
---
title: "Hands-On Threat Analysis: A Step-by-Step Tutorial"
description: Walk through analysing spiritual content with si-protocols — from CLI to API to Python library.
date: 2026-02-15
tags: [tutorial, getting-started]
---

## What We're Analysing

Let's walk through a complete threat analysis using a synthetic example. The file `examples/synthetic_suspicious.txt` ships with the repository — it was written specifically for testing and contains a concentration of common disinformation markers:

> The ancient and hidden cosmic truth has been veiled from humanity for millennia. The ascended masters say that only those who awaken to the divine frequency will transcend the matrix of illusion. The galactic federation confirms that a great shift is upon us. You must act now — the window is closing, and only the chosen will ascend to the fifth dimension.

This passage alone packs vague adjectives (*ancient*, *hidden*, *cosmic*, *divine*), authority claims (*the ascended masters say*, *the galactic federation confirms*), and urgency patterns (*you must act now*, *the window is closing*). The full file also includes emotional manipulation, logical contradictions, unfalsifiable source claims, and unnamed authority phrases.

All examples in this tutorial are synthetic. We never include real channelled or spiritual material in the repository.

## Method 1 — The CLI

The fastest way to analyse a file is the command-line interface.

**Setup** (if you haven't already):

```bash
git clone https://github.com/lemur47/si-protocols.git
cd si-protocols
uv sync --all-extras
uv pip install en_core_web_sm@https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl
```

**Run the analysis:**

```bash
uv run si-threat-filter examples/synthetic_suspicious.txt
```

The Rich output shows a colour-coded threat score (green for low, yellow for medium, red for high) with a breakdown of the tech and heuristic contributions. Below the score, a table lists every detected marker by category — authority claims, urgency patterns, emotion triggers, contradictions, source attribution, and escalation hits.

For machine-readable output, use `--format json`:

```bash
uv run si-threat-filter examples/synthetic_suspicious.txt --format json
```

This emits the full `ThreatResult` as indented JSON — useful for piping into other tools.

## Method 2 — The REST API

If you want to integrate threat analysis into a web application or call it from another language, the FastAPI-based API is the way to go.

**Start the server:**

```bash
uvicorn app.main:app --host 127.0.0.1 --port 8000
```

**Send a request:**

```bash
curl -X POST http://127.0.0.1:8000/analyse \
-H "Content-Type: application/json" \
-d '{"text": "The ascended masters say you must act now. The galactic federation confirms the window is closing.", "seed": 42}'
```

The response mirrors the `ThreatResult` dataclass — every hit category is listed so you can see exactly which markers fired. The `seed` parameter makes the heuristic layer reproducible, which is handy for testing.

You can also explore the API interactively at [http://127.0.0.1:8000/docs](http://127.0.0.1:8000/docs), where Swagger UI lets you paste text and execute requests from the browser.

See the [REST API Reference](/docs/api/) for full endpoint documentation.

## Method 3 — Python Library

For the most flexibility, import si-protocols directly into your Python code.

**Basic analysis:**

```python
from si_protocols.threat_filter import hybrid_score

text = open("examples/synthetic_suspicious.txt").read()
result = hybrid_score(text, seed=42)

print(f"Overall threat score: {result.overall_threat_score}/100")
print(f"Tech layer: {result.tech_contribution}")
print(f"Heuristic layer: {result.intuition_contribution}")

if result.authority_hits:
print(f"Authority claims: {', '.join(result.authority_hits)}")
if result.contradiction_hits:
print(f"Contradictions: {', '.join(result.contradiction_hits)}")
if result.escalation_hits:
print(f"Escalation: {', '.join(result.escalation_hits)}")
```

**Analysing multiple files:**

```python
from pathlib import Path
from si_protocols.threat_filter import hybrid_score

for path in Path("examples").glob("*.txt"):
result = hybrid_score(path.read_text(), seed=42)
print(f"{path.name}: {result.overall_threat_score}/100")
```

**Tech layer only** (deterministic, no heuristic):

```python
from si_protocols.threat_filter import tech_analysis

score, entities, auth, urgency, emotion, contra, source, escalation = tech_analysis(text)
print(f"Deterministic tech score: {score}/100")
```

See the [Python Library Reference](/docs/library/) for full API documentation.

## Understanding the Scores

The tech layer scores text across seven dimensions, each weighted independently:

| Dimension | Weight | What it measures |
|-----------|--------|------------------|
| Vagueness | 17% | Density of vague spiritual adjectives |
| Authority claims | 17% | Phrases that bypass critical thinking |
| Urgency/fear | 13% | Manufactured time pressure |
| Emotional manipulation | 13% | Fear and euphoria words, with a contrast bonus when both appear |
| Logical contradictions | 13% | Opposing claims that create dependency |
| Source attribution | 13% | Unfalsifiable and unnamed authority sources |
| Commitment escalation | 14% | Foot-in-the-door progression from mild to coercive |

A high score means the text contains many markers of potential disinformation — but **a high score is not a verdict**. Genuine spiritual content can trigger some of these patterns. The tool surfaces structure so you can examine it yourself; it does not tell you what to believe or disbelieve. For more on this philosophy, see [A Tool for Thinking, Not a Truth Oracle](/blog/beyond-nlp-detecting-deception-without-llms/#a-tool-for-thinking-not-a-truth-oracle) in our "Beyond NLP" post.

## What's Next

- **[Python Library Reference](/docs/library/)** — full documentation of `hybrid_score()`, `tech_analysis()`, and `ThreatResult`
- **[REST API Reference](/docs/api/)** — endpoint schemas, examples, and error handling
- **[GitHub](https://github.com/lemur47/si-protocols)** — file issues, contribute marker sets, or suggest new scoring dimensions
11 changes: 8 additions & 3 deletions site/src/content/blog/hello-world.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ The spiritual and metaphysical space has a disinformation problem. Vague authori

The threat filter analyses text across two layers:

1. **Tech layer** — a spaCy NLP pipeline that detects vagueness patterns (adjective density), authority claims (phrase matching), urgency/fear triggers, emotional manipulation (lemma-based fear/euphoria detection with a contrast bonus when both polarities appear), logical contradictions (detecting when both poles of common contradiction archetypes appear in the same text), and source attribution analysis (detecting unfalsifiable sources and unnamed authorities, offset by verifiable citations). Each dimension is scored independently, then combined with configurable weights (20/20/15/15/15/15).
1. **Tech layer** — a spaCy NLP pipeline that detects vagueness patterns (adjective density), authority claims (phrase matching), urgency/fear triggers, emotional manipulation (lemma-based fear/euphoria detection with a contrast bonus when both polarities appear), logical contradictions (detecting when both poles of common contradiction archetypes appear in the same text), source attribution analysis (detecting unfalsifiable sources and unnamed authorities, offset by verifiable citations), and commitment escalation (foot-in-the-door progression detection that splits text into thirds and measures whether commitment intensity increases from mild to coercive). Each dimension is scored independently, then combined with weights (17/17/13/13/13/13/14). Markers span six tradition-specific categories: generic New Age, prosperity gospel, conspirituality, New Age commercial exploitation, high-demand group rhetoric, and fraternal/secret society traditions.

2. **Heuristic layer** — a probabilistic dissonance scanner. Currently a randomised placeholder, this layer is designed to eventually integrate biofeedback signals for a more holistic analysis.

Expand All @@ -40,9 +40,14 @@ See the [quickstart guide](/docs/quickstart/) for full setup instructions.

## What's Next

This is v0.1 — the foundation. We're exploring:
Since launch we've shipped several major additions:

- **Commitment escalation detection** — a 7th scoring dimension that detects foot-in-the-door progression
- **Tradition-specific markers** — expanded dictionaries covering prosperity gospel, conspirituality, cult rhetoric, and more
- **REST API** — a FastAPI-based `POST /analyse` endpoint for HTTP integration ([API Reference](/docs/api/))

Still exploring:

- Expanded marker dictionaries for different spiritual traditions
- Biofeedback integration for the heuristic layer
- Browser extension for real-time analysis
- Community-contributed marker sets
Expand Down
122 changes: 122 additions & 0 deletions site/src/content/docs/api.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,122 @@
---
title: REST API Reference
description: Analyse text over HTTP with the FastAPI-based si-protocols API.
order: 3
---

## Running the Server

```bash
# Install dependencies (if not done already)
uv sync --all-extras
uv pip install en_core_web_sm@https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl

# Start the API server
uvicorn app.main:app --host 127.0.0.1 --port 8000
```

The server binds to `127.0.0.1` only — it is designed for local use. Never expose it to the public internet or use it to analyse third-party content.

## Interactive Docs

Once the server is running, open [http://127.0.0.1:8000/docs](http://127.0.0.1:8000/docs) for the Swagger UI. You can paste text directly into the request body and execute requests from the browser.

Multiline text is supported — a sanitising middleware escapes literal newlines inside JSON string values so you can paste passages without manual escaping.

## Endpoints

### `GET /health`

Liveness check.

**Response:**

```json
{
"status": "ok"
}
```

**Example:**

```bash
curl http://127.0.0.1:8000/health
```

### `POST /analyse`

Analyse text for spiritual disinformation markers. This is a synchronous endpoint — FastAPI runs the CPU-bound spaCy work in a thread pool.

**Request body:**

| Field | Type | Required | Default | Description |
|-------|------|----------|---------|-------------|
| `text` | `string` | Yes | — | Text to analyse (1–100,000 characters) |
| `density_bias` | `float` | No | `0.75` | Information density bias for heuristic layer (0.0–1.0) |
| `seed` | `int \| null` | No | `null` | RNG seed for reproducible heuristic scores |

**Response body:**

| Field | Type | Description |
|-------|------|-------------|
| `overall_threat_score` | `float` | Hybrid score (0–100) |
| `tech_contribution` | `float` | Tech layer score (0–100) |
| `intuition_contribution` | `float` | Heuristic layer score |
| `detected_entities` | `list[str]` | Named entities found by spaCy |
| `authority_hits` | `list[str]` | Matched authority claim phrases |
| `urgency_hits` | `list[str]` | Matched urgency/fear patterns |
| `emotion_hits` | `list[str]` | Matched fear and euphoria words/phrases |
| `contradiction_hits` | `list[str]` | Detected contradiction pair labels |
| `source_attribution_hits` | `list[str]` | Unfalsifiable and unnamed authority phrases |
| `escalation_hits` | `list[str]` | Commitment escalation labels by segment |
| `message` | `str` | Disclaimer |

**Example request:**

```bash
curl -X POST http://127.0.0.1:8000/analyse \
-H "Content-Type: application/json" \
-d '{"text": "The ascended masters say you must act now. Time is running out.", "seed": 42}'
```

**Example response:**

```json
{
"overall_threat_score": 27.74,
"tech_contribution": 23.36,
"intuition_contribution": 34.31,
"detected_entities": [],
"authority_hits": ["the ascended masters say"],
"urgency_hits": ["you must act now", "time is running out"],
"emotion_hits": [],
"contradiction_hits": [],
"source_attribution_hits": [],
"escalation_hits": [],
"message": "Run on your own texts only — this is a local tool."
}
```

## Error Handling

The API returns **422 Unprocessable Entity** for validation errors, such as:

- Empty `text` field
- `density_bias` outside the 0.0–1.0 range
- `text` exceeding 100,000 characters

Errors follow Pydantic's validation error format:

```json
{
"detail": [
{
"type": "string_too_short",
"loc": ["body", "text"],
"msg": "String should have at least 1 character",
"input": "",
"ctx": {"min_length": 1}
}
]
}
```
Loading