LogOcean

A self-hosted, agentless SIEM — parse, normalize, detect, and retain security logs from 29 vendors in PostgreSQL.

LogOcean ingests network, endpoint, cloud, identity, and OT/ICS logs from many vendors through three front doors — web upload, an HTTP ingest API, and a syslog receiver (UDP/TCP/TLS) — then auto-detects the format, parses and normalizes each record to one common schema, indexes it for full-text and structured search, matches it against a Sigma-based detection & correlation engine plus threat-intel feeds, and retains everything in PostgreSQL for ≥ 3 years. Alerts flow into triage, cases, notifications, and agentless response — with UEBA, kill-chain reconstruction, a detection-coverage scoreboard, an AI copilot, and passive OT/ICS monitoring on top.

Table of contents

• Overview
• Key features
• Architecture
• Quick start
  • Docker
  • Local (without Docker)
• Supported log sources
• Live ingestion
  • HTTP ingest API
  • Syslog receiver
  • Bulk import (large / historical backups)
• Detection & alerting
  • Rules
  • Triage & tuning
  • Cases / incidents
  • Saved searches
• Detection coverage
  • Coverage scoreboard
  • Import community SigmaHQ rules
• Notifications & response
• UEBA & entity risk
• Kill-chain reconstruction
• Detection-engineering workbench
• AI SOC copilot
• OT / ICS monitoring
• Agentless collectors & feeds
• Threat-intelligence enrichment
• Dashboards & reporting
• Configuration
• Project layout
• Testing & CI
• Data model & retention
• Parser accuracy notes
• Security
• Roadmap
• License & attribution

Overview

LogOcean is an agentless SIEM. Ingested events are matched against a Sigma-based detection & correlation engine and threat-intel IOC feeds, raising alerts you work at /alerts: triage them (assign, note, suppress noise) and group related ones into cases (/cases). New alerts are pushed to your channels and can trigger agentless response playbooks (audited at /responses). Collectors pull vendor / cloud / identity logs (Okta, GitHub, GitLab, AWS CloudTrail, Entra ID, Microsoft 365, GCP Cloud Audit Logs) while other tools push findings to the API.

UEBA entity-risk analytics (/risk) baseline every user / host / IP and flag new-entity / new-association anomalies beyond the rules; kill-chain reconstruction (/killchain) stitches related alerts across ATT&CK tactics into attack stories; and a detection workbench (/workbench) maps coverage, flags noisy / dead rules, and tests Sigma rules live. A /coverage scoreboard measures MITRE ATT&CK + ATLAS coverage, and a SigmaHQ importer brings in the community rule corpus. An optional AI SOC copilot (Claude) explains alerts, summarizes cases, and drafts Sigma rules from plain English. OT / ICS monitoring (/ot) ingests passive Zeek + ICSNPP telemetry (Modbus / DNP3 / S7comm / CIP …) with an ATT&CK-for-ICS rule pack, asset inventory, and master→controller baselining. Dashboards + /reports visualize it all (charts, top-N, ATT&CK-Navigator / CSV export), and auth / RBAC, an audit log, and a /compliance view (MITRE → NIST / CIS / ISO 27001 / SOC 2 / PCI / HIPAA + IEC 62443 / NERC CIP) round it out — all tested unit + integration against real PostgreSQL in CI.

Key features

• 29 parsers, auto-detected across network / firewall, endpoint / IDS, cloud / identity, private cloud (Nutanix), OT / ICS, and generic CEF / LEEF / syslog / JSON.
• One normalized schema (time, vendor, type, src/dst IP+port, user, host, action, severity, rule, bytes, message) with the full original record kept in jsonb.
• Three ingest paths — web upload, HTTP API (API-key auth, gzip-aware), and a syslog receiver (UDP/TCP/TLS) fronted by a bounded async queue.
• Sigma-based detection & correlation (90 detection + 10 correlation rules shipped) with the common Sigma modifiers, plus a community SigmaHQ importer.
• Detection-coverage scoreboard for MITRE ATT&CK (Enterprise + ICS) and MITRE ATLAS, with Navigator-layer export and a CI rule-linter.
• Threat-intelligence IOC enrichment, notifications (webhook / email), and agentless response playbooks (with time-boxed auto-revert).
• Triage workflow — assignment, notes, suppression/allowlists, case grouping, and per-user saved searches on the event + alert views.
• Investigation & analytics — UEBA entity risk, kill-chain reconstruction, a detection-engineering workbench, and an optional AI SOC copilot (Claude).
• Passive OT / ICS monitoring (Zeek + ICSNPP) with an ATT&CK-for-ICS rule pack, asset inventory, and IEC 62443 / NERC CIP compliance mapping.
• Agentless collectors for Okta, GitHub, GitLab, AWS CloudTrail, Entra ID, Microsoft 365, and GCP Cloud Audit Logs.
• PostgreSQL storage, RANGE-partitioned by month, with GIN full-text, a jsonb GIN index, and btree indexes on the common fields — plus ≥ 3-year retention (purge = instant partition drop).
• Auth / RBAC, security headers + CSRF, an audit log, and compliance reporting.

Architecture

Three inputs share one core: a source-agnostic detect → parse → normalize → store pipeline. Live sources buffer in a bounded async queue drained by writer workers, so a burst never blocks the receiver.

 upload (web) ───────────────┐
 POST /api/v1/ingest (key) ──┤─► auto-detect ─► parse ─► normalize ─► store (Postgres)
 syslog UDP/TCP/TLS ─► queue ┘     format                common      month-partitioned
                                                          schema      events + FTS index
                                                                           │
                                                  search ◄── filters + full-text ◄──┘

Every event is evaluated inline against per-event detection rules as it is stored; a background scheduler runs correlation (threshold) rules over the event store. Both raise rows in alerts, which fan out to notifications and response. The full original record is always kept in events.raw (jsonb) so nothing is lost and any field stays searchable.

• Stack: Python 3.11–3.13, FastAPI + Uvicorn, Jinja2 (server-rendered UI), PostgreSQL 16 via psycopg 3 (+ psycopg_pool), python-dateutil.
• No JS build step, no chart library — charts are server-rendered CSS/SVG.

Quick start

Docker

cp .env.example .env          # optional: adjust retention / limits
docker compose up --build     # starts Postgres + the app
# open http://localhost:8000

Then Upload a file (try the ones in samples/), and Search.

Local (without Docker)

python -m venv .venv && . .venv/bin/activate      # Windows: .venv\Scripts\activate
pip install -r requirements.txt

# point at a Postgres you control:
export DB_DSN="postgresql://logocean:logocean@localhost:5432/logocean"   # PowerShell: $env:DB_DSN=...
uvicorn app.main:app --reload

The schema (tables, partitions, indexes) is created automatically on startup.

Supported log sources

All 29 parsers are auto-detected on upload (or selectable explicitly). Grouped by domain:

Domain	Sources
Network / firewall	Palo Alto NGFW (CSV export & syslog), Fortinet FortiGate, Cisco ASA / Firepower, Cisco IOS / IOS-XE / NX-OS, Cisco Meraki, Zeek (TSV & JSON)
Endpoint / IDS / host	CrowdStrike Falcon (CSV & JSON), Windows Security Event Log, Microsoft Sysmon, Linux auditd, Apache / Nginx access logs, Suricata EVE
Cloud / identity	AWS CloudTrail, Google Cloud Audit Logs, Microsoft Azure Activity, Microsoft 365 Unified Audit, Microsoft Entra ID sign-ins, Okta System Log, GitHub audit, GitLab audit
Private cloud / virtualization	Nutanix Prism Central (audit / API / Flow), Nutanix Files / Data Lens (file-access audit)
OT / ICS (Zeek + ICSNPP)	Modbus, DNP3, S7comm, CIP / EtherNet-IP, BACnet … enriched into a control-plane action + ot.* fields (see OT / ICS monitoring)
Generic	CEF (ArcSight & many WAFs / proxies / AV), LEEF 1.0 / 2.0 (Tripwire, QRadar, Juniper, Check Point …), generic syslog (RFC 3164 / 5424), generic JSON / NDJSON (flat or ECS)

Every parser normalizes to one schema and keeps the full original record in jsonb.

How to export logs from each source

Source	How to export	Upload as
Palo Alto NGFW	Monitor ▸ Logs ▸ (Traffic/Threat/URL/System/Config) ▸ Export to CSV	Palo Alto CSV (auto)
Palo Alto NGFW	Syslog file from your collector / forwarder	Palo Alto syslog (auto)
Fortinet FortiGate	Syslog from your collector, or FortiAnalyzer ▸ Log download	Fortinet FortiGate (auto)
CrowdStrike Falcon	Endpoint security ▸ Detections / Incidents ▸ Export (CSV)	CrowdStrike CSV (auto)
CrowdStrike Falcon	Event Search / API / FDR export (JSON or NDJSON)	CrowdStrike JSON (auto)
Windows hosts	Get-WinEvent -LogName Security ▸ Export-Csv (or ConvertTo-Json); or Event Viewer ▸ Save All Events As CSV	Windows Security (auto)
Windows hosts (Sysmon)	Get-WinEvent -LogName 'Microsoft-Windows-Sysmon/Operational' ▸ ConvertTo-Json / Export-Csv	Sysmon (auto)
Linux hosts	/var/log/audit/audit.log (auditd)	Linux auditd (auto)
Apache / Nginx	access.log (Common / Combined Log Format)	Apache / Nginx access (auto)
Suricata IDS/IPS	eve.json (NDJSON) or an exported JSON array	Suricata EVE (auto)
Cisco ASA / Firepower	Syslog from your collector (lines with %ASA-…/%FTD-…)	Cisco ASA / Firepower (auto)
Cisco IOS / IOS-XE / NX-OS	Device syslog (lines with %FACILITY-SEV-MNEMONIC)	Cisco IOS (auto)
Cisco Meraki	Dashboard ▸ syslog server output (flows / urls / ids-alerts …)	Cisco Meraki (auto)
Zeek (Bro)	conn.log / dns.log / http.log — classic TSV (#fields) or JSON	Zeek TSV / JSON (auto)
AWS CloudTrail	S3/CloudWatch export or aws cloudtrail lookup-events (JSON)	AWS CloudTrail (auto)
Google Cloud	Cloud Logging export or gcloud logging read --format json	Google Cloud Audit (auto)
Microsoft Azure	Activity Log export ({"records":…}) or az monitor activity-log list	Microsoft Azure Activity (auto)
Microsoft 365	Search-UnifiedAuditLog ▸ AuditData, or Management Activity API (JSON)	Microsoft 365 (auto)
Microsoft Entra ID	Sign-in logs via Graph auditLogs/signIns or Azure Monitor export (JSON)	Microsoft Entra ID (auto)
Okta	System Log API export (JSON array / NDJSON)	Okta System Log (auto)
GitHub	Org/Enterprise ▸ audit log ▸ Export (JSON / NDJSON)	GitHub audit (auto)
GitLab	Admin ▸ /audit_events API (JSON)	GitLab audit (auto)
Nutanix Prism Central	Settings ▸ Syslog server (modules: Audit / API Audit / Flow) → your collector	Nutanix Prism Central (auto)
Nutanix Files / Data Lens	Partner server (vendor_name: syslog, :1468) file-audit notifications, or a File-Analytics / Data-Lens JSON export	Nutanix Files (auto)
Any CEF source	Syslog / file in Common Event Format (CEF:0|…)	CEF (auto)
Tripwire Log Center / Enterprise	Forwarder ▸ send events as LEEF or CEF (or the exported log file)	LEEF / CEF (auto)
IBM QRadar	Routing/forwarding rule ▸ export events as LEEF (its native format), or CEF / syslog. (A QRadar system-backup archive is proprietary — export events first.)	LEEF (auto)
Any LEEF source (QRadar, Juniper, Check Point …)	Syslog / file in Log Event Extended Format (LEEF:1.0|… / LEEF:2.0|…)	LEEF (auto)
Any syslog source	Plain RFC 3164 / 5424 syslog not matched above	Generic syslog (auto)
Any JSON source	Flat or ECS-style JSON / NDJSON not matched above	Generic JSON (auto)

Auto-detect inspects the header/content; if a file is ambiguous, pick the format explicitly in the upload form.

Live ingestion

Besides manual upload, LogOcean accepts logs in near-real-time through two front doors that share the same detect → parse → normalize → store pipeline.

HTTP ingest API

Create a key on the Admin page, then POST raw log content:

curl -X POST "http://localhost:8000/api/v1/ingest?format=auto&filename=fw.log" \
     -H "X-API-Key: lo_..." --data-binary @fw.log
# -> {"batch_id": 42, "format": "...", "total": N, "inserted": N, ...}

format may be auto or any format key; auth is X-API-Key or Authorization: Bearer. Only the sha256 of each key is stored (the plaintext is shown once). A gzip body is decompressed transparently (send .gz bytes directly, or --data-binary @fw.log.gz) — the MAX_UPLOAD_MB limit then applies to the decompressed size, with a decompression-bomb guard. Drag-dropping a .gz file into the web Upload page works the same way.

Syslog receiver

Set SYSLOG_ENABLED=true to listen on UDP+TCP (default port 5514; TLS optional). Point a collector or device at it:

logger -n localhost -P 5514 -d "<134>1 2026-06-24T10:00:00Z fw test message"

Messages flow through a bounded async queue with writer workers that batch-insert, so a burst never blocks the receiver; queue counters are on GET /health. TCP framing supports both octet-counting (RFC 6587) and newline-delimited streams.

Bulk import (large / historical backups)

To load a big multi-GB export — e.g. a 3-year IBM QRadar LEEF backup — use clients/logocean_import.py. It streams the file in size-bounded, line-aligned chunks (so no single request exceeds MAX_UPLOAD_MB), decompresses .gz locally, retries transient failures, and prints progress. Ingest is idempotent (per-event dedup hash), so re-running after an interruption is safe.

python clients/logocean_import.py --url http://localhost:8000 --key lo_... \
    --format leef --max-mb 400 qradar_3yr.leef.gz
# [chunk 1] 400.0 MB -> inserted=812345 duplicates=0 (batch 5)  ...

Events keep their original timestamps, so 3 years of history lands in the correct monthly partitions (not the upload date). For a historical backfill, consider DETECTION_ENABLED=false and UEBA_ENABLED=false during the load to avoid a flood of stale alerts and to speed ingest. --gzip compresses each POST body (the server gunzips it) to save bandwidth; --max-mb must be ≤ the server's MAX_UPLOAD_MB. Note: a proprietary QRadar system backup archive is not readable — export the events as LEEF/CEF/syslog first.

Detection & alerting

Every ingested event is evaluated against detection rules, and a background scheduler runs correlation rules over the event store. Matches raise alerts you can filter, drill into (down to the originating event), and triage at /alerts; open-alert counts show on the dashboard.

Rules

Rules are YAML files in rules/ (a Sigma-compatible subset), tagged with MITRE ATT&CK, and enable/disable from the Admin page (applies immediately). The engine supports the common Sigma field modifiers so many community rules load as-is: contains/startswith/endswith/re (with i/m/s flags)/cased, value lists with |all, cidr (IP-in-network), numeric lt/lte/gt/gte, exists, fieldref, and the base64/base64offset/windash encoding modifiers for command-line obfuscation — plus the and/or/not + N of … condition grammar.

# per-event rule
title: RDP Connection Allowed
id: lo-rdp-allowed
level: medium
fidelity: medium            # high | medium | hunt — drives the coverage scoreboard
data_source: [firewall]     # ATT&CK-style data-source key(s)
detection:
  selection: { dst_port: 3389 }
  permitted: { action: [allow, accept] }
  condition: selection and permitted
tags: [attack.t1021.001, attack.lateral_movement]

# correlation (threshold) rule — e.g. brute force
title: Brute Force - Failed Logon Burst
id: lo-corr-bruteforce-logon
level: high
correlation:
  match: { action: failed-logon }
  group_by: [src_ip]
  window: 5m
  threshold: 5
tags: [attack.t1110, attack.credential_access]

Correlation rules can also count distinct values of a second column instead of raw events (distinct_count:), which expresses behaviours defined by breadth — e.g. password spraying (one source failing against many distinct accounts) rather than a simple failed-logon burst:

# cardinality rule — one src_ip → 10+ distinct accounts = password spray
title: Password Spray - One Source, Many Accounts
id: lo-corr-password-spray
level: high
correlation:
  match: { action: failed-logon }
  group_by: [src_ip]
  distinct_count: user_name      # count DISTINCT users, not raw failures
  window: 10m
  threshold: 10
tags: [attack.t1110.003, attack.credential_access]

LogOcean ships a starter pack of 90 detection + 10 correlation rules covering failed-logon brute force, denied-connection floods, RDP exposure (incl. external RDP via cidr), ingress-tool transfer, event-log clearing, security-tool tampering, encoded/download PowerShell (base64offset/windash), AWS CloudTrail (logging disabled, root console login, world-open security groups, access-key creation), Entra ID (risky sign-in succeeded, legacy auth), Okta (admin grant, MFA deactivation), Microsoft 365 (mailbox forwarding rules), GitHub (repo made public), and these purpose-built packs:

• Tripwire file-integrity monitoring — critical system / credential file change, web-shell drop, persistence-mechanism change, integrity monitoring disabled, monitored-object deletion, plus a mass-change-burst correlation for ransomware / bulk tampering.
• Sysmon / endpoint — Office spawning a shell, LOLBin proxy execution, registry Run-key / WMI persistence, LSASS credential dumping, shadow-copy deletion, scheduled-task creation, command-line log clearing, plus a curated high-fidelity endpoint pack (Phase 2): LSASS memory access by mask, known credential-dumper tools, NTDS/SAM extraction, WDigest plaintext-caching, BYOVD vulnerable-driver load, Defender-disable & AMSI/ETW tampering, UAC registry hijack, LSA/AppInit persistence, remote-thread injection, PsExec / msiexec / BITS execution, Cobalt Strike named pipes, and local-account creation — each fidelity-tagged and IOC-verified.
• Cloud & identity (Phase 3) — a high-value pack across every cloud/identity source, adding first-time coverage for GCP, Azure and GitLab: AWS (threat-detection / Config disabled, admin-policy attach, public-S3 exposure), GCP (privileged IAM grant, service-account key created, logging-sink deleted), Azure (RBAC role assignment, diagnostic-settings deleted, Key Vault tamper), GitLab (2FA disabled, project made public), Okta (admin impersonation), Microsoft 365 (org transport rule, mailbox delegate), GitHub (org 2FA disabled, branch-protection removed) — every API event/operation name adversarially verified against provider docs.
• Linux & network / web-exploitation (Phase 4) — request-line detections for web attacks (T1190): SQL injection, path traversal / LFI, OS command injection, XSS, web-shell access, and scanner user-agents (each fires across Apache/Nginx, Zeek HTTP and Suricata HTTP); Suricata IDS passthrough (web-app-attack, trojan/C2, crypto-mining categories); and Linux auditd TTPs — reverse shells, setuid backdoors, sudoers / SSH-authorized_keys / cron persistence, /etc/shadow access, and security-control disabling — with every payload pattern and IDS category string adversarially verified.
• Behavioural correlation (Phase 5) — beyond simple event-count bursts, cardinality (distinct-count) rules catch behaviours defined by breadth: password spray (one source → many distinct accounts), distributed brute force (one account ← many distinct source IPs), port scan (one host → many distinct ports), and network sweep (one source denied to many distinct hosts).
• OT / ICS — Modbus write / diagnostic, S7comm program download / PLC stop, DNP3 device restart / disable-unsolicited, CIP set-attribute write, an IT→OT write conduit-violation (Purdue / IEC 62443 zone) rule, and an OT-protocol enumeration correlation — tagged with ATT&CK for ICS (see OT / ICS monitoring).
• Nutanix Prism Central — VM / cloud-instance deletion via the REST API, cluster unregister / detach, user / role / authentication change, plus a Flow microsegmentation drop-burst correlation for internal scanning / lateral movement.
• Nutanix Files / Data Lens — ransomware-encrypted-extension / ransom-note write, share ACL / permission change, plus a mass-file-deletion-per-user correlation for ransomware / wiper / insider destruction.

Detection can be turned off with DETECTION_ENABLED=false.

Triage & tuning

Each alert can be acknowledged / closed / reopened, assigned to an analyst, and annotated with threaded notes. To cut noise, build suppression / allowlist rules — by rule id, source IP/CIDR, user, host and/or vendor (each set field is an AND condition). A matching alert is stored as suppressed (kept for audit, hidden from the default queue, and never notified or actioned), and the suppression's hit count is tracked. The fastest way to make one is the Suppress similar form on any alert (pre-filled from its attributes); manage them all under Admin ▸ Suppressions.

Cases / incidents

Group related alerts into one investigation at /cases. Create a case from any alert (or add it to an open one), give it a status (open / investigating / closed), an assignee and threaded notes; its severity rolls up to the highest of its member alerts. The case page suggests related alerts — open, un-cased alerts sharing a source IP, user or host with the case — so a burst of activity folds into a single timeline with one click.

Saved searches

Name and re-run your common queries. On both /search and /alerts, a Save current box stores the active filter set; saved searches appear as one-click chips above the results (delete with the ×). They are per-user (when auth is on) and scoped to their page, so "my open high alerts" or "Fortinet denies this week" is always one click away.

Detection coverage

Coverage scoreboard

The /coverage page grows the detection pack against a measured MITRE ATT&CK (Enterprise + ICS) and MITRE ATLAS (adversarial-AI) coverage map, computed from the rules themselves — what LogOcean can detect, not what has fired. It rolls coverage up by tactic, fidelity (high / medium / hunt) and data source, renders the ATLAS matrix (a scaffold until AI/LLM telemetry lands), and exports MITRE ATT&CK Navigator layers scored by rule coverage (/coverage/attack-navigator.json, ?domain=ics for the ICS matrix). A CI rule-linter enforces per-rule quality (unique ids, valid level/fidelity, an attack.*/atlas.* tag). The programme is tracked in docs/DETECTION_COVERAGE_ROADMAP.md.

Import community SigmaHQ rules

scripts/import_sigma.py translates the thousands of open, ATT&CK-tagged SigmaHQ rules into LogOcean's engine — Sigma logsource becomes a gate over our normalized fields (a process_creation rule only fires on process-create events), Sigma field names pass through (our Sysmon/endpoint parsers lift them onto raw), and anything we can't faithfully run is skipped with a reason (unmapped logsource, unsupported modifier, Sigma aggregation/correlation, deprecated). Point it at a clone and it emits rules/imported/*.yml (loaded alongside rules/, generated per-deployment):

git clone https://github.com/SigmaHQ/sigma
python scripts/import_sigma.py --src sigma/rules --write   # then restart to load

Notifications & response

Newly-raised alerts at or above NOTIFY_MIN_LEVEL are delivered to notification channels — a webhook (Slack/Teams/Discord/generic, WEBHOOK_URL) and/or email (SMTP_*) — by a background worker, so slow delivery never blocks ingest. Set NOTIFY_ENABLED=true and configure a channel.

Alerts can also trigger response playbooks (playbooks/*.yml). A playbook matches alerts (by rule id / severity / technique) and runs an agentless action: a structured webhook POST to your automation/SOAR/firewall/IAM endpoint (RESPONSE_WEBHOOK_URL) carrying the intent, or a log action that only records. LogOcean stays agentless and lets your platform enforce. Every action is audited at /responses and on the alert's page.

A revert_after makes the action time-boxed: a background scheduler (RESPONSE_AUTO_REVERT, on by default) fires the inverse intent (block_ip→unblock_ip, disable_user→enable_user, …) once it expires, so temporary containment lifts itself — the revert appears as its own /responses row.

# playbooks/block_bruteforce_source.yml
title: Block brute-force source IP
id: pb-block-bruteforce
match: { rule_id: [lo-corr-bruteforce-logon], min_level: high }
action: { type: block_ip, target: src_ip }   # POSTs {action, target, alert} to your SOAR
revert_after: 600                             # auto-POST unblock_ip after 10 min

UEBA & entity risk

Beyond signature rules, LogOcean tracks behaviour. With UEBA_ENABLED (on by default), every ingested event updates lightweight entity baselines — one row per user / host / source IP with first-seen / last-seen, plus the associations between them (user↔IP, user↔host, host↔IP). The /risk page then shows:

• Riskiest entities (users, hosts, IPs), scored from their attributed alerts — severity-weighted and recency-decayed (RISK_HALF_LIFE_DAYS), so a recent critical outweighs an old low. Click through to a per-entity timeline, associations, and alerts (/entity?etype=user&value=…).
• Anomalies (24h) — new entities (first time we've ever seen this user/host/IP) and new associations (an established actor suddenly using a new IP or host — a classic account-takeover / lateral-movement signal), surfaced without any rule having to fire.

It's pure PostgreSQL (no ML dependency); the baselines are maintained incrementally in the ingest path.

Kill-chain reconstruction

A single alert is one step; an intrusion is a sequence of steps by the same actor across ATT&CK tactics over time. The /killchain page stitches related alerts into attack stories:

• alerts are linked when they share an entity (user / host / IP) and fall within KILLCHAIN_MAX_GAP_MINUTES of each other (single-linkage, so a long campaign chained through intermediate alerts stays one story);
• a linked group only qualifies when it spans ≥ KILLCHAIN_MIN_TACTICS distinct ATT&CK tactics — showing progression along the kill chain, not just a burst of one behaviour;
• each story is presented as kill-chain-ordered stages (Initial Access → Execution → Credential Access → …), the pivot entities that tie it together, a rolled-up severity, and a plain narrative.

Promote a story to an investigation case with one click (severity + alert linkage roll up exactly like a manually built case). With KILLCHAIN_AUTOCREATE=true, a background scheduler auto-promotes stories at or above KILLCHAIN_MIN_SEVERITY, de-duplicated by the story's alert-set signature. The reconstruction runs over recent un-cased alerts, so once a story is folded into a case it won't be re-surfaced. Like UEBA, it's pure PostgreSQL — the reconstructor (app/killchain.py) is dependency-free and fully unit-tested.

Detection-engineering workbench

The /workbench page helps you tune the detection pack itself:

• ATT&CK coverage map — per-tactic (kill-chain-ordered) view of which techniques the enabled rules cover, and the gaps (techniques only a disabled rule would catch, or none at all), plus an overall coverage %.
• Rule health — every rule with its firing counts over the last WORKBENCH_WINDOW_DAYS, bucketed into noisy (≥ WORKBENCH_NOISY_THRESHOLD alerts in the window), never-fired (enabled but no alert on record — untested or dead), and stale (fired historically, silent now).
• Rule tester — paste a Sigma-subset rule and a sample event and evaluate it with the same engine the pipeline uses; the result shows the final verdict, the logsource match, and each named selection's boolean so you can see exactly why it did or didn't fire. Event fields may be normalized names (user_name, src_ip…) or raw vendor fields.

The analytics (app/workbench.py) are pure functions over the rule registry, so they're fully unit-tested without a database.

AI SOC copilot

With COPILOT_ENABLED=true and an API key, LogOcean adds Claude-powered assistance at three points (off by default; the app runs fine without the anthropic package):

• Alert explainer — on any alert, Explain this alert sends the alert plus related activity (same entity) to Claude and returns a plain-language "what happened / why it fired / severity & likelihood / triage steps" briefing.
• Case summarizer — on a case, Summarize this case drafts an incident summary (attack narrative in ATT&CK order, impacted entities, recommended actions) from the member alerts and notes; one click saves it as a case note.
• Sigma-from-natural-language — on the workbench, describe a detection in English and Claude drafts a Sigma-subset rule, loaded straight into the rule tester so you can validate it before adding it to rules/.

The model is configurable (COPILOT_MODEL, default claude-opus-4-8) so operators can trade cost for capability (e.g. claude-sonnet-4-6). Prompt construction and the Sigma-extraction logic (app/copilot/prompts.py) are pure and fully unit-tested; the SDK call is a thin wrapper that degrades gracefully when unconfigured. Every AI action is RBAC-gated (analyst) and written to the audit log.

OT / ICS monitoring

LogOcean monitors operational-technology networks — PLCs, RTUs, and other controllers — the safe way: 100% passive and agentless. It never talks to a device or issues a control command; it ingests copies of telemetry produced by a network sensor, exactly the posture OT requires.

The telemetry source is Zeek running the ICSNPP (Industrial Control Systems Network Protocol Parsers, from CISA / INL) analyzers, which deep-packet-inspect the OT protocols and write modbus.log / dnp3.log / s7comm.log / cip.log … in the usual Zeek #fields shape. LogOcean's Zeek parsers enrich those records:

• the control-plane operation is lifted onto the normalized action (write-registers, write-coils, diagnostic, program-download, plc-stop, cold-restart, disable-unsolicited, write-attribute, …), and log_type is set to the canonical protocol (modbus / dnp3 / s7comm / cip / enip / …);
• an ot.* field set (ot.protocol, ot.operation = read/write/control, ot.is_write, ot.function_code, ot.unit_id, ot.address, …) is added to the event's raw, so every OT attribute is searchable and rule-matchable.

On top of that, the shipped OT rule pack flags the dangerous operations, tagged with ATT&CK for ICS techniques:

Rule	Technique
Modbus write command to a controller	T0855 / T0836
Modbus diagnostic / restart function	T0814
S7comm program / block download to a PLC	T0843 / T0889
S7comm PLC stop / operating-mode change	T0858 / T0813
DNP3 device restart (cold / warm)	T0816
DNP3 disable unsolicited reporting	T0878
CIP set-attribute (write) to a device	T0855 / T0836
IT→OT write/control across a zone boundary (Purdue / IEC 62443)	T0855 / T0886
OT-protocol enumeration from one source (correlation)	T0846

ATT&CK for ICS is wired through the rest of the platform too: kill-chain reconstruction understands the ICS tactics (inhibit-response-function, impair-process-control, …) in a merged IT→OT kill chain, so an intrusion that starts on IT and ends in process impact stitches into one attack story; and the Navigator export serves an ICS layer at GET /reports/attack-navigator.json?domain=ics-attack.

OT analysis (/ot). A dedicated page turns the OT telemetry into operator analytics: an asset inventory of the controllers (PLCs / RTUs) seen on the wire — the protocols each speaks, how many masters talk to it, and its control-op volume; a master → controller conversation map that flags a new-writer (a source first seen in the last 24h already issuing write / control commands to a controller — the top OT signal, an unexpected engineering client); and read / write / control activity per protocol. It's pure PostgreSQL aggregation over the events, no schema change. The IT→OT conduit-violation rule (rules/ot_it_to_ot_write.yml) enforces the Purdue / IEC 62443 zone model with cidr selections — a write/control command from outside the OT zone alerts (tune the CIDRs to your control network).

Compliance. OT rule coverage maps to IEC 62443-3-3 System Requirements and NERC CIP (plus NIST 800-53) on the /compliance page, alongside the existing IT frameworks.

Scope & limits. LogOcean is a log platform, not a sensor — it needs Zeek+ICSNPP (or a commercial DPI sensor) on the wire; it does no DPI itself. Serial Modbus and L2 GOOSE / Sampled-Values can't arrive over IP logs and need a sensor that sees them. OT response stays passive: alert / ticket / enforce at the IT boundary — never a device command.

Agentless collectors & feeds

Two agentless ways to get logs in without manual upload:

• Pull collectors — set COLLECTORS_ENABLED=true and a collector's credentials. A scheduler fetches new records every COLLECTOR_INTERVAL seconds, checkpointing a per-source cursor so each run only pulls what's new, and feeds them through the same parse→detect→alert pipeline. Status + enable/disable are on the Admin page. Built-in collectors (each activates only when its credentials are set):

  • Okta System Log (OKTA_*), GitHub audit log (GITHUB_*), GitLab audit events (GITLAB_*) — token-based REST.
  • AWS CloudTrail (AWS_REGION + AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY, optional AWS_SESSION_TOKEN) — LookupEvents signed with AWS SigV4 (stdlib).
  • Microsoft Entra ID sign-in logs and Microsoft 365 unified audit log — one Azure app registration (AZURE_TENANT_ID/AZURE_CLIENT_ID/AZURE_CLIENT_SECRET), OAuth2 client-credentials. Entra uses Microsoft Graph; M365 uses the Office 365 Management Activity API and additionally needs M365_ENABLED=true.
  • GCP Cloud Audit Logs (GCP_PROJECT_ID + GCP_CLIENT_EMAIL + GCP_PRIVATE_KEY from a service-account key) — Cloud Logging entries:list, authenticated with a service-account signed-JWT OAuth2 grant (RS256 signed via stdlib — no SDK). Grant the service account roles/logging.viewer.

• Push feeds — point your own tools (RHEL/Windows/SBOM/AWS audit scanners) at the ingest API. Copy clients/logocean_push.py:

  from logocean_push import push
  push("http://logocean:8000", "lo_...", findings)   # list of dicts -> generic_json

Threat-intelligence enrichment

Set THREATINTEL_ENABLED=true to match every ingested event against indicators of compromise — IPs, CIDRs, domains, file hashes, and URLs. A hit raises a Threat Intelligence Match alert that flows through the normal alert → notify → response path (its severity is the highest of the matched indicators).

• Feeds — THREATINTEL_FEEDS is a comma/space-separated list of local file paths or http(s) URLs, refreshed every THREATINTEL_REFRESH_MINUTES. Each feed may be a plain one-indicator-per-line list (# comments allowed), a CSV (indicator[,type[,severity[,description]]]), or JSON (an array of strings or of objects). Indicator type is inferred when not given.
• Manual indicators — add or remove individual IOCs from the Admin page; a Reload button re-fetches the feeds.
• Matching — indicators are held in an in-memory index for fast per-event lookup; the matcher checks the event's src/dst IPs (exact + CIDR) and pulls IPs/domains/URLs/hashes out of the normalized fields and raw (and from free text in message), so an indicator buried in a log line is still caught.

# a feed can be as simple as a blocklist file:
echo -e "# bad infra\n203.0.113.5\nevil.example\nhttp://drop.example/x" > feeds/iocs.txt
THREATINTEL_ENABLED=true THREATINTEL_FEEDS=feeds/iocs.txt  # ...then run as usual

Dashboards & reporting

The dashboard (/) shows headline counters (events, open alerts, open cases, on-disk size), an alert-volume time series, and top-N breakdowns — top firing rules, top MITRE ATT&CK techniques, and top alert/event source IPs — plus the existing per-vendor / per-log-type / partition tables. All charts are server-rendered CSS/SVG (no JS chart library, no extra dependencies).

The /reports page is a print-friendly summary over a selectable period (7–90 days) — headline metrics, alert status, the same time series and top-N charts, and the coverage span. Use Print / Save as PDF for a shareable PDF, and the toolbar buttons to export:

• ATT&CK Navigator layer — GET /reports/attack-navigator.json?days=N returns a Navigator (layer 4.5) JSON scoring each technique by alert volume; load it at the ATT&CK Navigator to visualize coverage. Add &domain=ics-attack for an ATT&CK for ICS layer (OT T0NNN techniques); the default is the Enterprise matrix.
• Alerts CSV — GET /alerts.csv streams the alert list (honours the /alerts filters), with spreadsheet-formula injection neutralized.

Configuration

All settings are environment variables (see .env.example).

Variable	Default	Meaning
DB_DSN	postgresql://logocean:logocean@localhost:5432/logocean	PostgreSQL connection
RETENTION_YEARS	3	Retention floor; purge cannot go below this
PAGE_SIZE	100	Search results per page
MAX_UPLOAD_MB	512	Reject larger uploads / API payloads
AUTO_PURGE	false	If true, drop partitions older than RETENTION_YEARS on startup
DETECTION_ENABLED	true	Evaluate detection + correlation rules and raise alerts
CORRELATION_INTERVAL	60	Seconds between correlation-rule evaluations
NOTIFY_ENABLED / NOTIFY_MIN_LEVEL	false / high	Send new alerts (>= level) to channels
WEBHOOK_URL / WEBHOOK_STYLE	— / slack	Notification webhook (Slack-text or full JSON)
SMTP_HOST … SMTP_TO	—	Email notification channel (host + from + to to activate)
RESPONSE_ENABLED / RESPONSE_WEBHOOK_URL	false / —	Run response playbooks; automation endpoint
RESPONSE_AUTO_REVERT / RESPONSE_REVERT_INTERVAL	true / 60	Auto-undo time-boxed actions; sweep period (s)
COLLECTORS_ENABLED / COLLECTOR_INTERVAL	false / 300	Scheduled pull collectors; poll period (s)
OKTA_* / GITHUB_* / GITLAB_* / AWS_* / AZURE_* / GCP_*	—	Per-collector credentials (a collector activates when set)
THREATINTEL_ENABLED / THREATINTEL_FEEDS	false / —	Match events against IOC feeds (paths or URLs)
THREATINTEL_REFRESH_MINUTES / THREATINTEL_DEFAULT_SEVERITY	60 / high	Feed refresh period; severity when a feed omits one
UEBA_ENABLED	true	Maintain entity baselines + the /risk page (behavioural analytics)
RISK_HALF_LIFE_DAYS / RISK_WINDOW_DAYS	7 / 30	Risk-score decay half-life; scoring window
KILLCHAIN_ENABLED	true	Reconstruct multi-tactic attack stories on the /killchain page
KILLCHAIN_WINDOW_HOURS / KILLCHAIN_MAX_GAP_MINUTES / KILLCHAIN_MIN_TACTICS	24 / 60 / 2	Look-back window; max gap to link alerts; min distinct tactics for a story
KILLCHAIN_AUTOCREATE / KILLCHAIN_INTERVAL / KILLCHAIN_MIN_SEVERITY	false / 300 / high	Auto-promote high-severity stories to cases; poll period (s); severity floor
WORKBENCH_WINDOW_DAYS / WORKBENCH_NOISY_THRESHOLD	30 / 50	Rule firing-stats window; alerts/window above which a rule is flagged noisy
COPILOT_ENABLED	false	Enable the Claude-powered alert/case explainers + Sigma-from-NL (needs anthropic + a key)
COPILOT_API_KEY / COPILOT_MODEL / COPILOT_MAX_TOKENS	— / claude-opus-4-8 / 1024	API key (else ANTHROPIC_API_KEY); Claude model; max response tokens
AUTH_ENABLED	false	Built-in login + RBAC (else front with SSO/proxy)
ADMIN_USER / ADMIN_PASSWORD	admin / —	Bootstrap admin on first run (random password logged if blank)
SESSION_TTL_HOURS / SESSION_COOKIE_SECURE	12 / false	Session lifetime; set secure cookie over HTTPS
INGEST_QUEUE_MAX	10000	Async ingest queue capacity (live sources)
INGEST_WORKERS	2	Writer workers draining the queue
INGEST_FLUSH_MAX / INGEST_FLUSH_MS	2000 / 1000	Flush a buffer at N events or N ms
SYSLOG_ENABLED	false	Listen for syslog (UDP+TCP)
SYSLOG_UDP_PORT / SYSLOG_TCP_PORT	5514 / 5514	Syslog ports (0 disables a transport)
SYSLOG_FORMAT	auto	Fixed parser for messages, or auto (per-message detect)
SYSLOG_TLS_CERT / SYSLOG_TLS_KEY	—	Enable TLS on syslog-over-TCP

Project layout

SIEM-Lite/                  # repo root (product: LogOcean)
├── docker-compose.yml      # Postgres + app
├── Dockerfile
├── schema.sql              # partitioned events table, FTS, indexes, batches
├── requirements.txt
├── app/
│   ├── main.py             # FastAPI routes + UI + lifespan
│   ├── api.py              # HTTP ingest API (POST /api/v1/ingest, API-key auth)
│   ├── config.py
│   ├── db.py               # pool, partitions, insert, search, stats, purge, api_keys
│   ├── pipeline.py         # source-agnostic parse → normalize → insert core
│   ├── ingest.py           # per-batch orchestration (sha, batch, source tagging)
│   ├── streaming.py        # bounded async ingest queue + batching writer workers
│   ├── receivers/syslog.py # UDP/TCP/TLS syslog receiver → queue
│   ├── detection/          # engine.py (per-event Sigma-subset), correlation.py, runtime.py
│   ├── alert_actions.py    # fan new alerts to notifications + response
│   ├── notify/             # webhook + email channels, background dispatcher
│   ├── response/           # agentless playbooks + audit log + stateful auto-revert
│   ├── collectors/         # pull connectors (Okta/GitHub/GitLab/AWS/Entra/M365/GCP) + scheduler
│   ├── threatintel/        # IOC matcher + feed loader + index runtime
│   ├── triage/             # suppression/allowlist matcher + index runtime
│   ├── navigator.py        # ATT&CK Navigator layer export (pure)
│   ├── risk.py             # UEBA entity extraction + risk scoring (pure)
│   ├── killchain.py        # kill-chain / attack-story reconstruction (pure)
│   ├── killchain_runtime.py # DB-backed reconstruction + auto-create scheduler
│   ├── ot.py               # OT/ICS analytics: asset/conversation classification (pure)
│   ├── workbench.py        # detection workbench: rule tester + coverage + health (pure)
│   ├── coverage.py         # ATT&CK (enterprise+ICS) + ATLAS detection-coverage scoreboard
│   ├── sigma_import.py     # translate community SigmaHQ rules → our engine (logsource gate)
│   ├── copilot/            # AI SOC copilot — prompts.py (pure) + client.py (Claude SDK wrapper)
│   ├── saved.py            # saved-search path/query validation + target URL (pure)
│   ├── severity.py         # canonical severity order + roll-up
│   ├── detect.py           # format auto-detection
│   ├── normalize.py        # dedup hash + full-text blob
│   ├── models.py           # NormalizedEvent
│   ├── auth.py             # password hashing (pbkdf2), roles, RBAC dependency
│   ├── compliance.py       # MITRE technique → framework control mapping + report
│   ├── util.py             # tolerant time/IP/int coercion; API-key helpers
│   ├── parsers/            # 29 vendor/format parsers (+ zeek_ics OT/ICS enrichment helper)
│   ├── templates/          # server-rendered Jinja2 pages (+ _macros chart partials)
│   └── static/style.css
├── rules/                  # detection + correlation rules (Sigma-subset YAML) · imported/ (SigmaHQ imports)
├── playbooks/              # agentless response playbooks
├── clients/                # logocean_push.py (push helper) · logocean_import.py (bulk file import)
├── scripts/                # coverage_report.py (coverage + Navigator layers) · import_sigma.py (SigmaHQ import)
├── docs/                   # DETECTION_COVERAGE_ROADMAP.md (living detection-coverage plan)
├── samples/                # one example file per format (+ samples/sigma/ importer fixtures)
└── tests/                  # unit (DB-free) + integration (real Postgres) tiers

Testing & CI

The suite has two tiers. Unit tests are DB-free and run anywhere; integration tests (marked integration) exercise a real PostgreSQL and self-skip when DB_DSN is unset.

pip install pytest python-dateutil
# Unit only (no database) — the default local experience:
PYTHONPATH=. python -m pytest tests/ -m "not integration" -q   # PowerShell: $env:PYTHONPATH="."

# Integration (needs Postgres + httpx); point DB_DSN at a throwaway database:
pip install httpx
DB_DSN=postgresql://logocean:logocean@localhost:5432/logocean \
  PYTHONPATH=. python -m pytest tests/ -m integration -q

The unit tier covers parsers + auto-detection (over the bundled samples), API-key auth, the async ingest queue (grouping, worker loop, backpressure), syslog TCP framing, gzip ingest decompression (bomb-guarded) + the bulk-import client's line-aligned chunker, the detection engine (Sigma-subset matching, all field modifiers + condition grammar, incl. the Tripwire-FIM, Sysmon/endpoint, Nutanix, and OT packs), the SigmaHQ importer (translation, logsource gating, skip reasons), the coverage scoreboard + rule-linter, inline detection in the pipeline, correlation-rule loading/dedup, notification routing + dispatcher, response playbook matching/execution, collector URL/cursor logic (incl. AWS SigV4 + Microsoft OAuth + GCP signed-JWT helpers), threat-intel (IOC classification, feed parsing, matching + alerting), suppression/allowlist matching, case severity roll-up, the ATT&CK Navigator builder, UEBA entity extraction + risk scoring, kill-chain reconstruction, the detection workbench, the AI copilot (prompt construction + Sigma extraction against a fake client), auth (pbkdf2, role ranking, the RBAC dependency), the audit helper, the OT/ICS enrichment + rule pack, and the compliance report — all without a database.

The integration tier runs against an actual PostgreSQL 16 and verifies what mocks can't: month-partition auto-creation, the GIN full-text index, inet/CIDR search, ON CONFLICT dedup, retention purge dropping whole partitions, the correlation SQL, the pipeline raising and suppressing alert rows, alert/case round-trips, the alert analytics aggregations, UEBA baselines / anomalies / risk ranking, kill-chain story→case creation, the workbench windowed stats, the coverage endpoints, and the HTTP stack end-to-end (TestClient → API-key auth → ingest → detect, plus the dashboard / report / Navigator / CSV endpoints). CI runs the unit tier on Python 3.11–3.13 and the integration tier against a Postgres service container (.github/workflows/tests.yml).

Data model & retention

• events is partitioned BY RANGE (event_time); partitions are monthly (events_YYYYMM) and created on demand at ingest. A events_default partition catches out-of-range timestamps. Time-range searches prune to the relevant months.
• Retention = dropping whole monthly partitions older than the cutoff (instant, no row-by-row delete). The Admin page exposes a guarded purge; the floor is RETENTION_YEARS. For 3-year retention you typically never purge — set AUTO_PURGE=true only when you want to stop keeping data beyond the floor.
• Idempotent ingest — every record has a dedup hash, so re-uploading the same file (or overlapping exports) does not create duplicates.
• Scale — tuned for manual-upload volumes (tens of millions of rows). For very high ingest, batch larger files, add a read replica, or move hot search to OpenSearch.

Parser accuracy notes

• Palo Alto CSV maps by column header (robust across PAN-OS versions).
• Palo Alto syslog uses documented positional field maps for Traffic/Threat/ System/Config (PAN-OS 10/11 common layout). The complete positional field list is preserved in raw.fields, so even if a field index drifts on your PAN-OS version, the data is retained and searchable, and the maps in app/parsers/paloalto_syslog.py are easy to adjust.
• CrowdStrike CSV/JSON resolve each field from multiple candidate names to cope with detection vs incident vs FDR shapes.
• Cisco ASA/Firepower mines the 5-tuple, bytes and user from the free-text message (best-effort src/dst, from/to, Built for/to); the full message is in raw.
• Zeek reads the #separator / #fields / #path header, so column order is taken from the file itself; a file may concatenate several logs (each with its own header).
• Cloud/identity JSON (CloudTrail, GCP, Azure, M365, Entra, Okta, GitHub, GitLab) is routed by record keys and resolves fields case-insensitively to tolerate camelCase (Graph) vs PascalCase (Azure Monitor) and wrapper shapes ({"Records":…}, {"records":…}, {"entries":…}, {"value":…}).
• Generic JSON is the JSON catch-all: it flattens one level so Elastic Common Schema keys (source.ip, event.action, user.name) resolve, and maps a wide set of candidate field names; anything unmapped stays in raw. It is the JSON fallback, so a recognized source is never shadowed by it.

Security

Set AUTH_ENABLED=true for built-in login + RBAC (roles: admin / analyst / viewer). An admin is bootstrapped on first run from ADMIN_USER/ADMIN_PASSWORD (a random password is logged if blank); manage users from the Admin page. Passwords are pbkdf2-hashed and sessions are server-side (revocable). Use SESSION_COOKIE_SECURE=true behind HTTPS. Every response carries security headers (X-Frame-Options: DENY, X-Content-Type-Options: nosniff, Referrer-Policy, and a CSP with frame-ancestors 'none'); with auth on, state-changing UI requests also get a CSRF same-origin (Origin/Referer) check, and the last enabled admin can't be demoted or disabled (no self-lockout). Security-relevant actions (login/logout, purge, key/rule/collector/user changes, alert triage, upload) are recorded in an audit log on the Admin page. With auth off, run behind your SSO/reverse proxy or on a trusted network. Either way, keep the Postgres volume backed up (it is your 3-year archive).

Input hardening. Ingest treats all log content as untrusted: uploads and the API body are size-capped (MAX_UPLOAD_MB) and streamed so a huge payload can't exhaust memory; a deeply-nested JSON "bomb" is rejected before parsing by an explicit depth guard (_MAX_JSON_DEPTH, version-stable — not reliant on the interpreter raising RecursionError); CSV exports neutralize spreadsheet formula injection; and correlation/search SQL uses whitelisted columns with fully parameterized values.

Roadmap

Detection coverage is grown as a phased programme — see docs/DETECTION_COVERAGE_ROADMAP.md for the living plan. Highlights: a measured ATT&CK (Enterprise + ICS) + ATLAS coverage scoreboard (done), a community SigmaHQ importer (done), curated high-fidelity endpoint / cloud / identity packs, a behavioural upgrade (temporal-sequence + distinct-count correlation, GeoIP enrichment for impossible-travel), and an adversarial-AI (ATLAS) track built on an LLM / AI-gateway telemetry parser.

License & attribution

Author: Krishnendu De · Co-author: Claude Fable 5.0.

See LICENSE for terms. Imported SigmaHQ rules retain their original authorship and are used under the Detection Rule License (DRL); MITRE ATT&CK and ATLAS are trademarks of The MITRE Corporation.