v0.12.0 - Solr write-health checks

Added

Solr write-health checks. Catches the failure mode that took out vfb_json on 2026-06-08: a closed Lucene IndexWriter (an EIO on write.lock when the soft NFSv3 mount backing /var/solr dropped) turns every /update into an HTTP 500 while /select and /admin/system keep returning 200. The existing liveness probe is a /select, so the page stayed green throughout. Two detectors now run against every core in solr_services:

• Passive (always on, read-only). Tracks UPDATE./update.serverErrors.count (5xx only) and flags a write outage when it climbs between checks. Computed from history before the new row is written, so each check seeds the next baseline; a post-restart counter reset reads as a negative delta and never false-alarms. No writes against prod.
• Active (opt-in, write_probe: true). Issues an empty commit — the only request that reliably forces IndexWriter.ensureOpen() — detecting a closed writer even with no other traffic. An empty commit changes no documents, but it is a write request, so it is off by default.

A container failing writes is marked not-ok (counts against uptime, surfaces in the cluster-degraded table) and the Solr card shows a writes failing badge. New fields u_server_errors, write_ok, write_detail on /api/solr, and a solr_history.u_server_errors column added via the standard _migrate() path.

v0.11.8 -- bundled patches (vfbquerycache + /api/version + mutation fix + probe logging)

Bundles three iterative patches from today: (a) vfbquerycache is moved from cache_services to solr_services (it's a Solr server, not an owl_cache), (b) error-string accumulation fixed on rancher_servers rows + new /api/version endpoint + run_checks bounded with asyncio.wait_for so a hung sub-probe can't freeze the scheduler, (c) every probe failure now writes an explicit WARNING log so failed attempts are visible in docker logs and not just in /api/status. See CHANGELOG.md.

v0.11.6 -- vfbquerycache moved to solr_services (vfb_json)

Same trap as v0.11.3: vfbquerycache.virtualflybrain.org is queryserver-vfbquery (1s348) running virtualflybrain/vfb-solr, not an owl_cache. Moved from cache_services to solr_services, core vfb_json. The only Solr of the four that takes writes. See CHANGELOG.md.

v0.11.5 -- swap Preview Solr for VFBquery cache

Drops Preview Solr (1s322) and adds VFBquery cache (vfbquerycache.virtualflybrain.org) to cache_services. LB-only for now -- the hostname has no LB rule yet and falls through to Hugo, so the card will show red until either the LB rule is added or the rancher: block is filled in. See CHANGELOG.md.

v0.11.4 -- Query Solr core fix (vfb_json)

Changes Query Solr's core from ontology to vfb_json. The rancher LB routes by path prefix independently of hostname, so the previous ontology setting was hitting the public ontology Solr (1s103) and double-counting its traffic onto the Query Solr card. See CHANGELOG.md.

v0.11.3 -- query-cache-server is Solr, not owl_cache

Drops the v0.11.2 cache_services entry (wrong shape -- probed /status) and re-adds the service correctly under solr_services as Preview Solr (core vfb_json, LB-fronted). See CHANGELOG.md.

v0.11.2 -- add query-cache-server to cache_services

Adds query-cache-server (1s322) to cache_services. It's the owl_cache fronting queryserver; per-container probe via the Rancher API hits the container's nginx on port 80 directly. See CHANGELOG.md.

v0.11.1 -- add Query Solr (query.virtualflybrain.org)

Adds the queryserver Solr (service 1s304) to solr_services. Carries the bulk of VFB Solr query traffic -- significantly busier than the public solr.virtualflybrain.org. Per-container probing via Rancher API; LB fallback at http://query.virtualflybrain.org. See CHANGELOG.md.

v0.11.0 -- Solr per-container monitoring

Per-container Solr probing for JVM memory, system load, and query/update rates. New solr_history table and Solr section on the page. See CHANGELOG.md.

v0.10.0 -- KB per-container probing + Neo4j latency chart

KB Neo4j now probed per cluster member (service 1s56, scale 2). New latency-over-time sparkline on every Neo4j card. Core user-facing group moved to the top of the page. See CHANGELOG.md.