From 390e1158dc7e08f42d35d5ee95837edba6f52270 Mon Sep 17 00:00:00 2001 From: Alex Moskowitz Date: Thu, 14 May 2026 12:12:24 -0400 Subject: [PATCH 1/7] fix(ontology): move :AnnexIII_Condition_1a and :AnnexIII_Condition_5b to governance extension Deduplicates universal regulatory content out of two per-fixture instance files into ARCO_governance_extension.ttl. Closes regulatory_alignment FAIL and traceability FAIL on the Adversarial Decoy and Blanknode Ghost fixtures by making the regulatory condition declarations visible to every fixture that imports the governance extension. Why Both audit queries (check_regulatory_alignment.sparql, check_assessment_ traceability.sparql) require the condition to be typed :RegulatoryContent in the merged graph. The conditions were declared only in ARCO_instances_sentinel.ttl (1(a)) and ARCO_instances_creditscoring.ttl (5(b)). Fixtures that don't import either file (Decoy, Ghost) referenced the conditions via iao:0000136 but the type assertion wasn't present, so both audit queries returned FAIL for fixture-distribution reasons, not fixture-semantics reasons. What changes - ARCO_governance_extension.ttl: new section "3a) REGULATORY CONTENT" adds :AnnexIII_List, :AnnexIII_Condition_1a, :AnnexIII_Condition_5b with the same triples previously in the two instance files. Triples preserved verbatim (rdfs:label, rdfs:comment, cco:prescribes, iao:0000136 targets). - ARCO_instances_sentinel.ttl: removes 10 lines (the declaration block); section header replaced with a brief migration comment. The three references via iao:0000136 :AnnexIII_Condition_1a are preserved. - ARCO_instances_creditscoring.ttl: removes 13 lines (the declaration block + the self-containedness comment); section header replaced with a brief migration comment. The three references via iao:0000136 :AnnexIII_Condition_5b are preserved. Tests (all 7 fixtures, pre vs post pipeline diff) - Sentinel, CreditScorer, VerificationKiosk: identical except entailed- triples count (+34 to +89 from the additional universal regulatory content now visible to every fixture). - DecoySystem_001: regulatory_alignment FAIL -> PASS; traceability FAIL -> PASS (closes the documented goal). - GhostSystem_001: regulatory_alignment FAIL -> PASS; traceability FAIL -> PASS; all_checks_passed false -> true. - FlagTest_BiometricSystem_WithDerogationClaim and FlagTest_CreditSystem_WithFraudProcess: no audit-row flip. Their :AssessmentDocumentation instances do not link to any regulatory condition via iao:0000136 in the source TTL, so the audit query's AssessmentDoc -> condition path is empty independent of where the condition is declared. The plan predicted these fixtures would flip; the actual cause is a separate fixture-authoring gap in ARCO_instances_flag_tests.ttl lines 90-92, 155-157. Closing that is a separate fixture edit outside this PR's scope. - Regression: test_gate_removal.py PASS; test_scenarios.py PASS (all 7 scenarios); test_kiosk_html_no_false_concretization.py PASS; test_output_provenance.py 1 failure (unchanged baseline). - HermiT vs OWL-RL cross-check: agree on every (fixture, system, query) tuple in the certificate-grade set. - SHACL conforms PASS on every fixture (unchanged). - No classification flip on any fixture; no SHACL change; no other audit-row change. Downstream consumer audit Grep across 03_TECHNICAL_CORE/, docs/, mcp/, .github/ for every reader of :AnnexIII_Condition_1a / _5b / _List / :RegulatoryContent. All reference sites either load ARCO_governance_extension.ttl (via every pipeline / test / cross-check loader) or are documentation mentions of the IRI itself. No consumer depends on the conditions being declared in a specific instance file. Deferred - :AnnexIII_Condition_1a_Exclusion in ARCO_instances_verification.ttl is a different class (verification-kiosk exclusion documentation per Recital 22 / Art 3(41)). Whether to also generalize the exclusion pattern is a separate future decision. - FlagTest fixtures' AssessmentDocs do not link to any regulatory condition; closing their regulatory_alignment FAIL is a separate fixture-authoring change. Revert git revert HEAD --- .../ontology/ARCO_governance_extension.ttl | 38 +++++++++++++++++++ .../ontology/ARCO_instances_creditscoring.ttl | 21 ++-------- .../ontology/ARCO_instances_sentinel.ttl | 16 ++------ 3 files changed, 46 insertions(+), 29 deletions(-) diff --git a/03_TECHNICAL_CORE/ontology/ARCO_governance_extension.ttl b/03_TECHNICAL_CORE/ontology/ARCO_governance_extension.ttl index 43e7fc1..57b17da 100644 --- a/03_TECHNICAL_CORE/ontology/ARCO_governance_extension.ttl +++ b/03_TECHNICAL_CORE/ontology/ARCO_governance_extension.ttl @@ -149,6 +149,44 @@ cco:Organization rdf:type owl:Class ; rdfs:label "Organization"@en ; rdfs:subClassOf bfo:0000027 . # Object Aggregate +################################################################# +# 3a) REGULATORY CONTENT — Annex III conditions (universal) +# +# These ICE instances describe what EU AI Act Regulation (EU) 2024/1689 +# Annex III prescribes for the modeled categories. They are universal +# regulatory content (one ICE per Annex III condition modeled), not +# per-fixture data. Every fixture references them via iao:0000136 +# from its :AssessmentDocumentation. +# +# Pattern: a regulatory condition ICE has type :RegulatoryContent, +# prescribes the regulated process type via cco:prescribes, and is_about +# the capability / process / role universals via iao:0000136. +# +# Moved from per-fixture files (Sentinel, CreditScoring) on 2026-05-14 +# to close regulatory_alignment FAIL on Adversarial and FlagTest fixtures. +################################################################# + +:AnnexIII_List rdf:type :RegulatoryContent ; + rdfs:label "Annex III List" ; + bfo:0000051 :AnnexIII_Condition_1a ; + bfo:0000051 :AnnexIII_Condition_5b . + +:AnnexIII_Condition_1a rdf:type :RegulatoryContent ; + rdfs:label "Annex III 1(a) (Biometric Rule)" ; + rdfs:comment "Annex III item 1(a): biometric identification of natural persons. cco:prescribes targets the regulated process TYPE (class IRI as concept-individual via OWL 2 punning) — the regulation prescribes process types, not deployment-specific tokens." ; + cco:prescribes :RemoteBiometricIdentificationProcess ; + iao:0000136 :BiometricIdentificationCapability ; + iao:0000136 :RemoteBiometricIdentificationProcess ; + iao:0000136 :NaturalPersonRole . + +:AnnexIII_Condition_5b rdf:type :RegulatoryContent ; + rdfs:label "Annex III 5(b) (Creditworthiness Rule)" ; + rdfs:comment "Annex III item 5(b): AI systems intended to evaluate the creditworthiness of natural persons or establish their credit score, with the exception of AI systems used for the purpose of detecting financial fraud. cco:prescribes targets the regulated process TYPE (class IRI as concept-individual via OWL 2 punning)." ; + cco:prescribes :CreditworthinessEvaluationProcess ; + iao:0000136 :CreditworthinessEvaluationCapability ; + iao:0000136 :CreditworthinessEvaluationProcess ; + iao:0000136 :NaturalPersonRole . + ################################################################# # 3b) REGULATORY BRIDGE AXIOMS # diff --git a/03_TECHNICAL_CORE/ontology/ARCO_instances_creditscoring.ttl b/03_TECHNICAL_CORE/ontology/ARCO_instances_creditscoring.ttl index ee9f700..6727edb 100644 --- a/03_TECHNICAL_CORE/ontology/ARCO_instances_creditscoring.ttl +++ b/03_TECHNICAL_CORE/ontology/ARCO_instances_creditscoring.ttl @@ -21,25 +21,12 @@ owl:imports . ################################################################# -# 1) REGULATORY LAYER — Annex III 5(b) +# 1) ANNEX III REGULATORY LAYER +# +# Annex III conditions moved to ARCO_governance_extension.ttl on 2026-05-14; +# references via iao:0000136 stay below. ################################################################# -# Mereological backbone (CLAUDE.md invariant 8): every modeled Annex III condition -# is `bfo:0000051` of `:AnnexIII_List`. The list is also re-asserted here with its -# rdf:type so this fixture is self-contained when loaded standalone (test_scenarios.py -# loads each fixture independently). Duplicate triples across fixtures are deduped -# at union time. See runs/loop/2026-05-09_beverley-research/audit_C_regulatory.md T2. -:AnnexIII_List rdf:type :RegulatoryContent ; - bfo:0000051 :AnnexIII_Condition_5b . - -:AnnexIII_Condition_5b rdf:type :RegulatoryContent ; - rdfs:label "Annex III 5(b) (Creditworthiness Rule)" ; - rdfs:comment "Annex III item 5(b): AI systems intended to be used to evaluate the creditworthiness of natural persons or establish their credit score, with the exception of AI systems used for the purpose of detecting financial fraud. cco:prescribes targets the regulated process TYPE (class IRI as concept-individual via OWL 2 punning) — the regulation prescribes process types, not deployment-specific tokens. This matches the Sentinel pattern and generalizes across multiple 5(b) assessments sharing this single regulatory ICE." ; - cco:prescribes :CreditworthinessEvaluationProcess ; - iao:0000136 :CreditworthinessEvaluationCapability ; - iao:0000136 :CreditworthinessEvaluationProcess ; - iao:0000136 :NaturalPersonRole . - ################################################################# # 2) SYSTEM LAYER (reality-side particulars) ################################################################# diff --git a/03_TECHNICAL_CORE/ontology/ARCO_instances_sentinel.ttl b/03_TECHNICAL_CORE/ontology/ARCO_instances_sentinel.ttl index 2e72cc2..650740f 100644 --- a/03_TECHNICAL_CORE/ontology/ARCO_instances_sentinel.ttl +++ b/03_TECHNICAL_CORE/ontology/ARCO_instances_sentinel.ttl @@ -13,20 +13,12 @@ owl:imports . ################################################################# -# 1) REGULATORY LAYER (ICE grounded to reality) +# 1) ANNEX III REGULATORY LAYER (mereological backbone) +# +# Annex III conditions moved to ARCO_governance_extension.ttl on 2026-05-14 +# as universal regulatory content; references via iao:0000136 stay below. ################################################################# -:AnnexIII_List rdf:type :RegulatoryContent ; - rdfs:label "Annex III List" ; - bfo:0000051 :AnnexIII_Condition_1a . # has part - -:AnnexIII_Condition_1a rdf:type :RegulatoryContent ; - rdfs:label "Annex III 1(a) (Biometric Rule)" ; - cco:prescribes :RemoteBiometricIdentificationProcess ; # directive: prescribes the regulated process type (Three D's — DirectiveICE → Process) - iao:0000136 :BiometricIdentificationCapability ; # is_about the capability universal - iao:0000136 :RemoteBiometricIdentificationProcess ; # is_about the regulated process type - iao:0000136 :NaturalPersonRole . # is_about the affected role - ################################################################# # 2) SYSTEM LAYER (reality-side particulars) - UPDATED ################################################################# From cffcf2e1c3993f271d3aef122d6e967580b4df03 Mon Sep 17 00:00:00 2001 From: Alex Moskowitz Date: Thu, 14 May 2026 13:29:21 -0400 Subject: [PATCH 2/7] docs(post-pr69): refresh fixture header and references after regulatory-content migration Three stale-doc fixes tied to the 2026-05-14 governance-extension move: - ARCO_instances_flag_tests.ttl header: replaces the pre-migration text ("classification PASS but audit FAIL ... minimal instances for flag testing only, without full regulatory content linkage") with the actual post-migration state. Classification and exception flag remain the test target; traceability and regulatory_alignment still FAIL but for a different reason now (local :AssessmentDocumentation -> :AnnexIII_Condition_* iao:0000136 link absent from this fixture, not fixture-distribution). - LIMITATIONS.md sec 9: file reference for the :AnnexIII_Condition_1a cco:prescribes :RemoteBiometricIdentificationProcess class-as-individual triple updated from ARCO_instances_sentinel.ttl to ARCO_governance_extension.ttl per the migration. Adds the 5(b) companion triple. Also notes that gate-removal coverage is now symmetric and adversarial- mechanism tests exist (next commit). - README.md "Gate independence is empirically verified" sentence: drops the "(Symmetric coverage for 5(b) is queued.)" parenthetical; corresponding row in the active-changes table moves from "Active work" to "Landed 2026-05-14". No pipeline behavior change. Co-Authored-By: Claude Opus 4.7 (1M context) --- .../ontology/ARCO_instances_flag_tests.ttl | 30 +++++++++++++++---- LIMITATIONS.md | 4 +-- README.md | 4 +-- 3 files changed, 29 insertions(+), 9 deletions(-) diff --git a/03_TECHNICAL_CORE/ontology/ARCO_instances_flag_tests.ttl b/03_TECHNICAL_CORE/ontology/ARCO_instances_flag_tests.ttl index 223c4a1..13c5451 100644 --- a/03_TECHNICAL_CORE/ontology/ARCO_instances_flag_tests.ttl +++ b/03_TECHNICAL_CORE/ontology/ARCO_instances_flag_tests.ttl @@ -11,11 +11,31 @@ rdfs:label "ARCO Flag Test Instances" ; rdfs:comment """Two test cases for the audit-layer exception flags. - NOTE: Running the full pipeline on these instances will show classification PASS - but audit FAIL (traceability and regulatory alignment). This is expected and correct — - these are minimal instances for flag testing only, without full regulatory content - linkage. The same pattern applies to ARCO_instances_adversarial_*.ttl. - The classification layer and flag behavior are the only results under test here. + TEST TARGET: simultaneous OWL classification + audit-layer flag detection + on the same system, demonstrating that classification and audit do not + bleed into each other. + + Expected audit-row outcomes (post 2026-05-14 migration of regulatory content + to ARCO_governance_extension.ttl): + - classification: PASS (all three Annex III gates satisfied) + - exception flag: FLAGGED (derogation or fraud, per fixture) + - traceability: FAIL + - regulatory_alignment: FAIL + The traceability and regulatory_alignment FAILs are NOT the test target. + They persist because the :AssessmentDocumentation instances below do not + carry an iao:0000136 link to :AnnexIII_Condition_1a or :AnnexIII_Condition_5b, + so the audit queries (which require ?doc iao:0000136 ?condition) return false. + This is a fixture-authoring gap, not a defect in the classification or flag + behavior under test. Adding those links would close the audit FAILs without + affecting the classification or flag entailments — separate change. + + Prior to the 2026-05-14 migration, the audit FAILs also covered fixture- + distribution effects (regulatory condition declarations were per-fixture + inside ARCO_instances_sentinel.ttl and ARCO_instances_creditscoring.ttl, so + the universal regulatory content was invisible to this fixture). The + migration moved those declarations to the governance extension, removing + the distribution issue; what remains is the local AssessmentDoc->condition + linkage gap described above. Test A — FlagTest_BiometricSystem_WithDerogationClaim: A system that IS classified as AnnexIII1aApplicableSystem (all three gates satisfied) diff --git a/LIMITATIONS.md b/LIMITATIONS.md index 4ad78cb..af3e666 100644 --- a/LIMITATIONS.md +++ b/LIMITATIONS.md @@ -303,9 +303,9 @@ The pipeline's output layer (`run_pipeline.py` from line 1699 onward, plus `writ ## 9. Engineering gaps -- **Negative test infrastructure is incomplete.** Gate-removal regression tests ([test_gate_removal.py](03_TECHNICAL_CORE/scripts/test_gate_removal.py)) verify each gate is independently necessary by mutating axioms and confirming entailment breaks. What does **not** exist is a parameterized test harness that loads an isolated deliberately-miscategorized instance file and runs the full pipeline against it, because the pipeline currently loads all TTL files into a single graph before reasoning — a negative-case file in the graph would contaminate positive cases. Building this harness is Step 2 of the ADR-001 work plan. Noted in [docs/agent/bfo_cco_alignment_audit.md](docs/agent/bfo_cco_alignment_audit.md) §"Unresolved Engineering Problem." +- **Negative test infrastructure is incomplete.** Gate-removal regression tests ([test_gate_removal.py](03_TECHNICAL_CORE/scripts/test_gate_removal.py)) verify each gate is independently necessary for both modeled Annex III categories (1(a) Sentinel and 5(b) CreditScorer) by mutating axioms and confirming entailment breaks. Symmetric coverage across the two categories was added 2026-05-14. Adversarial-mechanism tests ([test_adversarial_mechanism.py](03_TECHNICAL_CORE/scripts/test_adversarial_mechanism.py)) verify that DecoySystem_001 classifies via `owl:equivalentClass` (not direct IRI assertion) and that GhostSystem_001 classifies via blank-node `owl:someValuesFrom` (not a named individual). What does **not** exist is a parameterized test harness that loads an isolated deliberately-miscategorized instance file and runs the full pipeline against it, because the pipeline currently loads all TTL files into a single graph before reasoning — a negative-case file in the graph would contaminate positive cases. Building this harness is Step 2 of the ADR-001 work plan. Noted in [docs/agent/bfo_cco_alignment_audit.md](docs/agent/bfo_cco_alignment_audit.md) §"Unresolved Engineering Problem." - **Dependency pinning.** The pipeline is verified only on: `rdflib==7.6.0`, `pyshacl==0.31.0`, `owlrl==7.1.4`. Upgrading any of these requires re-running the full regression suite. The `owlrl` pin is especially load-bearing: Gate 2 and Gate 3 use anonymous inverse property restrictions whose entailment behavior could change across reasoner versions. -- **`AnnexIII_Condition_1a cco:prescribes :RemoteBiometricIdentificationProcess`** remains in [ARCO_instances_sentinel.ttl](03_TECHNICAL_CORE/ontology/ARCO_instances_sentinel.ttl) line 25 as a class-as-individual triple retained for regulatory traceability. It does not affect current classification. It is a known blocker for a future CCO import (see §4). +- **`AnnexIII_Condition_1a cco:prescribes :RemoteBiometricIdentificationProcess`** remains in [ARCO_governance_extension.ttl](03_TECHNICAL_CORE/ontology/ARCO_governance_extension.ttl) (moved 2026-05-14 from the Sentinel instance file to the governance extension as universal regulatory content) as a class-as-individual triple retained for regulatory traceability. The companion 5(b) triple `:AnnexIII_Condition_5b cco:prescribes :CreditworthinessEvaluationProcess` is in the same file. Neither affects current classification. Both are known blockers for a future CCO import (see §4). - **Single-reasoner portability.** The anonymous inverse property expressions in Gate 2 and Gate 3 equivalentClass axioms have been empirically verified on `owlrl==7.1.4` only. Other OWL-RL reasoners may or may not materialize these the same way. - **No automated tracking of regulatory amendments.** Annex III categories and Article 6(3) derogation criteria may change via EU delegated acts. Updates to the ontology are manual. diff --git a/README.md b/README.md index 642f368..0ae4ebc 100644 --- a/README.md +++ b/README.md @@ -128,7 +128,7 @@ The architecture grounds in BFO 2020 (ISO/IEC 21838-2:2021) and uses the seven-b **Layer separation is verified by fixtures.** Two flag-test fixtures present cases where all three Annex III gates are satisfied AND an audit-layer flag (a provider-asserted `:DerogationClaim`, or a `:FraudDetectionProcess` token) is also present. The OWL classification fires regardless of the audit flag; the flag fires alongside the classification. Classification and audit do not bleed into each other. -**Gate independence is empirically verified.** A regression test removes the supporting triples for each Annex III 1(a) gate in turn and confirms the classification fails. Each gate is independently necessary; removing any one breaks the entailment. (Symmetric coverage for 5(b) is queued.) +**Gate independence is empirically verified.** A regression test removes the supporting triples for each Annex III 1(a) and 5(b) gate in turn and confirms the classification fails. Each gate is independently necessary in both categories; removing any one breaks the entailment. Content-mutation variants (wrong process type, wrong designation target) verify that the gates check content, not just existence. **The certificate's classification binds to graph queries.** Classification field and evidence path are bound to SPARQL queries against the reasoned graph; the contract lives in `03_TECHNICAL_CORE/scripts/output_manifest_v2.yaml`, enforced by `test_output_provenance.py` (failing-by-design). Tightening provenance labels across surrounding fields is active work ([`OPEN_PROBLEMS.md L4.4-L4.6`](OPEN_PROBLEMS.md), [`LIMITATIONS.md §7.5`](LIMITATIONS.md)). @@ -151,7 +151,7 @@ The three-gate pattern (capability + intended use + affected role) generalizes b | Replacing the kiosk demo's hypothetical vendor packet with a real vendor document, so the demo runs on actual source evidence rather than hypothetical content | Active work | | Publishing the full reasoning output (around 20,000 derived facts per run) plus the second reasoner's separate result per fixture, so anyone can independently check both the conclusions and that two different reasoners agree | Active work | | Labeling every certificate field with where its value came from (a graph query result, the run's metadata, or a scope-disclosure note) and adding a CI check that verifies every field traces back to its declared source | Active work | -| Extending the test that confirms each Gate is necessary from Annex III 1(a) to also cover 5(b), so both classifications have the same proof that none of the three gates is decorative | Active work | +| Extending the test that confirms each Gate is necessary from Annex III 1(a) to also cover 5(b), so both classifications have the same proof that none of the three gates is decorative | Landed 2026-05-14 | Day-to-day rows live in `OPEN_PROBLEMS.md` (internal); the public roadmap with verified core, resolved modeling decisions, and execution sequence is at [`docs/MODELING_ROADMAP.md`](docs/MODELING_ROADMAP.md). From d74e76a10d1301a2f13d8a21d2c2fce1543bb00a Mon Sep 17 00:00:00 2001 From: Alex Moskowitz Date: Thu, 14 May 2026 13:29:40 -0400 Subject: [PATCH 3/7] test(coverage): add 5(b) gate-removal symmetry and adversarial-mechanism assertions Two coverage gaps closed against README claims: - test_gate_removal.py: parameterized over both modeled Annex III categories. CATEGORY_1A (Sentinel) preserves the original 7 tests (5 gate removals + 2 content mutations) verbatim by triple; CATEGORY_5B (CreditScorer) adds the symmetric 7 against AnnexIII5bApplicableSystem. README "Gate independence is empirically verified" previously disclosed the 5(b) gap as queued; closes that. - test_adversarial_mechanism.py (new): asserts that DecoySystem_001's Gate 1 entailment routes through owl:equivalentClass propagation (the disposition is typed only as :WeirdScanner pre-reasoning; :BiometricIdentificationCapability is absent from the asserted triples and entailed post-reasoning), and that GhostSystem_001's disposition is a blank node (no named individual) that still satisfies owl:someValuesFrom. test_scenarios.py asserts the entailment fires; this test asserts HOW. - .github/workflows/arco-smoke-test.yml and arco-demo.yml: both workflows run the new test alongside the existing three regression tests. Pipeline behavior unchanged. test_output_provenance.py failure count unchanged at 1 (baseline). Co-Authored-By: Claude Opus 4.7 (1M context) --- .github/workflows/arco-demo.yml | 11 +- .github/workflows/arco-smoke-test.yml | 3 + .../scripts/test_adversarial_mechanism.py | 237 ++++++++++++ .../scripts/test_gate_removal.py | 361 ++++++++++++------ 4 files changed, 484 insertions(+), 128 deletions(-) create mode 100644 03_TECHNICAL_CORE/scripts/test_adversarial_mechanism.py diff --git a/.github/workflows/arco-demo.yml b/.github/workflows/arco-demo.yml index ff74464..676909c 100644 --- a/.github/workflows/arco-demo.yml +++ b/.github/workflows/arco-demo.yml @@ -7,10 +7,11 @@ name: ARCO Demo Run # # What this gates (any failure fails the build): # - "ALL CHECKS PASSED" signal in the pipeline output. -# - Three regression test scripts return 0: -# * test_gate_removal.py (each Annex III 1(a) gate is independently necessary) +# - Four regression test scripts return 0: +# * test_gate_removal.py (each Annex III 1(a) and 5(b) gate is independently necessary) # * test_scenarios.py (multi-scenario classification correctness) # * test_kiosk_html_no_false_concretization.py (L4.7 regression) +# * test_adversarial_mechanism.py (decoy and ghost classification mechanism) # - Five expected artifact files exist in runs/demo/: # certificate.txt, summary.json, evidence.json, shacl_report.txt, # determination_view.html. @@ -95,6 +96,12 @@ jobs: set -euo pipefail python -u 03_TECHNICAL_CORE/scripts/test_kiosk_html_no_false_concretization.py + - name: Run adversarial-mechanism regression tests + shell: bash + run: | + set -euo pipefail + python -u 03_TECHNICAL_CORE/scripts/test_adversarial_mechanism.py + - name: Verify artifact files exist shell: bash run: | diff --git a/.github/workflows/arco-smoke-test.yml b/.github/workflows/arco-smoke-test.yml index 213e254..a98b296 100644 --- a/.github/workflows/arco-smoke-test.yml +++ b/.github/workflows/arco-smoke-test.yml @@ -46,3 +46,6 @@ jobs: - name: Run kiosk HTML no-false-concretization regression test (L4.7) run: python 03_TECHNICAL_CORE/scripts/test_kiosk_html_no_false_concretization.py + + - name: Run adversarial-mechanism regression tests + run: python 03_TECHNICAL_CORE/scripts/test_adversarial_mechanism.py diff --git a/03_TECHNICAL_CORE/scripts/test_adversarial_mechanism.py b/03_TECHNICAL_CORE/scripts/test_adversarial_mechanism.py new file mode 100644 index 0000000..4ce05c6 --- /dev/null +++ b/03_TECHNICAL_CORE/scripts/test_adversarial_mechanism.py @@ -0,0 +1,237 @@ +""" +Adversarial-mechanism regression tests. + +`test_scenarios.py` already asserts that DecoySystem_001 and GhostSystem_001 +classify correctly under OWL-RL. This file verifies HOW the classification +fires, not just that it fires. Each assertion catches a failure mode that +would let a pattern-matching pipeline pass test_scenarios.py with the wrong +mechanism in place. + + Decoy fixture (ARCO_instances_adversarial_decoy.ttl): + The disposition is typed only as :WeirdScanner. :WeirdScanner is declared + owl:equivalentClass :BiometricIdentificationCapability in the same fixture. + Pre-reasoning, no asserted triple types the disposition as + :BiometricIdentificationCapability. Post-reasoning under OWL-RL, + the equivalentClass propagation entails the typing and Gate 1 fires. + + If a pipeline did IRI-name pattern matching, it would see no + :BiometricIdentificationCapability triple in the input and miss the + classification. The OWL reasoner does not. + + Ghost fixture (ARCO_instances_adversarial_blanknode.ttl): + The disposition is an anonymous individual (blank node) typed as + :BiometricIdentificationCapability. owl:someValuesFrom requires + existence of an instance satisfying the restriction; it does not + require a named individual. + + If a pipeline required named individuals for evidence walks, it would + miss this case. The OWL reasoner does not require names; existential + quantification is satisfied by any witness, including blank nodes. + +Run from repo root: + python 03_TECHNICAL_CORE/scripts/test_adversarial_mechanism.py +""" + +from __future__ import annotations + +import sys +from pathlib import Path +from rdflib import Graph, Namespace, URIRef +from rdflib.term import BNode + +try: + import owlrl +except ImportError: + print("ERROR: owlrl is required. Install: pip install owlrl") + sys.exit(1) + +REPO_ROOT = Path(__file__).resolve().parents[2] +ONTOLOGY_DIR = REPO_ROOT / "03_TECHNICAL_CORE" / "ontology" + +BFO_2020 = ONTOLOGY_DIR / "imports" / "bfo-2020.owl" +IAO_BOT = ONTOLOGY_DIR / "imports" / "iao_bot.owl" +RO_BOT = ONTOLOGY_DIR / "imports" / "ro_bot.owl" +CCO_BOT = ONTOLOGY_DIR / "imports" / "cco_bot.owl" +CORE = ONTOLOGY_DIR / "ARCO_core.ttl" +GOV = ONTOLOGY_DIR / "ARCO_governance_extension.ttl" + +ARCO = Namespace("https://arco.ai/ontology/core#") +RDF = Namespace("http://www.w3.org/1999/02/22-rdf-syntax-ns#") +RO = Namespace("http://purl.obolibrary.org/obo/RO_") + + +def parse_fixture(instances_path: Path) -> Graph: + """Parse the full ontology + fixture WITHOUT running the reasoner.""" + g = Graph() + for p in (BFO_2020, IAO_BOT, RO_BOT, CCO_BOT, CORE, GOV, instances_path): + if not p.exists(): + raise FileNotFoundError(f"Missing: {p}") + fmt = "xml" if p.suffix == ".owl" else "turtle" + g.parse(p.as_posix(), format=fmt) + return g + + +def reason(g: Graph) -> Graph: + owlrl.DeductiveClosure(owlrl.OWLRL_Semantics).expand(g) + return g + + +def test_decoy_classifies_via_equivalent_class() -> bool: + """ + Verify that DecoySystem_001's Gate 1 entailment fires via + owl:equivalentClass propagation, not via direct IRI assertion. + + Pre-reasoning expectation: + :Decoy_Disposition rdf:type :WeirdScanner present + :Decoy_Disposition rdf:type :BiometricIdentificationCapability ABSENT + + Post-reasoning expectation: + :Decoy_Disposition rdf:type :BiometricIdentificationCapability present + :DecoySystem_001 rdf:type :AnnexIII1aApplicableSystem present + """ + print("\n--- DECOY: classification via owl:equivalentClass ---") + fixture = ONTOLOGY_DIR / "ARCO_instances_adversarial_decoy.ttl" + g_pre = parse_fixture(fixture) + + decoy_disp = ARCO["Decoy_Disposition"] + weird_scanner = ARCO["WeirdScanner"] + bio_cap = ARCO["BiometricIdentificationCapability"] + decoy_system = ARCO["DecoySystem_001"] + annex_1a = ARCO["AnnexIII1aApplicableSystem"] + + # Precondition 1: disposition is typed as :WeirdScanner (the decoy class) + has_weird_pre = (decoy_disp, RDF["type"], weird_scanner) in g_pre + # Precondition 2: disposition is NOT directly typed as + # :BiometricIdentificationCapability before reasoning + has_bio_pre = (decoy_disp, RDF["type"], bio_cap) in g_pre + # Precondition 3: System is NOT yet classified as 1(a) applicable + is_annex_pre = (decoy_system, RDF["type"], annex_1a) in g_pre + + ok = True + print(f" pre-reasoning :Decoy_Disposition rdf:type :WeirdScanner: {has_weird_pre} (expected True)") + if not has_weird_pre: + ok = False + print(f" pre-reasoning :Decoy_Disposition rdf:type :BiometricIdentificationCapability: {has_bio_pre} (expected False)") + if has_bio_pre: + print(" FAIL: decoy disposition asserted directly as BiometricIdentificationCapability; " + "test premise (equivalence-only routing) is invalid.") + ok = False + print(f" pre-reasoning :DecoySystem_001 rdf:type :AnnexIII1aApplicableSystem: {is_annex_pre} (expected False)") + if is_annex_pre: + ok = False + + # Reason and re-check + g_post = reason(g_pre) + has_bio_post = (decoy_disp, RDF["type"], bio_cap) in g_post + is_annex_post = (decoy_system, RDF["type"], annex_1a) in g_post + + print(f" post-reasoning :Decoy_Disposition rdf:type :BiometricIdentificationCapability: {has_bio_post} (expected True)") + if not has_bio_post: + ok = False + print(f" post-reasoning :DecoySystem_001 rdf:type :AnnexIII1aApplicableSystem: {is_annex_post} (expected True)") + if not is_annex_post: + ok = False + + if ok: + print(" RESULT: classification routed via owl:equivalentClass, not via direct IRI.") + return ok + + +def test_ghost_classifies_via_blank_node_disposition() -> bool: + """ + Verify that GhostSystem_001's Gate 1 entailment fires via a blank-node + disposition satisfying owl:someValuesFrom, not via a named individual. + + Pre-reasoning expectation: + :Ghost_Module ro:0000091 _:b where _:b is a blank node typed + as :BiometricIdentificationCapability + + Post-reasoning expectation: + :GhostSystem_001 rdf:type :AnnexIII1aApplicableSystem present + """ + print("\n--- GHOST: classification via blank-node owl:someValuesFrom ---") + fixture = ONTOLOGY_DIR / "ARCO_instances_adversarial_blanknode.ttl" + g_pre = parse_fixture(fixture) + + ghost_module = ARCO["Ghost_Module"] + has_disposition = RO["0000091"] + bio_cap = ARCO["BiometricIdentificationCapability"] + ghost_system = ARCO["GhostSystem_001"] + annex_1a = ARCO["AnnexIII1aApplicableSystem"] + + # Find disposition objects of :Ghost_Module via ro:0000091 + disposition_objs = list(g_pre.objects(ghost_module, has_disposition)) + + ok = True + print(f" :Ghost_Module ro:0000091 ? -> {len(disposition_objs)} disposition object(s)") + if not disposition_objs: + print(" FAIL: no disposition object found") + return False + + # The disposition must be a blank node, not a named IRI + blank_node_dispositions = [d for d in disposition_objs if isinstance(d, BNode)] + named_dispositions = [d for d in disposition_objs if not isinstance(d, BNode)] + + print(f" blank-node dispositions: {len(blank_node_dispositions)} (expected >= 1)") + print(f" named dispositions: {len(named_dispositions)} (expected 0 for this fixture's test target)") + if not blank_node_dispositions: + print(" FAIL: no blank-node disposition; test premise (anonymous individual) is invalid.") + ok = False + if named_dispositions: + print(" FAIL: a named disposition is present; test premise (anonymous individual) is invalid.") + ok = False + + # The blank-node disposition must be typed as :BiometricIdentificationCapability + if blank_node_dispositions: + bn = blank_node_dispositions[0] + bn_typed = (bn, RDF["type"], bio_cap) in g_pre + print(f" blank-node typed as :BiometricIdentificationCapability: {bn_typed} (expected True)") + if not bn_typed: + ok = False + + # Pre-reasoning: system is not yet classified + is_annex_pre = (ghost_system, RDF["type"], annex_1a) in g_pre + print(f" pre-reasoning :GhostSystem_001 rdf:type :AnnexIII1aApplicableSystem: {is_annex_pre} (expected False)") + if is_annex_pre: + ok = False + + # Reason and re-check + g_post = reason(g_pre) + is_annex_post = (ghost_system, RDF["type"], annex_1a) in g_post + print(f" post-reasoning :GhostSystem_001 rdf:type :AnnexIII1aApplicableSystem: {is_annex_post} (expected True)") + if not is_annex_post: + ok = False + + if ok: + print(" RESULT: classification satisfied by blank-node owl:someValuesFrom witness.") + return ok + + +def main() -> int: + print("=" * 72) + print("ARCO ADVERSARIAL-MECHANISM TEST") + print("=" * 72) + + all_pass = True + if not test_decoy_classifies_via_equivalent_class(): + all_pass = False + if not test_ghost_classifies_via_blank_node_disposition(): + all_pass = False + + print() + print("=" * 72) + if all_pass: + print("ALL ADVERSARIAL-MECHANISM TESTS PASSED") + print("Decoy classification routes via owl:equivalentClass.") + print("Ghost classification routes via blank-node owl:someValuesFrom witness.") + print("Neither relies on a direct IRI assertion or a named individual.") + else: + print("SOME ADVERSARIAL-MECHANISM TESTS FAILED") + print("=" * 72) + return 1 + print("=" * 72) + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/03_TECHNICAL_CORE/scripts/test_gate_removal.py b/03_TECHNICAL_CORE/scripts/test_gate_removal.py index 7f409db..ddc70f5 100644 --- a/03_TECHNICAL_CORE/scripts/test_gate_removal.py +++ b/03_TECHNICAL_CORE/scripts/test_gate_removal.py @@ -1,9 +1,13 @@ """ -Gate-removal regression test for the three-gate Annex III 1(a) classification. +Gate-removal regression test for the three-gate Annex III classification. -Removes each gate's key triple independently, re-reasons, and confirms -AnnexIII1aApplicableSystem disappears. Also verifies that HighRiskSystem -(capability-only axiom) is unaffected by gates 2 and 3. +For each modeled Annex III category (1(a) and 5(b)), removes each gate's key +triple independently, re-reasons, and confirms the category's applicable-system +class disappears. Also verifies that HighRiskSystem (capability-only axiom) is +unaffected by gates 2 and 3. + +Symmetric coverage across categories verifies that the three-gate pattern +generalizes; no gate in either category is decorative. """ from __future__ import annotations @@ -23,7 +27,6 @@ CORE = ONTOLOGY_DIR / "ARCO_core.ttl" GOV = ONTOLOGY_DIR / "ARCO_governance_extension.ttl" -INSTANCES = ONTOLOGY_DIR / "ARCO_instances_sentinel.ttl" ARCO = Namespace("https://arco.ai/ontology/core#") RDF = Namespace("http://www.w3.org/1999/02/22-rdf-syntax-ns#") @@ -31,106 +34,190 @@ RO = Namespace("http://purl.obolibrary.org/obo/RO_") CCO = Namespace("http://www.ontologyrepository.com/CommonCoreOntologies/") -# The triple removals that knock out each gate -GATE_REMOVALS = { - "gate1_capability": ( - ARCO["Sentinel_FaceID_Module"], - RO["0000091"], # has_disposition - ARCO["Sentinel_FaceID_Disposition"], - ), - "gate2_intended_use": ( - ARCO["Sentinel_IntendedUse_001"], - IAO["0000136"], # is_about - ARCO["Sentinel_ID_System"], - ), - "gate3_use_scenario": ( - ARCO["Sentinel_UseScenario_001"], - IAO["0000136"], # is_about - ARCO["Sentinel_ID_System"], - ), - # Content-based gate failures: - # Gate 2 must require cco:prescribes an instance of the regulated process class, not just - # existence of IUS. Triple references the token individual, not the class IRI. - "gate2_prescribes_removed": ( - ARCO["Sentinel_IntendedUse_001"], - CCO["prescribes"], # cco:prescribes - ARCO["Sentinel_RBIP_Process"], # the typed process token (not the class IRI) - ), - # Gate 3 must require cco:designates :NaturalPersonRole. The use scenario - # spec designates the affected role universal (class-level designation via - # the typed CCO designation property; same shape as Gate 2 with cco:prescribes). - "gate3_missing_role": ( - ARCO["Sentinel_UseScenario_001"], - CCO["designates"], - ARCO["NaturalPersonRole"], # the role universal as designation target - ), -} - -# Expected entailment results after each gate removal -EXPECTED = { - "gate1_capability": { - "AnnexIII1aApplicableSystem": False, - "HighRiskSystem": False, # capability-only axiom also breaks - }, - "gate2_intended_use": { - "AnnexIII1aApplicableSystem": False, - "HighRiskSystem": True, # HighRiskSystem only needs capability - }, - "gate3_use_scenario": { - "AnnexIII1aApplicableSystem": False, - "HighRiskSystem": True, # HighRiskSystem only needs capability - }, - "gate2_prescribes_removed": { - "AnnexIII1aApplicableSystem": False, # prescribes removed -> Gate 2 fails - "HighRiskSystem": True, # capability unchanged - }, - "gate3_missing_role": { - "AnnexIII1aApplicableSystem": False, # NaturalPersonRole removed -> Gate 3 fails - "HighRiskSystem": True, # capability unchanged - }, -} - -# Mutation tests: remove one triple and add a replacement (wrong value). -# These verify that wrong content, not just absence, also breaks the gate. -GATE_MUTATIONS = { - "gate2_wrong_process_type": { - "remove": ( +# --------------------------------------------------------------------------- +# Annex III 1(a) — Sentinel fixture +# --------------------------------------------------------------------------- +CATEGORY_1A = { + "label": "Annex III 1(a) — Sentinel (Biometric Identification)", + "instances": ONTOLOGY_DIR / "ARCO_instances_sentinel.ttl", + "system": ARCO["Sentinel_ID_System"], + "applicable_class": ARCO["AnnexIII1aApplicableSystem"], + "gate_removals": { + "gate1_capability": ( + ARCO["Sentinel_FaceID_Module"], + RO["0000091"], # has_disposition + ARCO["Sentinel_FaceID_Disposition"], + ), + "gate2_intended_use": ( ARCO["Sentinel_IntendedUse_001"], - CCO["prescribes"], - ARCO["Sentinel_RBIP_Process"], # the typed process token + IAO["0000136"], # is_about + ARCO["Sentinel_ID_System"], ), - "add": ( + "gate3_use_scenario": ( + ARCO["Sentinel_UseScenario_001"], + IAO["0000136"], # is_about + ARCO["Sentinel_ID_System"], + ), + # Content-based gate failures + "gate2_prescribes_removed": ( ARCO["Sentinel_IntendedUse_001"], CCO["prescribes"], - ARCO["SomeOtherProcess"], # wrong process type + ARCO["Sentinel_RBIP_Process"], # the typed process token ), - "expected": { - "AnnexIII1aApplicableSystem": False, # Gate 2 fails: wrong process - "HighRiskSystem": True, # capability unchanged - }, - }, - "gate3_wrong_designation_target": { - "remove": ( + "gate3_missing_role": ( ARCO["Sentinel_UseScenario_001"], CCO["designates"], - ARCO["NaturalPersonRole"], # the regulated role universal + ARCO["NaturalPersonRole"], ), - "add": ( - ARCO["Sentinel_UseScenario_001"], + }, + "expected": { + "gate1_capability": { + "applicable": False, + "HighRiskSystem": False, # capability-only axiom also breaks + }, + "gate2_intended_use": { + "applicable": False, + "HighRiskSystem": True, # HighRiskSystem only needs capability + }, + "gate3_use_scenario": { + "applicable": False, + "HighRiskSystem": True, + }, + "gate2_prescribes_removed": { + "applicable": False, + "HighRiskSystem": True, + }, + "gate3_missing_role": { + "applicable": False, + "HighRiskSystem": True, + }, + }, + "gate_mutations": { + "gate2_wrong_process_type": { + "remove": ( + ARCO["Sentinel_IntendedUse_001"], + CCO["prescribes"], + ARCO["Sentinel_RBIP_Process"], + ), + "add": ( + ARCO["Sentinel_IntendedUse_001"], + CCO["prescribes"], + ARCO["SomeOtherProcess"], + ), + "expected": {"applicable": False, "HighRiskSystem": True}, + }, + "gate3_wrong_designation_target": { + "remove": ( + ARCO["Sentinel_UseScenario_001"], + CCO["designates"], + ARCO["NaturalPersonRole"], + ), + "add": ( + ARCO["Sentinel_UseScenario_001"], + CCO["designates"], + ARCO["SomeOtherRole"], + ), + "expected": {"applicable": False, "HighRiskSystem": True}, + }, + }, +} + +# --------------------------------------------------------------------------- +# Annex III 5(b) — CreditScorer fixture +# Symmetric coverage with the 1(a) block above. Each gate-removal verifies a +# gate that is independently necessary for AnnexIII5bApplicableSystem. +# --------------------------------------------------------------------------- +CATEGORY_5B = { + "label": "Annex III 5(b) — CreditScorer (Creditworthiness Evaluation)", + "instances": ONTOLOGY_DIR / "ARCO_instances_creditscoring.ttl", + "system": ARCO["CreditScorer_001"], + "applicable_class": ARCO["AnnexIII5bApplicableSystem"], + "gate_removals": { + "gate1_capability": ( + ARCO["CreditScorer_Processing_Module"], + RO["0000091"], # has_disposition + ARCO["CreditScorer_Eval_Disposition"], + ), + "gate2_intended_use": ( + ARCO["CreditScorer_IntendedUse_001"], + IAO["0000136"], # is_about + ARCO["CreditScorer_001"], + ), + "gate3_use_scenario": ( + ARCO["CreditScorer_UseScenario_001"], + IAO["0000136"], # is_about + ARCO["CreditScorer_001"], + ), + # Content-based gate failures + "gate2_prescribes_removed": ( + ARCO["CreditScorer_IntendedUse_001"], + CCO["prescribes"], + ARCO["CreditScorer_EvalProcess_Token"], + ), + "gate3_missing_role": ( + ARCO["CreditScorer_UseScenario_001"], CCO["designates"], - ARCO["SomeOtherRole"], # wrong target (not the regulated role) + ARCO["NaturalPersonRole"], ), - "expected": { - "AnnexIII1aApplicableSystem": False, # Gate 3 fails: wrong designation target - "HighRiskSystem": True, # capability unchanged + }, + "expected": { + "gate1_capability": { + "applicable": False, + "HighRiskSystem": False, + }, + "gate2_intended_use": { + "applicable": False, + "HighRiskSystem": True, + }, + "gate3_use_scenario": { + "applicable": False, + "HighRiskSystem": True, + }, + "gate2_prescribes_removed": { + "applicable": False, + "HighRiskSystem": True, + }, + "gate3_missing_role": { + "applicable": False, + "HighRiskSystem": True, + }, + }, + "gate_mutations": { + "gate2_wrong_process_type": { + "remove": ( + ARCO["CreditScorer_IntendedUse_001"], + CCO["prescribes"], + ARCO["CreditScorer_EvalProcess_Token"], + ), + "add": ( + ARCO["CreditScorer_IntendedUse_001"], + CCO["prescribes"], + ARCO["SomeOtherProcess"], + ), + "expected": {"applicable": False, "HighRiskSystem": True}, + }, + "gate3_wrong_designation_target": { + "remove": ( + ARCO["CreditScorer_UseScenario_001"], + CCO["designates"], + ARCO["NaturalPersonRole"], + ), + "add": ( + ARCO["CreditScorer_UseScenario_001"], + CCO["designates"], + ARCO["SomeOtherRole"], + ), + "expected": {"applicable": False, "HighRiskSystem": True}, }, }, } +CATEGORIES = [CATEGORY_1A, CATEGORY_5B] -def load_graph() -> Graph: + +def load_graph(instances_path: Path) -> Graph: g = Graph() - for p in (CORE, GOV, INSTANCES): + for p in (CORE, GOV, instances_path): if not p.exists(): raise FileNotFoundError(f"Missing: {p}") g.parse(p.as_posix(), format="turtle") @@ -146,9 +233,9 @@ def check_type(g: Graph, individual: URIRef, cls: URIRef) -> bool: return (individual, RDF["type"], cls) in g -def run_mutation_test(gate_name: str, mutation: dict) -> dict: +def run_mutation_test(category: dict, gate_name: str, mutation: dict) -> dict: """Remove one triple, add a replacement with wrong content, reason, check entailments.""" - g = load_graph() + g = load_graph(category["instances"]) remove_triple = mutation["remove"] add_triple = mutation["add"] @@ -160,86 +247,93 @@ def run_mutation_test(gate_name: str, mutation: dict) -> dict: g.add(add_triple) reason(g) - system = ARCO["Sentinel_ID_System"] + system = category["system"] return { "gate": gate_name, "mutated": f"replaced <{o}> with <{add_triple[2]}>", - "AnnexIII1aApplicableSystem": check_type(g, system, ARCO["AnnexIII1aApplicableSystem"]), + "applicable": check_type(g, system, category["applicable_class"]), "HighRiskSystem": check_type(g, system, ARCO["HighRiskSystem"]), } -def run_test(gate_name: str, triple_to_remove: tuple) -> dict: +def run_test(category: dict, gate_name: str, triple_to_remove: tuple) -> dict: """Remove one triple, reason, check entailments.""" - g = load_graph() + g = load_graph(category["instances"]) s, p, o = triple_to_remove - # Verify the triple exists before removing if (s, p, o) not in g: return {"gate": gate_name, "error": f"Triple not found: ({s}, {p}, {o})"} g.remove((s, p, o)) reason(g) - system = ARCO["Sentinel_ID_System"] - results = { + system = category["system"] + return { "gate": gate_name, "removed": f"<{s}> <{p}> <{o}>", - "AnnexIII1aApplicableSystem": check_type(g, system, ARCO["AnnexIII1aApplicableSystem"]), + "applicable": check_type(g, system, category["applicable_class"]), "HighRiskSystem": check_type(g, system, ARCO["HighRiskSystem"]), } - return results -def main() -> None: - print("=" * 72) - print("ARCO GATE-REMOVAL REGRESSION TEST") +def run_category(category: dict) -> bool: + """Run baseline + gate-removal + mutation tests for one Annex III category. + + Returns True iff every assertion in this category passes. + """ + label = category["label"] + applicable_name = category["applicable_class"].split("#")[-1] + + print("\n" + "=" * 72) + print(label) print("=" * 72) - # Baseline: full graph should have both entailments + # Baseline print("\n--- BASELINE (all gates present) ---") - g_full = load_graph() + g_full = load_graph(category["instances"]) initial = len(g_full) reason(g_full) - system = ARCO["Sentinel_ID_System"] + system = category["system"] - annex_ok = check_type(g_full, system, ARCO["AnnexIII1aApplicableSystem"]) + applicable_ok = check_type(g_full, system, category["applicable_class"]) hr_ok = check_type(g_full, system, ARCO["HighRiskSystem"]) print(f" Triples: {initial} -> {len(g_full)}") - print(f" AnnexIII1aApplicableSystem: {annex_ok}") + print(f" {applicable_name}: {applicable_ok}") print(f" HighRiskSystem: {hr_ok}") - if not annex_ok or not hr_ok: + if not applicable_ok or not hr_ok: print("\nFAIL: Baseline entailments missing. Cannot test gate removal.") - sys.exit(1) + return False print(" Baseline: OK") - # Gate removal tests all_pass = True - for gate_name, triple in GATE_REMOVALS.items(): + + # Gate removal tests + for gate_name, triple in category["gate_removals"].items(): print(f"\n--- {gate_name.upper()} ---") - result = run_test(gate_name, triple) + result = run_test(category, gate_name, triple) if "error" in result: print(f" ERROR: {result['error']}") all_pass = False continue - expected = EXPECTED[gate_name] - for cls_name, expected_val in expected.items(): - actual = result[cls_name] + expected = category["expected"][gate_name] + for key, expected_val in expected.items(): + cls_label = applicable_name if key == "applicable" else key + actual = result[key] status = "OK" if actual == expected_val else "FAIL" if status == "FAIL": all_pass = False - print(f" {cls_name}: {actual} (expected {expected_val}) [{status}]") + print(f" {cls_label}: {actual} (expected {expected_val}) [{status}]") - # Mutation tests: wrong content (not just absence) also breaks the gate + # Mutation tests print("\n--- CONTENT-MUTATION TESTS ---") - for gate_name, mutation in GATE_MUTATIONS.items(): + for gate_name, mutation in category["gate_mutations"].items(): print(f"\n--- {gate_name.upper()} ---") - result = run_mutation_test(gate_name, mutation) + result = run_mutation_test(category, gate_name, mutation) if "error" in result: print(f" ERROR: {result['error']}") @@ -248,19 +342,34 @@ def main() -> None: print(f" Mutation: {result['mutated']}") expected = mutation["expected"] - for cls_name, expected_val in expected.items(): - actual = result[cls_name] + for key, expected_val in expected.items(): + cls_label = applicable_name if key == "applicable" else key + actual = result[key] status = "OK" if actual == expected_val else "FAIL" if status == "FAIL": all_pass = False - print(f" {cls_name}: {actual} (expected {expected_val}) [{status}]") + print(f" {cls_label}: {actual} (expected {expected_val}) [{status}]") + + return all_pass + + +def main() -> None: + print("=" * 72) + print("ARCO GATE-REMOVAL REGRESSION TEST") + print("=" * 72) + + all_pass = True + for category in CATEGORIES: + if not run_category(category): + all_pass = False - # Summary print("\n" + "=" * 72) if all_pass: print("ALL GATE-REMOVAL TESTS PASSED") - print("Each gate is independently necessary for AnnexIII1aApplicableSystem.") - print("Wrong content (not just absence) also breaks the gate.") + print("Each gate is independently necessary for AnnexIII1aApplicableSystem") + print("and AnnexIII5bApplicableSystem. Wrong content (not just absence) also") + print("breaks the gate. Coverage is symmetric across the two modeled Annex III") + print("categories.") else: print("SOME GATE-REMOVAL TESTS FAILED") sys.exit(1) From 01380217e5f737f15063965b6641d0d374a886c2 Mon Sep 17 00:00:00 2001 From: Amosk21 Date: Thu, 14 May 2026 22:08:51 -0400 Subject: [PATCH 4/7] fix(citations): correct EU AI Act anchors for verification carve-out (#70) Two regulatory citation defects across 6 tracked files (11 specific text edits). No schema, axiom, class, or property changes. Pipeline behavior byte-identical: ALL CHECKS PASSED, exit 0. Defect 1 (Recital 22 miscitation, 7 locations): Recital 22 of Regulation (EU) 2024/1689 governs extraterritorial scope (third-country operators), not the biometric verification carve-out. The correct anchor chain is Recital 15 (definition + carve-out), Recital 17 (RBI-specific carve-out + rationale), and Annex III item 1(a) operative clause. Fixed in: - 03_TECHNICAL_CORE/ontology/ARCO_instances_verification.ttl (lines 18, 27): kiosk fixture rdfs:comment surfaced via select_system_comment.sparql into the negative-case certificate panel - 03_TECHNICAL_CORE/scripts/run_pipeline.py (lines 445, 1243): docstring + inline comment describing the design pattern - 03_TECHNICAL_CORE/reasoning/select_system_comment.sparql (line 6): SPARQL header comment - docs/MODELING_ROADMAP.md (line 15): public-facing modeling narrative Defect 2 (Article 3(36) "intended to be used" misattribution, 4 locations in tracked files): Article 3(36) is the technical 1:1 verification definition; it does not contain the phrase "intended to be used." That framing appears verbatim in Recital 15, Recital 17, and Annex III item 1(a). Fixed in: - docs/MODELING_ROADMAP.md (line 68) - docs/kiosk_demo_v1/kiosk_demo.md (lines 65, 172) - LIMITATIONS.md (line 114) Defect 3 (Article 3(43) paraphrase fidelity, 1 location): Article 3(43) verbatim uses "system" (BFO Bucket 1) where ARCO's :PostRemoteBiometricIdentificationProcess class uses "process" (BFO Bucket 4). Light fix in LIMITATIONS.md:141 adds the verbatim regulatory text and explicitly flags the "process" framing as ARCO's modeling translation. The corresponding TTL skos:definition at ARCO_governance_extension.ttl:336 deliberately not modified: the existing rdfs:comment already discloses the unused-stub status and regulatory paraphrase context, so adding a redundant flag in the definition would violate Adequatism. Verification: - python 03_TECHNICAL_CORE/scripts/run_pipeline.py returns ALL CHECKS PASSED, exit 0 - grep "Recital 22" across the technical core returns zero hits - grep 'Article 3(36).{0,40}intended to be used' returns zero hits - All 14 fixes verified by per-fix backtest agents (deploy + verify pattern) Deferred: - Class-naming question (:PostRBI Process vs :PostRBI System for BFO Bucket 4 vs Bucket 1) tracked separately - /intake of the EU AI Act source into KB pending separate authorization - Output structure work (executive summary restructure, row-level audit-table interpretations, RAG colors, audit-trail anchor) is the separate next milestone Revert: each fix is a localized text edit in an annotation property, code comment, or prose paragraph. No cascading effects expected. Co-authored-by: Claude Opus 4.7 (1M context) --- .../ontology/ARCO_instances_verification.ttl | 4 ++-- .../reasoning/select_system_comment.sparql | 5 +++-- 03_TECHNICAL_CORE/scripts/run_pipeline.py | 12 +++++++----- LIMITATIONS.md | 4 ++-- docs/MODELING_ROADMAP.md | 4 ++-- docs/kiosk_demo_v1/kiosk_demo.md | 4 ++-- 6 files changed, 18 insertions(+), 15 deletions(-) diff --git a/03_TECHNICAL_CORE/ontology/ARCO_instances_verification.ttl b/03_TECHNICAL_CORE/ontology/ARCO_instances_verification.ttl index e507751..2a87cd3 100644 --- a/03_TECHNICAL_CORE/ontology/ARCO_instances_verification.ttl +++ b/03_TECHNICAL_CORE/ontology/ARCO_instances_verification.ttl @@ -15,7 +15,7 @@ Non-entailment claim: AnnexIII1aApplicableSystem is not entailed for VerificationKiosk_001 under current assertions (OWA — no closed-world guarantee). - Legal basis: Recital 22 and Art. 3(41) EU AI Act — 1:1 biometric verification is excluded from Annex III 1(a); only 1:N remote biometric identification triggers classification.""" ; + Legal basis: Recital 15, Recital 17, and Annex III item 1(a) of EU AI Act 2024/1689 — the verification carve-out ("intended to be used for biometric verification, which includes authentication, whose sole purpose is to confirm that a specific natural person is the person he or she claims to be") excludes 1:1 biometric verification from biometric identification (Recital 15) and from remote biometric identification systems specifically (Recital 17, rationale: minor impact on fundamental rights), and is reaffirmed in the Annex III 1(a) operative text. Article 3(36) defines biometric verification as the automated 1:1 modality; Article 3(41) defines the RBI system. Only 1:N remote biometric identification triggers Annex III 1(a) classification.""" ; owl:imports . ################################################################# @@ -24,7 +24,7 @@ :AnnexIII_Condition_1a_Exclusion rdf:type :RegulatoryContent ; rdfs:label "Annex III 1(a) — Verification Exclusion Note" ; - rdfs:comment "Documents the 1:1 verification exclusion from Annex III 1(a). Recital 22 and Art. 3(41): only 1:N remote biometric identification triggers the Annex III classification." ; + rdfs:comment "Documents the 1:1 verification exclusion from Annex III 1(a). Recital 15, Recital 17, and Annex III 1(a) operative carve-out: only 1:N remote biometric identification triggers the Annex III classification. Article 3(36) defines 1:1 verification; Article 3(41) defines the RBI system." ; iao:0000136 :BiometricVerificationCapability ; iao:0000136 :BiometricIdentificationCapability . diff --git a/03_TECHNICAL_CORE/reasoning/select_system_comment.sparql b/03_TECHNICAL_CORE/reasoning/select_system_comment.sparql index ddf5a60..4b5c75c 100644 --- a/03_TECHNICAL_CORE/reasoning/select_system_comment.sparql +++ b/03_TECHNICAL_CORE/reasoning/select_system_comment.sparql @@ -3,8 +3,9 @@ PREFIX rdfs: # EMISSION LAYER — binds rdfs:comment of the system under evaluation. # # Used by the negative-case output to surface fixture-supplied regulatory -# reasoning (e.g., the kiosk fixture's comment naming Recital 22 / Article -# 3(41) and the verification-vs-identification distinction). The comment +# reasoning (e.g., the kiosk fixture's comment naming Recital 15 + Recital +# 17 + Annex III 1(a) carve-out and the verification-vs-identification +# distinction). The comment # is fixture-authored ground truth and ships with the TTL; this query # avoids embedding fixture-specific legal text as Python literals in the # emitter (per CLAUDE.md Output discipline). diff --git a/03_TECHNICAL_CORE/scripts/run_pipeline.py b/03_TECHNICAL_CORE/scripts/run_pipeline.py index 37afbe4..70e1483 100644 --- a/03_TECHNICAL_CORE/scripts/run_pipeline.py +++ b/03_TECHNICAL_CORE/scripts/run_pipeline.py @@ -442,9 +442,10 @@ def get_system_comment(g: Graph, system_local: str) -> str: """Return rdfs:comment of the system instance, or empty string. Used by the negative-case Provider Obligations panel to surface fixture- - authored regulatory reasoning (e.g., the kiosk fixture's note on Recital - 22 / Article 3(41)) as a documentary scope text rather than a Python - literal in the emitter. Returns "" on query failure or empty result; + authored regulatory reasoning (e.g., the kiosk fixture's note on + Recital 15 + Recital 17 + Annex III 1(a) carve-out) as a documentary + scope text rather than a Python literal in the emitter. Returns "" on + query failure or empty result; emitter is responsible for skipping the panel when empty. """ try: @@ -1240,8 +1241,9 @@ def edge_el(rel, iri_label=""): # ARCO does not model Title IV obligations; see LIMITATIONS §2). It # is dropped here. # When the loaded fixture provides a system-level rdfs:comment with - # regulatory framing (e.g., the kiosk fixture's note on Recital 22 - # / Art. 3(41)), surface it via the documentary scope text path + # regulatory framing (e.g., the kiosk fixture's note on Recital 15 + + # Recital 17 + Annex III 1(a) carve-out), surface it via the + # documentary scope text path # declared in output_manifest_v2.yaml. Otherwise the panel reports # the OWA-bounded non-entailment note alone. if system_comment: diff --git a/LIMITATIONS.md b/LIMITATIONS.md index af3e666..08a50a3 100644 --- a/LIMITATIONS.md +++ b/LIMITATIONS.md @@ -111,7 +111,7 @@ Gate 1 locates the capability disposition on a `SystemComponent`, not on the `Sy A second, narrower simplification sits inside this gate: for a model-driven biometric module, the in-repo Sentinel fixture types the bearer as `:HardwareComponent` only, but a strict realist reading would locate the disposition on the *amalgam* of hardware plus the concretized model artifact running on it. The 2024 Capabilities paper's hardware-software-amalgam discussion treats software qua pattern as a generically dependent continuant, not itself a capable continuant; capabilities require a material bearer that concretizes the pattern. ARCO's classification result does not depend on which of these two bearer choices is taken — Gate 1 is satisfied as long as some component bearer instantiates a triggering-capability disposition — so the simplification is documented here rather than rebuilt as a structural change. -A third, related point: for software-configurable AI systems where the same hardware can be configured for different modes (e.g., 1:1 verification vs 1:N identification on the same biometric kiosk hardware), the disposition assertion describes what THIS specific deployment is intended to do under its current commitments, not what the hardware-in-isolation could theoretically do. ARCO does not make closed-world hardware-incapability claims; per-fixture disposition assertions reflect the configured-system commitments under OWA. A different deployment of the same hardware (different configuration, software, or database) would be modeled as a separate `:System` instance with its own asserted disposition. This matches the EU AI Act's classification on intended use (Article 3(36), Recital 15), not on raw hardware capability. Validated 2026-05-10 against vendor documentation for Suprema, ZKTeco, Matrix, HID, and IDEMIA biometric kiosks, all of which advertise the same hardware as configurable for both 1:1 and 1:N modes. +A third, related point: for software-configurable AI systems where the same hardware can be configured for different modes (e.g., 1:1 verification vs 1:N identification on the same biometric kiosk hardware), the disposition assertion describes what THIS specific deployment is intended to do under its current commitments, not what the hardware-in-isolation could theoretically do. ARCO does not make closed-world hardware-incapability claims; per-fixture disposition assertions reflect the configured-system commitments under OWA. A different deployment of the same hardware (different configuration, software, or database) would be modeled as a separate `:System` instance with its own asserted disposition. This matches the EU AI Act's classification on intended use (Recital 15, Recital 17, and the Annex III 1(a) operative carve-out, which together carry the "intended to be used for biometric verification" framing), not on raw hardware capability. Article 3(36) supplies the underlying 1:1 verification definition. Validated 2026-05-10 against vendor documentation for Suprema, ZKTeco, Matrix, HID, and IDEMIA biometric kiosks, all of which advertise the same hardware as configurable for both 1:1 and 1:N modes. ### 3.6 Reality/representation split @@ -138,7 +138,7 @@ The IUS subkind family is membership-fixed by Annex III categorial criterion, no #### 3.7.c Real-time vs. post RBI subclass declaration; Article 5 routing scoped future -ARCO declares `:PostRemoteBiometricIdentificationProcess` (Article 3(43): a remote biometric identification process other than real-time) and `:RealTimeRemoteBiometricIdentificationProcess` (Article 3(42): capturing, comparison and identification all occurring without a significant delay, comprising not only instant identification but also limited short delays in order to avoid circumvention) as disjoint subclasses of `:RemoteBiometricIdentificationProcess`. The disjointness is correct: the regulation defines the two as exhaustive subtypes within the parent term. +ARCO declares `:PostRemoteBiometricIdentificationProcess` and `:RealTimeRemoteBiometricIdentificationProcess` as disjoint subclasses of `:RemoteBiometricIdentificationProcess`. The regulatory verbatim text uses "system" rather than "process": Article 3(43) defines a post-RBI system as "a remote biometric identification system other than a real-time remote biometric identification system," and Article 3(42) defines a real-time RBI system as one whereby "the capturing of biometric data, the comparison and the identification all occur without a significant delay, comprising not only instant identification, but also limited short delays in order to avoid circumvention." ARCO models these as Process subclasses (BFO Bucket 4 / Occurrent) since classification reasons over the deployment-and-operation occurrence rather than the system artifact alone; the class-name "Process" is ARCO's modeling translation of the regulation's "system" term. The disjointness is correct: the regulation defines the two as exhaustive subtypes within the parent term. The Annex III 1(a) Gate 2 axiom continues to reference the parent class via `someValuesFrom`, so subclass propagation means an IUS prescribing either a real-time or a post RBI process particular satisfies Gate 2 and entails `:AnnexIII1aApplicableSystem`. No fixture is currently typed into either subclass; both are forward-declared. diff --git a/docs/MODELING_ROADMAP.md b/docs/MODELING_ROADMAP.md index a70f203..299f743 100644 --- a/docs/MODELING_ROADMAP.md +++ b/docs/MODELING_ROADMAP.md @@ -12,7 +12,7 @@ ARCO is the EU AI Act Annex III classifier I've been building. You hand it a str Mechanically: the classification axioms are written as `owl:equivalentClass` `owl:intersectionOf` definitions with three gates (capability disposition, intended use, affected role) at `03_TECHNICAL_CORE/ontology/ARCO_governance_extension.ttl`. The reasoning runs against BFO 2020, RO 2025-12-17, IAO 2026-03-30, and CCO v1.7-2024-11-03 commitments under OWL-RL. SHACL runs after reasoning to catch documentation inconsistencies. HermiT (under OWL 2 DL) runs in CI as an independent second reasoner so I'm not trusting a single inference profile. Outputs are gated through a failing-by-design provenance contract (`output_manifest_v2.yaml`, `test_output_provenance.py`) that refuses to ship a value unless its source is either graph-backed (named SPARQL query), run-metadata (declared sources), or a labeled documentary block. -Seven fixtures drive this today: two production positives (Sentinel ID for 1(a), Credit Scoring for 5(b)), one negative control (Verification Kiosk, a 1:1 biometric verification system that should not classify under 1(a), since verification is not identification per Recital 22 and Art. 3(41)), two adversarial fixtures (one decoy that injects an `owl:equivalentClass` to test whether the gates collapse on hostile input, one with an anonymous blank-node disposition to test whether the reasoner needs named individuals), and two flag-test fixtures (where all three Annex III gates are satisfied alongside an audit-layer flag like a provider-asserted derogation claim or a fraud-detection process token, to verify classification and audit don't bleed into each other). +Seven fixtures drive this today: two production positives (Sentinel ID for 1(a), Credit Scoring for 5(b)), one negative control (Verification Kiosk, a 1:1 biometric verification system that should not classify under 1(a), since verification is not identification per Recital 15, Recital 17, and the Annex III 1(a) operative carve-out (Article 3(36) defines biometric verification as 1:1; Article 3(41) defines the RBI system)), two adversarial fixtures (one decoy that injects an `owl:equivalentClass` to test whether the gates collapse on hostile input, one with an anonymous blank-node disposition to test whether the reasoner needs named individuals), and two flag-test fixtures (where all three Annex III gates are satisfied alongside an audit-layer flag like a provider-asserted derogation claim or a fraud-detection process token, to verify classification and audit don't bleed into each other). The EU AI Act lists 8 high-risk Annex III categories. I've modeled 2. The other 6 are deferred and I'm honest about that in `LIMITATIONS.md` §1. @@ -65,7 +65,7 @@ These are deliberate choices I've made. Each one is also disclosed in `LIMITATIO - BFO Bucket 5 (Site) and Bucket 6 (TemporalRegion) not modeled. ARCO is a design-time classifier, so runtime spatial/temporal regions don't carry weight at the entailment layer. If I extend ARCO to monitor jurisdiction-specific deployment or substantial modification, these activate (LIMITATIONS §3.7). - Provider-organization-to-ICE structural link not modeled (per PR #62). - Source documents do not auto-promote to reality-side commitments. Promotion is rare, conditional, and human-adjudicated per `docs/EVIDENCE_TO_COMMITMENT_POLICY.md`. I've been deliberate about not letting extraction wire directly to instance TTL. -- Closed-world hardware-incapability claims for software-configurable systems are forbidden. EU AI Act Annex III 1(a) keys on Article 3(36) "intended to be used" language, not on what the hardware in isolation can do (LIMITATIONS §3.5). +- Closed-world hardware-incapability claims for software-configurable systems are forbidden. EU AI Act Annex III 1(a) keys on the "intended to be used for biometric verification" language present in Recital 15, Recital 17, and the Annex III 1(a) operative carve-out, not on what the hardware in isolation can do. Article 3(36) supplies the technical 1:1 verification definition (LIMITATIONS §3.5). --- diff --git a/docs/kiosk_demo_v1/kiosk_demo.md b/docs/kiosk_demo_v1/kiosk_demo.md index 315601b..288c8f7 100644 --- a/docs/kiosk_demo_v1/kiosk_demo.md +++ b/docs/kiosk_demo_v1/kiosk_demo.md @@ -62,7 +62,7 @@ The source explicitly refuses two closed-world claims: - It does NOT claim the underlying hardware is structurally incapable of identification (the hardware is software-configurable). - It does NOT claim what the hardware could be configured to in some other deployment. -Article 3(36) of EU Regulation 2024/1689 keys Annex III 1(a) on intended use, not on raw hardware capability. The source packet describes THIS deployment under THIS configuration. +Annex III item 1(a) of EU Regulation 2024/1689 keys high-risk RBI classification on intended use ("intended to be used for biometric verification" carve-out per Recital 15, Recital 17, and the Annex III 1(a) operative clause), not on raw hardware capability. Article 3(36) supplies the technical 1:1 verification definition. The source packet describes THIS deployment under THIS configuration. (\*) **Real vendor document substitution is `OPEN_PROBLEMS.md L1.1`.** The next concrete move replaces this hypothetical with a real vendor packet (datasheet, conformity declaration, or equivalent), with each claim cited. @@ -169,7 +169,7 @@ The vision is the full input-mile-to-honest-certificate chain wired programmatic - (\*) **Configuration-level aboutness for regulatory ICEs** — `:AnnexIII_Condition_*` ICEs need explicit `iao:is_about` targets; canonical options surfaced (universal-only / particular continuant / multiple constituents). `M-Aboutness-Config-1 / L2.7`. - (\*) **`:CapabilityDisposition` rename to `:Capability`** — Smith-Against-Idiosyncrasy Principle 8 compositional naming fix; mechanical PR. `M-NameDiscipline-1 / L2.8`. - (\*) **Gate 3 role-relationship tightening** — current axiom designates the natural-person role universal but does not pin down whether natural persons are subjects of identification, operators, or another role-in-context. `L2.9`. -- (\*) **Gate 2 use-purpose proxy tightening** — current axiom maps Article 3(36) "intended to be used for" to `cco:prescribes someValuesFrom :Process`; defensible structural proxy but loose. `L2.10`. +- (\*) **Gate 2 use-purpose proxy tightening** — current axiom maps the "intended to be used for" framing (from Recital 15, Recital 17, and Annex III 1(a)) to `cco:prescribes someValuesFrom :Process`; Article 3(36) supplies the technical 1:1 verification definition the axiom presupposes. Defensible structural proxy but loose. `L2.10`. - (\*) **Reasoned-graph and HermiT classification artifact export** — surface the full entailment chain inspectably. `L4.8 parts 1-2`. - (\*) **Prose determination narrative emission** — generated from existing SPARQL gate evidence so a non-ARCO-glossary reader can read the conclusion + the reason in one paragraph. `L4.8 part 3`. - (\*) **Output provenance tightening** — G/M/D field labels, per-field source-query manifest, name/source mismatches across surrounding cert fields. `L4.4-L4.6`. From 4655ca83225fb777116b599e3aeca3686def15bd Mon Sep 17 00:00:00 2001 From: Alex Moskowitz Date: Wed, 20 May 2026 21:20:56 -0400 Subject: [PATCH 5/7] fix(output): separate scope metadata from graph-backed entailment Article 6(3) derogation scope qualifier no longer concatenated into `VERIFIED (ENTAILED)` literals (certificate text, summary print, summary.json, HTML badge, cert_lines builder). Scope surfaces as a separate field everywhere it appeared: - `derogation_evaluation_scope` object in summary.json - separate `ARTICLE 6(3) DEROGATION: NOT EVALUATED (run scope)` line in certificate.txt - pre-existing `derogation_scope_badge` HTML node now the canonical HTML disclosure surface Gate 3 display now requires the USS to designate `:NaturalPersonRole` (not just USS existence). Shared `gate3_designates_expected_role` helper applied to both HTML view and determination packet status, closing the display-weaker-than-OWL-axiom gap. `test_output_provenance.py` forbidden-pattern check passes 0/0. README and LIMITATIONS updated to remove stale "failing-by-design" language and "LIVE" markers on the now-closed Gate 3 and Article 6(3) defects. Closes OPEN_PROBLEMS L3.4 (Gate 3 truth-surface) and the Article 6(3) mixed-provenance forbidden-pattern. Co-Authored-By: Claude Opus 4.7 (1M context) --- .gitignore | 2 + 03_TECHNICAL_CORE/scripts/run_pipeline.py | 113 +++++++++++++----- .../scripts/test_output_provenance.py | 4 +- LIMITATIONS.md | 4 +- README.md | 2 +- 5 files changed, 88 insertions(+), 37 deletions(-) diff --git a/.gitignore b/.gitignore index 30926bc..a33dec5 100644 --- a/.gitignore +++ b/.gitignore @@ -4,6 +4,8 @@ # Obsidian knowledge base (local only, not versioned) KB/ +# Defensive: even if KB/ rule changes, never commit vendor node_modules trees +KB/**/node_modules/ # Git metadata (huge) .git/ diff --git a/03_TECHNICAL_CORE/scripts/run_pipeline.py b/03_TECHNICAL_CORE/scripts/run_pipeline.py index 70e1483..c3246fd 100644 --- a/03_TECHNICAL_CORE/scripts/run_pipeline.py +++ b/03_TECHNICAL_CORE/scripts/run_pipeline.py @@ -564,6 +564,20 @@ def select_gate_evidence(g: Graph, system_local: str) -> dict: return packet + +def gate3_designates_expected_role(gate_evidence: dict) -> bool: + """Return whether Gate 3 satisfies ARCO's current role-designation axiom. + + Current modeled Annex III branches both require the USS to designate the + NaturalPersonRole universal. OPEN_PROBLEMS L3.2 tracks future + category-parameterized role expectations. + """ + return ( + bool(gate_evidence["gate3"]["uss_uri"]) + and gate_evidence["gate3"]["role_uri"] == f"{ARCO_NS}NaturalPersonRole" + ) + + def verify_high_risk_inference(reasoned: Graph, source: Graph) -> tuple[bool, bool, bool, list[tuple[str, str]]]: """Returns (inference_ok, asserted_pre, entailed_post, bindings).""" hr("ARCO RESULT (ENTAILMENT + PROOF SKETCH)") @@ -860,9 +874,13 @@ def _annex(val): if val is None: return 'N/A' if val: - if not derogation_flagged: - return ('VERIFIED (ENTAILED, ' - 'Article 6(3) derogation not evaluated)') + # Pure graph-backed entailment value. The Article 6(3) derogation + # scope qualifier (when no DerogationClaim is asserted) is + # surfaced as the separate `derogation_scope_badge` rendered + # alongside the conclusion banner, per + # output_manifest_v2.yaml field `derogation_evaluation_scope` + # (forbidden_pattern: embedding the qualifier into a + # graph_backed value). return 'VERIFIED (ENTAILED)' return 'NOT APPLICABLE' @@ -1005,8 +1023,14 @@ def edge_el(rel, iri_label=""): # typed-content satisfaction by itself — its own header at # reasoning/check_intended_use.sparql:5-12 declares it documentary. gate2_ok = bool(gate_evidence["gate2"]["process_type_uri"]) - # Gate 3: USS designation is asserted (already evidence-coupled). - gate3_ok = bool(gate_evidence["gate3"]["uss_uri"]) + # Gate 3: USS designation is asserted AND designates the expected role + # universal. The OWL Gate 3 axiom uses + # `cco:designates owl:hasValue :NaturalPersonRole`; a fixture asserting + # a USS that designates a different role would otherwise display as + # Gate 3 OK while OWL correctly does not entail Gate 3 satisfaction. + # (L3.4 truth-surface fix; per-category role parameterization tracked + # at L3.2.) + gate3_ok = gate3_designates_expected_role(gate_evidence) # Pre-compute gate display labels from the determination packet. # These are used in axiom pattern text, gate answers, and counterfactuals. @@ -2055,11 +2079,20 @@ def main() -> None: print(" [classification layer — OWL-RL entailment]") print(f"SHACL: {_pf(shacl_ok)}") print(f"Entailment: {_pf(inference_ok)}") - _summary_qualifier = ", Article 6(3) derogation not evaluated" if not derogation_flagged else "" + # Pure graph-backed entailment values. Article 6(3) derogation scope + # is reported as a separate run-metadata line below, not embedded in + # the graph_backed VERIFIED literal (per output_manifest_v2.yaml + # field `derogation_evaluation_scope` forbidden_pattern). if annex_iii_1a_ok is not None: - print(f"Annex III 1a: {'VERIFIED (ENTAILED' + _summary_qualifier + ')' if annex_iii_1a_ok else 'NOT APPLICABLE'} (OWL-entailed)") + print(f"Annex III 1a: {'VERIFIED (ENTAILED)' if annex_iii_1a_ok else 'NOT APPLICABLE'} (OWL-entailed)") if annex_iii_5b_ok is not None: - print(f"Annex III 5b: {'VERIFIED (ENTAILED' + _summary_qualifier + ')' if annex_iii_5b_ok else 'NOT APPLICABLE'} (OWL-entailed)") + print(f"Annex III 5b: {'VERIFIED (ENTAILED)' if annex_iii_5b_ok else 'NOT APPLICABLE'} (OWL-entailed)") + # Separate run-scope disclosure for derogation evaluation. Surfaces + # only when a category is entailed and no DerogationClaim is asserted + # (when a claim IS asserted, the existing FLAG row signals the + # unevaluated derogation; do not double-disclose). + if (annex_iii_1a_ok or annex_iii_5b_ok) and not derogation_flagged: + print(" [run scope] Article 6(3) derogation: NOT EVALUATED") print() print(" [audit documentation layer — SPARQL ASK on reasoned graph]") print(f"Traceability: {_pf(traceability_ok)}") @@ -2268,26 +2301,29 @@ def main() -> None: # audit record. if intended_use_ok is not None: print(f" INTENDED USE: {_pf(intended_use_ok)}") - # Article 6(3) derogation scope qualifier: ARCO does not evaluate the + # Article 6(3) derogation scope: ARCO does not evaluate the # Article 6(3) carve-out conditions; it only detects whether a provider- - # supplied :DerogationClaim artifact is asserted. When such a claim is - # asserted, the FLAG line below signals the unevaluated derogation. When - # no claim is asserted, an Annex III ENTAILED conclusion otherwise reads - # as "high-risk classification confirmed" — append a same-line scope - # qualifier so the disclosure rides with the conclusion. - _annex_iii_entailed_qualifier = "VERIFIED (ENTAILED, Article 6(3) derogation not evaluated)" + # supplied :DerogationClaim artifact is asserted. The Annex III ENTAILED + # conclusion below is a pure graph_backed value; the derogation scope + # disclosure rides on its own line (see "ARTICLE 6(3) DEROGATION" below) + # per output_manifest_v2.yaml field `derogation_evaluation_scope` + # forbidden_pattern (Python concatenation that embeds the qualifier + # into a graph_backed value). if annex_iii_1a_ok is not None: - if annex_iii_1a_ok: - _line_1a = _annex_iii_entailed_qualifier if not derogation_flagged else "VERIFIED (ENTAILED)" - else: - _line_1a = "NOT APPLICABLE" + _line_1a = "VERIFIED (ENTAILED)" if annex_iii_1a_ok else "NOT APPLICABLE" print(f" ANNEX III 1(a): {_line_1a}") if annex_iii_5b_ok is not None: if annex_iii_5b_ok: - _line_5b = _annex_iii_entailed_qualifier if not derogation_flagged else "VERIFIED (ENTAILED)" + _line_5b = "VERIFIED (ENTAILED)" else: _line_5b = "NOT APPLICABLE" print(f" ANNEX III 5(b): {_line_5b}") + # Separate run-scope disclosure for derogation evaluation. Surfaces + # only when a category is entailed and no DerogationClaim is asserted + # (when a claim IS asserted, the existing FLAG line below already + # signals the unevaluated derogation; do not double-disclose). + if (annex_iii_1a_ok or annex_iii_5b_ok) and not derogation_flagged: + print(" ARTICLE 6(3) DEROGATION: NOT EVALUATED (run scope)") if obligation_ok is not None: print(f" OBLIGATION: {_cert_obligation_label}") if reg_alignment_ok is not None: @@ -2335,18 +2371,20 @@ def main() -> None: # summary.json["latent_risk"] for audit consumers. if intended_use_ok is not None: cert_lines.append(f" INTENDED USE: {_pf(intended_use_ok)}") + # Pure graph-backed entailment values. The Article 6(3) derogation + # scope rides on the separate "ARTICLE 6(3) DEROGATION" line below + # (per output_manifest_v2.yaml field `derogation_evaluation_scope` + # forbidden_pattern). if annex_iii_1a_ok is not None: - if annex_iii_1a_ok: - _cert_line_1a = _annex_iii_entailed_qualifier if not derogation_flagged else "VERIFIED (ENTAILED)" - else: - _cert_line_1a = "NOT APPLICABLE" + _cert_line_1a = "VERIFIED (ENTAILED)" if annex_iii_1a_ok else "NOT APPLICABLE" cert_lines.append(f" ANNEX III 1(a): {_cert_line_1a}") if annex_iii_5b_ok is not None: - if annex_iii_5b_ok: - _cert_line_5b = _annex_iii_entailed_qualifier if not derogation_flagged else "VERIFIED (ENTAILED)" - else: - _cert_line_5b = "NOT APPLICABLE" + _cert_line_5b = "VERIFIED (ENTAILED)" if annex_iii_5b_ok else "NOT APPLICABLE" cert_lines.append(f" ANNEX III 5(b): {_cert_line_5b}") + # Separate run-scope disclosure for derogation evaluation. Surfaces + # only when a category is entailed and no DerogationClaim is asserted. + if (annex_iii_1a_ok or annex_iii_5b_ok) and not derogation_flagged: + cert_lines.append(" ARTICLE 6(3) DEROGATION: NOT EVALUATED (run scope)") if obligation_ok is not None: cert_lines.append(f" OBLIGATION: {_cert_obligation_label}") if reg_alignment_ok is not None: @@ -2418,8 +2456,17 @@ def main() -> None: latent_ok, "DETECTED", "NOT_DETECTED", ), "intended_use": (_pf(intended_use_ok) if intended_use_ok is not None else "NOT_RUN"), - "annex_iii_1a": (("VERIFIED (ENTAILED, Article 6(3) derogation not evaluated)" if not derogation_flagged else "VERIFIED (ENTAILED)") if annex_iii_1a_ok else ("NOT APPLICABLE" if annex_iii_1a_ok is not None else "N/A")), - "annex_iii_5b": (("VERIFIED (ENTAILED, Article 6(3) derogation not evaluated)" if not derogation_flagged else "VERIFIED (ENTAILED)") if annex_iii_5b_ok else ("NOT APPLICABLE" if annex_iii_5b_ok is not None else "N/A")), + # Pure graph-backed entailment values. Article 6(3) derogation + # scope rides on the separate `derogation_evaluation_scope` + # field below, not embedded in these graph_backed literals + # (per output_manifest_v2.yaml field `derogation_evaluation_scope` + # forbidden_pattern). + "annex_iii_1a": ("VERIFIED (ENTAILED)" if annex_iii_1a_ok else ("NOT APPLICABLE" if annex_iii_1a_ok is not None else "N/A")), + "annex_iii_5b": ("VERIFIED (ENTAILED)" if annex_iii_5b_ok else ("NOT APPLICABLE" if annex_iii_5b_ok is not None else "N/A")), + "derogation_evaluation_scope": { + "evaluated": False, + "reason": "Article 6(3) derogation evaluation not modeled in current ARCO release; see LIMITATIONS.md §3.7", + }, "obligation": _status_label( obligation_ok, "PASS", "FAIL", not_applicable_label="NOT_APPLICABLE" if non_applicable_run else None, @@ -2538,7 +2585,6 @@ def main() -> None: # (run_pipeline.py:806): typed-evidence presence, not the # documentary ASK `intended_use_ok`. Closes the schema-incoherent # SATISFIED-with-empty-evidence state on non-applicable runs. - # Parallel to gate_3 below which is already evidence-coupled. "status": "SATISFIED" if bool(gate_evidence["gate2"]["process_type_uri"]) else "NOT_SATISFIED", "evidence": { "ius_uri": gate_evidence["gate2"]["ius_uri"], @@ -2550,7 +2596,10 @@ def main() -> None: { "id": "gate_3", "label": "Affected Role Category", - "status": "SATISFIED" if bool(gate_evidence["gate3"]["uss_uri"]) else "NOT_SATISFIED", + # Packet-side Gate 3 status mirrors HTML-side `gate3_ok`: + # USS existence is not enough; the designated role must match + # the current OWL Gate 3 target. + "status": "SATISFIED" if gate3_designates_expected_role(gate_evidence) else "NOT_SATISFIED", "evidence": { "uss_uri": gate_evidence["gate3"]["uss_uri"], "role_uri": gate_evidence["gate3"]["role_uri"], diff --git a/03_TECHNICAL_CORE/scripts/test_output_provenance.py b/03_TECHNICAL_CORE/scripts/test_output_provenance.py index 14a2ea5..666492b 100644 --- a/03_TECHNICAL_CORE/scripts/test_output_provenance.py +++ b/03_TECHNICAL_CORE/scripts/test_output_provenance.py @@ -282,8 +282,8 @@ def main() -> int: print(f"Pipeline: {PIPELINE_PATH.relative_to(REPO_ROOT)}") print(f"Manifest version: {manifest.get('manifest_version')}") print() - print("This test fails today by design. Each failure names a v1.2 emitter") - print("violation. PR2-PR6 each move at least one failure to passing.") + print("This test names output-provenance emitter violations when present.") + print("Current acceptance target: zero failures.") print() hr("Check 1 — forbidden source patterns") diff --git a/LIMITATIONS.md b/LIMITATIONS.md index 08a50a3..3d3e5b0 100644 --- a/LIMITATIONS.md +++ b/LIMITATIONS.md @@ -273,13 +273,13 @@ The pipeline's output layer (`run_pipeline.py` from line 1699 onward, plus `writ **Sentinel-shaped hardcoding** (the same affected-role pattern is hardcoded in three independent places, making the Gate 3 surface category-specific rather than parameterized): -- Python: `gate_evidence["gate3"]["role_uri"]` initializes to `:NaturalPersonRole` (`run_pipeline.py:371`); display labels fall back to biometric-identification strings when SPARQL returns zero rows (`run_pipeline.py:740-743`); `gate3_ok` checks USS existence only, not which role is designated (`run_pipeline.py:735`). **LIVE** (OPEN_PROBLEMS L3.2 and L3.4): the Python `role_uri` initialization and fallback display labels are still category-specific to biometric identification; `gate3_ok` is still a `bool(uss_uri)` Python predicate weaker than the OWL Gate 3 axiom. +- Python: Gate 3's display and determination-packet status now require the USS to designate `:NaturalPersonRole`; USS existence alone is not treated as Gate 3 satisfaction. **CLOSED 2026-05-20 (OPEN_PROBLEMS L3.4)** for the current modeled categories. **LIVE** (OPEN_PROBLEMS L3.2): the role target remains hardcoded to `:NaturalPersonRole` rather than parameterized by Annex III category. - SPARQL: `check_intended_use.sparql:31` hardcodes `cco:designates :NaturalPersonRole`, bundling Gates 2 and 3 into one ASK with the affected role baked in. **LIVE** (OPEN_PROBLEMS L3.2). - SHACL: `assessment_documentation_shape.ttl:96-100` hardcodes `sh:hasValue :NaturalPersonRole` at the shape level. **LIVE** (OPEN_PROBLEMS L3.2). The 5(b) HTML-emission counterpart was a separate locus and is **CLOSED 2026-05-10 (PR #34, OPEN_PROBLEMS L3.3)**: `write_html_view`'s 5(b) branch now reads `gate_evidence["gate2"]["process_type_label"]` and the parameterized gate-evidence helpers rather than Python literals. **Synthesized narrative**: -- The certificate's Annex III line carries a Python string literal "VERIFIED (ENTAILED, Article 6(3) derogation not evaluated)" (`run_pipeline.py:1725-1738`). The graph does not commit to "derogation not evaluated" as a state. **HEADLINE COMPOSITION CLOSED 2026-05-10 (PR #36, OPEN_PROBLEMS L4.3).** New `reasoning/select_primary_classification.sparql` returns the entailed class IRI list; the headline value is now the pure class IRI. Mode/scope text moved to separate fields. Composite qualifiers like `(ENTAILED, all three ARCO gates)` and `(Annex III Capability-Precondition Flag; ...; not the EU AI Act legal high-risk classification)` are dropped from the headline format functions. **REMAINING**: the polarity of the `(Article 6(3) derogation not evaluated)` qualifier when a `:DerogationClaim` is flagged is a queued semantic-correctness item (regulator-defensibility concern surfaced 2026-05-11; the honest qualifier should *strengthen*, not drop, when more uncertainty exists). Tracked separately. +- The certificate's Annex III line now emits pure graph-backed values such as `VERIFIED (ENTAILED)` or `NOT APPLICABLE`. Article 6(3) derogation scope is surfaced separately as run metadata (`ARTICLE 6(3) DEROGATION: NOT EVALUATED (run scope)` in the certificate and `derogation_evaluation_scope` in `summary.json`). **CLOSED 2026-05-20** for the mixed-provenance string caught by `test_output_provenance.py`. **REMAINING**: Article 6(3) validity is still not evaluated; `:DerogationClaim` is surfaced for human legal review only. **Operational**: diff --git a/README.md b/README.md index 0ae4ebc..976ae97 100644 --- a/README.md +++ b/README.md @@ -130,7 +130,7 @@ The architecture grounds in BFO 2020 (ISO/IEC 21838-2:2021) and uses the seven-b **Gate independence is empirically verified.** A regression test removes the supporting triples for each Annex III 1(a) and 5(b) gate in turn and confirms the classification fails. Each gate is independently necessary in both categories; removing any one breaks the entailment. Content-mutation variants (wrong process type, wrong designation target) verify that the gates check content, not just existence. -**The certificate's classification binds to graph queries.** Classification field and evidence path are bound to SPARQL queries against the reasoned graph; the contract lives in `03_TECHNICAL_CORE/scripts/output_manifest_v2.yaml`, enforced by `test_output_provenance.py` (failing-by-design). Tightening provenance labels across surrounding fields is active work ([`OPEN_PROBLEMS.md L4.4-L4.6`](OPEN_PROBLEMS.md), [`LIMITATIONS.md §7.5`](LIMITATIONS.md)). +**The certificate's classification binds to graph queries.** Classification field and evidence path are bound to SPARQL queries against the reasoned graph; the contract lives in `03_TECHNICAL_CORE/scripts/output_manifest_v2.yaml`, enforced by `test_output_provenance.py`. The former Article 6(3) mixed-provenance failure is closed; broader provenance-label and schema hardening remains active work ([`OPEN_PROBLEMS.md L4.4-L4.6`](OPEN_PROBLEMS.md), [`LIMITATIONS.md §7.5`](LIMITATIONS.md)). **The graph stays honest about what it doesn't know.** ARCO does not mint participant facts, temporal regions, role-bearer particulars, or other instance-level content that source evidence does not warrant. Under the Open World Assumption, absent triples mean "not asserted by the reviewed commitments," not "denied." Keeping the graph sparse where evidence is sparse is a project discipline, enforced in code review. From f3b53d93ddfab782504b05a5bdbc2b702e00eaa8 Mon Sep 17 00:00:00 2001 From: Alex Moskowitz Date: Wed, 20 May 2026 21:21:31 -0400 Subject: [PATCH 6/7] test(adversarial): add 5(b) equivalentClass decoy parity MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit New fixture `ARCO_instances_adversarial_decoy_5b.ttl` declares `:WeirdCalculator owl:equivalentClass :CreditworthinessEvaluationCapability`, types `:WeirdCalc_Disposition` only as the alias class, and asserts the three-gate participants (IUS prescribing a `:CreditworthinessEvaluationProcess` token, USS designating `:NaturalPersonRole`). Adversarial-purity discipline matches the 1(a) decoy: no provider organisation, no assessment documentation, no obligation, no determination ICE — only what the gate axiom needs to fire. New `test_credit_decoy_classifies_via_equivalent_class` in `test_adversarial_mechanism.py` verifies five assertions: 1. pre-reasoning disposition typed as alias only 2. pre-reasoning disposition NOT typed as :CreditworthinessEvaluationCapability 3. post-reasoning disposition IS typed as :CreditworthinessEvaluationCapability 4. post-reasoning system entails :AnnexIII5bApplicableSystem 5. post-reasoning system does NOT entail :AnnexIII1aApplicableSystem (cross-category isolation preserved) Alias path's IRIs and labels avoid Credit / Score / Assessment / Evaluation / 5b vocabulary so a grep/label-matching reader sees no regulated-class hint in the disposition, module, or system names. The `rdfs:comment` on the alias class necessarily names :CreditworthinessEvaluationCapability (that is what `owl:equivalentClass` documents). README and LIMITATIONS adversarial-coverage descriptions updated to reflect two equivalentClass decoys (1(a) + 5(b)) plus the blank-node ghost. Closes OPEN_PROBLEMS L3.7 (5(b) adversarial equivalentClass parity). Co-Authored-By: Claude Opus 4.7 (1M context) --- .../ARCO_instances_adversarial_decoy_5b.ttl | 77 ++++++++++++++++ .../scripts/test_adversarial_mechanism.py | 91 ++++++++++++++++++- LIMITATIONS.md | 2 +- README.md | 2 +- 4 files changed, 168 insertions(+), 4 deletions(-) create mode 100644 03_TECHNICAL_CORE/ontology/ARCO_instances_adversarial_decoy_5b.ttl diff --git a/03_TECHNICAL_CORE/ontology/ARCO_instances_adversarial_decoy_5b.ttl b/03_TECHNICAL_CORE/ontology/ARCO_instances_adversarial_decoy_5b.ttl new file mode 100644 index 0000000..3009460 --- /dev/null +++ b/03_TECHNICAL_CORE/ontology/ARCO_instances_adversarial_decoy_5b.ttl @@ -0,0 +1,77 @@ +@prefix : . +@prefix bfo: . +@prefix cco: . +@prefix ro: . +@prefix iao: . +@prefix rdf: . +@prefix rdfs: . +@prefix owl: . +@prefix skos: . + + rdf:type owl:Ontology ; + rdfs:label "ARCO Adversarial Test: Equivalency Decoy (5(b) parity)" ; + rdfs:comment """Tests that OWL-RL performs real equivalence reasoning on the Annex III 5(b) branch, mirroring the 1(a) decoy at ARCO_instances_adversarial_decoy.ttl. + + :WeirdCalculator is declared owl:equivalentClass to :CreditworthinessEvaluationCapability. + The disposition is typed ONLY as :WeirdCalculator — no Credit/Evaluation/Score/Assessment + vocabulary appears in the alias class, disposition, module, or system IRIs. + + If the reasoner is real, it infers CreditworthinessEvaluationCapability membership + via owl:equivalentClass propagation; the three-gate intersection then entails + :AnnexIII5bApplicableSystem. + + Cross-category isolation is also under test here: post-reasoning the system + must NOT entail :AnnexIII1aApplicableSystem (no biometric identification + capability is bound to any component). + + Adversarial-purity discipline (parallel to the 1(a) decoy): + - no provider organisation, no provider role + - no assessment documentation process, no assessment documentation artifact + - no compliance obligation + - no determination ICE asserted + Only the reality-side path the three-gate axiom needs to fire is asserted. + Closes OPEN_PROBLEMS L3.7 (5(b) adversarial equivalentClass parity).""" ; + owl:imports . + +################################################################# +# ADVERSARIAL SETUP: equivalence decoy class +################################################################# + +:WeirdCalculator rdf:type owl:Class ; + rdfs:label "Weird Calculator (decoy name)" ; + skos:prefLabel "Weird Calculator (decoy name)"@en ; + skos:definition "An adversarial test class declared owl:equivalentClass to :CreditworthinessEvaluationCapability. Used by the 5(b) equivalence-decoy fixture to verify that the OWL reasoner performs real semantic inference via equivalentClass propagation on the credit branch, rather than IRI pattern matching or class-name string matching." ; + rdfs:comment "Equivalent to CreditworthinessEvaluationCapability but with an unrelated name. Tests that the reasoner uses equivalence, not string matching, on the 5(b) branch." ; + owl:equivalentClass :CreditworthinessEvaluationCapability . + +################################################################# +# SYSTEM — typed using ONLY the decoy class name +################################################################# + +:WeirdCalc_Disposition rdf:type :WeirdCalculator ; + rdfs:label "Weird Calculator Disposition (typed as WeirdCalculator only)" . + +:WeirdCalc_Module rdf:type :HardwareComponent ; + rdfs:label "Weird Calculator Hardware Module" ; + ro:0000091 :WeirdCalc_Disposition . + +:WeirdCalcSystem_001 rdf:type :System ; + rdfs:label "Weird Calculator System 001" ; + bfo:0000051 :WeirdCalc_Module . + +################################################################# +# DOCUMENTARY MINIMUM (gates 2 + 3) +################################################################# + +:WeirdCalc_Process rdf:type :CreditworthinessEvaluationProcess ; + rdfs:label "Weird Calculator Process Token" . + +:WeirdCalc_IntendedUse_001 rdf:type :IntendedUseSpecification ; + rdfs:label "Weird Calculator Intended Use" ; + cco:prescribes :WeirdCalc_Process ; + iao:0000136 :WeirdCalcSystem_001 . + +:WeirdCalc_UseScenario_001 rdf:type :UseScenarioSpecification ; + rdfs:label "Weird Calculator Use Scenario" ; + iao:0000136 :WeirdCalcSystem_001 ; + cco:designates :NaturalPersonRole . diff --git a/03_TECHNICAL_CORE/scripts/test_adversarial_mechanism.py b/03_TECHNICAL_CORE/scripts/test_adversarial_mechanism.py index 4ce05c6..5ff163b 100644 --- a/03_TECHNICAL_CORE/scripts/test_adversarial_mechanism.py +++ b/03_TECHNICAL_CORE/scripts/test_adversarial_mechanism.py @@ -7,7 +7,7 @@ would let a pattern-matching pipeline pass test_scenarios.py with the wrong mechanism in place. - Decoy fixture (ARCO_instances_adversarial_decoy.ttl): + Decoy fixture (ARCO_instances_adversarial_decoy.ttl) — Annex III 1(a): The disposition is typed only as :WeirdScanner. :WeirdScanner is declared owl:equivalentClass :BiometricIdentificationCapability in the same fixture. Pre-reasoning, no asserted triple types the disposition as @@ -18,6 +18,16 @@ :BiometricIdentificationCapability triple in the input and miss the classification. The OWL reasoner does not. + Decoy 5(b) fixture (ARCO_instances_adversarial_decoy_5b.ttl) — Annex III 5(b): + Same shape as the 1(a) decoy, branch-swapped. :WeirdCalculator is declared + owl:equivalentClass :CreditworthinessEvaluationCapability. Disposition is + typed only as :WeirdCalculator. Post-reasoning the disposition entails as + :CreditworthinessEvaluationCapability and the three-gate intersection + entails :AnnexIII5bApplicableSystem. Cross-category isolation is also + under test: post-reasoning the system MUST NOT entail + :AnnexIII1aApplicableSystem (no biometric capability bound). Closes + OPEN_PROBLEMS L3.7. + Ghost fixture (ARCO_instances_adversarial_blanknode.ttl): The disposition is an anonymous individual (blank node) typed as :BiometricIdentificationCapability. owl:someValuesFrom requires @@ -137,6 +147,80 @@ def test_decoy_classifies_via_equivalent_class() -> bool: return ok +def test_credit_decoy_classifies_via_equivalent_class() -> bool: + """ + Verify that WeirdCalcSystem_001's Gate 1 entailment fires via + owl:equivalentClass propagation on the Annex III 5(b) branch, not via + direct IRI assertion. Mirrors test_decoy_classifies_via_equivalent_class + for the credit branch. Closes OPEN_PROBLEMS L3.7. + + Five assertions: + 1. Pre-reasoning: disposition rdf:type :WeirdCalculator (alias only). + 2. Pre-reasoning: disposition rdf:type :CreditworthinessEvaluationCapability + is ABSENT (proves alias-only routing). + 3. Post-reasoning: disposition rdf:type :CreditworthinessEvaluationCapability + is present (equivalentClass propagation fired). + 4. Post-reasoning: system rdf:type :AnnexIII5bApplicableSystem + (three-gate intersection entails). + 5. Post-reasoning: system rdf:type :AnnexIII1aApplicableSystem + is ABSENT (cross-category isolation; no biometric capability bound). + """ + print("\n--- DECOY 5(b): classification via owl:equivalentClass (credit branch) ---") + fixture = ONTOLOGY_DIR / "ARCO_instances_adversarial_decoy_5b.ttl" + g_pre = parse_fixture(fixture) + + calc_disp = ARCO["WeirdCalc_Disposition"] + weird_calculator = ARCO["WeirdCalculator"] + credit_cap = ARCO["CreditworthinessEvaluationCapability"] + calc_system = ARCO["WeirdCalcSystem_001"] + annex_5b = ARCO["AnnexIII5bApplicableSystem"] + annex_1a = ARCO["AnnexIII1aApplicableSystem"] + + # Precondition 1: disposition is typed as :WeirdCalculator (the decoy class) + has_alias_pre = (calc_disp, RDF["type"], weird_calculator) in g_pre + # Precondition 2: disposition is NOT directly typed as + # :CreditworthinessEvaluationCapability before reasoning + has_credit_pre = (calc_disp, RDF["type"], credit_cap) in g_pre + # Precondition 3: System is NOT yet classified as 5(b) applicable + is_5b_pre = (calc_system, RDF["type"], annex_5b) in g_pre + + ok = True + print(f" pre-reasoning :WeirdCalc_Disposition rdf:type :WeirdCalculator: {has_alias_pre} (expected True)") + if not has_alias_pre: + ok = False + print(f" pre-reasoning :WeirdCalc_Disposition rdf:type :CreditworthinessEvaluationCapability: {has_credit_pre} (expected False)") + if has_credit_pre: + print(" FAIL: decoy disposition asserted directly as CreditworthinessEvaluationCapability; " + "test premise (equivalence-only routing) is invalid.") + ok = False + print(f" pre-reasoning :WeirdCalcSystem_001 rdf:type :AnnexIII5bApplicableSystem: {is_5b_pre} (expected False)") + if is_5b_pre: + ok = False + + # Reason and re-check + g_post = reason(g_pre) + has_credit_post = (calc_disp, RDF["type"], credit_cap) in g_post + is_5b_post = (calc_system, RDF["type"], annex_5b) in g_post + is_1a_post = (calc_system, RDF["type"], annex_1a) in g_post + + print(f" post-reasoning :WeirdCalc_Disposition rdf:type :CreditworthinessEvaluationCapability: {has_credit_post} (expected True)") + if not has_credit_post: + ok = False + print(f" post-reasoning :WeirdCalcSystem_001 rdf:type :AnnexIII5bApplicableSystem: {is_5b_post} (expected True)") + if not is_5b_post: + ok = False + # Cross-category isolation check. + print(f" post-reasoning :WeirdCalcSystem_001 rdf:type :AnnexIII1aApplicableSystem: {is_1a_post} (expected False)") + if is_1a_post: + print(" FAIL: cross-category isolation broken; 5(b) decoy entailed into 1(a) class.") + ok = False + + if ok: + print(" RESULT: 5(b) classification routed via owl:equivalentClass, not via direct IRI.") + print(" Cross-category isolation preserved (no 1(a) entailment).") + return ok + + def test_ghost_classifies_via_blank_node_disposition() -> bool: """ Verify that GhostSystem_001's Gate 1 entailment fires via a blank-node @@ -215,6 +299,8 @@ def main() -> int: all_pass = True if not test_decoy_classifies_via_equivalent_class(): all_pass = False + if not test_credit_decoy_classifies_via_equivalent_class(): + all_pass = False if not test_ghost_classifies_via_blank_node_disposition(): all_pass = False @@ -222,7 +308,8 @@ def main() -> int: print("=" * 72) if all_pass: print("ALL ADVERSARIAL-MECHANISM TESTS PASSED") - print("Decoy classification routes via owl:equivalentClass.") + print("Decoy 1(a) classification routes via owl:equivalentClass.") + print("Decoy 5(b) classification routes via owl:equivalentClass; cross-category isolated.") print("Ghost classification routes via blank-node owl:someValuesFrom witness.") print("Neither relies on a direct IRI assertion or a named individual.") else: diff --git a/LIMITATIONS.md b/LIMITATIONS.md index 3d3e5b0..03041af 100644 --- a/LIMITATIONS.md +++ b/LIMITATIONS.md @@ -303,7 +303,7 @@ The pipeline's output layer (`run_pipeline.py` from line 1699 onward, plus `writ ## 9. Engineering gaps -- **Negative test infrastructure is incomplete.** Gate-removal regression tests ([test_gate_removal.py](03_TECHNICAL_CORE/scripts/test_gate_removal.py)) verify each gate is independently necessary for both modeled Annex III categories (1(a) Sentinel and 5(b) CreditScorer) by mutating axioms and confirming entailment breaks. Symmetric coverage across the two categories was added 2026-05-14. Adversarial-mechanism tests ([test_adversarial_mechanism.py](03_TECHNICAL_CORE/scripts/test_adversarial_mechanism.py)) verify that DecoySystem_001 classifies via `owl:equivalentClass` (not direct IRI assertion) and that GhostSystem_001 classifies via blank-node `owl:someValuesFrom` (not a named individual). What does **not** exist is a parameterized test harness that loads an isolated deliberately-miscategorized instance file and runs the full pipeline against it, because the pipeline currently loads all TTL files into a single graph before reasoning — a negative-case file in the graph would contaminate positive cases. Building this harness is Step 2 of the ADR-001 work plan. Noted in [docs/agent/bfo_cco_alignment_audit.md](docs/agent/bfo_cco_alignment_audit.md) §"Unresolved Engineering Problem." +- **Negative test infrastructure is incomplete.** Gate-removal regression tests ([test_gate_removal.py](03_TECHNICAL_CORE/scripts/test_gate_removal.py)) verify each gate is independently necessary for both modeled Annex III categories (1(a) Sentinel and 5(b) CreditScorer) by mutating axioms and confirming entailment breaks. Symmetric coverage across the two categories was added 2026-05-14. Adversarial-mechanism tests ([test_adversarial_mechanism.py](03_TECHNICAL_CORE/scripts/test_adversarial_mechanism.py)) verify three cases: DecoySystem_001 classifies via `owl:equivalentClass` on the Annex III 1(a) branch (not direct IRI assertion); WeirdCalcSystem_001 classifies via `owl:equivalentClass` on the 5(b) branch with cross-category isolation preserved (added 2026-05-20); and GhostSystem_001 classifies via blank-node `owl:someValuesFrom` (not a named individual). What does **not** exist is a parameterized test harness that loads an isolated deliberately-miscategorized instance file and runs the full pipeline against it, because the pipeline currently loads all TTL files into a single graph before reasoning — a negative-case file in the graph would contaminate positive cases. Building this harness is Step 2 of the ADR-001 work plan. Noted in [docs/agent/bfo_cco_alignment_audit.md](docs/agent/bfo_cco_alignment_audit.md) §"Unresolved Engineering Problem." - **Dependency pinning.** The pipeline is verified only on: `rdflib==7.6.0`, `pyshacl==0.31.0`, `owlrl==7.1.4`. Upgrading any of these requires re-running the full regression suite. The `owlrl` pin is especially load-bearing: Gate 2 and Gate 3 use anonymous inverse property restrictions whose entailment behavior could change across reasoner versions. - **`AnnexIII_Condition_1a cco:prescribes :RemoteBiometricIdentificationProcess`** remains in [ARCO_governance_extension.ttl](03_TECHNICAL_CORE/ontology/ARCO_governance_extension.ttl) (moved 2026-05-14 from the Sentinel instance file to the governance extension as universal regulatory content) as a class-as-individual triple retained for regulatory traceability. The companion 5(b) triple `:AnnexIII_Condition_5b cco:prescribes :CreditworthinessEvaluationProcess` is in the same file. Neither affects current classification. Both are known blockers for a future CCO import (see §4). - **Single-reasoner portability.** The anonymous inverse property expressions in Gate 2 and Gate 3 equivalentClass axioms have been empirically verified on `owlrl==7.1.4` only. Other OWL-RL reasoners may or may not materialize these the same way. diff --git a/README.md b/README.md index 976ae97..ab949ad 100644 --- a/README.md +++ b/README.md @@ -124,7 +124,7 @@ The architecture grounds in BFO 2020 (ISO/IEC 21838-2:2021) and uses the seven-b **The three layers do different jobs and are not interchangeable.** OWL-RL classifies (entails membership in the Annex III applicability classes). SHACL validates that the documentary record supporting a determination is structurally complete. SPARQL audits the post-reasoning graph and surfaces conditions for human review. A SHACL pass does not mean the system is high-risk. A SPARQL false does not overturn an OWL classification. The certificate's auditability turns on keeping the layers distinct. -**Actual OWL inference fires, not string matching.** One adversarial fixture types its capability disposition only as `:WeirdScanner`; the regulated class IRI never appears in the input data and the connection runs through an `owl:equivalentClass` declaration. Another fixture's disposition has no IRI at all (anonymous blank node, `ARCO_instances_adversarial_blanknode.ttl:28`). Both classify correctly because the reasoner performs actual OWL inference. An approach that did string matching on class names, or required named individuals at every position, would miss both. +**Actual OWL inference fires, not string matching.** Two adversarial fixtures, one per modeled Annex III category, type their capability dispositions only as a decoy class (`:WeirdScanner` for 1(a), `:WeirdCalculator` for 5(b)); the regulated class IRI never appears in the input data and the connection runs through an `owl:equivalentClass` declaration in each. A third fixture's disposition has no IRI at all (anonymous blank node, `ARCO_instances_adversarial_blanknode.ttl:28`). All three classify correctly because the reasoner performs actual OWL inference: the decoys via `owl:equivalentClass` propagation, the blank-node fixture via `owl:someValuesFrom` satisfaction. An approach that did string matching on class names, or required named individuals at every position, would miss them. **Layer separation is verified by fixtures.** Two flag-test fixtures present cases where all three Annex III gates are satisfied AND an audit-layer flag (a provider-asserted `:DerogationClaim`, or a `:FraudDetectionProcess` token) is also present. The OWL classification fires regardless of the audit flag; the flag fires alongside the classification. Classification and audit do not bleed into each other. From e8b04ffc587ac37ea0dc41d42f9c1b48572bb001 Mon Sep 17 00:00:00 2001 From: Alex Moskowitz Date: Wed, 20 May 2026 21:28:00 -0400 Subject: [PATCH 7/7] docs(readme): tighten adversarial-coverage claim MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit "the regulated class IRI never appears in the input data" is not literally true — the regulated class IRI does appear in the `owl:equivalentClass` declaration on the alias class. The safer claim is that the regulated class is not asserted as the disposition's type. The equivalentClass declaration is class-level, not instance-level, so the reasoner reaches the disposition's classification via class-equivalence propagation, not via direct type assertion. That distinction is what the test exercises. Co-Authored-By: Claude Opus 4.7 (1M context) --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index ab949ad..8cca2ed 100644 --- a/README.md +++ b/README.md @@ -124,7 +124,7 @@ The architecture grounds in BFO 2020 (ISO/IEC 21838-2:2021) and uses the seven-b **The three layers do different jobs and are not interchangeable.** OWL-RL classifies (entails membership in the Annex III applicability classes). SHACL validates that the documentary record supporting a determination is structurally complete. SPARQL audits the post-reasoning graph and surfaces conditions for human review. A SHACL pass does not mean the system is high-risk. A SPARQL false does not overturn an OWL classification. The certificate's auditability turns on keeping the layers distinct. -**Actual OWL inference fires, not string matching.** Two adversarial fixtures, one per modeled Annex III category, type their capability dispositions only as a decoy class (`:WeirdScanner` for 1(a), `:WeirdCalculator` for 5(b)); the regulated class IRI never appears in the input data and the connection runs through an `owl:equivalentClass` declaration in each. A third fixture's disposition has no IRI at all (anonymous blank node, `ARCO_instances_adversarial_blanknode.ttl:28`). All three classify correctly because the reasoner performs actual OWL inference: the decoys via `owl:equivalentClass` propagation, the blank-node fixture via `owl:someValuesFrom` satisfaction. An approach that did string matching on class names, or required named individuals at every position, would miss them. +**Actual OWL inference fires, not string matching.** Two adversarial fixtures, one per modeled Annex III category, type their capability dispositions only as a decoy class (`:WeirdScanner` for 1(a), `:WeirdCalculator` for 5(b)); the regulated class is not asserted as the disposition's type, and the connection runs through an `owl:equivalentClass` declaration in each. A third fixture's disposition has no IRI at all (anonymous blank node, `ARCO_instances_adversarial_blanknode.ttl:28`). All three classify correctly because the reasoner performs actual OWL inference: the decoys via `owl:equivalentClass` propagation, the blank-node fixture via `owl:someValuesFrom` satisfaction. An approach that did string matching on class names, or required named individuals at every position, would miss them. **Layer separation is verified by fixtures.** Two flag-test fixtures present cases where all three Annex III gates are satisfied AND an audit-layer flag (a provider-asserted `:DerogationClaim`, or a `:FraudDetectionProcess` token) is also present. The OWL classification fires regardless of the audit flag; the flag fires alongside the classification. Classification and audit do not bleed into each other.