feat: invert auth model — all routes protected, whitelist public paths

- Everything requires token by default when auth is enabled
- New WEB2API_PUBLIC_PATHS env var for whitelisting (glob patterns)
- Only / and /health are public by default
- Updated README with new auth model and examples
- Updated tests for new auth behavior

Author: Endogen

README.md

@@ -108,35 +108,50 @@ docker compose exec web2api web2api recipes catalog add hackernews --yes
 
 ## Access Token
 
-Web2API can protect the sensitive management and MCP surfaces with a shared access token.
+Web2API can protect all HTTP routes except selected public paths with a shared access token.
 
 Set one of:
 - `WEB2API_ACCESS_TOKEN`
 - `WEB2API_ACCESS_TOKEN_FILE` (path to a file containing the token)
 
-When configured, Web2API requires the token for:
-- `/api/recipes/manage*`
-- `/mcp*`
+By default, when configured, Web2API requires the token for everything except:
+- `/`
+- `/health`
+
+You can keep extra routes public while token auth stays enabled by setting
+`WEB2API_PUBLIC_PATHS` to a comma- or newline-separated list of exact paths or shell-style
+glob patterns matched against the request path.
+
+Common examples:
+- `/api/sites`
+- `/docs`
+- `/openapi.json`
+- `/allenai/*`
+- `/*/chat`
+
+Any path that matches one of those patterns skips token auth, so use this allowlist sparingly.
 
 Send the token as either:
 - `Authorization: Bearer <token>`
 - `X-Web2API-Key: <token>`
 
-Public scrape/discovery routes remain open:
-- `/`
-- `/health`
-- `/api/sites`
-- `/{slug}/{endpoint}`
+Example mixed setup:
+
+```bash
+export WEB2API_ACCESS_TOKEN="secret-token"
+export WEB2API_PUBLIC_PATHS="/api/sites,/allenai/*,/docs,/openapi.json"
+```
 
-Examples:
+Authenticated request examples:
 
 ```bash
+curl -H "Authorization: Bearer $WEB2API_ACCESS_TOKEN" http://localhost:8010/allenai/chat?q=example&page=1
 curl -H "Authorization: Bearer $WEB2API_ACCESS_TOKEN" http://localhost:8010/api/recipes/manage
 curl -H "Authorization: Bearer $WEB2API_ACCESS_TOKEN" http://localhost:8010/mcp/tools
 ```
 
 When token auth is enabled, the built-in web UI shows an access-token input and stores the token in
-browser local storage for repository/MCP actions.
+browser local storage for protected browser actions.
 
 ## CLI
 
@@ -387,15 +402,15 @@ A simpler HTTP-based tool bridge is also available for non-MCP clients:
 
 | Endpoint | Description |
 |---|---|
-| `GET /` | HTML index listing all recipes and endpoints |
-| `GET /health` | Service, browser pool, and cache health |
-| `GET /api/sites` | JSON list of all recipes with endpoint metadata |
-| `GET /api/recipes/manage` | JSON catalog + installed recipe state for UI/automation (protected when token auth is enabled) |
-| `POST /api/recipes/manage/install/{name}` | Install recipe by catalog entry name (protected when token auth is enabled) |
-| `POST /api/recipes/manage/update/{slug}` | Update installed managed recipe (protected when token auth is enabled) |
-| `POST /api/recipes/manage/uninstall/{slug}` | Uninstall recipe (add `?force=true` for unmanaged local recipes, protected when token auth is enabled) |
-| `POST /api/recipes/manage/enable/{slug}` | Enable installed recipe (protected when token auth is enabled) |
-| `POST /api/recipes/manage/disable/{slug}` | Disable installed recipe (protected when token auth is enabled) |
+| `GET /` | HTML index listing all recipes and endpoints (always public) |
+| `GET /health` | Service, browser pool, and cache health (always public) |
+| `GET /api/sites` | JSON list of all recipes with endpoint metadata (protected by default when token auth is enabled) |
+| `GET /api/recipes/manage` | JSON catalog + installed recipe state for UI/automation (protected by default when token auth is enabled) |
+| `POST /api/recipes/manage/install/{name}` | Install recipe by catalog entry name (protected by default when token auth is enabled) |
+| `POST /api/recipes/manage/update/{slug}` | Update installed managed recipe (protected by default when token auth is enabled) |
+| `POST /api/recipes/manage/uninstall/{slug}` | Uninstall recipe (add `?force=true` for unmanaged local recipes, protected by default when token auth is enabled) |
+| `POST /api/recipes/manage/enable/{slug}` | Enable installed recipe (protected by default when token auth is enabled) |
+| `POST /api/recipes/manage/disable/{slug}` | Disable installed recipe (protected by default when token auth is enabled) |
 
 `GET /api/recipes/manage` includes:
 - `catalog`: entries from the current catalog source
@@ -405,6 +420,7 @@ A simpler HTTP-based tool bridge is also available for non-MCP clients:
 ### Recipe Endpoints
 
 All recipe endpoints follow the pattern: `GET /{slug}/{endpoint}?page=1&q=...`
+and require the access token by default when token auth is enabled.
 
 - `page` — pagination (default: 1)
 - `q` — query text (required when `requires_query: true`)
@@ -644,8 +660,9 @@ Environment variables (with defaults):
 | `WEB2API_RECIPE_CATALOG_REF` | empty | Optional git ref for catalog source |
 | `WEB2API_RECIPE_CATALOG_PATH` | `catalog.yaml` | Catalog file path inside catalog source |
 | `PLUGIN_ENFORCE_COMPATIBILITY` | false | Skip plugin recipes outside declared `web2api` version bounds |
-| `WEB2API_ACCESS_TOKEN` | empty | Shared access token for admin API and MCP endpoints |
+| `WEB2API_ACCESS_TOKEN` | empty | Shared access token for all routes except public paths |
 | `WEB2API_ACCESS_TOKEN_FILE` | empty | Path to file containing the access token (alternative to `WEB2API_ACCESS_TOKEN`) |
+| `WEB2API_PUBLIC_PATHS` | empty | Extra public path patterns to allow without auth while token auth is enabled |
 | `BIRD_AUTH_TOKEN` | empty | X/Twitter auth token for `x` recipe |
 | `BIRD_CT0` | empty | X/Twitter ct0 token for `x` recipe |
 

docker-compose.yml

@@ -24,6 +24,7 @@ services:
       WEB2API_RECIPE_CATALOG_REF: "${WEB2API_RECIPE_CATALOG_REF:-}"
       WEB2API_RECIPE_CATALOG_PATH: "${WEB2API_RECIPE_CATALOG_PATH:-}"
       WEB2API_ACCESS_TOKEN: "${WEB2API_ACCESS_TOKEN:-}"
+      WEB2API_PUBLIC_PATHS: "${WEB2API_PUBLIC_PATHS:-}"
       LOG_LEVEL: "${LOG_LEVEL:-info}"
       BIRD_AUTH_TOKEN: "${BIRD_AUTH_TOKEN:-}"
       BIRD_CT0: "${BIRD_CT0:-}"

tests/integration/test_api.py

@@ -397,7 +397,7 @@ async def fake_scrape(
 
 
 @pytest.mark.asyncio
-async def test_access_token_protects_admin_and_mcp_surfaces(
+async def test_access_token_protects_all_routes_except_public_surfaces(
     tmp_path: Path,
     monkeypatch: pytest.MonkeyPatch,
 ) -> None:
@@ -427,17 +427,18 @@ async def fake_scrape(
         transport = ASGITransport(app=app)
         async with AsyncClient(transport=transport, base_url="http://testserver") as client:
             public_resp = await client.get("/alpha/read")
-            assert public_resp.status_code == 200
+            assert public_resp.status_code == 401
 
             sites_resp = await client.get("/api/sites")
-            assert sites_resp.status_code == 200
+            assert sites_resp.status_code == 401
 
             health_resp = await client.get("/health")
             assert health_resp.status_code == 200
 
             index_resp = await client.get("/")
             assert index_resp.status_code == 200
             assert "Paste access token" in index_resp.text
+            assert "public paths shown below" in index_resp.text
 
             manage_resp = await client.get("/api/recipes/manage")
             assert manage_resp.status_code == 401
@@ -470,6 +471,76 @@ async def fake_scrape(
             )
             assert authorized_mcp.status_code == 200
 
+            authorized_sites = await client.get(
+                "/api/sites",
+                headers={"Authorization": "Bearer secret-token"},
+            )
+            assert authorized_sites.status_code == 200
+
+            authorized_recipe = await client.get(
+                "/alpha/read",
+                headers={"Authorization": "Bearer secret-token"},
+            )
+            assert authorized_recipe.status_code == 200
+
+
+@pytest.mark.asyncio
+async def test_access_token_allows_configured_public_path_patterns(
+    tmp_path: Path,
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    recipes_dir = tmp_path / "recipes"
+    _write_recipe(recipes_dir, "alpha")
+    _write_recipe(recipes_dir, "beta")
+
+    async def fake_scrape(
+        *,
+        pool: FakePool,
+        recipe,
+        endpoint: str,
+        page: int = 1,
+        query: str | None = None,
+        extra_params: dict[str, str] | None = None,
+        scrape_timeout: float = 30.0,
+    ) -> ApiResponse:
+        _ = pool, endpoint, query, extra_params, scrape_timeout
+        return _success_response(slug=recipe.config.slug, endpoint="read", page=page)
+
+    monkeypatch.setattr("web2api.main.scrape", fake_scrape)
+    monkeypatch.setenv("WEB2API_ACCESS_TOKEN", "secret-token")
+    monkeypatch.setenv("WEB2API_PUBLIC_PATHS", "/api/sites,/alpha/*")
+
+    fake_pool = FakePool()
+    app = create_app(recipes_dir=recipes_dir, pool=fake_pool)
+
+    async with app.router.lifespan_context(app):
+        transport = ASGITransport(app=app)
+        async with AsyncClient(transport=transport, base_url="http://testserver") as client:
+            index_resp = await client.get("/")
+            assert index_resp.status_code == 200
+            assert "/alpha/*" in index_resp.text
+
+            health_resp = await client.get("/health")
+            assert health_resp.status_code == 200
+
+            sites_resp = await client.get("/api/sites")
+            assert sites_resp.status_code == 200
+
+            alpha_resp = await client.get("/alpha/read")
+            assert alpha_resp.status_code == 200
+
+            beta_resp = await client.get("/beta/read")
+            assert beta_resp.status_code == 401
+
+            manage_resp = await client.get("/api/recipes/manage")
+            assert manage_resp.status_code == 401
+
+            authorized_beta = await client.get(
+                "/beta/read",
+                headers={"Authorization": "Bearer secret-token"},
+            )
+            assert authorized_beta.status_code == 200
+
 
 @pytest.mark.asyncio
 async def test_response_cache_serves_fresh_and_stale_results(

tests/unit/test_auth.py

@@ -7,7 +7,7 @@
 import pytest
 from starlette.datastructures import Headers
 
-from web2api.auth import AuthConfig, load_auth_config, request_is_authorized
+from web2api.auth import AuthConfig, load_auth_config, public_auth_payload, request_is_authorized
 
 
 def test_auth_config_loads_token_from_file(
@@ -23,6 +23,29 @@ def test_auth_config_loads_token_from_file(
 
     assert config.enabled is True
     assert config.access_token == "secret-token"
+    assert config.public_path_patterns == ("/", "/health")
+
+
+def test_auth_config_loads_extra_public_paths_from_env(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    monkeypatch.delenv("WEB2API_ACCESS_TOKEN_FILE", raising=False)
+    monkeypatch.setenv("WEB2API_ACCESS_TOKEN", "secret-token")
+    monkeypatch.setenv(
+        "WEB2API_PUBLIC_PATHS",
+        "/api/sites, /allenai/*\n/docs,/openapi.json",
+    )
+
+    config = load_auth_config()
+
+    assert config.public_path_patterns == (
+        "/",
+        "/health",
+        "/api/sites",
+        "/allenai/*",
+        "/docs",
+        "/openapi.json",
+    )
 
 
 def test_request_is_authorized_supports_bearer_and_alt_header() -> None:
@@ -40,3 +63,50 @@ def test_request_is_authorized_supports_bearer_and_alt_header() -> None:
         Headers({"authorization": "Bearer wrong"}),
         config,
     ) is False
+
+
+def test_auth_config_requires_auth_for_all_non_public_routes() -> None:
+    config = AuthConfig(access_token="secret-token")
+
+    assert config.requires_auth("/") is False
+    assert config.requires_auth("/health") is False
+    assert config.requires_auth("/health/") is False
+    assert config.requires_auth("/api/sites") is True
+    assert config.requires_auth("/alpha/read") is True
+    assert config.requires_auth("/mcp/tools") is True
+
+
+def test_auth_config_matches_additional_public_path_patterns() -> None:
+    config = AuthConfig(
+        access_token="secret-token",
+        public_path_patterns=("/", "/health", "/api/sites", "/allenai/*", "/*/chat"),
+    )
+
+    assert config.requires_auth("/api/sites") is False
+    assert config.requires_auth("/allenai/chat") is False
+    assert config.requires_auth("/foo/chat") is False
+    assert config.requires_auth("/foo/read") is True
+
+
+def test_auth_config_rejects_invalid_public_path_patterns(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    monkeypatch.setenv("WEB2API_ACCESS_TOKEN", "secret-token")
+    monkeypatch.setenv("WEB2API_PUBLIC_PATHS", "api/sites")
+
+    with pytest.raises(ValueError, match="WEB2API_PUBLIC_PATHS entries must start with '/'"):
+        load_auth_config()
+
+
+def test_public_auth_payload_describes_public_and_protected_surfaces() -> None:
+    payload = public_auth_payload(
+        AuthConfig(
+            access_token="secret-token",
+            public_path_patterns=("/", "/health", "/api/sites"),
+        )
+    )
+
+    assert payload["enabled"] is True
+    assert payload["protected_surfaces"] == ["all routes except configured public path patterns"]
+    assert payload["public_surfaces"] == ["/", "/health", "/api/sites"]
+    assert payload["public_paths_env"] == "WEB2API_PUBLIC_PATHS"