feat: add plugin lifecycle CLI and release packaging assets

Author: Endogen

README.md

@@ -2,13 +2,14 @@
 
 Turn any website into a REST API by scraping it live with Playwright.
 
-Web2API loads recipe folders from `recipes/` at startup. Each recipe defines endpoints with selectors, actions, fields, and pagination in YAML. Optional Python scrapers handle interactive or complex sites. Drop a folder — get an API.
+Web2API loads recipe folders from `recipes/` at startup. Each recipe defines endpoints with selectors, actions, fields, and pagination in YAML. Optional Python scrapers handle interactive or complex sites. Optional plugin metadata can declare external dependencies and required env vars. Drop a folder — get an API.
 
 ## Features
 
 - **Arbitrary named endpoints** — recipes define as many endpoints as needed (not limited to read/search)
 - **Declarative YAML recipes** with selectors, actions, transforms, and pagination
 - **Custom Python scrapers** for interactive sites (e.g. typing text, waiting for dynamic content)
+- **Optional plugin metadata** (`plugin.yaml`) for recipe-specific dependency requirements
 - **Shared browser/context pool** for concurrent Playwright requests
 - **In-memory response cache** with stale-while-revalidate
 - **Unified JSON response schema** across all recipes and endpoints
@@ -29,6 +30,82 @@ curl -s http://localhost:8010/health | jq
 curl -s http://localhost:8010/api/sites | jq
 ```
 
+## CLI
+
+Web2API ships with a management CLI:
+
+```bash
+web2api --help
+```
+
+### Plugin Commands
+
+```bash
+# List all recipe folders with plugin readiness
+web2api plugins list
+
+# Check missing env vars/commands/packages
+web2api plugins doctor
+web2api plugins doctor x
+web2api plugins doctor x --no-run-healthchecks
+web2api plugins doctor x --allow-untrusted
+
+# Install plugin recipe from source
+web2api plugins add ./my-recipe
+web2api plugins add https://github.com/acme/web2api-recipes.git --ref v1.2.0 --subdir recipes/news
+
+# Update managed plugin from recorded source
+web2api plugins update x --yes
+web2api plugins update x --ref v1.3.0 --subdir recipes/x --yes
+
+# Install plugin recipe from catalog
+web2api plugins catalog list
+web2api plugins catalog add hackernews --yes
+
+# Install declared dependencies for a plugin recipe (host)
+web2api plugins install x --yes
+web2api plugins install x --apt --yes   # include apt packages
+
+# Generate Dockerfile snippet for plugin dependencies
+web2api plugins install x --target docker --apt
+
+# Remove plugin recipe + manifest record
+web2api plugins uninstall x --yes
+
+# Disable/enable a recipe (writes/removes recipes/<slug>/.disabled)
+web2api plugins disable x --yes
+web2api plugins enable x
+```
+
+`plugins install` does not run `apt` installs unless `--apt` is explicitly passed.
+Install-state records are stored in `recipes/.web2api_plugins.json`.
+Default catalog path is `plugins/catalog.yaml` in a source checkout, with a bundled fallback
+inside the installed package.
+`plugins update` works only for plugins tracked in the manifest.
+
+Plugins installed from untrusted sources (for example git URLs) are blocked from executing
+install/healthcheck commands unless `--allow-untrusted` is passed.
+
+### Self Update Commands
+
+```bash
+# Show current version + recommended update method
+web2api self update check
+
+# Apply update using auto-detected method (pip/git/docker)
+web2api self update apply --yes
+
+# Pin explicit method or target version/ref
+web2api self update apply --method pip --to 0.1.0 --yes
+web2api self update apply --method git --to v0.1.0 --yes
+```
+
+For `--method git`, `self update apply` checks out a tag:
+- if `--to` is provided, that tag/ref is used
+- if `--to` is omitted, the latest sortable git tag is used
+
+After `self update apply`, the CLI automatically runs `web2api plugins doctor`.
+
 ## Discover Recipes
 
 Recipe availability is dynamic. Use discovery endpoints instead of relying on a static README list.
@@ -137,11 +214,13 @@ recipes/
   <slug>/
     recipe.yaml     # required — endpoint definitions
     scraper.py      # optional — custom Python scraper
+    plugin.yaml     # optional — dependency metadata and runtime checks
     README.md       # optional — documentation
 ```
 
 - Folder name must match `slug`
 - `slug` cannot be a reserved system route (`api`, `health`, `docs`, `openapi`, `redoc`)
+- Recipe folders containing `.disabled` are skipped by discovery
 - Restart the service to pick up new or changed recipes
 - Invalid recipes are skipped with warning logs
 
@@ -254,6 +333,43 @@ class Scraper(BaseScraper):
 - `params` also includes validated extra query params (for example `count`)
 - Endpoints not handled by the scraper fall back to declarative YAML
 
+### Plugin Metadata (Optional)
+
+Use `plugin.yaml` to declare install/runtime requirements for a recipe:
+
+```yaml
+version: "1.0.0"
+web2api:
+  min: "0.2.0"
+  max: "1.0.0"
+requires_env:
+  - BIRD_AUTH_TOKEN
+  - BIRD_CT0
+dependencies:
+  commands:
+    - bird
+  python:
+    - httpx
+  apt:
+    - nodejs
+  npm:
+    - "@steipete/bird"
+healthcheck:
+  command: ["bird", "--version"]
+```
+
+Version bounds in `web2api.min` / `web2api.max` use numeric `major.minor.patch` format.
+
+`GET /api/sites` now includes a `plugin` block (or `null`) with:
+
+- declared metadata from `plugin.yaml`
+- computed `status.ready` plus missing env vars/commands/python packages
+- unverified package declarations (`apt`, `npm`) for operators
+
+Compatibility enforcement:
+- `PLUGIN_ENFORCE_COMPATIBILITY=false` (default): incompatible plugins are loaded but reported as not ready.
+- `PLUGIN_ENFORCE_COMPATIBILITY=true`: incompatible plugins are skipped at discovery time.
+
 ## Configuration
 
 Environment variables (with defaults):
@@ -270,7 +386,8 @@ Environment variables (with defaults):
 | `CACHE_TTL_SECONDS` | 30 | Fresh cache duration in seconds |
 | `CACHE_STALE_TTL_SECONDS` | 120 | Stale-while-revalidate window in seconds |
 | `CACHE_MAX_ENTRIES` | 500 | Maximum cached request variants |
-| `RECIPES_DIR` | `./recipes` | Path to recipes directory |
+| `RECIPES_DIR` | `./recipes` (or bundled defaults in installed package) | Path to recipes directory |
+| `PLUGIN_ENFORCE_COMPATIBILITY` | false | Skip plugin recipes outside declared `web2api` version bounds |
 | `BIRD_AUTH_TOKEN` | empty | X/Twitter auth token for `x` recipe |
 | `BIRD_CT0` | empty | X/Twitter ct0 token for `x` recipe |
 

plugins/catalog.yaml

@@ -0,0 +1,15 @@
+plugins:
+  hackernews:
+    description: "Built-in Hacker News recipe."
+    source: "../recipes/hackernews"
+    trusted: true
+
+  deepl:
+    description: "Built-in DeepL translation recipe."
+    source: "../recipes/deepl"
+    trusted: true
+
+  x:
+    description: "Built-in X/Twitter recipe (requires bird CLI and auth env vars)."
+    source: "../recipes/x"
+    trusted: true

pyproject.toml

@@ -13,6 +13,7 @@ dependencies = [
   "playwright>=1.50,<2.0",
   "pydantic>=2.10,<3.0",
   "pyyaml>=6.0,<7.0",
+  "typer>=0.12,<1.0",
   "uvicorn[standard]>=0.34,<1.0",
 ]
 
@@ -26,9 +27,23 @@ dev = [
   "ruff>=0.9,<1.0",
 ]
 
+[project.scripts]
+web2api = "web2api.cli:main"
+
+[tool.setuptools]
+include-package-data = true
+
 [tool.setuptools.packages.find]
 include = ["web2api*"]
 
+[tool.setuptools.package-data]
+web2api = [
+  "templates/*.html",
+  "bundled/plugins/*.yaml",
+  "bundled/recipes/*/*.yaml",
+  "bundled/recipes/*/*.py",
+]
+
 [tool.ruff]
 line-length = 100
 target-version = "py312"

recipes/x/plugin.yaml

@@ -0,0 +1,17 @@
+version: "1.0.0"
+web2api:
+  min: "0.1.0"
+requires_env:
+  - BIRD_AUTH_TOKEN
+  - BIRD_CT0
+dependencies:
+  commands:
+    - bird
+  apt:
+    - nodejs
+  npm:
+    - "@steipete/bird"
+healthcheck:
+  command:
+    - bird
+    - --version

recipes/x/scraper.py

@@ -89,12 +89,13 @@ async def scrape(self, endpoint: str, page: Page, params: dict[str, Any]) -> Scr
 
         items: list[dict[str, Any]] = []
         for tweet in tweets_data[:count]:
+            author_username = tweet.get("author", {}).get("username", username)
             items.append({
                 "text": tweet.get("text", ""),
-                "author": tweet.get("author", {}).get("username", username),
+                "author": author_username,
                 "author_name": tweet.get("author", {}).get("name", ""),
                 "timestamp": tweet.get("createdAt", ""),
-                "url": f"https://x.com/{tweet.get('author', {}).get('username', username)}/status/{tweet.get('id', '')}",
+                "url": f"https://x.com/{author_username}/status/{tweet.get('id', '')}",
                 "replies": tweet.get("replyCount"),
                 "reposts": tweet.get("retweetCount"),
                 "likes": tweet.get("likeCount"),

tests/integration/test_api.py

@@ -51,6 +51,7 @@ def _write_recipe(
     recipes_dir: Path,
     slug: str,
     endpoints: dict[str, dict] | None = None,
+    plugin: dict[str, object] | None = None,
 ) -> None:
     if endpoints is None:
         endpoints = {
@@ -75,6 +76,11 @@ def _write_recipe(
         ),
         encoding="utf-8",
     )
+    if plugin is not None:
+        (recipe_dir / "plugin.yaml").write_text(
+            yaml.safe_dump(plugin),
+            encoding="utf-8",
+        )
 
 
 def _success_response(
@@ -147,20 +153,31 @@ async def test_api_routes_and_index(
     caplog: pytest.LogCaptureFixture,
 ) -> None:
     recipes_dir = tmp_path / "recipes"
+    missing_env = "WEB2API_TEST_ALPHA_TOKEN_UNLIKELY"
+    monkeypatch.delenv(missing_env, raising=False)
 
-    _write_recipe(recipes_dir, "alpha", endpoints={
-        "read": {
-            "url": "https://example.com/items?page={page}",
-            "items": {"container": ".item", "fields": {"title": {"selector": ".title"}}},
-            "pagination": {"type": "page_param", "param": "page"},
+    _write_recipe(
+        recipes_dir,
+        "alpha",
+        endpoints={
+            "read": {
+                "url": "https://example.com/items?page={page}",
+                "items": {"container": ".item", "fields": {"title": {"selector": ".title"}}},
+                "pagination": {"type": "page_param", "param": "page"},
+            },
+            "search": {
+                "url": "https://example.com/search?q={query}&page={page}",
+                "requires_query": True,
+                "items": {"container": ".item", "fields": {"title": {"selector": ".title"}}},
+                "pagination": {"type": "page_param", "param": "page"},
+            },
         },
-        "search": {
-            "url": "https://example.com/search?q={query}&page={page}",
-            "requires_query": True,
-            "items": {"container": ".item", "fields": {"title": {"selector": ".title"}}},
-            "pagination": {"type": "page_param", "param": "page"},
+        plugin={
+            "version": "1.0.0",
+            "requires_env": [missing_env],
+            "dependencies": {"commands": ["missing-web2api-plugin-command"]},
         },
-    })
+    )
     _write_recipe(recipes_dir, "beta")  # read only
 
     async def fake_scrape(
@@ -251,6 +268,16 @@ async def fake_scrape(
                 alpha_site = next(s for s in sites_resp.json() if s["slug"] == "alpha")
                 ep_names = {ep["name"] for ep in alpha_site["endpoints"]}
                 assert ep_names == {"read", "search"}
+                assert alpha_site["plugin"]["version"] == "1.0.0"
+                assert alpha_site["plugin"]["status"]["ready"] is False
+                assert missing_env in alpha_site["plugin"]["status"]["checks"]["env"]["missing"]
+                assert (
+                    "missing-web2api-plugin-command"
+                    in alpha_site["plugin"]["status"]["checks"]["commands"]["missing"]
+                )
+
+                beta_site = next(s for s in sites_resp.json() if s["slug"] == "beta")
+                assert beta_site["plugin"] is None
 
                 # Health check
                 health_resp = await client.get("/health")