refactor: split python backend and plugin orchestration boundaries

[maehwasoo]

Problem statement

packages/obsidian-plugin/src/main.ts, apps/api/src/ailss_api/agent.py, and apps/api/src/ailss_api/retrieval_common.py currently mix multiple concerns in a way that makes the Python-first local backend baseline hard to evolve safely.

• main.ts owns plugin bootstrap, settings/token persistence, MCP and Python service lifecycle wiring, Python backend command handlers, and result formatting.
• agent.py owns workflow entry, graph topology, guard/policy logic, retrieval orchestration, answer construction, local note reads, and artifact writing.
• retrieval_common.py owns DB metadata access, chunk stitching, file preview I/O, lexical scoring, and text parsing helpers.
• pythonApiServiceController.ts and utils/spawn.ts are also accumulating startup/runtime concerns and duplicate service-supervision patterns.

This increases divergent change risk: backend workflow changes, plugin UX changes, and runtime/service changes all require edits across the same large files.

Refactor goal

Split the current Python-backend-related code by responsibility while preserving existing user-visible behavior.

• Keep the Obsidian plugin entrypoint thin.
• Isolate Python backend command flows and result formatting from plugin lifecycle code.
• Separate backend workflow topology/policy from retrieval/file-I/O concerns.
• Break retrieval helpers into focused modules for DB access, preview/stitching, and scoring/text helpers.
• Reduce duplicated service-supervision growth across Python and MCP controllers.

In scope

• Extract Python backend command handlers and text formatters out of packages/obsidian-plugin/src/main.ts.
• Extract workflow/policy helpers from apps/api/src/ailss_api/agent.py into smaller modules.
• Split apps/api/src/ailss_api/retrieval_common.py into focused helpers for metadata queries, chunk/preview assembly, and lexical scoring/text utilities.
• Reduce responsibility width in packages/obsidian-plugin/src/pythonApi/pythonApiServiceController.ts and packages/obsidian-plugin/src/utils/spawn.ts.
• Deliver this incrementally in small PRs:
  • Phase 1: main.ts
  • Phase 2: agent.py and retrieval_common.py
  • Phase 3: pythonApiServiceController.ts and spawn.ts

Out of scope

• No schema change.
• No request/response contract change for the Python API.
• No public CLI behavior change.
• No new write-capable agent behavior.
• No retrieval ranking/model behavior change beyond extraction required for refactoring.
• No UX redesign.

Validation

• pnpm check
• Manual smoke checks for plugin load/unload, MCP service start/stop, Python backend start/stop, health check, eval, retrieval, and agent commands
• Existing Python tests remain green via pnpm py:test
• Existing TypeScript tests remain green via pnpm test

Done criteria

• The Python backend baseline no longer depends on a single large plugin entry file for orchestration and formatting.
• Agent workflow, retrieval helpers, and service/runtime helpers are split into focused modules with clearer boundaries.
• Existing behavior remains unchanged from a user perspective.
• Existing checks/tests pass.
• Follow-up refactor PRs can be scoped to one responsibility area at a time instead of touching multiple large files at once.