Prod Clerk cutover: follow-up cleanup and verification

[ChesterSchendel]

Context

Tonight (2026-05-28) we cut runladder.com production over from dev Clerk (pk_test_) to prod Clerk (pk_live_, clerk.runladder.com). The cutover itself works (Google branding confirmed, prod webhook wired, provisioning service account in place — user_3ENqFcFXNNjstCfR5SCiMX2fWhV), but the prod Clerk database starts empty. Everything that lived on dev Clerk — users, orgs, memberships, tokens — does not exist on prod.

This issue tracks the cleanup and verification work needed to fully stand up prod. Items marked blocking should be done before the client demo (~7-10 days out).

Checklist

Internal Drawbackwards org

• Blocking. Re-create the internal Drawbackwards org on prod cleanly (the dev version was cobbled together via the Comps feature). Provision via /admin/clients so it uses the service account createdBy pattern.
• Set the org's internal: true metadata so the dashboard suppresses the "Complimentary Team" plan strip (see isInternalOrg in src/lib/orgs.ts). Without this, internal members will see the comp-team framing meant for external clients.
• Re-invite all Drawbackwards team members (Ward, Chester, Michael, Jordan, Sara, Sean) with appropriate org:admin / org:member roles. Each one needs to create a fresh prod Clerk account when they accept.

Figma Plugin (drawbackwards/ai-design-assistant)

• Blocking. Verify the plugin works end-to-end against prod Clerk. Plugin tokens are issued via POST /api/plugin/issue-token (Clerk session required) and verified via POST /api/plugin/verify-token. Existing plugin tokens issued under dev Clerk userIds are orphaned — anyone using the plugin needs to re-auth.
• Confirm /api/plugin/analyze and /api/plugin/persist-score work against prod.
• Test the in-plugin auth flow: log in to prod runladder.com, generate token, paste into Figma plugin, score a frame.
• Update the plugin marketplace listing if anything changed (probably nothing — same endpoints).

Claude Skill (Ladder for Claude)

• Blocking. Verify the Skill works end-to-end against prod Clerk. Skill tokens issued via POST /api/skill/token (Clerk session required). Same story as plugin: existing tokens orphaned.
• Confirm /api/skill/score works against prod.
• Re-issue Ward's skill token from his new prod account so his Claude.ai install keeps working.

Stripe

• Blocking. Audit Stripe customer / subscription records. Stripe customer IDs are linked to Clerk userIds in our system. Any paid subscription created during the dev Clerk period is now orphaned (the new prod userIds won't match the Stripe customer records).
• Decide policy: re-link manually for the small number of paid users we have, or accept resub for everyone.
• Verify Stripe is in live mode in prod env vars (not test mode). This is the analog to the Clerk dev/prod split — same audit. See related dev-environments task Stand up a Dev, Stage, and Prod Environment #249.

Webhooks

• Confirm the prod Clerk webhook is firing on the right events. Hit it manually with a test event from the Clerk dashboard if available.
• Confirm organizationMembership.created actually flips new members to tier: team end-to-end on prod (the tier flip path is what makes invited designers see the Team UI).
• Audit any unprocessed dev Clerk webhook events — they're gone, but check if anything was mid-flight.

Admin / role gating

• Verify ADMIN_EMAILS env var is set in Vercel Production scope and contains the right Drawbackwards email addresses. The env var is not Clerk-tied, but the gate runs the lookup against the Clerk session, so it needs to match the email on each user's new prod account.
• Confirm /admin/* access works for Ward + Chester after they create their new prod accounts.

Beta codes / Comps

• Beta codes live in Redis (separate from Clerk), so they survive the cutover. Verify the codes list at /admin/beta-codes still resolves correctly.
• Re-issue any active comp subscriptions that were attached to dev Clerk userIds (likely zero outside the Drawbackwards org).

Site password gate

• Confirm SITE_PASSWORD still works on prod (it's environment-scoped but not Clerk-tied).
• Verify the bypass-for-signed-in-users logic (/hq/decisions entry) still works after the Clerk swap.

In-flight dev invitations

• Any org invitations sent via dev Clerk before tonight's cutover are now broken — they point at the dev Clerk Accounts portal, which can't sign users into prod runladder.com. Tell anyone with a pending invite to ask for a fresh one.

Other surfaces (lower priority)

• Ladder API + MCP — roadmap, not built yet. No cutover impact today. See /hq/journeys API/MCP section.
• Ladder Pulse — runs in a separate codebase (ladder-beta), not affected by this cutover.

Closing tasks

• Close Onboarding flow is completely broken due to incorrect Clerk config #181 (post-accept redirect dead-end — should be fixed by prod Clerk having a real Home URL).
• Close Configure Clerk settings so that connecting to Google says, "Connect to Ladder" (instead of Clerk) #230 (custom Google OAuth credentials — done tonight, just needs verification).
• Close Write a better error message for this (and style it better) so that Clerk isn't referenced. #218 (org invitation deliverability — should improve with prod Clerk + verified domain).
• Close Update Ladder logo in all emails so it includes the DB bug #214 if applicable.
• Update /hq/journeys to remove the dev-Clerk caveats now that prod is real.
• Update /hq/decisions with a new entry summarizing the cutover and what changed.

Notes

This list is comprehensive but every item is small. Bulk of the work is the Drawbackwards org re-setup and the plugin/Skill verification. Stripe audit is the riskiest because money is involved — do that one carefully.

[ChesterSchendel]

Shared Redis across surfaces (Chester, 2026-05-29)

Confirmed while splitting runladder's non-prod Redis: the Figma plugin (ai-design-assistant) and the web app share the same prod Redis instance (redis-champagne-house, 30MB). This is by design — per CLAUDE.md, all surfaces share one engine, one auth, one billing, one database, so a user's lifetime cap / score history / usage meter is unified regardless of which surface they score from.

Implications for the cutover + future dev environment work:

• Prod Redis is shared by runladder web + Figma plugin (and the Claude Skill writes here too). Anything that touches Redis schema or key namespacing affects all surfaces.
• The two team-level Upstash DBs (redis-champagne-house = real Redis/30MB, upstash-kv-camel-drum = Free) are both connected to ai-design-assistant via the Vercel integration. runladder reaches Redis via manual env vars (no integration link on the runladder project), pointing at the shared instance.
• Plugin/Skill verification (existing checklist items): because tokens are now issued against prod Clerk, plugin/Skill scores write to the shared prod Redis keyed by the new prod Clerk userIds. Verify a plugin score and a web score from the same prod account both increment the same user:{userId}:lifetime_scans_used counter.
• Future dev environment for the plugin: when ai-design-assistant gets a dev environment, it must point at the same runladder-dev Redis as the web app's non-prod scopes, or cross-surface testing breaks (a plugin score wouldn't count against the same user's cap as a web score). Coordinate the plugin's dev Redis with runladder's. See related env work in Stand up a Dev, Stage, and Prod Environment #249.

[ChesterSchendel]

Env var audit findings (2026-05-29)

Grepped code for process.env.* and compared to Vercel. New gaps found:

• TEAM_EMAILS — not set in Vercel. Gates /hq access. With it unset, only ADMIN_EMAILS members can see the internal hub; non-admin teammates (Sean, etc.) get a 403. Set it on Vercel (comma-separated emails). Defined in src/lib/admin.ts:46.
• ADMIN_KEY — not set in Vercel. Guards POST /api/top-100 auto-scoring. Fails closed (safe), but the endpoint is dead until set. Low priority unless Top 100 auto-scoring is needed. src/app/api/top-100/route.ts:23.
• PLUGIN_BACKEND_URL / PROVISIONING_EMAIL — both have safe fallbacks; no action needed.
• Stripe not split test/live (reconfirmed): STRIPE_SECRET_KEY / STRIPE_WEBHOOK_SECRET / STRIPE_PRO_PRICE_ID share one value across Production+Preview. Same fix pattern as the Clerk split. Verify prod is live mode and preview can't create real charges. (Highest priority of these.)
• Dev-scope backfill: many vars (ADMIN_EMAILS, ANTHROPIC_API_KEY, RESEND_API_KEY, STRIPE_*, etc.) lack the Development scope — root cause of the vercel env pull local-dev friction. Batch-fix later.

[ChesterSchendel]

E2E test findings — invite email + redirect (2026-05-29)

Ran the invite flow on a preview deploy against the dev Clerk instance. Findings:

Invite email content (needs polish):

• Body reads "Ladder Provisioning has invited you to join …" — the inviter is the provisioning service account, whose display name is "Ladder Provisioning." Reads robotically. Fix by renaming the provisioning service account user's first/last name (e.g. to "Drawbackwards" or the Team Lead's own org context). NOT paid-gated. Quick win.
• Subject prefixed "[Development]" and sender is invitations@accounts.dev — both are dev-Clerk-instance artifacts. They disappear on the prod Clerk instance once the verified domain is in use. No action beyond confirming on prod.
• Bare visual styling (logo too plain, etc.) — deeper template customization is the Clerk paid-plan dependency tracked in Plan Clerk paid-tier upgrade timing (membership cap + features) #255/Export Ladder wordmark PNG for Clerk email templates (light-bg variant) #256.

Redirect dead-end (#181) confirmed reproducible on dev Clerk:

• Accepting the invite on the accounts.dev hosted portal dead-ends on Clerk's generic "Welcome / Development mode — cannot redirect to your application" page instead of returning to the app.
• This is inherent to development Clerk instances (they use the accounts.dev portal with no app domain). The membership IS created and the user IS signed in — only the browser redirect dead-ends.
• Now fixable for real on prod: prod Clerk is a true production instance as of the 2026-05-28 cutover, so we can set its Home URL / paths to close Onboarding flow is completely broken due to incorrect Clerk config #181 properly. Verify on prod, then close Onboarding flow is completely broken due to incorrect Clerk config #181.

[ChesterSchendel]

Prod email deliverability (found during prod signup test, 2026-05-29)

Verified prod signup works and email now sends from notifications@runladder.com (verified domain — good). One real issue:

Gmail spam/suspicious flag → blocks images. The verification email triggered Gmail's "This message appears suspicious / Images in this message are hidden" warning. Root cause is email authentication for runladder.com — SPF / DKIM / DMARC records need to be set up so Gmail trusts mail sent as @runladder.com via Clerk's sending infra. Clerk provides the DNS records; they go in Cloudflare.

Note on the "broken logo image": it is almost certainly a symptom of the spam flag, not a broken URL — Gmail hides ALL images on a message it flags as suspicious (the screenshot showed a "Show images" button). So this is most likely ONE issue, not two. Fix order: set up SPF/DKIM/DMARC first, then re-check whether the logo renders. Only chase a logo-asset/URL fix if the image is STILL broken after clicking "Show images" / after deliverability is fixed. Don't pre-emptively hunt a logo bug that may not exist.

High priority before the client demo: if invite + verification emails land in spam, clients can't onboard.

Not bugs (confirmed working as intended)

• Admin menu item is correctly gated to platform admins only (UserMenu.tsx line 351, isAdmin from /api/admin/status → ADMIN_EMAILS). Non-admins never see it.
• Admin showing 'Free' tier is expected for a fresh account with no org. Resolves once the Drawbackwards internal org is recreated on prod and the member is added. Admin status and subscription tier are independent axes.