Skip to content

Add research CSV ZIP exports#8

Open
Spbd1 wants to merge 1 commit into
codex/create-json-config-files-and-validatorfrom
codex/add-research-data-export-service
Open

Add research CSV ZIP exports#8
Spbd1 wants to merge 1 commit into
codex/create-json-config-files-and-validatorfrom
codex/add-research-data-export-service

Conversation

@Spbd1

@Spbd1 Spbd1 commented May 10, 2026

Copy link
Copy Markdown
Owner

Motivation

  • Provide researchers with de-identified, analysis-ready CSV exports for reproducible research by generating normalized datasets and packaging them into a single ZIP file.
  • Avoid leaking direct personal identifiers by exporting only internal player_id references and omitting emails, IPs, names, comprehension answers, and other sensitive data.
  • Offer both server-scoped and global bulk exports so admins can download a single session or the whole study inventory for analysis.

Description

  • Added a research export service that queries Prisma and composes eight normalized CSV files then packs them into a ZIP: players.csv, parcels.csv, decisions.csv, contracts.csv, server_events.csv, treasury_transactions.csv, round_outcomes.csv, and server_summary.csv (apps/web/lib/services/researchExport.ts).
  • Implemented a streaming-compatible ZIP builder and robust CSV escaping and headers to produce stable, machine-readable CSVs (CSV/ZIP encoding logic in researchExport.ts).
  • Added admin API endpoints: GET /api/admin/servers/:serverId/export.zip (apps/web/app/api/admin/servers/[serverId]/export.zip/route.ts) and GET /api/admin/export/all.zip (apps/web/app/api/admin/export/all.zip/route.ts) that require admin auth and return the ZIP as application/zip with appropriate Content-Disposition headers.
  • Updated the admin Exports UI to link to the new ZIP downloads (apps/web/app/admin/exports/page.tsx).
  • Added a complete docs/data-dictionary.md documenting every exported column, types, missingness rules, and privacy notes (sensitive fields explicitly omitted).

Testing

  • Ran git diff --check to validate no whitespace/merge errors and it succeeded.
  • Attempted project dependency install with pnpm install, but the run failed due to an external registry authorization error (403 for @tailwindcss/postcss), preventing full dependency installation and project-wide typechecking.
  • Ran pnpm --filter @parcel-society/web lint, pnpm --filter @parcel-society/web typecheck, and pnpm --filter @parcel-society/web test but these could not complete because pnpm install was blocked and required dev/runtime packages (e.g., vitest, type declarations) were missing.
  • The new export code and endpoints were exercised locally via static inspection and targeted edits in the repository; errors reported by tsc in this environment are primarily due to missing node modules / type definitions rather than the added export logic itself.

Codex Task

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant