Poputi

A small web app and data pipeline to measure what counts as “along the way”. Respondents see travel scenarios and report the maximum extra time still considered acceptable. Data is stored in Firestore, exported for analysis, and a population threshold f(x) is estimated.

Repository layout

• survey/web/: static front-end (vanilla JS + CSS)
  • index.html, app.js, questions.json (pre-generated scenarios)
  • Expects firebase-config.js at runtime (generated by CI or created locally)
• survey/questions/: question generation tools (Python)
• survey/export/: CSV exports produced by the export script
• analysis/: Jupyter notebook and fitted curve outputs (CSV + JSON)
• infrastructure/: OpenTofu (Terraform-compatible) IaC
  • Modules by service: firestore/, firebase/, iam/
  • Root wires modules and exposes outputs
• .github/workflows/: CI for infra deploy and static site (Pages)

Front-end

• Pure static app served from survey/web/.
• Submissions are written directly to Firestore collection survey using the Firebase Web SDK (compat) and public “create-only” rules.
• Client-side reads are blocked by rules (privacy-first; reads via exporter SA only).

Local run (simple dev server):

cd survey/web
python3 -m http.server 8080
# open http://localhost:8080

Provide Firebase config locally by creating survey/web/firebase-config.js:

window.POPUTI_FIREBASE_CONFIG = {
  apiKey: "...",
  authDomain: "...",
  projectId: "...",
  storageBucket: "...",
  messagingSenderId: "...",
  appId: "..."
};

Infrastructure (OpenTofu)

Prereqs: OpenTofu, gcloud/gsutil, a GCP project with billing enabled.

• State backend: GCS (configured in CI/init). Create bucket once and set versioning (free for small state).
• Modules:
  • firestore/: enables API and provisions default DB in Native mode
  • firebase/: links Firebase, deploys Firestore rules, creates a Web App and outputs client config
  • iam/: creates a read-only exporter Service Account and key (for CSV export)

Run locally (summary):

cd infrastructure
# authenticate (user ADC or a deployer SA JSON)
# gcloud auth application-default login
# export GOOGLE_APPLICATION_CREDENTIALS=/abs/path/deployer-sa.json

# init with your state bucket
tofu init \
  -backend-config="bucket=$TF_STATE_BUCKET" \
  -backend-config="prefix=terraform/state"

# set variables
export TF_VAR_project_id=your-gcp-project
export TF_VAR_region=us-central1

# apply
tofu apply -auto-approve

# outputs
tofu output -json firebase_web_config | jq .

Firestore rules (deployed by infra):

• Anyone can create docs in survey (public write for submissions)
• read/update/delete are denied from clients

CI/CD

• Infra: .github/workflows/deploy-infrastructure.yml
  • Runs on push to master; applies infra with OpenTofu
  • Exports Firebase Web config as an artifact firebase-web-config
• Static site: .github/workflows/static.yml
  • Triggers after infra completes (workflow_run)
  • Downloads the artifact and generates survey/web/firebase-config.js
  • Publishes survey/web/ to GitHub Pages

Export data (CSV)

A separate read-only Service Account (IAM) is created for exports; it bypasses client rules via IAM.

Steps:

# After apply, fetch SA key as JSON
cd infrastructure
tofu output -raw export_service_account_key_json > ../exporter_key.json
cd ..

# Install dependency
pip install google-cloud-firestore

# Run exporter
export GOOGLE_APPLICATION_CREDENTIALS="$PWD/exporter_key.json"
python3 survey/export/export_firestorm_csv.py --project "$GCP_PROJECT_ID" --out survey/export/YYYY-MM-DD/survey_export.csv

CSV rows: one response per row, including respondent metadata and scenario fields.

Analysis

Notebook: analysis/survey_analysis.ipynb

• Loads CSV, deduplicates by (user_id, question_id) (keeps latest)
• Normalizes times (minutes), plots scatter (x capped at 300 minutes for clarity)
• Fits a monotone threshold via isotonic regression on binned medians
• Exports the fitted curve to:
  • analysis/fitted_threshold.csv (x,y points in minutes)
  • analysis/fitted_threshold.json (piecewise-linear, cross-language)

Piecewise-linear JSON format:

{
  "representation": "piecewise_linear",
  "units": {"x": "minutes", "y": "minutes"},
  "breakpoints": [x0, x1, ...],
  "values": [y0, y1, ...],
  "interp": "linear",
  "monotone": true
}

Evaluate f(x) by linear interpolation between neighboring breakpoints.

Generate questions

Questions are generated by survey/questions/ with a backwards‑linear distribution favoring 10–25 minute bases; hard caps: walk ≤ 3h, bicycle ≤ 6h, global max ≤ 7 days. Regenerate:

python3 -m survey.questions.generate_questions 2000 --format json --out survey/web/questions.json --seed 42

Privacy

• Public writes only; client reads denied. Use the exporter Service Account to access data.

License

TBD.