Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
ffd79f2
docs(demo): add end-to-end demo presentation and tutorial
May 27, 2026
cf05e8e
docs(demo): apply brand header/footer from official pptx
May 27, 2026
a9d39a2
docs(demo): add SVG component icons and use them in the pieces slide
May 27, 2026
ae84316
docs(demo): use more formal slide titles
May 27, 2026
6cc588e
docs(demo): replace ASCII diagram with on-brand C4 landscape (SVG)
May 27, 2026
b10a434
docs(demo): polish the component-interactions diagram
May 27, 2026
4aa0faf
docs(demo): redraw component interactions as a linear journey
May 27, 2026
1f46463
docs(demo): fix Affinities role — it is a relationship registry, not …
May 27, 2026
fbc1cba
docs(demo): make component-interactions diagram horizontal
May 27, 2026
1ec6662
docs(demo): move interactions explanation to speaker notes
May 27, 2026
5169c46
docs(demo): add a 'step by step' slide for component interactions
May 27, 2026
de022df
docs(demo): de-personalize narrative and reframe installation
May 27, 2026
5fb0ca2
docs(demo): remove remaining gendered pronouns in narrative
May 27, 2026
3477542
docs(demo): rename section dividers from 'Act' to 'Step'
May 27, 2026
c1593fd
docs(demo): clarify NDP-EP install (Docker Hub image, backends separate)
May 27, 2026
e1bbf6c
docs(demo): NDP-EP install via git clone + compose, with sysadmin notice
May 27, 2026
90f4cb0
docs: add .env configuration reference; split install slide
May 27, 2026
c41e2ad
docs(demo): hide screenshot placeholders; link .env reference on inst…
May 27, 2026
36296a6
docs(demo): reword .env reference to '.env variables are ...'
May 27, 2026
c880b3c
docs: rewrite .env configuration reference (purpose + where to get va…
May 27, 2026
bed8df2
docs(demo): turn weak 'what's shared' slide into a responsibility-bou…
May 27, 2026
d2d1c7d
docs(demo): fix AAI install steps (git clone, .env_template, SSL certs)
May 28, 2026
d165788
docs(demo): add git clone + correct env templates to full-stack steps
May 28, 2026
3bfa73a
docs(demo): don't repeat NDP-EP install in full-stack step 4
May 28, 2026
e9c7ee3
docs(demo): tighten full-stack step 4 wording
May 28, 2026
d0f820a
docs(demo): explain Compose profiles on the NDP-EP install slide
May 28, 2026
9ffac3d
docs(demo): add Compose profiles slide explaining what each one starts
May 28, 2026
b0c5698
docs(demo): split Compose profiles table across two slides
May 28, 2026
0574abf
docs(demo): note CKAN local catalog needs an accessible CKAN + admin …
May 28, 2026
6234a12
docs(demo): professionalize CKAN backend prerequisite wording
May 28, 2026
e48bdb0
docs(demo): professional, technical tone across the deck
May 28, 2026
437a90f
docs(demo): add 'bootstrap the first admin' slide to Step 2
May 28, 2026
e43e980
docs(demo): add prerequisites slide (incl. EP_UUID before install)
May 28, 2026
733a8d0
docs: add appendix on obtaining EP_UUID; correct Affinities path to /ep
May 28, 2026
79e9031
docs(demo): split EP_UUID appendix into two slides and cross-referenc…
May 28, 2026
e5c573d
docs(demo): EP_UUID web-app slide follows the Add Endpoint flow + emb…
May 28, 2026
0ed5e43
docs(demo): constrain EP_UUID screenshot by height so it fits the slide
May 28, 2026
3160739
docs(demo): split 'Bootstrap the first admin' into two slides
May 28, 2026
31e6f0d
docs(demo): appendix — Keycloak for user+password, AAI API for groups…
May 28, 2026
34c40a0
docs(demo): unify appendix slide titles and split the AAI API slide
May 28, 2026
88e293f
docs(demo): split admin bootstrap by deployment (NDP infra vs full st…
May 29, 2026
550ae01
docs(demo): clarify where users come from (NDP accounts vs self-hoste…
May 29, 2026
ed53764
docs(demo): role granting via the Access Requests UI (request + approve)
May 29, 2026
00d5a2e
docs(demo): give each Access Requests screenshot its own full-size slide
May 29, 2026
902c2b9
docs(demo): add screenshots (request access, approve, Affinities add …
May 29, 2026
66017a9
docs(demo): note NDP-EP access (web UI at /ui, HTTP API) at end of in…
May 29, 2026
fb661ee
docs(demo): polish 'The three roles' slide (AAI source + hierarchy note)
May 29, 2026
73a1fdf
docs(demo): Step 3 leads with Search, with all its options
May 29, 2026
b7c9bd6
docs(demo): drop redundant 'Log in' slide from Step 3
May 31, 2026
6d6a975
docs(demo): add full-size Search UI screenshot slide after Search opt…
May 31, 2026
d552ddc
docs(demo): add Search UI screenshot (search options)
May 31, 2026
53b0030
docs(demo): consolidate '+ New' flow into overview + grouped field sl…
May 31, 2026
c5ce947
docs(demo): expand '+ New' fields — one slide per item with field mea…
May 31, 2026
46b09f5
docs: add the original NDP-EP PowerPoint as the brand-asset source
May 31, 2026
bebb8ce
docs(demo): expand '+ New' fields with purpose + realistic examples
May 31, 2026
5e49ce7
docs(demo): split overflowing S3 and Kafka '+ New' slides
May 31, 2026
a1f2b81
docs(demo): drop redundant 'Role-based access in practice' slide
May 31, 2026
08f28b4
docs(demo): expand S3 Management section (requirements + bucket/objec…
May 31, 2026
805e26f
docs(demo): drop empty 'Conclusion' divider slide
Jun 1, 2026
41f58f9
docs(demo): spell out the federation metrics payload
Jun 1, 2026
d586d37
docs(demo): note that NDP-managed onboarding also stores catalog tokens
Jun 1, 2026
2b61fe6
docs(demo): drop 'Federation benefits' slide
Jun 1, 2026
5e115b1
docs(demo): give each NetBird screenshot its own full-size slide
Jun 1, 2026
3998c46
docs(demo): drop 'Thank you / Questions' slide
Jun 1, 2026
30bca57
docs(demo): collapse NetBird bonus into a single page (link to netbir…
Jun 1, 2026
486d46a
docs(demo): drop 'Summary' slide
Jun 1, 2026
54b1d47
ci(demo): render Marp deck and publish to GitHub Pages
Jun 1, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
71 changes: 71 additions & 0 deletions .github/workflows/demo-deck.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
name: Build & publish demo deck

# Renders docs/demo/NDP-demo-presentation.md with Marp and publishes the slides
# as a GitHub Pages site (HTML, navigable in the browser) plus a PDF that is
# accessible from the published site at /NDP-demo-presentation.pdf.

on:
push:
branches: [main, feature/179-demo-documentation]
paths:
- 'docs/demo/**'
- '.github/workflows/demo-deck.yml'
workflow_dispatch:

permissions:
contents: read
pages: write
id-token: write

concurrency:
group: pages
cancel-in-progress: false

jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- uses: actions/setup-node@v4
with:
node-version: '20'

- name: Install Marp CLI
run: npm install -g @marp-team/marp-cli@latest

- name: Prepare site/ (copy local assets so relative URLs work)
run: |
mkdir -p site
cp -r docs/demo/assets site/assets
# Screenshots may not all be present yet; copy what's there.
if [ -d docs/demo/screenshots ]; then
cp -r docs/demo/screenshots site/screenshots
fi

- name: Render HTML (slides)
run: |
marp --allow-local-files --html \
--output site/index.html \
docs/demo/NDP-demo-presentation.md

- name: Render PDF (also published next to the HTML)
run: |
marp --allow-local-files --pdf \
--output site/NDP-demo-presentation.pdf \
docs/demo/NDP-demo-presentation.md

- name: Upload Pages artifact
uses: actions/upload-pages-artifact@v3
with:
path: site

deploy:
needs: build
runs-on: ubuntu-latest
environment:
name: github-pages
url: ${{ steps.deployment.outputs.page_url }}
steps:
- id: deployment
uses: actions/deploy-pages@v4
314 changes: 314 additions & 0 deletions docs/configuration.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,314 @@
# Configuration reference (`.env`)

NDP-EP is configured entirely through environment variables, normally placed in
a `.env` file next to `docker-compose.yml`. The annotated template is
[`example.env`](../example.env) — copy it to `.env` and edit.

This page explains **every** variable: what it is, what it controls, where to get
its value, and whether you need it. Booleans are `True` / `False`.

- **Required** = the Endpoint will not work correctly without it for the feature it
controls.
- **Optional** = safe to leave at its default.
- Unknown variables are ignored (settings allow extras), so a shared `.env` across
components is fine.

### Minimum to get started (connect to the National Data Platform)

You really only need to set:

- `AUTH_API_URL` → the platform's AAI (so logins/roles work).
- `LOCAL_CATALOG_BACKEND` + its backend vars (`MONGODB_*` **or** `CKAN_*`) → where this EP stores its catalog.
- `CKAN_LOCAL_ENABLED=True` → if this EP should allow publishing (not read-only).
- `ORGANIZATION`, `EP_NAME` → how your EP identifies itself.

Everything else turns optional features on/off.

---

## General / API

#### `ROOT_PATH`
*Optional · default: empty (served at `/`).*
URL path prefix when the EP runs behind a reverse proxy at a sub-path (e.g.
`/ep-api`). It makes the API and UI build correct links. **Where:** match the path
your reverse proxy mounts the EP at; leave empty if it is served at the domain root.

#### `ORGANIZATION`
*Optional · default: `Unknown Organization`.*
Display name of the organization running this Endpoint (shown in the UI and
reported to Federation). **Where:** you choose it.

#### `EP_NAME`
*Optional · default: `Unknown EP`.*
Display name of this Endpoint (UI + Federation registry). **Where:** you choose it.

#### `IS_PUBLIC`
*Optional · default: `True`.*
Whether this Endpoint advertises itself as public. **Where:** set `False` for a
private/internal EP.

#### `USE_JUPYTERLAB` / `JUPYTER_URL`
*Optional · defaults: `False` / `https://jupyter.org/try-jupyter/lab/`.*
Show a "JupyterLab" link in the UI and where it points. **Where:** set `USE_JUPYTERLAB=True`
and `JUPYTER_URL` to your JupyterLab instance if you offer one.

#### `SWAGGER_TITLE` / `SWAGGER_DESCRIPTION` / `SWAGGER_VERSION`
*Advanced · usually leave default.*
Metadata shown on the `/docs` (Swagger) page. `SWAGGER_VERSION` tracks the release
and normally should not be overridden.

---

## Authentication & access (AAI)

#### `AUTH_API_URL`
*Required · default: `https://idp.nationaldataplatform.org/temp/information`.*
The AAI endpoint the EP calls to **validate bearer tokens** and read the user's
identity, groups and **roles**. This is the single source of authentication.
**Where:** your NDP/AAI administrator. For the central platform use the default;
for a self-hosted AAI use its `…/information` URL.

#### `TEST_TOKEN`
*Optional · dev only · default: `testing_token`.*
A fixed token the EP accepts without contacting AAI, for local testing.
**Where:** you set it for dev. **Leave blank in production.**

#### `ENABLE_GROUP_BASED_ACCESS`
*Optional · default: `False`.*
If `True`, write operations (POST/PUT/DELETE) additionally require the user to
belong to one of `GROUP_NAMES`. This is an extra gate on top of the role model.
**Where:** enable only if you want group-scoped writes.

#### `GROUP_NAMES`
*Optional · default: empty.*
Comma-separated list of groups allowed to write when `ENABLE_GROUP_BASED_ACCESS=True`
(e.g. `admins,data-managers`). Matching is case-insensitive; empty + enabled =
all writes denied. **Where:** the group names defined in Keycloak/AAI — ask your
AAI administrator.

> **Roles** (`viewer` / `writer` / `admin`) are **not** set here — they travel in
> the token issued by AAI. See [roles-and-permissions.md](roles-and-permissions.md).

---

## Local catalog (where this EP stores its data)

#### `LOCAL_CATALOG_BACKEND`
*Required · default: `ckan`.*
Backend for **this EP's own** catalog: `ckan` or `mongodb`. (The global and
Pre-CKAN catalogs always use CKAN regardless.) **Where:** you choose — `mongodb`
is the simplest to self-host; `ckan` if you already run a CKAN.

#### `CKAN_LOCAL_ENABLED`
*Optional · default: `False`.*
Master switch for **write** operations on the local catalog (create/update/delete
organizations, datasets, resources), for **any** backend. `False` makes the EP
read-only. *(The name keeps "CKAN" for historical reasons; it applies to MongoDB
too.)* **Where:** set `True` if this EP should accept publishing.

### CKAN backend — only if `LOCAL_CATALOG_BACKEND=ckan`

#### `CKAN_URL`
*Required (CKAN backend) · default: `http://localhost:5000`.*
Base URL of your CKAN instance. **Where:** your CKAN deployment URL.

#### `CKAN_API_KEY`
*Required (CKAN backend).*
API key used to write to CKAN. **Where:** in CKAN, log in → your user profile →
**API Tokens** → create a token.

#### `CKAN_VERIFY_SSL`
*Optional · default: `True`.*
Verify CKAN's TLS certificate. **Where:** set `False` only for self-signed/dev
CKAN instances (less secure).

#### `CKAN_GLOBAL_URL`
*Optional · default: `https://nationaldataplatform.org/catalog`.*
Read-only global NDP catalog the EP can search alongside the local one.
**Where:** usually leave the default.

### MongoDB backend — only if `LOCAL_CATALOG_BACKEND=mongodb`

#### `MONGODB_CONNECTION_STRING`
*Required (MongoDB backend) · default: `mongodb://localhost:27017`.*
MongoDB connection URI for the local catalog. **Where:** your MongoDB. With the
bundled Compose (`--profile mongodb`) use `mongodb://admin:admin123@mongodb:27017`.

#### `MONGODB_DATABASE`
*Optional · default: `ndp_local_catalog`.*
Database name used for the local catalog. **Where:** you choose.

### Pre-CKAN (optional staging catalog)

#### `PRE_CKAN_ENABLED`
*Optional · default: `False`.*
Enable a Pre-CKAN staging target (publish to a review instance before the global
catalog).

#### `PRE_CKAN_URL` / `PRE_CKAN_API_KEY` / `PRE_CKAN_VERIFY_SSL` / `PRE_CKAN_ORGANIZATION`
*Required only if `PRE_CKAN_ENABLED=True`.*
URL, API key (same place as `CKAN_API_KEY`, but on the Pre-CKAN instance), TLS
verification, and the organization all staged datasets are published under
(overrides their `owner_org`). **Where:** from whoever operates your Pre-CKAN
(e.g. the SDSC staging instance).

---

## S3 / object storage (MinIO)

#### `S3_ENABLED`
*Optional · default: `False`.*
Enable S3 object storage and the **S3 Management** tool in the UI (writers only).

#### `S3_ENDPOINT`
*Required if `S3_ENABLED=True` · default: `localhost:9000`.*
`host:port` of the S3-compatible service. **Where:** your MinIO/S3 endpoint. With
the bundled Compose (`--profile s3`) use `minio:9000`.

#### `S3_ACCESS_KEY` / `S3_SECRET_KEY`
*Required if `S3_ENABLED=True` · dev defaults: `minioadmin` / `minioadmin123`.*
Credentials for the S3 service. **Where:** your MinIO/S3 admin console (Access
Keys). The dev MinIO ships with `minioadmin` / `minioadmin123`.

#### `S3_SECURE`
*Optional · default: `False`.*
`True` for HTTPS, `False` for HTTP. **Where:** `True` when your S3 endpoint serves
TLS.

#### `S3_REGION`
*Optional · default: `us-east-1`.*
Region label sent to the S3 API. **Where:** match your provider; the default is
fine for MinIO.

---

## Streaming (Kafka)

#### `KAFKA_CONNECTION`
*Optional · default: `False`.*
Enable Kafka connectivity (managing/streaming topics).

#### `KAFKA_HOST` / `KAFKA_PORT`
*Required if `KAFKA_CONNECTION=True` · defaults: `localhost` / `9092`.*
Kafka broker address. **Where:** your broker. With the bundled Compose
(`--profile kafka`) use `kafka` and `9093` (internal) / `9092` (external).

#### `KAFKA_PREFIX`
*Optional · default: `data_stream_`.*
Prefix applied to topics the EP manages. **Where:** you choose.

#### `MAX_STREAMS`
*Optional · default: `10`.*
Maximum number of concurrent managed streams.

---

## Affinities integration

#### `AFFINITIES_ENABLED`
*Optional · default: `False`.*
When `True`, datasets/services created here are auto-registered in Affinities
(non-blocking — the EP keeps working if Affinities is down).

#### `AFFINITIES_URL`
*Required if `AFFINITIES_ENABLED=True`.*
Base URL of the Affinities API (e.g. `http://affinities-api:8000`). **Where:** the
platform's Affinities URL, or your local one.

#### `AFFINITIES_EP_UUID`
*Required if `AFFINITIES_ENABLED=True`.*
This Endpoint's UUID inside Affinities. **Where:** the endpoint's `uid` in
Affinities — list/create it via `GET`/`POST /ep` on the Affinities API, or from
the **Endpoints** page of the Affinities web app. See
[affinities-integration.md](affinities-integration.md).

#### `AFFINITIES_TIMEOUT`
*Optional · default: `30`.*
HTTP timeout (seconds) for Affinities calls.

---

## Federation (metrics reporting)

#### `METRICS_ENDPOINT`
*Optional · default: `https://federation.ndp.utah.edu/metrics/`.*
The Federation endpoint this EP periodically reports health/usage metrics to,
which is how the EP becomes discoverable in the federation. **Where:** the
platform's Federation `/metrics/` URL, or your local Federation when self-hosting.

#### `METRICS_INTERVAL_SECONDS`
*Optional · default: `3300` (55 min).*
How often metrics are reported.

---

## Pelican federation (external data access)

#### `PELICAN_ENABLED`
*Optional · default: `False`.*
Enable browsing/downloading from Pelican federations (OSDF, etc.) and support for
`pelican://` resource URLs.

#### `PELICAN_FEDERATION_URL`
*Optional · default: empty (uses OSDF).*
Default Pelican federation, format `pelican://host` (e.g. `pelican://osg-htc.org`).
**Where:** the federation you target; leave empty for OSDF.

#### `PELICAN_DIRECT_READS`
*Optional · default: `False`.*
Read straight from origin servers instead of caches. **Where:** keep `False` for
better performance unless you have a reason.

---

## Remote execution (Rexec)

#### `REXEC_CONNECTION`
*Optional · default: `False`.*
Enable the Remote Execution Deployment API integration.

#### `REXEC_DEPLOYMENT_API_URL`
*Required if `REXEC_CONNECTION=True`.*
URL of the Remote Execution Deployment API. **Where:** from whoever operates that
service.

---

## Access requests

#### `ENABLE_ACCESS_REQUESTS`
*Optional · default: `False`.*
Enable the access-request workflow (a user requests access; an admin
approves/rejects). **Requires MongoDB** reachable via `MONGODB_CONNECTION_STRING`.
**Where:** turn on if you want self-service access requests.

#### `ACCESS_REQUESTS_COLLECTION`
*Optional · default: `access_requests`.*
MongoDB collection used to store access requests.

---

## Telemetry — OpenTelemetry (advanced)

#### `OTEL_ENABLED`
*Optional · default: `False`.* Enable OpenTelemetry tracing.

#### `OTEL_SERVICE_NAME`
*Optional · default: `ep-api`.* Service name shown in traces.

#### `OTEL_EXPORTER_TYPE`
*Optional · default: `console`.* Where traces go: `console`, `otlp`, or `none`.

#### `OTEL_OTLP_ENDPOINT`
*Optional · default: `http://localhost:4317`.* OTLP collector endpoint (when
`OTEL_EXPORTER_TYPE=otlp`). **Where:** your collector's address.

#### `OTEL_OTLP_INSECURE`
*Optional · default: `True`.* Use a non-TLS OTLP connection.

---

> **Tip:** start from [`example.env`](../example.env) and override only what your
> deployment needs. When in doubt about a value for a shared platform service
> (AAI, Affinities, Federation), ask your NDP administrator.
Loading
Loading