Skip to content

342 add dap id field and wire up new dag api endpoints #344

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mehalter merged 54 commits into from
May 18, 2026
Merged

342 add dap id field and wire up new dag api endpoints #344

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
fa8d51b
added pipelineId and piplineRunnable keys/values to all pipeline prof…
thecaffiend May 8, 2026
4eadae0
first couple (untested) configs for workflow api endpoints
thecaffiend May 8, 2026
6bfc89a
first couple (untested) workflow api endpoints
thecaffiend May 8, 2026
e1b3681
added pipelineId and pipelineRunnable keys/values to dap registry ite…
thecaffiend May 8, 2026
2d2ad85
wired up workflow meta registry to the environment variables sent off…
thecaffiend May 8, 2026
6d5effa
first (untested) list workflow api endpoint
thecaffiend May 8, 2026
389e0c4
first (untested) get profiles for workflow api endpoint
thecaffiend May 8, 2026
13509f5
spelling
thecaffiend May 8, 2026
98a99a0
moved dapregistry module to registry as it now has 2 classes
thecaffiend May 8, 2026
3fede03
(untested) speculative additions to get profiles for workflow api end…
thecaffiend May 8, 2026
8f0b474
added config for 1st 4 workflow endpoints. fixed longstanding typo
thecaffiend May 12, 2026
a853c51
added eps for triggering and halting workflows. fixed error schema fo…
thecaffiend May 12, 2026
12a8eca
made the mwaa config a single env mapping instead of a list. if we en…
thecaffiend May 12, 2026
3285922
changed to use correct env var for the mwaa environment. fixed a var …
thecaffiend May 12, 2026
3bd1e5d
removed code moved to capepy. changed to handle 404 for both workflow…
thecaffiend May 12, 2026
6b33b87
added policy enum and supported policies for mwaa env
thecaffiend May 12, 2026
d86fdf4
changed cape rest api to handle old style resource grants as well as …
thecaffiend May 12, 2026
046f94b
removed base class mwaa env tracking (this is private swimlane only now)
thecaffiend May 12, 2026
39b42e8
moved all mwaa env managent to private swimlane. changed mwaa env con…
thecaffiend May 12, 2026
92ed38c
untested ep handlers for triggering and halting a workflow run
thecaffiend May 12, 2026
39d7b3d
added pipelineRunnable: false to bactopia base dev profile.
thecaffiend May 12, 2026
116defd
updated capepy to 2.3.0
thecaffiend May 12, 2026
0aa18b3
fixed bad query param name. removed useless word from error msg
thecaffiend May 14, 2026
54cc8ff
fixed missing param to function call. changed get_pipeline_by_id to g…
thecaffiend May 14, 2026
c72d4ca
changed to use env var for mwaa env name
thecaffiend May 14, 2026
dcad8ec
removed range key for dap registry (it's now by id). added secondary …
thecaffiend May 14, 2026
2c308fe
added vpc config and vpc access role for lambdas. they need to be in …
thecaffiend May 14, 2026
f234906
added (untested) endpoints for getting a workflow run, workflow tasks…
thecaffiend May 14, 2026
4b08683
added config for new workflow endpoints
thecaffiend May 14, 2026
5a954d9
wired up a vpc endpoint for the airflow env (so we can hit the api in…
thecaffiend May 14, 2026
857d66d
got away from hard coded aws account number
thecaffiend May 14, 2026
fd3b867
update capepy wheel
thecaffiend May 15, 2026
a3433b7
set the airflow api endpoints to timeout at 10 sec instead of 3 as th…
thecaffiend May 15, 2026
4bd6e61
fixed path and handler alias for workflows/run/taskinstances endpoint
thecaffiend May 15, 2026
9b872c6
added query perms for the new dap registry index (in a way that grant…
thecaffiend May 15, 2026
50ac539
updated to capepy 3.0.0
thecaffiend May 15, 2026
0aefc85
updated to capepy 3.0.0
thecaffiend May 15, 2026
dd9e20a
added ignore for changes to the gsi for dap registry table as pulumi …
thecaffiend May 15, 2026
2360cd7
adde query perms to database read policy (previously only had getitem)
thecaffiend May 15, 2026
3c1dbf3
removed code that has been moved to capepy. handled change in capepy …
thecaffiend May 15, 2026
872ebd5
removed code that was moved to capepy
thecaffiend May 15, 2026
61d3a8c
defined variables early as some code paths didn't do that.
thecaffiend May 15, 2026
07a371a
fixed a bug in the api path to get tasks for dag runs
thecaffiend May 15, 2026
29e35c9
imported missing functions from capepy. fixed bug in the naming of qu…
thecaffiend May 15, 2026
ebc541f
fixed error in the timestamp format expected by airflow. imported mis…
thecaffiend May 15, 2026
71dc5e1
spelling errors
thecaffiend May 15, 2026
3b95424
fixed version of capepy
thecaffiend May 15, 2026
d049ccc
trying to make pyright happy feels so wrong
thecaffiend May 15, 2026
9b7230f
fixed bactopia 3.2.0 base id
thecaffiend May 18, 2026
9e62296
removed pipelineRunnable from all profiles
thecaffiend May 18, 2026
45eb156
hacky fix for pyright typing with new pulumi version
mehalter May 18, 2026
81200cb
removed pipeline runnable
thecaffiend May 18, 2026
36b2fa0
Merge remote-tracking branch 'refs/remotes/gh/342-add-dap-id-field-an…
thecaffiend May 18, 2026
02d7366
removed duplicated options and obe todo
thecaffiend May 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
182 changes: 139 additions & 43 deletions Pulumi.cape-cod-dev.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -759,15 +759,109 @@ config:
logging_enabled: True
env_vars:
- "DAP_REG_DDB_TABLE"
- "WORKFLOW_REG_DDB_TABLE"
- "DDB_REGION"
- "USER_ATTRS_DDB_TABLE"
- "ETL_ATTRS_DDB_TABLE"
- "CRAWLER_ATTRS_DDB_TABLE"
- "CANNED_REPORT_DDB_TABLE"
- "MWAA_ENVIRONMENT"
# env_vars: TODO: add env vars for this API if needed.
# TODO: memory and timeouts for these functions need
# some love
handlers:
- id: "get_workflow_dags_handler"
name: "getdags"
code: "assets/api/capi/handlers/get_workflow_dags.py"
layers:
- capi-all
funct_args:
handler: "index.index_handler"
runtime: "python3.10"
architectures:
- "x86_64"
description: "getdags Lambda Function"
memory_size: 128
timeout: 10
- id: "get_workflow_pipeline_profiles_handler"
name: "getdagprofiles"
code: "assets/api/capi/handlers/get_workflow_pipeline_profiles.py"
layers:
- capi-all
funct_args:
handler: "index.index_handler"
runtime: "python3.10"
architectures:
- "x86_64"
description: "getdagprofiles Lambda Function"
memory_size: 128
timeout: 10
- id: "post_workflow_run_handler"
name: "postdagrun"
code: "assets/api/capi/handlers/post_workflow_run.py"
layers:
- capi-all
funct_args:
handler: "index.index_handler"
runtime: "python3.10"
architectures:
- "x86_64"
description: "postdagrun Lambda Function"
memory_size: 128
timeout: 10
- id: "patch_workflow_run_handler"
name: "patchdagrun"
code: "assets/api/capi/handlers/patch_workflow_run.py"
layers:
- capi-all
funct_args:
handler: "index.index_handler"
runtime: "python3.10"
architectures:
- "x86_64"
description: "patchdagrun Lambda Function"
memory_size: 128
timeout: 10
- id: "get_workflow_run_handler"
name: "getdagrun"
code: "assets/api/capi/handlers/get_workflow_run.py"
layers:
- capi-all
funct_args:
handler: "index.index_handler"
runtime: "python3.10"
architectures:
- "x86_64"
description: "getdags Lambda Function"
memory_size: 128
timeout: 10
- id: "get_workflow_tasks_handler"
name: "getdagtasks"
code: "assets/api/capi/handlers/get_workflow_tasks.py"
layers:
- capi-all
funct_args:
handler: "index.index_handler"
runtime: "python3.10"
architectures:
- "x86_64"
description: "getdagtasks Lambda Function"
memory_size: 128
timeout: 10
- id: "get_workflow_run_task_instances_handler"
name: "getdagruntaskinsts"
code: "assets/api/capi/handlers/get_workflow_run_task_instances.py"
layers:
- capi-all
funct_args:
handler: "index.index_handler"
runtime: "python3.10"
architectures:
- "x86_64"
description:
"getdagruntaskinsts Lambda Function"
memory_size: 128
timeout: 10
- id: "get_daps_handler"
name: "getdaps"
code: "assets/api/capi/handlers/get_daps.py"
Expand All @@ -778,7 +872,7 @@ config:
runtime: "python3.10"
architectures:
- "x86_64"
description: "getdaps Lambda Funnction"
description: "getdaps Lambda Function"
memory_size: 128
timeout: 3
- id: "get_dap_profile_handler"
Expand All @@ -791,7 +885,7 @@ config:
runtime: "python3.10"
architectures:
- "x86_64"
description: "getdapprofiles Lambda Funnction"
description: "getdapprofiles Lambda Function"
memory_size: 128
timeout: 3
- id: "get_dap_status_handler"
Expand All @@ -804,7 +898,7 @@ config:
runtime: "python3.10"
architectures:
- "x86_64"
description: "getdapstatus Lambda Funnction"
description: "getdapstatus Lambda Function"
memory_size: 128
timeout: 3
- id: "get_dap_logs_handler"
Expand All @@ -817,9 +911,11 @@ config:
runtime: "python3.10"
architectures:
- "x86_64"
description: "getdaplogs Lambda Funnction"
description: "getdaplogs Lambda Function"
memory_size: 128
timeout: 3
# TODO: if not used as part of workflows (probably
# won't be) this should be removed
- id: "submit_dap_run_handler"
name: "submitdaprun"
code: "assets/api/capi/handlers/submit_dap_run.py"
Expand All @@ -830,7 +926,7 @@ config:
runtime: "python3.10"
architectures:
- "x86_64"
description: "submitdaprun Lambda Funnction"
description: "submitdaprun Lambda Function"
memory_size: 128
timeout: 3
- id: "get_raw_objstore_authz_handler"
Expand Down Expand Up @@ -1273,9 +1369,10 @@ config:
# and no compute environments will be deployed
compute:
# `environments` (mapping, optional)
# Environments for aws batch and mwaa (aairflow) are configured
# Environments for aws batch and mwaa (airflow) are configured
# in this section. There are mappings for both types of
# environments, and both mappings contain lists of mappings.
# environments, with the airflow mapping being singular and the
# batch environments being a list of mappings.
#
# Both types have some shared config keys and then keys specific
# to that type. Shared keys are as follows:
Expand All @@ -1290,9 +1387,8 @@ config:
# config file. At least one subnet must be defined here.
#
environments:
# Each mwaa mapping in the list has the
# following schema in addition to the shared keys mentioned
# above:
# The mwaa mapping has the following schema in addition to
# the shared keys mentioned above:
# * `dag_path` (string, required)
# The path in the CAPE meta assets s3 bucket where dags
# will be stored for airflow to load. MWAA syncs these
Expand Down Expand Up @@ -1332,39 +1428,39 @@ config:
# here:
# https://www.pulumi.com/registry/packages/aws/api-docs/mwaa/environment/
mwaa:
- name: airflow-env
dag_path: airflow/dags
airflow_version: 3.0.6
airflow_config:
core.default_task_retries: 2
core.parallelism: 20
celery.worker_autoscale: 5,5
environment_class: mw1.small
subnet_types:
- compute
ingress_subnet_types:
- compute
- vpn
extra_env_args:
min_workers: 2
max_workers: 10
logging_configuration:
# NOTE: INFO is the lowest level supported in MWAA
dag_processing_logs:
enabled: True
log_level: INFO
scheduler_logs:
enabled: True
log_level: INFO
task_logs:
enabled: True
log_level: INFO
webserver_logs:
enabled: True
log_level: INFO
worker_logs:
enabled: True
log_level: INFO
name: airflow-env
dag_path: airflow/dags
airflow_version: 3.0.6
airflow_config:
core.default_task_retries: 2
core.parallelism: 20
celery.worker_autoscale: 5,5
environment_class: mw1.small
subnet_types:
- compute
ingress_subnet_types:
- compute
- vpn
extra_env_args:
min_workers: 2
max_workers: 10
logging_configuration:
# NOTE: INFO is the lowest level supported in MWAA
dag_processing_logs:
enabled: True
log_level: INFO
scheduler_logs:
enabled: True
log_level: INFO
task_logs:
enabled: True
log_level: INFO
webserver_logs:
enabled: True
log_level: INFO
worker_logs:
enabled: True
log_level: INFO
# Each batch mapping in the list has the
# following schema in addition to the shared keys mentioned
# above:
Expand Down
1 change: 1 addition & 0 deletions assets/analysis-pipelines/bactopia/bactopia-base-3.2.0.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
{
"pipelineType": "nextflow",
"pipelineName": "Bactopia",
"pipelineId": "bactopia-bactopia-base-v3.2.0",
"pipelineDescription": "Execute Bactopia v3.2.0",
"project": "bactopia/bactopia",
"version": "v3.2.0",
Expand Down
1 change: 1 addition & 0 deletions assets/analysis-pipelines/bactopia/bactopia-base-dev.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
{
"pipelineType": "nextflow",
"pipelineName": "Bactopia",
"pipelineId": "bactopia-bactopia-base-dev",
"pipelineDescription": "Execute Bactopia development release",
"project": "bactopia/bactopia",
"version": "dev",
Expand Down
1 change: 1 addition & 0 deletions assets/analysis-pipelines/bactopia/kraken2-bactopia-3.2.0.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
{
"pipelineType": "nextflow",
"pipelineName": "Bactopia Kraken2",
"pipelineId": "bactopia-kraken2-v3.2.0",
"pipelineDescription": "Execute Bactopia's Kraken2 workflow with the development release",
"project": "bactopia/bactopia",
"version": "v3.2.0",
Expand Down
1 change: 1 addition & 0 deletions assets/analysis-pipelines/bactopia/kraken2-bactopia-dev.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
{
"pipelineType": "nextflow",
"pipelineName": "Bactopia Kraken2",
"pipelineId": "bactopia-kraken2-dev",
"pipelineDescription": "Execute Bactopia's Kraken2 workflow with the development release",
"project": "bactopia/bactopia",
"version": "dev",
Expand Down
1 change: 1 addition & 0 deletions assets/analysis-pipelines/bactopia/ont-bactopia-3.2.0.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
{
"pipelineType": "nextflow",
"pipelineName": "Bactopia ONT Sample",
"pipelineId": "bactopia-ont-v3.2.0",
"pipelineDescription": "Execute Bactopia's ONT sample sequencing workflow with v3.2.0",
"project": "bactopia/bactopia",
"version": "v3.2.0",
Expand Down
1 change: 1 addition & 0 deletions assets/analysis-pipelines/bactopia/ont-bactopia-dev.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
{
"pipelineType": "nextflow",
"pipelineName": "Bactopia ONT Sample",
"pipelineId": "bactopia-ont-dev",
"pipelineDescription": "Execute Bactopia's ONT sample sequencing workflow with the development release",
"project": "bactopia/bactopia",
"version": "dev",
Expand Down
Loading
Loading