feat: enable AWS Marketplace distribution support#1186
Conversation
Configure AMI build pipeline to comply with AWS Marketplace requirements, enabling commercial distribution through AWS Marketplace seller program. Changes: - Disable EBS encryption (AWS Marketplace prohibits encrypted AMIs) - Replace public AMI sharing with Marketplace tagging workflow - Update release notifications to indicate "Ready for AWS Marketplace" - Split oversized TNConfigComponent to meet 16KB ImageBuilder limit - Add --network flag for developer custom network initialization resolves: trufnetwork/truf-network#1249 resolves: trufnetwork/truf-network#1247
WalkthroughReplaces AMI public exposure with AWS Marketplace tagging in the CI workflow, adds a Welcome component to the AMI image recipe and exports its ARN, extends TN configuration to support NETWORK and NETWORK_TYPE, updates docker-compose init flow to branch on NETWORK_TYPE, and adjusts test scripts to provide temporary .env files for config validation. Changes
Sequence Diagram(s)sequenceDiagram
autonumber
participant GH as GitHub Actions
participant AWS as AWS EC2 API
GH->>AWS: build AMI
AWS-->>GH: AMI ID, Region
note over GH: Previous: modify-image-attribute (public)
GH->>AWS: create-tags on AMI (marketplace=ready, version, stage)
AWS-->>GH: Tags applied
GH-->>GH: Release summary updated (Marketplace readiness)
sequenceDiagram
autonumber
participant User
participant VM as AMI Instance
participant Init as TN Init Scripts
participant DC as docker-compose
User->>VM: First boot / login
VM->>Init: Load .env (NETWORK, NETWORK_TYPE, CHAIN_ID, TN_PRIVATE_KEY)
alt NETWORK_TYPE = custom
Init->>Init: kwild init new network (use CHAIN_ID)
else NETWORK_TYPE != custom
Init->>Init: kwild init from mainnet genesis + bootnodes<br/>enable state-sync/trusted providers
end
Init->>DC: Start services (kwild, postgres, etc.)
Note over VM,DC: Welcome component sets MOTD and guidance
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Possibly related PRs
Suggested reviewers
Poem
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✨ Finishing touches
🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Time Submission Status
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
.github/workflows/ami-build.yml (1)
25-27: Critical: Build region must be us-east-1 for Marketplace source AMIs.Marketplace policy: “Source AMIs for AWS Marketplace must be provided in the US East (N. Virginia) Region.” The workflow sets AWS_REGION=us-east-2, which will block submission. Switch prod builds to us-east-1 (keep dev in us-east-2 if desired).
Apply one of these:
- Simple (prod-only project): set env.AWS_REGION to us-east-1.
-env: - AWS_REGION: us-east-2 +env: + AWS_REGION: us-east-1
- Conditional by stage:
- AWS_REGION: us-east-2 + AWS_REGION: ${{ (inputs.stage || 'prod') == 'prod' && 'us-east-1' || 'us-east-2' }}Based on AWS Marketplace “AMI-based product requirements.” (docs.aws.amazon.com)
deployments/infra/stacks/docker-compose.template.yml (1)
72-75: Copy preserves only non-dotfiles; usecp -awith trailing dot.
cp /root/.kwild-new/* /root/.kwild/skips dotfiles and may miss metadata. Prefercp -a /root/.kwild-new/. /root/.kwild/.- cp /root/.kwild-new/* /root/.kwild/ + cp -a /root/.kwild-new/. /root/.kwild/
🧹 Nitpick comments (10)
scripts/test-ami.sh (4)
110-115: Avoid global /tmp/.env; use mktemp and pass via --env-file to prevent collisions and TDZ issues.Using a single /tmp/.env is racy across concurrent runs and brittle if the script exits before rm. Prefer a unique temp env file and pass it explicitly to Compose. Also update cleanup to remove it on EXIT.
Example diff:
-# Create dummy .env file for validation (docker-compose expects it) -cat > /tmp/.env << 'EOF' +ENV_FILE_COMPOSE="$(mktemp -p /tmp tn-env-XXXXXX)" +cat > "$ENV_FILE_COMPOSE" << 'EOF' CHAIN_ID=tn-v2.1 NETWORK_TYPE=mainnet EOF -# Create .env file in same directory (docker-compose expects .env relative to compose file) -cat > /tmp/.env << 'EOF' +ENV_FILE_PG="$(mktemp -p /tmp tn-env-XXXXXX)" +cat > "$ENV_FILE_PG" << 'EOF' CHAIN_ID=tn-v2.1 NETWORK_TYPE=mainnet EOFAnd call Compose with:
"$COMPOSE" --env-file "$ENV_FILE_COMPOSE" -f /tmp/test-docker-compose.yml configand--env-file "$ENV_FILE_PG"for the PG section.Also applies to: 405-410
26-33: Strengthen cleanup: remove temp env files and test compose files on any exit.Currently, trap only brings down containers. If the script fails before explicit rm, /tmp/.env persists. Extend cleanup to remove temp artifacts defensively.
cleanup() { - if [ -f /tmp/tn-test-compose.yml ]; then - eval "$COMPOSE -f /tmp/tn-test-compose.yml down -v" >/dev/null 2>&1 || true - fi + # Best-effort teardown and file cleanup + [ -f /tmp/tn-test-compose.yml ] && "$COMPOSE" -f /tmp/tn-test-compose.yml down -v >/dev/null 2>&1 || true + [ -f /tmp/test-docker-compose.yml ] && rm -f /tmp/test-docker-compose.yml || true + [ -n "${ENV_FILE_COMPOSE:-}" ] && rm -f "$ENV_FILE_COMPOSE" || true + [ -n "${ENV_FILE_PG:-}" ] && rm -f "$ENV_FILE_PG" || true } -trap cleanup EXIT +trap cleanup EXITAlso, after Test 3 and PG test, you can drop the explicit
rm -flines since trap covers them.Also applies to: 123-123, 443-445
116-122: Drop eval; build a Compose command array to avoid shell injection and quoting bugs.
eval "$COMPOSE ..."is unnecessary and risky. Define an array once and reuse it.-if eval "$COMPOSE -f /tmp/test-docker-compose.yml config" > /dev/null 2>&1; then +COMPOSE_CMD=(docker compose) +# or: COMPOSE_CMD=(docker-compose) when v1 detected +if "${COMPOSE_CMD[@]}" --env-file "$ENV_FILE_COMPOSE" -f /tmp/test-docker-compose.yml config > /dev/null 2>&1; then @@ -if eval "$COMPOSE -f /tmp/tn-test-compose.yml up -d tn-postgres"; then +if "${COMPOSE_CMD[@]}" --env-file "$ENV_FILE_PG" -f /tmp/tn-test-compose.yml up -d tn-postgres; then @@ - if eval "$COMPOSE -f /tmp/tn-test-compose.yml exec -T tn-postgres pg_isready -U postgres" > /dev/null 2>&1; then + if "${COMPOSE_CMD[@]}" -f /tmp/tn-test-compose.yml exec -T tn-postgres pg_isready -U postgres > /dev/null 2>&1; then @@ - if eval "$COMPOSE -f /tmp/tn-test-compose.yml exec -T tn-postgres psql -U postgres -d kwild -c \"SELECT version();\"" > /dev/null 2>&1; then + if "${COMPOSE_CMD[@]}" -f /tmp/tn-test-compose.yml exec -T tn-postgres psql -U postgres -d kwild -c "SELECT version();" > /dev/null 2>&1; then[ suggest_recommended_refactor ]
Also applies to: 411-441
374-389: Align image tags across tests and compose for determinism.Compose uses
ghcr.io/trufnetwork/kwil-postgres:latest, while the test checks:16.8-1. Consider pinning the same tag (or digest) in both to reduce flakiness..github/workflows/ami-build.yml (1)
46-51: Go 1.24.x may not exist; pin to a known stable.To avoid setup-go resolution failures, pin to a released minor (e.g., 1.23.x) or use a repo variable.
deployments/infra/stacks/docker-compose.template.yml (2)
76-89: Private key validation looks solid; minor hardening suggestion.Consider writing nodekey.json atomically to avoid partial writes: write to a temp file then mv.
tmp=$(mktemp) printf '{"key":"%s","type":"secp256k1"}' "$CLEAN_KEY" > "$tmp" && chmod 600 "$tmp" && mv -f "$tmp" /root/.kwild/nodekey.json
21-35: Ports expose services publicly; confirm intended exposure.
- tn-node exposes 26656/26657/6600/8484; postgres-mcp exposes 8000. Ensure docs and SG defaults reflect this, and consider binding to 0.0.0.0 only when required.
Also applies to: 98-116
deployments/infra/stacks/ami_pipeline_stack.go (3)
177-421: Configuration script improvements are on point; two nits.
- Good: split components to stay under 16KB; MCP/profile handling; reconfig guardrails.
- Nits:
- For reconfigure, erroring on
--networkis correct. Consider echoing existing CHAIN_ID/NETWORK_TYPE before exit for clarity.- Use atomic write for
.env(write to temp then mv) to avoid partial writes if interrupted.
549-563: Distribution config limited to current region; align with prod region change and distribution needs.You’re distributing only to
Aws_REGION(); once prod builds move to us-east-1, this matches Marketplace requirements. If you plan to copy across regions post-approval, add a separate distribution config and ensure any encryption remains disabled.Also applies to: 565-577
592-602: Outputs complete; consider surfacing Region as an output for workflows.Expose
awscdk.Aws_REGION()as an output to validate the region from Actions before tagging.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (4)
.github/workflows/ami-build.yml(2 hunks)deployments/infra/stacks/ami_pipeline_stack.go(11 hunks)deployments/infra/stacks/docker-compose.template.yml(1 hunks)scripts/test-ami.sh(3 hunks)
🧰 Additional context used
🧠 Learnings (1)
📚 Learning: 2025-09-23T02:13:19.979Z
Learnt from: MicBun
PR: trufnetwork/node#1170
File: deployments/infra/stacks/ami_pipeline_stack.go:297-384
Timestamp: 2025-09-23T02:13:19.979Z
Learning: In the trufnetwork/node repository, the chain ID should always be "tn-v2.1" regardless of network configuration (mainnet/testnet). There are no other chains currently supported.
Applied to files:
deployments/infra/stacks/ami_pipeline_stack.go
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
- GitHub Check: lint
- GitHub Check: acceptance-test
🔇 Additional comments (3)
.github/workflows/ami-build.yml (1)
221-239: Nice touch on release notes. Ensure SSH/usage guidance matches Marketplace policies.Usage instructions should avoid requiring PII and must allow admin access per policy. Your commands look fine; just ensure the listing’s usage section mirrors this text.
Based on AWS Marketplace checklist. (docs.aws.amazon.com)
deployments/infra/stacks/ami_pipeline_stack.go (2)
27-36: Export additions LGTM; verify downstream consumers.New WelcomeComponentArn and re-ordered fields look fine. Ensure any code relying on positional construction or JSON tags adjusts accordingly.
520-543: EBS encryption disabled aligns with Marketplace policy.Setting
Encrypted: falseis required for AMIs submitted to Marketplace. Keep this explicit comment in place.Based on AWS Marketplace AMI policies (no encrypted EBS snapshots). (docs.aws.amazon.com)
| - name: Tag AMI for Marketplace (prod only) | ||
| if: env.AMI_ID && env.STAGE == 'prod' | ||
| run: | | ||
| echo "🌍 Making AMI public for production release..." | ||
| aws ec2 modify-image-attribute \ | ||
| --image-id "$AMI_ID" \ | ||
| --launch-permission "Add=[{Group=all}]" \ | ||
| echo "🏷️ Tagging AMI for AWS Marketplace distribution..." | ||
| aws ec2 create-tags \ | ||
| --resources "$AMI_ID" \ | ||
| --tags \ | ||
| Key=marketplace,Value=ready \ | ||
| Key=version,Value=${{ github.ref_name || 'latest' }} \ | ||
| Key=stage,Value=${{ env.STAGE }} \ | ||
| --region "${{ env.AWS_REGION }}" | ||
|
|
||
| echo "✅ AMI is now publicly available in Community AMIs" | ||
|
|
||
| # Verify public status | ||
| aws ec2 describe-image-attribute \ | ||
| --image-id "$AMI_ID" \ | ||
| --attribute launchPermission \ | ||
| --region "${{ env.AWS_REGION }}" | ||
| echo "✅ AMI is ready for Marketplace submission" | ||
| echo "📝 AMI is private and unencrypted (Marketplace requirement)" | ||
| echo "🔗 Next: Submit to AWS Marketplace Seller Console" | ||
| echo " https://aws.amazon.com/marketplace/management/products" | ||
|
|
There was a problem hiding this comment.
🛠️ Refactor suggestion | 🟠 Major
Fix job condition and harden tagging.
if: env.AMI_ID && env.STAGE == 'prod'treats AMI_ID as a string, which isn’t a boolean. Use an explicit non-empty check.- Keep the expression for version fallback.
- - name: Tag AMI for Marketplace (prod only)
- if: env.AMI_ID && env.STAGE == 'prod'
+ - name: Tag AMI for Marketplace (prod only)
+ if: ${{ env.AMI_ID != '' && env.STAGE == 'prod' }}
run: |
echo "🏷️ Tagging AMI for AWS Marketplace distribution..."
aws ec2 create-tags \
--resources "$AMI_ID" \
--tags \
Key=marketplace,Value=ready \
Key=version,Value=${{ github.ref_name || 'latest' }} \
Key=stage,Value=${{ env.STAGE }} \
--region "${{ env.AWS_REGION }}"Also, keep EBS unencrypted for Marketplace (already handled in the recipe). (docs.aws.amazon.com)
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| - name: Tag AMI for Marketplace (prod only) | |
| if: env.AMI_ID && env.STAGE == 'prod' | |
| run: | | |
| echo "🌍 Making AMI public for production release..." | |
| aws ec2 modify-image-attribute \ | |
| --image-id "$AMI_ID" \ | |
| --launch-permission "Add=[{Group=all}]" \ | |
| echo "🏷️ Tagging AMI for AWS Marketplace distribution..." | |
| aws ec2 create-tags \ | |
| --resources "$AMI_ID" \ | |
| --tags \ | |
| Key=marketplace,Value=ready \ | |
| Key=version,Value=${{ github.ref_name || 'latest' }} \ | |
| Key=stage,Value=${{ env.STAGE }} \ | |
| --region "${{ env.AWS_REGION }}" | |
| echo "✅ AMI is now publicly available in Community AMIs" | |
| # Verify public status | |
| aws ec2 describe-image-attribute \ | |
| --image-id "$AMI_ID" \ | |
| --attribute launchPermission \ | |
| --region "${{ env.AWS_REGION }}" | |
| echo "✅ AMI is ready for Marketplace submission" | |
| echo "📝 AMI is private and unencrypted (Marketplace requirement)" | |
| echo "🔗 Next: Submit to AWS Marketplace Seller Console" | |
| echo " https://aws.amazon.com/marketplace/management/products" | |
| - name: Tag AMI for Marketplace (prod only) | |
| if: ${{ env.AMI_ID != '' && env.STAGE == 'prod' }} | |
| run: | | |
| echo "🏷️ Tagging AMI for AWS Marketplace distribution..." | |
| aws ec2 create-tags \ | |
| --resources "$AMI_ID" \ | |
| --tags \ | |
| Key=marketplace,Value=ready \ | |
| Key=version,Value=${{ github.ref_name || 'latest' }} \ | |
| Key=stage,Value=${{ env.STAGE }} \ | |
| --region "${{ env.AWS_REGION }}" | |
| echo "✅ AMI is ready for Marketplace submission" | |
| echo "📝 AMI is private and unencrypted (Marketplace requirement)" | |
| echo "🔗 Next: Submit to AWS Marketplace Seller Console" | |
| echo " https://aws.amazon.com/marketplace/management/products" |
🤖 Prompt for AI Agents
.github/workflows/ami-build.yml lines 187-203: the job uses a truthy test for
AMI_ID which treats the string as a boolean; change the job condition to an
explicit non-empty check (e.g. if: env.AMI_ID != '' && env.STAGE == 'prod') so
the step only runs when AMI_ID is present and stage is prod, preserve the
existing version fallback expression for the version tag (${ { github.ref_name
|| 'latest' } }) exactly as-is, and ensure AMI_ID remains quoted in the aws cli
call (e.g. --resources "$AMI_ID") so tagging is robust; no change needed for EBS
encryption because Marketplace requires unencrypted AMIs.
| # Determine network type and generate appropriate configuration | ||
| NETWORK_TYPE=\"$${NETWORK_TYPE:-mainnet}\" | ||
|
|
||
| if [ \"$$NETWORK_TYPE\" = \"custom\" ]; then | ||
| echo 'Creating new custom network with chain ID: '\"$$CHAIN_ID\" | ||
| # Create new network without genesis file | ||
| /app/kwild setup init --root /root/.kwild-new --chain-id \"$$CHAIN_ID\" --rpc.private --db.host tn-postgres $$EXTERNAL_FLAG | ||
| else | ||
| echo 'Joining existing mainnet (tn-v2.1)...' | ||
| # Join existing network using genesis file | ||
| /app/kwild setup init --genesis /opt/tn/configs/network/v2/genesis.json --root /root/.kwild-new --p2p.bootnodes \"4e0b5c952be7f26698dc1898ff3696ac30e990f25891aeaf88b0285eab4663e1#ed25519@node-1.mainnet.truf.network:26656,0c830b69790eaa09315826403c2008edc65b5c7132be9d4b7b4da825c2a166ae#ed25519@node-2.mainnet.truf.network:26656\" --state-sync.enable --state-sync.trusted-providers \"4e0b5c952be7f26698dc1898ff3696ac30e990f25891aeaf88b0285eab4663e1#ed25519@node-1.mainnet.truf.network:26656\" --rpc.private --db.host tn-postgres $$EXTERNAL_FLAG | ||
| fi | ||
|
|
There was a problem hiding this comment.
Validate CHAIN_ID when NETWORK_TYPE=custom to avoid broken init.
If NETWORK_TYPE=custom but CHAIN_ID is unset, /app/kwild setup init --chain-id "" will misconfigure. Add a guard.
- if [ "$$NETWORK_TYPE" = "custom" ]; then
- echo 'Creating new custom network with chain ID: '\"$$CHAIN_ID\"
+ if [ "$$NETWORK_TYPE" = "custom" ]; then
+ if [ -z "$$CHAIN_ID" ]; then
+ echo 'Error: CHAIN_ID must be set when NETWORK_TYPE=custom' >&2
+ exit 1
+ fi
+ echo 'Creating new custom network with chain ID: '\"$$CHAIN_ID\"
# Create new network without genesis file
/app/kwild setup init --root /root/.kwild-new --chain-id \"$$CHAIN_ID\" --rpc.private --db.host tn-postgres $$EXTERNAL_FLAG📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| # Determine network type and generate appropriate configuration | |
| NETWORK_TYPE=\"$${NETWORK_TYPE:-mainnet}\" | |
| if [ \"$$NETWORK_TYPE\" = \"custom\" ]; then | |
| echo 'Creating new custom network with chain ID: '\"$$CHAIN_ID\" | |
| # Create new network without genesis file | |
| /app/kwild setup init --root /root/.kwild-new --chain-id \"$$CHAIN_ID\" --rpc.private --db.host tn-postgres $$EXTERNAL_FLAG | |
| else | |
| echo 'Joining existing mainnet (tn-v2.1)...' | |
| # Join existing network using genesis file | |
| /app/kwild setup init --genesis /opt/tn/configs/network/v2/genesis.json --root /root/.kwild-new --p2p.bootnodes \"4e0b5c952be7f26698dc1898ff3696ac30e990f25891aeaf88b0285eab4663e1#ed25519@node-1.mainnet.truf.network:26656,0c830b69790eaa09315826403c2008edc65b5c7132be9d4b7b4da825c2a166ae#ed25519@node-2.mainnet.truf.network:26656\" --state-sync.enable --state-sync.trusted-providers \"4e0b5c952be7f26698dc1898ff3696ac30e990f25891aeaf88b0285eab4663e1#ed25519@node-1.mainnet.truf.network:26656\" --rpc.private --db.host tn-postgres $$EXTERNAL_FLAG | |
| fi | |
| # Determine network type and generate appropriate configuration | |
| NETWORK_TYPE="$${NETWORK_TYPE:-mainnet}" | |
| if [ "$$NETWORK_TYPE" = "custom" ]; then | |
| # Ensure a chain ID is provided for custom networks | |
| if [ -z "$$CHAIN_ID" ]; then | |
| echo 'Error: CHAIN_ID must be set when NETWORK_TYPE=custom' >&2 | |
| exit 1 | |
| fi | |
| echo 'Creating new custom network with chain ID: '"$$CHAIN_ID" | |
| # Create new network without genesis file | |
| /app/kwild setup init \ | |
| --root /root/.kwild-new \ | |
| --chain-id "$$CHAIN_ID" \ | |
| --rpc.private \ | |
| --db.host tn-postgres $$EXTERNAL_FLAG | |
| else | |
| echo 'Joining existing mainnet (tn-v2.1)...' | |
| # Join existing network using genesis file | |
| /app/kwild setup init \ | |
| --genesis /opt/tn/configs/network/v2/genesis.json \ | |
| --root /root/.kwild-new \ | |
| --p2p.bootnodes "4e0b5c952be7f26698dc1898ff3696ac30e990f25891aeaf88b0285eab4663e1#ed25519@node-1.mainnet.truf.network:26656,0c830b69790eaa09315826403c2008edc65b5c7132be9d4b7b4da825c2a166ae#ed25519@node-2.mainnet.truf.network:26656" \ | |
| --state-sync.enable \ | |
| --state-sync.trusted-providers "4e0b5c952be7f26698dc1898ff3696ac30e990f25891aeaf88b0285eab4663e1#ed25519@node-1.mainnet.truf.network:26656" \ | |
| --rpc.private \ | |
| --db.host tn-postgres $$EXTERNAL_FLAG | |
| fi |
2 participants
Configure AMI build pipeline to comply with AWS Marketplace requirements, enabling commercial distribution through AWS Marketplace seller program.
Changes:
resolves: https://github.com/trufnetwork/truf-network/issues/1249
resolves: https://github.com/trufnetwork/truf-network/issues/1247
The dev AMI is already available too
Summary by CodeRabbit