Skip to content

Project-Tree.md

Latest commit

 

History

History
121 lines (101 loc) · 4.14 KB

Project-Tree.md

File metadata and controls

121 lines (101 loc) · 4.14 KB

Project Directory Tree

Projecy directory tree inspired by Yasen structure of Transitional-Master-Payer project which my inpired by other former projext but evolve to be used and resused for other similar automation pipelines.

Directory tree structure should be consistant for different but similar projects to:

  • Easier to understand who's who
  • Easier to know which doing what
  • Easier to know look where for what
  • Easier to know where to look for what

Ironman

Ironman is the name I choose for Yasen development of execution manifest which addresses two issues:

  • Extend AWS CodeBuild single build phase up to 100 Tasks or more
  • Replaces Ansible and eleminated its limitations of:
    • Elemninate requiring result for task with output
    • Eleminat limited screen width in CodeBuild builds
    • Free of limiting to only json esacped outputs or yaml

Data folder config

Transitional-Master-Payer used DATAFOLDER as a data folder off of the project itsels in /tmp/<proj-name>. It is a best practice to separate execution outcome and logs off of the project and locally store in in /tmp and in the pipeline copy it to a persistanst storage from execution server /tmp/<path> to respective persistanst storage path.

# Local / project data folder
export PROJ="dev-divest"
export DATAFOLDER="/tmp/${PROJ}"
mkdir -p "$DATAFOLDER"
export DATAFOLDER="$DATAFOLDER"

# Local / project builds data
export DATAFOLDER="/tmp/${PROJ}/${ACCOUNT_ALIAS}"
mkdir -p "$DATAFOLDER"

# Server local file / each build data folder / in build phase istall
export DATAFOLDER="/tmp"

# Pre only since we capture current state and creating backup to consume it later
[ 'pre' == "$STAGE" ] && export DATAFOLDER="/tmp/pre"

export S3_BUCKET="<project-bucket>"

# Server Storage / each build data folder / in build phase finally
# Pre only since we capture current state and creating backup
s3 cp "$DATAFOLDER" "${S3_BUCKET}/${PROJ}/${ACCOUNT_ALIAS}"

Directory Tree Structure

Tree Purpose File Types
docs Documentation .md, ...
images Image files .jpg, .png, ...
src Source code code, config, input
src/bin bin folder .py, ...
src/bin/not-used Deprecated or replaced code .py, ...
src/buildspec BuildSpec Yaml files .yml
src/buildtask Build Task Yaml files .yml
src/config Configurations .yml, .json, .toml, ...
src/exec Execution files .sh, ...
src/exec/OLD Deprecated or replaced code .sh, ...
src/inputs Input files .dat, .tab, ...
src/mappings Mapping files .json, .py, .sh, .yml, ...
src/pre pre tasks,
src/pre/tests dev testing ...
src/tags not-used .json
src/tests Py Tests & dependent files .json, .py, .sh, .yml
utils Other utilities
verify-accounts Verif AWS Leave Org Requirements .py

Creating Project Directory Tree

Edit and add or comment not relevant lines. Lines which starts with # in PATHS will be excluded from tree structure.

Example:

# execute once
# PROJ='Export-CMC-SecOps-Data'
# mkdir -p "${PROJ}"
# cd "${PROJ}"; 

PROJ_PATHS="
docs
images
src
src/bin
# src/bin/not-used
# src/bin/lambda
# src/buildspec
# src/buildtask
src/config
src/exec
# src/exec/OLD
src/inputs
# src/mappings
# src/pre
# src/pre/tests
# src/tags
src/tests
# utils
# verify-accounts
"

# dryrun
while read -r iPATH; do echo "mkdir -p ${iPATH}"; done < <(printf '%s\n' "${PROJ_PATHS}" | grep -Ev '^#|^$' )

# execute
while read -r iPATH; do mkdir -p "${iPATH}"; done < <(printf '%s\n' "${PROJ_PATHS}" | grep -Ev '^#|^$' )

Build Task

# from project src/ run
export OPT1='--definition-file buildtask/divest.pre.tasks.yml'
export OPT2='--folder buildtask'
export OPT3='--keep-relative-path'

python bin/generate-shell-script.py $OPT1 $OPT2 $OPT3 1> /dev/null

# other build tasks
buildtask/divest.pre.tasks.yml  # pre
buildtask/divest.run.tasks.yml  # process
buildtask/divest.set.tasks.yml  # post