ATP Processor - Package Documentation

Overview

atp_processor is a CLI-driven Python package for performing bulk operations against the TargetProcess (TP) API v1. It reads the JSON schemas generated by the internal generator module to validate input, coerce types, and construct API requests. All mutating operations are logged to a rollback journal so they can be reversed.

python -m atp_processor --resource Epic --action create --input epics.csv

Package Structure

atp_processor/
    __init__.py
    __main__.py
    config.py
    client.py
    schema.py
    generator.py
    io_handlers.py
    result.py
    rollback.py
    operations/
        __init__.py
        crud.py
        query.py
        references.py
        advanced.py
        audit.py
        relations.py
        reporting.py
        migration.py

Dependency Graph

__main__.py
    |
    +-- config.py           (auth + paths)
    +-- schema.py           (schema loading + validation)
    +-- generator.py        (schema generation)
    +-- client.py           (HTTP requests)
    +-- rollback.py         (operation journaling)
    |
    +-- operations/
            |
            +-- crud.py         --> io_handlers, result, client, schema, rollback
            +-- query.py        --> io_handlers, client, schema
            +-- references.py   --> io_handlers, result, client, schema, rollback
            +-- advanced.py     --> io_handlers, result, client, schema, rollback
            +-- audit.py        --> io_handlers, client, schema
            +-- relations.py    --> io_handlers, result, client, schema, rollback
            +-- reporting.py    --> io_handlers, client, schema
            +-- migration.py    --> io_handlers, result, client, schema, rollback

Core Modules

`init.py`

Package marker. Exposes __version__.

Symbol	Type	Description
`__version__`	`str`	Package version tag

`main.py`

CLI entry point. Invoked via python -m atp_processor.

Function	Description
`build_parser()`	Constructs the `argparse.ArgumentParser` with all global flags, action choices, and action-specific args.
`main()`	Parses args, initializes `Config`, `SchemaRegistry`, `TPClient`, and `RollbackLog`, then dispatches.
`_dispatch(args, ...)`	Routes the `--action` value to the correct operation function in `operations/`.
`_describe_resource()`	Prints a human-readable summary of a resource schema (properties, references, CRUD permissions).

Global CLI flags:

Flag	Description
`--token`	API token override (highest priority)
`--dry-run`	Validate without making API calls
`--output` / `-o`	Output file path for exports (.csv or .json)
`--results`	Path to write per-row operation results JSON
`--no-rollback`	Disable rollback journal for this session
`--list-resources`	Print all resource names and exit
`--describe`	Print schema details for a named resource
`--resource` / `-r`	Target resource type (e.g., `Epic`, `Bug`)
`--action` / `-a`	Operation to perform (19 choices)
`--input` / `-i`	Input file (.csv or .json)
`--ids`	Comma-separated entity IDs or path to ID file

`config.py`

Resolves authentication and filesystem paths.

Class	`Config`
Init	`Config(token_cli=None)` - Loads `.env`, resolves token from CLI > `ATP_TOKEN` > `ACCESS_TOKEN`.

Attribute	Type	Description
`token`	`str`	Resolved API token
`schema_ref`	`str`	Absolute path to `schema_ref.json`
`schema_dir`	`str`	Absolute path to `schemas/` directory
`BASE_URL`	`str`	Class-level constant: TP API v1 base URL

Method	Returns	Description
`resource_url(resource_name)`	`str`	Fallback URL builder: `{BASE_URL}/{name}s`

Auth priority (first found wins):

--token CLI flag
ATP_TOKEN environment variable
ACCESS_TOKEN from .env

`client.py`

HTTP client wrapping requests.Session. Handles auth, retries, and pagination.

Class	`TPClient`
Init	`TPClient(config)` - Creates a session with token and JSON headers.

Method	Returns	Description
`get(url, params=None)`	`Response`	GET request with retry.
`post(url, json_body=None)`	`Response`	POST request with retry.
`delete(url)`	`Response`	DELETE request with retry.
`_request(method, url, **kwargs)`	`Response`	Internal: executes request with exponential backoff on 429/5xx (3 tries).
`get_entity(base_url, entity_id, include=None)`	`Response`	GET `/Resource/{id}` with optional include fields.
`search(base_url, where, include, order_by, take, skip)`	`Response`	GET `/Resources` with query params.
`search_all(base_url, where, include, order_by, page_size=100)`	`Generator[dict]`	Auto-paginating generator yielding all matching items.
`create_entity(base_url, body)`	`Response`	POST to create a new entity.
`update_entity(base_url, entity_id, body)`	`Response`	POST to `/Resource/{id}` to update fields.
`delete_entity(base_url, entity_id)`	`Response`	DELETE `/Resource/{id}`.

Associations: Receives Config for token. Used by every operation module.

`schema.py`

Loads generated JSON schemas from disk and provides validation/introspection.

Class	`SchemaRegistry`
Init	`SchemaRegistry(config)` - Loads `schema_ref.json`, lazily caches per-resource.

Method	Returns	Description
`list_resources()`	`list[str]`	Sorted list of all known resource names (186+).
`has_resource(name)`	`bool`	Check if a resource name exists.
`get_schema(resource_name)`	`dict`	Full schema dict (resource, canCreate/Update/Delete, properties, references).
`get_uri(resource_name)`	`str`	API collection endpoint URL (strips `/meta` suffix from schema URI).
`get_settable_properties(resource_name)`	`dict`	Properties where `canSet=True`.
`get_required_properties(resource_name)`	`dict`	Properties and references where `isRequired=True`.
`get_settable_references(resource_name)`	`dict`	References where `canSet=True`.
`validate_row(resource_name, row_data, mode)`	`list[str]`	Validates a data row. Returns list of error strings (empty = valid).
`coerce_value(field_name, raw_value, resource_name)`	`any`	Converts a CSV string to the correct Python type based on schema type info.

validate_row checks (by mode):

Check	`create`	`update`	`delete`
CRUD permission	Yes	Yes	Yes
Unknown fields	Yes	Yes	-
Non-settable fields	Yes	Yes	-
Required fields present	Yes	-	-
`Id` field present	-	Yes	-
Reference format	Yes	Yes	-

Associations: Receives Config for file paths. Used by every operation module and __main__.py.

`generator.py`

Fetches resource metadata from the TP API index and meta endpoints to generate JSON schemas.

Function	Description
`generate_all(config)`	Orchestrates the full schema generation process. Fetches index, loops resources, fetches metadata, and writes schema files.
`SchemaFactory`	Class handling the API interaction and schema generation logic.

`io_handlers.py`

Reads and writes CSV/JSON files. Handles type coercion for CSV input.

Function	Returns	Description
`detect_format(filepath)`	`str`	Returns `"csv"` or `"json"` based on file extension.
`read_input(filepath, resource_name=None, schema_registry=None)`	`list[dict]`	Reads CSV or JSON input. CSV values are coerced via schema if registry given.
`write_output(filepath, data, fields=None)`	`None`	Writes data to CSV or JSON based on extension.
`read_id_list(filepath_or_string)`	`list[int]`	Parses `"1,2,3"` string or reads file with one ID per line.

CSV coercion rules (when schema_registry is provided):

Schema Type	Python Type	Example
`Int32`, `Int64`	`int`	`"42"` -> `42`
`Double`,`Decimal`	`float`	`"3.14"` -> `3.14`
`Boolean`	`bool`	`"true"` -> `True`
`String`,`DateTime`	`str`	`"hello"` -> `"hello"`
Reference fields	`dict`	`"42"` -> `{"Id": 42}`

Associations: Used by all operation modules. Optionally receives SchemaRegistry for coercion.

`result.py`

Tracks per-row success/failure for bulk operations.

`OperationResult`

Represents the outcome of a single row/entity operation.

Attribute	Type	Description
`row_index`	`int`	Position in the input file
`input_data`	`dict`	The data that was sent
`success`	`bool`	Whether the operation succeeded
`response_data`	`dict	None`
`error`	`str	None`
`entity_id`	`int	None`
`action`	`str	None`
`timestamp`	`str`	ISO 8601 UTC timestamp

Method	Returns	Description
`to_dict()`	`dict`	Serializable dict of result

`ResultSet`

Collection of OperationResult objects for a batch.

Property/Method	Returns	Description
`succeeded`	`list`	Results where `success=True`
`failed`	`list`	Results where `success=False`
`total`	`int`	Total number of results
`add(result)`	`None`	Append an `OperationResult`
`print_summary(file=...)`	`None`	Print operation/resource/counts/failures to console
`to_json(filepath=None)`	`dict`	Serialize to JSON. Writes to file if filepath given.

Associations: Used by all mutating operation modules (crud, references, advanced, relations, migration).

`rollback.py`

Append-only JSONL journal for reversing operations.

Class	`RollbackLog`
Init	`RollbackLog(log_dir=None)` - Creates `.atp_logs/` dir and session log file.

Attribute	Type	Description
`log_dir`	`str`	Directory for log files (default: `.atp_logs`)
`session_id`	`str`	Timestamp-based session identifier
`log_file`	`str`	Path to current session's JSONL file
`entries`	`list`	In-memory list of journal entries

Method	Description
`log_create(resource, entity_id, sent_data)`	Records a create. Rollback action: DELETE the entity.
`log_update(resource, entity_id, old_data, new_data)`	Records an update. Rollback action: POST old_data back.
`log_delete(resource, entity_id, old_data)`	Records a delete. Rollback action: POST old_data to recreate.
`_append(entry)`	Internal: writes entry to JSONL file and in-memory list.
`load_log(filepath)` (static)	Reads a JSONL log file into a list of dicts.
`list_logs(log_dir=None)` (static)	Lists all rollback files, newest first.

JSONL entry format:

{"action":"create","resource":"Epic","entity_id":5001,"old_data":null,"new_data":{...},"timestamp":"...","session_id":"..."}

Associations: Created by __main__.py, passed to all mutating operation functions. Read by migration.execute_rollback.

Operation Modules

All operation functions share a common signature pattern:

def operation_name(args, registry, client, rollback_log):

args - Parsed CLI arguments (argparse.Namespace)
registry - SchemaRegistry instance
client - TPClient instance
rollback_log - RollbackLog instance or None

Read-only operations omit rollback_log.

`operations/crud.py`

Core create, update, and delete operations.

Function	Action	Input Source	HTTP Method
`bulk_create(args, registry, client, rollback_log)`	`create`	`--input` (CSV/JSON)	POST `/Resources`
`bulk_update(args, registry, client, rollback_log)`	`update`	`--input` (CSV/JSON)	POST `/Resources/{id}`
`bulk_delete(args, registry, client, rollback_log)`	`delete`	`--ids` or `--input`	DELETE `/Resources/{id}`

bulk_create - Reads rows from --input, validates each against the schema (required fields, settable fields), POSTs each row. Logs created IDs to rollback.

bulk_update - Reads rows from --input (must include Id). GETs old data for rollback, POSTs update body (Id excluded from body). Validates with mode="update".

bulk_delete - Accepts IDs from --ids (comma-separated or file) or --input (Id column). GETs old data for rollback before each DELETE.

Associations: Imports read_input, read_id_list from io_handlers. Imports OperationResult, ResultSet from result.

`operations/query.py`

Read-only search and export operations.

Function	Action	Output
`search(args, registry, client)`	`search`	stdout (JSON) or `--output` file
`export(args, registry, client)`	`export`	`--output` file (required)

search - GETs entities with --where, --include, --order-by, --take, --skip. Use --all for auto-pagination. Prints JSON to stdout or writes to --output.

export - Always auto-paginates (fetches all). Always writes to --output. Supports CSV and JSON output.

CLI flags used: --where, --include, --order-by, --take, --skip, --all, --output

Associations: Imports write_output from io_handlers.

`operations/references.py`

Update reference fields and entity state on existing entities.

Function	Action	Description
`set_references(args, registry, client, rollback_log)`	`set-ref`	Set a reference field on one or more entities.
`set_state(args, registry, client, rollback_log)`	`set-state`	Bulk update `EntityState` on entities.

set_references - Two modes:

Bulk mode: --ids "1,2,3" --ref-field Milestone --ref-id 42 applies the same reference to all IDs.
File mode: --input refs.csv where each row has Id plus reference columns.

set_state - Accepts --ids and --state. State can be an integer ID or a name string (resolved via EntityState API query).

Associations: Imports read_id_list, read_input from io_handlers. Imports OperationResult, ResultSet from result.

`operations/advanced.py`

Composite operations that chain multiple API calls.

Function	Action	Description
`clone(args, registry, client, rollback_log)`	`clone`	Copy an entity N times with optional field overrides.
`hierarchical_create(args, registry, client, rollback_log)`	`hier-create`	Create parent + children from a JSON tree.
`upsert(args, registry, client, rollback_log)`	`upsert`	Search-then-create-or-update (idempotent sync).

clone - GETs entity by --source-id, extracts settable fields, applies --overrides (JSON string or file), POSTs --count copies. Appends (Copy N) to Name when count > 1.

hierarchical_create - Reads a JSON tree from --input with nested resource, data, and children keys. Creates parent first, captures its ID, auto-links children via the parent's resource name as a reference field. Recursively processes nested children. Aborts a branch if the parent creation fails.

upsert - For each input row, searches by --match-field (default: Name). If found, updates. If not, creates. Makes the input file idempotent.

CLI flags used: --source-id, --count, --overrides, --match-field, --input

Associations: Imports read_input from io_handlers. Imports OperationResult, ResultSet from result.

`operations/audit.py`

Non-mutating operations: no writes to the API.

Function	Action	API Calls	Description
`diff(args, registry, client)`	`diff`	Read-only	Compare local file vs API state.
`validate(args, registry)`	`validate`	None	Check input against schema (fully offline).

diff - Reads a local file (--local-file), indexes by Id. Fetches matching entities from the API (using --where or by ID list). Compares field-by-field and reports: only-local, only-remote, modified (with per-field diffs), and identical.

validate - Reads --input, runs schema.validate_row() on each row. Prints pass/fail per row. Makes zero API calls -- works without network.

CLI flags used: --local-file, --where, --input, --output

Associations: Imports read_input, write_output from io_handlers. validate only needs SchemaRegistry (no client).

`operations/relations.py`

Create join-table entities: Relation, Assignment, Comment.

Function	Action	TP Resource	Description
`add_dependencies(args, registry, client, rollback_log)`	`add-dependency`	`Relation`	Link entities together via Relation.
`assign_users(args, registry, client, rollback_log)`	`assign-users`	`Assignment`	Assign users to work items.
`add_comments(args, registry, client, rollback_log)`	`add-comment`	`Comment`	Post comments to entities.

add_dependencies - Reads --input CSV/JSON with InboundId, OutboundId, RelationTypeId columns. Creates Relation entities.

assign_users - Two modes:

--ids + --user-ids (+ optional --role-id): creates the cross-product of entity IDs x user IDs.
--input file with AssignableId, UserId, RoleId columns.

add_comments - Two modes:

--ids + --comment: same comment text posted to each entity.
--input file with EntityId (or GeneralId) and Description (or Comment) columns.

Associations: Imports read_input, read_id_list from io_handlers. Imports OperationResult, ResultSet from result.

`operations/reporting.py`

Multi-resource data retrieval.

Function	Action	Description
`cross_resource_report(args, registry, client)`	`cross-report`	Query multiple resources, merge into one output.

cross_resource_report - Takes --resources "Epic,Feature,Bug", queries each with the same --where filter, tags each item with _ResourceType, merges all into a single --output file (CSV or JSON).

CLI flags used: --resources, --where, --include, --output

Associations: Imports write_output from io_handlers.

`operations/migration.py`

Project migration and rollback execution.

Function	Action	Description
`migrate(args, registry, client, rollback_log)`	`migrate`	Move entities between projects by updating `Project`.
`execute_rollback(args, registry, client)`	`rollback`	Reverse operations from a rollback journal.

migrate - Accepts --ids or --where to select entities, plus --to-project. Updates the Project reference on each entity. GETs old data for rollback before each update.

execute_rollback - Loads the most recent JSONL log (or --log-file). Processes entries in reverse order:

Original Action	Rollback Action
`create`	DELETE the created entity
`update`	POST the old_data back (restore original field values)
`delete`	POST old_data to recreate (entity gets a new ID)

Only settable fields are included in restore/recreate bodies.

CLI flags used: --ids, --where, --to-project, --from-project, --log-file

Associations: Imports read_id_list from io_handlers. Imports RollbackLog from rollback. Imports OperationResult, ResultSet from result.

Data Flow

Input File (.csv / .json)
        |
        v
   io_handlers.read_input()  ----> schema.coerce_value()  (CSV only)
        |
        v
   schema.validate_row()     ----> errors[] or pass
        |
        v
   client.create/update/delete_entity()
        |
        +---> result.OperationResult (per row)
        +---> rollback.RollbackLog   (per mutation)
        |
        v
   result.ResultSet.print_summary()
   result.ResultSet.to_json()       (if --results flag)

All 19 CLI Actions

#	Action	Module	Function	Mutates API	Uses Rollback
1	`create`	`crud.py`	`bulk_create`	Yes	Yes
2	`update`	`crud.py`	`bulk_update`	Yes	Yes
3	`delete`	`crud.py`	`bulk_delete`	Yes	Yes
4	`search`	`query.py`	`search`	No	No
5	`export`	`query.py`	`export`	No	No
6	`set-ref`	`references.py`	`set_references`	Yes	Yes
7	`set-state`	`references.py`	`set_state`	Yes	Yes
8	`clone`	`advanced.py`	`clone`	Yes	Yes
9	`hier-create`	`advanced.py`	`hierarchical_create`	Yes	Yes
10	`upsert`	`advanced.py`	`upsert`	Yes	Yes
11	`diff`	`audit.py`	`diff`	No	No
12	`validate`	`audit.py`	`validate`	No (offline)	No
13	`add-dependency`	`relations.py`	`add_dependencies`	Yes	Yes
14	`assign-users`	`relations.py`	`assign_users`	Yes	Yes
15	`add-comment`	`relations.py`	`add_comments`	Yes	Yes
16	`cross-report`	`reporting.py`	`cross_resource_report`	No	No
17	`rollback`	`migration.py`	`execute_rollback`	Yes	No
18	`migrate`	`migration.py`	`migrate`	Yes	Yes
19	`generate-schemas`	`generator.py`	`generate_all`	No (Files)	No

Name		Name	Last commit message	Last commit date
Latest commit History 3 Commits
archive		archive
atp_processor		atp_processor
docs		docs
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
requirements.txt		requirements.txt

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

ATP Processor - Package Documentation

Overview

Package Structure

Dependency Graph

Core Modules

`init.py`

`main.py`

`config.py`

`client.py`

`schema.py`

`generator.py`

`io_handlers.py`

`result.py`

`OperationResult`

`ResultSet`

`rollback.py`

Operation Modules

`operations/crud.py`

`operations/query.py`

`operations/references.py`

`operations/advanced.py`

`operations/audit.py`

`operations/relations.py`

`operations/reporting.py`

`operations/migration.py`

Data Flow

All 19 CLI Actions

About

Uh oh!

Releases

Packages

Uh oh!

Contributors

Uh oh!

Languages

Folders and files

Latest commit

History

Repository files navigation

ATP Processor - Package Documentation

Overview

Package Structure

Dependency Graph

Core Modules

__init__.py

__main__.py

config.py

client.py

schema.py

generator.py

io_handlers.py

result.py

OperationResult

ResultSet

rollback.py

Operation Modules

operations/crud.py

operations/query.py

operations/references.py

operations/advanced.py

operations/audit.py

operations/relations.py

operations/reporting.py

operations/migration.py

Data Flow

All 19 CLI Actions

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

`init.py`

`main.py`

`config.py`

`client.py`

`schema.py`

`generator.py`

`io_handlers.py`

`result.py`

`OperationResult`

`ResultSet`

`rollback.py`

`operations/crud.py`

`operations/query.py`

`operations/references.py`

`operations/advanced.py`

`operations/audit.py`

`operations/relations.py`

`operations/reporting.py`

`operations/migration.py`

Packages