Skip to content

Commit d91143b

Browse files
committed
feat: delegate generate to sl-5-8 via INPUT_DIR, add prereqs + docs
- Simplify root generate target to delegate to sl-5-8-asset-tools via INPUT_DIR/OUTPUT_DIR variables (removes duplicated pipeline invocation). - Add Prerequisites section with install instructions for Windows (PowerShell, Git Bash), macOS, and Linux. - Add direct Python invocation examples for Make-free usage (PowerShell). - Add debug logging section with Bash and PowerShell syntax. - Add Podman/Docker Desktop troubleshooting table. - Update Option B (manual pipeline) to show paths work from any CWD. - Add custom INPUT_DIR usage to make help output. - Note make clean all as fix for stale QC checker packages. Signed-off-by: jdsika <carlo.van-driesten@bmw.de>
1 parent e76b05e commit d91143b

3 files changed

Lines changed: 104 additions & 31 deletions

File tree

Makefile

Lines changed: 9 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -131,12 +131,11 @@ import pathlib, sys; \
131131
im = pathlib.Path('$(GEN_INPUT)') / 'input_manifest.json'; \
132132
sys.exit('[ERR] No input_manifest.json in $(GEN_INPUT)/. Stage input files first.') if not im.exists() else None; \
133133
"
134-
@mkdir -p "$(GEN_OUTPUT)" 2>/dev/null || "$(PYTHON)" -c "import pathlib; pathlib.Path('$(GEN_OUTPUT)').mkdir(parents=True, exist_ok=True)"
135-
@echo "[INFO] Running asset creation pipeline..."
136-
@cd "$(GEN_INPUT)" && "$(CURDIR)/$(PYTHON)" -m asset_extraction.main \
137-
input_manifest.json \
138-
-config "$(CURDIR)/$(GEN_CONFIGS)" \
139-
-out "$(CURDIR)/$(GEN_OUTPUT)"
134+
@"$(MAKE)" -C "$(ASSET_TOOLS)" generate \
135+
INPUT_DIR="$(CURDIR)/$(GEN_INPUT)" \
136+
OUTPUT_DIR="$(CURDIR)/$(GEN_OUTPUT)" \
137+
VENV="$(CURDIR)/$(VENV)" \
138+
PYTHON="$(CURDIR)/$(PYTHON)"
140139
@echo ""
141140
@echo "[OK] Asset generated in $(GEN_OUTPUT)/"
142141
endif
@@ -186,6 +185,10 @@ help:
186185
@echo " make generate validate Validate the generated asset"
187186
@echo " make generate clean Remove generated/output/ directory"
188187
@echo ""
188+
@echo " You can also run the pipeline for a custom input directory:"
189+
@echo " make -C submodules/sl-5-8-asset-tools generate \\"
190+
@echo " INPUT_DIR=/path/to/input OUTPUT_DIR=/path/to/output"
191+
@echo ""
189192
@echo " make lint Lint (validates asset JSON-LD)"
190193
@echo " make validate Validate generated/output/ asset against SHACL"
191194
@echo ""

README.md

Lines changed: 94 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,50 @@ The asset is generated from source files in `generated/input/` using the pipelin
88

99
All ENVITED-X Dataspace assets are defined according to [EVES-003](https://ascs-ev.github.io/EVES/EVES-003/eves-003.html).
1010

11+
## Prerequisites
12+
13+
| Tool | Why | Install |
14+
|------|-----|---------|
15+
| **Python 3.12+** | Pipeline runtime | [python.org](https://www.python.org/downloads/), Microsoft Store, or `winget install Python.Python.3.12` |
16+
| **Git** | Submodule management | [git-scm.com](https://git-scm.com/downloads) (includes Git Bash on Windows) |
17+
| **GNU Make** | Task runner | see below |
18+
19+
<details>
20+
<summary><strong>Windows — installing Make</strong></summary>
21+
22+
**Git Bash / MSYS2** — Make is included if you selected the MSYS2 option during Git install, or install it with:
23+
24+
```bash
25+
# Using Scoop (recommended)
26+
scoop install make
27+
28+
# Using Chocolatey
29+
choco install make
30+
```
31+
32+
**PowerShell** — The same `scoop` or `choco` commands work. After installing, `make` uses Git's bundled `sh.exe` for recipe execution, so it works from PowerShell, Git Bash, and CMD alike.
33+
34+
**Without Make** — You can run the pipeline directly with Python (see [Option B](#option-b--run-the-pipeline-directly) below).
35+
36+
</details>
37+
38+
<details>
39+
<summary><strong>macOS / Linux</strong></summary>
40+
41+
Make and Git are typically pre-installed. Install Python 3.12+ via your package manager:
42+
43+
```bash
44+
# macOS
45+
brew install python@3.12
46+
47+
# Ubuntu/Debian
48+
sudo apt install python3.12 python3.12-venv
49+
```
50+
51+
</details>
52+
53+
> **Podman** is only needed for `make wizard` (the SD Creation Wizard web UI). The pipeline itself does not require it. Podman is auto-installed by `make setup wizard` if missing.
54+
1155
## Quick Start
1256

1357
```bash
@@ -160,11 +204,12 @@ To create a new HD-Map asset from **your own** `.xodr` file, you have two option
160204

161205
The pipeline will run all steps and produce a fresh asset in `generated/output/`.
162206

163-
### Option B — Run the pipeline manually
207+
### Option B — Run the pipeline directly
164208

165-
Create a working directory with all your input files **and** an `uploadedFiles.json` manifest:
209+
You can invoke the pipeline from any directory without `make`. This works in **any shell** — Bash, PowerShell, CMD:
166210

167211
```bash
212+
# Create a working directory with your input files
168213
mkdir -p my_asset/input my_asset/output
169214

170215
# Copy your files into the input directory
@@ -174,7 +219,11 @@ cp path/to/documentation.pdf my_asset/input/
174219
cp LICENSE my_asset/input/
175220
```
176221

177-
Create `my_asset/input/uploadedFiles.json`:
222+
Create `my_asset/input/input_manifest.json` (JSON-LD format — recommended):
223+
224+
> See `generated/input/input_manifest.json` in this repo for a complete example.
225+
226+
Or create `my_asset/input/uploadedFiles.json` (legacy array format):
178227

179228
```json
180229
[
@@ -202,28 +251,23 @@ Create `my_asset/input/uploadedFiles.json`:
202251
]
203252
```
204253

205-
Run the pipeline:
254+
Run the pipeline (paths can be relative or absolute — no need to `cd` into the input directory):
206255

207256
```bash
208-
cd my_asset/input
209-
210-
# Windows:
211-
../../.venv/Scripts/python -m asset_extraction.main \
212-
uploadedFiles.json \
213-
-config ../../submodules/sl-5-8-asset-tools/configs \
214-
-out ../output
215-
216-
# macOS/Linux:
217-
../../.venv/bin/python -m asset_extraction.main \
218-
uploadedFiles.json \
219-
-config ../../submodules/sl-5-8-asset-tools/configs \
220-
-out ../output
257+
# Bash / Git Bash / macOS / Linux:
258+
.venv/bin/python -m asset_extraction.main \
259+
my_asset/input/uploadedFiles.json \
260+
-config submodules/sl-5-8-asset-tools/configs \
261+
-out my_asset/output
221262
```
222263

223-
> ⚠️ **Important path rules:**
224-
> - All input files **must be in the same directory** as `uploadedFiles.json`
225-
> - Use **bare filenames** (no subdirectories) in `uploadedFiles.json`
226-
> - The **output directory must be separate** from the input directory
264+
```powershell
265+
# PowerShell:
266+
.\.venv\Scripts\python.exe -m asset_extraction.main `
267+
my_asset\input\uploadedFiles.json `
268+
-config submodules\sl-5-8-asset-tools\configs `
269+
-out my_asset\output
270+
```
227271

228272
Supported file types and categories:
229273

@@ -308,11 +352,27 @@ You need to use the following ontology from [Ontology Management Base Repository
308352

309353
### What does the pipeline need to run?
310354

311-
- **Python 3.12+** and `make`
312-
- The `sl-5-8-asset-tools` submodule (initialized via `git submodule update --init --recursive`)
355+
- **Python 3.12+**, **Git**, and **GNU Make** (see [Prerequisites](#prerequisites))
356+
- The `sl-5-8-asset-tools` submodule (initialized via `git submodule update --init --recursive` or `make setup`)
313357
- **Internet connection** — required only for reverse geocoding (Nominatim API); ontology schemas and SHACL shapes are provided locally by the `ontology-management-base` submodule
314358
- For quality checking (optional): run `make setup qc` to install the ASAM and OpenMSL checker tools from GitHub
315359

360+
### Debug logging
361+
362+
```bash
363+
# Bash / Git Bash / macOS / Linux:
364+
SL58_LOG_MODE=debug make generate
365+
366+
# PowerShell:
367+
$env:SL58_LOG_MODE = "debug"; make generate
368+
369+
# Or with direct Python invocation:
370+
$env:SL58_LOG_MODE = "debug"
371+
.\.venv\Scripts\python.exe -m asset_extraction.main ...
372+
```
373+
374+
Shows full subprocess command lines, stdout/stderr, and tracebacks.
375+
316376
### What is the `uploadedFiles.json`?
317377

318378
It's a simple JSON array that tells the pipeline which files to process and how to categorize them. Each entry has:
@@ -326,5 +386,15 @@ When using `make generate`, this file is created automatically.
326386
### Known Limitations
327387

328388
- **Geocoding timeouts** — The `meta_data_extractor` uses the Nominatim API for reverse geocoding. If the API is slow or rate-limited, the pipeline may fail. Simply retry.
329-
- **Quality checkers require extra install** — The `[qc]` optional dependencies install from Git branches and may take a while. Without them, quality checking is skipped (no validation reports generated).
389+
- **Quality checkers require extra install** — The `[qc]` optional dependencies install from Git branches and may take a while. Without them, quality checking is skipped (no validation reports generated). If you see stale checker errors after switching branches, run `make clean all && make setup` to get a fresh environment.
330390
- **Ontology versions** — The pipeline generates metadata using the **latest** ontology versions. Both current and older versions are valid but produce different JSON-LD structures.
391+
392+
### Podman / Wizard Troubleshooting
393+
394+
The SD Creation Wizard (`make wizard`) uses Podman containers. Common issues on Windows:
395+
396+
| Error | Cause | Fix |
397+
|-------|-------|-----|
398+
| `CreateFile \\.\pipe\docker_engine: All pipe instances are busy` | Docker Desktop is running and holds the Docker API pipe | Quit Docker Desktop before starting Podman, or run `podman machine stop && podman machine start` |
399+
| `machine not in running state` | Podman machine failed to start | Run `podman machine rm` then `podman machine init && podman machine start` |
400+
| Port 4200 or 8080 already in use | Another process holds the port | `make wizard stop` first, or stop the other process |

0 commit comments

Comments
 (0)