Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
49 commits
Select commit Hold shift + click to select a range
64109c1
add the first version of the preprocessing using edge instead of coll…
godinlu Mar 25, 2026
a882df3
add a strategy based model for the image rectification and a pdf report
godinlu Mar 27, 2026
664108b
add the first version of the collimation strategy
godinlu Apr 3, 2026
f8b597a
improve the model and the qc generation
godinlu Apr 3, 2026
b9de5d0
add the compute grid for collimation
godinlu Apr 3, 2026
01e814d
small correction
godinlu Apr 3, 2026
1d72c23
add fitted time and fitted date
godinlu Apr 3, 2026
711363d
refactor of the class model to able pipeline
godinlu Apr 8, 2026
b0ffe74
small update for qc
godinlu Apr 8, 2026
4fdc6d2
update the image mosaic function and add a pipeline for rectification
godinlu Apr 9, 2026
298ee4f
add the first step of the fiducial_rectification_strategy
godinlu Apr 16, 2026
976114c
remove strategy
godinlu Apr 17, 2026
24656ca
first draft of the kh9pc pipeline
godinlu Apr 17, 2026
7ab759a
add the final pdf report
godinlu Apr 22, 2026
e7e5efd
add a CLI entrie point for the pipeline and clean tqdm for log file
godinlu Apr 22, 2026
76bb706
change logger config to set only hipp loger to choose level
godinlu Apr 22, 2026
5c9fde5
improve pbar logging
godinlu Apr 22, 2026
67137bf
replace output-dir by output
godinlu Apr 23, 2026
3e350bf
improve pipeline
godinlu Apr 23, 2026
8542f84
save big refactor
godinlu Apr 25, 2026
3af2c8a
improve the code quality
godinlu Apr 29, 2026
b2cb9e9
add the first version of fiducial strategy
godinlu Apr 30, 2026
ff779b6
improve the vertical detection by adding a additional criteria of gra…
godinlu May 6, 2026
ac943e0
add the outlier filtering for fiducials
godinlu May 20, 2026
9237903
improve code quality
godinlu May 21, 2026
64f7ab7
change intern class result store
godinlu May 21, 2026
12a5463
add transform and improve quality control of fiducial strategy
godinlu May 21, 2026
806a04e
improve quality_control and include poly_strategy in collimation_stra…
godinlu May 21, 2026
6265f49
code review
godinlu May 21, 2026
bce3c6f
code review
godinlu May 21, 2026
4b40517
improve the robsutness of strategy
godinlu May 22, 2026
1f2d530
small changes
godinlu May 27, 2026
47a95c9
increase the right edge detection window and otpimize clustering on t…
godinlu May 28, 2026
ad675ca
invert y axis
godinlu May 28, 2026
d14a24d
improve qc plot
godinlu May 28, 2026
e8ed5f4
be more robust on transform
godinlu May 28, 2026
446a37f
add an adaptative template choice depend on mission number
godinlu Jun 5, 2026
caf9ce8
big codebase refactor
godinlu Jun 10, 2026
dbaed08
adapt plot functions and clean
godinlu Jun 11, 2026
bbcdfb6
improving the score function for segmented
godinlu Jun 12, 2026
1f1af95
restrict the pattern research depend on mission and improve pattern s…
godinlu Jun 17, 2026
e9c22f2
file refactor
godinlu Jun 17, 2026
29cfa90
improving vertical detection with a succes ratio of 254/255
godinlu Jun 17, 2026
c3b4f84
switch the strategy model for a function based fiducials pattern
godinlu Jun 19, 2026
cf413e8
fix the transform function and test the fiducial restitution
godinlu Jun 24, 2026
b7f85a5
add metrics
godinlu Jun 24, 2026
d2ad9c4
implement batch process and process single image
godinlu Jun 24, 2026
9cb02c5
correct the cropping
godinlu Jun 24, 2026
2a1a293
small imrpovement of the pipeline
godinlu Jun 25, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -185,4 +185,6 @@ notebooks/data/*/*
# ignore the data test_images
tests/aerial/data

dev/
dev/
*.local*
.claude
105 changes: 105 additions & 0 deletions CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,105 @@
# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Overview

**HIPP (Historical Image Pre-Processing)** is a Python library for preprocessing scanned historical aerial and satellite images for photogrammetric analysis (Structure from Motion). It supports:
- Aerial images with fiducial markers (USGS, NAGAP datasets)
- Declassified US reconnaissance satellite images (KH-9 Hexagon panoramic/mapping camera, KH-4/4A/4B Corona)

## Commands

The project uses **Hatch** as the project manager.

```bash
hatch shell dev # Enter development environment
hatch run dev:check # Type check (mypy --strict) + lint (ruff)
hatch run dev:pytest # Run tests
hatch run dev:lab # Start Jupyter Lab on port 8333
hatch run dev:kernel # Install IPython kernel
```

Direct commands (inside `hatch shell dev`):
```bash
pytest # All tests
pytest tests/aerial/test_core.py # Single test file
ruff check . # Lint
mypy src/ --strict --ignore-missing-imports --no-warn-unused-ignores --allow-untyped-calls
pre-commit install # Install git hooks (run once after cloning)
```

Line length is 120 characters. Pre-commit hooks run ruff + mypy on every commit.

## Architecture

### Package Layout (`src/hipp/`)

```
hipp/
├── image.py # Low-level image I/O, CLAHE, resizing
├── math.py # Geometric transforms, matrix ops
├── intrinsics.py # Intrinsics class (camera calibration parameters)
├── tools.py # GUI point picking, archive extraction, quickviews
├── aerial/ # Fiducial-based aerial image preprocessing
│ ├── core.py # Main pipeline: template creation → detection → restitution
│ ├── fiducials.py # Fiducial marker detection, matching, transformation
│ └── quality_control.py
├── kh9pc/ # KH-9 panoramic camera preprocessing
│ ├── pipeline.py # End-to-end orchestration (PipelineStep, KH9Pipeline, PipelineConfig)
│ ├── image_mosaic.py # ORB keypoint matching, RANSAC, image stitching (ImageAlignment)
│ ├── batch.py # Batch join_images
│ ├── quality_control.py
│ ├── utils.py
│ └── restitution/ # Image rectification
│ ├── types.py # StepResult, StrategyAttempt data classes
│ ├── strategy.py # RectificationStrategy + Collimation/Poly/Flat strategies
│ ├── vertical.py # VerticalDetector (collimation line detection)
│ └── plotters.py
└── dataquery/ # USGS/NAGAP data download
```

### Data Flow

**Aerial pipeline** (`hipp.aerial.core`):
1. `create_fiducial_templates()` — user picks fiducial locations on reference image
2. `iter_detect_fiducials()` — OpenCV template matching on input images
3. `filter_detected_fiducials()` — removes low-confidence matches
4. `compute_transformations()` — estimates affine/similarity transforms
5. `iter_image_restitution()` — crops, applies CLAHE, outputs standardized images

**KH-9 pipeline** (`hipp.kh9pc.pipeline.KH9Pipeline`):
1. Extract archive → list of TIF scan strips
2. `join_images()` — stitch strips via ORB keypoints + RANSAC affine alignment
3. Restitute (rectify):
- Detect vertical collimation edges (`VerticalDetector`)
- Detect horizontal edges with strategy fallback: `CollimationStrategy` → `PolyStrategy` → `FlatStrategy`
- Apply analytical inverse-map transform (bilinear interpolation between fitted polynomial curves via `build_inverse_map`)
4. Generate QC reports

Valid `PipelineConfig.steps` names (in order): `extract`, `join_images`, `quickview_mosaic`, `restitution`, `quickview_final`, `qc_report`.

**CLI** (`python -m hipp.kh9pc`):
```bash
python -m hipp.kh9pc --input scan.tgz --output /out/images/DZB1215.tif --qc-dir /out/qc
python -m hipp.kh9pc --input t1.tif t2.tif t3.tif --output /out/DZB1215.tif --qc-dir /out/qc
python -m hipp.kh9pc --input scan.tgz --output /out/DZB1215.tif --qc-dir /out/qc --config cfg.toml
```
`PipelineConfig.from_toml()` accepts keys: `overwrite`, `cleanup`, `steps`, and `output_height` (integer, default `22064`).

### Key Patterns

- **`PipelineStep`**: declarative step class with `inputs`/`outputs`/`overwrite` — enables skip-if-done logic
- **Strategy pattern** in `kh9pc/restitution/strategy.py`: multiple fallback strategies for edge detection; all inherit `RectificationStrategy` ABC which chains `_fit()` + `make_inverse_map()` + centering translation into `transform(output_height)`; strategies expose `detected_region() → ((col_off, row_top), (width, height))`
- **Pandas Series for fiducials**: coordinate data stored with named keys like `corner_top_left_x`, `midside_left_x`
- **`Intrinsics` class**: wraps focal length, pixel pitch, true fiducial coordinates in mm, principal point
- **3×3 homogeneous matrices** throughout for image transforms
- **Rasterio** for all geospatial raster I/O; **OpenCV** for image operations; `build_inverse_map` in `utils.py` for the analytical curve-interpolation warp
- **Intermediate files persisted as `.joblib`**: `vertical.joblib`, `horizontal.joblib`, `alignments.joblib` — individual steps can be re-run by loading these directly

### Notebooks

Practical usage examples live in `notebooks/`:
- `aerial_preprocessing.ipynb` — aerial fiducial workflow
- `kh9pc_preprocessing.ipynb` — full KH-9 pipeline
- `kh9pc_collimation_rectification.ipynb` — detailed rectification walkthrough
10 changes: 8 additions & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -33,9 +33,13 @@ dependencies = [
"rasterio",
"scikit-image",
"scikit-learn",
"requests"
"requests",
"click",
]

[project.scripts]
hipp-kh9pc = "hipp.kh9pc.cli:main"

[project.urls]
Documentation = "https://github.com/godinlu/hipp#readme"
Issues = "https://github.com/godinlu/hipp/issues"
Expand All @@ -53,13 +57,15 @@ dependencies = [
"pre-commit",
"usgsxplore",
"ipykernel",
"jupyter",
]
[tool.hatch.envs.dev.scripts]
check = [
"ruff check .",
"mypy . --strict --ignore-missing-imports --no-warn-unused-ignores --allow-untyped-calls"
"mypy src/ --strict --ignore-missing-imports --no-warn-unused-ignores --allow-untyped-calls"
]
kernel = "python -m ipykernel install --user --name hipp --display-name 'Python (hipp)'"
lab = "jupyter lab --no-browser --ip=127.0.0.1 --port=8333"


[tool.coverage.run]
Expand Down
Loading