Droplet reconstruction, patch_pixels auto-detect, XAS fixes, and XCS/MFX pipelines#95
Merged
lg345 merged 12 commits intoJul 1, 2026
Merged
Conversation
Pipeline output HDF5 files (per-shot spectra, binned arrays) can reach hundreds of MB per run and must not be tracked in git. Add examples/results/ to .gitignore so regenerate_*.py and Pipeline.run() output is never staged.
droplet_reconstruction reads sparse photon coordinates by absolute HDF5 shot index, not by relative position within a batch. Two new attributes (abs_start_index, abs_end_index) are now injected by _make_batch_run and _process_batch_parallel so each batch worker knows its exact slice in the source file. Both are listed in _INFRASTRUCTURE_ATTRS to prevent them from being collected as result keys. precomputed_attrs solves a batch consistency problem for axis steps (make_ccm_axis, time_binning): when those steps run independently on each batch they can produce different energy/time grids from incomplete data slices. The pipeline now runs a scalar-only pre-pass on the full run, collects the resulting axis arrays, and injects them via _inject_precomputed before each batch pipeline executes. Per-shot arrays (shape[0] == total_shots) are sliced to the batch range; static axis arrays (ccm_bins, ccm_energies, time_bins) are copied as-is. reconverge_results is tightened: geometry scalars ending in _angle are averaged (not summed) across batches; axis/bin key detection now matches on exact key names first, then suffix patterns.
…ecomputed pipeline.py: when a run is large enough to batch, the pipeline now executes a lightweight scalar-only pre-pass (run_pipeline without detector data) before entering the batched loop. Axis-derivation steps (make_ccm_axis, time_binning, ccm_binning) fire here on the full set of scalar diagnostics and produce a globally consistent ccm_bins/ccm_energies/time_bins. _collect_scalar_precomputed then harvests those attributes — excluding raw input key names so that pre-pipeline-filtered copies of ipm/xray/encoder never overwrite the freshly-loaded batch-slice arrays — and passes them to run_batched as precomputed_attrs. max_shots (data.max_shots in YAML) caps the number of shots loaded per run, which is essential when testing droplet_reconstruction pipelines where each reconstructed shot expands into a full-size 2D detector array. The cap is forwarded to spectroscopy_run(end_index=max_shots) and also stored on the run object so it is picked up correctly by the fallback constructor path. config_parser.py: parse data.max_shots from YAML (int or null).
Reads per-shot sparse photon coordinates stored by smalldata's
droplet2photon algorithm and scatters them back onto dense 2D images,
yielding a (N_shots, rows, cols) array that is drop-in compatible with
all downstream pipeline steps (filter_detector_adu, patch_pixels,
rotate_detector, reduce_detector_spatial, etc.).
Two HDF5 layouts are auto-detected:
Fixed-length — <det>/droplet_droplet2phot_sparse_{row,col,data}
shape (Nshots, nData), zero-padded (valid entries != 0)
Variable-length — <det>/var_droplet_droplet2phot_sparse/{row,col,data}
flat arrays + <det>/var_droplet_droplet2phot_sparse_len
ROI-direct scattering (_scatter_roi) filters photons to the crop window
before allocating the output, so the full 704×768 panel is never
materialised when a roi is specified. For a typical (60, 300) XES
window this reduces memory by ~30x vs full-panel scatter-then-crop.
Batch-awareness: the step reads abs_start_index / abs_end_index injected
by the batch manager so each worker slices the correct HDF5 rows. In
the non-batched path it falls back to start_index / end_index.
Motivation: stochastic RIXS (spooktroscopy) requires the cleanest
possible per-shot emission spectra. Standard ADU integration accumulates
read noise on every pixel on every shot; photon counting eliminates all
counts below the single-photon threshold and produces integer photon maps
that have much better signal-to-noise for the spook correlation matrix.
…d_rotation_angle patch_pixels gains auto_detect mode for ASIC panel-gap columns. A two-method approach handles the full range of defect severity: Method 1 (ratio): compares column profile against median-filtered baseline. Columns with ratio > threshold (spikes) or < 1/threshold within the signal band (dead gaps) are flagged. Catches extreme charge-sharing spikes that dwarf the spectral signal. Method 2 (z-score + width filter): computes a robust z-score using MAD-based local sigma. Bright and dark outliers above nsigma are separately clustered and kept only when cluster width <= max_gap_width. Separating bright/dark prevents merging a dark gap with its bright charge-sharing neighbors into a single wide cluster that would be rejected as a signal gradient. This catches subtle 50%-reduced dead columns that the ratio method misses. The polynomial fit is vectorised: instead of calling np.polyfit for every (shot, row) pair, the Vandermonde matrix is solved once via np.linalg.solve to get a projection vector, then all rows are evaluated with a single dot product. The fit region now also excludes the ±patch_range neighborhood of every known bad column, not just the target pixel, so elevated charge-sharing neighbors don't bias the polynomial anchor. find_rotation_angle wraps XSpectDetectorProcessor to auto-detect the tilt of a dispersed spectral signal via Canny edge detection → DBSCAN clustering → per-cluster PCA. Stores the angle on the run for use by rotate_detector. XSpectDetectorProcessor.find_optimal_rotation_angle: fix arctan2 argument order (was arctan2(dy,dx), should be arctan2(dx,dy) for row-dominant principal vector); weight cluster angles by cluster size so small edge-detection artifacts don't bias the result. filter_detector_adu: guard getattr with a None check so the step skips gracefully when the detector key is absent on the pre-pipeline pass.
…gies length reduce_detector_ccm previously only handled 1D (scalar) and 2D (spectrum) detectors. A new 3D branch accumulates full 2D images per energy bin — producing a (n_energy, rows, cols) stack — enabling RIXS-plane pipelines where the detector is a 2D spectrometer image not yet collapsed spatially. Off-by-one in bin count: both reduce_detector_ccm and reduce_detector_ccm_temporal were allocating n_bins = len(ccm_bins) where ccm_bins is the n+1-element edge array from make_ccm_axis. They now use len(ccm_energies) (= n edges - 1) so the output axis length matches the energy axis. NaN-guarded accumulation: shots that produce NaN values (e.g. empty bins, missing HDF5 rows) are now skipped rather than poisoning the running sum. For 2D/3D detectors np.where(np.isnan(...), 0.0, ...) replaces NaN pixels element-wise so a single bad shot doesn't zero an entire spectrum row. reduce_detector_ccm_temporal gets the same ccm_energies length fix and NaN guarding for both 1D and 2D detector dims. make_energy_axis and normalize_xes: minor formatting only.
…erance
Four new test cases cover the full auto_detect surface:
- test_auto_detect_spike: confirms a 100× bright column is flagged and
patched via the ratio method (Method 1)
- test_auto_detect_subtle_gap: confirms a 50%-reduced narrow 2-col dip
is caught by the z-score method (Method 2) which the ratio threshold misses
- test_auto_detect_ignores_wide_gradient: verifies that a smooth
intensity gradient across 200 columns produces zero flagged pixels
- test_auto_detect_merges_manual: checks that auto-detected and
manually specified pixel lists are both applied
test_patch_interpolate tolerance relaxed from exact equality to atol=1e-3
because the vectorized polynomial fit (Vandermonde + np.linalg.solve)
differs from np.polyfit in numerical precision by ~1e-10 — the test was
comparing against a specific polyfit rounding, not a scientific threshold.
…nalysis notebook
Three YAML pipeline configs for the polariton / FeNO pump-probe XAS
experiment at XCS (LCLS run 26, runs 187-216 and 339-347):
xcs101591326_ultrafast_xas.yaml — static DCCM energy scan, auto CCM
axis from setpoints, laser-on vs laser-off, 1D XAS with reduce_detector_ccm
xcs101591326_temporal_xas.yaml — time-delay scan at fixed DCCM energy,
enc/lasDelay binning (-10 to 25 ps, 35 bins), reduce_detector_temporal,
combine_runs reduction across 7 runs with uncertainty propagation
xcs101591326_2d_xas.yaml — simultaneous DCCM energy × time-delay scan,
21-bin time axis (-2 to 8 ps), 2D binning via reduce_detector_ccm_temporal,
produces transient absorption map Δμ(E,t)
Analysis notebook (xcs101591326_temporal_xas_analysis.ipynb) computes
mu = If/I0 per delay bin from the pipeline output and plots normalised
difference spectra for kinetic analysis.
…eline configs
mfx101609126_pershot_xes.yaml — stochastic RIXS input preparation:
filters to xray shots, ADU thresholds both detectors, patches ASIC
gap columns (manual on epix100_0; auto_detect on epix100_1 SEER),
rotates -2.0° (spec) / -1.6° (SEER), projects rows → 1D per-shot
spectra for spook correlation. Produces:
xrt_hproj (N×2048) and epix_spec_ROI_1 / epix_seer_ROI_1 (N×n_cols)
mfx101609126_seer_xrt.yaml — static comparison of both spectrometers
across runs 78-82: sums all xray shots to 2D image then collapses
rows → 1D spectrum for each detector independently.
mfx101609126_static_rixs.yaml — RIXS plane for run 97 DCCM scan:
reduce_detector_ccm accumulates full 2D epix images at each incident
energy → (n_energy, 60, 300) stack, then reduce_detector_spatial
collapses the cross-dispersion axis → (n_energy, 300) RIXS plane.
mfx101609126_droplet_pershot_xes.yaml — per-shot XES using photon-
counting via droplet_reconstruction (epix100_0 fixed-length sparse
format), same downstream steps as pershot_xes.
mfx101609126_droplet_recon_test.yaml — 100-shot smoke test for
droplet_reconstruction; max_shots: 100 avoids materialising the full
~36k-shot array during development.
rotation_diagnostic — finds optimal detector tilt angle by running the pipeline on a subset of shots and inspecting the 2D sum image with overlaid rotation angle estimates from XSpectDetectorProcessor. Fixed angles (-2.0° / -1.6°) were determined here and hardcoded into YAMLs. pixel_patch_diagnostic — compares raw vs patched column profiles for epix100_0 and epix100_1, visualises the ASIC gap positions and the effect of auto_detect patching on the baseline ratio and z-score. seer_shot_browser — per-shot SEER image browser: scan forward/backward through individual shots to inspect photon distributions and confirm the ASIC gap correction is working correctly. seer_vs_xrt — quantifies SEER as an alternative incident-energy monitor for stochastic RIXS. Computes correlation between SEER and XRT per- shot total-intensity and center-of-mass energy. Conclusion: SEER tracks intensity well (r=0.90) but has low COM energy correlation (0.41) because it encodes sample transmission/emission, not purely the incident beam. Gap patching dramatically improves the spook AtA condition number. spook_rixs — stochastic RIXS extraction using the spook library: builds A (XRT hproj) and B (epix_spec per-shot spectra) matrices and solves for the RIXS cross-section X via regularised least squares. droplet_spook_rixs — same as spook_rixs but using photon-counted spectra from droplet_reconstruction instead of raw ADU integration. droplet_spook_transmission — stochastic RIXS using SEER (transmission geometry) as the B matrix; photon-counted for improved S/N. rixs_plane — static RIXS plane visualisation for the DCCM scan run.
regenerate_run0079.py — runs mfx101609126_pershot_xes.yaml (standard ADU integration) and saves per-shot spectra to examples/results/; used as the baseline to compare against droplet-reconstructed spectra. regenerate_droplet_pershot.py — runs mfx101609126_droplet_pershot_xes.yaml for runs 78-82 sequentially, saving one HDF5 per run to results/. Measures reconstruction throughput (shots/sec) and reports memory usage. droplet_reconstruct.py — standalone utility script that reconstructs a single run's sparse photon data without the full pipeline framework; used for rapid iteration and debugging of the reconstruction step. droplet_reconstruction.ipynb — step-by-step walkthrough of the sparse format: reads raw HDF5, shows fixed-length vs variable-length layouts, runs _scatter and _scatter_roi helpers manually, compares output images against the pre-processed ROI_area dataset for bit-identity verification. droplet_recon_pipeline_eval.ipynb — end-to-end comparison of ADU integration vs photon-counting via the pipeline API. Runs both YAMLs on run 79, overlays 1D emission spectra, and evaluates peak S/N ratios to quantify the photon-counting improvement for spook inputs.
…c assets mfx101080524_static_xes.yaml: remove hitfinding step. The hit-finder was removing shots that had low per-frame median signal, which is expected for single-photon-level XES data. Removing it recovers the correct shot count and avoids discarding weak-signal runs. seer_full_detector_run78.png: full-panel SEER detector image from run 78 showing the spatial distribution of signal across all ASIC tiles; used in rotation_diagnostic.ipynb and seer_shot_browser.ipynb to orient the cross-dispersion ROI selection. c.ipynb: scratch notebook for static RIXS pipeline diagnostics (run 97); shows RIXS plane shape, energy range, and 2D sum-image ROI inspection. xcs101591326.ipynb, Untitled.ipynb: updated experiment analysis notebooks.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes out a major round of pipeline development driven by stochastic RIXS (spooktroscopy) analysis on
mfx101609126and pump-probe XAS onxcs101591326.Infrastructure
batch_manager: injectabs_start_index/abs_end_indexon every batch run (required bydroplet_reconstructionto slice the correct HDF5 rows); addprecomputed_attrsmechanism so CCM/time axis steps run once on the full run and their results are broadcast into every batch worker, preventing inconsistent energy/time grids.pipeline: scalar pre-pipeline pass before entering the batched loop;_collect_scalar_precomputedexcludes raw input keys to avoid injecting filtered-down arrays into batch workers.config_parser:max_shotsYAML field caps shots per run — essential for testing droplet pipelines where each shot materialises a full 2D detector array.New step:
droplet_reconstructionReads sparse photon coordinates from smalldata HDF5 (fixed-length zero-padded or variable-length
_lenpointer format) and scatters them back onto dense 2D images. ROI-direct scatter avoids allocating the full 704×768 panel. Motivation: photon counting eliminates read noise below the single-photon threshold, improving thespookAtAcondition number for stochastic RIXS.patch_pixelsoverhaulAuto-detection of ASIC panel-gap columns via a dual method:
Polynomial fit vectorised (Vandermonde solve → single dot product replaces per-row
np.polyfitloop). Also addsfind_rotation_anglestep wrappingXSpectDetectorProcessorPCA tilt detection; fixesarctan2argument order and cluster-size weighting inXSpectDetectorProcessor.XAS/XES analysis fixes
reduce_detector_ccm: 3D detector support (produces(n_energy, rows, cols)RIXS-plane stack); NaN-guarded accumulation; off-by-one fix usinglen(ccm_energies)notlen(ccm_bins).reduce_detector_ccm_temporal.New YAML configs
xcs101591326_ultrafast_xas.yamlxcs101591326_temporal_xas.yamlxcs101591326_2d_xas.yamlmfx101609126_pershot_xes.yamlmfx101609126_seer_xrt.yamlmfx101609126_static_rixs.yamlmfx101609126_droplet_pershot_xes.yamlmfx101609126_droplet_recon_test.yamlNotebooks & scripts
Rotation diagnostic, pixel-patch diagnostic, shot browser, SEER-vs-XRT correlation analysis, spook RIXS (ADU + photon-counted + transmission), static RIXS plane, droplet reconstruction evaluation, and regeneration scripts.
Tests
4 new
patch_pixelsauto-detect test cases covering spike detection, subtle gap detection, gradient rejection, and manual+auto merge.