Experimental/exact gradient workspace

examples/NMFS/afsc_walleye_pollock/walleye_pollock_analysis.md

@@ -0,0 +1,143 @@
+# Synthetic Walleye Pollock Functional Analysis
+
+Synthetic and public-data-safe. Not an official assessment.
+
+## Executive Summary
+
+- **Overall status:** `HEALTHY`.
+- **Confidence:** `HIGH`.
+- **Optimization quality:** `EXCELLENT`.
+- **Uncertainty structure:** `LOCAL`.
+- **Optimization:** converged = `yes`, gradient norm = `5.19827398754203e-06`.
+- **Curvature health:** positive definite = `yes`, condition number = `10.9840248198155`.
+- **Latent structure:** `20` random effects were estimated.
+- **Symbolic vs numerical structure:** structural density = `0.91`, but 95% of curvature is retained by `58` entries.
+- **Spectral complexity:** entropy effective rank = `16.3107626636197`, with 90% curvature requiring `14` eigen-directions.
+
+## Model Health Assessment
+
+| Check | Status | Evidence |
+|---|---:|---|
+| Optimization | `PASS` | converged = `yes` |
+| Gradient quality | `PASS` | gradient norm = `5.19827398754203e-06` |
+| Curvature | `PASS` | positive definite = `yes` |
+| Conditioning | `EXCELLENT` | condition number = `10.9840248198155` |
+| Overall status | `HEALTHY` | rule-based v1 diagnostic |
+| Confidence | `HIGH` | based on convergence, gradient, PD status, and conditioning |
+
+**Interpretation:** the rule-based health check is intentionally simple. It flags obvious numerical issues quickly, but it does not replace scientific review or model-specific diagnostics.
+
+## Model Complexity
+
+| Quantity | Value |
+|---|---:|
+| Fixed effects | `2` |
+| Random effects | `20` |
+| Total estimated quantities | `22` |
+| Structural nonzeros | `364` |
+| Structural density | `0.91` |
+| Entries for 95% curvature | `58` |
+| Effective bandwidth for 95% curvature | `1` |
+| 95% curvature compression | `6.27586x` |
+
+## Optimization
+
+- Quality: `EXCELLENT`
+- Objective value: `-14.3868675854755`
+- Gradient norm: `5.19827398754203e-06`
+- Converged: `yes`
+- Max gradient parameter: `log_r0`
+
+## Curvature
+
+- Positive definite: `yes`
+- Condition number: `10.9840248198155`
+- Minimum eigenvalue: `10.2183884027573`
+- Maximum eigenvalue: `112.239031834401`
+
+## Spectral Structure
+
+- Largest eigenvalue share: `0.095050325395682`
+- Entropy effective rank: `16.3107626636197`
+- Eigenvectors needed for 90% curvature: `14`
+- Eigenvectors needed for 95% curvature: `16`
+
+**Interpretation:** curvature is distributed across many latent-state directions rather than being dominated by one or two modes. That is a good sign for numerical stability.
+
+## Effective Structure
+
+- Structural density: `0.91`
+- Structural nonzeros: `364`
+- Entries for 95% curvature: `58`
+- Effective bandwidth for 95% curvature: `1`
+- 95% curvature compression: `6.27586x`
+
+**Interpretation:** symbolic density alone overstates practical complexity. The detailed Laplace report below shows that large amounts of curvature can be retained with far fewer entries or a narrow effective bandwidth.
+
+## Correlation Graph
+
+- Classification: `LOCAL`
+- Average degree: `1.5`
+- Maximum degree: `2`
+- Connected components: `5`
+- Largest component size: `14`
+- Graph diameter: `13`
+
+**Interpretation:** a LOCAL graph means the strongest uncertainty relationships are neighborhood-like rather than globally tangled.
+
+## Latent State Summary
+
+- Count: `20`
+- Mean: `0.0824010336165072`
+- Standard deviation: `0.0500130055529492`
+
+## Key Takeaway
+
+This report demonstrates why Quadra's functional analysis diagnostics are useful: a model can look dense from a symbolic Hessian pattern, while numerical curvature, graph structure, and effective bandwidth reveal a simpler local-dependence structure.
+
+## Full Laplace Structure Report
+
+```text
+Laplace Structure Report
+========================
+
+Random effects:              20
+Matrix size:                 20 x 20
+Total entries:               400
+Structural nonzeros:         364 / 400 (91%)
+Nonzero tolerance:           1e-08
+Max |H_ij|:                  61.5960537686533
+Positive definite:           yes
+Min eigenvalue:              10.2183884027573
+Max eigenvalue:              112.239031834401
+Condition number:            10.9840248198155
+
+Effective sparsity
+------------------
+curvature_retained,entries_required,entry_share,compression_vs_structural
+90%,54,0.135,6.74074074074074
+95%,58,0.145,6.27586206896552
+97%,100,0.25,3.64
+98%,133,0.3325,2.73684210526316
+99%,183,0.4575,1.98907103825137
+99.5%,224,0.56,1.625
+99.9%,293,0.7325,1.24232081911263
+100%,364,0.91,1
+
+Effective bandwidth
+-------------------
+curvature_retained,bandwidth,entry_count_if_banded,entry_share_if_banded
+90%,1,58,0.145
+95%,1,58,0.145
+97%,2,94,0.235
+98%,3,128,0.32
+99%,5,190,0.475
+99.5%,7,244,0.61
+99.9%,10,310,0.775
+100%,19,400,1
+
+Interpretation
+--------------
+This report measures numerical curvature concentration, not only symbolic sparsity.
+A dense structural Hessian can still be effectively sparse if most curvature is carried by relatively few entries or bands.
+```

examples/NMFS/afsc_walleye_pollock/walleye_pollock_fit_summary.csv

@@ -0,0 +1,7 @@
+field,value
+objective,-14.3868675854755
+grad_norm,5.19827398754203e-06
+iterations,17
+converged,yes
+message,converged to requested fixed-effect gradient tolerance
+random_effects,20

examples/NMFS/afsc_walleye_pollock/walleye_pollock_fixed_gradient_diagnostics.csv

@@ -0,0 +1,3 @@
+parameter,gradient,abs_gradient
+log_r0,5.11138687136689e-06,5.11138687136689e-06
+log_fbar,-9.46454806514431e-07,9.46454806514431e-07

examples/NMFS/afsc_walleye_pollock/walleye_pollock_fixed_hessian_diagnostics.csv

@@ -0,0 +1,13 @@
+field,value
+fixed_effects,2
+available,yes
+fd_step,0.0001
+profile_objective,-14.386867585475
+min_diagonal,0.271912625748882
+max_diagonal,134.948869856544
+eigen_success,yes
+min_eigenvalue,0.258341875865381
+max_eigenvalue,134.962440606427
+positive_definite,yes
+condition_number_abs,522.417978712652
+eigenvalues,0.258341875865381;134.962440606427

examples/NMFS/afsc_walleye_pollock/walleye_pollock_fixed_hessian_matrix.csv

@@ -0,0 +1,3 @@
+parameter,log_r0,log_fbar
+log_r0,134.948869856544,-1.35198057193975
+log_fbar,-1.35198057193975,0.271912625748882

examples/NMFS/afsc_walleye_pollock/walleye_pollock_fixed_parameter_estimates.csv

@@ -0,0 +1,3 @@
+parameter,estimate,exp_estimate
+log_r0,7.40479763896654,1643.85215079671
+log_fbar,-5.06778215474103,0.00629636903905652

examples/NMFS/afsc_walleye_pollock/walleye_pollock_functional_analysis_report.csv

@@ -0,0 +1,106 @@
+section,metric,target,value,extra
+optimization,objective_value,,-14.3868675854755,
+optimization,gradient_norm,,5.19827398754203e-06,
+optimization,max_gradient_parameter,,log_r0,
+optimization,max_gradient_value,,5.11138687136689e-06,
+optimization,max_abs_gradient,,5.11138687136689e-06,
+optimization,iterations,,17,
+optimization,converged,,yes,
+optimization,message,,converged to requested fixed-effect gradient tolerance,
+curvature,positive_definite,,yes,
+curvature,min_eigenvalue,,10.2183884027573,
+curvature,max_eigenvalue,,112.239031834401,
+curvature,condition_number_abs,,10.9840248198155,
+spectral_structure,available,,yes,
+spectral_structure,largest_eigen_share,,0.095050325395682,
+spectral_structure,effective_rank_entropy,,16.3107626636197,
+spectral_structure,eigen_count_for_50%,,6,
+spectral_structure,eigen_count_for_90%,,14,
+spectral_structure,eigen_count_for_95%,,16,
+spectral_structure,eigen_count_for_99%,,19,
+huu_structure,random_effects,,20,
+huu_structure,total_entries,,400,
+huu_structure,structural_nonzeros,,364,
+huu_structure,structural_density,,0.91,
+effective_sparsity,entries_required,90%,54,compression_vs_structural=6.74074074074074
+effective_sparsity,entries_required,95%,58,compression_vs_structural=6.27586206896552
+effective_sparsity,entries_required,97%,100,compression_vs_structural=3.64
+effective_sparsity,entries_required,98%,133,compression_vs_structural=2.73684210526316
+effective_sparsity,entries_required,99%,183,compression_vs_structural=1.98907103825137
+effective_sparsity,entries_required,99.5%,224,compression_vs_structural=1.625
+effective_sparsity,entries_required,99.9%,293,compression_vs_structural=1.24232081911263
+effective_sparsity,entries_required,100%,364,compression_vs_structural=1
+effective_bandwidth,bandwidth,90%,1,entry_count_if_banded=58
+effective_bandwidth,bandwidth,95%,1,entry_count_if_banded=58
+effective_bandwidth,bandwidth,97%,2,entry_count_if_banded=94
+effective_bandwidth,bandwidth,98%,3,entry_count_if_banded=128
+effective_bandwidth,bandwidth,99%,5,entry_count_if_banded=190
+effective_bandwidth,bandwidth,99.5%,7,entry_count_if_banded=244
+effective_bandwidth,bandwidth,99.9%,10,entry_count_if_banded=310
+effective_bandwidth,bandwidth,100%,19,entry_count_if_banded=400
+uncertainty,covariance_available,,yes,
+uncertainty,correlation_available,,yes,
+uncertainty,min_variance,,0.027865965883578,index=3
+uncertainty,max_variance,,0.0350480485905882,index=19
+uncertainty,max_abs_correlation,,0.598351221654447,pair=18;19
+uncertainty,count_abs_corr_gt_0_5,,15,
+uncertainty,count_abs_corr_gt_0_8,,0,
+uncertainty,count_abs_corr_gt_0_9,,0,
+parameter_influence,importance,log_rec_dev_17,44.5488428332677,index=16;sd=0.183360328347728;variance=0.0336210100117867;variance_share=0.0554843153726583;correlation_centrality=2.42549081701974;curvature_column_norm=70.9264530813492;importance_share=0.0609918078147027
+parameter_influence,importance,log_rec_dev_16,44.2615446289814,index=15;sd=0.180556501484316;variance=0.0326006502282559;variance_share=0.0538004288980058;correlation_centrality=2.46325967176437;curvature_column_norm=70.7829044051472;importance_share=0.0605984679264609
+parameter_influence,importance,log_rec_dev_18,43.2618345586047,index=17;sd=0.185464578784088;variance=0.0343971099835591;variance_share=0.0567651030580797;correlation_centrality=2.28245716879513;curvature_column_norm=71.0632207342914;importance_share=0.0592297651587808
+parameter_influence,importance,log_rec_dev_15,42.6803069211919,index=14;sd=0.177501684946827;variance=0.0315068481589625;variance_share=0.0519953415747319;correlation_centrality=2.40347164485272;curvature_column_norm=70.6484954006843;importance_share=0.0584335958389004
+parameter_influence,importance,log_rec_dev_14,40.7022095683089,index=13;sd=0.17471439426877;variance=0.0305251195647032;variance_share=0.050375207649096;correlation_centrality=2.30268400109302;curvature_column_norm=70.5378594805887;importance_share=0.0557253833262352
+parameter_influence,importance,log_rec_dev_19,39.5757904308082,index=18;sd=0.186696793808497;variance=0.0348556928183725;variance_share=0.0575218963436578;correlation_centrality=1.97827497081705;curvature_column_norm=71.1750770735403;importance_share=0.0541832032114705
+parameter_influence,importance,log_rec_dev_13,38.3629205908768,index=12;sd=0.172552949412713;variance=0.0297745203510261;variance_share=0.0491365035329637;correlation_centrality=2.15515537253594;curvature_column_norm=70.4641999522685;importance_share=0.0525226634650575
+parameter_influence,importance,log_rec_dev_12,36.954537302952,index=11;sd=0.171102186326852;variance=0.0292759581658288;variance_share=0.0483137328456264;correlation_centrality=2.06650457767272;curvature_column_norm=70.4317683832589;importance_share=0.0505944463136481
+parameter_influence,importance,log_rec_dev_10,36.9349887676045,index=9;sd=0.169560165113695;variance=0.0287506495933834;variance_share=0.0474468229434163;correlation_centrality=2.09114199974086;curvature_column_norm=70.4685255303029;importance_share=0.0505676824195682
+parameter_influence,importance,log_rec_dev_9,36.9235537937839,index=8;sd=0.169053168481611;variance=0.0285789737736718;variance_share=0.0471635085718553;correlation_centrality=2.09710336077563;curvature_column_norm=70.5219771472488;importance_share=0.0505520268002237
+parameter_influence,importance,log_rec_dev_11,36.7110052728577,index=10;sd=0.170182113018083;variance=0.0289619515912994;variance_share=0.047795531881284;correlation_centrality=2.06255374123372;curvature_column_norm=70.4366426325561;importance_share=0.050261026681811
+parameter_influence,importance,log_rec_dev_8,36.5579094790204,index=7;sd=0.168555129685161;variance=0.0284108317431813;variance_share=0.0468860259666669;correlation_centrality=2.0723774999475;curvature_column_norm=70.5935143908074;importance_share=0.0500514232748289
+parameter_influence,importance,log_rec_dev_7,35.8623310634345,index=6;sd=0.1680214128777;variance=0.0282311951854186;variance_share=0.0465895740926789;correlation_centrality=2.0195748276892;curvature_column_norm=70.6851341925052;importance_share=0.0490991070675987
+parameter_influence,importance,log_rec_dev_4,35.2815189643711,index=3;sd=0.166931021333897;variance=0.027865965883578;variance_share=0.0459868409279241;correlation_centrality=1.97256613521238;curvature_column_norm=71.1014893750246;importance_share=0.0483039173910652
+parameter_influence,importance,log_rec_dev_3,35.1733312007332,index=2;sd=0.167672150352856;variance=0.0281139500039506;variance_share=0.0463960858952032;correlation_centrality=1.9416127474987;curvature_column_norm=71.3127309936786;importance_share=0.0481557975552167
+parameter_influence,importance,log_rec_dev_6,35.0391789469444,index=5;sd=0.167477586740294;variance=0.0280487420603527;variance_share=0.0462884740743174;correlation_centrality=1.95520531014378;curvature_column_norm=70.7961381811328;importance_share=0.047972129743426
+parameter_influence,importance,log_rec_dev_5,34.5564405955384,index=4;sd=0.167030715015572;variance=0.0278992597586132;variance_share=0.0460417853767012;correlation_centrality=1.91673643224868;curvature_column_norm=70.9309009284492;importance_share=0.0473112128063926
+parameter_influence,importance,log_rec_dev_2,33.2137603001327,index=1;sd=0.170015622385115;variance=0.0289053118549979;variance_share=0.0477020600614166;correlation_centrality=1.72998327242739;curvature_column_norm=71.55981868112;importance_share=0.0454729496030033
+parameter_influence,importance,log_rec_dev_20,23.5210608030845,index=19;sd=0.187211240556191;variance=0.0350480485905882;variance_share=0.0578393385717653;correlation_centrality=1.42403028275588;curvature_column_norm=51.8306842643709;importance_share=0.0322026775301188
+parameter_influence,importance,log_rec_dev_1,20.2839251690113,index=0;sd=0.17488116356916;variance=0.0305834213713032;variance_share=0.0504714223619514;correlation_centrality=1.20266181741194;curvature_column_norm=52.6576098027071;importance_share=0.0277707160714908
+parameter_influence,correlation_pair,log_rec_dev_19__log_rec_dev_20,0.598351221654447,i=18;j=19;abs_correlation=0.598351221654447
+parameter_influence,correlation_pair,log_rec_dev_18__log_rec_dev_19,0.594581208456059,i=17;j=18;abs_correlation=0.594581208456059
+parameter_influence,correlation_pair,log_rec_dev_17__log_rec_dev_18,0.586512046951335,i=16;j=17;abs_correlation=0.586512046951335
+parameter_influence,correlation_pair,log_rec_dev_16__log_rec_dev_17,0.573942598352128,i=15;j=16;abs_correlation=0.573942598352128
+parameter_influence,correlation_pair,log_rec_dev_15__log_rec_dev_16,0.558210612518368,i=14;j=15;abs_correlation=0.558210612518368
+parameter_influence,correlation_pair,log_rec_dev_14__log_rec_dev_15,0.541906091575975,i=13;j=14;abs_correlation=0.541906091575975
+parameter_influence,correlation_pair,log_rec_dev_1__log_rec_dev_2,0.534149026853994,i=0;j=1;abs_correlation=0.534149026853994
+parameter_influence,correlation_pair,log_rec_dev_13__log_rec_dev_14,0.527798397745733,i=12;j=13;abs_correlation=0.527798397745733
+parameter_influence,correlation_pair,log_rec_dev_12__log_rec_dev_13,0.517641393962613,i=11;j=12;abs_correlation=0.517641393962613
+parameter_influence,correlation_pair,log_rec_dev_11__log_rec_dev_12,0.511255943807131,i=10;j=11;abs_correlation=0.511255943807131
+correlation_graph,abs_correlation_threshold,,0.5,
+correlation_graph,node_count,,20,
+correlation_graph,edge_count,,15,
+correlation_graph,average_degree,,1.5,
+correlation_graph,maximum_degree,,2,parameter=log_rec_dev_2
+correlation_graph,connected_components,,5,
+correlation_graph,largest_component_size,,14,
+correlation_graph,graph_diameter,,13,
+parameter_geometry,available,,yes,
+parameter_geometry,dominant_parameter,,log_r0,index=0;curvature_column_norm=221.843719224663
+parameter_geometry,curvature_column_norm,log_r0,221.843719224663,index=0;gradient=5.11138687136689e-06;abs_gradient=5.11138687136689e-06;curvature_diagonal=221.827904700423;curvature_share=0.988135761774757
+parameter_geometry,curvature_column_norm,log_fbar,2.66360841847077,index=1;gradient=-9.46454806514431e-07;abs_gradient=9.46454806514431e-07;curvature_diagonal=0.279918123434981;curvature_share=0.0118642382252428
+gradient_volatility,available,,no,
+gradient_volatility,perturbation_scale,,0,
+gradient_volatility,samples,,0,
+gradient_volatility,baseline_gradient_norm,,nan,
+gradient_volatility,mean_gradient_norm,,nan,
+gradient_volatility,sd_gradient_norm,,nan,
+gradient_volatility,max_gradient_norm,,nan,
+gradient_volatility,gradient_norm_cv,,nan,
+gradient_volatility,most_volatile_parameter,,,sd=nan
+gradient_volatility,most_sign_flips_parameter,,,sign_flips=0
+latent_states,count,,20,
+latent_states,mean,,0.0824010336165072,
+latent_states,sd,,0.0500130055529492,
+latent_states,min_value,,-0.00979634350742143,index=0
+latent_states,max_value,,0.145654153363153,index=10
+latent_states,l2_norm,,0.431073800305889,