Fix dashboard and resolve various conflicts#244
Draft
martjanz wants to merge 123 commits into
Draft
Conversation
Add unit tests for DuckDB Arrow-registration memory safety, verifying every register call is balanced by unregister and that save_legs uses zero Arrow registrations via parquet staging.
Replace stale `h3_o` column references with `h3` in geo equivalence and desire-line helpers. Compute polygon centroid from geometry instead of relying on dropped `polygon_lat/lon` mean columns.
Resolves 18 files with conflict markers. Strategy: keep HEAD's StorageContext/ DuckDB architecture throughout, integrate VV's functional improvements: - Rename assign_gps_destination → assign_time_distances; now computes distance_od, distance_route, distance_route_gps and saves to travel_times_legs/trips tables - preparo_dashboard: query etapas/viajes with LEFT JOIN to travel_times_* to pull pre-computed distance and speed columns - kpi.py: read_data_for_daily_kpi joins travel_times_legs; add inf cleanup and rounding to compute_kpi_by_line_day; add pd.set_option for pandas - misc.py: persist_indicators queries use distance_od from travel_times_trips - run_process.py: procesar_transacciones uses assign_time_distances, moves rearrange_trip_id_same_od before it, removes redundant add_distance calls - routes.py: improved assert error message from VV - kpi_lineas.py: column renames (distance_km→distance_route, distancia→ distance_od, travel_speed→kmh_od) and vehiculos_operativos logic - configs: alias_db_insumos key, new GPS column names preserved from VV
…run_basic_kpi output
…r first-run tables
… fo_mean) to VV names throughout dashboard and schema
…n resumen_x_linea
…configs without alias_db_insumos
…s, not autogenerated config
…s inline from corrida name
- Query only needed columns from gps and transacciones (largest tables) in kpi_lineas and preparo_dashboard - Slice viajes/etapas to used columns before merges and assigns - Skip defensive .copy() when no mutations follow - Add del + gc.collect() after large frames go out of scope - Deduplicate h3toparent calls with a per-unique-value map
Configure the implicit `duckdb.sql()` connection used as a SQL compute engine over pandas frames (e.g. `calculate_weighted_means`) so it respects the same memory_limit and thread cap as the adapter connections. Without this, the default connection ran with 80% RAM and all cores, making peak memory unbounded and machine-dependent. Also set a temp_directory so queries that exceed the limit can spill to disk instead of failing hard.
Iterate over `(dia, id_polygon)` pairs instead of polygons only, slicing one day at a time from the caller's frame. NONE-mode now defers the `.assign` to the per-day slice so the full etapas frame is never copied upfront.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
…in crear_mapa_folium
Comment on lines
+260
to
+266
| try: | ||
| stamped = ctx.data.query( | ||
| "SELECT MAX(batch_id) AS m FROM transacciones" | ||
| ).iloc[0, 0] | ||
| except _duckdb.CatalogException: | ||
| stamped = None | ||
| n = int(stamped) + 1 |
Comment on lines
1367
to
1368
| seleccion = '' | ||
| return seleccion |
|
|
||
| # leo el config autogenerado | ||
| configs_usuario = leer_configs_generales(autogenerado=False) | ||
| configs_base = revise_configs(configs_usuario) |
Comment on lines
+76
to
+83
| try: | ||
| with open(config_file, encoding="utf-8") as f: | ||
| raw = yaml.safe_load(f) or {} | ||
| for key in ("input_dir", "db_dir", "output_dir"): | ||
| if key in raw and raw[key]: | ||
| overrides[key] = raw[key] | ||
| except FileNotFoundError: | ||
| pass # config file optional — defaults apply |
Comment on lines
201
to
+211
| if "wkt" in tabla.columns and not tabla.empty: | ||
| tabla["geometry"] = tabla.wkt.apply(wkt.loads) | ||
| tabla = gpd.GeoDataFrame(tabla, crs=4326) | ||
| tabla = tabla.drop(["wkt"], axis=1) | ||
| tabla["geometry"] = tabla["wkt"].apply(wkt.loads) | ||
| tabla = tabla.drop(columns=["wkt"]) | ||
|
|
||
| elif "geometry" in tabla.columns and not tabla.empty: | ||
| sample_geom = tabla["geometry"].dropna().iloc[0] if tabla["geometry"].notna().any() else None | ||
|
|
||
| if isinstance(sample_geom, str): | ||
| tabla["geometry"] = tabla["geometry"].apply( | ||
| lambda x: wkt.loads(x) if pd.notna(x) else None | ||
| ) |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces several improvements and refactors to the dashboard codebase, most notably migrating the "Líneas de Deseo" map visualization from Folium to Pydeck for better performance and interactivity. It also enhances configuration handling for legacy compatibility, improves geometry handling in SQL table loading, and updates dependencies accordingly.
Dashboard Visualization Refactor:
pydeck), replacing all Folium-based logic and color mapping with Pydeck's WebGL layers for significantly improved rendering performance and interactivity. Added a utility for RGBA color conversion and adapted the data preparation and layer creation logic to Pydeck. (urbantrips/dashboard/pages/2_Líneas de Deseo.py)pydeck==0.9.1to the project dependencies. (pyproject.toml)Configuration and Compatibility Improvements:
alias_dbfields and ensuring optional fields set toNonedo not override dataclass defaults. (urbantrips/config/config.py)n_batchesto be zero (for auto-tuning), instead of requiring it to be strictly positive. (urbantrips/config/config.py)urbantrips/dashboard/__init__.py)Geometry and Data Handling Enhancements:
_load_table_sql: now robustly detects and parses both WKT and string geometries, raising errors for invalid types, and ensures resulting DataFrames are always valid GeoDataFrames. (urbantrips/dashboard/dash_utils.py)configurar_selector_dia, likely for debugging or future rework. (urbantrips/dashboard/dash_utils.py)Minor Fixes and Cleanups:
urbantrips/dashboard/pages/1_Datos Generales.py)urbantrips/dashboard/dashboard.py)