Change ftt for package coupling w mset#317
Conversation
…rdcoded tech_to_resource to a csv read for more flexibility on the number of tech used 3) add necessary __init__.py files
…option to disable sector_coupling (V2G, batteries in power) which are not in MSET by default
…ng mode. - get_gen_tech_indices() used to allocate the different technologies to the correct index, so far hard-coded with 2 possibilities: 22 or 12 technologies: further work could make that more flexible. - battery cost not updated when FTT:Transport not run - MSET coupling mode: LCOE reacts to changing fuel costs (FPIX) - update `pyproject.toml to include ftt_source*` - changed name of `converters_ftt.csv` to `converters.csv`
…) and add_vre_integration_costs()
There was a problem hiding this comment.
Pull request overview
This PR refactors the FTT Stand-alone codebase to support coupling with MSET by reducing hardcoding (notably technology/region schemas) and introducing CSV-driven mappings/configuration, while reorganizing the internal code into a dedicated ftt_source package.
Changes:
- Rename/relocate core implementation package to
ftt_sourceand update imports/packaging accordingly. - Make key FTT:Power mappings configurable via new CSV files (tech/resource mappings, wind/solar roles, CF multipliers, ERTI↔JTI mapping) and thread new settings through
settings.ini. - Add/extend Transport, Freight, and sector-coupling helpers to support cross-sector feedbacks and additional model features.
Reviewed changes
Copilot reviewed 38 out of 71 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| Utilities/titles/VariableListing.csv | Adds new variables (e.g., LAT, RLDCMAP) to support configurable RLDC logic. |
| Utilities/titles/T2TI_roles.csv | New CSV defining wind/solar/CSP role-to-technology mapping for RLDC. |
| Utilities/titles/T2TI_CF_multiplier.csv | New CSV for tech-specific capacity-factor multipliers (e.g., CSP). |
| Utilities/titles/ERTI_JTI_map.csv | New CSV defining how non-power fuel carriers map into resource demand. |
| Utilities/titles/converters.csv | New CSV mapping generation technologies (T2TI) to resources (ERTI). |
| settings.ini | Updates enabled modules and adds new run-time configuration parameters used by Power/coupling. |
| run_file.py | Updates entrypoint import to ftt_source.model_class.RunFTT. |
| pyproject.toml | Ensures ftt_source is included in package discovery. |
| future_technology_transformation/init.py | Updates public API imports/docs to reference ftt_source internals. |
| ftt_source/Transport/ftt_tr_survival.py | New Transport survival/age-tracking implementation. |
| ftt_source/Transport/ftt_tr_main.py | Updates module imports to the new ftt_source package layout. |
| ftt_source/Transport/ftt_tr_lcot.py | New levelised-cost-of-transport (LCOT) calculations and carbon tax application. |
| ftt_source/Transport/ftt_tr_emission_corrections.py | New CO2/biofuel correction utilities and emissions/fuel-use computation. |
| ftt_source/Transport/init.py | New package initializer for Transport submodule. |
| ftt_source/support/titles_functions.py | Updates utilities-path import to ftt_source.paths. |
| ftt_source/support/read_support.py | New helper to read support spreadsheets (Converters/Lagged_sales). |
| ftt_source/support/paths_append.py | New utility that appends the module directory to sys.path. |
| ftt_source/support/output_functions.py | New (incomplete) output scaffolding for saving results. |
| ftt_source/support/input_functions.py | Updates debug/paths imports to ftt_source. |
| ftt_source/support/get_vars_to_copy.py | New helper(s) to determine which variables need copying across timesteps/domains. |
| ftt_source/support/gamma_value_automation.py | Updates import paths/comments to the new ftt_source layout. |
| ftt_source/support/docstring_templates.py | New internal templates/examples for docstring structure. |
| ftt_source/support/divide.py | New bespoke divide helper (safe divide-by-zero behavior). |
| ftt_source/support/dimensions_functions.py | Updates utilities-path import to ftt_source.paths. |
| ftt_source/support/debug_messages.py | New debug message helpers for input loading. |
| ftt_source/support/cross_section.py | New cross-section slicer used for per-year model execution. |
| ftt_source/support/convert_masterfiles_to_csv.py | Updates imports to ftt_source for masterfile conversion. |
| ftt_source/support/check_market_shares.py | New validation helper to ensure market shares are well-formed. |
| ftt_source/support/init.py | New package initializer for support submodule. |
| ftt_source/sector_coupling/transport_batteries_to_power.py | New V2G/second-hand battery logic for Power↔Transport coupling. |
| ftt_source/sector_coupling/electricity_price.py | Updates divide import to ftt_source.support.divide. |
| ftt_source/sector_coupling/electricity_demand.py | New electricity-demand feedback from other sectors/models. |
| ftt_source/sector_coupling/battery_lbd.py | Updates coupling imports to ftt_source modules. |
| ftt_source/sector_coupling/init.py | New package initializer and documentation for sector coupling. |
| ftt_source/Power/ftt_p_surv.py | New (currently unused) power-sector survival function module. |
| ftt_source/Power/ftt_p_rldc.py | Refactors RLDC to use CSV-driven tech indices and input-driven region/latitude mappings. |
| ftt_source/Power/ftt_p_phase_out.py | New coal phase-out policy module. |
| ftt_source/Power/ftt_p_main.py | Threads new settings/mappings through Power solve loop and RLDC/dispatch/cost-curve calls. |
| ftt_source/Power/ftt_p_lcoe.py | Adds gamma_mode and optional FPIX fuel-cost scaling for MSET coupling. |
| ftt_source/Power/ftt_p_integration_costs.py | Generalizes VRE integration costs to use configurable wind/solar indices. |
| ftt_source/Power/ftt_p_fuel_price.py | Generalizes fuel-price mapping and electricity price computation with dynamic indices. |
| ftt_source/Power/ftt_p_early_scrapping_costs.py | New early scrappage cost calculation helper. |
| ftt_source/Power/ftt_p_dspch.py | Updates dispatch signature to accept a configurable nuclear index and dynamic electricity index. |
| ftt_source/Power/ftt_p_costc.py | Refactors cost curves to be CSV-driven (tech↔resource, ERTI↔JTI, CF multipliers, schema detection). |
| ftt_source/Power/init.py | New package initializer for Power submodule. |
| ftt_source/paths.py | Updates documentation strings to reference ftt_source classes and layout. |
| ftt_source/model_class.py | Updates module imports to ftt_source and precomputes Power settings once per run. |
| ftt_source/initialise_csv_files.py | Updates imports to ftt_source for CSV initialization workflow. |
| ftt_source/Industrial_Heat/ftt_ois_main.py | Updates divide import to ftt_source.support.divide. |
| ftt_source/Industrial_Heat/ftt_nmm_main.py | Updates divide import to ftt_source.support.divide. |
| ftt_source/Industrial_Heat/ftt_mtm_main.py | Updates divide import to ftt_source.support.divide. |
| ftt_source/Industrial_Heat/ftt_fbt_main.py | Updates divide import to ftt_source.support.divide. |
| ftt_source/Industrial_Heat/ftt_chi_main.py | Updates divide import to ftt_source.support.divide. |
| ftt_source/Industrial_Heat/init.py | New package initializer for Industrial Heat submodule. |
| ftt_source/Heat/ftt_h_main.py | Updates imports to ftt_source modules. |
| ftt_source/Heat/ftt_h_lcoh.py | Updates divide import to ftt_source.support.divide. |
| ftt_source/ftt_core/ftt_shares.py | New core shares-equation implementation (incl. njit path). |
| ftt_source/ftt_core/ftt_sales_or_investments.py | New shared sales/investment calculation utilities. |
| ftt_source/ftt_core/ftt_mandate.py | New centralized mandate/seeding implementation for multiple sectors. |
| ftt_source/ftt_core/ftt_exogenous_sales.py | New exogenous sales policy helper. |
| ftt_source/ftt_core/ftt_exogenous_capacity.py | New exogenous capacity and regulation-correction helper. |
| ftt_source/ftt_core/ftt_emissions_regulation.py | New centralized declining-emissions regulation implementation. |
| ftt_source/ftt_core/init.py | New package initializer for ftt_core. |
| ftt_source/Freight/ftt_fr_main.py | Updates imports to ftt_source modules. |
| ftt_source/Freight/ftt_fr_lcof.py | New freight levelised-cost implementation (incl. optional feebate logic). |
| ftt_source/init.py | Updates internal package public exports to ftt_source paths. |
| docs/conf.py | Updates Sphinx config to document ftt_source instead of SourceCode. |
| Backend_FTT.py | Updates RunFTT import to ftt_source.model_class. |
| .gitignore | Adds build/ to ignored outputs. |
Comments suppressed due to low confidence (2)
ftt_source/Power/ftt_p_rldc.py:97
- RLDC now depends on data['LAT'] and data['RLDCMAP'], but there are no LAT.csv / RLDCMAP.csv inputs in Inputs/S0 (or General). With the current loader, both variables default to all zeros, which silently changes behavior vs the previous hardcoded defaults (all regions map to RLDC region 0, and seasonality_index becomes 0 everywhere). Please ship default input CSVs for LAT and RLDCMAP (or load defaults from Utilities) so standalone runs reproduce prior behavior.
ftt_source/Power/ftt_p_costc.py:400 - update_capacity_factors_original() returns inside the outer region loop due to incorrect indentation, so it would only process r=0 if ever used.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
|
@ADanneaux: did you test that the changes in code give the same output as before? And what tests did you do? @copilot: could you rename the variable solar_t2ti to solar_tech_idx, solar_erti to solar_res_idx and oil_jti to oil_fuel_idx? Including for other technologies, resources and fuel types? That should make it more human readable. |
Done in commit |
|
@copilot resolve the merge conflicts in this pull request |
Resolved. The merge conflicts were fixed and merged in commit |
|
Overall, this PR contains a lot of changes that are perfectly fine. Fewer elements are hard-coded. I struggle with a few:
|
…ub.com/cpmodel/FTT_StandAlone into change_ftt_for_package_coupling_w_mset
|
@ADanneaux: could you commit the LAT and RLDC files. That seems to be the thing that makes the check fail. You may have to adjust elements in the .gitignore file if these two files are not showing up to be committed. |
| # %% tech-to-resource mapping and related lookups | ||
| # ----------------------------------------------------------------------------- | ||
|
|
||
| def get_tech_to_resource(titles): |
There was a problem hiding this comment.
All these calls are quite expensive. Reading in data takes ages. I don't know if this will fail the regression test as I cannot run the code without the missing LAT and RLDC file, but it's might be better to load this once.
| return [erti.index(tech_resource_map[tech]) for tech in t2ti] | ||
|
|
||
|
|
||
| def get_erti_jti_map(titles): |
There was a problem hiding this comment.
Rename as "get_resource_to_fuel_map". At some point, we'll get rid of erti as a variable name. in the meantime, let's keep it simple for readers.
| for _, row in df.iterrows()} | ||
|
|
||
|
|
||
| def _detect_power_schema(t2ti): |
There was a problem hiding this comment.
Does this mean we're hardcoding that there's only two options? I prefer our code to be flexible for new classifications too. For instance, Carlos is building a system to add rooftop solar.
| MEPD[:, 2, 0] = MEWD[:, 0, 0] + MEWD[:, 1, 0] | ||
| # Gas | ||
| MEPD[:, 3, 0] = MEWD[:, 6, 0] | ||
| n_t, n_e = gen_tech_indices['nuclear'] |
There was a problem hiding this comment.
n_t and n_e etc are cryptic variable names. Please spell out
| data_dt[var] = np.copy(data[var]) | ||
|
|
||
| data = get_marginal_fuel_prices_mewp(data, titles, Svar) | ||
| if not mset_coupling: |
There was a problem hiding this comment.
Interesting, so electricity costs are not coupled back to MSET?
|
@copilot : why is the output slightly different from main? This should just be a refactor when run without mset. |
| MEPD[:, 5, 0] = (MEWG[:, 4, 0] / BCET[:, 4, 13] | ||
| + MEWG[:, 5, 0] / BCET[:, 5, 13] ) * 3.6e-3 | ||
| RERY[:, 5, 0] = MEPD[:, 5, 0] # In PJ | ||
| bg_ts, bg_e = gen_tech_indices['biogas'] |
There was a problem hiding this comment.
Same: use longer variable names
| data = battery_costs(data, time_lag, year, t, titles, histend) | ||
| # Only when FTT:Transport is coupled (SCA dimension present AND sector_coupling enabled) | ||
| if 'SCA' in titles and sector_coupling: | ||
| data = battery_costs(data, time_lag, year, t, titles, histend) |
There was a problem hiding this comment.
What happens to battery costs if either of these are not enabled? Can use a comment.
| carbon_price_conv_factor = 1.3281 | ||
| gamma_mode = multiplicative | ||
| sector_coupling = True | ||
|
|
There was a problem hiding this comment.
I wouldn't want all this information added to the settings file: I can imagine people may want to change the gamma mode in the period of phasing out the old additive ones, and sector coupling should also be a setting, but the details of how we initialise the base year of cost curves and residual load duration curves, that's a detail which does not matter much.
Change FTT code to enable coupling with MSET using the FTT package.
This includes taking inputs from MSET as well as flexibility in reading setting files.
Core modifications include the need to make the typically 22 technology / 71 regions FTT more flexible.
MSET uses 12 technologies (open source OWID data) and a flexible number of regions.
The core issue is to remove hardcoding as much as possible and read from csv files for mapping. Default csv files are committed along with the code, but when coupling to another model, it is possible to define other input files.