Ftt power integration#1
Draft
ADanneaux wants to merge 14 commits into
Draft
Conversation
Replace all year literals (2013, 2015, 2018) and the hard-coded USD/EUR conversion rate (1.0018) with module-level constants parsed once at import time from settings.ini via configparser. Fallback values keep existing behaviour when the keys are absent. The PRSC/EX variable names (PRSC13, EX18, ...) are derived from the year values rather than duplicated as strings, so changing a base year in the .ini file is the only edit required. The BCET copy-range upper bound (previously 17) and the MERC starting values (previously four hard-coded floats) are also externalised: MERC is now seeded from time-lagged data instead of literals.
titles_functions.py:
- Replace openpyxl workbook loading with pd.read_csv on
classification_titles.csv, removing the openpyxl dependency.
- Raise FileNotFoundError instead of silently printing when the file is
missing.
- Add a sentinel entry titles_dict[''] = ('',) so that VariableListing.csv
rows that use an empty string as a placeholder dimension key pass the
dimension-filter check in input_functions.py.
- Override T2TI_ERTI in converters with ftt_t2ti_erti.csv when present,
aligning numbered technology names with classification_titles.csv.
input_functions.py:
- Filter the variable dimension table to entries whose every dimension key
is known in titles, preventing KeyError for future or incomplete-feature
variables listed in VariableListing.csv.
… state
Switch imports from the MINDSET_FTT_Power package path to the ftt_source
standalone package (ftt_source.paths.set_paths, ftt_source.Power.ftt_p_main
.solve). Use pathlib for robust settings.ini path resolution instead of a
CWD-relative string.
After load_data(), extend BCET and C2TI to add '22 Gamma' and '23 Value
factor' columns expected by ftt_p_lcoe in the standalone solver. The CSV
data ends at column 20 ('21 Gamma ($/MWh)'); column 21 is seeded as a copy
and column 22 defaults to 1.0.
Introduce three coupling-state attributes written by ftt_power.py before
each solve_year() call and consumed inside it:
_mset_reppx – per-(region, tech) carbon price in EUR2015/tCO2
_mset_fpi – year-on-year fuel price index
_fpix_carry – cumulative FPIX across years (persisted between calls)
In solve_year(), inject these into variables before calling ftt_p_solve:
CO2taxP ← _mset_reppx converted to USD2013/tCO2 via exchange-rate vars
BCET[:,22] ← MGAM (gamma values for current year)
FPIX ← _fpix_carry * (1 + _mset_fpi)
Remove the within-year iteration loop; FTT is now called once per year.
Drop the iter_lags and conv arguments from the ftt_p_solve call; add
settings_path so the standalone solver reads the same .ini file.
…growth elec_dem is a Series indexed by RTI_short (short region codes). MEWD was being assigned using RTI (full names), causing a silent all-NaN assignment. Fix to use RTI_short and populate MEWDX with the same values. ftt_energy_dem_growth is a per-region growth factor merged onto power_ener_sec via REG_imp, which also uses short codes. Change the pd.Series index from RTI to RTI_short so the map() join finds its keys. Add _rti_short = list(ftt_model.titles['RTI_short']) at function top to avoid repeated attribute lookups.
The investment calculation multiplied MWIY (shape: n_reg × n_tech) by ftt_inv_converter (shape: n_mrio_sectors × n_tech) via a direct broadcast. This assumed MWIY columns and ftt_inv_converter columns shared the same technology order, which is not guaranteed when FTT_Standalone or the CSV files are updated. Replace with an explicit pd.DataFrame.reindex so the column alignment is enforced by name rather than by position. Apply the same fix in both the annual solve loop (ftt_power.py) and the historical setup loop (model_class.py).
Previously, the year-on-year fuel price index was written directly into ftt_model.input['S0']['FPI'][:, tech_idx, 0, y], requiring FPI to exist as a named variable in FTT_Standalone's VariableListing.csv. Instead, accumulate the (n_reg × n_tech × 1) FPI array and store it on ftt_model._mset_fpi. model_class.solve_year() reads this attribute, multiplies it into the cumulative _fpix_carry, and injects the result as FPIX into variables before each ftt_p_solve call. The attribute is consumed (reset to None) after injection so a missing write from ftt_power.py is detectable. Add n_reg and n_tech helpers at function top. Tighten FPI block comments.
…ppx_arr The previous implementation looped over every region, called Scenario.carbon_tax_rate_loop(reg) inside the loop (one DataFrame filter per region), and wrote each result individually into ftt_model.input['S0']['REPPX']. Replace with a single pass over Scenario.tax_rate (accessed once), building a (n_reg × n_tech × 1) array reppx_arr and storing it on ftt_model._mset_reppx. model_class.solve_year() converts the EUR2015/tCO2 value to USD2013/tCO2 using the current-year FTT exchange-rate variables (PRSCX, EXX, PRSC18, EX18), which are only available there. When no carbon price is active, _mset_reppx stays None and CO2taxP remains zero — the correct behaviour for no-tax years.
The energy_flows update loop was calling read_var_df and write_var_df on every iteration over fuel sectors. For N fuel sectors this means N full DataFrame reads and N full writes to storage each year. Move the read outside the loop, accumulate all sector updates in-place with _ef.update(), and perform a single write after the loop completes. No change to correctness; pure I/O reduction.
…ar_ftt Move the top-level imports of SourceCode.ftt_power.solve_year_ftt and MINDSET_FTT_Power.SourceCode.model_class.ModelRun inside the `if self.ftt_run:` block in __init__, so they are only executed when FTT is actually enabled. Store the function as self._solve_year_ftt so solve_year() does not need to close over a module-level name. Additional housekeeping: - Fix missing trailing comma in the DYNAMIC dict literal (syntax hazard when adding entries). - Remove three stale commented-out debug print/plot lines that were left after earlier debugging sessions.
…scope Electricity investment de-duplication: Before: elec_idx = 92 * np.array(range(0, n_reg)) This multiplied the sector offset (92) by the region index instead of stepping by the number of sectors per region (120). The resulting indices pointed to the wrong rows in the flattened investment vector. After: np.arange(92, n_reg * n_sectors, n_sectors) Labour supply constraint: dempl_labour_supply_constraint was declared inside the while loop and therefore reset to zeros on every iteration, discarding any value set in the previous pass. Move the declaration before the loop; only re-zero it when iter_run > 1 (the first pass should keep whatever value the preceding code has set). Also fix an f-string where inner double-quotes conflicted with the outer f-string delimiter, causing a SyntaxWarning in Python 3.12+.
…inal After merging emission_cost_ with consumption_shares, the blanket .fillna(0) incorrectly zeroed out price_index for non-matched rows. price_index should default to 1.0 (no price change), not 0.0 (which would zero all nominal values downstream). Apply separate fillna calls: emission_cost → 0 price_index → 1.0 Protect the subsequent division by VIGA_nominal against rows where government expenditure is zero: replace zeros with NaN before dividing, then fillna(0) on the result so those rows contribute nothing to the emission cost share rather than producing inf. Add `import numpy as np` which was missing from the module header.
Femkemilene
reviewed
Jun 8, 2026
| ftt_model.output[ftt_model.scenarios]['MWIY'][:, :, 0, y][:, np.newaxis, :]).sum(axis = 2) | ||
| # Reorder MWIY columns to match ftt_inv_converter, then map to MRIO sectors. | ||
| _mwiy_raw = ftt_model.output[ftt_model.scenarios]['MWIY'][:, :, 0, y] | ||
| _mwiy12 = pd.DataFrame(_mwiy_raw, columns=list(ftt_model.titles['T2TI'])) \ |
There was a problem hiding this comment.
We have some legacy four-letter names such as MWIY. There is a long-term plan to replace these with human-readable names. In the meantime, they should not be added in new places.
There was a problem hiding this comment.
Pull request overview
This PR integrates MINDSET’s power coupling with the external FTT_Standalone “FTT:Power” implementation, shifting key inputs (fuel prices, carbon tax, settings) to a more explicit MSET→FTT handoff while addressing several stability/performance issues in the coupling and iteration logic.
Changes:
- Switch MINDSET’s power module call path to use FTT_Standalone’s
ftt_p_solveand introduce stateful coupling hooks for fuel price and carbon tax inputs. - Make multiple calibration constants (base years, FX rate, BCET copy range) settings-driven rather than hardcoded, and migrate title loading from XLSX to CSV.
- Fix/adjust several MRIO/CBAM/government indexing and edge-case behaviors (e.g., price_index NA handling, electricity investment index construction, batched energy_flows updates).
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| SourceCode/model_class.py | Lazily imports/initializes FTT:Power and routes yearly solves through an instance method; adjusts investment mapping to match converter column order. |
| SourceCode/initiate_modules.py | Fixes electricity investment indexing, adjusts CBAM/final-demand column naming and types, and tweaks within-year loop variables and logging. |
| SourceCode/government.py | Improves robustness of government demand price index/emission cost handling (NA fills and divide-by-zero protection). |
| SourceCode/ftt_power.py | Updates coupling for RTI_short usage, adds fuel-price and carbon-tax injection via model state, batches energy_flows writes, and reindexes investment mapping. |
| MINDSET_FTT_Power/SourceCode/support/titles_functions.py | Replaces XLSX title loading with CSV parsing; adds placeholder dimension key support and tweaks converter handling. |
| MINDSET_FTT_Power/SourceCode/support/input_functions.py | Filters out variables whose dimension keys aren’t present in titles to avoid crashes on incomplete/future variables. |
| MINDSET_FTT_Power/SourceCode/Power/ftt_p_main.py | Reads calibration base years and FX rate from settings.ini; removes hardcoded initialization values in favor of lagged data. |
| MINDSET_FTT_Power/SourceCode/model_class.py | Uses FTT_Standalone entrypoints, wires settings/paths, extends BCET/C2TI for 23-column expectation, and implements carbon/fuel price coupling state. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
54
to
+55
| tech_idx = ftt_model.titles['T2TI'].index(tech) | ||
| ftt_model.input['S0']['FPI'][:, tech_idx, 0, y] = weighted_dp[list(ftt_model.titles['RTI'])].values | ||
| # Overwrite carbon prices (USD / tCO2) | ||
| for reg in ftt_model.titles['RTI']: | ||
| c_price = Scenario.carbon_tax_rate_loop(reg) | ||
| c_price = c_price.loc[c_price.PROD_COMM == 93] | ||
| for fuel, sectors in ftt_model.ftt_fuel_converter.groupby('ERTI'): | ||
| # Get relevant sectoral data (prices and output for weighting) | ||
| sec_c_price = c_price.loc[c_price.TRAD_COMM.isin(sectors.TRAD_COMM)] | ||
| if (len(sec_c_price) > 0) & (fuel in ftt_model.conv['T2TI_ERTI'].ERTI.values): | ||
| # Since carbon price is the same for all REG_exp, simply take avg. | ||
| avg_sec_c_price = sec_c_price.groupby('REG_imp')['ctax'].mean() | ||
| # Keep if weighted avreage will be needed | ||
| # weighted_dp = (sec_c_price.groupby("REG_imp", group_keys=False, dropna=False) | ||
| # .apply( | ||
| # lambda g: (g["dp"] * g["z_bp"]).sum() / g["z_bp"].sum() | ||
| # if g["z_bp"].sum() != 0 | ||
| # else 0, | ||
| # include_groups=False)) | ||
|
|
||
| # Map fuel to tech | ||
| tech = ftt_model.conv['T2TI_ERTI'].index[ftt_model.conv['T2TI_ERTI'].ERTI == fuel].values[0] | ||
| tech_idx = ftt_model.titles['T2TI'].index(tech) | ||
| reg_idx = ftt_model.titles['RTI'].index(reg) | ||
| ftt_model.input['S0']['REPPX'][reg_idx, tech_idx, 0, y] = avg_sec_c_price.values[0] | ||
|
|
||
| fpi[:, tech_idx, 0] = weighted_dp[_rti_short].values |
Comment on lines
+63
to
+67
| _conv_csv = pd.read_csv(Path('MINDSET_FTT_Power/Utilities/titles/converters.csv')) | ||
| _t2ti_list = list(ftt_model.titles['T2TI']) | ||
| _erti_to_t2ti_idxs = {} | ||
| for _, row in _conv_csv.iterrows(): | ||
| _erti_to_t2ti_idxs.setdefault(row['ERTI'], []).append(_t2ti_list.index(row['T2TI'])) |
Comment on lines
+245
to
+249
| for d in (variables, time_lags): | ||
| if d['BCET'].shape[2] < n_c2ti: | ||
| ext = np.zeros((*d['BCET'].shape[:2], n_c2ti)) | ||
| ext[:, :, :d['BCET'].shape[2]] = d['BCET'] | ||
| d['BCET'] = ext |
Comment on lines
672
to
+675
| dev_profit_rate_L1[abs(dev_profit_rate_L1)==np.inf] = 0.0 | ||
| dev_profit_rate_L1[abs(dev_profit_rate_L1)>10] = 0.0 | ||
| if len(dev_profit_rate_L1[abs(dev_profit_rate_L1)>10]) > 0: | ||
| print(f"profit rate issues (>10) sectors: {", ".join(map(str, np.where(dyn_qbase < 0)[0]))}") | ||
| print(f"profit rate issues (>10) sectors: {', '.join(map(str, np.where(dyn_qbase < 0)[0]))}") |
Femkemilene
reviewed
Jun 8, 2026
Femkemilene
left a comment
Femkemilene
left a comment
There was a problem hiding this comment.
Initial review; I'll leave the more substantative comments to Aron
| reg_idx = ftt_model.titles['RTI'].index(reg) | ||
| ftt_model.input['S0']['REPPX'][reg_idx, tech_idx, 0, y] = avg_sec_c_price.values[0] | ||
|
|
||
| fpi[:, tech_idx, 0] = weighted_dp[_rti_short].values |
There was a problem hiding this comment.
What is fpi? Ensure clear variable names. If not possible, include comment explaining abbreviation
| cost_curves_impact_old = cost_curves_impact_old.rename(columns={'input_cost_change':'input_cost_change_old'}) | ||
|
|
||
| dempl_labour_supply_constraint = np.zeros_like(dempl_total) | ||
| if iter_run > 1: |
There was a problem hiding this comment.
You are defining dempl_labour_supply constraint above already; this seems duplicative. Do you know what this means? If so, could you please rename into a more human-readable term?
| if self.ftt_run: | ||
|
|
||
| from SourceCode.ftt_power import solve_year_ftt | ||
| from MINDSET_FTT_Power.SourceCode.model_class import ModelRun as ftt |
There was a problem hiding this comment.
Normally, you wouldn't put imports under an if statement. Put them on top first to keep the code clean
There was a problem hiding this comment.
I guess they are here because the FTT package doesn't need to be imported if MSET isn't being run with FTT?
Author
There was a problem hiding this comment.
This was indeed the rationale. Without this, you would need to install FTT even for MSET runs that do not require it.
| self.ftt_model.investment[year] = (np.array(self.ftt_model.ftt_inv_converter[list(self.ftt_model.titles['T2TI'])])[np.newaxis, :, :] * | ||
| self.ftt_model.output[self.ftt_model.scenarios]['MWIY'][:, :, 0, y][:, np.newaxis, :]).sum(axis = 2) | ||
| # Reorder MWIY columns to match ftt_inv_converter, then map to MRIO sectors. | ||
| _mwiy_raw = self.ftt_model.output[self.ftt_model.scenarios]['MWIY'][:, :, 0, y] |
There was a problem hiding this comment.
Same comment as above. We're slowly phasing out the cryptic variable names, so they should not be repeated.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
MINDSET_FTT_Power/model_class.pynow callsftt_source.Power.ftt_p_solveinstead of the internalftt_p.solve, using a new power_settings object. BCET/C2TI is extended at init to bridge MINDSET's 21-column layout with FTT_Standalone's 23-column expectation._mset_reppx,converted from EUR2015/tCO₂ to USD2013/tCO₂ insidemodel_class.solve_year()using current-year exchange rates. Fuel price index accumulates year-over-year via_fpix_carry.ftt_p_main.pyare now read fromsettings.ini, enabling scenario-specific calibration base years.classification_titles.xlsxreplaced byclassification_titles.csv; openpyxl dependency removed from the titles loader. Dimension filtering ininput_functions.pyprevents crashes from future/incomplete variables inVariableListing.csv.Bug fixes
government.py: price_index now fills NA with 1.0 (not 0); zero VIGA_nominal no longer causes division errors.initiate_modules.py: fixed electricity investment index (92 * range(n_reg)→np.arange(92, n_reg*n_sectors, n_sectors)); FD emission/CBAM cost variables renamed PROD_COMM → FD; CBAM intermediates cast to int.ftt_power.py:RTI lookup uses RTI_short (short names) everywhere; energy_flows write batched to a single read + write per year; MWIY investment mapping reindexed to handle T2TI column order.Caution
At the moment this seems to be essentially a one-way coupling: MSET →FTT. Two preexisting features have not been changed:
Note
pip install . "pandas=2.3.3"Warning
MSET does not work with pandas >3.0