Offline-safe Calibration and IMF SDMX 3.0 refresh

.github/workflows/build_and_test.yml

@@ -61,3 +61,32 @@ jobs:
           token: ${{ secrets.CODECOV_TOKEN }}
           fail_ci_if_error: true
           verbose: true
+  pip-import-smoke:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.12", "3.13"]
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v5
+        with:
+          persist-credentials: false
+
+      - name: Setup Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - name: Install package and pytest
+        run: |
+          python -m pip install --upgrade pip
+          python -m pip install . pytest
+
+      - name: Run import smoke tests outside checkout
+        shell: bash
+        run: |
+          tmpdir="$(mktemp -d)"
+          cp "${{ github.workspace }}/tests/test_import_smoke.py" "$tmpdir/test_import_smoke.py"
+          cd "$tmpdir"
+          python -m pytest "$tmpdir/test_import_smoke.py"

AGENTS.md

@@ -0,0 +1,30 @@
+## Workflow
+
+- Read-only inspection does not require prior approval. This includes actions like checking status, listing branches or worktrees, inspecting tracking refs, reviewing merged status, reading files, and preparing a proposed cleanup list.
+- Present a plan and wait for explicit approval before making changes to repo state. This includes file edits, branch deletion, pruning refs or worktrees, rebases, merges, commits, pushes, and similar mutating actions.
+
+## Project: OG-ETH
+
+OG-ETH is an Ethiopia country calibration of the OG-Core overlapping-generations model of demographics and fiscal policy.
+
+## Environment
+
+- Conda environment: `ogeth-dev`
+
+## Python formatting
+
+- Run `black` on all touched `.py` files before staging and pushing.
+- Do not run `black` on non-Python files (e.g. README.md will fail to parse).
+- Re-run tests after formatting to confirm nothing broke.
+- Format command: `conda run -n ogeth-dev python -m black <files>`
+
+## Testing
+
+- Full suite: `conda run -n ogeth-dev python -m pytest tests/ -q`
+- Targeted: `conda run -n ogeth-dev python -m pytest tests/test_calibrate.py tests/test_input_output.py tests/test_macro_params.py -q`
+
+## Repo conventions
+
+- The packaged JSON default parameters are the standard baseline input for offline/default runs.
+- Calibration-related changes can affect macro parameters, demographics, earnings distribution, and industry I/O behavior.
+- Changes in calibration or data-source behavior should be validated with targeted tests and, where feasible, the relevant example flows.

CLAUDE.md

@@ -0,0 +1 @@
+@AGENTS.md

README.md

@@ -50,7 +50,7 @@ Once the package is installed, one can adjust parameters in the OG-Core `Specifi
 from ogcore.parameters import Specifications
 from ogeth.calibrate import Calibration
 p = Specifications()
-c = Calibration(p)
+c = Calibration(p, update_from_api=True)
 updated_params = c.get_dict()
 p.update_specifications({'initial_debt_ratio': updated_params['initial_debt_ratio']})
 ```

docs/book/content/calibration/macro.md

@@ -39,13 +39,17 @@ where $\tau_d$ is the scale parameter and $\mu_d$ is the level shift parameter.
 
 ### Aggregate transfers
 
-Aggregate (non-Social Security) transfers to households are set as a share of GDP with the parameter $\alpha_T$. We exclude Social Security from transfers since it is modeled specifically. With this definition, the share of transfers to GDP in 2015 is 0.034 according to [IMF data](https://data.imf.org/en/Data-Explorer?datasetUrn=IMF.STA:GFS_SOO(12.0.0)&INDICATOR=G271_T).
+Aggregate (non-Social Security) transfers to households are set as a share of GDP with the parameter $\alpha_T$. We exclude Social Security from transfers since it is modeled specifically. We compute this from the IMF GFS Statement of Operations data as
+    <center>$\alpha_T = (\texttt{G27\_T} - \texttt{G271\_T}) / 100$</center>
+For Ethiopia, the available IMF GFS percent-of-GDP series for this calibration are published under the `Budgetary central government` sector (`S1311B`), so the OG-ETH calibration uses that sector rather than mechanically reusing another country repo's sector code. Using the 2024 IMF values for `G27_T` and `G271_T`, the default calibration sets $\alpha_T = 0.0$.
 
 ### Government expenditures
 
 Government spending on goods and services are also set as a share of GDP with the parameter $\alpha_G$. We define government spending as:
     <center>Government Spending = Total Outlays - Transfers - Net Interest on Debt - Social Security</center>
-With this definition, the share of government expenditure to GDP is 0.095 based on [data from the IMF](https://data.imf.org/en/Data-Explorer?datasetUrn=IMF.RES:WEO(9.0.0)&INDICATOR=GGX).
+In IMF GFS Statement of Operations terms, we compute
+    <center>$\alpha_G = (\texttt{G2\_T} - \texttt{G24\_T} - \texttt{G27\_T}) / 100$</center>
+Using Ethiopia's 2024 `S1311B` IMF GFS percent-of-GDP series, the default calibration sets $\alpha_G = 0.0453$.
 
 
 (SecLWI_footnotes)=

docs/create_doc_figures.py

@@ -11,7 +11,6 @@
 from ogcore import parameter_plots as pp
 from ogcore import demographics as demog
 
-
 CUR_DIR = os.path.dirname(os.path.realpath(__file__))
 UN_COUNTRY_CODE = "231"
 plot_path = os.path.join(CUR_DIR, "book", "content", "calibration", "images")

environment.yml

@@ -36,7 +36,6 @@ dependencies:
 - pip
 - pip:
   - openpyxl>=3.1.2
-  - pandas-datareader
   - linecheck
   - ogcore>=0.14.11
   - sphinx-exercise

ogeth/calibrate.py

@@ -1,6 +1,7 @@
 from ogeth import macro_params, income
 from ogeth import input_output as io
 import os
+import warnings
 import numpy as np
 import datetime
 from ogcore import demographics
@@ -16,7 +17,7 @@ def __init__(
         macro_data_end_year=datetime.datetime(2024, 12, 31),
         demographic_data_path=None,
         output_path=None,
-        update_from_api=True,  # Set True to update from World Bank and UN APIs
+        update_from_api=False,  # Set True to update from World Bank and UN APIs
     ):
         """
         Constructor for the Calibration class.
@@ -37,75 +38,98 @@ def __init__(
             if not os.path.exists(output_path):
                 os.makedirs(output_path)
 
-        # Macro estimation
-        self.macro_params = macro_params.get_macro_params(
-            macro_data_start_year,
-            macro_data_end_year,
-            update_from_api=update_from_api,
-        )
-        print("Calibrated macro parameters.")
-        print(self.macro_params)
+        # Only return successfully refreshed values; packaged defaults on p
+        # remain the baseline when update_from_api is False or a source fails.
+        self.macro_params = {}
+        self.demographic_params = {}
+        self.e = None
+        self.alpha_c = np.array([1.0]) if p.I == 1 else None
+        self.io_matrix = np.array([[1.0]]) if p.M == 1 else None
+
+        if not update_from_api:
+            return
+
+        try:
+            self.macro_params = macro_params.get_macro_params(
+                macro_data_start_year,
+                macro_data_end_year,
+                update_from_api=update_from_api,
+            )
+            print("Calibrated macro parameters.")
+            print(self.macro_params)
+        except Exception as exc:
+            warnings.warn(f"Macro params update failed: {exc}", stacklevel=2)
 
-        # io matrix and alpha_c
         if p.I > 1:  # no need if just one consumption good
-            alpha_c_dict = io.get_alpha_c()
-            # check that model dimensions are consistent with alpha_c
-            assert p.I == len(list(alpha_c_dict.keys()))
-            self.alpha_c = np.array(list(alpha_c_dict.values()))
-        else:
-            self.alpha_c = np.array([1.0])
+            try:
+                alpha_c_dict = io.get_alpha_c()
+                # check that model dimensions are consistent with alpha_c
+                assert p.I == len(list(alpha_c_dict.keys()))
+                self.alpha_c = np.array(list(alpha_c_dict.values()))
+            except Exception as exc:
+                warnings.warn(f"alpha_c update failed: {exc}", stacklevel=2)
         if p.M > 1:  # no need if just one production good
-            io_df = io.get_io_matrix()
-            # check that model dimensions are consistent with io_matrix
-            assert p.M == len(list(io_df.keys()))
-            self.io_matrix = io_df.values
-        else:
-            self.io_matrix = np.array([[1.0]])
+            try:
+                io_df = io.get_io_matrix()
+                # check that model dimensions are consistent with io_matrix
+                assert p.M == len(list(io_df.keys()))
+                self.io_matrix = io_df.values
+            except Exception as exc:
+                warnings.warn(f"io_matrix update failed: {exc}", stacklevel=2)
 
-        # demographics
-        self.demographic_params = demographics.get_pop_objs(
-            p.E,
-            p.S,
-            p.T,
-            0,
-            99,
-            country_id="231",
-            initial_data_year=p.start_year - 1,
-            final_data_year=p.start_year + 1,
-            GraphDiag=False,
-            download_path=demographic_data_path,
-        )
+        try:
+            self.demographic_params = demographics.get_pop_objs(
+                p.E,
+                p.S,
+                p.T,
+                0,
+                99,
+                country_id="231",
+                initial_data_year=p.start_year - 1,
+                final_data_year=p.start_year + 1,
+                GraphDiag=False,
+                download_path=demographic_data_path,
+            )
 
-        # demographics for 80 period lives (needed for getting e below)
-        demog80 = demographics.get_pop_objs(
-            20,
-            80,
-            p.T,
-            0,
-            99,
-            country_id="231",
-            initial_data_year=p.start_year - 1,
-            final_data_year=p.start_year + 1,
-            GraphDiag=False,
-        )
+            # demographics for 80 period lives (needed for getting e below)
+            demog80 = demographics.get_pop_objs(
+                20,
+                80,
+                p.T,
+                0,
+                99,
+                country_id="231",
+                initial_data_year=p.start_year - 1,
+                final_data_year=p.start_year + 1,
+                GraphDiag=False,
+            )
 
-        # earnings profiles
-        self.e = income.get_e_interp(
-            p.E,
-            p.S,
-            p.J,
-            p.lambdas,
-            demog80["omega_SS"],
-            plot_path=output_path,
-        )
+            # earnings profiles
+            self.e = income.get_e_interp(
+                p.E,
+                p.S,
+                p.J,
+                p.lambdas,
+                demog80["omega_SS"],
+                plot_path=output_path,
+            )
+        except Exception as exc:
+            warnings.warn(
+                f"Demographics/income update failed: {exc}", stacklevel=2
+            )
+            self.demographic_params = {}
+            self.e = None
 
     # method to return all newly calibrated parameters in a dictionary
     def get_dict(self):
-        dict = {}
-        dict.update(self.macro_params)
-        dict["e"] = self.e
-        dict["alpha_c"] = self.alpha_c
-        dict["io_matrix"] = self.io_matrix
-        dict.update(self.demographic_params)
+        d = {}
+        d.update(self.macro_params)
+        d.update(self.demographic_params)
+        if self.e is not None:
+            d["e"] = self.e
+        if self.alpha_c is not None:
+            d["alpha_c"] = self.alpha_c
+        if self.io_matrix is not None:
+            d["io_matrix"] = self.io_matrix
 
-        return dict
+        return d

ogeth/input_output.py

@@ -3,20 +3,27 @@
 import os
 from ogeth.constants import CONS_DICT, PROD_DICT
 
-
 CUR_DIR = os.path.dirname(os.path.realpath(__file__))
-"""
-Read in Social Accounting Matrix (SAM) file
-"""
-# Read in SAM file
-# SAM file:
 sam_path = os.path.join(CUR_DIR, "data", "IFPRI_SAM_ETH_2022_SAM.csv")
-SAM = pd.read_csv(sam_path, index_col=1, thousands=",")
-# replace NaN with 0
-SAM.fillna(0, inplace=True)
 
 
-def get_alpha_c(sam=SAM, cons_dict=CONS_DICT):
+def read_SAM():
+    """
+    Read the packaged Ethiopia SAM file.
+
+    Returns:
+        pd.DataFrame | None: parsed SAM table, or None if unavailable
+    """
+    try:
+        sam = pd.read_csv(sam_path, index_col=1, thousands=",")
+        sam.fillna(0, inplace=True)
+        return sam
+    except Exception as exc:
+        print(f"Failed to read packaged SAM file: {exc}")
+        return None
+
+
+def get_alpha_c(sam=None, cons_dict=CONS_DICT):
     """
     Calibrate the alpha_c vector, showing the shares of household
     expenditures for each consumption category
@@ -28,6 +35,11 @@ def get_alpha_c(sam=SAM, cons_dict=CONS_DICT):
     Returns:
         alpha_c (dict): Dictionary of shares of household expenditures
     """
+    if sam is None:
+        sam = read_SAM()
+    if sam is None:
+        raise RuntimeError("SAM data is unavailable. Cannot compute alpha_c.")
+
     hh_cols = [
         "hhd-r1",
         "hhd-r2",
@@ -55,7 +67,7 @@ def get_alpha_c(sam=SAM, cons_dict=CONS_DICT):
     return alpha_c
 
 
-def get_io_matrix(sam=SAM, cons_dict=CONS_DICT, prod_dict=PROD_DICT):
+def get_io_matrix(sam=None, cons_dict=CONS_DICT, prod_dict=PROD_DICT):
     """
     Calibrate the io_matrix array.  This array relates the share of each
     production category in each consumption category
@@ -68,6 +80,13 @@ def get_io_matrix(sam=SAM, cons_dict=CONS_DICT, prod_dict=PROD_DICT):
     Returns:
         io_df (pd.DataFrame): Dataframe of io_matrix
     """
+    if sam is None:
+        sam = read_SAM()
+    if sam is None:
+        raise RuntimeError(
+            "SAM data is unavailable. Cannot compute io_matrix."
+        )
+
     # Create initial matrix as dataframe of 0's to fill in
     io_dict = {}
     for key in prod_dict.keys():