73 Selecting specific Input folder (and/or Euromod Policy Schedule) per-run via config file by andrewbaxter439 · Pull Request #207 · simpaths/SimPaths

andrewbaxter439 · 2025-06-23T12:21:00Z

What

This allows arguments passed via a config file to set input directory, initial population files and euromod output folders differently for each run

Why

This would allow multiple scenarios to be tested to have separate input directories
a script runs through the config files which access respective directories in turn

…/73-selecting-specific-input-folder

…folder

Copilot

Pull Request Overview

This PR enables per-run configuration of input, initial population, and Euromod policy schedule directories via a config file, allowing multiple scenarios with separate data folders.

Introduces parameter_args for directory settings in YAML and updates path constructions to use Parameters.getInputDirectory() and similar methods.
Adds reflection-based setExperimentFolders to configure Microsim Experiment input/output paths.
Updates tests and CSV/database parsers to use the new directory parameters.

Reviewed Changes

Copilot reviewed 6 out of 8 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
`PersonTest.java`	Sets custom input directory for EQ-5D parameter tests
`TaxDonorDataParser.java`	Uses `Parameters.getInputDirectory()` in JDBC and Hibernate URL
`SimPathsStart.java`	Replaces hard-coded “input” paths with parameters and updates exception message
`SimPathsMultiRun.java`	Adds handling of `input_directory` and `working_directory` parameters, and reflection helpers
`DataParser.java`	Uses `Parameters.getInputDirectory()` for H2 connection
`config/default.yml`	Adds `parameter_args.INPUT_DIRECTORY` entry

Comments suppressed due to low confidence (1)

config/default.yml:65

YAML key INPUT_DIRECTORY is uppercase but the code expects "input_directory". Rename the key to match the switch-case or adjust the code to accept the uppercase variant.

  INPUT_DIRECTORY: input

Copilot · 2025-06-23T16:30:18Z

src/test/java/simpaths/model/PersonTest.java

            @BeforeEach
            public void setupLawrenceCoefficients() {

+                Parameters.setInputDirectory("src/test/java/simpaths/testinput");


[nitpick] Modifying a global Parameters directory in @beforeeach without resetting it may lead to state leakage between tests; consider using JUnit's @tempdir or resetting to a default value in an @AfterEach.

Copilot · 2025-06-23T16:30:18Z

src/main/java/simpaths/model/taxes/database/TaxDonorDataParser.java

        try {
            Class.forName("org.h2.Driver");
-            conn = DriverManager.getConnection("jdbc:h2:file:./input" + File.separator + "input;TRACE_LEVEL_FILE=0;TRACE_LEVEL_SYSTEM_OUT=0;AUTO_SERVER=TRUE", "sa", "");
+            conn = DriverManager.getConnection("jdbc:h2:file:" + Parameters.getInputDirectory() + "input;TRACE_LEVEL_FILE=0;TRACE_LEVEL_SYSTEM_OUT=0;AUTO_SERVER=TRUE", "sa", "");


Missing file separator between getInputDirectory() and input; this may produce an invalid path. Use Parameters.getInputDirectory() + File.separator + "input".

Suggested change

conn = DriverManager.getConnection("jdbc:h2:file:" + Parameters.getInputDirectory() + "input;TRACE_LEVEL_FILE=0;TRACE_LEVEL_SYSTEM_OUT=0;AUTO_SERVER=TRUE", "sa", "");

conn = DriverManager.getConnection("jdbc:h2:file:" + Parameters.getInputDirectory() + File.separator + "input;TRACE_LEVEL_FILE=0;TRACE_LEVEL_SYSTEM_OUT=0;AUTO_SERVER=TRUE", "sa", "");

Copilot · 2025-06-23T16:30:19Z

src/main/java/simpaths/model/taxes/database/TaxDonorDataParser.java

-            EntityManager em = Persistence.createEntityManagerFactory("tax-database").createEntityManager();
+            // access database and obtain donor pool
+            Map propertyMap = new HashMap();
+            propertyMap.put("hibernate.connection.url", "jdbc:h2:file:" + Parameters.getInputDirectory() + "input" + ";TRACE_LEVEL_FILE=0;TRACE_LEVEL_SYSTEM_OUT=0;AUTO_SERVER=TRUE");


Path concatenation omits a separator before input, leading to an incorrect JDBC URL. Insert File.separator between the directory and "input".

Suggested change

propertyMap.put("hibernate.connection.url", "jdbc:h2:file:" + Parameters.getInputDirectory() + "input" + ";TRACE_LEVEL_FILE=0;TRACE_LEVEL_SYSTEM_OUT=0;AUTO_SERVER=TRUE");

propertyMap.put("hibernate.connection.url", "jdbc:h2:file:" + Parameters.getInputDirectory() + File.separator + "input" + ";TRACE_LEVEL_FILE=0;TRACE_LEVEL_SYSTEM_OUT=0;AUTO_SERVER=TRUE");

Copilot · 2025-06-23T16:30:19Z

src/main/java/simpaths/experiment/SimPathsStart.java


 		//Adjust the country and year to the value read from Excel, which is updated when the database is rebuilt. Otherwise it will set the country and year to the last one used to build the database
-		MultiKeyCoefficientMap lastDatabaseCountryAndYear = ExcelAssistant.loadCoefficientMap("input" + File.separator + Parameters.DatabaseCountryYearFilename + ".xlsx", "Data", 1, 1);
+		MultiKeyCoefficientMap lastDatabaseCountryAndYear = ExcelAssistant.loadCoefficientMap(Parameters.getInputDirectory() + Parameters.DatabaseCountryYearFilename + ".xlsx", "Data", 1, 1);


Concatenating getInputDirectory() and filename without a separator may cause missing-path errors; consider adding File.separator between them.

Suggested change

MultiKeyCoefficientMap lastDatabaseCountryAndYear = ExcelAssistant.loadCoefficientMap(Parameters.getInputDirectory() + Parameters.DatabaseCountryYearFilename + ".xlsx", "Data", 1, 1);

MultiKeyCoefficientMap lastDatabaseCountryAndYear = ExcelAssistant.loadCoefficientMap(

java.nio.file.Paths.get(Parameters.getInputDirectory(), Parameters.DatabaseCountryYearFilename + ".xlsx").toString(),

"Data", 1, 1);

Copilot · 2025-06-23T16:30:19Z

src/main/java/simpaths/experiment/SimPathsStart.java

+			throw new FileNotFoundException("Policy Schedule file '"+ File.separator + Parameters.getInputDirectory() +
 					Parameters.EUROMODpolicyScheduleFilename + ".xlsx` doesn't exist. " +
 					"Provide excel file or use `--rewrite-policy-schedule` to re-construct from available policy files.");


The exception message is confusing: it prefixes the path with File.separator and includes a stray backtick (). Refactor to show the full path correctly, e.g., "Policy Schedule file '" + fullPath + "' doesn't exist."`.

Suggested change

throw new FileNotFoundException("Policy Schedule file '"+ File.separator + Parameters.getInputDirectory() +

Parameters.EUROMODpolicyScheduleFilename + ".xlsx` doesn't exist. " +

"Provide excel file or use `--rewrite-policy-schedule` to re-construct from available policy files.");

throw new FileNotFoundException("Policy Schedule file '" + Parameters.getInputDirectory() +

Parameters.EUROMODpolicyScheduleFilename + ".xlsx' doesn't exist. " +

"Provide the Excel file or use `--rewrite-policy-schedule` to re-construct it from available policy files.");

Copilot · 2025-06-23T16:30:19Z

src/main/java/simpaths/experiment/SimPathsStart.java


 			// load previously stored values for policy description and initiation year
-			MultiKeyCoefficientMap previousEUROMODfileInfo = ExcelAssistant.loadCoefficientMap("input" + File.separator + Parameters.EUROMODpolicyScheduleFilename + ".xlsx", country.toString(), 1, 3);
+			MultiKeyCoefficientMap previousEUROMODfileInfo = ExcelAssistant.loadCoefficientMap(Parameters.getInputDirectory() + Parameters.EUROMODpolicyScheduleFilename + ".xlsx", country.toString(), 1, 3);


Missing path separator before the filename; add File.separator between getInputDirectory() and the schedule filename.

Suggested change

MultiKeyCoefficientMap previousEUROMODfileInfo = ExcelAssistant.loadCoefficientMap(Parameters.getInputDirectory() + Parameters.EUROMODpolicyScheduleFilename + ".xlsx", country.toString(), 1, 3);

MultiKeyCoefficientMap previousEUROMODfileInfo = ExcelAssistant.loadCoefficientMap(Parameters.getInputDirectory() + File.separator + Parameters.EUROMODpolicyScheduleFilename + ".xlsx", country.toString(), 1, 3);

Copilot · 2025-06-23T16:30:20Z

src/main/java/simpaths/experiment/SimPathsMultiRun.java

+		try {
+			Field inputDir = Experiment.class.getDeclaredField("inputFolder");
+			inputDir.setAccessible(true);
+			inputDir.set(Experiment.class, Parameters.getInputDirectory());


For setting a static field via reflection, the first argument to Field.set should be null, not Experiment.class. Use inputDir.set(null, Parameters.getInputDirectory()).

Suggested change

inputDir.set(Experiment.class, Parameters.getInputDirectory());

inputDir.set(null, Parameters.getInputDirectory());

Copilot · 2025-06-23T16:30:20Z

src/main/java/simpaths/experiment/SimPathsMultiRun.java

 		SimPathsModel model = new SimPathsModel(Country.getCountryFromNameString(countryString), startYear);
 		if (persist_population) model.setPersistPopulation(true);
-		if (persist_root) model.setPersistDatabasePath("./input/input");
+		if (persist_root) model.setPersistDatabasePath(Parameters.getInputDirectory() + "input");


Concatenation lacks a separator before input; include File.separator (or ensure the directory string ends with one) to avoid malformed paths.

Suggested change

if (persist_root) model.setPersistDatabasePath(Parameters.getInputDirectory() + "input");

if (persist_root) model.setPersistDatabasePath(ensureTrailingSeparator(Parameters.getInputDirectory()) + "input");

Copilot · 2025-06-23T16:30:20Z

src/main/java/simpaths/data/startingpop/DataParser.java

 		try {
 			Class.forName("org.h2.Driver");
-			conn = DriverManager.getConnection("jdbc:h2:file:./input" + File.separator + "input;TRACE_LEVEL_FILE=0;TRACE_LEVEL_SYSTEM_OUT=0;AUTO_SERVER=TRUE", "sa", "");
+			conn = DriverManager.getConnection("jdbc:h2:file:" + Parameters.getInputDirectory() + "input;TRACE_LEVEL_FILE=0;TRACE_LEVEL_SYSTEM_OUT=0;AUTO_SERVER=TRUE", "sa", "");


Missing file separator when building the H2 JDBC URL; add File.separator between the configured directory and input.

Suggested change

conn = DriverManager.getConnection("jdbc:h2:file:" + Parameters.getInputDirectory() + "input;TRACE_LEVEL_FILE=0;TRACE_LEVEL_SYSTEM_OUT=0;AUTO_SERVER=TRUE", "sa", "");

conn = DriverManager.getConnection("jdbc:h2:file:" + java.nio.file.Paths.get(Parameters.getInputDirectory(), "input").toString() + ";TRACE_LEVEL_FILE=0;TRACE_LEVEL_SYSTEM_OUT=0;AUTO_SERVER=TRUE", "sa", "");

andrewbaxter439 · 2025-06-23T17:54:16Z

@justin-ven @pbronka - this is mostly a structural change, would allow parameters to point runs to different places to get input parts (can even set working directory outside of directory with code in it). tested manually with a folder called "impoot" and it successfully ran!

justin-ven · 2025-07-18T11:50:13Z

Looks OK to me Andy - we can always debug later if necessary

andrewbaxter439 added 23 commits November 22, 2024 11:43

de-finalise input directory and input filenames

974c191

getters and setters for amendable input dirs

8d82eef

replace all mentions in Parameters.java of input with input dir getter

0204764

de-finalise some potential changing parameters

bfeae9c

simpaths start calls dynamic input folder loc

d31cef7

reads in parameter args

641f407

setting paramters including input directory in Experiment

55926f2

update parameters (when building model)

c42085d

experimental configs

166e4b6

add logging

7cdf68b

correct file finding

66dec30

parse input folder before looking up country

78855ae

Merge remote-tracking branch 'sphsu-origin/develop' into experimental…

8963b4a

…/73-selecting-specific-input-folder

Merge branch 'develop' into experimental/73-selecting-specific-input-…

3ab6723

…folder

some safe checking for last database country file if doesnt exist

bcfabdd

correct persist root database ref

008221a

correct database parsing path

c173ef8

override all calls to static root database in parameters

6043d3e

detect if input filder is absolute or not

cde7ba7

prefer lower-case args to parameters

5bb70b7

fixes to find correct input folder (maybe?)

2fc7419

eq5d test file moved to test folder

f25063c

caught one last reference to static root database

f377ed7

andrewbaxter439 requested a review from Copilot June 23, 2025 16:27

Copilot AI reviewed Jun 23, 2025

View reviewed changes

andrewbaxter439 added 2 commits June 23, 2025 17:55

change yaml file to commented out defaults

c64aa5b

ensure parsing of aboslute/relative paths in pop/euromod args

e8d9e5d

andrewbaxter439 self-assigned this Jun 23, 2025

andrewbaxter439 requested review from justin-ven and pbronka June 23, 2025 17:52

andrewbaxter439 marked this pull request as ready for review June 23, 2025 17:53

justin-ven merged commit b10aa96 into simpaths:develop Jul 18, 2025
5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

73 Selecting specific Input folder (and/or Euromod Policy Schedule) per-run via config file#207

73 Selecting specific Input folder (and/or Euromod Policy Schedule) per-run via config file#207
justin-ven merged 25 commits intosimpaths:developfrom
MRC-CSO-SPHSU:experimental/73-selecting-specific-input-folder

andrewbaxter439 commented Jun 23, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Jun 23, 2025

Uh oh!

Copilot AI Jun 23, 2025

Uh oh!

Copilot AI Jun 23, 2025

Uh oh!

Copilot AI Jun 23, 2025

Uh oh!

Copilot AI Jun 23, 2025

Uh oh!

Copilot AI Jun 23, 2025

Uh oh!

Copilot AI Jun 23, 2025

Uh oh!

Copilot AI Jun 23, 2025

Uh oh!

Copilot AI Jun 23, 2025

Uh oh!

andrewbaxter439 commented Jun 23, 2025

Uh oh!

justin-ven commented Jul 18, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	conn = DriverManager.getConnection("jdbc:h2:file:" + Parameters.getInputDirectory() + "input;TRACE_LEVEL_FILE=0;TRACE_LEVEL_SYSTEM_OUT=0;AUTO_SERVER=TRUE", "sa", "");
	conn = DriverManager.getConnection("jdbc:h2:file:" + Parameters.getInputDirectory() + File.separator + "input;TRACE_LEVEL_FILE=0;TRACE_LEVEL_SYSTEM_OUT=0;AUTO_SERVER=TRUE", "sa", "");

	propertyMap.put("hibernate.connection.url", "jdbc:h2:file:" + Parameters.getInputDirectory() + "input" + ";TRACE_LEVEL_FILE=0;TRACE_LEVEL_SYSTEM_OUT=0;AUTO_SERVER=TRUE");
	propertyMap.put("hibernate.connection.url", "jdbc:h2:file:" + Parameters.getInputDirectory() + File.separator + "input" + ";TRACE_LEVEL_FILE=0;TRACE_LEVEL_SYSTEM_OUT=0;AUTO_SERVER=TRUE");

	MultiKeyCoefficientMap previousEUROMODfileInfo = ExcelAssistant.loadCoefficientMap(Parameters.getInputDirectory() + Parameters.EUROMODpolicyScheduleFilename + ".xlsx", country.toString(), 1, 3);
	MultiKeyCoefficientMap previousEUROMODfileInfo = ExcelAssistant.loadCoefficientMap(Parameters.getInputDirectory() + File.separator + Parameters.EUROMODpolicyScheduleFilename + ".xlsx", country.toString(), 1, 3);

	inputDir.set(Experiment.class, Parameters.getInputDirectory());
	inputDir.set(null, Parameters.getInputDirectory());

	if (persist_root) model.setPersistDatabasePath(Parameters.getInputDirectory() + "input");
	if (persist_root) model.setPersistDatabasePath(ensureTrailingSeparator(Parameters.getInputDirectory()) + "input");

Conversation

andrewbaxter439 commented Jun 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI Jun 23, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jun 23, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jun 23, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jun 23, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jun 23, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jun 23, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jun 23, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jun 23, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Jun 23, 2025

Choose a reason for hiding this comment

Uh oh!

andrewbaxter439 commented Jun 23, 2025

Uh oh!

justin-ven commented Jul 18, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

andrewbaxter439 commented Jun 23, 2025 •

edited

Loading