Skip to content

Speeding up database handling in initial populations #249

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
andrewbaxter439 merged 23 commits into from
Sep 25, 2025
Merged

Speeding up database handling in initial populations #249

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
c4367bf
Code changes to accommodate new data
dav-sonn Aug 6, 2025
4aaa5ee
Update of the code with the latests estimates
dav-sonn Aug 28, 2025
3bdeba0
Update persistence.xml
dav-sonn Sep 4, 2025
5a7e6e2
updated initial population to include new variables
andrewbaxter439 Sep 17, 2025
db211f7
updated statistics for new population
andrewbaxter439 Sep 17, 2025
6c33fce
convert all health variables to numeric
andrewbaxter439 Sep 20, 2025
4d105e2
foreign keys buid and household id cast to bigint
andrewbaxter439 Sep 20, 2025
82a79c4
correct idhousehold
andrewbaxter439 Sep 20, 2025
e6f1a7a
missing semicolons
andrewbaxter439 Sep 20, 2025
1da2cc7
lazy fetching of BUs and Persons to speed things up
andrewbaxter439 Sep 21, 2025
d9b0846
double rather than decimal
andrewbaxter439 Sep 21, 2025
62bfcac
save intellij h2 sql dialect setting
andrewbaxter439 Sep 22, 2025
cccda4a
Merge pull request #250 from andrewbaxter439/update-code/new-data
andrewbaxter439 Sep 22, 2025
29e8f52
add employment stats test
andrewbaxter439 Sep 22, 2025
a4a0dbf
update statistics
andrewbaxter439 Sep 22, 2025
0a6b661
change data types of scores to decimal format
andrewbaxter439 Sep 22, 2025
1ae4a52
change persist option
andrewbaxter439 Sep 9, 2025
96b6f41
oops, didn't actually change default
andrewbaxter439 Sep 23, 2025
5031caa
Update src/main/java/simpaths/model/Person.java
dav-sonn Sep 24, 2025
3a3b7bc
Update src/main/java/simpaths/model/Person.java
dav-sonn Sep 24, 2025
08b500f
Merge branch 'develop' into bugfix/tidy-database-creation
andrewbaxter439 Sep 24, 2025
2344435
updated statistics for joined branch
andrewbaxter439 Sep 24, 2025
1dde04d
deleted duplicated input files
andrewbaxter439 Sep 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .idea/sqldialects.xml
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion config/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ executeWithGui: false
randomSeed: 606
startYear: 2019
endYear: 2026
popSize: 170000
popSize: 50000

# Arguments passed to the SimPathsModel
model_args:
Expand Down
Binary file modified input/DatabaseCountryYear.xlsx
View file
Open in desktop
Binary file not shown.
Binary file modified input/EUROMODpolicySchedule.xlsx
View file
Open in desktop
Binary file not shown.
74,046 changes: 37,063 additions & 36,983 deletions input/InitialPopulations/training/population_initial_UK_2019.csv
View file
Open in desktop

Large diffs are not rendered by default.

Binary file modified input/reg_RMSE.xlsx
View file
Open in desktop
Binary file not shown.
Binary file modified input/reg_education.xlsx
View file
Open in desktop
Binary file not shown.
Binary file modified input/reg_employmentSelection.xlsx
View file
Open in desktop
Binary file not shown.
Binary file modified input/reg_fertility.xlsx
View file
Open in desktop
Binary file not shown.
Binary file modified input/reg_health.xlsx
View file
Open in desktop
Binary file not shown.
Binary file modified input/reg_home_ownership.xlsx
View file
Open in desktop
Binary file not shown.
Binary file modified input/reg_income.xlsx
View file
Open in desktop
Binary file not shown.
Binary file modified input/reg_leaveParentalHome.xlsx
View file
Open in desktop
Binary file not shown.
Binary file added input/reg_leave_parental_home.xlsx
View file
Open in desktop
Binary file not shown.
Binary file modified input/reg_partnership.xlsx
View file
Open in desktop
Binary file not shown.
Binary file modified input/reg_retirement.xlsx
View file
Open in desktop
Binary file not shown.
Binary file modified input/reg_wages.xlsx
View file
Open in desktop
Binary file not shown.
Binary file modified input/scenario_parametricMatching.xlsx
View file
Open in desktop
Binary file not shown.
18 changes: 9 additions & 9 deletions src/main/java/simpaths/data/ManagerRegressions.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -196,15 +196,6 @@ public static OrderedRegression getOrderedRegression(RegressionName regression)
throw new RuntimeException("requested ordered regression is not recognised: " + regression.name());

switch (regression) {
case HealthH1a -> {
return Parameters.getRegHealthH1a();
}
case HealthH1b -> {
return Parameters.getRegHealthH1b();
}
case EducationE2a -> {
return Parameters.getRegEducationE2a();
}
default -> {
throw new RuntimeException("unrecognised regression (1)");
}
Expand All @@ -217,6 +208,15 @@ public static GeneralisedOrderedRegression getGeneralisedOrderedRegression(Regre
throw new RuntimeException("requested generalised ordered regression is not recognised: " + regression.name());

switch (regression) {
case HealthH1a -> {
return Parameters.getRegHealthH1a();
}
case HealthH1b -> {
return Parameters.getRegHealthH1b();
}
case EducationE2a -> {
return Parameters.getRegEducationE2a();
}
default -> {
throw new RuntimeException("unrecognised regression (1)");
}
Expand Down
105 changes: 54 additions & 51 deletions src/main/java/simpaths/data/Parameters.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,13 +119,16 @@ public class Parameters {
"ded", //in education dummy
"der", //return to education dummy
"dot", //ethnicity
"dot01", //ethnicity 6 categories
"dhe", //health status
"dhm", //mental health status
"dhe_mcs", //mental health - SF12 score MCS
"dhe_pcs", //physical health - SF12 score PCS
"dhe_mcssp", //mental health - SF12 score MCS (partner)
"dhe_pcssp", //physical health - SF12 score PCS (partner)
"dhm", //mental health status
"scghq2_dv", //mental health status case based
"dhm_ghq", //mental health status case based dummy (1 = psychologically distressed)
"dls", //life satisfaction
"dhe_mcs", //mental health - SF12 score MCS
"dhe_pcs", //physical health - SF12 score PCS
"financial_distress", //financial distress
"dcpyy", //years in partnership
"dcpagdf", //partners age difference
Expand All @@ -137,7 +140,7 @@ public class Parameters {
"ypnoab", //gross personal pension (public / occupational) income
"yplgrs_dv", //gross personal employment income
"ynbcpdf_dv", //difference partner income
"dlltsd", //long-term sick or disabled
"dlltsd01", //long-term sick or disabled (we use this -and not dlltsd- in the DataParser)
"sedex", //year left education
"stm", //system variable - year
"swv", //system variable - wave
Expand Down Expand Up @@ -278,8 +281,8 @@ else if(numberOfChildren <= 5) {

//public static int MAX_AGE_IN_EDUCATION;// = MAX_AGE;//30; // Max age a person can stay in education //Cannot set here, as MAX_AGE is not known yet. Now set to MAX_AGE in buildObjects in Model class.
//public static int MAX_AGE_MARRIAGE;// = MAX_AGE;//75; // Max age a person can marry //Cannot set here, as MAX_AGE is not known yet. Now set to MAX_AGE in buildObjects in Model class.
private static int MIN_START_YEAR = 2015; //Minimum allowed starting point. Should correspond to the oldest initial population.
private static int MAX_START_YEAR = 2019; //Maximum allowed starting point. Should correspond to the most recent initial population.
private static int MIN_START_YEAR = 2011; //Minimum allowed starting point. Should correspond to the oldest initial population.
private static int MAX_START_YEAR = 2023; //Maximum allowed starting point. Should correspond to the most recent initial population.
public static int startYear;
public static int endYear;
private static final int MIN_START_YEAR_TESTING = 2019;
Expand All @@ -300,7 +303,7 @@ else if(numberOfChildren <= 5) {

public static final boolean systemOut = true;

//Bootstrap all the regression coefficients if true, or only the female labour participation regressions when false
//Bootstrap all the regression coefficients if true
public static final boolean bootstrapAll = false;

//Scheduling
Expand Down Expand Up @@ -679,8 +682,8 @@ else if(numberOfChildren <= 5) {
/////////////////////////////////////////////////////////////////// REGRESSION OBJECTS //////////////////////////////////////////

//Health
private static OrderedRegression regHealthH1a;
private static OrderedRegression regHealthH1b;
private static GeneralisedOrderedRegression regHealthH1a;
private static GeneralisedOrderedRegression regHealthH1b;
private static BinomialRegression regHealthH2b;

//Social care
Expand Down Expand Up @@ -739,7 +742,7 @@ else if(numberOfChildren <= 5) {
//Education
private static BinomialRegression regEducationE1a;
private static BinomialRegression regEducationE1b;
private static OrderedRegression regEducationE2a;
private static GeneralisedOrderedRegression regEducationE2a;

//Partnership
private static BinomialRegression regPartnershipU1a;
Expand Down Expand Up @@ -1110,8 +1113,8 @@ public static void loadParameters(Country country, int maxAgeModel, boolean enab
columnsIncomeI3b_selection = 22;
columnsLeaveHomeP1a = 19;
columnsHomeownership = 32;
columnsRetirementR1a = 19;
columnsRetirementR1b = 24;
columnsRetirementR1a = 33;
columnsRetirementR1b = 37;
columnsChildcareC1a = 37;
columnsChildcareC1b = 37;
columnsValidationStudentsByAge = 10;
Expand All @@ -1131,14 +1134,14 @@ public static void loadParameters(Country country, int maxAgeModel, boolean enab
columnsValidationLabourSupplyByEducation = 3;
}
else if(country.equals(Country.UK)) {
columnsWagesMalesNE = 30;
columnsWagesMalesE = 31;
columnsWagesFemalesNE = 30;
columnsWagesFemalesE = 31;
columnsEmploymentSelectionMalesNE = 30;
columnsEmploymentSelectionMalesE = 29;
columnsEmploymentSelectionFemalesNE = 30;
columnsEmploymentSelectionFemalesE = 29;
columnsWagesMalesNE = 33;
columnsWagesMalesE = 34;
columnsWagesFemalesNE = 33;
columnsWagesFemalesE = 34;
columnsEmploymentSelectionMalesNE = 32;
columnsEmploymentSelectionMalesE = 32;
columnsEmploymentSelectionFemalesNE = 32;
columnsEmploymentSelectionFemalesE = 32;
columnsLabourSupplyUtilityMales = 19;
columnsLabourSupplyUtilityFemales = 12;
columnsLabourSupplyUtilityMalesWithDependent = 15;
Expand All @@ -1148,9 +1151,9 @@ else if(country.equals(Country.UK)) {
columnsLabourSupplyUtilityCouples = 64;
columnsLabourCovid19_SE = 1;
columnsLabourCovid19_2a_processes = 1;
columnsHealthH1a = 28;
columnsHealthH1b = 35;
columnsHealthH2b = 35;
columnsHealthH1a = 31;
columnsHealthH1b = 110;
columnsHealthH2b = 36;
columnsHealthHM1 = 30;
columnsHealthHM2Males = 15;
columnsHealthHM2Females = 15;
Expand Down Expand Up @@ -1187,31 +1190,31 @@ else if(country.equals(Country.UK)) {
columnsUnemploymentU1c = 19;
columnsUnemploymentU1d = 19;
columnsFinancialDistress = 50;
columnsEducationE1a = 19;
columnsEducationE1b = 25;
columnsEducationE2a = 11;
columnsPartnershipU1a = 27;
columnsPartnershipU1b = 31;
columnsEducationE1a = 24;
columnsEducationE1b = 29;
columnsEducationE2a = 24;
columnsPartnershipU1a = 16;
columnsPartnershipU1b = 29;
columnsPartnershipU2b = 38;
columnsFertilityF1a = 6;
columnsFertilityF1b = 26;
columnsFertilityF1b = 36;
columnsIncomeI1a = 19; //*
columnsIncomeI1b = 31; //*
columnsIncomeI3a = 20;
columnsIncomeI3b = 29;
columnsIncomeI3a = 26;
columnsIncomeI3b = 36;
columnsIncomeI3c = 28; //*
columnsIncomeI4a = 24; //*
columnsIncomeI4b = 25;
columnsIncomeI5a = 25;
columnsIncomeI5b = 25;
columnsIncomeI4b = 31;
columnsIncomeI5a = 30;
columnsIncomeI5b = 31;
columnsIncomeI6a = 22; //*
columnsIncomeI6b = 22; //*
columnsIncomeI3a_selection = 20;
columnsIncomeI3b_selection = 29;
columnsIncomeI3a_selection = 26;
columnsIncomeI3b_selection = 36;
columnsLeaveHomeP1a = 25;
columnsHomeownership = 33;
columnsRetirementR1a = 26;
columnsRetirementR1b = 31;
columnsHomeownership = 41;
columnsRetirementR1a = 33;
columnsRetirementR1b = 37;
columnsChildcareC1a = 37;
columnsChildcareC1b = 37;
columnsValidationStudentsByAge = 10;
Expand Down Expand Up @@ -1410,7 +1413,7 @@ else if (country.equals(Country.IT)) {
}

//Partnership - parameters for matching based on wage and age differential
meanCovarianceParametricMatching = ExcelAssistant.loadCoefficientMap(Parameters.getInputDirectory() + "scenario_parametricMatching.xlsx", countryString, 1, 1);
meanCovarianceParametricMatching = ExcelAssistant.loadCoefficientMap(Parameters.getInputDirectory() + "scenario_parametricMatching.xlsx", "Parameters", 1, 1);

//Fertility
if (country.equals(Country.UK)) {
Expand All @@ -1422,11 +1425,11 @@ else if (country.equals(Country.IT)) {
}

//Income
coeffCovarianceIncomeI3a = ExcelAssistant.loadCoefficientMap(Parameters.getInputDirectory() + "reg_income.xlsx", countryString + "_I3a", 1, columnsIncomeI3a);
coeffCovarianceIncomeI3b = ExcelAssistant.loadCoefficientMap(Parameters.getInputDirectory() + "reg_income.xlsx", countryString + "_I3b", 1, columnsIncomeI3b);
coeffCovarianceIncomeI4b = ExcelAssistant.loadCoefficientMap(Parameters.getInputDirectory() + "reg_income.xlsx", countryString + "_I4b", 1, columnsIncomeI4b);
coeffCovarianceIncomeI5a_selection = ExcelAssistant.loadCoefficientMap(Parameters.getInputDirectory() + "reg_income.xlsx", countryString + "_I5a", 1, columnsIncomeI5a);
coeffCovarianceIncomeI5b_amount = ExcelAssistant.loadCoefficientMap(Parameters.getInputDirectory() + "reg_income.xlsx", countryString + "_I5b", 1, columnsIncomeI5b);
coeffCovarianceIncomeI3a = ExcelAssistant.loadCoefficientMap(Parameters.getInputDirectory() + "reg_income.xlsx", countryString + "_I3a_amount", 1, columnsIncomeI3a);
coeffCovarianceIncomeI3b = ExcelAssistant.loadCoefficientMap(Parameters.getInputDirectory() + "reg_income.xlsx", countryString + "_I3b_amount", 1, columnsIncomeI3b);
coeffCovarianceIncomeI4b = ExcelAssistant.loadCoefficientMap(Parameters.getInputDirectory() + "reg_income.xlsx", countryString + "_I4b_amount", 1, columnsIncomeI4b);
coeffCovarianceIncomeI5a_selection = ExcelAssistant.loadCoefficientMap(Parameters.getInputDirectory() + "reg_income.xlsx", countryString + "_I5a_selection", 1, columnsIncomeI5a);
coeffCovarianceIncomeI5b_amount = ExcelAssistant.loadCoefficientMap(Parameters.getInputDirectory() + "reg_income.xlsx", countryString + "_I5a_amount", 1, columnsIncomeI5b);
coeffCovarianceIncomeI3a_selection = ExcelAssistant.loadCoefficientMap(Parameters.getInputDirectory() + "reg_income.xlsx", countryString + "_I3a_selection", 1, columnsIncomeI3a_selection);
coeffCovarianceIncomeI3b_selection = ExcelAssistant.loadCoefficientMap(Parameters.getInputDirectory() + "reg_income.xlsx", countryString + "_I3b_selection", 1, columnsIncomeI3b_selection);

Expand Down Expand Up @@ -1558,8 +1561,8 @@ else if (country.equals(Country.IT)) {
}

//Health
regHealthH1a = new OrderedRegression<>(RegressionType.GenOrderedLogit, Dhe.class, coeffCovarianceHealthH1a);
regHealthH1b = new OrderedRegression<>(RegressionType.GenOrderedLogit, Dhe.class, coeffCovarianceHealthH1b);
regHealthH1a = new GeneralisedOrderedRegression<>(RegressionType.GenOrderedLogit, Dhe.class, coeffCovarianceHealthH1a);
regHealthH1b = new GeneralisedOrderedRegression<>(RegressionType.GenOrderedLogit, Dhe.class, coeffCovarianceHealthH1b);
regHealthH2b = new BinomialRegression(RegressionType.Probit, Indicator.class, coeffCovarianceHealthH2b);

//Social care
Expand Down Expand Up @@ -1613,7 +1616,7 @@ else if (country.equals(Country.IT)) {

regEducationE1a = new BinomialRegression(RegressionType.Probit, Indicator.class, coeffCovarianceEducationE1a);
regEducationE1b = new BinomialRegression(RegressionType.Probit, Indicator.class, coeffCovarianceEducationE1b);
regEducationE2a = new OrderedRegression<>(RegressionType.GenOrderedLogit, Education.class, coeffCovarianceEducationE2a);
regEducationE2a = new GeneralisedOrderedRegression<>(RegressionType.GenOrderedLogit, Education.class, coeffCovarianceEducationE2a);

//Partnership
if (country.equals(Country.UK)) {
Expand Down Expand Up @@ -2076,8 +2079,8 @@ public static void setEmploymentsFurloughedFlex(MultiKeyCoefficientMap employmen
Parameters.employmentsFurloughedFlex = employmentsFurloughedFlex;
}

public static OrderedRegression getRegHealthH1a() { return regHealthH1a; }
public static OrderedRegression getRegHealthH1b() { return regHealthH1b; }
public static GeneralisedOrderedRegression getRegHealthH1a() { return regHealthH1a; }
public static GeneralisedOrderedRegression getRegHealthH1b() { return regHealthH1b; }
public static BinomialRegression getRegHealthH2b() { return regHealthH2b; }

public static BinomialRegression getRegReceiveCareS1a() { return regReceiveCareS1a; }
Expand Down Expand Up @@ -2127,7 +2130,7 @@ public static void setEmploymentsFurloughedFlex(MultiKeyCoefficientMap employmen

public static BinomialRegression getRegEducationE1a() {return regEducationE1a;}
public static BinomialRegression getRegEducationE1b() {return regEducationE1b;}
public static OrderedRegression getRegEducationE2a() {return regEducationE2a;}
public static GeneralisedOrderedRegression getRegEducationE2a() {return regEducationE2a;}

public static LinearRegression getRegEQ5D() { return regHealthEQ5D; };

Expand Down
6 changes: 3 additions & 3 deletions src/main/java/simpaths/data/RegressionName.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ public enum RegressionName {

EducationE1a(RegressionType.Probit),
EducationE1b(RegressionType.Probit),
EducationE2a(RegressionType.OrderedLogit),
EducationE2a(RegressionType.GenOrderedLogit),

FertilityF1a(RegressionType.Probit),
FertilityF1b(RegressionType.Probit),
Expand All @@ -18,8 +18,8 @@ public enum RegressionName {
PartnershipU1b(RegressionType.Probit),
PartnershipU2b(RegressionType.Probit),

HealthH1a(RegressionType.OrderedLogit),
HealthH1b(RegressionType.OrderedLogit),
HealthH1a(RegressionType.GenOrderedLogit),
HealthH1b(RegressionType.GenOrderedLogit),
HealthH2b(RegressionType.Probit),

HealthHM1Level(RegressionType.Linear),
Expand Down
Loading