Labour supply is currently producing strange estimates for employment transitions

[andrewbaxter439]

In previous meetings we’d noted a problem that employment rates were seen to be decreasing across the years. Additionally, transitions into and out of employment were much greater than seen in observational data. These anomalies didn’t appear in a previous version (tagged 2023.12.08) so it seems a change introduced in recent updates that has caused this.

I ran a git bisect to try and narrow this down between the 2023.12.08 tag and the main branch (where the issue was still presenting). I think it boils down to two changes made to the code at two time points.

Firstly, the labour supply estimates are updated in #59 (commit 94bd8ba). These seem to change both the transitions into and out of employment, so they’re different from previous versions (and expected trends), but not by much:

(Understanding Society data for comparison)

A second commit (1ec4f85) then seems to change things completely, making both employment transition estimates much larger:

This second one is labelled 'reformatted random draws to facilitate replication of simulations'. My only guess here is that somehow the random draws of labour supply choices are being changed in a way that makes a person more likely to enter/exit employment than before?

Suggestions of how to fix this are at the minute beyond me and I think would likely be two stages:

• re-estimation of labour supply (currently underway), with both cross-sectional and longitudinal (into/out of work) estimates to match
• checking over labour supply draws/probabilities to ensure these are working consistently

Can discuss at future meeting but hopefully this is a start. @justin-ven @pbronka @igelstorm @dkopasker - please do add thoughts and correct my (very likely) misunderstandings.

[dkopasker]

@andrewbaxter439 just for clarification, from the additional graphs that you have shown me, the issue of a declining employment rate over time (i.e. failure of cross-sectional validation) is first observed after the labour supply estimates are updated, and this is regardless of alignment. Is this correct?

[andrewbaxter439]

Hi @dkopasker - yeah the employment rates themselves going into declines seems to begin with the changing labour supply estimates. Haven't tested all with alignment yet, as alignment seems to make longitudinal transitions worse in an attempt to correct cross-sectional rates. Can re-run tests on the two pre-post comparisons for graphing employment rates themselves at some point. Figure that fixing labour supply choices for realistic transitions is a necessary step in any case!

[justin-ven]

Commit (1ec4f85) replaced a series of attributes describing random draws by a new Innovations object. This change was designed to ensure that all random draws for each individual in each period are updated in a systematic fashion - in the previous code, it was possible to update some random draws a variable number of times within a single period, or not update others, which generated variable results.

In the commit, BenefitUnit.java, original line 1346, randomDrawLabourSupply is only updated if the existing value is negative - it looks like this is just done once for each simulated benefit unit. In the revised code, this random draw is refreshed each period - this difference may be responsible for the altered behaviour.

I am not totally clear how the regression model for labour should work, and the respective role played by the variable randomDrawLabourSupply. If the random draw should be a unique lifetime characteristic of an individual (like a random preference parameter), then the above aspect of the commit needs to be amended to avoid updating the respective random draw each period. Otherwise, it is possible that the error that was in the original (pre-commit) code was masking some other problem.

It should be easy enough to check if this aspect of the model is responsible for the behaviour that is of concern: e.g. add a new BenefitUnit attribute, initialise this attribute with a random draw once, and then use the new attribute to set the value for labourInnov in the updateLabourSupplyAndIncome method of the existing BenefitUnit code.

[pbronka]

Hi All, sorry I wasn't able to join the meeting where this was discussed.

If the random draw should be a unique lifetime characteristic of an individual (like a random preference parameter), then the above aspect of the commit needs to be amended to avoid updating the respective random draw each period.

That was the idea to reduce the churn otherwise visible in @andrewbaxter439's graphs. There may be ways of implementing that which are more consistent with the theory, so perhaps it should remain an open question whether this is the best way of increasing the longitudinal stability of predictions.

I also became aware of another issue related to the updated labour supply estimates - the process runs in all years and controls for lag(1) of hours of work interacted with labour supply choices (L1_lhw...), but this variable is not properly initialised in the initial populations (there is currently no L1_lhw variable loaded in the UK). We can correct this fairly easily but probably only worth doing if this variable will be used by new labour supply estimates which are currently underway?

How to proceed depends on timelines, I think.

If we need to generate results immediately:

1. Keep the random draw fixed to avoid excessive churn.
2. Add L1_lhw to initial populations.

If we have some time:

1. Discuss whether keeping the random draw fixed is consistent with the theory and whether we can come up with a better solution.
2. Consider whether updated labour supply estimated will make use of L1_lhw; if not, do not add to the initial populations.

[justin-ven]

@pbronka raises two important points here:

1. Initialisation of lag variables
2. Random draws assumed for the labour process.

On (1), this has been a recurrent problem, which could be addressed by standardising initialisation before proceeding with each year's simulation. In that regard, it is somewhat similar to the motivation for introducing the Innovations object that I discuss above.

On (2), this seems like something that requires further investigation - probably just an hour or so, but we should clarify what is best supported by the theory before proceeding.

[justin-ven]

I have finished running two test simulations, and the quick summary is that I haven't found anything really concerning using the most recent version of the model.

Specifically, I did the following:

1. Started with (development) model version 6bb5102
2. Compiled new input data using most recent UKHLS
   • Required updating 00_master to allow for a new wave of the survey - I pushed these changes to github as pull request 168
3. Copied data from (2) to the model directory SimPaths\input\InitialPopulations
4. Ran UKMOD version B2025.05 using the UK_2019_b1.txt input data for system years 2011 to 2029
5. Copied the data from (4) to the model directory SimPaths\input\EUROMODoutput
6. Ran SimPathsStart and directed the model to "Change country...", "Load new input data for stating populations", "Load new input data fror tax and benefit systems", and "Select tax and benefit systems for analysis".
7. Ran SimPathsStart, forward from 2020 to 2030 for the base specification of the model.
8. Analysed output from (7) using attached Stata do file.
9. Editted the model:
   • Innovations.java adjusted to allow for public Innovations(int nDoubleInnovs, int nSingleDrawDoubleInnovs, long seed) constructor
   • BenefitUnit.java adjusted:
     • innovations = new Innovations(9, seed) > innovations = new Innovations(9, 1, seed);
     • double labourInnov = innovations.getDoubleDraw(5) > double labourInnov = innovations.getSingleDrawDoubleInnov(0)
10. Saved model and ran from 2019 to 2030 (2019 used here because running the model from 2020 only allows transitions to be reported from 2021, which I didn't appreciate at (7)).
11. Analysed output from (10) using attached Stata do file.

Results for the population aged 18 to 64 are reported in the attached Excel file and copied below. These results do not indicate the very high transition rates set out above.

It would be helpful if @andrewbaxter439 can confirm whether I have missed anything. If not, then I think the current statistics provide some reassurance concerning the most recent model specification.

test statistics.xlsx

employment analysis.txt

[andrewbaxter439]

Hi Justin,

Just back to desk after a week of a workshop, will take a look at this! It looks as if in these runs you're finding that the current setup (annual random draws) is providing a more stable rate of entering/exiting employment? That seems odd! Will attempt on a re-compiled population when I get round to it and let you know what I find.

One minor thing to clarify is that we calculated the proportions as "proportion of those who were employed at L1 leaving employment" and vice versa, rather than proportion of whole population which would be giving the higher numbers.

The EmploymentStatistics.java is aiming to do proportions as denominators of lagged subgroups:

https://github.com/centreformicrosimulation/SimPaths/blob/31c88f5d844aaff6c528a4842be9e272db32a75e/src/main/java/simpaths/data/statistics/EmploymentStatistics.java#L47-L56

The Stata code that @dkopasker used for calculating across survey/simpaths that I aimed to replicate:

* Stage 2: summarise SimPaths prediction

import delimited "`output'\`tag'\csv\Person.csv", clear 

rename time Year

* SimPaths outputs string variable for economic status
* Convert to numerical category
encode les_c4, gen(emp_stat)



xtset id_person Year

foreach year in 2012 2013 2014 2015 2016 2017 {
preserve
local lag = scalar(`year'-1)


* Task 1
* Summarise employment transitions by year
di in red "`lag'-`year'"
xttrans2 emp_stat if inrange(Year, `lag', `year'), prob matcell(p) 
* column is current status
* row is lagged status. Sums to 100%.

* Different value labels due to conversion from string
* employed to not employed = 1 to 2
* not employed to employed = 2 to 1

matrix list p

* save element of matrix
gen EmptoNotemp=p[1,2]
gen NotemptoEmp=p[2,1]

drop if Year!=`year'

collapse (mean) EmptoNotemp NotemptoEmp, by(Year)

[andrewbaxter439]

Hi Justin, have done one run following the same procedure as yourself and getting roughly similar results:

But in using the same data and the calculation method to compare with USoc observations (count becoming unemployed divided by count in employment at t-1; count becoming employed divided by count not in employment at t-1), these simulations are producing the similarly high rates from above (10-12% and 30-35%, rather than the expected 5-7% and 20-25%)

This seems to point to the high churn rates in/out of employment still persisting?

[andrewbaxter439]

Brief update, tried with alignEmployment = true and it seems to reverse the trends but still have them outside the expected ranges:

[andrewbaxter439]

Hi @justin-ven, tried out your suggested solution of switching it to lifetime draws (new Innovation constructor). This seems to have the effect of bringing the longitudinal transition estimates in line with the 11th June results - EmpToNotEmp about what was expected bot NotEmpToEmp a bit lower than expected.

Turning on alignment here again creates very high rates of entering unemployment and very low rates of entering employment:

So lifetime draws do seem to reverse some of the issue we had, and perhaps then labour supply estimates are the parts still effecting the entering employment stages?

[pbronka]

Hi @andrewbaxter439 , could you clarify how the employment alignment is configured in the above runs? By default, in the development branch of the model, it’s only active up to and including 2019 when enabled: https://github.com/centreformicrosimulation/SimPaths/blob/e1e3c79b65bc704053898b3061a557f693e669c5/src/main/java/simpaths/model/LabourMarket.java#L237

So the large effect it has on transition rates is surprising.

Have you looked into the variable initialisation? (#163 (comment))

Currently, if the lagged value is unavailable the model will use contemporaneous value in it's place, e.g.
https://github.com/centreformicrosimulation/SimPaths/blob/e1e3c79b65bc704053898b3061a557f693e669c5/src/main/java/simpaths/model/Person.java#L452-L456

https://github.com/centreformicrosimulation/SimPaths/blob/e1e3c79b65bc704053898b3061a557f693e669c5/src/main/java/simpaths/model/Person.java#L573

we can remove these parts of the code which will then result in a Runtime Exception being thrown:

https://github.com/centreformicrosimulation/SimPaths/blob/e1e3c79b65bc704053898b3061a557f693e669c5/src/main/java/simpaths/model/Person.java#L3539-L3543

So to avoid that we will need to initialise L1_lhw in the initial populations.

[pbronka]

Draft PR introducing L1_lhw to initial populations: #172