Add probability of re-sampling random draw from labour transitions to stabilise employment transitions

[andrewbaxter439]

From Justin's email:

a quick work-around might involve the following.

1. Allow for two time (t) specific random draws: x1(t) and x2(t).
2. At the beginning of life, the random draw used to identify labour transitions is set xl(t) = x1(t)
3. In each subsequent year, x2(t) is compared against an exogenous probability p;
   a. If x2(t) < p, then xl(t) = x1(t)
   b. Otherwise, xl(t) = xl(t-1).

Attempt as a first-stage fix of #163

[andrewbaxter439]

Update from 03/06 meeting

• seems to be doing the right thing (image below), with transition in/out rates more normalised
• next steps: different probabilities for transitions in/out of employment?

[andrewbaxter439]

Update on progress

The process of adding a probability of taking a new labourInnov each year seems to be working in some sense to reduce the probability of entering/exiting employment. Setting this to 0% would mean that all benefit units in every year take a new innovation value, which in turn is more likely to change their employment status. 100% means no benefit unit takes a new value - probabilities of selections from labour supply choices remain stable (and random draws within those stable probabilities?). The transition rates for entering employment come down into the expected range quickly; exiting employment seems further off (though 2019 is a strange anomaly?):

However, in latest checks this is still producing changing slopes in employment/unemployment, with higher persistence probabilities actually performing worse:

This seems to not be fixing the problem?

Alternative solution

I reverted the labour supply estimates to the pre-June ones (changed cd31a06). This seems to give almost bang-on transition estimates when persistence = 100% (and 2019 still looks weird?):

And very stable cross-sectional rates at any persistence setting:

This overall seems to suggest that:

• old labour supply estimates are sufficient for the time being to give stable employment rates and rational transition rates.
• the persistence probability mechanism may be redundant if 100% seems defensible. But can keep it in in this pull request for later flexibility?

Does this seem a workable change to have stable employment estimates in the develop branch which we can further look through in adding validation statistics @matteorichiardi @justin-ven @dkopasker? I would anticipate that @ANBurdett's model would be giving fresh estimates to replace these stand-ins, and these values would provide validators for checking.

[justin-ven]

Given the time constraint Andy, I'd work with whatever seemed to best address your needs at present. That said, not sure what others think about the question of what estimates to use more generally. I'd be inclined to use the newer estimates, and work toward a model that we have confidence in.

[andrewbaxter439]

• turn off labour supply in 1st year of run
• add alignment up to 2023/24
• start a few years earlier

[matteorichiardi]

A few updates and recommendations:

1. I have spoken with @dariaple - the new labour supply estimates will be ready towards the end of the month.
2. The new estimates will not be drastically different from the most recent ones we have produced, but will contain some further refinement.
3. If the model is to be used before the new estimates become available, either the June 2024 or the most recent estimates can be used.
4. In both cases and going forward, as we have discussed on our meeting of 4/6/2025 the recommendation is to have two separate probabilities to determine whether a new innovation or an old innovation needs to be used in the choice process, one for employed individuals and one for those not employed.
5. With these two extra parameters we should be able to match the two transition targets (employed-to-non-employed and non-employed-to-employed) exactly.
6. As discussed in the June 4 meeting, these two parameters are not to be regarded as ad hoc adjustments (aka "cheating"), but as proxies of a more accurate model where individual effects are fully considered.
7. In that "ideal" individual effect model, the disturbances would be composed of two parts, d_{i,t} = e_i + u_{i,t}, where d_{i,t} is the overall disturbance, e_i is the individual (time persistent) effect and u_{i,t} is the individual- and time-specific innovation, with two distinct distributions to be estimated in the data.
8. The suggested proxy involves using d_{i,t} = u_{i,t-1} + u_{i,t}, and blanking out one of the two components in each period.
9. An alternative - more closely related to the "ideal" individual effect model - would be to use e_i + u_{i,t} rather than u_{i,t-1} + u_{i,t}, while still blanking out one of the two components in each period. This would imply reverting to an individual effect drawn once for all when the individual is initialised, whenever a new innovation is not selected.
10. This however would imply much less persistency in the individual disturbances d_{i,t}, as the innovations u_{i,t} are uncorrelated with the individual effects e_i, differently from the individual effect model where considering both terms introduces correlation between d_{i,t} and d_{i,t-1}.
11. For this reason, I would recommend using the approach discussed on the June 4 meeting (point 8 above).
12. We will start looking for ways of estimating the labour supply model with individual effects over the summer.

For housekeeping, may I also suggest to @andrewbaxter439 to consolidate this issue with issue #163.

[andrewbaxter439]

Thanks for update Matteo. With regards point 4 - having two different probabilities for unemployed and employed individuals - my understanding from the code below is that labour supply choices are made at benefit unit level, with pairs of labour supply hours grouped by probability:

https://github.com/centreformicrosimulation/SimPaths/blob/e1e3c79b65bc704053898b3061a557f693e669c5/src/main/java/simpaths/model/BenefitUnit.java#L1075-L1083

I'm not clear on how I should detect then which of two potential probabilities to apply, basing on one or both benefit unit members' previous employment?

[matteorichiardi]

Good point. For hh where there is only one labour-supply flexible individual (adult children, singles, couples where one of the partners is not at risk of working) that would not be a problem. For couples where both partners are LS-flexible, I would follow the head of the BU (the one with higher income).

…

On Thu, 12 Jun 2025, 15:15 Andrew Baxter, ***@***.***> wrote: *andrewbaxter439* left a comment (simpaths/SimPaths#176) <#176 (comment)> Thanks for update Matteo. With regards point 4 - having two different probabilities for unemployed and employed individuals - my understanding from the code below is that labour supply choices are made at benefit unit level, with pairs of labour supply hours grouped by probability: https://github.com/centreformicrosimulation/SimPaths/blob/e1e3c79b65bc704053898b3061a557f693e669c5/src/main/java/simpaths/model/BenefitUnit.java#L1075-L1083 I'm not clear on how I should detect then which of two potential probabilities to apply, basing on one or both benefit unit members' previous employment? — Reply to this email directly, view it on GitHub <#176 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACOK4OB46CSUQ4WN3A3YPOT3DGDRHAVCNFSM6AAAAAB6ADBEYCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDSNRXGAYTAMBUGU> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[andrewbaxter439]

Update

I've updated with Daria's October estimates for labour supply and with the varying probabilities in #221. I experimented with these probabilities and it seemed that somewhere between 0.8 and 0.9 for persistence of innovation value for employed individuals was stable and around 0.5 fitted for unemployment:

Running with 0.85 and 0.5 gives a seemingly stable rate of entering/exiting employment:

Cross-sectional estimates of proportions in employment and out of employment (calculated in line with validation statistics from initial populations) is a little off but at least stable:

Does this seem to be approaching a useable set of estimates and calibration statistics for labour supply going forward?