Automated Profiling

[mxndrwgrdnr]

We need to generate detailed profiles of module runtimes and memory usage. ActivitySim already captures most of this information, if not all of it. We just need to establish a process to extract the data we want and store it in a centrally located place where we (Paul) can look at the profiles from successive runs in one place.

Some existing code to get started:

• memory profiling
  • https://github.com/ual/activitysim/blob/main/activitysim/core/mem.py and
  • https://github.com/ual/activitysim/blob/main/activitysim/core/mp_tasks.py#L737
  • https://github.com/ual/activitysim/blob/main/activitysim/core/mp_tasks.py#L1179
  • https://github.com/ual/activitysim/blob/main/activitysim/core/mp_tasks.py#L845-L1028 <-- probably more detail than we need.
  • https://github.com/ual/activitysim/blob/main/activitysim/core/chunk.py#L88-L141 <-- object (dataframe) size profiling
• runtime logging
  • https://github.com/ual/activitysim/blob/main/activitysim/core/pipeline.py#L472

Feel free to post any questions or issues here or on slack.