Decorator lifecycle hooks and common patterns

[saikonen]

Metaflow decorators have strict lifecycle hooks, but they do not impose guarantees on order between the execution of decorators. This means that any cross-reliance on hooks executing between decorators is unreliable, e.g. there is no guarantee that step_init of decorator A has executed before step_init of decorator B every time, therefore B can not depend on A during the same lifecycle stage.

There are also some commonly occurring patterns around runtime_step_cli and task_pre_step that could be abstracted away.

Goals

• Explore existing lifecycle hooks and determine if new ones should be added, or existing ones modified due to common patterns.
• Propose changes if any, with technical details and backwards compatibility as a first priority.
  • The way hooks are executed in order to achieve dependable ordering, if possible.
  • Abstractions for common patterns
  • Possible other findings

Instructions

This issue is meant to be a venue for asking further questions on the topic. Implementation details and proposals are to be kept separate.

When you have met the goals of the issue, it is time to show and tell.
You can do this for example with your own fork with an issue or pull request and messaging a mentor on Slack with a link.

[LuisJG8]

Hi @saikonen, I did some research and read the code in the codebase that is related to this issue (decorators.py, task.py, runtime.py etc). I think a good solution for the lifecycle hooks problem is to create a hook barrier. Essentially have a hook that is going to run after an earlier phase/stage finished for all decorators. In this way we can enforce an order based on the hook barrier and make sure the step_init of decorator A has executed before step_init of decorator B. What do you think about this approach?

Reference: https://docs.python.org/3/library/threading.html#threading.Barrier

Questions

• Do you think hook barriers could be a good way of having like a checkpoint where all decorators need to finish before they keep running other processes
• What did you meant by commonly occurring patterns around runtime_step_cli and task_pre_step
• When you said occurring patterns that could be abstracted way, did you meant that we could write like a function that we can call multiple times so that we do not repeat logic (DRY methodology)
• How would issues like this change with Python 3.13/3.14, which removed the GIL

[rawadhossain]

Hi @saikonen, I wrote up a detailed exploration and proposal here: Proposal Link
The proposal contains my findings and experiments, and I implemented a few changes on my fork while investigating:

• Introduced an optional ORDER_PRIORITY attribute to make decorator ordering deterministic when needed.
• Abstracted repeated metadata registration logic into a reusable _register_metadata helper.
• Shared lifecycle logic used by Batch and Kubernetes decorators into small helper methods.

I kept the details separate from the issue comment to avoid cluttering the discussion here.

A few questions:

• Does introducing an optional ORDER_PRIORITY align with the intended direction for handling decorator ordering, or would you prefer a different approach?
• Should decorators be able to rely on a specific execution order, or should they always remain independent of each other?

[rawadhossain]

@saikonen I opened the PR from my fork since master branch of this repository is currently behind Netflix/metaflow:master

[saikonen]

Heya! Sorry for the delay, here are some answers to the questions presented here.

Common patterns

One example of a common pattern is often binding the metadata to the decorator object in task_pre_step and relying on this in task_finished, which requires introducing guards for the case that task_pre_step fails, and self.metadata does not get set. One solution to this might be to introduce metadata as a default input to the methods requiring it.

There might be more, feel free to report any findings :)

Execution order of lifecycles

Basically the problem to tackle here is that of cross-decorator dependencies

• if deco_A.step_init depends on deco_B.step_init executing first, then there needs to be a mechanism to reliably check and wait for that to happen.
• What also needs to be handled is a dependency loop, where deco_B.step_init were to in turn, depend on deco_A.step_init. Such loops can never be resolved.

As decorators can be introduced as part of the core package, but also through extensions, we want to avoid any static order definitions (f.ex. ORDER_PRIORITY in the decorator itself)

On hook barries, I'm not sure if the exact structure lends itself to solve this as we're not dealing with concurrency issues, but something akin to it could work. Feel free to experiment :)

[rawadhossain]

Hi @saikonen, I’ve changed my approach and updated the PR to as per your feedback. Here's a summary of what changed:

PR: Link

1. Decorator ordering via DEPENDS_ON

   • Decorators can now declare DEPENDS_ON = ["other_decorator"] to specify ordering.
   • _sort_step_decorators builds a DAG, runs topological sort (Kahn’s algorithm) with source-order tie-break, and raises a MetaflowException if there’s a cycle.
   • Default DEPENDS_ON = [] means no constraints and all existing decorators are unaffected.

2. Explicit state in task_finished

   • Following your suggestion on the self.metadata pattern I audited all 7 plugin decorators.
   • Previously, task_pre_step was setting state on self (metadata, task_datastore, run_id, task_id) for use in task_finished.
   • Now these are passed directly from task.py as kwargs. So, no hasattr checks, and no risk if task_pre_step fails.

I left step_init → runtime_step_cli self-assignments intentionally since those hooks run in a guaranteed order and don’t have the same failure risk.