Idea for RFC to represent task-task dependencies, between individual tasks of steps

[mwiebe]

Render farms often support dependencies between two steps that divide per frame instead of being across all the the frames together. Open Job Description could support this by extending the representation of a step dependency. This idea requires #79 to be able to express many patterns by writing expressions.

OpenCue supports a LayerOnAny mode where any of the frames of the dependency unblock a layer instead of a specific frame or the whole layer. This is the inspiration for adding anyOf and allOf optionally in the parameterSubspace. The default would be allOf behavior if the list is at the parameterSubspace level.

Here are a few examples to try showing how a general syntax could work for a variety of different cases.

Example: scene export -> frame render.

Suppose you have a scene in Windows-only software, that supports export to a renderer that runs on Linux. You can structure this as a two-step job. Generally, when a frame is finished exporting, it should render immediately, not wait until all the frames are exported.

...
parameterDefinitions:
  - name: Frames
    type: STRING
    default: 1-10

steps:
- name: ExportPerFrame
  parameterSpace:
    taskParameterDefinitions:
      - name: Frame
        type: INT
        range: "{{Param.Frames}}"
  script:
    ...
- name: RenderFrame
  dependencies:
    - dependsOn: ExportPerFrame
      # A task-task dependency specifies task parameters from the ExportPerFrame parameter space
      parameterSubspace:
        - name: Frame
          # Each Frame in RenderFrame depends on the same Frame in ExportPerFrame
          range: "{{Task.Param.Frame}}"
  parameterSpace:
    taskParameterDefinitions:
      - name: Frame
        type: INT
        range: "{{Param.Frames}}"
  script:
    ...

Example: simulation with uniform time steps -> render every Nth as an image

Suppose you have a simulation of a world defining its physics and perhaps agents within it. For each timestep, you take the state of the world at the previous timestep, and advance it. You can represent this as a chain of tasks, each one depending on the previous one. You could compose this with a per-timestep dependency

...
parameterDefinitions:
- name: FrameCount
  type: INT
  default: 100
  minValue: 1
- name: TimeStepsPerFrame
  type: INT
  default: 10
  minValue: 1

steps:
- name: SimulateTimeStep
  parameterSpace:
    taskParameterDefinitions:
      - name: TimeStepNumber
        type: INT
        # There are FrameCount * TimeStepsPerFrame total number of time steps
        range: "0-{{Param.FrameCount * TimeStepsPerFrame  - 1}}"
  dependencies:
  # This step depends on itself
  - dependsOn: SimulateTimeStep
    parameterSubspace:
      - name: StepNumber
        # Each task depends on the task for the previous TimeStep. Out of bounds values are discarded, so
        # TimeStepNumber 0 has no dependency.
        range: ["{{Task.Param.TimeStepNumber - 1}}"]
  script:
    ...
- name: RenderFrame
  dependencies:
    - dependsOn: SimulateTimeStep
      parameterSubspace:
        - name: TimeStepNumber
          # Each Frame in RenderFrame depends on the time step corresponding to the frame
          range: ["{{Task.Param.Frame * Param.TimeStepsPerFrame}}"]
  parameterSpace:
    taskParameterDefinitions:
      - name: Frame
        type: INT
        range: "0-{{Param.FrameCount - 1}}"
  script:
    ...

Example: Directed acyclic graph (DAG) uniform processing of a dataset.

Suppose you want to process your data set in a DAG, running the same processing at each node of that graph. You can structure this as a single-step job, with a self-dependency that holds a graph as a dependency list. There might be a better way to represent this, but this example needs an ARRAY[ARRAY[INT]] (or LIST[LIST[INT]], however #79 ends up) job parameter data type to accept the adjacency list as a parameter.

...
parameterDefinitions:
- name: GraphAdjList
  type: LIST[LIST[INT]]
  # Default adjacency list for this graph:
  # N0 -> N1 -> N4
  #   \-> N3   \
  #     \--------> N2
  default: [[], [0], [0, 1], [0], [1]]

steps:
- name: ProcessGraphNode
  parameterSpace:
    taskParameterDefinitions:
      - name: NodeIndex
        type: INT
        # There's one task for each graph node
        range: "0-{{len(Param.GraphAdjList)}}"
  dependencies:
  # This step depends on itself
  - dependsOn: SimulateStep
    parameterSubspace:
      - name: ProcessGraphNode
        # Each graph node depends on the list of nodes in its adjacency list entry
        range: "{{Param.GraphAdjList[Task.Param.NodeIndex]}}"
  script:
    ...

Example: scene where any frame creates sim cache -> render depends on the sim cache

...
parameterDefinitions:
- name: Frames
  type: STRING
  default: "1-100"
  minLength: 1

steps:
- name: RenderWithSim
  parameterSpace:
    taskParameterDefinitions:
      - name: Frame
        type: INT
        range: "{{Param.Frames}}"
  script:
    ...
- name: RenderNeedsSim
  dependencies:
    - dependsOn: RenderWithSim
      parameterSubspace:
        # Depend on any frame of the step RenderWithSim
        anyOf:
          - name: Frame
            range: ["{{Step.RenderWithSim.ParameterSpace.Frame.Range}}"]
  parameterSpace:
    taskParameterDefinitions:
      - name: Frame
        type: INT
        range: "{{Param.Frames}}"
  script:
    ...

[mwiebe]

In the course of creating AcademySoftwareFoundation/OpenCue#2180, I've taken a closer look at OpenCue's frame-level dependency modes. They are FrameByFrame, PreviousFrame, LayerOnSimFrame, and LayerOnAny. They're defined in the DependType class of pyoutline/outline/depend.py.

FrameByFrame

This is like the "scene export -> frame render" example.

LayerOnLayer

This is a dependency on another step, no per-task subspace.

PreviousFrame

This is like the "simulation with uniform time steps -> render every Nth as an image" example, for the SimulateTimeStep self-dependency or an analogous dependency on a different step.

LayerOnSimFrame

Behavior defined by pyoutline/outline/backend/cue.py function build_dependencies. It takes the first frame of the dependency layer, and makes every frame of the current layer depend on that frame.

My understanding for this is that the scene has a "cache sim" process implicit, that the first frame will perform, and you don't want a different layer to start until that is processed. Similar to LayerOnAny but just for the first frame.

This is like the "simulation with uniform time steps -> render every Nth as an image" example, with a different dependency structure for how RenderFrame depends on SimulateTimeStep.

- name: RenderFrame
  dependencies:
    - dependsOn: SimulateTimeStep
      dependsOnSubspace:
        - name: TimeStepNumber
          # Each Frame in RenderFrame depends on the first time step.
          # This could be `"{{Param.Frames[0]}}"` if the first frame is defined by a Frames range expression.
          range: [0]

LayerOnAny

Starts the layer when any frame of the dependency layer finishes. Similar to LayerOnSimFrame but for any frame not the first frame number.

Created the example to illustrate this: scene where any frame creates sim cache -> render depends on the sim cache