In the current implementation of metrics (#18), metrics are collected and aggregated from all tasks involved at the completion of a session.Run. This has a few drawbacks. First, all tasks may not be available at the end of session.Run (e.g., due to failed machines). Second, it ties stats collection to evaluation: we may in the future, persist task outputs to, e.g., s3, and clear task outputs from individual machines.
The solution to this is clear: we can flow Scopes along with task outputs; the executor can then aggregate them as the data flows.
In the current implementation of metrics (#18), metrics are collected and aggregated from all tasks involved at the completion of a
session.Run. This has a few drawbacks. First, all tasks may not be available at the end ofsession.Run(e.g., due to failed machines). Second, it ties stats collection to evaluation: we may in the future, persist task outputs to, e.g., s3, and clear task outputs from individual machines.The solution to this is clear: we can flow Scopes along with task outputs; the executor can then aggregate them as the data flows.