Skip to content

very slow with tensorflow2.17.0 and upper #20

Description

@alexandrecuer

solution is described here
tensorflow/tensorflow#67033

with this kind of warning

WARNING:tensorflow:5 out of the last 5 calls to <function TensorFlowTrainer.make_train_function.<locals>.one_step_on_iterator at 0x796ef9fc58a0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to 
(1) creating @tf.function repeatedly in a loop, 
(2) passing tensors with different shapes, 
(3) passing Python objects instead of tensors. 

For (1), please define your @tf.function outside of the loop. 
For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. 
For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for  more details.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions