error during training

[CalvinLi01]

024-04-15 12:11:22.419625: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:533] layout failed: Invalid argument: MutableGraphView::SortTopologically error: detected edge(s) creating cycle(s) {'new_train/person_pred/encoder_grid_class_1/while/Switch_4' -> 'new_train/person_pred/encoder_grid_class_1/while/encoder_grid_class_1/enc_grid_1/concat', 'new_train/person_pred/encoder_grid_class_1/while/Switch_4-1-0-TransposeNCHWToNHWC-LayoutOptimizer' -> 'new_train/person_pred/encoder_grid_class_1/while/Select_2', 'new_train/person_pred/encoder_grid_class_1/while/Switch_3' -> 'new_train/person_pred/encoder_grid_class_1/while/encoder_grid_class_1/enc_grid_1/mul', 'new_train/person_pred/encoder_grid_class_1/while/Switch_3-1-1-TransposeNCHWToNHWC-LayoutOptimizer' -> 'new_train/person_pred/encoder_grid_class_1/while/Select_1', 'new_train/person_pred/encoder_grid_class_0/while/Switch_4' -> 'new_train/person_pred/encoder_grid_class_0/while/encoder_grid_class_0/enc_grid_0/concat', 'new_train/person_pred/encoder_grid_class_0/while/Switch_4-1-0-TransposeNCHWToNHWC-LayoutOptimizer' -> 'new_train/person_pred/encoder_grid_class_0/while/Select_2', 'new_train/person_pred/encoder_grid_class_0/while/Switch_3' -> 'new_train/person_pred/encoder_grid_class_0/while/encoder_grid_class_0/enc_grid_0/mul', 'new_train/person_pred/encoder_grid_class_0/while/Switch_3-1-1-TransposeNCHWToNHWC-LayoutOptimizer' -> 'new_train/person_pred/encoder_grid_class_0/while/Select_1', 'gradients/AddN_35-0-TransposeNHWCToNCHW-LayoutOptimizer' -> 'gradients/AddN_35', 'gradients/AddN_28-0-TransposeNHWCToNCHW-LayoutOptimizer' -> 'gradients/AddN_28', 'gradients/AddN_45-0-TransposeNHWCToNCHW-LayoutOptimizer' -> 'gradients/AddN_45', 'gradients/AddN_20-0-TransposeNHWCToNCHW-LayoutOptimizer' -> 'gradients/AddN_20', 'gradients/AddN_33-0-TransposeNHWCToNCHW-LayoutOptimizer' -> 'gradients/AddN_33', 'gradients/AddN_25-0-TransposeNHWCToNCHW-LayoutOptimizer' -> 'gradients/AddN_25', 'gradients/AddN_44-0-TransposeNHWCToNCHW-LayoutOptimizer' -> 'gradients/AddN_44', 'gradients/AddN_18-0-TransposeNHWCToNCHW-LayoutOptimizer' -> 'gradients/AddN_18', 'gradients/AddN_43-0-TransposeNHWCToNCHW-LayoutOptimizer' -> 'gradients/AddN_43', 'gradients/AddN_39-0-TransposeNHWCToNCHW-LayoutOptimizer' -> 'gradients/AddN_39', 'gradients/AddN_51-0-TransposeNHWCToNCHW-LayoutOptimizer' -> 'gradients/AddN_51', 'gradients/AddN_37-0-TransposeNHWCToNCHW-LayoutOptimizer' -> 'gradients/AddN_37', 'gradients/AddN_37-3-TransposeNHWCToNCHW-LayoutOptimizer' -> 'gradients/AddN_37', 'gradients/AddN_37-2-TransposeNHWCToNCHW-LayoutOptimizer' -> 'gradients/AddN_37', 'gradients/AddN_42-0-TransposeNHWCToNCHW-LayoutOptimizer' -> 'gradients/AddN_42', 'gradients/AddN_38-0-TransposeNHWCToNCHW-LayoutOptimizer' -> 'gradients/AddN_38', 'gradients/AddN_50-0-TransposeNHWCToNCHW-LayoutOptimizer' -> 'gradients/AddN_50', 'gradients/AddN_36-0-TransposeNHWCToNCHW-LayoutOptimizer' -> 'gradients/AddN_36', 'gradients/AddN_36-3-TransposeNHWCToNCHW-LayoutOptimizer' -> 'gradients/AddN_36', 'gradients/AddN_36-2-TransposeNHWCToNCHW-LayoutOptimizer' -> 'gradients/AddN_36', 'new_train/person_pred/encoder_grid_reg_1/while/NextIteration_4-0-0-TransposeNCHWToNHWC-LayoutOptimizer' -> 'new_train/person_pred/encoder_grid_reg_1/while/Merge_4', 'new_train/person_pred/encoder_grid_reg_1/while/Switch_3' -> 'new_train/person_pred/encoder_grid_reg_1/while/encoder_grid_reg_1/enc_grid_regress_1/mul', 'new_train/person_pred/encoder_grid_reg_1/while/Switch_3-1-1-TransposeNCHWToNHWC-LayoutOptimizer' -> 'new_train/person_pred/encoder_grid_reg_1/while/Select_1', 'new_train/person_pred/encoder_grid_reg_0/while/NextIteration_4-0-0-TransposeNCHWToNHWC-LayoutOptimizer' -> 'new_train/person_pred/encoder_grid_reg_0/while/Merge_4', 'new_train/person_pred/encoder_grid_reg_0/while/Switch_3' -> 'new_train/person_pred/encoder_grid_reg_0/while/encoder_grid_reg_0/enc_grid_regress_0/mul', 'new_train/person_pred/encoder_grid_reg_0/while/Switch_3-1-1-TransposeNCHWToNHWC-LayoutOptimizer' -> 'new_train/person_pred/encoder_grid_reg_0/while/Select_1'}.

[JunweiLiang]

Can you post the exact command you used? And are you using the right tensorflow version (v1.15)?

[CalvinLi01]

python code/train.py actev_preprocess multiverse-models new_train/ --wd 0.001 --runId 0 --obs_len 8 --pred_len 12 --emb_size 32 --enc_hidden_size 256 --dec_hidden_size 256 --activation_func tanh --keep_prob 0.8 --num_epochs 80 --batch_size 20 --init_lr 0.3 --use_gnn --use_scene --learning_rate_decay 0.95 --num_epoch_per_decay 2.0 --grid_loss_weight 1.0 --grid_reg_loss_weight 0.2 --save_period 2000 --scene_h 36 --scene_w 64 --scene_conv_kernel 3 --scene_conv_dim 64 --scene_grid_strides 2,4 --use_grids 1,1 --val_grid_num 0 --train_w_onehot --gpuid 0

the tensorflow version is correct.
this error also happen when I do the muti-future inference but it seems that this error doesn't stop the code running but the progress of train will not be displayed on the progress bar