Skip to content

NaN Output When running on Ubuntu 16.04 #21

Description

@xupei0610

We just complied it as the instruction on Ubuntu 16.04, and run it with argument -arg_file= args/train_llc_args.txt. A NaN output will be got only after several iterations, as shown following.

`Actor Iter 4
Update Net 0:
I0808 12:46:07.273452 25515 solver.cpp:228] Iteration 10, loss = 0.0132303
I0808 12:46:07.273478 25515 solver.cpp:244] Train net output #0: loss = 0.0132303 (* 1 = 0.0132303 loss)
I0808 12:46:07.273486 25515 sgd_solver.cpp:106] Iteration 10, lr = 0.01
I0808 12:46:07.299680 25515 solver.cpp:228] Iteration 4, loss = inf
I0808 12:46:07.299707 25515 solver.cpp:244] Train net output #0: loss = inf (* 1 = inf loss)
I0808 12:46:07.299715 25515 sgd_solver.cpp:106] Iteration 4, lr = 0.001

Actor Iter 5
Update Net 0:
I0808 12:46:07.632088 25517 solver.cpp:228] Iteration 12, loss = 0.0219066
I0808 12:46:07.632109 25517 solver.cpp:244] Train net output #0: loss = 0.0219066 (* 1 = 0.0219066 loss)
I0808 12:46:07.632114 25517 sgd_solver.cpp:106] Iteration 12, lr = 0.01
I0808 12:46:07.651747 25517 solver.cpp:228] Iteration 5, loss = -nan
I0808 12:46:07.651769 25517 solver.cpp:244] Train net output #0: loss = -nan (* 1 = -nan loss)
I0808 12:46:07.651777 25517 sgd_solver.cpp:106] Iteration 5, lr = 0.001`

We tried it on different Linux machines. The output is similar. But on Windows 10, no NaN will be got.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions