Low fps on jetson nano

Hi. I am trying to run PatchNet on Jetson Nano. Measured fps is aroud 15-16(demo_skipframe.py) when number of skipframes is set to 1000(so only PatchNet is running). Gpu load is <=50%. What can be done?
Jetpack-4.6.1
CUDA-10.2
Python-3.6 
Torch-1.10
