Inference time slow

[siddagra]

I just added a timer using time library and the inference speed is much worse than the one in the paper.

timer code:

t1 = time()
        for i in tqdm.tqdm(range(1, num_samples+1)):
            image_key = b'image-%09d' % i
            label_key = b'label-%09d' % i

            imgbuf = txn.get(image_key)
            buf = six.BytesIO()
            buf.write(imgbuf)
            buf.seek(0)

            img_pil = Image.open(buf).convert('RGB')
            img = np.array(img_pil)
            label = txn.get(label_key).decode()
            labels.append(label)
            img, la, width, img_ = data_preprocess(img, label, char2id, args)

            pred_value, attention_weights_value = sess.run([enc_pred, enc_attention_weights], feed_dict={input_images: [img], input_labels: [la]})

            pred_value_str = idx2label(pred_value, id2char, char2id)[0]

            predicts.append(pred_value_str)
            if args.vis_dir != None and args.vis_dir != "":
                os.makedirs(args.vis_dir, exist_ok=True)
                os.makedirs(os.path.join(args.vis_dir, "errors"), exist_ok=True)
                _ = heatmap_visualize(img_, attention_weights_value, pred_value_str, args.vis_dir, "{}.jpg".format(i))
                if pred_value_str.lower() != label.lower():
                    _ = heatmap_visualize(img_, attention_weights_value, pred_value_str,
                                       os.path.join(args.vis_dir, "errors"), "{}.jpg".format(i))
        metrics_value = calc_metrics(predicts, labels, args.metrics_type)
        print("Done, {}: {}".format(args.metrics_type, metrics_value))
        t2 = time()
        print("time taken:", t2-t1, "inference time (ms):", (t2-t1) / num_samples * 1000, "FPS:", 1/ ((t2-t1) / num_samples))

logs:

2022-07-14 01:25:41.839559: I tensorflow/stream_executor/cuda/cuda_dnn.cc:384] Loaded cuDNN version 8401
2022-07-14 01:25:41.840187: I tensorflow/stream_executor/cuda/cuda_blas.cc:1786] TensorFloat-32 will be used for the matrix multiplication. This will only be logged once.
100%|█████████████████████████████████████████| 79/79 [00:07<00:00, 10.18it/s]
Done, accuracy: 0.013
time taken: 7.767025709152222 inference time (ms): 98.31678112850913 FPS: 10.17120361876888

Is this suppose to happen? Any way to optimise inference time to be better? Let me know if there are any ways, mentioned in the paper or otherwise, for improving inference time of this model. Because as it stands, it seems to be very unideal in terms of accuracy/inference time tradeoff.

[siddagra]

Even after putting the timers on specifically the model inference:

            t1 = time()
            pred_value, attention_weights_value = sess.run([enc_pred, enc_attention_weights], feed_dict={input_images: [img], input_labels: [la]})
            t2 = time()
            print("inference time (ms):", (t2-t1) )

inference time (ms): 0.03200173377990723
inference time (ms): 0.03399825096130371
inference time (ms): 0.03300046920776367
inference time (ms): 0.03300309181213379
inference time (ms): 0.032997846603393555
inference time (ms): 0.0319981575012207
inference time (ms): 0.03499794006347656
inference time (ms): 0.0330042839050293
inference time (ms): 0.032996177673339844
inference time (ms): 0.033998727798461914
inference time (ms): 0.031998634338378906
inference time (ms): 0.03300333023071289
inference time (ms): 0.03300333023071289
inference time (ms): 0.03200268745422363
inference time (ms): 0.03600025177001953
inference time (ms): 0.03300023078918457
inference time (ms): 0.035999298095703125
inference time (ms): 0.03399920463562012
inference time (ms): 0.03299999237060547
inference time (ms): 0.03300333023071289
inference time (ms): 0.03400111198425293
inference time (ms): 0.033998727798461914
inference time (ms): 0.03399944305419922
inference time (ms): 0.03399991989135742
inference time (ms): 0.03299593925476074
inference time (ms): 0.03399944305419922
inference time (ms): 0.03399968147277832
inference time (ms): 0.03399658203125
inference time (ms): 0.033002376556396484
inference time (ms): 0.034000396728515625
inference time (ms): 0.031998395919799805
inference time (ms): 0.0330042839050293
inference time (ms): 0.03299975395202637
inference time (ms): 0.03200340270996094
inference time (ms): 0.03399968147277832
inference time (ms): 0.03300023078918457
inference time (ms): 0.03199911117553711
inference time (ms): 0.030997037887573242
inference time (ms): 0.03300142288208008
inference time (ms): 0.0319981575012207
inference time (ms): 0.03199911117553711
inference time (ms): 0.03300285339355469
inference time (ms): 0.03099966049194336
inference time (ms): 0.03399801254272461
inference time (ms): 0.03600621223449707
inference time (ms): 0.03200030326843262
inference time (ms): 0.033998966217041016
inference time (ms): 0.038999080657958984
inference time (ms): 0.03500175476074219
inference time (ms): 0.032999277114868164
inference time (ms): 0.03500080108642578
inference time (ms): 0.03399658203125
inference time (ms): 0.03400230407714844
inference time (ms): 0.03500247001647949
inference time (ms): 0.03299975395202637
inference time (ms): 0.03499937057495117
inference time (ms): 0.03399944305419922
inference time (ms): 0.03400301933288574
inference time (ms): 0.03399920463562012
inference time (ms): 0.03299903869628906
inference time (ms): 0.03499794006347656
inference time (ms): 0.03300166130065918
inference time (ms): 0.03300046920776367
inference time (ms): 0.03499913215637207
inference time (ms): 0.03500080108642578
inference time (ms): 0.034003496170043945
inference time (ms): 0.03399825096130371
inference time (ms): 0.03400063514709473
inference time (ms): 0.034000396728515625
inference time (ms): 0.034003496170043945
inference time (ms): 0.03499889373779297
inference time (ms): 0.03500199317932129
inference time (ms): 0.03399848937988281
inference time (ms): 0.03499865531921387
inference time (ms): 0.033997297286987305
inference time (ms): 0.03399777412414551

average inference time is still pretty low