SimLingo eval performance: ~0.05x realtime (visible/offscreen) on RTX 4060 Ti — is realtime possible?

[Akumar201]

Summary

I ran SimLingo Bench2Drive evaluation and observed very slow closed-loop performance (~0.05x realtime).
I want to report exact setup/results and ask whether realtime is expected/possible with this model/config.

Environment

• Repo: RenzKa/simlingo
• CARLA: 0.9.15
• Python env: simlingo (Python 3.8)
• GPU: NVIDIA GeForce RTX 4060 Ti (16 GB)
• OS: Linux (Ubuntu)

Evaluation command used

python -u ${WORK_DIR}/Bench2Drive/leaderboard/leaderboard/leaderboard_evaluator.py \
  --routes=${WORK_DIR}/leaderboard/data/bench2drive_split/bench2drive_32.xml \
  --repetitions=1 \
  --track=SENSORS \
  --checkpoint=./eval_result.json \
  --timeout=600 \
  --agent=${WORK_DIR}/team_code/agent_simlingo.py \
  --agent-config=${WORK_DIR}/models/simlingo/simlingo/checkpoints/epoch=013.ckpt/pytorch_model.pt \
  --traffic-manager-seed=1 \
  --port=2000 \
  --traffic-manager-port=8000

I tested both modes:

1. Visible simulation: with --existing-server + manually launched ./CarlaUE4.sh
2. No rendering window: without --existing-server (evaluator launches CARLA with -RenderOffScreen)

Observed performance

From evaluator logs (=== [Agent] ... Ratio = ...):

• Visible mode: roughly 0.045x to 0.048x
• No-render mode: roughly 0.053x to 0.065x

So no-render improves performance, but still far from realtime.

GPU utilization during run

nvidia-smi during evaluation:

• GPU utilization around 98%
• VRAM usage around 11.4 GB / 16.3 GB
• python process around 95% SM utilization
• CarlaUE4-Linux process also active on GPU

This suggests the run is GPU-bound and using CUDA properly (not CPU-only fallback).

Setup/debug notes (in case relevant)

• Initially checkpoint files were Git LFS pointers; fixed by pulling actual model files.
• I also had to ensure PYTHONPATH includes:
  • ${WORK_DIR}
  • ${WORK_DIR}/Bench2Drive/scenario_runner
  • ${WORK_DIR}/Bench2Drive/leaderboard
  • CARLA Python API paths

Question

• Is this performance expected for team_code/agent_simlingo.py + InternVL2-1B checkpoint on Bench2Drive closed-loop eval?
• Is realtime (or near realtime) possible in this setup?
• Are there recommended settings for a significantly faster eval (e.g., lower sensor resolution, specific flags, model variant, or benchmark settings)?

[johnreb20]

Hi,

I tried it on T4 without flash attention headless, I just getting 0.008x as the ratio!

Thanks,
John.

[rvxfahim]

rtx 4090, getting 0.032x

is this expected ?