Enhancement: Dynamic Batching

[ryanznie]

• Added dynamic batching configs
• Minor: cleaned comments

[github-actions]

Coverage report

Click to see where and how coverage changed

File	Statements	Missing	Coverage	Coverage (new stmts)	Lines missing
scripts
setup_triton_repo.py					65-66, 82-90
Project Total

This report was generated by python-coverage-comment-action

[ryanznie]

@gemini-cli /review

[ryanznie]

@gemini-cli /review

[github-actions]

🤖 Hi @ryanznie, I've received your request, and I'm working on it now! You can track my progress in the logs for more details.

[github-actions]

📋 Review Summary

This pull request focuses on improving the Triton inference backend and its configuration. The changes include code cleanup by removing unnecessary comments and a more robust client implementation. The dynamic batching configuration for the Triton server has been refined, which should lead to better performance. Overall, the changes are positive and improve the quality of the codebase.

🔍 General Feedback

• The code is cleaner and more readable after the removal of commented-out code and explanatory comments.
• The Triton client is more robust as it now dynamically determines the output name instead of assuming a fixed name.
• The refined dynamic batching settings are a good step towards performance optimization.
• In src/inference.py, the _get_triton_datatype function has a fallback to "FP32" for unknown numpy dtypes. This could hide potential data type mismatches. It would be more robust to raise a TypeError for unsupported types to ensure that only expected data types are handled.