How much GPU memory required to run live streaming demo?

[WangyiNTU]

Thanks for your wonderful framework. May I know how much GPU memory is required to run a live-streaming demo? Do you have a small pre-trained model for low-memory inferencing? Thanks.

[hyf015]

Hi, thank you for your interest in our work! If generation is not required, Vince needs 18G GPU memory. If the generation module is loaded, it will need extra 6G GPU memory, in total a little more than 24G.

In our experiments, we run Vinci for live-streaming demo on one 4090 GPU.

[WangyiNTU]

Hi, thank you for your interest in our work! If generation is not required, Vince needs 18G GPU memory. If the generation module is loaded, it will need extra 6G GPU memory, in total a little more than 24G.

In our experiments, we run Vinci for live-streaming demo on one 4090 GPU.

Thanks for your prompt reply. Is there an alternate solution if I want to run it on a GPU less than 18G? Like InternLM2.5-1.8B as fixed LLM?

[hyf015]

Thank you for your suggestion! In fact, we originally used smaller models but the performance did not seem optimal. We will add this option to our code soon, within this week. I will notify you in this thread once it is done.

[WangyiNTU]

@hyf015 There? In February, you mentioned that one smaller model would be provided in the code. As we plan to use your code in our research, can you please release that model? Many thanks.