Simple and stable Inference APIs#4697

Open

YangFei1990 wants to merge 18 commits into

NVIDIA:mainfrom

YangFei1990:inference_apis

Commits on May 7, 2026

feat(inference): add ServeConfig and _EventLoopManager

YangFei1990
and
claude
committed
feat(inference): add coordinator runtime and _MegatronLLMBase

YangFei1990
and
claude
committed
feat(inference): add MegatronAsyncLLM, slim base class

YangFei1990
and
claude
committed
feat(inference): add MegatronLLM

YangFei1990
and
claude
committed
refactor(inference): drop model_name fields from ServeConfig

YangFei1990
and
claude
committed
feat(inference): add MegatronAsyncLLM.serve(), drop ServeConfig.role

YangFei1990
and
claude
committed
feat(inference): add offline_inference example, fix high-level API bugs

YangFei1990
and
claude
committed

Commits on May 8, 2026