chore: nightly sync main into dev (08_05_2026) by svcnvidia-nemo-ci · Pull Request #4708 · NVIDIA/Megatron-LM

svcnvidia-nemo-ci · 2026-05-08T21:39:03Z

Summary

Automated nightly sync of main → dev for 08_05_2026.

Commits synced from main: 116
Files changed: 482
Total lines: +68,337 / -87,809
Python lines: +35,086 / -10,517 across 266 files

Files where main's version was taken (intent-driven resolutions)

The merge used -X theirs to favor main on conflicts. The following are
notable conflict resolutions:

Files deleted from main (kept deleted)

megatron/legacy/model/__init__.py — main removed in remove legacy tranformer and modules #4207 ("remove legacy transformer and modules")
megatron/legacy/model/transformer.py — same removal commit
tools/checkpoint/loader_legacy.py — same removal commit
tools/checkpoint/loader_llama_mistral.py — same removal commit

Files dev-only and intentionally kept removed

.github/workflows/multi-approval-bot.yml — dev removed in ci: Remove multi-approval action from dev branch #3576 ("Remove multi-approval action from dev branch"); main still has it but dev's intent is clear

Files restored from dev (NOT taken from main)

Per the sync skill, these are kept at dev's version:

pyproject.toml — preserves dev-only dependencies (fast-hadamard-transform, dev's pinned nvidia-resiliency-ext rev)
uv.lock — kept consistent with dev's pyproject.toml
docker/Dockerfile.ci.dev — kept consistent with dev's dependency-management triple
.github/CODEOWNERS — never touched by sync bot

No new git sources from main needed to be added (dev already has all required packages).

API Mismatches Detected and Fixed

HybridCPDataLoaderWrapper: main's training.py import line for this
class was merged in (line 203) but the class no longer exists in dev's
data_schedule.py (renamed/replaced by wrap_data_iterator API in dev's
[Dev] feat: Dynamic CP (part 2) rewrite). Removed the dead import line.
No call sites to fix — dev's wrap_data_iterator is already used
throughout merged training.py.

Pre-push Invariants

✅ .github/CODEOWNERS unchanged from origin/dev
✅ pyproject.toml, uv.lock, docker/Dockerfile.ci.dev unchanged from origin/dev
✅ All 250 changed Python files pass black==24.4.2 --skip-magic-trailing-comma --skip-string-normalization --check
✅ All 250 changed Python files pass isort==5.13.2 --check
✅ 90 changed megatron/core/ files: pylint==3.2.6 rated 10.00/10
✅ All changed files: ruff==0.9.10 check passed

Remerge-diff (conflict resolutions only)

76 files had non-trivial merge conflicts resolved by -X theirs. Below is
the file-level stat from git show --remerge-diff --stat HEAD. The full
diff is omitted from this PR body for brevity (it is ~12,000 lines); use
git show --remerge-diff HEAD locally to inspect.

Conflict resolution stats (76 files)

 .github/workflows/cicd-main.yml                    |    5 -
 .github/workflows/multi-approval-bot.yml           |   74 -
 docker/Dockerfile.ci.dev                           |    4 -
 docs/conf.py                                       |   18 +-
 .../detxoify_lm/generate_samples_gpt.py            |   76 +-
 .../gpt/gpt_dynamic_inference_with_coordinator.py  |    6 +-
 examples/mimo/train.py                             |    6 +-
 examples/multimodal/layer_specs.py                 |    2 +-
 examples/multimodal/model.py                       |   85 +-
 examples/post_training/modelopt/convert_model.py   |   19 +-
 examples/post_training/modelopt/export.py          |    5 +-
 examples/post_training/modelopt/finetune.py        |   67 +-
 examples/post_training/modelopt/generate.py        |   27 +-
 examples/post_training/modelopt/mmlu.py            |   45 +-
 .../modelopt/offline_feature_extract.py            |   56 +-
 examples/post_training/modelopt/prune.py           |   13 +-
 examples/post_training/modelopt/quantize.py        |   55 +-
 examples/post_training/modelopt/validate.py        |   32 +-
 gpt_builders.py                                    |   77 +-
 hybrid_builders.py                                 |    4 +-
 megatron/core/datasets/readme.md                   |   64 -
 megatron/core/distributed/param_and_grad_buffer.py |    4 -
 megatron/core/transformer/mlp.py                   |    4 -
 megatron/core/transformer/moe/fused_a2a.py         |   13 -
 megatron/core/transformer/moe/moe_layer.py         |    8 -
 megatron/core/transformer/moe/token_dispatcher.py  |    4 -
 megatron/core/transformer/transformer_config.py    |   27 -
 megatron/core/transformer/transformer_layer.py     |   13 -
 megatron/elastification/arguments.py               |    6 +-
 megatron/elastification/flextron_utils.py          |   11 +-
 megatron/elastification/pretrain_hybrid_flex.py    |  136 +-
 .../elastification/router/hybrid_flex_router.py    |    7 +-
 megatron/legacy/model/__init__.py                  |    6 -
 megatron/legacy/model/transformer.py               | 1645 --------------------
 megatron/post_training/arguments.py                |    7 +-
 megatron/post_training/model_builder.py            |   55 +-
 megatron/training/activation_logging.py            |   37 +-
 megatron/training/argument_utils.py                |   90 +-
 megatron/training/arguments.py                     |  589 +------
 megatron/training/async_utils.py                   |    4 +-
 megatron/training/checkpointing.py                 |   33 +-
 megatron/training/config/__init__.py               |   27 +-
 megatron/training/config/container.py              |   40 +-
 megatron/training/config/instantiate_utils.py      |   46 +-
 megatron/training/config/training_config.py        |   24 +-
 megatron/training/config/utils.py                  |   13 +-
 megatron/training/config/yaml_utils.py             |   10 +-
 megatron/training/datasets/data_samplers.py        |    9 +-
 megatron/training/gpu_sniff_test.py                |   81 +-
 megatron/training/training.py                      |  153 +-
 megatron/training/utils.py                         |    4 -
 model_provider.py                                  |   12 +-
 pretrain_bert.py                                   |   32 +-
 pretrain_gpt.py                                    |   42 +-
 pretrain_hybrid.py                                 |   65 +-
 pretrain_mamba.py                                  |  363 -----
 pretrain_t5.py                                     |    2 +-
 pretrain_vlm.py                                    |   10 +-
 pyproject.toml                                     |   23 +-
 .../golden_values_dev_dgx_h100.json                |    8 -
 .../golden_values_dev_dgx_h100.json                |    8 -
 .../unit_tests/fusions/test_mla_yarn_rope_apply.py |   10 -
 tests/unit_tests/models/test_hybrid_moe_model.py   |   16 -
 tools/checkpoint/checkpoint_inspector.py           |    9 +-
 tools/checkpoint/convert.py                        |   62 +-
 tools/checkpoint/dist_checkpoint_io.py             |   45 +-
 tools/checkpoint/gpt_hybrid_conversion.py          |  171 +-
 tools/checkpoint/loader_legacy.py                  |  416 -----
 tools/checkpoint/loader_llama_mistral.py           |  751 ---------
 tools/checkpoint/loader_mixtral_hf.py              |   12 +-
 tools/checkpoint/remap_gpt_dsa_to_mamba.py         |    5 -
 tools/prepare_cache.py                             |    9 +-
 tools/preprocess_data.py                           |  217 ++-
 tools/preprocess_mmdata.py                         |  160 +-
 train_rl.py                                        |   20 +-
 uv.lock                                            | 1182 ++------------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Xin Yao <xiny@nvidia.com>

Co-authored-by: john2 <john2@jrlogin01.jureca>

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

Co-authored-by: root <root@nvl72098-T17.cm.cluster> Co-authored-by: William Dykas <wdykas@oci-hsg-cs-001-vscode-03.cm.cluster> Co-authored-by: root <root@nvl72160-T13.cm.cluster>

…classmethod (#3812) Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>

Signed-off-by: oliver könig <okoenig@nvidia.com>

#4403) Co-authored-by: Antoni-Joan Solergibert <asolergibert@nvidia.com>

Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com>

Signed-off-by: Maanu Grover <maanug@nvidia.com> Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

Signed-off-by: dimapihtar <dpykhtar@nvidia.com>

Co-authored-by: Siddharth Singh <sidsingh@nvidia.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…nference cuda graph scope for hybrid models (#4440)

…ss curve gaps for latent MoE models (#4433) Signed-off-by: root <jiemingz@nvidia.com>

…4158) Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…4422) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Signed-off-by: rprenger <rprenger@nvidia.com>

Signed-off-by: qiyuw <qiyuw@nvidia.com> Co-authored-by: Antoni-Joan Solergibert <asolergibert@nvidia.com>

Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com>

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

Signed-off-by: oliver könig <okoenig@nvidia.com>

…q_len (#4094)

copy-pr-bot · 2026-05-08T21:39:07Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

svcnvidia-nemo-ci · 2026-05-08T21:39:21Z

/ok to test d268c6f

svcnvidia-nemo-ci · 2026-05-08T22:56:22Z

/ok to test c776aad

svcnvidia-nemo-ci · 2026-05-08T23:35:28Z

/ok to test 9d98047

svcnvidia-nemo-ci · 2026-05-09T00:04:37Z

/ok to test dcf9563

svcnvidia-nemo-ci · 2026-05-09T01:32:14Z

/ok to test dcf9563

svcnvidia-nemo-ci · 2026-05-09T01:49:32Z

/ok to test 81ec280

svcnvidia-nemo-ci · 2026-05-09T03:18:09Z

/ok to test 6a17219

svcnvidia-nemo-ci · 2026-05-09T21:13:14Z

Superseded by today's nightly sync.

minitu and others added 30 commits April 22, 2026 18:02

Fix nvtx_decorator to check _nvtx_enabled at call time (#4184)

9a3c927

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Co-authored-by: Xin Yao <xiny@nvidia.com>

fix merges_file typo in megatron_hf_tokenizer (#4392)

60f71e1

Co-authored-by: john2 <john2@jrlogin01.jureca>

Enable NullTokenizer for pretraining to reduce I/O access (#4057)

c9dfe34

docs: Add SECURITY.md (#4431)

7073492

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

Mamba inference opt (#4414)

40627d0

Co-authored-by: root <root@nvl72098-T17.cm.cluster> Co-authored-by: William Dykas <wdykas@oci-hsg-cs-001-vscode-03.cm.cluster> Co-authored-by: root <root@nvl72160-T13.cm.cluster>

DDP refactoring: Extract parameter layout computation into optimizer …

55b8111

…classmethod (#3812) Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

Update PR template with explicit request for issue (#4409)

90e09b6

Misc inference fixes (#4397)

ab2b33d

Rename Mamba to Hybrid outside megatron/core (#4159)

60408d5

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Include mtp layers in token per expert logging (#4412)

a52014c

fix: NVRx async compatibility and defer resiliency import (#4420)

32275b2

Co-authored-by: Dmytro Pykhtar <37850217+dimapihtar@users.noreply.github.com>

ci: add base_sha to codecov/codecov-action upload step (#4445)

9bb35a8

Signed-off-by: oliver könig <okoenig@nvidia.com>

Update copy-pr-bot.yaml [skip ci]

3034d86

fix(checkpoint_inspector): allow empty --param-to-param-group-map-json (

f78ed05

#4403) Co-authored-by: Antoni-Joan Solergibert <asolergibert@nvidia.com>

Add the YARN support for hybrid_model (#4244)

4d6cdd5

Co-authored-by: Philip Petrakian <ppetrakian@nvidia.com>

[training migration] Add container class for config dataclasses (#4227)

41ffa83

Signed-off-by: Maanu Grover <maanug@nvidia.com> Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>

Inference: Fix broken functional tests on gitlab (#4454)

a1165fa

SafeUnpickler class for safe pickle usage (#4319)

d4cacef

Signed-off-by: dimapihtar <dpykhtar@nvidia.com>

get rid of weights_only=False (#4434)

109feda

Signed-off-by: dimapihtar <dpykhtar@nvidia.com>

Inference | Per-block MoE routing storage for prefix caching (#4301)

64870c1

Co-authored-by: Siddharth Singh <sidsingh@nvidia.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Add troubleshooting tip for 'access forbidden' (#4449)

017e684

Fix checkpoint loading with rerun state machine (#4448)

3d7bcd3

Add misc CUDA graph sugar to CudaGraphManager (#4425)

9b02206

Inference: Add the embedding and output layer in the full_iteration_i…

35f76df

…nference cuda graph scope for hybrid models (#4440)

Important bugfixes in local CG implementation that were leading to lo…

481efd0

…ss curve gaps for latent MoE models (#4433) Signed-off-by: root <jiemingz@nvidia.com>

fix: Replace polynomial rolling hash with SHA-256 for prefix caching (#…

e9abb6c

…4158) Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

feat(ckpt): expose validate_access_integrity knob on dist-ckpt load (#…

377af02

…4422) Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Fix multivalidation (#3388)

241a5ca

Signed-off-by: rprenger <rprenger@nvidia.com>

Add missing knob for reduce_scatter_with_fp32_accumulation (#4410)

f2dcd42

Signed-off-by: qiyuw <qiyuw@nvidia.com> Co-authored-by: Antoni-Joan Solergibert <asolergibert@nvidia.com>

Enable CUDA graphs for MTP inference (#4260)

03f4111

Signed-off-by: Keshav Santhanam <ksanthanam@nvidia.com>

deepakn94 and others added 7 commits May 8, 2026 03:55

Add periodic GPU sniff tests to detect hardware stragglers (#4662)

932d9ee

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

ci: Bump GHA versions (#4606)

8079f37

Signed-off-by: Charlie Truong <chtruong@nvidia.com>

build: widen flashinfer-python pin to <0.7.0 (#4700)

a7bd379

Signed-off-by: oliver könig <okoenig@nvidia.com>

Add Python-side guardrail for HybridEP InfiniBand limit and rename se…

a08e259

…q_len (#4094)

Switch oncall (#4702)

dfd4a1a

Update golden values for various functional tests (#4703)

4933b24

chore: nightly sync main into dev (08_05_2026)

d268c6f

svcnvidia-nemo-ci added Run functional tests Run MBridge tests Attach this for testing this PR against MBridge main labels May 8, 2026

copy-pr-bot Bot temporarily deployed to test May 8, 2026 21:40 Inactive

copy-pr-bot Bot temporarily deployed to test May 8, 2026 22:57 Inactive

svcnvidia-nemo-ci force-pushed the main2dev/08_05_2026 branch from c776aad to 9d98047 Compare May 8, 2026 23:35

copy-pr-bot Bot temporarily deployed to test May 8, 2026 23:36 Inactive

svcnvidia-nemo-ci force-pushed the main2dev/08_05_2026 branch from 9d98047 to dcf9563 Compare May 9, 2026 00:04

copy-pr-bot Bot temporarily deployed to test May 9, 2026 00:06 Inactive

svcnvidia-nemo-ci force-pushed the main2dev/08_05_2026 branch from dcf9563 to 81ec280 Compare May 9, 2026 01:49

copy-pr-bot Bot temporarily deployed to test May 9, 2026 01:50 Inactive

fix: post-CI corrections

6a17219

svcnvidia-nemo-ci force-pushed the main2dev/08_05_2026 branch from 81ec280 to 6a17219 Compare May 9, 2026 03:18

copy-pr-bot Bot temporarily deployed to test May 9, 2026 03:19 Inactive

svcnvidia-nemo-ci closed this May 9, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: nightly sync main into dev (08_05_2026)#4708

chore: nightly sync main into dev (08_05_2026)#4708
svcnvidia-nemo-ci wants to merge 118 commits into
devfrom
main2dev/08_05_2026

svcnvidia-nemo-ci commented May 8, 2026

Uh oh!

copy-pr-bot Bot commented May 8, 2026

Uh oh!

svcnvidia-nemo-ci commented May 8, 2026

Uh oh!

svcnvidia-nemo-ci commented May 8, 2026

Uh oh!

svcnvidia-nemo-ci commented May 8, 2026

Uh oh!

svcnvidia-nemo-ci commented May 9, 2026

Uh oh!

svcnvidia-nemo-ci commented May 9, 2026

Uh oh!

svcnvidia-nemo-ci commented May 9, 2026

Uh oh!

svcnvidia-nemo-ci commented May 9, 2026

Uh oh!

svcnvidia-nemo-ci commented May 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Conversation

svcnvidia-nemo-ci commented May 8, 2026

Summary

Files where main's version was taken (intent-driven resolutions)

Files deleted from main (kept deleted)

Files dev-only and intentionally kept removed

Files restored from dev (NOT taken from main)

API Mismatches Detected and Fixed

Pre-push Invariants

Remerge-diff (conflict resolutions only)

Uh oh!

copy-pr-bot Bot commented May 8, 2026

Uh oh!

svcnvidia-nemo-ci commented May 8, 2026

Uh oh!

svcnvidia-nemo-ci commented May 8, 2026

Uh oh!

svcnvidia-nemo-ci commented May 8, 2026

Uh oh!

svcnvidia-nemo-ci commented May 9, 2026

Uh oh!

svcnvidia-nemo-ci commented May 9, 2026

Uh oh!

svcnvidia-nemo-ci commented May 9, 2026

Uh oh!

svcnvidia-nemo-ci commented May 9, 2026

Uh oh!

svcnvidia-nemo-ci commented May 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants