test: Regression tests show increased use of memory causing 1 additional experiment to OOM over last tests

[kmehant]

OOM experiments

epoch	framework_config	gradient_accumulation_steps	mem_nvidia_mem_reserved	model_name_or_path	num_gpus	per_device_train_batch_size	torch_dtype	train_loss	train_runtime	train_samples_per_second	train_steps_per_second	train_tokens_per_second
	none	16	78783.5	mistralai/Mixtral-8x7B-Instruct-v0.1	8	1	bfloat16

Failed experiments - number of gpus not divisible by ep degree

epoch	framework_config	gradient_accumulation_steps	mem_nvidia_mem_reserved	model_name_or_path	num_gpus	per_device_train_batch_size	torch_dtype	train_loss	train_runtime	train_samples_per_second	train_steps_per_second	train_tokens_per_second
	moe-scattermoe-granite-ep2	16	0	ibm-granite/granite-3.0-3b-a800m-instruct	1	8	bfloat16
	moe-scattermoe-granite-ep4	16	0	ibm-granite/granite-3.0-3b-a800m-instruct	1	8	bfloat16
	moe-scattermoe-granite-ep2-padding-free-foak	16	0	ibm-research/moe-7b-1b-active-shared-experts	1	8	bfloat16
	moe-scattermoe-granite-ep2-padding-free	16	0	ibm-granite/granite-3.0-3b-a800m-instruct	1	8	bfloat16
	moe-scattermoe-granite-ep2-padding-free-foak	16	0	ibm-granite/granite-3.0-3b-a800m-instruct	1	8	bfloat16
	moe-scattermoe-granite-ep4-padding-free	16	0	ibm-granite/granite-3.0-3b-a800m-instruct	1	8	bfloat16
	moe-scattermoe-granite-ep4-padding-free-foak	16	0	ibm-granite/granite-3.0-3b-a800m-instruct	1	8	bfloat16
	moe-scattermoe-granite-ep2	16	0	ibm-research/moe-7b-1b-active-shared-experts	1	8	bfloat16
	moe-scattermoe-granite-ep2-padding-free	16	0	ibm-research/moe-7b-1b-active-shared-experts	1	8	bfloat16
	moe-scattermoe-granite-ep4	8	2276	ibm-granite/granite-3.0-3b-a800m-instruct	2	8	bfloat16
	moe-scattermoe-granite-ep4-padding-free	8	2276	ibm-granite/granite-3.0-3b-a800m-instruct	2	8	bfloat16
	moe-scattermoe-granite-ep4-padding-free-foak	8	2277	ibm-granite/granite-3.0-3b-a800m-instruct	2	8	bfloat16

Failed experiments - number of experts not divisible by ep degree

epoch	framework_config	gradient_accumulation_steps	mem_nvidia_mem_reserved	model_name_or_path	num_gpus	per_device_train_batch_size	torch_dtype	train_loss	train_runtime	train_samples_per_second	train_steps_per_second	train_tokens_per_second
	moe-scattermoe-granite-ep4	16	0	ibm-research/moe-7b-1b-active-shared-experts	1	8	bfloat16
	moe-scattermoe-granite-ep4	8	2276	ibm-research/moe-7b-1b-active-shared-experts	2	8	bfloat16
	moe-scattermoe-granite-ep4	4	2564	ibm-research/moe-7b-1b-active-shared-experts	4	8	bfloat16
	moe-scattermoe-granite-ep4-padding-free	16	0	ibm-research/moe-7b-1b-active-shared-experts	1	8	bfloat16
	moe-scattermoe-granite-ep4-padding-free	8	2276	ibm-research/moe-7b-1b-active-shared-experts	2	8	bfloat16
	moe-scattermoe-granite-ep4-padding-free	4	2564	ibm-research/moe-7b-1b-active-shared-experts	4	8	bfloat16
	moe-scattermoe-granite-ep4-padding-free-foak	16	0	ibm-research/moe-7b-1b-active-shared-experts	1	8	bfloat16
	moe-scattermoe-granite-ep4-padding-free-foak	8	2277	ibm-research/moe-7b-1b-active-shared-experts	2	8	bfloat16
	moe-scattermoe-granite-ep4-padding-free-foak	4	2564.5	ibm-research/moe-7b-1b-active-shared-experts	4	8	bfloat16

Delta with previous experiments on OOM

epoch	framework_config	gradient_accumulation_steps	mem_nvidia_mem_reserved	model_name_or_path	num_gpus	per_device_train_batch_size	torch_dtype	train_loss	train_runtime	train_samples_per_second	train_steps_per_second	train_tokens_per_second
	none	16	78783.5	mistralai/Mixtral-8x7B-Instruct-v0.1	8	1	bfloat16

Regression test is done as part of the PR - #126. The change in metrics may be attributed to transformers==4.49 but needs investigation.

[fabianlim]

@kmehant are you saying that the experiments in your first OOM table, also OOM in the previous committed benchmark. And the single mixtral experiment in the second table, is the only one that OOMed now, but did not in the preivous committed benchmark?

[kmehant]

@fabianlim Yes @fabianlim I have checked.

[fabianlim]

@kmehant sorry im very confused you have this line here

epoch	framework_config	gradient_accumulation_steps	mem_nvidia_mem_reserved	model_name_or_path	num_gpus	per_device_train_batch_size	torch_dtype	train_loss	train_runtime	train_samples_per_second	train_steps_per_second	train_tokens_per_second
	none	128	81174	mistralai/Mixtral-8x7B-Instruct-v0.1	1	1	bfloat16

This is a single gpu run with ep8. But as I mentioned here #126 (comment) that you must follow the instructions how the benches are run.

# The mixtral bench must be run by a single command, and this is a 8 gpu run
tox ... \
    8 128 benchmark_outputs scenarios-moe.yaml accelerated-moe-full-mixtral

Hence we obviously dont have a bench for single gpu for mixtral, so you need to follow how the benches are run, and you should not be reporting or documenting extraneous benches as OOM, because they were not meant to be run that way.

I think now there is a lot of confusion, it has to be cleared up and I need to feel confident that the benches are properly updated, and the issues are properly documented, before I can merge. Otherwise down the road it will be alot of confusion

So orginally we had 21 lines of granite and 3 lines of mixtral. In your new file, you have

• 21 lines of ibm-granite/granite-3.0-3b-a800m-instruct
• 18 lines of ibm-research/moe-7b-1b-active-shared-experts <- but here you dont have any ep4 runs. It is very strange there is ep2 but no ep4, since the latter is even more memory efficient
• 2 lines of mixtra. <- for this I can accept that framework_config = None OOMs.

@willmj

[kmehant]

Thank you for being meticulous, I agree its important.

Let me jot down the details below

Mixtral bench results as shown below has run 6 experiments (3 - 1 gpu and 3 - 8 gpu). Its my mistake on reporting 1 GPU runs, so let me remove them from the tables in this message and the attached PR.

I hope this alleviates confusion on mixtral runs.

On the granite moe shared missing ep4 experiments. My bad I will remove them from the OOM table. They failed since number of experts are not divisible by ep degree.

[fabianlim]

@kmehant ok please update the tables and the descriptions of this issue and #126 @willmj

[kmehant]

@fabianlim @willmj Have updated the details.

[fabianlim]

@kmehant thanks, but can you check your OOM table again, you have entries like these

epoch	framework_config	gradient_accumulation_steps	mem_nvidia_mem_reserved	model_name_or_path	num_gpus	per_device_train_batch_size	torch_dtype	train_loss	train_runtime	train_samples_per_second	train_steps_per_second	train_tokens_per_second
	moe-scattermoe-granite-ep2	16	0	ibm-granite/granite-3.0-3b-a800m-instruct	1	8	bfloat16

This is a case where we are trying to do ep2 with only 1 gpu, so its probably throwing because of an inappropriate setting. The ep_degree must divide the number of gpus, and here its not the case

[kmehant]

@fabianlim made the updates, thank you.

[fabianlim]

@kmehant No its still not correct, you still three entries that are num_gpus = 2 but ep4 on your first table

[kmehant]

@fabianlim Catch!

I have updated it.