Gradient accumulation causing different training curves

[ZhaofengWu]

For a pretraining experiment using the built-in Trainer, setting batch size 32 x 16 accumulation steps seems to yield a different training loss curve from 64 x 8. Even though they converge to approximately the same value, shouldn't the curves be exactly the same? What would cause the difference?

I've also seen similar things with using 1 vs. multiple GPUs (under DataParallel), which may or may not be relevant.

[gugarosa]

Theoretically speaking, it is expected because each configuration "differently" feed the gradients. Though is also expected that they converge to a common point.

[ZhaofengWu]

What do you mean by they differently feed the gradients? Calculating the loss based on 32 examples and then another 32 examples and then calculating the gradients should be exactly the same as calculating the loss based on 64 examples and then calculating the gradients, no?

[gugarosa]

They are very almost the same. Though, they are being repeatedly calculated and might lead to some numerical difference that explain the divergence at some points. The amount of updates that are performed when propagating the gradients, along with precision, might also been causing some influence.

If you exponentiate the loss, the differences at the end might be easier to be spotted.

[LysandreJik]

cc @sgugger

[sgugger]

I think @gugarosa analysis is correct in the sense that it could be explained by precision differences when computing the gradients vs accumulating partial gradients.

[ZhaofengWu]

Do you think that is suffcient to explain the not-too-minor difference in the plot?

[sgugger]

It's hard to say given you shared basically nothing on the training script :-)

[ZhaofengWu]

Ok, I think I'll take this explanation. Thanks! I'm not doing anything extraordinary in my training script and posting it in its entirety will probably bore everyone :)

[ZhaofengWu]

@sgugger I found one source of difference: the grouping by length done in LengthGroupedSampler depends on the batch size, so the data ordering would be different. I argue that ideally in all (batch size, gradient accumulation) settings, as long as the effective batch size is the same, they should have an identical training process, except for numerical issues.

[sgugger]

Good point, will try to see if there is a quick way to fix this!

[HuangChiEn]

@sgugger I found one source of difference: the grouping by length done in LengthGroupedSampler depends on the batch size, so the data ordering would be different. I argue that ideally in all (batch size, gradient accumulation) settings, as long as the effective batch size is the same, they should have an identical training process, except for numerical issues.

hello, i wonder why the different order for feeding data will make the gradient different ?