I actually think I was wrong and the issue is not full srcset, but rather the combination of width / height attribute values and width / height gallery CSS and full srcset:
That’s why I guess setting width and height attributes with selected file size dimensions is better.
You force the file you need and you play with CSS sizes.
(See Image size (if srcset is removed or not) - #5 by yankiara)