Hi,
Very good thoughts here.
I very much agree that it is currently a pain to browse through components content and I personally wish it was more seamless like all other container type blocks with a permanent arrow to expand, which would remove the need to click on Modify component, be it for browsing component structure or editing parameters.
An additional global option could also be useful to prevent components from being expanded when expanding the whole structure, because one could need to expand the whole structure without seeing inside components which can be considered as more atomic elements, and only expand a component when needed.
As I’m still not very into them for different reasons, including this one, I don’t have a lot and I can’t yet speak about performances, but I agree this is an important point to keep in mind, as bloated pages can already slow down the editor a lot, particularly at load time.