This is a minor issue because with normal use of WordPress you won’t encounter it but maybe you want to take a look and decide how you want to deal with it. It is a bit difficult to explain with text so please see the video below.
no need to hurry. I just noticed this because of some experiments. In normal day use I would have never noticed since removing categories from posts is rather uncommon
The categories are shared with both post types. So lets say I have a category term “Fantasy” and there are Movies as well as Books with this term.
I have an archive with all books that the user can sort for category. Empty category terms are set to “hide”. This works as long as there is no other post type with the term.
So when there is no Book with the term “Fantasy” but there is a Movie with this term the term won’t hide on the Books list page. The count item element works though, meaning on the books list page the term “Fantasy” is disabled but not hidden.