Struggling with the concept of "Groups" - limitations or misunderstanding?

The real usefulness of groups is as you already said not in Sitemap as they don’t give you much control but it is in rules and persistence. You can do many things with the availability of the group based rule triggers and so on.
@rlkoshak has few great design patterns to give you some inspiration for use cases of groups.

Best regards Johannes