Hi Thomas! Thanks for your reply and input!
Due to i have a couple of years in experience in GUI and UX my thoughts are matching with your understanding and also the faced problems!
I see the basic problems in the inconsistence as well. IMHO this is related to the fact, that most of all open plattform concepts lacks in the adressing of an audience. As i have seen in OpenHAB, i see the efforts in many parts of UI and UX to split the interfaces in two basic groups of users: a) Programming based users and b) WYSIWYG users.
For those several UIs i think first there is the need to clarify the adressed users to specific groups. Like: users with low programming and tech-skills -> more WYSSIWIG and also more experienced Users -> programming mode & experts mode. I see already a lot of efforts in both ways and - personally for me now - i can handle openHAB quiet well.
And i also agree with you on the lack in guidance for Users. My magic (Buzz)words would be: Contextual menues and informations and structurization of the documentation/help.
The UI as understood is a great base for letting the users build on their individual system. Nethertheless here comes the basic problem with unxeperienced users (which are in general the majority of users): the more options - the more decisions are to make. Here i like the thought of concentration on two approaches with the existing UIs: having a) an expert mode where experienced users have control of all the system and b) a simple mode with step up form based set-up process and and eased UI with less options but lots of examples (like readymade panels for import) or set-ups (f.e. 80-90% of all beginners in SmartHome area starts with lightbulbs OR wall plugsā¦). Said short: The UX work i have in mind, is mostly related to create basic templates and advanced templates.
And i all in all agree with your opinion to narrow down UX improvments in general which have to be discussed.
To start the whole thing 
i would start to sketch a workframe for UX which names the different users (e.g. Expert, Beginner) with use cases for some first ideas to get closer to a structure. So widely known that is a UX Guideline / GUI Guideline. And due to it always a good idea for smarter systems: invite the audience here to work also on this, bring input. This is in general also the starting point for any UX projects i do in business! Having an roadmap (as detailed as possible) with a clearly defined vision and aim, its also easier to inspire and aboard devs and techies to a project (the vision is my only magic key for my 3 dev teams i develop software since yearsā¦:). even though, creating a script with a clear bearing for the project and having some visualizations helps a lot to avoid a lot of wasted time in experimenting with programmed prototypes (and anyway, nowadays there are tons of slick prototyping solutions like adobe XD to visualize ideas). So i would suggest to start with exactly this: Concetrate on a (communitywide open?) Paper with defines the vision and aims for openHAB clear and identifies lacks and difficulties for the different audiences.
I would really like to work with you and others on improvements in UX, GUI and also share knowledge. Morelike my questions is now (i think you know the openHAB community better) with witch development tools we could start?
btw. for personal further communication i am native german speaking as well 
I might also respond timeshifted due to i have currently two softwares in hot beta-states here and move around in europe a lot.
I wait to continue with you and others! Its a challenge, for sure - but the harder the challenge, the sweeter the berries you gain in the end.
I wish you a nice day!
Manuel
)