@steve1
Look e.g. at games. They do far more processing and run smoothly. I think if really the processing of layers, etc. is the bottleneck in OH, there shall be a lot of room for optimization.
But anyway. I want to thank everyone who tried to help. My issue more or less solved itself by upgrading to OH 1.8. I see a long delay for the first few switches, after that it seems to settle and the delay is more or less 200-300ms, which is perfectly acceptable for me.