They are using GraalVM. GraalVM is able to fully compile java bytecode ahead of time (AOT), so they basically start x86 binaries. Of course that leads to very good start up times. But I have read that GraalVM does not work well with OSGi which is used by openHAB.
And by speaking of OSGi. That existed way before Java9 jigsaw aka modules. So openHAB by itself is very modular actually. And it is a simple compile switch that converts an OSGi bundle into a java9 module (almost).
OSGi is a runtime modular system though. The advantage is that any binding can be thrown in at any time and the system will adapt. So we cannot simply link a static runtime and have a smaller memory footprint, because the next user will require exactly the method that you have removed. But openHAB already requires that you only use methods from the Compact Profile 2.