Generally depending on an external language that is implemented within Java is not ideal, as typically the implementation can’t be maintained to catch up with with the official native implementation. This is the case of Jython; it will be forever stuck at version 2.7. GraalVM is the solution, but it will take a while.
In OH, the official Java is in API. My guess is that the REST API is meant to be the official non-Java API initially. Then we have the various languages implemented with the JVM and can access the Java APIs directly (rather than going through the REST API). One thing to note is that all these scripting environments have a thin layer that wrap the interaction with the Java API. As such, if you structure your rules in such away that they stay purely within that language (e.g JS or Python), then it is less painful for you to have to adapt to changes later. This can be done by introducing an additional layer of abstraction.
I understand you’re not interested in HABapps, but I think this post I created couple days ago is still somewhat informative, as I did come from the Jython side.