Wondering why you use Ubuntu on ARM. It’s 64bit presumably which is consuming a lot more RAM than 32bits do. Consider using openHABian. It’ll work with RPi4 and eventually with your machine, too (your description reads as if you don’t dare to name that as that’s not a RPi).
OpenHABian comes with a serious set of optimizations on memory such as ZRAM and Java parameters.
You say you gave Java more mem but your problem description reads as if it’s Java garbage collection.
Did you increase -Xms ? -Xmx is the wrong one. And did you account for the additional mem 64bit requires?