How to send a HEX command on the serial port

That’s no answer, there’s more than one way to do that. We can send characters “1” “0” “H” “space” easily enough.

But you are probably looking to send binary. Binary is the keyword. Then it’s all about how to encode binary in openHAB, which is unicode.

There’s some detail about handling bytes as unicode in this post

which in your case would be something like

var testoutput = '\u0010\u0001\u00FF\u00FF ...'
mySerialItem.sendCommand(testoutput)