Can it be any javascript file? Can I use node libraries? Can I use js libraries? Does it have to follow a specific format or is just like any other javascript program?
No, at least for now, it must be Nashorn JavaScript which is ECMAScript 5.1 which also has access to all of the standard Java classes should you need it.
The format does matter, as documented: JavaScript - Transformation Services | openHAB
You need to wrap it in a function call. input will be a String and what ever is returned needs to be a String or something that can be coerced to a String.
If you are doing something where you need node libraries and the like, you probably should be using a rule anyway. The JS Scripting automation add-on does support the use of some node libraries but there are limitations so not just any arbitrary one is guaranteed to work. Formatting there matters too.