As you are the only one reporting this problem to date, at this point you are the expert on what does and does not take a long time to run.
There is a PR open that didn’t make it into OH 4 to allow publishing future values to persistence through the persistence API so this sort of work around should not be required in the future.
No doubt but you are also the only one who can generate this extreme difference in runtimes. As you saw, while my machine ran the rule more slowly than I would have expected, it’s certainly not so slow as to be a problem. I am running on a VM on an Intel processor but all things considered, it’s not 50 times the processing power of an RPi 4.
I don’t know what other loggers you could enable. But I do know that while the rule is running, the add-on and rule engine is blocked waiting for the rule to complete. I wouldn’t expect there to be any logging taking place after the rule is triggered so I’m not sure how much can be learned in this case from the logs.
Comparison simple math code between Python and JS. Without for loops, there was no difference or even a slightly better performance of JS. With for loops, JS was slightly slower, but that summed up to only around 400ms for a 10000 times loop, so usually nothing to worry about.
Then I analyzed my code of the respective rule which I had ported from Python to JS. Without any changes, it was 5s in Python vs. 30s in JS. The rule received PV forecast data from a HTTP request, then wrote them into InfluxDB using a for loop (144 times).
In each loop, I used 6x ZonedDateTime conversions which I eliminated. That gave 15s, so 15s left.
Then, each loop had a ZonedDateTime.parse of a JSONPATH transformation of a large string. I also eliminated that, then I was at 7s.
Each loop has a toEpoch conversion, I eliminated also that one. More or less no effect on performance.
Then, each loop had another two JSONPATH transformations of the same large string. I succeeded to eliminate those ones, too, and this is now done only two times before the loop. Now, I am at 4.6s vs Python’s 5s.
Conclusion: Those ZonedDateTime operations and the JSONPATH transformations have to be way more inefficient than those in Python. I don’t know what is the reason for that, but I can live with it, I am just forced to write much more efficient and optimized code.
You could experiment with using java.time.ZonedDateTime or even the native JS Date to get at the epoch.
JSON stands for “Java Script Object Notation”. It’s almost certainly going to be way more efficient to use JSON.parse(string) compared to the JSONPATH transformation. This is doubly true if you are using multiple JSONPATH operations on the same large JSON string. That will make it so the JSON only has to be parsed once instead of dozens or hundreds of times.
In general each programming language/environment is optimized for different things based on what sorts of problems they are aimed at solving. It’s not surprising that Python is better at some things compared to JS and I would expect JS to be better than Python at others. There’s also differences in the underlying interface between Java and Jython/JS where Jython uses JSR223 (or what it’s called now) whereas JS uses GraalVM. There might be differences there. Finally, JS Scripting uses a completely separate library for ZonedDateTime whereas Jython uses the Java Class which could account for more differences.
I want to extract only the pv_estimate values. I ended up in: JSON.parse(actions.Transformation.transform("JSONPATH", "$.forecasts[0:144].pv_estimate", dataNW));
I tried several things, but with JSON.parse(dataNW).forecasts[???].pv_estimate;
For reusability you should assign the result of the pars to a variable. Once parsed the JSON becomes a real JS Object so you can do stuff like looping and the like.
When in doubt, there is a utils.dumpObject() function in openhab-js that will show you what any given Object looks like.
The following to works for me:
var str = '{"forecasts":[{"pv_estimate":1.4225,"pv_estimate10":1.1069,"pv_estimate90":1.5612,"period_end":"2023-08-08T16:00:00.0000000Z","period":"PT30M"},{"pv_estimate":1.7246,"pv_estimate10":1.1804,"pv_estimate90":1.7979,"period_end":"2023-08-08T16:30:00.0000000Z","period":"PT30M"}]}';
var parsed = JSON.parse(str);
utils.dumpObject(parsed);
console.log(parsed.forecasts[0].pv_estimate);
One more question: The JS scripting uses a new implementation of the time namespace. Why is time.ZonedDateTime 5x slower than java.time.ZonedDateTime? 6s vs. 30s with these lines:
Because one of the driving principals for the development of openhab-js is to provide a native JavaScript interface for everything. java.time.ZonedDateTime is a Java class, not a JavaScript class. Therefore openhab-js imports and uses the joda-js for time.
On one project I worked on I implemented a custom logger that explicitly didn’t use the date time classes built into Java for the timestamps because that was too slow.
Why is it slower? I didn’t implement it and don’t have enough to compare between the two implementation. But I can say that back when I was a pretty heavy Java developer, the two slowest operations you could do were 1. building Strings using + and 2. anything to do with date times. I suspect over the years some really smart people have managed to add optimizations to the Java libraries and the JVM itself to improve the performance of using and manipulating date times. I doubt the same can be said for joda-js.
In addition, unlike the Java byte code which is already half compiled, and even Jython which is compiled after being loaded, the JS code is interpreted at runtime every time. The JS is always going to be slower.
Maybe, as an explanation: Why am I asking all that?
I have a couple of quite usual rules like - let’s say - everybody who does a little bit of smart home stuff like timer-based control of shutter, heating thermostats, ventilation control, smart metering, and so on. For all that, I can port my rules from Jython to Javascript, maybe I will have to do optimizations as described for this PV forecast stuff, but performance will be ok, I think. And in fact I already did a couple of successful ports.
But there is one Jython rule that is really complicated and performance-sensitive. It is an optimization algorithm to determine the best EV charging strategy based on planned drives and PV forecast, in order to optimize charging cost. It is using “large” matrices. I realized that one in Jython (and I am really not a professional programmer, and far away from being able to write efficient code from scratch) and I am trying now to determine which scripting language is suited best. I wouldn’t be so excited to spend the effort of days to port it, just to understand that performance is insufficient. In Jython, this rule is running every 30min, and takes about 1-3 minutes, depending on the forecast period between a few hours and three days. So here, I cannot spend a factor of 5 more calculating time. Most important operations are accessing single elements of the current vector of the matrix, and copying that to the next one. It is rather not classic matrix operations. At least that is how I implemented the algorithm right now.
So far, I could imagine JRuby to be as performant as Jython (from my simple tests). As I don’t use much date time operations in loops, I could also imagine JS, but I jave some doubts. What do you think?
It might be an easier path to move this to an external Python script either called by OH using the exec binding or executeCommandLine from a rule, or as a stand alone service that runs independently and pushes data to OH over the REST API or MQTT.
I don’t really have a lot of experience comparing the runtimes of the various rules languages. Based on what I know about them I’d expect Java and Jython to be close to tied for the fastest, jRuby to be next and JS to be slowest.
But even at 1-3 minutes I think you’ll come close to running into problems as I believe OH 4 will start to timeout the rule before that amount of time. IIRC it will kill any rule that takes longer than 5 minutes.
How big are these matrices? It might be worth while taking your coding skills to the next level and see if there are easy things you can do to make it more efficient. For example, simple things like taking expensive operations out of loops (e.g. calls to the transform action to parse out a value) can probably shave minutes off of the runtime even for the Jython implementation and possibly make it within reach of JS.
Even with matrices of hundreds of thousands of elements shouldn’t take 3 minutes to process.
Don’t have the code here in vacation, but I think the two matrixes are SOC-Steps x time steps, which is ~150 x max. 144. Those are filled column by column, and during that process lines are removed or new lines are added based on existing ones. So lot of operations just deleting or copying elements back and forth.
It could really be that just the handling of those operations (allocating memory for the whole maximum matrix vs. dynamically increasing or decreading the matrix, how to mark lines as deleted, how to copy lines and columns) is very inefficient.
I don’t think I have any transform or DateTime operations inside the loops. Time is an integer, SOC is an integer. But lots of nested loops.
Certainly, that is a good proposal to optimize the code. But will take quite some hours to get back into the code and understand it well enough to optimize it. Good thing is that the Jython rule is running stable since a year, so I can always check and benchmark against Jython results and performance.
Those sorts of situations are almost always more efficient when you you allocate the space needed for the matrix and just move stuff around in the already allocated memory. Creating and deleting Objects is expensive.
That’s how all the scientific computing stuff does it. Change values, don’t delete values. It doesn’t take anything at all to change a value in memory. It takes a whole lot more to delete allocated memory and create a new Object.
The caching is there to speed up the library injection/evaluation the first time a script is run. When you create a script, is is evaluated once and the following runs do not need to re-evaluate the script code.
We’ve already talked about this in the issue, I neither can reproduce nor do I understand it. As Rich said: Rule Builder uses JSRule under the hood. When I think back to the time I used a Pi 3 for openHAB 3.2, when JS Scripting was relatively new, I already had a several file-based rules with loops, all ram immediately. (I moved to a VM on my NAS for faster network, safer storage and easier backups, and x86 also has a few benefits sometimes.)
As Rich already said, using JSONPATH in JS is not needed. JSON is a data format that comes from JS (JavaScript Object Notation), just use JSON.parse(string) on your JSON and access properties the JS way.
ZonedDateTime in JS is based on JS-Joda to provide the user a pure JS environment without Java type pollution, but (as Rich also said) the Jython helper library uses Java‘s ZonedDateTime. Given that the Java ZonedDateTime is part of the JDK, I would expect it to be highly performance optimized (at least today) — JS-Joda in contrast is a JS port of java.time and I would not think it is that optimized. If you need max. performance, either use Java‘s ZonedDateTime or probably the native JS Date.
IIRC, GraalJS compiles the JS source code to some byte code as well. However the best performance is only reached if you run GraalJS on GraalVM instead of any OpenJDK build.
Unfortunately, the item receives a float number (1.0, 2.0, …) as command. Then the sitemap mapping does not work any more. If I change the sitemap mapping to 1.0=…, 2.0=…, the mapping works for the item states set by the Javascript rule, but is not compatible any more to other Jython rules.
Is there a way to send integer values to the item in Javascript? I would really find it “unclean” to do everything with floats. I know that Javascript number variables are always float, but in the OpenHAB world there is still a difference betwenn integer and float.
First, you are not commanding the day to become Tuesday, you are merely recording that it is. An update is the more appropriate way to update this Item.
Have you tried converting it to a String like you did in Jython?
That did the trick. Thanks.
Well, of course as always, you are right. I could also use only strings. I use these items to store regular weekly “events” and I sort them to get the upcoming events of the next three days. This information is used in a sitemap and also in rules. I think I chose numbers because of the handling of the periodicity in rules. And other reason is that I am coming rather from the microcontroller world and I am somehow trained to store information in an efficient way. But I understand that this is different in the Java, Javascript and openHAB world.
By the way, I have succeeded now to port that most complex rule (running an optimization algorithm) from Jython to Javascript. It is not using much ZonedDateTime stuff, so no problems there. I even refactored the array handling and used “inactivating” objects instead of array.splice() and array.push(). That did not improve performance, probably because the loop that is searching for “free” object spaces is too time-consuming.
Anyway, I am now at 6min JS runtime vs. 2min Jython runtime. I will now set up a completely fresh openhabian installation because there were already some strange things happening (e.g. that issue with rule builder triggers). If the performance difference is still there on the fresh setup, I will open up a new discussion about performance because the discussion here becomes more and more off-topic.
Apparently, I never entirely understood the philosophy behind “update” and “command”. I am using the difference at some points in order to distinguish between user set and otherwise changed items.
In this particular case, I understand it as a command because I am telling the optimization rule that there is a regular drive that needs e.g. 40% of my EV battery e.g. on every Monday starting at 7am.
