Xtend Scripts vs JSR223?

I’ve got a rule working in Jython which is nice. Its faster than the equivalent rule. So I might try it a bit more here and there.

I currently have two thousand lines of rules written with Xtend and feel like I’ve given it a good try but I think there are some areas around groups and map reduce jobs I could utilise a bit more.

I have pretty much all the various home automation systems so my system is a way of trying to integrate an make everything run opaquely with the same behaviour. I’m still trying to figure out or come up with a proper architecture before I start reworking it all. But I’m really don’t understand Xtend.

@rlkoshak I think my issue with lambdas was it felt tacked on since they aren’t proper functions and kind of something else complately but can be made to look if you squint enough like a function.

I think the bugs I was encountering is know, something about not being about to use vars. The advice was to use a map reduce job or something.

I think my main issue is it’s not clear if the DSL is suppose to just make everything much simpler for beginners and just do basic stuff, which is fine and I should just jump over to Jython. Or if it’s just a different way of thinking about rules and is actually much more powerful when you get the hang of it. They are two completely different positions and I’m not really sure what people are saying about Xtend.

Lambdas are a function that is also an Object. If they feel tacked on it is because they kind of are tacked on by Java and Xtend runs on top of Java and inherits them from Java. But the design of Xtend is highly dependent on lambdas. Just about every method call one makes on collections requires a lambda as an argument.

Where we run into some awkwardness is the Rules DSL, which is built on Xtend, was not created with lambdas in mind as something that would be used independently from Rules. So I think some design decisions were made early on that imposed a couple of limitations.

A lambda is just an object. To make a lambda accessible to more than one Rule it needs to be created as a global variable. But the Rules DSL imposes some significant restrictions on global variables:

  • global variables can only be accessed by Rules in the same file
  • global variables cannot see each other
  • for some reason, lambdas only support up to seven arguments, which is imposed by Xtend and I don’t know why except that it might be a limitation of Java (i.e Java doesn’t support functions with variable arguments so Xtend had to draw the line somewhere).

The other awkwardness is how lambdas are defined. And just to make sure that I am not lying I went to the Xtend docs and I’ve discovered we’ve been doing it wrong all this time.

This may be the result of a recent change. First of all I discovered we no longer need to import Functions or Procedures. Second the warning has gone away when you don’t define everything in the < >. so we can define the following lambda:

import org.eclipse.xtext.xbase.lib.Functions

val Functions$Function1<HttpsURLConnection, String> readResponse = [ connection |
    // a bunch of code, last line returns a String
]

can be defined as:

val readResponse = [ HttpsURLConnection connection |
    // a bunch of code, last line returns a String
]

Or if you want to be a bit more explicit

val (HttpsURLConnection)=>String readResponse = [ connection |
    // a bunch of code, last line returns a String
]

If you want a Procedure, the last line need only return a void.

So while I will agree that lambdas feel somewhat like an outlier in Rules DSL, they are not. And if someone felt strongly enough about it, one should probably write a PR to provide some other mechanism to make real functions. Though I should say, this weirdness is not a problem with the Experimental Rules Engine which is designed to allow Rules to call each other and support libraries (no more copy and paste from DP postings and Examples on the Forum).

That isn’t a bug. That is just how it is designed to work. I suspect you wanted to reference a variable inside a forEach. The problem is you cannot reference non-final variables inside a lambda. I remember reading a big long thread at some point explaining why that is the case but I don’t remember the specifics. But it was a deliberate design choice.

Part of it was to remember, the stuff between [ | ] is an Object. It is instantiated and lives in memory. You can add it to a Collection, pass it to other lambdas or in method calls and such. So, when this new Object comes into existence, it is given a copy of the stack up to that point. But how should it actually deal with variables on that stack?

Take Timers as an example. A Rule runs, creates a lambda Object and that Object is added to the collection the Quartz timer owns to run later. When the timer does go off and the lambda is executed, the instance of the Rule that created it is long since gone. Thus it is meaningless to modify a variable that was defined outside the lambda. That variable no longer exists. But you can access final variables (i.e. vals) because you know that nothing outside the lambda can change it (nothing can change it, that’s what final means) so you can make a copy of it to give to the lambda.

But that does become inconvenient when you are wanting to loop through a List and sum the values or something. But the proper way to do that is a map/reduce anyway.

I don’t know what “people” are saying about the Rules DSL but what I say is that both are powerful and equally capable in the home automation space. Indeed the Rules DSL is simplified in a lot of ways. No regular functions, no classes, no structs, etc which do help non-programmers get started much more quickly. But I’ve found that trading some lines in my .items files I can write Rules code that is every bit as capable and as powerful in as few number of lines as I can in Python. For example: OAuth2 using just OH Rules and myopenhab.org compared to this Python example OAuth2 Python Example · reddit-archive/reddit Wiki · GitHub.

And honestly, this example is way more complex than 99% of the Rules DSL code most users, advanced or not, would ever need to write and all of it is dealing with making custom HTTP requests with my own header values which the sendHttp* Actions don’t support. Most stuff like this would be half the number of lines.

I’ve had a whole lot of success in finding code efficient (i.e. fewer lines of easier to understand code to do the same thing) ways to solve lots of user’s problems. If you have a particularly messy set of code you want to see what is possible with Rules DSL to make it less complex I’m willing to take a look.

I am usually able to find ways to reduce the LOC count by about 50%.

1 Like

Ooh, how about this. I was trying to figure out how to simply change the colour of any lights that are on when a scene changes. I’m a bit tired and probably just missing something obvious but maybe some help in the approach here.

Group gRoom
Group gLivingroom (gRoom)
Group gBedroom (gRoom)
Color LivingroomLight (gLivingroom, gColor)
Color LivingroomProxy (gLivingroom, gProxy)
Color BedroomLight (gBedroom, gColor)
Color BedroomProxy (gBedroom, gProxy)
.....

I want a rule that updates any lights that is currently on. The update should be via the proxy.


Scene.postUpdate("Morning")

rule "Update lights for scene"
when Item Scene changed
then
     gColor.members.filter[ f | (f.state as HSBType).brightness as DecimalType).intValue > 0

Then turn on the proxy light in the same room group. 

end

I kind of want to get a list of items that are in the two groups. e.g. if the light in the bedroom is on, then I want to get a list of proxy items that are in the same room (group) as a light that is on. It all feels like it should be very simple and it’s probably just because I’m unfamiliar with it but I seem to be going in circles.

I’ll take a look at this tomorrow morning band see if there is something I can suggest. Though I will admit, scenes are one area where OH is a little weak.

OK so your first requirement: “change the colour of any lights that are on when a scene changes” is pretty simple, assuming that all your Color Lights are in gColor.

gColor.members.filter[ f | f.getStateAs(OnOffType) == ON ].forEach[ f | f.sendCommand(<new color>) ]

where <new color> is the color to change the lights to. Filter so we have a List containing all the lights that are ON then send that light the new color.

You’re second requirement though is a bit different but not too bad. If we assume that all your Color proxy Items are in the group gProxy, and you have Rules that keep the proxy and real Item in sync (which should be the case for a proxy Item) then you need to filter on and loop through the gProxy instead.

gProxy.members.filter[ f | f instanceof ColorItem && f.getStateAs(OnOffType) == ON ].forEach[ f | f.sendCommand(<new color>) ]

If your proxy is acting as a true proxy then when the light is ON, the proxy will be ON and vise versa so we don’t really even need to search across multiple Groups.

So this solves your particular case, but what if there are other cases where you may want to filter and find Items like your original question. We can do this a number of ways using your existing Group structure. BTW, the following are implementations of Design Pattern: Associated Items.

    gColor,members.filter[ f | f.getStateAs(OnOffType) == ON ].forEach[ f |
        val proxy = gProxy.findFirst[ p | p.name == f.name.replace("Light", "Proxy") ]
        proxy.sendCommand(<new color>)
    ]

Filter to get a List of those lights that are ON and for each of those find the associated proxy Item based on name and send the new color to the proxy.

This assumes a one to one. If you had lots of proxy Items associated with this one light (not sure why one would have that but let’s just assume it for now):

    gColor,members.filter[ f | f.getStateAs(OnOffType) == ON ].forEach[ f |
        val proxies = gProxy.members.findFirst[ p | p.name.contians(f.name.replace("Light", "") ]
        proxies.forEach[ p | p.sendCommand(<new color>) ]
    ]

So for this we get the List of lights that are ON, and then get a list of proxies that have a matching name (assumes the naming scheme in the Items above) and send the new color to those proxies.

Or in another approach you can use the Groups, though this is a little more challenging because we don’t really have a hint as to what Group the member of gColor is also member of is that we care about. If we can assume that the members of gColor are only ever a member of one other Group then we can do something like:

    gColor,members.filter[ f | f.getStateAs(OnOffType) == ON ].forEach[ f |
        val relatedGroup = f.getGroupNames.findFirst[ gn | gn != "gColor" ]
        val proxies = gProxy.members.filter[ p | p.getGroupNames.contains(relatedGroup) ]
        proxies.forEach[ p | p.sendCommand(<new color>) ]
    ]

Filter to get a List of lights that are ON and for each of those get the name of the first Group it is a member of that isn’t “gColor”. Then filter gProxy for all Items that are also a member of the just found Group. Finally, for each proxy send the new color to the proxy.

If the lights can be a member of more than two Groups, then it gets a bit more complicated because we don’t have anything to tell us which of the two or more additional Groups that are not gColor we care about. You can use the name trick in this case again in some cases.

    gColor,members.filter[ f | f.getStateAs(OnOffType) == ON ].forEach[ f |
        val relatedGroup = f.getGroupNames.findFirst[ gn | gn.contains("room")  ]
        val proxies = gProxy.members.filter[ p | p.getGroupNames.contains(relatedGroup) ]
        proxies.forEach[ p | p.sendCommand(<new color>) ]
    ]

Which gives us the first Group name that contains “room” in the name. Of course this assumes that only one Group will have “room” in its name.

When Xtend and the Rules DSL started to click for me was when I realized that it really likes to work with Lists. All of the powerful operations (filter, forEach, findFirst, map, reduce, et. al.) are List operations. So if you can convert your problem into a List operation the language will help you solve it rather than feel like its fighting against you. So you will find that half of the DPs revolve around representing your problem as a List, usually using Groups, for which the language provides the best tools.

Thanks. The fourth/fifth example is exactly how I wanted to write the rule but I didn’t realise you could get a list of the groups an item was in. Where is the best place to get a list of what function/properties I can use in Xtend. I was just going off the group properties in the following post post so I didn’t realise I could do what I wanted. Design Pattern: Working with Groups in Rules.

I’ll try and implement your rule and also write a python equivalent.

Xtend - Expressions // only the Expressions section is relevant
Xtend - The Movies Example
Help - Eclipse Platform
Archived Projects | The Eclipse Foundation

But for the most part I just use VSCode and let the code completion tell me what is available:

image

The Working with Groups in Rules DP is my attempt to summarize the List operators that are not explained in the Xtend docs but demonstrated in the Xtend Movies Example.

Wow those links are a goldmine that are well and truly berried from normal users. I found out there are classes which could be game changing especially if I could use them with items. I can’t wait until I’m next at home to try it out.

Note my comment. Only the Expressions section applies. Rules DSL does not support classes. Rules DSL is similar to Xtend but != Xtend.

The first link is the official rules docs. The second link is linked to from the rules docs. So it isn’t exactly hidden.

The movies example doesn’t fully apply to rules DSL which is why I wrote the groups dp. So I wouldn’t usually point people to those links. I only posted them here because the documentation for working with lists is so poor for them in the Xtend

The javadocs are really the programmers interface. Not all of the methods shown are supported in Rules. This is why I recommend VSCode with openHAB extension.

Python/Jython is a programming language. Xtend is a programming language. People that write code in either one are “programming”. The declarative aspect of the Rules DSL is mostly limited to defining triggers, which can also be represented declaratively using decorators in Python. The “then” part of a the Rule DSL is a script (a program). Any nontrivial automation rule will require programming.

So, the question is which programming language do you want to use?

As a programming language, Xtend is not very popular, even among JVM-based languages. As one blog article put it, “xTend is not dead, because it has never been living.” I personally found the DSL/Xtend approach to be clunky, crippled and buggy. According to Rich, it’s just because I have had too much experience with many programming languages. According to him, if it didn’t know better, I’d be happier with Xtend. He might be correct, but that’s not a strong argument in favor of Xtend.

I agree there is more information in the OH forums about how to use the DSL and Xtend than for JSR223 Jython. However, Python is one of the most popular programming languages in the world. There is much, much more documentation about how to use Python as a language compared to Xtend. There are thousands of times more Python programmers than Xtend programmers. Multitudes of existing Python libraries can be used with Jython and OH2. And ironically, it’s even easier to integrate Java libraries using Jython. For those reasons and others (like performance), I think Jython is a net win compared to Xtend.

I’ve never said that and never will say that. I have and will say that Xtend, or more properly the Rules DSL is well suited to this specific problem space but if you feel like you are fighting it you should use JSR223. I can’t count the number of times I’ve posted people to JSR223. I’ve never once to my knowledge said anything negative about JSR223. I’ve never once to my recollection told anyone they should not move to JSR223.

But perhaps you know better what I think and what I’ve said than I do.

Agreed. But there are a large minority if not a majority of OH users who are not programmers and who will never be programmers. The barrior to those users for Python is quite a bit higher than Rules DSL. That is why Rules DSL is the default.

If we throw out Rules DSL before the Experimental Rules Engine is ready we will be abandoning a sizable chunk of the current and future OH users who will never even give OH a try let alone preservere to the point they have a working system.

My point then and my point now is the barrior to entry for none programmers is relatively lower with Rules DSL verses Jython et. al. and once past that barrior there is very little one cannot accomplish in the OH space by continuing to use the Rules DSL.

But if the limitations of the language are posing a barrior to you getting started because you are used to having programming features not available, you should use JSR223.

Though perhaps I’m wrong. Can you tell me what I think again? I keep forgetting.

You have actually made that point to me in the past (about programming preconceptions being a reason people have issues with Xtend, to be clear). It’s been awhile so it’s very possible you believe otherwise now. I never claimed you were saying negative things about JSR223 Jython or advising users to avoid it. On the other hand, I don’t agree with some of your points in favor of Xtend.

For example…

If OH users are writing DSL rules with nontrivial “then” Xtend scripts (or standalone Xtend scripts)… they are programmers. Maybe not professional programmers and maybe not self-identified as such, but they are programming and that’s what a programmer does. (We’ve had this discussion several times too.) Given that reality, it’s reasonable to discuss what programming tools are most effective for that purpose.

I’ll tell you what. If you will sign up to explain what an Object is, what a Class is, what a structure is, what a library is, what an annotation is explain why you can’t mix tabs and spaces in indents why you have to use indents in the first place and agree to do so over and over and over and over in this forum I’ll not say another word about Xtend and completely ignore every “should I use rules DSL” posting from now on. I’d be happy to let someone else take that on.

If you will agree to bring the OH docs up to at least the same level as the docs in the rules DSL I won’t even help other people to work with the rules DSL any more.

Hell if you do all that and want me to I’ll delete my account on the forum and move on. I’ll even drop OH itself if it makes you happy.

But as it stands I can’t even figure out how to call an Action in Jython without looking at source code. I’ve seen multiple questions on the forum asking really basic questions like are Timers supported? And often these questions go unanswered.

[One user is at a stand still] (How to use `itemRegistry` within a jython-**module**?) because he can’t get his rules to trigger. 7 days and no response. But he’s a programmer now and there are tons of python resources out there. I guess he can figure it out for himself. That’s what programmers do.

Honestly there have been so few answers to jsr223 questions on the forum I thought most of the users like you and spacemanspiff had moved on and jsr223 had largely been abandoned. In fact someone even posted that you had moved on in one thread. And indeed you haven’t posted since Feb.

So are you back? Are you willing to do what it takes to make Jython (let’s just pick one of the three languages) a viable alternative for new users?

As everything stands right now, saying that there should be no Rules DSL and everyone needs to be a full up programmer to get started is unreasonable and a big fu to the new users. And yes that is how I interpret your position. I’m not sure how else I could but if I’m wrong please correct me because I don’t want to put words in your mouth.

I get it. You think the Rules DSL is shit and shouldn’t exist. But you offer no viable alternative to a big chunk of OH users.

So I’ll continue to answer questions about the Rules DSL. I’ll continue to post DPs. I’ll continue to push those who are frustrated by the limitations of the language to one of the JSR223 languages. And when a viable alternative that has a low barrior to entry for non programmers comes along I’ll probably switch to supporting that. The Experimental Rules Engine is looking promising. But it doesn’t support timers and actions yet, or if it does us mere mortals don’t know how to do it. If over if the JSR223 languages was documented where I can’t figure out how to do basic stuff by looking at the docs, not the code is jump on that.

Node-red has dedicated openHAB nodes that caninteract directly with OH via the REST API.
MQTT works great in node-red (It is kind of made for it) but there are other options to interact with OH.

We’ve quickly moved quite far from an objective discussion, so I’ll make a few final comments and be on my way. Feel free to have the final rant.

Ironically, there are DSL/Xtend questions about timers in the forum within the last 24 hours (and there have been many in the past). With Jython I can point them to comprehensive documentation since timers are part of the Python standard library (along with schedulers and other time-related functionality). But that doesn’t really matter. Users will have endless questions about both approaches.

The biggest practical disadvantage of JRS223 Jython in OH is not directly technical, although it has significant technical implications. It is that it is not well understood or supported by the core developers. It took over a year of effort to get the PR merged in to OH2 and mostly not for technical reasons. They appear to not understand the potential of platform scripting. There’s a tendency to compare DLS/Xtend and JSR223 only in the context of rule creation, but that is a misleading way to compare the two technologies. JSR223 Jython can also be used to create new Triggers, Things, Bindings, Actions, and so on, and those components can be packaged and shared. Supporting this type of development with Jython would open up component development to a much larger developer community that isn’t interested in investing long hours installing, learning, and trying to keep the very complex OH Eclipse and Maven build environment working.

A clear advantage of DSL/Xtend is that it works “out of the box” where OH requires extra installation steps for JSR223 Jython (rather than bundling it). The out-of-the-box experimental rule engine API that JSR223 must use is not great. Fortunately, there are many possibilities to build easy-to-use API’s in Jython to help, but those are not endorsed by the OH team and they must be maintained external to OH. I suspect that, even for you, some your frustrations are from using the low level OH JSR223 APIs rather than using the available convenience libraries that allow “rules” (event-driven functions) to be written in a much more concise way than the DSL without needing to know about classes and objects or being a “full up” programmer (whatever that means).

You are correct that I have mostly moved on from OH. I have voted with my feet. The DSL was not a primary factor in my decision since I didn’t use it anyway. I still run a server, but OH is not the core component in my home automation system now.

2 Likes

Good to know. Thanks!

@steve1, I disagree with very little of your last post. JSR223 is a fantastic way to extend the basic functionality of OH. But that doesn’t help me help new users write automation rules. I’m sure there have been questions about rules DSL timers in the last 24 hours. But the difference is those questions have been or will get answers. Many or most of the JSR223 questions do not. And there are no easy ways to find and read docs that provide the answers.

I can’t speak to the maintainer’s. I don’t work on OH as a developer and have no insight there. From what I have seen I’ve no doubt you are correct. I’ve seen them push away a number of long time contributors. The complex maven eclipse environment is indeed a big factor in why I’ve not tried developing a binding it some such.

And I was indeed angry in my responses. From my perspective you showed up after not paying to the forum for almost half a year only to put words into my mouth that do not represent what I’ve said or believe. Replace “Rich” with “Steve” and swap Xtend for Python and tell me you wouldn’t have been angered? Especially since you left I’ve been one of the biggest promoters of using JSR223. I’m not calling blame, just explaining my responses.

1 Like

For whoever is interested, it’s easy to type “jython timer” in this community forum search form or to type “openhab jython timers” in Google. Both techniques lead quickly to related questions with answers.

Hi Rich,
I’ve been using Openhab for the past 7 months and just wanted to say Thank You.

I’m not big on posting (this being my first) but I just wanted you to know that it’s people like you, that gives folks such as myself the info to succeed. I’ve noticed in a few post you mentioning a white board would help explain the situation. Any thoughts or plans on making a Youtube tutorial?

1 Like

Yeah! We want Rich and his hat on video! We love you really.

I’m a professional software developer and I’ve been able to survive using xtend, but when I found jsr223 in openhab 1 I was so much happier. I shared a rule I wrote in Javascript that I just couldn’t do in a similar way in xtend without a lot of extra and duplicate code. Here it is: General Scene Rule for jsr223

One of the things I really appreciated about it was the fact that I could define any arbitrary data structure for a particular problem and then use that to create rules, even based on the data structure. So in the example above I had an array of objects that included item names. I could then use those item names in each object to create a rule trigger. As far as I know (and I could be ignorant) it is not possible to do in xtend and each item I want to trigger the rule would need to be explicitly included as a trigger somewhere in one of the rules.

After I upgraded to openhab 2, jsr223 wasn’t quite ready to use at the time so I went back to xtend. Most of my rules are not bad in xtend, but I just recently had a doozie. I was setting up my house to open or close vents based on temperature sensors in each room and the operating state of the thermostat. I ended up copying and pasting the same complex rule for each room, after making sure the first was well debugged. I just did a text search and replace for the copies to replace office with bedroom for example. Using jsr223, which I believe is working better now and I will probably eventually switch back to, I could create a data structure defining each room and make one rule with various triggers determined from the data structure. This is just much easier for me and the way that I like to think about problems.