Journey to JSR223 Python 1 of 9

Tags: #<Tag:0x00007f617e5423b0> #<Tag:0x00007f617e542130> #<Tag:0x00007f617e541ed8>

This is the first of a series of posts where I will provide the before and after, commentary, and lessons learned I’ve made while converting my Rules DSL to JSR223 Jython.

Purpose

These will not be fully realized docs. You will have to reference https://openhab-scripters.github.io/openhab-helper-libraries/index.html repeatedly. Keep that page open in your browser as you code. These will also not be and are not intended to be the best way to implement the Rules in Jython. I’m learning as I go and cannot be considered an expert.

My hope is that these posts will show examples the the thought process behind migrating that I took. Also, it should be useful to see a Rules DSL Rule and a Jython Rule that does the same thing side by side.

My intent is to have one Rule per post, but for really simple Rules I’ll include more than one per post. I’ll have the posts linked to each other at the bottom of each post. Expect the quality of the Jython to improve as I go.

Overall Approach

This is going to be a learning process and, against better advice, I am going to be making the changes on my production system. Therefore I’m going to start small and start around the edges. I have a number of very simple Rules or Rules that are there mainly for monitoring. I’m unlikely to break things with the simple Rules and if the monitoring Rules stop working it’s no big deal.

Some New Concepts

configuration.py

See the docs for how to install and use the JSR223 Helper Libraries. Don’t forget to do step 8, rename configuration.py. This is our first new concept. JSR223 provides a way to have code and variables that are shared across all your Rules. configuration.py’s purpose is to provide values to your Rules that either cut across all your Rules, or are values that you do not want to share (e.g. usernames and passwords). By putting those into configuration.py you can share your code without needing to redact them.

$OH_CONF/automation/lib/python/personal

This is one folder down from where you will find configuration.py. Here is where you will place your personally developed libraries. Helper methods, abstract classes, and other stuff like that will go here. There is an important distinction between the code placed here and the code placed in $OH_CONF/automation/JSR223/python/personal. Code placed here is statically compiled which means you need to force Jython to reload them (either through a restart of OH or an import directive). Simply saving the file will not cause your changed to be reloaded.

Take note, unlike with Rules DSL, functions and classes are very well supported in Jython.

First Rule Translation, Alerting

We are starting small so let’s take on my alerting Rule. This is an implementation of Design Pattern: Separation of Behaviors (note this DP has been updated with the JSR223 code) and how I centralize my alerting code.

val logName = "alert"

rule "Send message"
when
  Item aAlert received command or
  Item aInfo received command
then
  logInfo(logName, receivedCommand.toString)
  val night = if(vTimeOfDay.state.toString == "NIGHT" || vTimeOfDay.state.toString == "BED") true else false
  var alert = false
  if(triggeringItem.name == "aAlert" && !night) alert = true

  if(alert) {
    sendBroadcastNotification(receivedCommand.toString)
    sendMail("<phone number>@vtext.com", "", receivedCommand.toString)
  }
  else {
    sendNotification("<email>@email.com", receivedCommand.toString)
    sendMail("<email>@email.com", "openHAB Message", receivedCommand.toString)
  }
end

This is a great example of a Rule that would be better implemented as a library function rather than a Rule. That’s not possible in Rules DSL (and have access to it in all of your .rules files) but it is possible with Jython.

from core.jsr223 import scope
from core.actions import NotificationAction, Mail
from configuration import admin_email, alert_email

def send_info(message, logger):
    out = str(message)
    logger.info("[INFO ALERT] {}".format(message))
    NotificationAction.sendNotification(admin_email, out)
    Mail.sendMail(admin_email, "openHAB Info", out)

def send_alert(message, logger):
    out = str(message)
    night = True if scope.items.vTimeOfDay == "NIGHT" or scope.items.vTimeOfDay == "BED" else False

    if not night:
        logger.warning("[ALERT] {}".format(message))
        NotificationAction.sendBroadcastNotification(out)
        Mail.sendMail(alert_email, "", out)
    else:
        send_info(message)

Because this is in a library, we have to import the scope to get access to stuff like Items. We also need to import the logger. I’ve broken the Rule into two separate functions, one to send info alerts and another to send serious alerts.

Notice how I’ve replaced the email addresses for the sendMails with admin_email and alert_email, which are both defined in configuration.py.

We pass in the logger from the Rule to this function because it’s easier and it will tie the alerts to the Rule that generated it.

Using the library methods is as follows:

from core.rules import rule
from core.triggers import when
import personal.util
reload(personal.util)
from personal.util import send_info, send_alert

# -----------------------------------------------------------------------------
# todo: remove once Rules are all transitioned
@rule("Publish alerts and info", description="Centralizes alerting logic.", tags=["admin"])
@when("Item aAlert received command")
@when("Item aInfo received command")
def send_alert_rule(event):
    if event.itemName == "aAlert":
        send_alert(event.itemCommand, send_alert_rule.log)
    else:
        send_info(event.itemCommand, send_alert_rule.log)

Notice how I import personal.util (the functions above are in util.py. the reload(personal.util) causes Jython to recompile util.py to pick up any changes made there. The alternative is to restart openHAB.

I’ve kept my Separation of Behaviors Rule for now but will delete it once all my Rules are transitioned to JSR223.

Note that strange reload of personal.util. When you make changes to library files, those changes will not be picked up until OH restarts. That can be darned inconvenient. But there is a way you can cause the library to be reloaded when the file the above code is in get’s loaded with that reload command. This frees you from needing to restart every time you change the library. But once your changes have been loaded, remove that reload command, it’s unnecessary.

Is it Cloudy?

This Rule is another example of the Separation of Behaviors DP and it is an example of a Rule that it makes sense to keep as a Rule rather than a library function. This Rule triggers when OpenWeatherMap updates the cloudiness percentage and sets a Design Pattern: Unbound Item (aka Virtual Item) Switch.

rule "Is it cloudy outside?"
when
  Item vCloudiness changed
then
  val newState = if(vCloudiness.state > 50) ON else OFF
  if(newState != vIsCloudy.state) vIsCloudy.sendCommand(newState)
end

One of the things to notice here is that we only command vIsCloudy if the new state is different from the current state. Thankfully there is a method in the library that does this for you.

The Rule becomes:

from core.rules import rule
from core.triggers import when
from core.util import sendCommandCheckFirst

@rule("Is Cloudy", description="Generates an event when it's cloudy or not", tags=["weather"])
@when("Item vCloudiness changed")
def is_cloudy(event):
    newState = "ON" if items["vCloudiness"] > QuantityType(u"50.0 %") else "OFF"
    sendCommandCheckFirst("vIsCloudy", newState)

Initial Lessons Learned

  • If you have clear and concise Rules DSL Rules, the Python equivalent is probably going to be the same or more lines of code.

  • One Rule per file feels like it may make more sense with Python Rules, but I’ll explore that later.

  • The syntax is different but the over all look and feel of the Rules are the same as Rules DSL.

Next post: Journey to JSR223 Python 2 of?

EDIT: Corrections and improvements based on recommendations from CrazyIvan359.
EDIT: Corrections and improvements based on recommendations from 5iver
EDIT: Corrected to use QuantityType/Units of Measure

17 Likes

Just to be precise, this is not a problem with python but rather with jython/jrs223 implementation and the interpreter scope.
Python:

import logging

class Test:
    def __init__(self):
        self.log = logging.getLogger('Test')
t = Test()
t.log('asdf')

raises

TypeError: 'Logger' object is not callable

:sob: I just lost my second posting which was twice as long as this one.

Never write a long post in the browser.

configuration.py should actually be in $OH_CONF/automation/lib/python.

$OH_CONF/automation/jsr223/python/personal can be thought of as where you put rules code.

Logging

I would suggest a slightly different approach to logging. Using the logger provided by the rule decorator, you will know which rule is outputting the log entry.

def send_info(message, log):
    out = str(message)
    log.info(out.format(5 + 5))
    NotificationAction.sendNotification(admin_email, out)
    Mail.sendMail(admin_email, "openHAB Info", out)
@rule("Publish alerts and info", description="Centralizes alerting logic.", tags=["admin"])
@when("Item aAlert received command")
@when("Item aInfo received command")
def send_alert_rule(event):
    if event.itemName == "aAlert":
        send_alert(event.itemCommand, send_alert_rule.log)
    else:
        send_info(event.itemCommand, send_alert_rule.log)

Correct. That was a mistake. I’ll correct it.

I thought of that but I wanted to minimize the number of differences between the two versions of the Rules. The Rules DSL uses it’s own logger so I had the library functions use their own logger. But in practice I had already planned on passing in the logger.

If this is considered best practice I’ll update the OP accordingly.

Sorry - this is the worst!

I don’t know that it is best practice, but for beginners it is definitely suggested to use the logger provided by the rule decorator. This means you know the name of the rule the log message came from, but also requires less code because you don’t have to spawn your own logger. Depending on the content of message it may or may not be easy to identify which rule called the function.

Just to clarify… configuration.py should be located up one directory from this.

For emphasis, since anyone modifying modules will run into this… https://openhab-scripters.github.io/openhab-helper-libraries/Python/Reference.html#modifying-and-reloading-modules

core.utils has similar functions (migrated from lucid) used for the same purpose… postUpdateCheckFirst and sendCommandCheckFirst.

The doc for UoM (QuanityType) should probably be separated out…

https://openhab-scripters.github.io/openhab-helper-libraries/Guides/But%20How%20Do%20I.html#convert-a-value-to-a-state-for-comparison

items["vCloudiness"] > QuantityType(u"50 %")# in this case, you shouldn't need the unicode

I’m not saying you’re wrong… the logging is a little weak. But in this particular case, I would have expected to have seen something like this…

2019-07-22 20:24:55.939 [ERROR] [org.openhab.core.automation.internal.RuleEngineImpl] - Failed to execute rule '0c985c34-a3d2-4c9b-a38f-0ec407ecd1ef': Fail to execute action: 1

…and a traceback logged to the function’s logger. I’ll take a look… this particular case may not through an exception.

Here is an example of a bit more involved person.utils notification function…

from core.jsr223 import scope
from core.actions import NotificationAction
from configuration import scott_email, lisa_email
from core.log import logging, LOG_PREFIX

log = logging.getLogger("{}.personal.utils".format(LOG_PREFIX))

def notification(prefix, message, priority=0, audio=True, kodi=True, android=True):# priority 1 is urgent
    log.info("Notification: {}: {}".format(prefix, message))
    current_mode = scope.items["Mode"].toString()
    if android:
        if priority > 0 or scope.items["Presence"] == scope.StringType("Away") or current_mode in ["Night", "Late"]:
            NotificationAction.sendBroadcastNotification(message)
        else:
            if scope.items["Lisa_Region"] != scope.StringType("Home"):
                NotificationAction.sendNotification(lisa_email, message)
            if scope.items["Scott_Region"] != scope.StringType("Home"):
                NotificationAction.sendNotification(scott_email, message)
    if audio:
        if priority > 0 or (scope.items["Disable_Audio_Notification"] == scope.OFF and scope.items["Presence"] == scope.StringType("Home") and (current_mode not in ["Night", "Late"] or (current_mode == "Night" and "Time: " in message))):
            scope.events.sendCommand("Audio_Notification", message)
    if kodi:
        scope.events.sendCommand("Kodi_Notification", message)

Should we maybe change these to updateIfDifferent and commandIfDifferent and create aliases for the Lucid names? Their current names are a bit vague.

I’m OK with changing the names. Submit an issue and we can discuss in the repo.

I looked and didn’t find them. One of the challenges I’m finding with the Helper docs is you have to know what to search for to find anything useful. As I gain experience this will get better but it isn’t always clear of obvious what to search for or where to browse to to find what I need. Even now, I can’t find anything even when I search for sendCommandCheckFirst and there is no util entry under core in the docs.

I do remember those methods from lewie’s JS library but when I couldn’t find them (I forgot what they were called) I assumed that they didn’t make the cut, or you would pipe in and tell me where they are. :wink:

I thought I remember reading that UoM wasn’t supported yet so I didn’t even think to look. I still don’t see that at the link. Do you mean there needs to be an Issue and eventual PR to document UoM?

So I cheated a little bit. The actual place I observed this was in the body of a Timer. I had a timer function that just had a logging statement and I messed that statement up as described. There was nothing at all in the logs. So perhaps it is only completely silent when it errors in a Timer lambda, which raises the question “What other errors get suppresses in Timer lambdas?” I should probably move this “lesson learned” to my second posting (when I rewrite it) and make the context a bit more clear.

I’m glad you guys are chiming in and seeing these posts. Granted, I know more than most so am not a perfect “noob” user. But I will provide more documentation on my progress translating my Rules and therefore learning the language and libraries. I hope my experiences are useful to both other new users and the maintainers to identify potential trouble spots.

2 Likes

It will be there tonight after I push. A lot of the docs are fed by docstrings, so when in doubt, you might try checking in the modules if you don’t find something in the docs. Having trouble finding these as a beginner is good feedback. Do you think a layout change might help, or is this just something that will come with experience?

Definitely ping me if there’s anything I can help with! There’s nothing too small. That goes for anyone trying to get into Jython/JS/Groovy/etc. rules!

It’s in there… third example down. Just type ‘QuantityType’ in the search. I’ll add a separate entry.

Yes… timers do eat errors!

I’m not sure. Maybe providing enough synonyms in the docs so the search function in the docs works would be helpful. Let me do some more and I’ll see if I can think of something better. Right now I’m spending 80% of all my time on the “But How do I…” page and I suspect I’m not alone in that. So if I were to put any major focus it would be on that page. Maybe it would be useful to have some links to the reference docs from there which will help some users orient themselves to the doc’s overall organization.

Not sure how I missed that. But this is a good example where synonyms would help the search. I suspect I was searching for “UoM” or “Unit of Measure” or variations of that.

1 Like

Sorry to hear.
Hope you find time and the spirit to do it over … looking forward to read it.

Thanks for this tutorial … how-to

Would you please tell me why you are converting your rules?
Will Rules DSL be obsolete in the near future?
Or is JSR223 so much better?

What would you recommend?
Should a beginner (me) with little coding skills try to learn the new language and do the covertion too?

1 Like

Lots of reasons.

  • I like to learn new things. I can write Rules DSL Rules in my sleep now.
  • More and more people are using JSR223 Python on the forum and I’d like to be a resource to them too.
  • Reports are that they run faster, though that hardly matters in the home automation context.
  • To help contribute to the docs for them (maybe the code someday).
  • To provide a comprehensive set of before and after examples for others looking to migrate as well.
  • Prime myself for the development of Rule Templates that others in the community can use. No more copy and paste of code from the forum.
  • While I have defended the Rules DSL over the years, as a coder, I myself am more comfortable using Python or JavaScript than Rules DSL.

I expect that Rules DSL will be deprecated in OH 3. That doesn’t mean unsupported but it does mean that all the docs, all the example, and all the defaults will be geared towards UI created Rules and JSR223 Rules, which though they have some differences, are executed by the same Rules Engine.

When a more usable UI is released for developing Rules, that will be the primary way new users and particularly non-coder users will develop their own Rules. The UI developed Rules will use JSR223 scripts like the above in cases where custom code is required.

For a certain class of OH user, namely those who already know how to code, JSR223 Rules are objectively better. For others, it isn’t so clear. The UI in PaperUI is kind of unusable at the moment (a replacement is under development). Setting up JSR223 with the helper libraries is a few extra steps which some users may not be willing to do. And tooling for development (i.e. JSR223 equivalent of VSCode with the openHAB Extension) is much less capable.

You will be fine sticking with Rules DSL. It isn’t going anywhere in the near term or mid term. I’d recommend waiting until the replacement for PaperUI’s Rule UI is released and then consider migrating your Rules to that.

But if you want to get more into coding or don’t want to wait then by all means. Take the plunge.

2 Likes

Thank you for the detailed comments.

from core.utils

And I was struggling with the literal translation of the rules DSL code as I couldn’t find the simple sendCommand.
Sure I finally found it in the ‘how do I’ section but it would have been better to point there this early.

Then a line like events.sendCommand("Terrasse_Bewegung_Timer",ON) for me still results in the error below, again making me wonder why. Ok, found out “ON” needs to be in quotation marks, but why is ON no string ? Why does sendCommandCheckFirst work with either ON or "ON" then ?
These little subleties such as implicit data types are really confusing because you never know if it’s your Python (lack of) knowledge or specific of (not) having imported the right package and most important, which package to import. And where’s the list of those ?

Actually I find the most challenging part for a migrator to date is to find out which lib/class commonly used stuff is in that you didn’t need to explicitly import in Rules DSL.

2020-01-01 16:40:22.905 [ERROR] [on.Weihnachtsbeleuchtung einschalten] - Traceback (most recent call last):
  File "/etc/openhab2/automation/lib/python/core/log.py", line 51, in wrapper
    return fn(*args, **kwargs)
  File "<script>", line 23, in Weihnachtsbeleuchtung_An
TypeError: sendCommand(): 1st arg can't be coerced to String, org.eclipse.smarthome.core.items.Item

2020-01-01 16:40:22.938 [WARN ] [e.automation.internal.RuleEngineImpl] - Fail to execute action: 1
org.python.core.PyException: null
        at org.python.core.Py.AttributeError(Py.java:207) ~[?:?]
        at org.python.core.PyObject.noAttributeError(PyObject.java:1032) ~[?:?]
1 Like

Well, these posts are not and never were intended as documentation. There are just what it says in the title, my journey. They show my experiences and struggles and successes. As I was writing these posts, I had the “How do I” page open at the same time next to my VSCode.

By default, in the Python Rules there is only the sendCommand Action. And as with Rules DSL, the sendCommand Action only accepts to Strings as arguments. Unlike Roles DSL, Python doesn’t do the half assed only right some of the time autoconversion of types (e.g. from OnOffType ON to String “ON”).

The Helper Library core.utils has a sendCommand function you can call that works more like what you are used to when calling MyItem.sendCommand() in Rules DSL (i.e. it accepts an Item it the name of an Item as the first argument and a String or Type such as ON as the second argument. It then does all the conversions for you.

When I wore theses posts, I didn’t really understand that aspect so I neither mentioned it nor used the core.util.sendCommand function.

Hopefully the above explanation makes this clear. events.sendCommand is like calling sendCommand(“MyItem”, “ON”) in Rules DSL. Consequently both arguments must be Strings. And Python won’t try to guess and convert them for you. It works with the core.utils versions because those functions are written in such a way that they accept both Types and Strings.

I know that Scott is working to make the Helper Libraries just be available when you install the Rules Engine but I don’t know if core.utils is planned on being automatically imported into the context so you don’t have to import them. @5iver, any comments?

The real challenge is in Rules DSL the tires are implicit. In Python you at l have to be explicit. So a call that you used to be able to get away with in Rules DSL like sendCommand(“MyItem”, ON) you can’t get away with in Python because Python won’t automatically convert the ON to “ON”. That’s why the core.utils functions were written.

They are all documented in the Helper Library Docs here.

Because it is using an Item rather than a String name… https://github.com/openhab-scripters/openhab-helper-libraries/blob/master/Core/automation/lib/python/core/utils.py#L130.

Without core.utils…

events.sendCommand(itemRegistry.getItem("Terrasse_Bewegung_Timer"), ON)

I’m not clear on what you are referring to. Do you mean these?

https://openhab-scripters.github.io/openhab-helper-libraries/Guides/Event%20Object%20Attributes.html

It’s important to understand these too… https://www.openhab.org/docs/configuration/jsr223.html#events-operations.

Everything you need should be included in the default script scope or one of the other presets. Let me know what you are missing and I can add them.