Idea: use python to directly generate rules files

Hai,

Some weeks ago a friend of mine came with the idea to “read” the local government office website and extract the dates on which the various types of garbage are collected. Great job by Corne! He made a script to do so and output the results to his domoticz system, which in turn would send telegram messages the night before to remind his family to set the correct type of trash outside. Very nice idea but done in java & domoticz. But all is working great (after some debugging).

So I took up the challenge and re-wrote the thing in Python + Openhab-Python interface. The end-result is shown below. An overview of all garbage collecting moments, daily updated and a telegram message the evening before and early in the morning.

Result

afval

How does it work?
Basically crontab on my PI runs a python script named “afval.py” once a day (afval = trash in Dutch). The script is attached below. It basically gets the webpage of the city hall, uses PyQuery to understand the page and then finds the correct data. Then do some processing as the year is not included in the data. Then the script connects to my openhab and sets the items shown in the figure above.

afval.py (6.9 KB)

Generated rules file
Then I started to write some generic rules to trigger a Telegram message based on the dates written to certain openhab items, but while I was writing the rules came up with a different approach. Why not directly generate the rules file?

Python code snippet below writes the actual rules file. (Download the entire file if you want to try yourself)

Python code

def add_openhab_rules(soort_afval, file, ophaal_datum, ophaal_datum_dag_eerder):
   day     = ophaal_datum.day
   month   = ophaal_datum.month
   day2    = ophaal_datum_dag_eerder.day
   month2  = ophaal_datum_dag_eerder.month

   # crontab format:
   #  Seconds Minute Hour Day Month Year
   file.write('// auto generated rules file by afval.py script @' + str(datetime.now()) + '\n')

   file.write('rule \"afval_notificatie_' + soort_afval +'\" \n')
   file.write('when\n')
   file.write('    Time cron \"0 30 07 ')
   file.write(str(day) + ' ' + str(month) + ' ?\" or\n')
   file.write('    Time cron \"0 30 19 ')
   file.write(str(day2) + ' ' + str(month2) + ' ?\" \n')
   file.write('then\n')
   file.write('    logInfo(\"Afvalkalender\", \"Afvalkalender         : ' + soort_afval + '\") \n')
   file.write('    sendTelegram(\"domoticahajerstijn\", \"Afval notificatie: ' + soort_afval + '\") \n')
   file.write('end\n\n')

The output is following: a file called afval_generated.rules

// auto generated rules file by afval.py script @2019-12-31 13:05:26.804218
rule "afval_notificatie_Restafval" 
when
    Time cron "0 30 07 31 12 ?" or
    Time cron "0 30 19 30 12 ?" 
then
    logInfo("Afvalkalender", "Afvalkalender         : Restafval") 
    sendTelegram("domoticahajerstijn", "Afval notificatie: Restafval") 
end

At the end of the python script, the generated file is copied to the correct folder and openhab automatically processes the new file.

# copy the rules files. This will not work on Windows, so a try:finally mechanism is needed.
try:
    os.system('cp afval_generated.rules /etc/openhab2/rules/')
    os.system('chmod 777 /etc/openhab2/rules/afval_generated.rules')
finally:
    return        

Must say I am very pleased with the result as there is no need to use “hidden” openhab items to store the dates and there is no need to write generic rules which in turn uses these hidden items as input.

Maybe this will trigger one of you to try a similar approach.

Grtz Matthijs

5 Likes

We already have Jython which is basically Python 2 based. Is that not sufficient? @rlkoshak and @5iver have been working with that for quite some time.

Thanks for the quick response!

Basically I have been working a lot with python of over the last year, so I really wanted to do this in (pure) Python. Fun part was that I was able to develop this script from my Windows computer using the Anaconda environment. All python packages needed to connect to OpenHab are also available for windows.

As Python 2.x is end-of-life Tomorrow (it’s now New Year’s Eve 2019-2020), there is no reason to spent any time on learning 2.x I guess.

Also I have more python scripts running to complement openhab. Especially the 433 MHz modules I use, need a python glue between openhab and the transmitter.

But going to have a look at Jython, maybe I am wrong.

Greetings Matthijs

CPython is EOL but Jython is not. In my limited experience, except for possibly some minor syntax changes and additional libraries there is not much differences. There are even some Python 2 libraries with Python 3 features to help eventual migration.

https://python-future.org/quickstart.html

Ahhh. Just learnt something. Thanks. Valuable addition.

Will keep Jython in mind for cases where using Python directly is not possible (e.g. scripting from openhab).

Grtz Matthijs

If you don’t like python2 you could use HABApp to automatically do these things which runs from python 3.6 on.

Wow. Why no one ever told me about HABApp. It looks amazing.

Rather than write coffee to generate code, which rarely works well in the long run, why not just code The rules in Python to begin with? In OH 3, Python will be there default rules language.

So far the difference between python 2.7 and 3, in an OH context, is almost exactly zero. And Scott is working so that we are not stuck with Jython but have access to pure Python and lots of other languages too.

End of support just means no more updates. It doesn’t mean that all the python 2 scripts it there will stop working tomorrow. But if your are really bothered, there is @Spaceman_Spiff’s Habapp which is a rules engine that runs along side OH written in OH 3.

Why not let the built in rule engine be the glue?

3 Likes

I did not write OH3 (but only because I don’t know java :smile: :wink:).
It’s python 3.

1 Like

I am not sure ANYBODY knows OH3 at this point, including the developers. :face_with_raised_eyebrow:

This is great news!!! Where can I read more about it?

I think this is for openHAB 3 where it will use Java 11 instead of Java 8, if I recall correctly.

Typo. :flushed:

There are numerous issues and threads but ultimately I think we need to get off of Java 8.

2 Likes

Just to be clear, you meant Python 3 which, if used, will require OH3.
You were just a little too concise. :wink:

Habapp is a wholly separate service written in Python3. It’s kind of like Node Red in that respect. It interacts with OH though it’s REST API. Consequently, it works with any version of OH.

2 Likes

The framework and tools for you to do this have been provided… why use the old technology? The vision of a robot that works furiously with a printing press comes to mind :smiley:.

Using a Python script to generate a DSL rule sounds like a lack of awareness of the new rule engine, scripted automation, and the helper libraries. With these, the original Java code could have been used, rather than rewriting it in Python. But if you prefer Python, it could all be done within the new rule engine.

Rich, were you referring to GraalVM. This does not allow for the use of “pure python” though, but an implementation of Python written specifically for GraalVM, similar to Jython. I recently made a comment about this here. IMO, it will be a long while before it would be worth the effort to integrate GraalVM into OH for automation. However, I do plan to build bundles for other languages.

Thanks to all,

This thread gave a lot of new insights. Did quite some reading the last days.
To summarize my use cases it is something like this:

  1. Download a website once a day.
  2. Process the site to extract some information
  3. Update OH2 items with the new information
  4. Make sure timers are triggered anywhere between “today” and “in 3 months”
  5. (make sure item 4. still works after each reload/restart of OH2.)

So, what I learnt:

HABapp
Much similarities between the “ptyhon-openhab” package I used. But can do a lot more, for example being triggered by OH2 events. All python packages can be added. So using PyQuery is possible (for step 2 in my use case). But also some drawbacks as it is a “side” project. Not maintained by the OH community if I understand right. Still under development.

–> might test this solution for the next idea that comes up and requires some python (according my vision)

new rule engine
Looked at it, but as I have no programming background, I have no clue how to get items 1 & 2 done with it.

–> no go. Have no clue where to start and no idea if my project is possible at all.

scripted automation & helper libraries
Probably should invest some time in this solution. But will be another steep learning curve I guess.

–> depending on the time I have will invest time to learn about it.

But as always in life, you cannot do everything the best possible way. The solution I made in the 1st post worked and was done within 4 hours from scratch. So the trade-off between spending time on domotica and construction work on the house, was acceptable (according my wife :joy: )

As a Mechatronics engineer I already discovered there are way to many programming languages & software tools important these days and there is only time to learn a few up to a level I can use it as part of making a living. So on the programming field, for now, I limited myself to:
C is for microprocessor (some C++ for arduino/ esp8266)
Python for scripting & dataprocessing (in case Matlab is not available)
C# for windows UI programs.

(not to mention the software tools I need to learn for mechanical design & electrical design).

OH2 is working fine now as I was able to start over in our new house, but it has been a steep learning curve so far. My 2 best friends both run Domoticz and they seem to have a lot of stuff up and running fast (I did not say faster). I always need a lot of googling for examples to get things correctly running --> so a general remark: keep in mind that complete working examples, and not code snippets or API interface definitions are, the most useful source of information for users like me.

Basically this forum is the best documentation there is.

Grtz Matthijs

Pretty much the same way as you would in Rules DSL or in native Python. Call the sendHttpGetRequest Action to download the page and depending on the format of the downloaded contents use the openHAB transformation action, either JSONPATH, XPATH, or REGEX.

There are tons of examples on the forum and we can help if you run into trouble. For the most part you don’t need to know how to program this sort of thing, you can just use the tools openHAB provides.

One of the goals of Rules (any version of rules) and openHAB is that you don’t need to know the details about how to connect to devices or web pages or specific technologies.

:+1: for your quick reply.

But in one of the above post following link was given for the new rule engine:
new rules engine

It pretty well illustrates the steep learning curve issue of OH. On that page, there is no information at all which gives me the impression that my “use case” can be done. So, my conclusion was / is : no go.

So, I will search the forum and come back if needed :slight_smile:

Grtz Matthijs

You can’t put every use case in every page of documentation. See How to get started (there is no step-by-step tutorial)

So having a familiarity of what OH can do is important. That’s why we emphasize reading the docs so much. That page doesn’t say anything about getting web pages but it does take about openHAB Actions. And there is a page in the docs that goes over all the built in actions including the sendHttpGetRequest.

To be successful with any home automation system you have to be able to piece together tutorials and reference guides that address on small party of what you are trying to achieve because there never will be an end to end tutorial for your specific system. Your system is unique.