The rule file provider was finally merged, meaning that it is now possible to include rules and rule templates in the YAML configuration files together with Things, Items, tags etc.
Documentation must be written, but I haven’t started thinking too much about that yet, because I have a feeling that some things might still change. As of now, rules and rule templates are supported in the files, but not from the marketplace. In addition, the UI still shows the “old YAML format” for rules, and there’s no functionality to convert to/from YAML (and potentially DSL in a very limited way) like there is for Things and Items.
I’ve decided to start looking at the “conversion” part first, since it clearly is the most complicated. Fixing the marketplace should be easy, but the marketplace itself must first be fixed first.
When looking at the conversion part, I’ve met my first “challenge”. When studying the code that parses DSL rules, I can see that the rule UID seems to simply be the name of the .rules file with a -<index> suffix. I assume that the index is added to ensure uniqueness, but I’m not sure that I understand exactly why. It seems like the index starts at 1 for each file independently, so if you have two .rules files with the same name, they would generate the same UID. I’m not even sure that’s possible thought, is it allowed to have subfolders inside /rules? If not, there can’t be a file name collision anyway.
Which leads me to the next question: Can there be more than one rule in a .rules file? That’s the only thing that can really make sense out of what I’m seeing.
I need to understand this to make the correct changes, since doing the file conversion means changing some of this code to support dealing with “isolated”/temporary “rule files” (not actually files, but the content is the same as it would have been in a file). I must enable this without botching the existing rule UID scheme.
Yes, and you are correct, then the index of the rule ID increments for each individual rule in the file.
It’s quite common for users that use DSL to have multiple rules in a single file. This allows sharing global values between rules and is often just used to organize systems with lots of rules (e.g., all light rules are in lighting.rules and all door alerts are in doors.rules).
This didn’t used to be possible. None of the DSL folders supported subfolders. I can’t speak to whether this has changed in the last few years. It should be relatively easy to prove one way or the other through experimentation. But it definitely wouldn’t be supported to have two files with the same name for all the reasons you describe.
I suspect it’s still the case that subfolders are not supported. Otherwise, we would have heard about problems already concerning rules disappearing because they get overridden by a new rule with the same ID.
I’ve always bemoaned the fact that one cannot set the uid of a DSL rule. I think it’s possible in all the other rules approaches to provide your own uid for the rule. It’s certainly possible in JS, jRuby, and Python. I’m pretty sure it’s possible in Groovy too.
I doubt anybody has changed it, and it looks like this is the assumption with the assignment of UIDs. This code is just hard to get a “complete overview” of, so I’d rather get some insight this way in addition to what I interpret from the code. I think it’s fine to just leave it without subfolder support, it might not be organized “as neat” as some would wish - but you can do much the same by using prefixes in the filenames.
The shared context between the rules makes sense with regard to some of the code in here.
I know that I can just test all of this, but I’m afraid that I’ll draw the wrong conclusions because of some other circumstance. I can also figure it out by digging deep enough into the code, but that can be hard work, and getting some info “on the side” can save me some time and effort.
I won’t pretend to understand how Xtext/base/tend works, but it doesn’t seem like it would be too difficult to make a way one could specify the rule UID also for DSL. I think it’s mostly a matter of figuring where it would fit in “when”, “then” etc.:
…it could then be read out the same way name is, and the rest is “easy” (for me). I guess nobody ever saw the need? I can’t quite see that there’s any technical reason why that couldn’t be done.
I don’t think I will go after that now in any case, I want to just focus on making this file conversion possible.
Conversion to DSL will have to be quite limited. I see that DSL rules supports any number of triggers, no conditions and only a single action. But, it also only supports a predefined subset of triggers. And, of course, it only support the “DSL language” for the action. So, I will have to check all these things before even “offering” to export one as DSL. I’m not going to try to make something that can convert JS to DSL
I’m willing to bet that only a minority of managed rules would qualify.
Where this conversion is really useful though is to load a text file DSL rule to make it managed. But I wonder if it makes sense to export anything other than yaml because of these limitations. Note the same limitations apply to all rules languages.
The only true one-to-one conversion would be to yaml.
That will come and is quite easy (it’s basically working already in my branch through the API - haven’t made the UI bit).
I realize that “to DSL” has a somewhat limited value, but since I try to “mirror” what @Lolodomo has done for Things and Items, it kind of follows from that. I decided to give it an attempt, and abort it if it became too hard to do. I think I’ve made some progress, but at the moment I can’t get it to run for some extremely frustrating OSGi reason, where the bundle fails to start because OSGi claims that a class has no constructor (which it does). It’s just one of those OSGi things that can be extremely frustrating and consume many, many hours, but the “problem” is probably tiny - something somewhere that isn’t defined as it should - if I could only find it.
Since the current code was never meant to write DSL files, only read them, I’m not really sure how many modifications are needed. But, it looks to me, from what was done with Things and Items, like “the fundamentals” for export is in the Xbase/text/tend somewhere once the formats have been defined (and they are because they are read/parsed).
Making it work would make the whole thing fit more neatly with what’s there, and it would also allow the Code tab to show the complete Rule DSL code, not only the action (in effect - to include the triggers).
I guess that’s still far into “nice to have land”, so I’ll have to try to hold back in how much effort I put into exporting to DSL…
The other languages “don’t have a format”, so that part is simple. YAML will be the only option, as we have nothing else (except JSON) that can store them. When it comes to JSR223/SimpleRules, no export will be possible, not even to YAML. That’s because of how they “cheat” in that the action isn’t actually a part of the rule, it’s just a pointer/reference to some bytecode that has been compiled by the scripting language when the rule was “created”. So, it’s impossible to export that, unless you want to venture into decompiling the bytecode
I’ve been fighting OSGi for a long time now. I just can’t find what the problem is. Basically, the REST interface can’t “find” the DSL rule file converter. I’ve tried to do everything identical to what has been done for Things and Items, but this is all that happens (it never starts - it stays at “SATISFIED”):
08:35:00.220 [WARN ] ( (change controller)) [ternal.fileformat.FileFormatResource] - bundle org.openhab.core.io.rest.core:5.2.0.202602230709 (75)[org.openhab.core.io.rest.core.internal.fileformat.FileFormatResource(196)] : Could not get service from ref {org.openhab.core.automation.fileconverter.RuleSerializer, org.openhab.core.automation.fileconverter.RuleParser}={service.id=305, service.bundleid=87, service.scope=bundle, osgi.ds.satisfying.condition.target=(osgi.condition.id=true), component.name=org.openhab.core.model.rule.internal.fileconverter.DslRuleFileConverter, component.id=226}
Any hints are appreciated. @Lolodomo do you remember if you did something “special” to get this to work? It’s a bit “strange” that it’s in an “internal” package, but that’s the same for Things and Items.
In my experience when working with multiple core bundles at the same time, especially when working with the core core bundles (e.g. org.openhab.core) the only thing that worked for me was to do a distro development build, see Karaf debugging | openHAB
I know and use that, I made the Pdev functionality and wrote the docs
But, it this case, there must be something missing somewhere. I have cleaned and built core in its entirety several times, I’ve tried running it both under Karaf and Equinox, the exact messages I get differ a little bit, but generally, the problem is the same. I’ve “cleaned” every conceivable thing, caches etc. in Eclipse. The problem remains. So, I’m fairly confident that there is a real problem, there is some definition that is wrong or missing somewhere, I just can’t figure out where.
According to Googe, the name being $000 is an indication that OSGi failes to resolve the actual parameter type, and that this is probably the reason that the components “fails” - although it doesn’t say that it fails, it weirdly stays as “satisfied” when it should have gone to “active”. Trying to debug this is really “unfriendly” from where I stand, if just one of the errors/warnings/indications I find could just point to the actual cause of the probem, things would be so much easier. There are probably things I don’t know, so an “OSGi expert” would figure out what happens immediately and understand what to do. But, to me, it fees like clairvoyance is the only way to figure out what the problem is.
As usual, what Google claims seems to be just BS. I’ve looked at other files, and they have exactly the same type of references, and they seem to be working. Here is an example where some references are named and others aren’t:
To me, it looks like the references that are injected in the constructor have the $nnn form, while those injected via setters have the actual class name as name. So, this is probably another dead end.
I have reorganized a bit (moved the new class/component to a different bundle), which seems to have resolved this issue. I’m guessing that there was some kind of circular dependency involved that prevent it from working, but I must say that the wild goose chase the feedback, and in particular Google, was very frustrating and didn’t really mention circular dependency as an option at all.
I still don’t know if that was the cause, but I consider it likely because simply moving the class to a different bundle made things start working.
Edit: I can’t make another post until somebody else has posted, so I’ll just have to keep adding to this one.
I’m trying to figure out how to turn a DSL Action (the THEN part) into a XBlockExpression. I’ve tried to debug what happens when .rules files are parsed, but that’s a very complex process full of injections and reflection, which means that I haven’t succeeded. I’ve also tried various other approaches, but no success this far (again, Google suggests a lot of crazy stuff that doesn’t come close to working).
@dpa-openhab I have the impression that you’ve figured out a lot of the Xbase/text/tend stuff, do you have any idea of how I should do that? I’ve looked at ScriptParser, but isn’t at all sure that it’s the right thing, or how to use it.
That’s weird. I would have expected you to have made it to “regular” status on the forum by now but you were still just a “member”. I bumped your status to regular. That should remove that restriction.
Ok, it seems like I’ve figured out a way to parse a DSL script into a XBlockExpression. Don’t know if it’s the most effective way, but I’m basically doing what is done when parsing “DSL scripts” (not rules).
So a real XBlockExpression is surrounded by { and }, while Script is an XBlockExpression not surrounded by { and } — that is the only difference. Currently between then and end no { brackets are required }. The above could be changed not to utilize Script after Rule then… by creating a new EvaluationContext after then and using as grammar (expressions+=XExpressionOrVarDeclaration ';'?)* instead of script=Script`.
This code is just hard to get a “complete overview” of
I can’t get it to run for some extremely frustrating OSGi reason, where the bundle fails to start because OSGi claims that a class has no constructor (which it does)
It would also allow the Code tab to show the complete Rule DSL code
Any chance that the Code tab shows the complete code for rules written in JavaScript, Groovy, Python, Jython?
Ok, it seems like I’ve figured out a way to parse a DSL script into a XBlockExpression
It is with internal/engine/ScriptEngineImpl::parseScriptIntoXTextEObject().
For me converting DSL scripts and DSL rules into YAML format makes as much sense as converting Groovy, Python and JavaScript openHAB-automation code from a file to an YAML format. This essentially will mean wrapping the existing code into a multi-line tag, like
context: console.log("I am not a file-based JavaScript automation, but much better YAML-JavaScript-file-based code!")
My reading of the discussion is that the existing DSL Rules and Script will just be retained, as written, but the code will be wrapped in a YAML directive.
As in Javascript/Groovy/Python outside of a rule can be defined variables, which are used in the rules, and are shared between several rules in the same file, DSL rules can have also a variable defined outside of a rule, and import statements (e.g. for UnDefType) defined outside of a rule: all these will have to be mapped to YAML.
I see as advantage for YAML instead ot XText configuration that the parser is smaller and if no Xtext configurations are used, the XText bundles can be stopped, so not loaded by openHAB/Karaf. However running a DSL rule, wrapped in YAML, requires anyway both XBase and XText for parsing and executing the rule (written in XBase) even if the XBase-lines are wrapped in YAML.
Apart from being unable to set user-provided UID, DSL Rules from files also cannot set tags. But I consider this as okay, becasue these DSL Rules have much bigger problems.
Currenlty DSL Rules and Scripts are loaded as Eclipse Models - a concept which I do not understand. But I think the direction should be to abolish the difference between “DSL Rules” and “DSL Scripts” by converting the construct “Rule … when … then … end” to an expression and allowing both rules and scripts to just list expressions in any order. This would mean merging the six bundles org.openhab.core.model.{rule,script}{,.ide,.runtime} into three bundles. This would make the code simpler. However I am not sure in case such a change is proposed, that the change will ever be reviewed.
The YAML format is essentially a file based version of a managed rule in terms of overall structure and support. As such, there really isn’t an “outside of the rule” to define such variables and imports. And here we are talking about translating one of these rules to DSL file based format. So I’m not sure this would be an issue. It certainly would be an issue to translate a rule from a .rules file to a managed/YAML format though.
Note, a managed rule and a YAML rule could support multiple different rules languages in the same rule. But of course that type of rule would be impossible to translate to DSL anyway.
To translate a managed/YAML rule to DSL it must:
have exactly one action, a script action using DSL as the language
have no conditions
not use any of the triggers that are not supported by DSL rules (e.g. time of day).
Such a managed rule already cannot have global variables nor can it have imports, so those concerns do not apply.
To translate a DSL rule to YAML/managed it must:
not reference any global variables (could be replaced with cache)
not use any imports
If a rule doesn’t meet these restrictions, that rule cannot be converted to YAML.
Thank you for the explanations regarding the structure, my problem was rather how to turn a string with a script into an XBlockExpression, but it seems like I’ve managed that.
Believe me, I checked and double-checked Import-Package and Export-Package. That’s not where the problem was, I also checked the OSGi headers and scr:info time and time again. Nothing made sense, until I moved the class from org.openhab.core.model.rule to org.openhab.core.model.rule.runtime. I think it can stay there, it’s not entirely clear for me what’s different about the runtime bundles anyway, except that they don’t seem to contain generated code.
You can already do that to some extent, but it’s complicated. The Action of any SimpleRule is impossible to display, unless you want to display the bytecode (I doubt users would find that useful). This is because OH simply don’t know/have the uncompiled Action. If it’s not a SimpleRule, you can already view the code for scripted Conditions and Actions by clicking on them (they will open a “script viewer”) in any language. In addition, if the Rule contains a source configuration string element, that will be shown in a separate “Source” tab. Helper libraries can include the source for the rule, also for SimpleRules, and they will show there. This only works if the Rule contains the source code, obviously - if it doesn’t, the tab isn’t shown.
So, we’re already showing the code to the extent possible. There is a different between a script and a Rule. A Rule consists of various fields and modules, all contained as one “entity”. The modules are Triggers, Conditions and Actions. A Rule can have any number of any of them. You can’t express Rules in JavaScript, Groovy, Python etc. simply because there is no way to define a Rule. DSL is different, because it is both a “scripting language/Xbase”, and it’s a file format .rules. The “DSL file format” has the structure to define a simplified rule, that is, it can only supply the bare minimum of information (the name, the UID is auto-generated), one Action (in Xbase) and any number of predefined Triggers. It doesn’t support all Triggers, only those that are defined in DSL.
More or less. I don’t have access to that method, but I have made a copy that I have tweaked slightly. I’m not “done” with this, I’m not sure that it’s currently “optimal”, but it does work.
It’s not the YAML in itself that is the point. The point is that you can define every part of a rule, and isn’t limited to a small subset of properties. With the YAML format, you can define everything, so it’s the only file-based format where you have the possibility to use all aspects of a Rule (except JSON, but that’s very inconvenient to work with because the scripts must be stored as strings with \n on one line).
Here is an example of a YAML containing multiple Rules in various languages:
version: 1
rules:
mode-tv-rule:
label: Mode TV
templateState: instantiated
config:
sourceItem: None
configDescriptions:
- name: sourceItem
type: TEXT
context: item
label: Source Item
description: The source Item whose state to monitor
required: true
tags:
- tag1
- tag2
- Tag 3
triggers:
- config:
itemName: TvPower
state: ON
previousState: OFF
type: ItemChanged
actions:
- id: script
config:
type: DSL
script: |
logInfo("Rule","mode TV")
// Stop Item
ItemStop.sendCommand(ON)
type: Script
stub:mode-tv-rule:
label: Template based mode TV
template: mode-tv-template
templateState: pending
config:
sourceItem: 'TvPower'
rules_tools:tsm:
actions:
- config:
script: >
// Version 1.0
var {TimerMgr, helpers} = require('openhab_rules_tools');
console.loggerName =
'org.openhab.automation.rules_tools.TimeStateMachine';
type: JavaScript
id: '3'
type: Script
description: >-
Creates timers to transition a state Item to a new state at defined times of
day.
label: Time Based State Machine Test Rule
triggers:
- config:
groupName: DemoSwitchGroup
id: '1'
type: MemberChanged
- config:
startlevel: 100
id: '2'
type: StartLevel
- config:
time: '00:05'
id: '4'
type: TimeOfDay
I won’t try to cover the full syntax here, but the point is that a YAML rule can do anything that a OH Rule can, including use Rule templates, which none of the other file formats can (except JSON).
As I’ve explained above, nothing is removed, this is just an additional way to provision Rules from files. What I’m working on now is make the “export” function, that lets you save a Rule as YAML from the UI. So, if you so wish, you can easily “move” your managed rules to YAML files for easier backup/control.
No, there will be no “shared context” in YAML. If you want to have that, you will have to use one of the other ways to create rules. The whole “shared context” thing causes a lot of headache with threading, and isn’t possible here anyway. To have a “shared context”, there must be something residing outside the rule itself, that is still “available” to the Rule. To achieve that, the rule must be created by a script that can establish that context, and it must be established/created every time OH starts, before any of the Rules that rely on it comes into existence.
You can say that shared context is the major benefit of using scripts to create rules, but it also has a lot of drawbacks. I’m not entirely sure what people use this shared context for, since it’s not threadsafe to do so, but if you just want to share parts of the Rule between multiple Rules, you can do that using Rule templates instead, where “the Rule” only defines the “variables” that is then injected into the Rule itself, allowing essentially writing one Rule that can handle many different Items or any other kind of “reuse”. In addition, @jimtng is working on a very powerful YAML preprocessor that let you do all kind of evaluations and substitutions when you build the YAML itself. I don’t know if these things completely “replace” the need for shared context, it certainly doesn’t if you want multiple Rules to “work together”, but as said before, doing so isn’t thread-safe anyway, so it will be buggy. If you use the shared cache, you can make Rules “work together” without the shared context, so that can be done from YAML as well.
There are many, many other things that they can’t do. Look at the YAML examples I posted, and you will see some of all the things that aren’t available from Rules DSL.
I don’t follow your logic here. A Rule and a script are two very different concepts, a Rule has a certain structure, and can contain any number of scripts, both as Conditions and Actions. A script is just a script, a piece of code. You can “create a Rule programmatically” from a script, which is done when you use e.g. the “helper libraries” to create Rules, but then the Rule and the script aren’t the same. The Rule is what the script creates, not the script itself.
This is actually an important point that I hadn’t thought about (I had actually forgotten that DSL could have “shared context”). I’m wondering how that looks today, and I must probably make some checks to prevent exporting these to YAML (as they won’t work). Some example DSL Rules with shared context would be helpful if anybody has some lying around, as I must study exactly how to “detect” this. It must be treated similarly to SimpleRules which are restricted with what you can do.
What is the difference between a managed rule and a YAML rule, apart from the obvious that the YAML rule is not managed? For me here it is unclear what is supposed to change and why. Is the idea to store UI rules not in jsondb, but as YAML instead, possibly handling them afterwards as read-only rules? If this is the case then indeed UI rules can have “timeofday trigger” and DSL Rules cannot specify “hh:mm” as timer.TimeOfDayTrigger, but the available cron expression trigger offers the same functionality.
In UI rules, whose body is “DSL”, the term “DSL Rule” is not involved, but the body is “DSL Script”, because it is not handled by the org.openhab.core.model.rule* bundles, but by the org.openhab.core.model.script* bundles.
Yes, essentially, a YAML rule is a file-based, read-only “UI Rule” if you want to call it that. It doesn’t have anything to do with the UI, but it’s a rule that can utilize all the possibilities of a OH Rule.
Yes, this is what I tried to describe above, we just don’t have precise terminology here, but there are two different concepts, it’s the “DSL file format” and the “DSL script language”. The “DSL file format” is a very limited subset of the OH Rule structure.
edit: Please note, there’s no “instead” here, it’s “in addition”. YAML won’t replace the JSON DB, it’s just something you can choose to use as an alternative. The YAML format also has some other benefits, like that you can define related Things, Items and Rules in the same file if you think that makes more sense.
Timers mostly. You need to store the timer somewhere or else you lose access to it when the rule exits. You can’t cancel it later from the same rule or a different rule, for example.
Other stuff like timestamps, counters, locks, flags, etc. are also stored there. For Rules DSL there’s also lambdas and the other languages have functions. But as you point out, the cache is there for most of that. Even though it’s been around for a while, I think few file rules developers know about it or use it.
That really wasn’t a practical option until very recently.
It also doesn’t help with imports in Rules DSL. If I created a Rules DSL rule that uses java.nio.* to read/write to some files, that would not be supportable in YAML or as a managed rule (i.e. in the UI).
Any variable defined before the first rule in the file is the “global” context. So you could look for any var or val between the last import (or the top of the file if there is none) and the first line that starts with rule.
That is unless it’s been changes since the last time I used it to allow defining “global” rules between the rules. But in that case you’d just need to look for val and var between an end and rule in addition to before the first rule. But I’m pretty sure this is still not allowed and all such variables and constants must be defined before the first rule in the file.
If you wanted to get fancy you could figure out how to replace those with use of the cache instead, but I’m not sure the juice is worth the squeeze.
To be fair, I think this statement is true of all the rules language’s file formats. JS and jRuby have some concept of conditions, but they are implemented wholly in the helper libraries (as I understand it) and not defining a condition like you could in the YAML and in the UI. Rules DSL is just a little more limited than the rest, but they all have their limitations. I think the new YAML format is the only one that will have the ability to define/use all aspects of an OH rule.
Isn’t it possible to do the imports inside the “script section”? I have no experience doing this, so I’m just asking. I understand that shared context is impossible to achieve, but imports doesn’t sound like they would, at least in theory, be impossible.
That’s definitely way beyond anything I want to try to achieve
I need to figure out how to detect this after the rule has been parsed, not in the “source”. That’s why I need some examples, so I can let the system parse them and inspect the result. When “decisions are made” as to what can and cannot be done, I don’t have the source, I just have the Rule instance that results from the parsing.
Any hints as to where I can find one is appreciated, because if I am to hack something together myself, it’s hard to know if what I do is actually “working” and that the result is representative.
Absolutely, this is what makes it versatile and a good alternative to managed rules. I’m not so sure that we’ll see a lot of “script created rules” moved to YAML. But, the ability to “create” the rule in the UI and then “save it” as YAML is also something I would think that could be worth something for some users.
