OH4 Blockly Persistence Calculation error

Mark_VG · May 24, 2023, 8:30am

Hi All

Current SNAPSHOT - openHAB 4.0.0 Build #3470

I had a similar Blockly Script to this working yesterday (simplified to demonstrate the issue), but now cannot get the script to work as required. (I am getting NaN as result for any calculations). I have tried all the tricks I can see to try and work out what is wrong - any assistance would be greatly appreciated.

Which generates:

var Previous_Y1, Previous_Y2, m1;


//log:set DEBUG org.openhab.automation.script.ui.Test_Persistence
Previous_Y1 = items.getItem('Shelly_UNI_Persist_1Min_Voltage_ADC').history.historicState(time.ZonedDateTime.now().minusMinutes(0))?.state;
Previous_Y2 = items.getItem('Shelly_UNI_Persist_1Min_Voltage_ADC').history.historicState(time.ZonedDateTime.now().minusMinutes(1))?.state;
console.error(('Previous Y1   ' + String(Previous_Y1)));
console.error(('Previous Y2   ' + String(Previous_Y2)));
m1 = Previous_Y1 * 1;
m1 = Previous_Y1 - Previous_Y2;
console.error(('M1 is ' + String(m1)));
console.error(('M1 is ' + String(m1)));

The output I get is:

10:14:56.320 [ERROR] [.openhab.automation.script.ui.Test123] - Previous Y1   27.63 V
10:14:56.321 [ERROR] [.openhab.automation.script.ui.Test123] - Previous Y2   27.63 V
10:14:56.322 [ERROR] [.openhab.automation.script.ui.Test123] - M1 is NaN
10:14:56.322 [ERROR] [.openhab.automation.script.ui.Test123] - M1 is NaN

So the values are retrieved correctly from persistence, however any calculation with those Variables fail and produce NaN, even a simple multiply by 1.

I also tried the UoM blocks and got errors.

var Previous_Y1, Previous_Y2, m1;


//log:set DEBUG org.openhab.automation.script.ui.Test_Persistence
Previous_Y1 = items.getItem('Shelly_UNI_Persist_1Min_Voltage_ADC').history.historicState(time.ZonedDateTime.now().minusMinutes(0))?.state;
Previous_Y2 = items.getItem('Shelly_UNI_Persist_1Min_Voltage_ADC').history.historicState(time.ZonedDateTime.now().minusMinutes(1))?.state;
console.error(('Previous Y1   ' + String(Previous_Y1)));
console.error(('Previous Y2   ' + String(Previous_Y2)));
m1 = 1 * 1;
m1 = Previous_Y1 - Previous_Y2;
console.error(('M1 is ' + String(m1)));
console.error(('M1 is ' + String(m1)));
m1 = (.multiply(1));
console.error(('UoM M1 is ' + String(m1)));

10:25:12.511 [ERROR] [.internal.handler.ScriptActionHandler] - Script execution of rule with UID 'Test123' failed: org.graalvm.polyglot.PolyglotException: SyntaxError: <eval>:13:6 Expected an operand but found .
m1 = (.add(1));
      ^

The only changes I can think I made was to remove RRD4j and MAPDB as persistence services. I am now using InfluxDB - but that has been my default for a few days now.

rlkoshak · May 24, 2023, 1:51pm

That’s not a number. As in everywhere else in JS Scripting’s helper library, the .state of an Item or HistoricItem is a String.

Furthermore, that state carries units.

You are on the right track in trying to use the UoM blocks but you need to actually have UoM values before you can use the UoM math block.

You need to wrap the result of the call to historic state inside a Qty block.

And then you need to use the UoM math block to multiply/subtract the two UoM variables.

Mark_VG · May 24, 2023, 2:13pm

What is really weird is that this was working…

I tried using your suggestion:

var Previous_Y1, Previous_Y2, m1;


//log:set DEBUG org.openhab.automation.script.ui.Test_Persistence
Previous_Y1 = 0;
Previous_Y2 = items.getItem('Shelly_UNI_Persist_1Min_Voltage_ADC').history.historicState(time.ZonedDateTime.now().minusMinutes(1))?.state;
console.error(('Previous Y1   ' + String(Previous_Y1)));
console.error(('Previous Y2   ' + String(Previous_Y2)));
m1 = 1 * 1;
m1 = Previous_Y1 - Previous_Y2;
console.error(('M1 is ' + String(m1)));
console.error(('M1 is ' + String(m1)));

This however produce 0 for “Previous Y1”?

16:08:41.683 [ERROR] [.openhab.automation.script.ui.Test123] - Previous Y1   0
16:08:41.684 [ERROR] [.openhab.automation.script.ui.Test123] - Previous Y2   27.64 V
16:08:41.684 [ERROR] [.openhab.automation.script.ui.Test123] - M1 is NaN
16:08:41.684 [ERROR] [.openhab.automation.script.ui.Test123] - M1 is NaN

These blocks where recently worked on - so wonder of there is an issue in my SNAPSHOT?

rlkoshak · May 24, 2023, 2:18pm

When though. If in OH 3.4, well Blockly now converts to a whole new version of JavaScript. If a snapshot of OH more than a week or so ago, the way that units are managed on Items was completely reworked and now perhaps Items that didn’t have units now do.

That looks like a bug in the UoM block. All I can suggest there is to file an issue.

It should generate something that looks like

Previous_Y1 = Quantity(items.getItem('Shelly_UNI_Persist_1Min_Voltage_ADC').history.historicState(time.ZonedDateTime.now().minusMinutes(0))?.state);

Though now that I notice it, why bother pulling anything out of persistence with a data that is effectively now? Just use the Item’s current state.

Mark_VG · May 24, 2023, 2:32pm

I had this working yesterday on the same 4.0.0 SNAPSHOT. All I have done in between is removed RRD4J and MAPDB. I have tried re-installing them but made no difference.

That is just a subset of what I am retrieving to demonstrate the issue. The actual rule is longer with more calculations in it. I need Now, 1 minute ago, 2 minutes ago, 3 Minutes ago.

rlkoshak · May 24, 2023, 2:40pm

So what’s configured as the default persistence?

Mark_VG · May 24, 2023, 2:42pm

InfluxDB, which it has been for about a week. I was running them together till InfluxDB had enough data for me to cut across.

rlkoshak · May 24, 2023, 2:52pm

And HistoricState is returning something valid?

If it worked before and doesn’t now, something changed. If all you changed was uninstalling two persistence engines you have your culprit. Something about removing those two engines changed what historic state is returning.

Mark_VG · May 24, 2023, 2:57pm

I am going to try reboot the server. Uninstalling MAPDB did cause OH to “crash”. I restarted and all seemed to be OK. But maybe not.

It does look like something other than persistence though:

var Previous_Y1, Previous_Y2, m1;


//log:set DEBUG org.openhab.automation.script.ui.Test_Persistence
Previous_Y1 = items.getItem('Shelly_UNI_Persist_1Min_Voltage_ADC').history.historicState(time.ZonedDateTime.now().minusMinutes(1))?.state;
Previous_Y2 = items.getItem('Shelly_UNI_Persist_1Min_Voltage_ADC').history.historicState(time.ZonedDateTime.now().minusMinutes(2))?.state;
console.error(('Previous Y1   ' + String(Previous_Y1)));
console.error(('Previous Y2   ' + String(Previous_Y2)));
m1 = 1 * 1;
m1 = (items.getItem('Shelly_UNI_Persist_1Min_Voltage_ADC').history.historicState(time.ZonedDateTime.now().minusMinutes(0))?.state) - (items.getItem('Shelly_UNI_Persist_1Min_Voltage_ADC').history.historicState(time.ZonedDateTime.now().minusMinutes(1))?.state);
console.error(('M1 is ' + String(m1)));
console.error(('M1 is ' + String(m1)));

Should return 1 for m1, but it is NaN:

16:53:37.092 [ERROR] [.openhab.automation.script.ui.Test123] - Previous Y1   27.6 V
16:53:37.093 [ERROR] [.openhab.automation.script.ui.Test123] - Previous Y2   27.63 V
16:53:37.116 [ERROR] [.openhab.automation.script.ui.Test123] - M1 is NaN
16:53:37.117 [ERROR] [.openhab.automation.script.ui.Test123] - M1 is NaN

rlkoshak · May 24, 2023, 3:13pm

No, you set m1 to 1*1 and then set m1 to (items.getItem('Shelly_UNI_Persist_1Min_Voltage_ADC').... before logging m1 for the first operation. You wipe out the 1 before you get a chance to log it.

Mark_VG · May 24, 2023, 3:15pm

Sorry, yes, all the changes must have missed that one - thanks for pointing out.

Mark_VG · May 24, 2023, 3:36pm

Yes, the logs show valid values. Fails on the calulations.

Mark_VG · May 24, 2023, 3:58pm

Seems to be an issue with the UoM QTY blcoks?

Produces:

m1 = (.subtract());
m1 = Previous_Y1 - Previous_Y2;

So I am pretty sure the first calculation is wrong. Could this be the root of the problems?

And even

Produces:

Previous_Y1 = 0;
Previous_Y1 = items.getItem('Shelly_UNI_Persist_1Min_Voltage_ADC').history.historicState(time.ZonedDateTime.now().minusMinutes(1))?.state;

And again the QTY version is incorrect.

EDIT:

Logged issue:

github.com/openhab/openhab-webui

[blockly] Latest UoM QTY blocks with variables no longer work

opened 05:49PM - 24 May 23 UTC

Mark-VG

bug main ui

## The problem Running on openHAB 4.0.0 Build #3470 The latest UoM blocks …produce incorrect code when working with variables. ![image](https://github.com/openhab/openhab-webui/assets/84638697/99c0ea87-cf6c-410b-a9b2-eaa77f89f73e) Produces the following code ``` var Test1; Test1 = Quantity('10 V'); console.debug((Quantity('100 V').add())); console.debug((Quantity('100 V').add(Quantity('10 V')))); ``` As can be seen from this example the first formula has the variable missing and produces an invalid function. ``` console.debug((Quantity('100 V').add(***Missing***))); ``` ## Expected behavior Produced code from Blockly to be correct and functional ## Steps to reproduce Create Blockly Script as per above screen shot and check resultant code. Sample rule included below. ## Your environment ``` Configuration folder C:\openHAB\conf User data folder C:\openHAB\userdata Logs folder C:\openHAB\userdata\logs Operating system Windows 10/10.0 (amd64) Java Runtime Azul Systems, Inc. 17.0.7 (Zulu17.42+19-CA) openHAB 4.0.0 Build #3470 ``` Test Rule Code ``` Rules Loading... Title of the rule status badge Tags, Schedule, Scene... Description of the rule Title of the rule status badge Tags, Schedule, Scene... Description of the rule Title of the rule status badge Tags, Schedule, Scene... Description of the rule Title of the rule status badge Tags, Schedule, Scene... Description of the rule Title of the rule status badge Tags, Schedule, Scene... Description of the rule Title of the rule status badge Tags, Schedule, Scene... Description of the rule Title of the rule status badge Tags, Schedule, Scene... Description of the rule Title of the rule status badge Tags, Schedule, Scene... Description of the rule Title of the rule status badge Tags, Schedule, Scene... Description of the rule Title of the rule status badge Tags, Schedule, Scene... Description of the rule Title of the rule status badge Tags, Schedule, Scene... Description of the rule Title of the rule status badge Tags, Schedule, Scene... Description of the rule Title of the rule status badge Tags, Schedule, Scene... Description of the rule Title of the rule status badge Tags, Schedule, Scene... Description of the rule Title of the rule status badge Tags, Schedule, Scene... Description of the rule Title of the rule status badge Tags, Schedule, Scene... Description of the rule Title of the rule status badge Tags, Schedule, Scene... Description of the rule Title of the rule status badge Tags, Schedule, Scene... Description of the rule Title of the rule status badge Tags, Schedule, Scene... Description of the rule Title of the rule status badge Tags, Schedule, Scene... Description of the rule Test UoM 1 configuration: {} 2 triggers: 3 - id: "1" 4 configuration: 5 itemName: Testing_Switch 6 type: core.ItemStateChangeTrigger 7 conditions: [] 8 actions: 9 - inputs: {} 10 id: "2" 11 configuration: 12 blockSource: <xml 13 xmlns="https://developers.google.com/blockly/xml"><variables><variable 14 id="CT7z)^UGiUA(a76IsM:$">Test1</variable></variables><block 15 type="variables_set" id=":iV4T[iuFW+cy6KC([h?" x="45" y="49"><field 16 name="VAR" id="CT7z)^UGiUA(a76IsM:$">Test1</field><value 17 name="VALUE"><block type="oh_quantity" id="xPzls{{)@9)7x*}c|mRv"><value 18 name="quantity"><shadow type="text" id="AuQ73xk/%R6$@X.+({qU"><field 19 name="TEXT">10 W</field></shadow><block type="text" 20 id="D+[g*~hRODXW5f7DXLiK"><field name="TEXT">10 21 V</field></block></value></block></value><next><block type="oh_log" 22 id="mEiA99CDb5q1T;vbP;vy"><field name="severity">debug</field><value 23 name="message"><shadow type="text" id="Q(`Wk5N[n~sZc@]d#(%7"><field 24 name="TEXT">abc</field></shadow><block type="oh_quantity_arithmetic" 25 id="qS`I@wv(9,`Ty#RrbGrL"><field name="operand">add</field><value 26 name="first"><shadow type="oh_quantity" 27 id="%D`X+rWrZ7K5/3V.$`u8"></shadow><block type="oh_quantity" 28 id="[}bkJgD0298LLTrufH=k"><value name="quantity"><shadow type="text" 29 id="Rg*ihu5k76vFl5Vz[1_3"><field name="TEXT">10 W</field></shadow><block 30 type="text" id="MDRHDmjqlN*np=qhk/-K"><field name="TEXT">100 31 V</field></block></value></block></value><value name="second"><shadow 32 type="oh_quantity" id="N]Ud`r+6Qoqcf6=b)TpZ"></shadow><block 33 type="variables_get" id=";GyI7hnYXW^u]]S+M:6K"><field name="VAR" 34 id="CT7z)^UGiUA(a76IsM:$">Test1</field></block></value></block></value><next><block 35 type="oh_log" id=":UXrU[e2pMocUi0=LE`k"><field 36 name="severity">debug</field><value name="message"><shadow type="text" 37 id="Q(`Wk5N[n~sZc@]d#(%7"><field name="TEXT">abc</field></shadow><block 38 type="oh_quantity_arithmetic" id="`Lw6!Ya.;:?9YtX!EbS["><field 39 name="operand">add</field><value name="first"><shadow type="oh_quantity" 40 id="%D`X+rWrZ7K5/3V.$`u8"></shadow><block type="oh_quantity" 41 id="Gsi=M:#=o:QZ4E+%@PA["><value name="quantity"><shadow type="text" 42 id="Rg*ihu5k76vFl5Vz[1_3"><field name="TEXT">10 W</field></shadow><block 43 type="text" id=",^(%:k7oOoNNDjpz2yR]"><field name="TEXT">100 44 V</field></block></value></block></value><value name="second"><shadow 45 type="oh_quantity" id="N]Ud`r+6Qoqcf6=b)TpZ"></shadow><block 46 type="oh_quantity" id="U?jd4P8Jd0^wa=c4s!N7"><value 47 name="quantity"><shadow type="text" id="AuQ73xk/%R6$@X.+({qU"><field 48 name="TEXT">10 W</field></shadow><block type="text" 49 id="uRHpZ@.NQ6WnBg+vPL7u"><field name="TEXT">10 50 V</field></block></value></block></value></block></value></block></next></block></next></block></xml> 51 type: application/javascript 52 script: | 53 var Test1; 54 55 56 Test1 = Quantity('10 V'); 57 console.debug((Quantity('100 V').add())); 58 console.debug((Quantity('100 V').add(Quantity('10 V')))); 59 type: script.ScriptAction 60 ``` Community Forum Topic for further information: https://community.openhab.org/t/oh4-blockly-persistence-calculation-error/146841/13

rlkoshak · May 24, 2023, 6:55pm

Yes, that’s why I said above it looks like a bug and an issue should be filed. The UoM blocks are not producing valid code. Hopefully your issue will be seen and fixed quickly.

Mark_VG · May 25, 2023, 7:12am

Thanks for the support.

So to try and get back up and running I have added inline scripts to perform the required functions.

I was expecting that once the Quantity is assigne dto a variable, that that Quantity would be carried through to the later calculations etc - But it appears that this is not the case - I end up with 0 as a result?

So I have added the following inline script to retrieve the persisted values:

PreviousY1 = Quantity(items.getItem('Shelly_UNI_Persist_1Min_Voltage_ADC').history.historicState(time.ZonedDateTime.now().minusMinutes(0))?.state);
PreviousY2 = Quantity(items.getItem('Shelly_UNI_Persist_1Min_Voltage_ADC').history.historicState(time.ZonedDateTime.now().minusMinutes(1))?.state);
PreviousY3 = Quantity(items.getItem('Shelly_UNI_Persist_1Min_Voltage_ADC').history.historicState(time.ZonedDateTime.now().minusMinutes(2))?.state);

And the following to perform the calulations:

m1 = (Quantity(PreviousY1).subtract(Quantity(PreviousY2)));
m2 = (Quantity(PreviousY2).subtract(Quantity(PreviousY3)));

I would however like to round the result to 1 decimal place, which also requires the Quantity again, so trying to use inline script with (and a few variants):

m1 = (Quantity(m1).toFixed(1));

But I always get an error:

09:11:44.186 [ERROR] [ab.automation.script.javascript.stack] - Failed to execute script:
org.graalvm.polyglot.PolyglotException: TypeError: (intermediate value)(...).toFixed is not a function
        at <js>.:program(<eval>:17) ~[?:?]
        at org.graalvm.polyglot.Context.eval(Context.java:399) ~[?:?]
        at com.oracle.truffle.js.scriptengine.GraalJSScriptEngine.eval(GraalJSScriptEngine.java:458) ~[?:?]
        at com.oracle.truffle.js.scriptengine.GraalJSScriptEngine.eval(GraalJSScriptEngine.java:426) ~[?:?]
        at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:262) ~[java.scripting:?]
        at org.openhab.automation.jsscripting.internal.scriptengine.DelegatingScriptEngineWithInvocableAndAutocloseable.eval(DelegatingScriptEngineWithInvocableAndAutocloseable.java:53) ~[?:?]

I am sure my syntax is wrong - But I have no idea why.

rlkoshak · May 25, 2023, 3:00pm

I don’t know if there is support for toFixed() on a QuantityType. Quantity is just a wrapper around the Java QuantityType. I don’t see such a function anywhere.

Why do you need to round it in the rule? You shouldn’t round numbers until the last moment (e.g. when you display it on a UI). This is particularly important if you are doing further calculations with that number because rounding errors can become huge in only a couple of operations.

If you must round/truncate here, you’re going to have to convert it to a number and then back to a Quantity. m1’s already a Quantity so it would be this convoluted path.

m1 = Quantity(String(m1.toFloat().toFixed(1)), 'V')

Mark_VG · May 25, 2023, 5:04pm

Thanks for the help so far Rich.

I think maybe part of my issue (besides the “broken” blocks, is my understanding of String vs Quantity vs Number and where they need /have to be used.

So from what I understand:

Having read Units Of Measurement | openHAB

An Item STATE is stored as a string, so before any “calculations” can be performed it needs to be converted to a “numerical” format being either a Quantity or a Number?

A Quantity is a number with a UoM attached, such as 100kW, 100V, etc?

A Number is a plain number with no UoM attached, such as 100, 200, 159.764 etc?

What I don’t seem to get is:

If a state of an item is retrieved from the item itself or from persistence and stored in a variable, it is still a string
Unless it is converted to a quantity? Is it then always a Quantity going forward in that variable? So in Blockly one would have to use the UoM blocks - which are currently broken (even in 4.0.0.M3)
The Plain Math Blocks are only used for numbers with no UoM?

So this extract:

Which produces (and works):

temp = (3.123).toFixed(1);

Resulting in:

3.1

Would not work for a Quantity type? It would only work for a Plain Number?

Maybe I need to back off a bit and wait for the UoM blocks to be fixed and then try again?

rlkoshak · May 25, 2023, 5:15pm

Not quite.

Blockly “compiles” to GraalVM JavaScript Scripting: JavaScript Scripting - Automation | openHAB

The Item class has four “state” properties:

.state - always a String
.numericState - always a JavaScript number or null if the Item’s state cannot be represented as a number
.quantityState - always returns a Quantity or null if the Item’s state cannot be represented as a Quantity
.rawState - the original Java State Object

You can get all but this last one using the “get X from Item” block.

But you can only get state from the “get the historic state of the Item…” block. So if you want to do math with it, you need to convert it to a numericState or a quantityState.

No, it’s just the .state property of the Item is always a String. And by default Blockly uses .state unless told otherwise and it doesn’t have the option to be told otherwise when getting the historic state.

Yes with the caveat that if you reassign the variable some other value, that new value might not be a Quantity.

Correct.

Correct. If you have a Quantity, you can only use the block from the Units of Measurement section to do comparisons and math.

stefan.hoehn · May 29, 2023, 9:14pm

It is a pity I have only seen this thread by chance.

When you say that the UOM blocks are broken, are you referring to the variable usage?

I have provided a fix for that yesterday

github.com/openhab/openhab-webui

[blockly] Fix UoM variable support

openhab:main ← stefan-hoehn:blockly_fix_uom_vars

opened 08:12PM - 28 May 23 UTC

stefan-hoehn

+1 -0

Regression from #1864. Fixes #1907. fixes variable support for the blockly u…nit of measurement blocks which did not work at all -> var defaults to item object Blockly cannot detect the type that is contained in a variable so it expects an item object. This allows to iterate over a group of item members which returns a list of item objects

because I stumbled over it myself during documenting the latest changes.
Sorry for the inconvenience.

Update: Now I saw your filed issue on github: [blockly] Latest UoM QTY blocks with variables no longer work · Issue #1907 · openhab/openhab-webui · GitHub

Unfortunately I need to check this out again.

The problem is that blockly cannot detect the type of the content of the variables which is a pity which is why I need to make assumptions, so I went for

Blockly cannot detect the type that is contained in a variable so it expects an item object.
This allows to iterate over a group of item members which returns a list of item objects

which of course is debatable…

Mark_VG · June 1, 2023, 11:12am

I just updated to Build #3485 which seems to resolve the issues with the UoM Blcoks using variables.

Thank You.