JSONPATH Transformation Service vs. JSONPATH

Add-ons Other
,
1

Purpose

  1. Explain recent history of the functionality, along with faults
  2. Describe a breaking change that is committed to ESH and is currently in the OH 2.3.0 snapshot
  3. Describe what I have proposed in the issue that the breaking PR was trying to resolve
  4. My current workaround

History of functionlity from OH 2.0.0 to OH 2.3.0 snapshot 1194 (January 23) [approximately]
The JSONPATH Transformation Service uses Jayway JSONPATH. When the path is definite, JSONPATH returns a string value, and the JSONPATH transform would return a string. The JSONPATH transform functionality was inline with what is returned by JSONPATH.

When the path is indefinite, JSONPATH returns a List, and the JSONPATH transform would return List.toString. This is not inline with what is returned by JSONPATH and caused a number of issues. For items/labels/bindings, indefinite paths could not be used (ESH issue 4862). For use in a rule, the List.toString that was returned by the transform could be manipulated to return it back to a List. Depending on what was being returned, even double quotes would need to be removed from the result.

val List<String> zwaveThingListFinal = transform("JSONPATH","$..[?(@.UID =~ /zwave:device.*/)].UID",thingResultFinal).replace("[","").replace("]","").replace("\"","").split(",")

Breaking change in ESH
The current OH 2.3.0 snapshots include the ESH PR to resolve the issue linked above. This will allow indefinite paths to be used for items/labels/bindings, if only one element was in the list returned by JSONPATH. If more than one element is returned in the list, the transform will return NULL and log the warning “JsonPath expressions with more than one result are not allowed, please adapt your selector. Result: {}”.

Proposed change
The ESH PR removed more functionality that it created. IMO, the JSONPATH transform should return a list if an indefinite path is used, just as is returned by JSONPATH. This would simplify the use of the transform in rules, but it would break the current functionality for items/labels/bindings unless something could be written to handle this scenario, as the ESH PR has done. A rule could also be used with the JSONPATH transform to populate item values.

Workaround
I’m sure others will run into this, especially if the current change makes it into the production OH build. Until this functionality is implemented, I am using jq where the ESH change had broken several of my rules, but I don’t see why it couldn’t also be used with an EXEC transform for items/labels/bindings too. Hmmm, JQ Transformation Service… :thinking:

val List<String> thingListFinal = executeCommandLine("/bin/sh@@-c@@/usr/bin/curl -s --connect-timeout 10 -m 10 -X GET --header \"Accept: application/json\" \"http://<ohserver>:8080/rest/things\" | /usr/bin/jq '.[].UID | select(contains(\"zwave:device\"))'",5000).split("\n")

At the very least, I hope this helps some people out, but I’m really hoping this post will incite some discussion in the ESH issue too. As of yet, I have not received any response.

3 Likes
2

Would it not be sufficient to just remove that?
If the jsonpath is not a single element it should return the list and this would be obvious for the user that his jsonpath ist not a single element.
It will make it necessary to check if it is a single element or not in the rule but keep the transformation working.

3

That is a partial solution, which would bring back the previous awkward functionality for rules that has caused so many people trouble with using the transform. I feel that the best solution is for the JSONPATH transform to actually behave like JSONPATH.

4

The problem with this change, is that functionality is removed because of convenience in one usecase.

I have not much knowledge in Jsonpath, but as far as i understand, with this change the ability to process a resultset with more than one element is no longer existing (*), just to make selection of a single item easier. It’s quite easy wrap the list result into a regex transformation to get the desired result in the original issue.

(*) without the unix tool, windows is out of luck

5

So the best solution would be a JSONPATH and a JSONOBJ filter so jsonpath stays as full fledged jsonpath and JSONOBJ would return an evaluated object.

This would be my idea as a solution.

As the convinience to be able to parse a object out of the json is very high.

6

Edit: Ok, now i understand. Didn’t see the issue description at first. :wink:

Although i would name the JSONOBJ better JSONVALUE.

Still, i don’t think an inflation of transformation services is the best solution. As i said earlier, the desired result on which the first issue was based, could be easily solved with a regex transformation.

At least as far as i understand.

7

Then this would increase the effort for everyone to add an additional regex.

What then could make life easier would be nested transformation.

Additionally
http://goessner.net/articles/JsonPath/

Please note, that the return value of jsonPath is an array, which is also a valid JSON structure. So you might want to apply jsonPath to the resulting structure again or use one of your favorite array methods as sort with it.

So my argument is make jsonpath to be jsonpath. Add a wrapper which brings the functionality all are accustomed back with an additional transformation or an aditional parameter. Which defaults to Yes.

One notice, it as wrong in OH1 and in the early implementation of jsonpath. And know there is allready a big ruckus why not take this chance to make it as jsonpath is defined.
Imho if its called jsonpath it should be as jsonpath is defined, yes this is not always convinient.
But writting a dokumentation for the jsonpath transformation just to list all differences is also not nice. And every new user will search for jsonpath and then think ah this jsonpath is different!

transform("JSONPATH", "$.path", jsonstring, evaluate)

So JSONOBJ is meant to remove the object out of the array, and maybe remove the enclosing double quotes. the result can then be set to a Number, string etc.

8

Well, not for everyone. :smiley:

For my use case, the current stable implementation is good. I use a transformation to strip a huge json array down to its relevant ids. If the resulting id-string has changed, a rule triggers to process the array.

@5iver, if the JSONPATH result would be adjusted to a List in case of an indefinite path, what would happen, if this transformation is used for a simple String item? Would implicitely toString() called on the List? Or will an exception be thrown?

Nested transformation would be great, not only for this case. :wink:

9

Should i write a feature request? I have the feeling a spammed smarthome/openHAB in the last days. :joy:

10

I’m not so sure it would be an easy transform with a regexp… JS would probably be easier.

I keep having a similar thought. I assume you can’t currently stack transforms, but that would be a good thing to test…

That sounds promising, but I think nested transformations would be a better option, since it could be used with all transformations.

11

Maybe just reuse the JSONOBJ one and change the title?

12

Ok, didn’t think of a wrapper earlier, just had the mind set to many new transformation services, one returning a list, the other a string next a float and a bool. Sometimes blind…

13

\d+\.\d+

Or am missing something? Original use case was an indefinite path resulting in definitive one temperature value.

Nested transformation would be nice, but thinking about it, i am not sure about the readability of those in items files. Thinking of PaperUI, i can’t think of a useful editing method.
Another idea to solve this, would be an “item binding”. A binding which uses other items as source, where another transformation could be applied. Step-by-step.

14

Testing JSONPATH and REGEX in a rule.

rule "my Test no 1234"
when
    Item TestScenes changed
then

    val TestString1 ="{ \"temperature\": \"23.2\" }"

    val result1 = transform("JSONPATH", "$.temperature", TestString1)
    logInfo("JsonPath", result1.toString)

    val result2 = transform( "REGEX",
                            "s/\\[\\s*\"?(.*?)\"?\\s*\\]/$1/g",
                            transform("JSONPATH", "$.temperature", TestString1) 
                            )
    logInfo("JsonPath", result2.toString)


    val TestString2 = "{
                        \"firstName\": \"John\",
                        \"lastName\" : \"doe\",
                        \"age\"      : 26,
                        \"address\"  : {
                            \"streetAddress\": \"naist street\",
                            \"city\"         : \"Nara\",
                            \"postalCode\"   : \"630-0192\"
                        },
                        \"phoneNumbers\": [
                            {
                            \"type\"  : \"iPhone\",
                            \"number\": \"0123-4567-8888\"
                            },
                            {
                            \"type\"  : \"home\",
                            \"number\": \"0123-4567-8910\"
                            }
                        ]
                        }"

    val result3 = transform("JSONPATH", "$.phoneNumbers[:1].type", TestString2)
    logInfo("JsonPath", result3.toString)

    val result4 = transform( "REGEX",
                            "s/\\[\\s*\"?(.*?)\"?\\s*\\]/$1/g",
                            transform("JSONPATH", "$.phoneNumbers[:1].type", TestString2) 
                            )
    logInfo("JsonPath", result4.toString)

    Test_3.postUpdate(TestString1)
    Test_4.postUpdate(TestString2)
end
end

String Test_3 "[JSONPATH($.temperature):%s]"  // works but is no an array ["23.3"] it's 23.2
String Test_4 "[JSONPATH($.phoneNumbers[:1].type):%s]"  //does not work

Nested transformation in labels would also be a nice feature.

String Test_3 "[JSONPATH($.temperature):REGEX(s/\\[\\s*\"?(.*?)\"?\\s*\\]/$1/g):%s]" // would be nice
15

I would think that an exception would be thrown. I don’t know where in the code that data from a transform is handled for an Item. Maybe it is here that the first element of a List could be taken? I would think the same exception would occur when any transform returned an Object type that couldn’t be cast to the Item type.

:bulb:! If I’m understanding this correctly, it looks like TransformationService.java and Transformation.java are expecting all transforms to return a String! The plot thickens…

That looks like it would return all of the temperatures. @ThomDietrich (I hope that’s you!) was looking for the “Temperature” value of a sibling that has an “Id” value of “000005FB25D0”. It may be possible with a regexp, but I think JS would be an easier alternative.

16

I thought about using the regex on the result of the jsonpath transformation. To extract the value from the returned list.

17

After thinking through this again, and assuming there is a very real reason why transforms need to return a String, it seems to me that the best option right now is to revert the commit from PR 4951, and if JSONPATH is returning a List, continue doing the List.toString conversion but then also strip the square brackets. This would return a String usable for items/labels/bindings (this resolves Issue 4862), and it could easily be converted back to a List in a rule. This still is not returning a true JSONPATH result, but I think it is the closest we can currently get. There may still be some cases where double quotes would be in the String, but they could be removed, if needed.

1 Like
18

Further testing shows.

  1. Nice format breaks JSONPATH in label transformation.
// Is working in label and rule transformation
val TestString1 =   "{ \"temperature\": \"23.2\",\"lastName\" : \"doe\" }"

// Only works in rule, in label transformation whole string is displayed.
val TestString1 =   "{ 
                    \"temperature\": \"23.2\",
                    \"lastName\" : \"doe\"
                    }"
  1. Some jsonpaths work in rules put not in the label.
val TestString2 = "{ \"firstName\" : \"John\", \"lastName\" : \"doe\",\"age\" : 26, \"address\" : { \"streetAddress\" : \"naist street\",\"city\" : \"Nara\", \"postalCode\" : \"630-0192\" }, \"phoneNumbers\" : [ { \"type\" : \"iPhone\", \"number\" : \"0123-4567-8888\"}, { \"type\" : \"home\", \"number\" : \"0123-4567-8910\" } ] }"


// works
String Test_4 "[JSONPATH($.phoneNumbers):%s]" 
// does not work in label but in rule.
String Test_4 "[JSONPATH($.phoneNumbers..[?(@.type == 'iPhone')].number):%s]" 
String Test_4 "[JSONPATH($.phoneNumbers[:1].type):%s]"
19

Which OH version are you using?

20

2.2.0