Url not accepted by http binding

Guido_van_Haasteren · January 10, 2019, 3:57pm

Hi, I must be doing something wrong but I don’t see it.
I already have this url working in my http.cfg file:

Buienradar_JSON.url=https://api.buienradar.nl/data/public/2.0/jsonfeed
Buienradar_JSON.updateInterval=180000

but adding this url:

Neerslag.url=https://gpsgadget.buienradar.nl/data/raintext/?lat=51.82&lon=4.6
Neerslag.updateInterval=60000

results in this warning/error:

[WARN ] [ab.binding.http.internal.HttpBinding] - given config key 'Buienradar_neerslag' does not follow the expected pattern '&lt;id&gt;.&lt;url|updateInterval&gt;'
[WARN ] [ab.binding.http.internal.HttpBinding] - given config key 'Buienradar_neerslag.updateinterval' does not follow the expected pattern '&lt;id&gt;.&lt;url|updateInterval&gt;'

This doesn’t seem to be related to the encoding of the ? and & because when I strip the url just after ‘raintext’ I still get this error:

[WARN ] [ab.binding.http.internal.HttpBinding] - given config key 'Buienradar_neerslag' does not follow the expected pattern '<id>.<url|updateInterval>'
[WARN ] [ab.binding.http.internal.HttpBinding] - given config key 'Buienradar_neerslag.updateinterval' does not follow the expected pattern '<id>.<url|updateInterval>'

It’s probably some small detail I am missing, but I cannot find it. I hope someone spots my error!

EDIT: I now see that the url in the error is the same both times. So the services file is not read in correctly after saving the file, therfore the encoding probably is the reason. Then my question changes to: how to escape the ? and & correctly. I already tried % infront, but if the file is read out of cache and not refreshed, I don’t know what I effectively tried anymore.

H102 · January 10, 2019, 4:07pm

Have you tried using \\ to escape the ? and &

Guido_van_Haasteren · January 10, 2019, 4:11pm

Yes, but I now realise my services/http.cfg file is not refreshed correctly. I have just put in this url, but I still get the old url-error!

https://gpsgadget.buienradar.nl/data/raintext/lat%3D51.82%26lon%3D4.6

So maybe putting \ infront should work, but my test failed because of caching older http.cfg file.

H102 · January 10, 2019, 4:14pm

You may need to clean the cache after making file changes.

Commands for apt-get install

sudo systemctl stop openhab2

sudo openhab-cli clean-cache

sudo systemctl start openhab2

Guido_van_Haasteren · January 10, 2019, 4:42pm

Thanx, but I still get an error that is related to a version of the file the is no longer there. I guess the clean-cache procedure does not take http.cfg into account. I’ll try this way: https://community.openhab.org/t/http-binding-doesnt-seem-to-care-whats-in-the-http-cfg-file/39407

H102 · January 10, 2019, 4:50pm

Deleting /var/lib/openhab2/config/org/openhab/http.config would have been my next suggestion. When troubleshooting I try to avoid messing with any files outside of /etc/openhab2 but sometimes it’s needed.

Guido_van_Haasteren · January 10, 2019, 6:15pm

This works for deleting the cache, but it still doesn’t work. (sorry, had to edit this post, first I thought it was working)

[ERROR] [b.core.service.AbstractActiveService] - Error while executing background thread HTTP Refresh Service

java.lang.IllegalArgumentException: Invalid uri 'https://gpsgadget.buienradar.nl/data/raintext\?lat=51.82\&lon=4.6': escaped absolute path not valid

But if I paste this url in a browser and remove the , it does give me the desired data…

Guido_van_Haasteren · January 10, 2019, 7:46pm

Ok, using encoding like this does remove the errors scrolling over my log file, unless I put in the item. So I guess the error is caused by the item-definition:

String  Neerslag_nu "Neerslag_nu [%s]" {http="<[Neerslag:10000]" }

esteevens · January 10, 2019, 9:32pm

There seems to be something ‘special’ about the URL. I can’t get it to work either. I do not get the exact same message. If I host the ‘rain’ content statically on another site, it works. But if I set the url to the buienradar site it doesn’t. It’s not clear to me yet.

http.cfg:

http.items

The Item ‘Regen’ gets updated:

The Item ‘Regen_Echt’ does not work. It looks as if the URL doesn’t get parsed correctly somewhere:

Guido_van_Haasteren · January 11, 2019, 10:43am

Pity you cannot get it to work either, but huge THANX for taking your time to test it yourself! And you figured out how to regex to the first datapoint! That was the next thing on my todo list .
Actually I would like to extract the first three datapoints, each in their own Item and I thought I should do it with the split function. But as long as I don’t get the URL working, it’s no use.

Guido_van_Haasteren · January 16, 2019, 8:40pm

So I had some time experimenting again and tried the url without encoding again, combined with the Item defenition like @esteevens had (with the regex). Now I get this error which may explan something, but I don’t know what exacly:

Cookie rejected: "$Version=0; ARRAffinity=f634bd8f83bdc06d691b42f5d1eef43ea4d8d93a9970533b1b99a165ee1e4d1e; $Path=/; $Domain=br-gpsgadget.azurewebsites.net". Illegal domain attribute "br-gpsgadget.azurewebsites.net". Domain of origin: "gpsgadget.buienradar.nl"

Google helped me understand: the cookie is set for a different url then the url I got the info from.
So I changed the url to:

https://br-gpsgadget.azurewebsites.net/data/raintext?lat=51.82&lon=4.6

And that is working!!!
With the regex from @esteevens I can get the first datapoint. Now the next step is to get the second and third datapoint from this file that I get from the above url:

000|21:35
000|21:40
000|21:45
000|21:50
000|21:55
077|22:00
077|22:05

I will start reading about the split function, but any help is appreciated.
Thanx Eddy for setting me in the right direction.

noppes123 · January 17, 2019, 2:51pm

This works quite nice. Thanks!
Especially usefull now most weather API’s either have shutdown (like Yahoo) or moved behind a payed subscription.

I suppose a JavaScript transform or some complex rule is needed to process all the Neerslagintensiteit data. Will give it a try next week…

noppes123 · January 19, 2019, 4:44pm

OK. A first attempt at making something out of the ‘Neerslagintensiteit’ received from Buienradar (Dutch weather).
Assuming a String Item named WB_RainInput that receives the data from the http binding and an Item WB_RainJson to store the JSON formatted output in. Alternatively it could be done with a group of of Items, for instance. I’m sure there are better ways, but I had a hard time with DSL and Arrays.

rule "Parse rain intensity key-value pairs"
when
	Item WB_RainInput received update
then
	val tag = "Rain.Intensity.Test"
    logInfo(tag, ">>> Enter rule >>>")

	var String[] rainLines = WB_RainInput.state.toString.split("\\n")
	var String jsonOut = "{\"rainintensity\":["
	var i = 0
	for (String pair:rainLines)
	{
		val keyValue = transform("REGEX", "(.*)\\|.*", pair)
		val keyString = transform("REGEX", ".*\\|(.*)", pair)
		logInfo(tag, "key: {}, value: {}", keyValue, keyString)
		jsonOut += "{ \"intensity\": \" " + keyValue + "\", \"time\": \" " + keyString + "\" }"
		if (i < rainLines.length - 1)
			jsonOut += ", "
		else
			jsonOut += "]}"
		i++ 
	}
	WB_RainJson.postUpdate(jsonOut)
	logInfo(tag, "<<< Exit Rule with: {}", jsonOut)
end

Gives something like:

 { "rainintensity": [ { "intensity": " 000", "time": " 07:50" }, { "intensity": " 000", "time": " 07:55" }, { "intensity": " 090", "time": " 08:00" }, { "intensity": " 130", "time": " 08:05" }, { "intensity": " 077", "time": " 08:10" }, ... ] }

Skipped the calculation for now as that is the easy part.

Guido_van_Haasteren · January 19, 2019, 8:58pm

So this turns the weird text file into a json-file? That is cool. I am particular interested in the first three values, but I think I can manage to get them from the JSON.

esteevens · January 19, 2019, 9:06pm

I was wondering why you used the REGEX to split a rainline in 2 parts. A simple split should be sufficient I guess. I was playing with a JS transform this afternoon. I’ve adapted it based upon your code now.

String RainDataJson { http="<[buienradar:300000:JS(convertBuienradarRaindata2Json.js)]" }

The corresponding convertBuienradarRaindata2Json.js code:

(function(i) {
    var rainLines = i.split("\n");
    var rainintensity = [];
    rainLines.forEach(function(line)
    {
        var array = line.split("|");
        var data = {
            "intensity": Math.pow(10, (parseInt(array[0]) - 109) / 32),
            "time": array[1]
        }
        rainintensity.push(data);
    });
    
    var output = {};
    output.rainintensity = rainintensity;
    return JSON.stringify(output);
})(input)

I’m still playing with my fake data, because I cannot get it to work with the real URL. Not even the azure URL. Which Openhab version are you guys using? I’m on 2.4.0. This is the error I get:

2019-01-19 22:10:02.277 [ERROR] [b.core.service.AbstractActiveService] - Error while executing background thread HTTP Refresh Service
java.lang.NullPointerException: null
	at java.util.regex.Matcher.getTextLength(Matcher.java:1283) ~[?:?]
	at java.util.regex.Matcher.reset(Matcher.java:309) ~[?:?]
	at java.util.regex.Matcher.<init>(Matcher.java:229) ~[?:?]
	at java.util.regex.Pattern.matcher(Pattern.java:1093) ~[?:?]
	at org.openhab.io.net.http.HttpUtil.extractCredentials(HttpUtil.java:278) 226:org.openhab.core.compat1x:2.4.0]
	at org.openhab.io.net.http.HttpUtil.executeUrl(HttpUtil.java:181) ~[226:org.openhab.core.compat1x:2.4.0]
	at org.openhab.io.net.http.HttpUtil.executeUrl(HttpUtil.java:130) ~[226:org.openhab.core.compat1x:2.4.0]
	at org.openhab.binding.http.internal.HttpBinding.getCacheData(HttpBinding.java:423) ~[?:?]
	at org.openhab.binding.http.internal.HttpBinding.execute(HttpBinding.java:169) ~[?:?]
	at org.openhab.core.binding.AbstractActiveBinding$BindingActiveService.execute(AbstractActiveBinding.java:144) ~[226:org.openhab.core.compat1x:2.4.0]
	at org.openhab.core.service.AbstractActiveService$RefreshThread.run(AbstractActiveService.java:166) [226:org.openhab.core.compat1x:2.4.0]
2019-01-19 22:10:03.292 [DEBUG] [ab.binding.http.internal.HttpBinding] - item 'RainDataJson' is fetched from cache
2019-01-19 22:10:03.294 [DEBUG] [ab.binding.http.internal.HttpBinding] - updating cache for 'buienradar' ('null')
2019-01-19 22:10:03.298 [ERROR] [b.core.service.AbstractActiveService] - Error while executing background thread HTTP Refresh Service

It seem to get into problems when trying to extractCredentials from the URL.

Guido_van_Haasteren · January 19, 2019, 9:46pm

I have had this error several times too when I did not use any transform. This was solved when I used your regex!
Did you try your example above but with the azure url?

Guido_van_Haasteren · January 19, 2019, 9:50pm

You two are so much better with coding! I’ll better wait and see what comes out!

esteevens · January 19, 2019, 10:17pm

OK. I found my problem. Just a stupid typo in the http.cfg file (I had buienrader.url= instead of buienradar.url=). Now I have tested with the real URL and it turned out I had to do some tweaking for the line endings:

(function(i) {
    var rainLines = i.replace(/\r\n|\r|\n/g, '\n').split('\n');
    var rainintensity = [];
    rainLines.forEach(function(line)
    {
        var array = line.split("|");
        var data = {
            "intensity": Math.pow(10, (parseInt(array[0]) - 109) / 32),
            "time": array[1]
        }
        if (data.intensity) 
            rainintensity.push(data);
    });
    
    var output = {};
    output.rainintensity = rainintensity;
    return JSON.stringify(output);
})(input)

2019-01-19 23:19:46.431 [DEBUG] [ab.binding.http.internal.HttpBinding] - item 'RainDataJson' is fetched from cache
2019-01-19 23:19:46.705 [DEBUG] [ab.binding.http.internal.HttpBinding] - transformed response is '{"rainintensity":[{"intensity":0.0003924189758484536,"time":"23:20"},{"intensity":0.0003924189758484536,"time":"23:25"},{"intensity":0.0003924189758484536,"time":"23:30"},{"intensity":0.0003924189758484536,"time":"23:35"},{"intensity":0.0003924189758484536,"time":"23:40"},{"intensity":0.0003924189758484536,"time":"23:45"},{"intensity":0.0003924189758484536,"time":"23:50"},{"intensity":0.0003924189758484536,"time":"23:55"},{"intensity":0.0003924189758484536,"time":"00:00"},{"intensity":0.0003924189758484536,"time":"00:05"},{"intensity":0.0003924189758484536,"time":"00:10"},{"intensity":0.0003924189758484536,"time":"00:15"},{"intensity":0.0003924189758484536,"time":"00:20"},{"intensity":0.0003924189758484536,"time":"00:25"},{"intensity":0.0003924189758484536,"time":"00:30"},{"intensity":0.0003924189758484536,"time":"00:35"},{"intensity":0.0003924189758484536,"time":"00:40"},{"intensity":0.0003924189758484536,"time":"00:45"},{"intensity":0.0003924189758484536,"time":"00:50"},{"intensity":0.0003924189758484536,"time":"00:55"},{"intensity":0.0003924189758484536,"time":"01:00"},{"intensity":0.0003924189758484536,"time":"01:05"},{"intensity":0.0003924189758484536,"time":"01:10"},{"intensity":0.0003924189758484536,"time":"01:15"}]}'

esteevens · January 20, 2019, 9:49pm

@Guido_van_Haasteren

services/http.cfg:

buienradar.url=https://br-gpsgadget.azurewebsites.net/data/raintext?lat=51.00&lon=3.00
buienradar.updateInterval=300000

items/rain.items:

String RainDataJson { http="<[buienradar:300000:JS(convertBuienradarRaindata2Json.js)]" }
Number Rain1
Number Rain2
Number Rain3

transform/convertBuienradarRaindata2Json.js

(function(i) {
    var rainLines = i.replace(/\r\n|\r|\n/g, '\n').split('\n');
    var rainintensity = [];
    rainLines.forEach(function(line)
    {
        var array = line.split("|");
        var data = {
            "intensity": Math.round(Math.pow(10, (parseInt(array[0]) - 109) / 32) * 100) / 100,
            "time": array[1]
        }
        if (!isNaN(data.intensity)) 
            rainintensity.push(data);
    });
    
    var output = {};
    output.rainintensity = rainintensity;
    return JSON.stringify(output);
})(input)

rules/rain.rules:

rule "Rain"
when
	Item RainDataJson changed
then
	var String json = (RainDataJson.state as StringType).toString

	var String regen1 = transform("JSONPATH", "$.rainintensity[0].intensity", json);
	var String regen2 = transform("JSONPATH", "$.rainintensity[1].intensity", json);
	var String regen3 = transform("JSONPATH", "$.rainintensity[2].intensity", json);

	logInfo(RFN, "Rain1 = {}", regen1)
	logInfo(RFN, "Rain2 = {}", regen2)
	logInfo(RFN, "Rain3 = {}", regen3)

	Rain1.postUpdate(regen1)
	Rain2.postUpdate(regen2)
	Rain3.postUpdate(regen3)
end

Now that it works, I’m really curious to know what you plan on doing with it

Guido_van_Haasteren · January 21, 2019, 7:13pm

Sweet! Thank you very much!

Haha, I definitely owe you that one! I am going to show them all three on my sitemap for reference, trigger a push message through pushbullet when Rain3 >0 mm and I am going to coil up my sun shades on my veranda to prevent it from getting wet.
Actually for my veranda I only need Rain1, but I like to check up front to see how accurate it is going to be, hence the warning 15 minutes before.