Updating items using XML data encoded in ISO-8859-1

Tags: #<Tag:0x00007f61799bb8b0> #<Tag:0x00007f61799bb7e8> #<Tag:0x00007f61799bb630>

Here in Canada, our government weather service helpfully publishes their data in an XML format. Unfortunately, it’s encoded in ISO-8859-1, and the special characters seem to cause some issues with the XPATH transformation, apparently because Java String objects are encoded in UTF-16.

What I would like to do is fetch the relevant XML file and update the state of items with the relevant values, e.g. temperature. However, I haven’t yet hit on a working combination of commands to fetch the XML, change the character encoding, and parse the data out.

I did manage to get it working in bash:

curl -s http://dd.weatheroffice.ec.gc.ca/citypage_weather/xml/ON/s0000430_e.xml | 
iconv -f ISO-8859-1 -t UTF-8 | 
xmlstarlet sel -t -v /siteData/currentConditions/temperature

As well as in Python:

import requests
from xml.etree import ElementTree
response = requests.get('http://dd.weatheroffice.ec.gc.ca/citypage_weather/xml/ON/s0000430_e.xml')
xml = ElementTree.fromstring(response.content)
print xml.find('./currentConditions/temperature').text

Thanks in advance for your help. An excerpt of the XML data is shown below, and the full file is here:

<?xml version='1.0' encoding='ISO-8859-1'?>
<siteData xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://dd.weather.gc.ca/citypage_weather/schema/site.xsd">
<license>http://dd.weather.gc.ca/doc/LICENCE_GENERAL.txt</license>
<dateTime name="xmlCreation" zone="UTC" UTCOffset="0">...</dateTime>
<dateTime name="xmlCreation" zone="EST" UTCOffset="-5">...</dateTime>
<location>...</location>
<warnings/>
<currentConditions>
<station code="yow" lat="45.32N" lon="75.67W">Ottawa Macdonald-Cartier Int'l Airport</station>
<dateTime name="observation" zone="UTC" UTCOffset="0">...</dateTime>
<dateTime name="observation" zone="EST" UTCOffset="-5">...</dateTime>
<condition>Mainly Clear</condition>
<iconCode format="gif">31</iconCode>
<temperature unitType="metric" units="C">0.7</temperature>
<dewpoint unitType="metric" units="C">-2.0</dewpoint>
<pressure unitType="metric" units="kPa" change="0.21" tendency="rising">100.2</pressure>
<visibility unitType="metric" units="km">24.1</visibility>
<relativeHumidity units="%">82</relativeHumidity>
<wind>...</wind>
</currentConditions>
...

Looks like your really close. Could you just call the bash script from a time triggered cron rule?

Dear heavens, why?

You should at least tell us what errors you’re seeing…

Assuming your reply was to my post, and your reaction was in regards to updating the items through the API, my thinking was that it would be easier to use API calls in the shell script to update multiple items (the XML has more than just temperature) then to pull the values back into the rule and have to iterate through them to set the state of the items. Although, multiple scripts could be used. Anyhow, I edited my previous post to keep it simple.

Sure. I’ve been through a number of iterations, and here are some of the errors thrown:

[WARN ] [.core.transform.TransformationHelper] - Cannot get service reference for transformation service of type <?xml version='1.0' encoding='ISO-8859-1'?>

org.osgi.framework.InvalidSyntaxException: Invalid value at "(Kanata - Orl�ans)</name>[...the rest of the XML file])": (smarthome.transform=<?xml version='1.0' encoding='ISO-8859-1'?>

I tried calling a bash pipeline, but it seemed like it was returning only the value of the curl command, rather than the whole pipeline. I haven’t tried using a bash script on disk.

I’d like to use the rules engine itself, but I’m often frustrated trying to figure out how to get things done using Xtend and the Rules DSL.

Fiddling with executeCommandLine is tricky, but much easier when just calling a shell script that can be tested on the command line. Something like this:

rule "Test"
when
    //Item Virtual_Switch_1 changed to ON
    //or
    Time cron "0 0/5 * * * ?"
then
    val String currentTemp = executeCommandLine("/opt/openhab2/conf/scripts/temperature.sh",30000)
    logDebug("Rules", "test: currentTemp=[{}]",currentTemp)
    Virtual_Number_1.postUpdate(currentTemp)
end

temperature.sh

#!/bin/sh
response=$(curl -s http://dd.weatheroffice.ec.gc.ca/citypage_weather/xml/ON/s0000430_e.xml | iconv -f ISO-8859-1 -t UTF-8 | xmlstarlet sel -t -v /siteData/currentConditions/temperature)
echo "$response"
1 Like

I don’t have much to offer beyond what has already been said except to say that XSLT might work better than XPATH.

And what rule or script are you using to generate this?

Thanks, I ended up doing something similar but using Python and the REST API instead. It seems to be working, and with no drama about the encoding.

It might be heretical, but I can see a lot of advantages to this approach. I’ve spent a lot of time trying to decipher the nuances of the not-quite-Java, not-quite-Xtend DSL used for rules, and it’s definitely a lot easier to find answers to the same questions for Python.

Rule:

/*** Update weather from Environment Canada ***/

rule "Update EC weather"
  when
    Time cron "0 0/5 * * * ?"
  then

  executeCommandLine('python /etc/openhab2/rules/ec-weather.py', 10000)
end

Python:

import requests
from xml.etree import ElementTree

### Fetch data and extract values ###

response = requests.get('http://dd.weatheroffice.ec.gc.ca/citypage_weather/xml/ON/s0000430_e.xml')
xml = ElementTree.fromstring(response.content)

temp = xml.find('./currentConditions/temperature').text
humidity = xml.find('./currentConditions/relativeHumidity').text
windChill = xml.find('./currentConditions/windChill').text
forecastLow = xml.find('.//forecast/temperatures/temperature[@class="low"]').text

### Update OpenHAB items via REST API ###

s = requests.Session()
s.headers = {"Content-Type": "text/plain", "Accept": "application/json"}

s.put(url="http://192.168.0.121:8080/rest/items/Temperature/state", data = str(temp))
s.put(url="http://192.168.0.121:8080/rest/items/Humidity/state", data = str(humidity))
s.put(url="http://192.168.0.121:8080/rest/items/WindChill/state", data = str(windChill))
s.put(url="http://192.168.0.121:8080/rest/items/ForecastLow/state", data = str(forecastLow))
1 Like

One thing to be aware of is that the JSR223 add-on will let you write all your Rules in Jython, JavaScript, or Groovy if you are more comfortable in those languages.

But coding something up in Python and calling it from the Rules DSL is far from heretical.