Retrieving data from NWS URL

Tags: #<Tag:0x00007f6173354ce8>

OpenHAB: 2.5
Raspbian: 9

Does anyone have any ideas as to how I could parse the XML from this web page? It seems that it only displays the XML after being viewed by a browser. If I use a sendHttpGetRequest(URL) in a rule, it does not get the page properly. If I could just get the XML data from the page just as displayed in Firefox or Chrome, I could do the rest. I want to use the data to automatically update weather-related items in OpenHAB for various purposes.

I have tried ParseHub, but it seems to choke on it. I would need a way to pull the data automatically on a schedule.

Here is the link: https://www.wrh.noaa.gov/mesowest/getobextXml.php?sid=KSLC&num=2

I searched the forum and found this. Will it work for you?

I had looked at the alerts before. The problem I am running into with the NWS API which provides JSON-LD and XML formatted weather observations is that they are delayed (significant delays regularly) and often have null values. However, this page (which is adjustable to specific weather stations) gives the observations every 5 minutes. Sometimes, there is a 5-minute interval that is missed, but it is much more reliable. However, I have not yet found a way to pull the XML into a rule so it can be used/parsed. Import.io seems possible, but I have not learned how to make it work yet. ParseHub would be a good option if it would actually work with this URL.

The RSS feeds do not provide this same regular weather observation updates.

Since you mentioned XML, would this work?

Thank you. The problem, however, is not with XPath. I need a way to programmatically gain access to the XML as displayed when the web page is opened in a web browser. That is the core issue. If there is a tool that will allow access to the XML and can also parse the data automatically, that would be fine. I can figure out how to parse the data from the XML if I can just get the XML available to parse in an OpenHAB rule.

So in summary, I want to be able to programmatically read the XML from the URL provided and parse the values, such as temperature, wind speed, wind direction, relative humidity, etc. and update OpenHAB items with these values. Also, I only need to be able to parse the first values given in the URL from the most recent observation node for my current purposes.

Perhaps using Jython which is basically Python 2?

Again, the issue is not really one of parsing the data. The issue is finding some way to programmatically obtain the XML data automatically so that it can be parsed using one of the tools available.

I do not agree with your argument about the API being delayed, but if that is what you want, you can pull the data using executeCommandLine and curl. Just be aware that https://www.wrh.noaa.gov/mesowest/ gives…

2 Likes

Are you using the old API? NOAA has two APIs, a new one and an old one. I did some messing around with NOAA awhile back which might be able to get you started. See

Notice the significantly different URL I’m using there.

I recommend using Scripted Automation as you will have better tools available to you to parse out the complicated parts than XPath provides (unless you are some sort of XSLT guru in which case you can use the XSLT transformation).

1 Like

I got the current NWS API working just fine in an OpenHAB rule that updates several weather items and have been watching it on a regular basis this week. That is how I know there are delays; plus, it regularly provides null values, and this happens from the two main NWS stations at the two airports that are in close proximity. I have looked at every option the NWS provides for free, and in some cases, it is only provided every hour or can be delayed at times for multiple hours. They even acknowledge the potential delays on their own website (I don’t have that reference at the moment). Also, the frequency can depend on which station is reporting. Maybe the NWS API in other areas provides more reliable data, but that doesn’t seem to be the case here. The SLC International Airport is one of the stations in which I was most interested, but I want the data to be closer to real-time. I have gotten poor results from OpenWeatherMap and Meteoblue does not provide current weather observations for free. I have obtained a free API key for several weather services so that I could try them. AccuWeather is still one that I have to spend more time with, but their free tier only allows 50 calls per day. I may combine them with other sources once I figure out how to query their API to find the specific location(s) I would want to query. I don’t know if it will be possible with AccuWeather to select a specific monitoring station, if desired.

There is a reason why I am focused on the NWS URL I have posted here, and that is because it does not ever seem to have null values and is the only place I have found where the NWS seems to provide near real-time current weather observations. I wonder if the other commercial services are getting a different feed from the NWS or if they are just combining the NWS API data with many other data points for their databases. I wish the NWS API were more reliable. Then, I would just use it, but it is not good for my use.

I would just like to know if there is a way that someone could tell me how to regularly pull the XML from the URL I provided in the first post.

Perhaps set up a bounty on bounty-source to attract help?

I believe Scott did.

Having gone down this path myself, my only advice is if you want accurate, local, and timely observations you have to build your own weather station.

2 Likes

Is that what you did? We have a purchased one that is only standalone.

Not yet. For now I’m living with inaccurate but timely (i.e. OpenWeatherMap) which as long as it’s within about 5 degrees F of actual is accurate enough for my home automation purposes. The current outside temperature is used as part of the calculation on whether to turn on the house fan or not. Eventually I’ll probably build one but there are many higher priority projects for me right now.

1 Like

Thanks for the responses. I was able to find a way to use PowerShell to grab the XML data. Now, I have many options, and I may just write a PowerShell script to take the data and then use the OpenHAB REST API to update my OpenHAB weather items or I will just save the output to an XML file on the OpenHAB server and then use a rule to parse the data and update the OpenHAB items. It was surprisingly easy to do.

The basic two lines that I used in PowerShell ISE for testing are these:

$webContent = Invoke-WebRequest -Uri 'https://www.wrh.noaa.gov/mesowest/getobextXml.php?sid=KSLC&num=2' -Headers @{'Accept' = 'application/xml'; 'X-My-Header' = 'Hello World'}
$webContent.Content

This is the output I received from the above commands:

<?xml version="1.0" encoding="ISO-8859-1"?>
<station id="KSLC" name="Salt Lake City, Salt Lake City International Airport" elev="4226" lat="40.77069" lon="-111.96503" provider="NWS/FAA">
<ob time="10 Apr 5:15 pm" utime="1586560500">
<variable var='T' description='Temp' unit='F' value='64'/>
<variable var='TD' description='Dewp' unit='F' value='25'/>
<variable var='RH' description='Relh' unit='%' value='22'/>
<variable var='DD' description='Direction' unit='deg' value='340'/>
<variable var='DDCARD' description='Wind Card' unit='direction' value='NNW'/>
<variable var='FF' description='Wind' unit='mph' value='12'/>
<variable var='VV' description='Visibility' unit='miles' value='10'/>
<variable var='SKY' description='Clouds' unit='' value='CLR'/>
<variable var='SLP' description='SL Pres' unit='mb' value='1007.5'/>
<variable var='ALTSE' description='Altimeter' unit='inches' value='29.93'/>
<variable var='P' description='Station Pressure' unit='inches' value='25.63'/>
<variable var='STAQUAL' description='Station Quality' unit='' value='OK'/>
</ob>
<ob time="10 Apr 5:10 pm" utime="1586560200">
<variable var='T' description='Temp' unit='F' value='64'/>
<variable var='TD' description='Dewp' unit='F' value='25'/>
<variable var='RH' description='Relh' unit='%' value='22'/>
<variable var='DD' description='Direction' unit='deg' value='320'/>
<variable var='DDCARD' description='Wind Card' unit='direction' value='NW'/>
<variable var='FF' description='Wind' unit='mph' value='10'/>
<variable var='VV' description='Visibility' unit='miles' value='10'/>
<variable var='SKY' description='Clouds' unit='' value='CLR'/>
<variable var='SLP' description='SL Pres' unit='mb' value='1007.5'/>
<variable var='ALTSE' description='Altimeter' unit='inches' value='29.93'/>
<variable var='P' description='Station Pressure' unit='inches' value='25.63'/>
<variable var='STAQUAL' description='Station Quality' unit='' value='OK'/>
</ob>
<ob time="10 Apr 5:05 pm" utime="1586559900">
<variable var='T' description='Temp' unit='F' value='63'/>
<variable var='TD' description='Dewp' unit='F' value='25'/>
<variable var='RH' description='Relh' unit='%' value='24'/>
<variable var='DD' description='Direction' unit='deg' value='320'/>
<variable var='DDCARD' description='Wind Card' unit='direction' value='NW'/>
<variable var='FF' description='Wind' unit='mph' value='9'/>
<variable var='VV' description='Visibility' unit='miles' value='10'/>
<variable var='SKY' description='Clouds' unit='' value='CLR'/>
<variable var='SLP' description='SL Pres' unit='mb' value='1007.8'/>
<variable var='ALTSE' description='Altimeter' unit='inches' value='29.93'/>
<variable var='P' description='Station Pressure' unit='inches' value='25.63'/>
<variable var='STAQUAL' description='Station Quality' unit='' value='OK'/>
</ob>
<ob time="10 Apr 5:00 pm" utime="1586559600">
<variable var='T' description='Temp' unit='F' value='63'/>
<variable var='TD' description='Dewp' unit='F' value='25'/>
<variable var='RH' description='Relh' unit='%' value='24'/>
<variable var='DD' description='Direction' unit='deg' value='320'/>
<variable var='DDCARD' description='Wind Card' unit='direction' value='NW'/>
<variable var='FF' description='Wind' unit='mph' value='8'/>
<variable var='VV' description='Visibility' unit='miles' value='10'/>
<variable var='SKY' description='Clouds' unit='' value='CLR'/>
<variable var='SLP' description='SL Pres' unit='mb' value='1007.8'/>
<variable var='ALTSE' description='Altimeter' unit='inches' value='29.93'/>
<variable var='P' description='Station Pressure' unit='inches' value='25.63'/>
<variable var='STAQUAL' description='Station Quality' unit='' value='OK'/>
</ob>
</station>

I removed many of the XML nodes to save space here.

After some further work on this issue since last week, I have found a way to get the NWS XML data using an OpenHAB rule without having to bring it in from an external Powershell script (although I have that working, as well). Here are the basic commands that allow me to capture, parse the XML data with XPath expressions, and update the OpenHAB items.

	var String xmlkslc = executeCommandLine("curl -s -H accept:application/json -X GET 'https://www.wrh.noaa.gov/mesowest/getobextXml.php?sid=KSLC&num=2'", 3000) //Executes the curl string
//	logInfo("ResponseNWSDataXML", xmlkslc) // Displays full output of URL
	logInfo("ResponseNWSDataXML", "NWS Data Received")
	var String testStationID = transform("XPATH", "string(//station/@id)", xmlkslc)
	var String testTime = transform("XPATH", "string(//ob[1]/@time)", xmlkslc)
	var String testUTime = transform("XPATH", "string(//ob[1]/@utime)", xmlkslc)
	var DateTimeType testCurTime = new DateTimeType(testUTime)
	var Number testTempF = Float::parseFloat(transform("XPATH", "number(//ob[1]/variable[@var='T']/@value)", xmlkslc))
	var Number testRH = Float::parseFloat(transform("XPATH", "number(//ob[1]/variable[@var='RH']/@value)", xmlkslc))
	var String testWindDir = transform("XPATH", "string(//ob[1]/variable[@var='DDCARD']/@value)", xmlkslc)
	var Number testWindSpeedMPH = Float::parseFloat(transform("XPATH", "number(//ob[1]/variable[@var='FF']/@value)", xmlkslc))
	var Number testWindGustMPH = Float::parseFloat(transform("XPATH", "number(//ob/variable[@var='FFGUST']/@value)", xmlkslc))
	var Number testVisibility = Float::parseFloat(transform("XPATH", "number(//ob[1]/variable[@var='VV']/@value)", xmlkslc))
	var String testWeatherCond = transform("XPATH", "string(//ob/variable[@var='WEA']/@value)", xmlkslc)

	logInfo("testStationID", testStationID)
	logInfo("testTime", testTime)
	logInfo("testUTime", testUTime)
	logInfo("testCurTime", testCurTime.toString)
	logInfo("testTempF", testTempF.toString)
	logInfo("testRH", testRH.toString)
	logInfo("testWindDir", testWindDir)
	logInfo("testWindSpeedMPH", testWindSpeedMPH.toString)
	logInfo("testWindGustMPH", testWindGustMPH.toString)
	logInfo("testVisibility", testVisibility.toString)
	logInfo("testWeatherCond", testWeatherCond)

	NWSCurrentTemperature.postUpdate(testTempF)
	NWSCurrentWindDirection.postUpdate(testWindDir)
	NWSCurrentWindSpeed.postUpdate(testWindSpeedMPH)
	NWSCurrentRelativeHumidity.postUpdate(testRH)
	NWSCurrentWeatherCondition.postUpdate(testWeatherCond)
	NWSTimestamp.postUpdate(testCurTime.toString)

The log file output looks like this:

==> /var/log/openhab2/openhab.log <==
2020-04-13 19:16:01.185 [INFO ] [home.model.script.ResponseNWSDataXML] - NWS Data Received
2020-04-13 19:16:01.499 [INFO ] [smarthome.model.script.testStationID] - KSLC
2020-04-13 19:16:01.503 [INFO ] [ipse.smarthome.model.script.testTime] - 13 Apr 6:55 pm
2020-04-13 19:16:01.506 [INFO ] [pse.smarthome.model.script.testUTime] - 1586825700
2020-04-13 19:16:01.510 [INFO ] [e.smarthome.model.script.testCurTime] - 2020-04-14T00:55:00.000+0000
2020-04-13 19:16:01.513 [INFO ] [pse.smarthome.model.script.testTempF] - 45.0
2020-04-13 19:16:01.516 [INFO ] [clipse.smarthome.model.script.testRH] - 14.0
2020-04-13 19:16:01.519 [INFO ] [e.smarthome.model.script.testWindDir] - NNW
2020-04-13 19:16:01.523 [INFO ] [rthome.model.script.testWindSpeedMPH] - 16.0
2020-04-13 19:16:01.526 [INFO ] [arthome.model.script.testWindGustMPH] - 26.0
2020-04-13 19:16:01.530 [INFO ] [marthome.model.script.testVisibility] - 10.0
2020-04-13 19:16:01.533 [INFO ] [arthome.model.script.testWeatherCond] - Clear