OH3: HTTP-Binding / XPATH - no values

boogie · June 28, 2021, 7:55am

Hello community,
i have a problem with XPath on OH3. I would like to read a value from a website and have created the following thing/channel for it:

UID: http:url:xpath_zamg
label: xpath_zamg
thingTypeUID: http:url
configuration:
  authMode: BASIC
  ignoreSSLErrors: false
  baseURL: https://www.zamg.ac.at/cms/de/wetter/wetterwerte-analysen/tirol
  refresh: 30
  commandMethod: GET
  timeout: 3000
  bufferSize: 2048
location: system
channels:
  - id: location
    channelTypeUID: http:string
    label: Standort
    description: ""
    configuration: {}

According to the log file, i only get the following message:

[INFO ] [inding.http.internal.HttpThingHandler] - Using the secure client for thing 'http:url:xpath_zamg'
[INFO ] [openhab.event.ItemStateChangedEvent  ] - Item 'xpath_zamg_location' changed from NULL to

I can successfully determine the value via http://xpather.com/

My Link looks like this:

However, no value is displayed:

The XPath Transformation Service is already installed. Does anyone have any idea why this could be?

Many thanks,
greetings Christoph

ubeaut · June 28, 2021, 8:35am

The documentation only references xml and that is html so maybe the transform doesn’t do html?

I haven’t tried html but I did get xml to work.

rlkoshak · June 28, 2021, 5:05pm

It should work on HTML but only if it’s xhtml, meaning that it conforms to the XML standard. That means every open tag must have a closing tag and a few other similar requirements. Browsers are not as strict and will be happy to render a page that is missing such things.

But the header shows that the page is claiming to be XHTML so XPath shouldn’t have a problem with it.

Given the data is pulled by the HTTP binding, why not apply the transform there? If there is an error it will be more apparent there as I think the Profile fails silently.

boogie · June 28, 2021, 7:10pm

Hi @rlkoshak , thank you for your feedback.
I’ve also read that XHTML should work with XPATH. You mean the XPATH command can be executed directly via the binding? I’ve tried this now and i’m one step further. But now the whole XHTML code is downloaded and the desired value is not selected.

UID: http:url:xpath_zamg
label: xpath_zamg
thingTypeUID: http:url
configuration:
  authMode: BASIC
  ignoreSSLErrors: false
  baseURL: https://www.zamg.ac.at/cms/de/wetter/wetterwerte-analysen/tirol
  delay: 0
  stateMethod: GET
  refresh: 60
  commandMethod: GET
  stateTransformation: XPath://html/body/div/div[2]/div[2]/div[1]/div[1]/div/div[2]/div/div/div[3]/table/tr[3]/td[1]
  timeout: 3000
  bufferSize: 2048
location: system
channels:
  - id: xpath_zamg_location
    channelTypeUID: http:string
    label: Standort
    description: ""
    configuration: {}

From the HTTP-binding i get this log-output with the code from the whole website:

19:16:45.652 [INFO ] [inding.http.internal.HttpThingHandler] - Using the secure client for thing 'http:url:xpath_zamg'.

I’ve already tried the ending /text() /@Value /@* …, unfortunately without success.

rlkoshak · June 28, 2021, 7:15pm

That indicates that the actual xpath is not being found. I can’t explain why the online tester works and this does not. Often different programming languages will have slightly different behaviors. Perhaps Java works a little differently.

boogie · June 29, 2021, 3:41pm

Hi @rlkoshak, thanks, then i will try another way…
I have now extracted the data via Python-script and BeautifulSoup and saved it in a file. Now i would like to import this output (file) using HTTP binding and output it using an item. In your opinion, is it better to save the data in a csv-file or in a txt-file? (only the values are listed one below the other) Unfortunately, i haven’t found much for OH3 - can i configure everything via the GUI or does a script have to be created for this?

rlkoshak · June 29, 2021, 3:53pm

The HTTP binding doesn’t load files. What exactly are you trying to do with the binding here?

The best depends on the nature of the data and your ability to code what needs to load and process that data in a rule.

That’s not an either or. You can create Script Actions in UI rules which is where you write code for a rule. But to load and parse a file you will have to write some code.

boogie · June 29, 2021, 4:32pm

I thought, i might be able to load the file via the binding and process the content in OH.

But what i actually want to do, i fetch the content of the website once an hour using a Python-script and save the content in a file. I store this file in a defined location, from which OH reads the content of the file and displays it in/with an item. For example, the content of the file looks like this:

Kitzbühel
767m
28.5°
29 %
Südost, 13 km/h
29 km/h
0.0 mm
81 %
1002.3 hPa

Unfortunately, i don’t really know how to approach this matter. I can certainly look up the individual steps via the forum, but i don’t know what options there are in general and which of them are also useful for my project.
I am currently in the process of familiarizing myself with HABApp, so I would like to implement it in Python if possible.

rlkoshak · June 29, 2021, 4:37pm

I can’t help with HABApp. That’s a completely separate rule engine. You definitely cannot use the OH UI to create or manage HABApp rules.

The HTTP binding can only interact with HTTP servers. It can’t load some random file from your file system.

If you are planning on using HABApp you’ll need to learn how to load a file using Python 3 and parse its contents. Then use HABApp features to issue the updates and commands to the relevant Items.

boogie · June 29, 2021, 4:57pm

Okay i understand, thanks for the explanation. And if i wanted to do everything with OH, how would that go about here? The rule engine HABApp is not so important to me now, if i could solve this via the internal rule engine, i would honestly prefer that. The only thing that matters to me is that i can do it with Python. The OH Python-binding has already been installed and also works.

rlkoshak · June 29, 2021, 5:05pm

Pretty much the same thing. You’ll need to figure out how to load and parse a file in Python and issue updates and commands to the relevant Items.

You’ll most likely want to use GitHub - CrazyIvan359/openhab-helper-libraries: JSR223-Jython scripts and modules for use with openHAB.

You can potentially write a Script Action using Python through MainUI but there are a few limitations and you’ll want the Helper Libraries anyway.

boogie · June 29, 2021, 5:12pm

Okay thank you very much! Then i’ll give it a try and if i manage to do it, i’ll share the procedure here.

rossko57 · June 29, 2021, 6:47pm

I’m not sure why you’re messing with files.

You’ve done the hard part, getting the raw data into openHAB.
Manipulating it to extract a value can be done directly in rules, or via a transformation. If you can’t make XPATH work with it, the whole of javascript is available too.

EDIT - the XPATH trouble is about namespace.
The returned webpage includes
<html xmlns="http://www.w3.org/1999/xhtml" lang="de">
so all the enclosed html tags are in that namespace.

I had a play in a rule; XPath with
"/html/head/title/text()"
returns nothing, because it searches for no-namespace tags.
"/*[name()='html']/*[name()='head']/*[name()='title']/text()"
returns Tirol ZAMG from the
<title>Tirol — ZAMG</title>
section, using namespace.

namespace is discussed in the transformation doc

but I find it hard to follow.

boogie · July 4, 2021, 12:23pm

Thank you very much @rossko57 you gave me the right hint, now it works!

For all those who find it a little difficult to create the path, chrome has the option of displaying the XPATH path in the developer tools. You still have to modify the output a bit, but it makes the creation of the path a lot easier.

Dave_Baldwin · April 22, 2022, 7:09pm

@boogie, are you able to able to provide any more details on what your final XPATH expression was here?
In the XHTML I am trying to parse through (from my old-ish Brother printer), the XPATH expression should ideally be something like:

/html/body/div/div/div[2]/div[2]/div[2]/div/div/div[2]/form/div[2]/dl/dd[3]/div/table/tbody/tr[2]/td/img/@height

… the specific XHTML fragment of interest looks like:
(in the interests of brevity I have left out much of the rest of the XHTML)

<dt>Toner&#32;Level</dt>
<dd>
	<div id="ink_level">
		<table id="inkLevelMono"
			   summary="ink level">
			<tr>
				<th/>
			</tr>
			<tr>
				<td>
					<img src="../common/images/black.gif"
						 alt="Black"
						 class="tonerremain"
						 height="50"/>
				</td>
			</tr>
			<tr>
				<th>BK</th>
			</tr>
		</table>
	</div>
</dd>

The value that I want is the ‘50’. It is representative of my printer toner quantity remaining.

I have tried expanding out my XPATH string, something like:
"/*[name()='html']/*[name()='body']/*[name()='div']/*[name()='div']/*[name(2)='div']/*[name(2)='div']/ … etc. but that did not seem to work.

The XPATH documentation does not provide any details on how to delve into duplicate tags on the same level of the tree, or how to retrieve properties of tags rather than the text contained within the tags…

Thanks for any insight you can provide!

digdeep76 · April 22, 2022, 9:42pm

I have got exactly the same problem… maybe @rossko57 can help…

Dave_Baldwin · May 15, 2022, 12:47am

Quick nudge … any feedback here?

Rob-VF · November 14, 2022, 5:03pm

Hello everyone, I’m in trouble trying to retrieve the status of a PIC-WEB relay using XPATH, I would appreciate any help in solving the issue.

Calling for the status this is what i see on web browser:

<rel_status>
<element id="1014" type="3" status="1" currentval="0" title="porch light" label_val="mV" id_group="1012" currenttextval="" des_group="Lights"/>
</rel_status>

if retrieved by curl

<?xml version="1.0" encoding="UTF-8"?>
<rel_status>
<element id="1014" type="3" status="1" currentval="0" title="porch light" label_val="mV" id_group="1012" currenttextval="" des_group="Lights"/>
</rel_status>

I need to get the “status” 1 or 0 (it means ON or OFF).

On OH3 I created a HTTP Binding Thing, I’m able to control the relay using the commands but I’m not able to poll for the status, can anyone try to help me please?

Regards

R

rlkoshak · November 14, 2022, 5:09pm

There are lots of Xpath testers online where you can paste in the XML and experiment with Xpath expressions until you get it right. They are often also associated with a tutorial. https://codebeautify.org/Xpath-Tester is one but there are others.

It’s been too long since I’ve done Xpath to guess at the right syntax.

Rob-VF · November 14, 2022, 5:31pm

Thanks for your prompt reply, I spent some time (a lot, I’m not a programmer) probably on wrong tutorials/testers.

I.e., using the following expression I was able to read the 1/0 status with several online tools/formatter, but not with the one you linked

/rel_status/element/@status/number()

I will try more, tnx for the hint