Web Scraping behind login page

I am trying to get info from the web page of my energy company. To access the page with the info, I have to login first.
I tried accessing it with a direct POST on the target page, with the id/passwords fields from the login form, but it didn’t work.
How do you usually get info behind a login page?

There are different approaches to web logins, and your energy company is likely using cookies and sessions. You would have to research how your particular energy company is accepting the login and then maintaining the login session, and achieve the equivalent with code on the openHAB side. You could use your browser’s development mode to examine HTTP responses to see what it’s doing, and then trying to achieve something similar with the HTTP binding, action calls, and/or curl calls.

Thanks for your help. My first approach was to code something, which would then call openHAB’s REST API, but it didn’t seem clean. I’ll look into the HTTP Binding.

Don’t abandon your code yet. I’ve found a lot of cases where the default implementation of HTTP client libraries do not handle more than basic logins (e.g. the logins where a dialog pops up asking for username and password and where you can put the username and password on the URL) and you may have to build something yourself. Before I started working with openHAB I built some scripts on Google Drive to do a first pass analysis of my finances (current and projected account balances, upcoming bills and scheduled transactions, etc) and send me an email summary every night. But to pull down my information from Mint I ended up needing to write a Python script to log in and get the data from Mint and then use the Google Drive API to push it up the csv files to Drive. From there the spreadsheet Javascript imports the files, populates the spreadsheet and generates and emails the report.

I was lucky enough to find a library to figure out the Mint login stuff but lacking that you may need to hook up a local web proxy that you can use to examine the traffic as it goes by. Doing this I learned that Mint not only uses cookies but it generates a token that you need to provide in your subsequent requests.