JSON to pull Weather data from metoffice.gov.uk

1

Hi,

Trying to pull data from the metoffice.gov website and started out by using this as reference

https://the.cyclingengineer.co.uk/2014/10/04/metoffice-observations-in-openhab/

However, I believe it’s missing a rule to actually go and get the data and ‘transform’ it out? I’ve created all of the config files as stated - and understand how to write the rules to control my things once I have each of the parameters stored as items.

Can anyone nudge me in the correct direction for getting the data and updating the items please?

2

If you are following the procedure mentioned on some other site, did you ask for assistance there? You may get better feedback. I know openHAB has changed a lot just in the past year.

3

Good point! I’ve just sent the chap a comment - his post is from 2014 though ;0

1 Like
4

That tutorial does not use Rules. It uses one Item with the HTTP binding and a JS transformation to pull the data. Neither the HTTP binding nor the JS transformation have changed much if at all in all this time so it should work as written.

5

Thanks for giving me confident Rikoshak. I re-built all of the files, ensuring I have a seperate .js transform for all entries and voila! it worked!.
I think the difference was not having the HTTP binding installed last time -

I do have two questions which I’ll try and find out:

  • Does openhab only request the data once per hour - or does it request the data multiple time (for each item)

  • the metoffice give the Weather Type an int (I’ve put the item as Number) and that translates out to a String of human type: I’m assuming I now need a MAP to translate the weather type number they give me to text?

W int Significant weather as a code:
NA Not available
0 Clear night
1 Sunny day
2 Partly cloudy (night)
3 Partly cloudy (day)
4 Not used
5 Mist
6 Fog
7 Cloudy
8 Overcast
9 Light rain shower (

6

Depends on how you have it configured. As that tutorial is written, it only pulls the data down once every 3600000 msec for each Item individually. If you only want to pull the web page once for lots of Items look at the caching configuration for the HTTP binding in the binding’s readme.

Sounds reasonable. Use the Map transform in the Item label.

7

Rik,

Thanks to your help I sorted out the caching and fetch the items from the hourly cache - just as I did I got a load of “URI is too large >8192” log entries - I gathered because of the cahces filling up, me trying to fetch all that data every hour.

Cleared cache, fixed permissions all sorted - All of this was done from your posts, scattered around openHAB forums - thank you.

MAP went swimmingly.

I’m now going to play around with some

https://api.spacexdata.com/v3/launches/latest

SpaceX data, in the same vein as the gentlemen that did the metoffice integration. I’ve got my hourly pull to cache done, I’ve setup my items and pointed them to the transform.js files for each item.

Now I need some guidance on what to read to understand the syntax for the expression to pull data out in his .js - I’m trying to ‘reverse’ engineer’ it - however if I simply knew what I was trying to reverse engineer I could then learn it for real.

Specifically the below:

var metofficeData = JSON.parse(input)

metofficeData.SiteRep.DV.Location.Period[metofficeData.SiteRep.DV.Location.Period.length-1].Rep[metofficeData.SiteRep.DV.Location.Period[metofficeData.SiteRep.DV.Location.Period.length-1].Rep.length - 1].T

8

I haven’t looked very closely at the data, but if it’s JSON formatted data, as it appears, the JSONPATH transformation would probably be a better choice over JavaScript. A JavaScript transform is only really needed if there is something about the JSON that JSONPATH can’t handle. The Metooffice data is an array of entries of arbitrary size so the JavaScript is required because you can’t say (the last element in the array) in JSONPATH.

Takes the JSON formatted text and parses it into a JavaScript Object.

Access one value from the JSON. JSON is hierarchical so it’s grabbing:

SiteRep
  DV
    Location
      Period[n-1] // where n is the number of periods
        Rep[m-1] // where m is the number of Reps
          T

You will have to examine the JSON returned by SpaceX to see if it uses arrays and whether those arrays are of an arbitrary size. If not than just use the JSONPATH transformation.

9

Does the JSONPATH transform route require a rule to run periodically?

The example in the literature doesnt make (much) to me clear.

10

No, it’s a transformation like any other transformation.

String RocketType "Rocket Type [%s]" { http="<[https://api.spacexdata.com/v3/launches/latest:3600000:JSONPATH($.rocket.rocket_type)"}
11

Sorry dont meant to bother you but - where there is a 0 in the path, such as rocket.second_stage.payloads.0.payload_id simply a 0 does not work - Is this what you meant by an array? And I should be using the JS method but mandating the first entry of the array - (And not sure how that is denoted)

As well as providing the answer (which is awefully kind), do you have a resource or a keyword I can search to learn these syntax’s

12

JSON is really simple.

Start a hierarchy with { and close it with }.

Elements have a name and a value.

The name is always in quotes.

The value can be:

  • number
  • String (in quotes)
  • an array of values, denoted by ["value1", "value2"], the first value is at location [0]
  • a sub-section of JSON denoted by { "element1": "foo", "element2": "bar"}

That’s it. That’s all there is to it.

JSONPATH is a way to navigate down the hierarchy of name/value pairs. $ means the root of the JSON documents. Then reference the elements by name. If the value of an element is an array, use [n] to reference the nth element of the array.

Given:

{
  "flight_number": 93,
  "mission_name": "Starlink 6",
  "mission_id": [],
  "launch_year": "2020",
  "launch_date_unix": 1587583800,
  "launch_date_utc": "2020-04-22T19:30:00.000Z",
  "launch_date_local": "2020-04-22T15:30:00-04:00",
  "is_tentative": false,
  "tentative_max_precision": "hour",
  "tbd": false,
  "launch_window": null,
  "rocket": {
    "rocket_id": "falcon9",
    "rocket_name": "Falcon 9",
    "rocket_type": "FT",
    "first_stage": {
      "cores": [
        {
          "core_serial": "B1051",
          "flight": 4,
          "block": 5,
          "gridfins": true,
          "legs": true,
          "reused": true,
          "land_success": true,
          "landing_intent": true,
          "landing_type": "ASDS",
          "landing_vehicle": "OCISLY"
        }
      ]
    },
    "second_stage": {
      "block": 5,
      "payloads": [
        {
          "payload_id": "Starlink 6",
          "norad_id": [
            72000
          ],
          "reused": false,
          "customers": [
            "SpaceX"
          ],
          "nationality": "United States",
          "manufacturer": "SpaceX",
          "payload_type": "Satellite",
          "payload_mass_kg": 15400,
          "payload_mass_lbs": 33951.2,
          "orbit": "VLEO",
          "orbit_params": {
            "reference_system": "geocentric",
            "regime": "very-low-earth",
            "longitude": null,
            "semi_major_axis_km": null,
            "eccentricity": null,
            "periapsis_km": null,
            "apoapsis_km": null,
            "inclination_deg": null,
            "period_min": null,
            "lifespan_years": null,
            "epoch": null,
            "mean_motion": null,
            "raan": null,
            "arg_of_pericenter": null,
            "mean_anomaly": null
          }
        }
      ]
    },
    "fairings": {
      "reused": true,
      "recovery_attempt": false,
      "recovered": null,
      "ship": "GOMSTREE"
    }
  },
  "ships": [
    "OCISLY",
    "GOMSTREE",
    "GOMSCHIEF"
  ],
  "telemetry": {
    "flight_club": null
  },
  "launch_site": {
    "site_id": "ksc_lc_39a",
    "site_name": "KSC LC 39A",
    "site_name_long": "Kennedy Space Center Historic Launch Complex 39A"
  },
  "launch_success": true,
  "links": {
    "mission_patch": "https://images2.imgbox.com/d2/3b/bQaWiil0_o.png",
    "mission_patch_small": "https://images2.imgbox.com/9a/96/nLppz9HW_o.png",
    "reddit_campaign": "https://www.reddit.com/r/spacex/comments/fxkc7k/starlink6_launch_campaign_thread/",
    "reddit_launch": "https://www.reddit.com/r/spacex/comments/g5jmx0/rspacex_starlink_6_official_launch_discussion/",
    "reddit_recovery": "https://www.reddit.com/r/spacex/comments/g6kztd/rspacex_starlink_v1_l6_recovery_discussion/",
    "reddit_media": "https://www.reddit.com/r/spacex/comments/g5fqka/rspacex_starlink6_media_thread_photographer/",
    "presskit": "https://www.spacex.com/sites/spacex/files/seventh_starlink_mission_overview.pdf",
    "article_link": "https://spaceflightnow.com/2020/04/22/spacexs-starlink-network-surpasses-400-satellite-mark-after-successful-launch/",
    "wikipedia": "https://en.wikipedia.org/wiki/Starlink",
    "video_link": "https://youtu.be/wSge0I7pwFI",
    "youtube_id": "wSge0I7pwFI",
    "flickr_images": [
      "https://live.staticflickr.com/65535/49673373182_93a517e140_o.jpg",
      "https://live.staticflickr.com/65535/49672551378_fabc17ef6f_o.jpg",
      "https://live.staticflickr.com/65535/49672551303_564ce21658_o.jpg",
      "https://live.staticflickr.com/65535/49806771628_fef13c852d_o.jpg",
      "https://live.staticflickr.com/65535/49807633862_e5abcb41a6_o.jpg"
    ]
  },
  "details": "This mission will launch the sixth batch of operational Starlink satellites, which are expected to be version 1.0, from SLC-40, Cape Canaveral AFS. It is the seventh Starlink launch overall. The satellites will be delivered to low Earth orbit and will spend a few weeks maneuvering to their operational altitude of 550 km. The booster for this mission is expected to land on OCISLY.",
  "upcoming": false,
  "static_fire_date_utc": "2020-04-17T11:48:00.000Z",
  "static_fire_date_unix": 1587687810,
  "timeline": null,
  "crew": null,
  "last_date_update": "2020-04-22T17:41:33.000Z",
  "last_ll_launch_date": "2020-04-22T19:30:00.000Z",
  "last_ll_update": "2020-04-22T17:41:33.000Z",
  "last_wiki_launch_date": "2020-04-22T19:30:00.000Z",
  "last_wiki_revision": "7eb594bf-84a3-11ea-b698-0e120e32abbf",
  "last_wiki_update": "2020-04-22T14:13:53.000Z",
  "launch_date_source": "launch_library"
}
JSONPATH Expression value notes
$.flight_number 93
$.rocket.rocket_id “falcon9”
$.rocket.first_stage.cores[0].flight 4 Only one value in the array value for cores
$.ships[0] “OCISLY” three values in the array, get the first one
$.ships[2] “GOMSCHIEF”. three values in the array, get the third one
$.telemetry “flight_club”: null notice this is itself a piece of JSON, not just a single string or number

It’s only if there is an array value you want to select from and you need to do some math to determine which element you want to select that you need to use JavaScript to extract the values.

Search for “JSON syntax” and “JSONPATH tutorial”. Both are standards and neither were invented by openHAB.

13

again Rik,

A massive help. I’ve now ditched the previous .js files and JS approach - have everything being transformed within the items file on a JSON basis - most values are coming through perfect - except for anything under an array- even at position 0 for instance:

String payload_id "Payload ID : [%s]" { http="<[spaceXLastCache:3600000:JSONPATH($.rocket.second_stage.payloads[0].payload_id)]"}

says in the logs that it ‘threw an exception’ and the response = the entire file.

org.openhab.core.transform.TransformationException: An error occurred while transforming JSON expression.

at org.openhab.core.transform.TransformationHelper$TransformationServiceDelegate.transform(TransformationHelper.java:71) ~[bundleFile:?]

at org.openhab.binding.http.internal.HttpBinding.execute(HttpBinding.java:218) [bundleFile:?]

at org.openhab.core.binding.AbstractActiveBinding$BindingActiveService.execute(AbstractActiveBinding.java:146) [bundleFile:?]

at org.openhab.core.service.AbstractActiveService$RefreshThread.run(AbstractActiveService.java:169) [bundleFile:?]

I’ve been racking the internet for an answer - however, everywhere suggests the same format as you

https://jsonpathfinder.com/

for instance gives the same

x.rocket.second_stage.payloads[0].payload_id

14

Worked it out - needed to wait for an hour to refresh - and some of the items I was trying to pull through I appended payload_ by accident.

Again, many thanks for your help - You are making my venture in to openhab an exciting one and without your help I’d be left stuck in the water

15

This topic was automatically closed 41 days after the last reply. New replies are no longer allowed.