How to use time-series from rules (DSL or JS)?

ollo · February 25, 2024, 11:56am

Hi,

is there a way to use time-series from rules (DSL or JS)?
I have a few HTTP services which return time series information in JSON format. I’d like to populate that data into a time-series for later use. How do I do that using rules?
For persistence I’m using rrd4j. OH is running v4.1 in Docker container.

Thanks!

JimT · February 25, 2024, 3:19pm

AFAIK, you can call Item.setTimeSeries(timeseriesobject) and as long as you’ve set up an update policy (I think), this will persist the given timeseries data.

JRuby library recently added support for working with timeseries. With jsscripting you can probably work with raw java class.

Example (JRuby):

# create and populate a TimeSeries object
ts_data = TimeSeries.new
                    .add(Time.now, DecimalType.new(1))

# Send it to the item
MyItem.time_series = ts

This PR will add support so you can persist timeseries directly.

ollo · March 17, 2024, 9:29am

Thanks for your input. Meanwhile I came across the InMemory persistence service which should work for what I have in mind. Still, how do I format the incoming JSON time series into an Item that can be persisted as a forecast?

The HTTP binding has a channel which pulls the JSON that looks like this:

{"elements":[{"begin":"2024-03-17T00:00:00+01:00","end":"2024-03-17T01:00:00+01:00","price":6.499},{"begin":"2024-03-17T01:00:00+01:00","end":"2024-03-17T02:00:00+01:00","price":6.049},{"begin":"2024-03-17T02:00:00+01:00","end":"2024-03-17T03:00:00+01:00","price":5.597},{"begin":"2024-03-17T03:00:00+01:00","end":"2024-03-17T04:00:00+01:00","price":5.875},{"begin":"2024-03-17T04:00:00+01:00","end":"2024-03-17T05:00:00+01:00","price":6.045},{"begin":"2024-03-17T05:00:00+01:00","end":"2024-03-17T06:00:00+01:00","price":6.049},{"begin":"2024-03-17T06:00:00+01:00","end":"2024-03-17T07:00:00+01:00","price":6.26},{"begin":"2024-03-17T07:00:00+01:00","end":"2024-03-17T08:00:00+01:00","price":5.66},{"begin":"2024-03-17T08:00:00+01:00","end":"2024-03-17T09:00:00+01:00","price":5.058},{"begin":"2024-03-17T09:00:00+01:00","end":"2024-03-17T10:00:00+01:00","price":4.715},{"begin":"2024-03-17T10:00:00+01:00","end":"2024-03-17T11:00:00+01:00","price":4.54},{"begin":"2024-03-17T11:00:00+01:00","end":"2024-03-17T12:00:00+01:00","price":4.591},{"begin":"2024-03-17T12:00:00+01:00","end":"2024-03-17T13:00:00+01:00","price":4.181},{"begin":"2024-03-17T13:00:00+01:00","end":"2024-03-17T14:00:00+01:00","price":4.015},{"begin":"2024-03-17T14:00:00+01:00","end":"2024-03-17T15:00:00+01:00","price":4.806},{"begin":"2024-03-17T15:00:00+01:00","end":"2024-03-17T16:00:00+01:00","price":5.612},{"begin":"2024-03-17T16:00:00+01:00","end":"2024-03-17T17:00:00+01:00","price":7.176},{"begin":"2024-03-17T17:00:00+01:00","end":"2024-03-17T18:00:00+01:00","price":8.544},{"begin":"2024-03-17T18:00:00+01:00","end":"2024-03-17T19:00:00+01:00","price":9.431},{"begin":"2024-03-17T19:00:00+01:00","end":"2024-03-17T20:00:00+01:00","price":8.829},{"begin":"2024-03-17T20:00:00+01:00","end":"2024-03-17T21:00:00+01:00","price":7.789},{"begin":"2024-03-17T21:00:00+01:00","end":"2024-03-17T22:00:00+01:00","price":7.233},{"begin":"2024-03-17T22:00:00+01:00","end":"2024-03-17T23:00:00+01:00","price":7.21},{"begin":"2024-03-17T23:00:00+01:00","end":"2024-03-18T00:00:00+01:00","price":6.591}],"next_offset":null,"last_modified":"2024-03-16T11:45:02+00:00"}

Is there any transformation to make that a forecast timeseries item that can be persisted with the InMemory service?

Thanks!

JimT · March 17, 2024, 2:26pm

One item linked to that json channel. Don’t apply any transformation / profile to it. Let’s call it JsonData
Another item for storing the time series into your persistence. Let’s call it Whatever

Using jruby

require "json"

rule "Persist Time Series" do
  updated JsonData
  run do |event|
    time_series = TimeSeries.new

    data = JSON.parse(event.state.to_s)
    data["elements"].each do |element|
      timestamp = ZonedDateTime.parse(element["begin"])
      time_series.add(timestamp, element["price"])
    end

    Whatever.time_series = time_series
  end
end

higgers · March 19, 2024, 4:41pm

Do you happen to know if it’s possible to do the same thing in JavaScript scripting? I’ve just written some code to read the electricity unit prices from the JSON at this URL and put the values in a series of items:

rules.JSRule({
    name: "Get agile unit prices",
    description: "Get agile unit prices",
    triggers: [triggers.ItemStateChangeTrigger('OctopusAgilePrices_AgilePricesJSON')],
    // triggers: [triggers.ItemStateChangeTrigger('OctopusAgileRefreshPrices')],
    execute: data => {
        var logger = log('OCTOPUS');
  
        logger.info("Octopus agile unit price forecast refresh rule...");
        var agileprices = String(items.getItem("OctopusAgilePrices_AgilePricesJSON").state);

        var obj = JSON.parse(agileprices);

        for (var i = 0, l = 99; i < l; i++) {
            var agilePriceItemName = String("Agile_Price_").concat(String(i));
            items.getItem(agilePriceItemName).sendCommand(0);
        }

        var today = time.ZonedDateTime.now();
        var now_dayOfMonth = today.dayOfMonth();

        for (var i = 0, l = obj.results.length; i < l; i++) {
            var valid_from = obj.results[i].valid_from;
            var valid_from_zdt = time.toZDT(valid_from);
            var valid_from_ldt = valid_from_zdt.toLocalDateTime();                     //get valid_from as LocalDateTime
            var valid_from_ldt_dayOfMonth = valid_from_ldt.dayOfMonth();
            if (valid_from_ldt_dayOfMonth >= now_dayOfMonth)
            {
                var item_day_index = Number(valid_from_ldt_dayOfMonth - now_dayOfMonth);
                var item_index = (item_day_index * 48) + (Number(valid_from_ldt.hour()) * 2) + (Number(valid_from_ldt.minute()) >= 30 ? 1 : 0);
                logger.info("Item index = ({}).", String(item_index));
                var agilePriceItemName = String("Agile_Price_").concat(String(item_index));
                items.getItem(agilePriceItemName).sendCommand(obj.results[i].value_inc_vat);
                logger.info("Item name = ({}), valid_from = ({}), unit price = ({}).", agilePriceItemName, String(valid_from_ldt), obj.results[i].value_inc_vat);
            }
        }

        logger.info("count = ({}).", String(obj.count));
        logger.info("number of results = ({}).", String(obj.results.length));
    }
  });

It would be nice if I could also get the code to persist these future values so I could display them as a graph. The parts I don’t know how to write in JS are the declaration of the timeseries like

time_series = TimeSeries.new

And how I define the strategy for the Whatever item so I do the JS equivalent of:

Whatever.time_series = time_series

I use influxDB and a strategies file that I’ve been using since forever:

Strategies {
    everyMinute : "0 * * * * ?"
    everyHour   : "0 0 * * * ?"
    everyDay    : "0 0 0 * * ?"
    default = everyChange
}

Items {
    gGraphing*  : strategy = everyUpdate, everyHour
}

and I can’t find where in the UI I would create a strategy for the future timeseries.

I’ve had a google but it seems like it’s such a new feature that there are many posts about it yet.

higgers · March 19, 2024, 8:22pm

I just installed the Ruby Scripting addon and ever so slightly edited your rule above and it worked! I edited the influxdb.persist file as follows:

Strategies {
    everyMinute : "0 * * * * ?"
    everyHour   : "0 0 * * * ?"
    everyDay    : "0 0 0 * * ?"
    default = everyChange
}

Items {
    gGraphing*  : strategy = everyUpdate, everyHour
    gForecast*  : strategy = forecast
}

The only line I added was gForecast* : strategy = forecast

The rule is as follows:

require "json"

rule "Persist Time Series" do
  updated OctopusAgilePrices_AgilePricesJSON
  run do |event|
    time_series = TimeSeries.new

    logger.info "Triggered item: #{event.item.name}"

    data = JSON.parse(event.state.to_s)
    data["results"].each do |element|
      timestamp = ZonedDateTime.parse(element["valid_from"])
      time_series.add(timestamp, element["value_inc_vat"])
    end

    Agile_Price_Forecast.time_series = time_series
  end
end

Many many thanks for posting this rule!

There is one tweak I need to make to it, the times in the JSON feed of the electricity unit prices are in UTC. This works fine for the moment as here in the UK we’re in GMT for the next 12 days after which we’ll be in BST so the unit prices will be shifted by one hour. How would I go about editing the rule to add a conversion of the valid_from element to local time?

(As an aside, this is my first time ever looking in any detail at ruby code and from what I can see it looks very compact, i.e. you don’t seem to need as much code to get things done as the other supported languages.)

JimT · March 19, 2024, 9:46pm

Assuming you’ve set your system time zone to BST:

      timestamp = ZonedDateTime.parse(element["valid_from"])
                               .with_zone_same_instant(ZoneId.system_default)

Otherwise, adjust the ZoneId.xxx accordingly.

See ZonedDateTime (Java SE 17 & JDK 17)

ollo · March 23, 2024, 8:08am

Hi,

I got it working now too - Thanks for your help!

Now that I got the time series, how do I access the n-th future element of it? The Number item I created to store the time series contains the most recent one. Also, how to get the min/max/avg of the time series?

Thanks again!

JimT · March 23, 2024, 12:01pm

There’s a pr waiting to be reviewed

github.com/openhab/openhab-core

Return units in persistence extension commands and support future persisted states

openhab:main ← mherwege:persistence_extensions

opened 02:16PM - 31 Jul 23 UTC

mherwege

+3467 -978

Closes https://github.com/openhab/openhab-core/issues/3732 Closes https://githu…b.com/openhab/openhab-core/issues/3896 This PR implements unit support in all PersistenceExtension commands, returning the full value including unit for Number item types with a dimension. It therefore changes return types from DecimalType to a State type, whereby the state can include the unit. This has been tested to work with rules DSL, and it should be possible to extend other scripting languages to support this, making calculations with historic states of items with dimensions more straightforward. This may require adapting user rules and is therefore a breaking change. Secondly, this PR makes sure persistence extensions work properly with persisted items in the future: - all `Between` commands have been adapted to be able to properly work across the `now` boundary, - a series of `Till` commands have been added, equivalent to the `Since` commands. - `nexUpdate` and `nexState` commands have been added. - the `historicState` command has been deprecated (but still available for backward compatibility) and replaced with `persistedState` also accepting future dates. - the `evolutionRate` command has been renamed to `evolutionRateSince`, `evolutionRateTill` and `evolutionRateBetween`, bringing it in line with the other commands. This change is required to allow one date signatures to differentiate between `Since` and `Till`. `evolutionRate` is deprecated and kept for backward compatibility, and interpreted as `evolutionRateSince`. - added `removeAllStates` `Since`, `Till` and `Between` commands. The code now also has full null annotations and checks. In doing so, changes were required to avoid potential null issues. Many of these null issues where due to null being interpreted as now for end dates before, which is not possible when time can be in the future. The test classes where extended and refactored to also test for all future commands. Signed-off by: Mark Herwege mark.herwege@telenet.be

ollo · March 25, 2024, 8:20am

Hi Jim,

thanks for the pointer, guess I must wait for 4.2 then.

While the JRuby rule is working, I see a strange behavior. Everytime the rule runs, a copy of it gets created and not terminated until I manually save the rule again. Is that a known issue in 4.1? Any idea how to fix that? I fear that behavior will take up memory until OH crashes?!?

JimT · March 25, 2024, 11:08am

Can you show me the rule in which the Ruby rule is created? Is it a script, or a rule. If it’s the latter, what trigger does it have?

ollo · March 25, 2024, 11:26am

Hi @JimT,

this is what I have created (disabled for now):

And this is from the code tab:

configuration: {}
triggers:
  - id: "1"
    configuration:
      itemName: HTTP_URL_Thing_Voltego_VoltegoVerguetung_copy
    type: core.ItemStateChangeTrigger
conditions: []
actions:
  - inputs: {}
    id: "2"
    configuration:
      type: application/x-ruby
      script: >
        require "json"

        rule "Persist Time Series" do
          updated HTTP_URL_Thing_Voltego_VoltegoVerguetung_copy
          run do |event|
            time_series = TimeSeries.new
            
            data = JSON.parse(event.state.to_s)
            data["elements"].each do |element|
              timestamp = ZonedDateTime.parse(element["begin"])
                                       .with_zone_same_instant(ZoneId.system_default)
              time_series.add(timestamp, element["price"])
            end
            
            VoltegoVerguetungTS.time_series = time_series
          end
        end
    type: script.ScriptAction

Thanks for checking! Not sure what I’m doing here…

JimT · March 25, 2024, 11:47am

Your rule is incorrectly configured.
There are several ways to fix this.

Change the When (trigger) to “system reached start level 40”. If you do this, optionally also change the Ruby script from rule to rule! (i.e. add an exclamation mark)

This is the least preferred method

Or

Save the Ruby script as a .rb file inside conf/automation/ruby/ folder and remove the UI rule.

Method #2 is more straight forward and what I’d suggest. However, you cannot edit the rule using the UI.

Or

Change the Ruby script to:

(Excuse the indentation, typed on a phone)
This is perhaps the best solution of you prefer to use the UI

require "json"

time_series = TimeSeries.new
            
            data = JSON.parse(event.state.to_s)
            data["elements"].each do |element|
              timestamp = ZonedDateTime.parse(element["begin"])
                                       .with_zone_same_instant(ZoneId.system_default)
              time_series.add(timestamp, element["price"])
            end
            
            VoltegoVerguetungTS.time_series = time_series

ollo · March 25, 2024, 2:51pm

Hi Jim,

I went for Method #2 and it works now as intended! Thanks for the guidance!

Regards, Thomas

ollo · April 7, 2024, 11:58am

Hi @JimT ,

I noticed the time-series Item is running 1h behind. My OH instance on Docker runs on UTC:

root@dba4bb64487a:/openhab# date
Sun 07 Apr 2024 09:10:53 AM UTC

This is what I get from the HTTP item:

2024-04-07 10:35:39.279 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'HTTP_URL_Thing_Preise_copy' changed from NULL to {"elements":[{"begin":"2024-04-07T10:00:00+01:00","end":"2024-04-07T11:00:00+01:00","price":-0.325},{"begin":"2024-04-07T11:00:00+01:00","end":"2024-04-07T12:00:00+01:00","price":-0.994},{"begin":"2024-04-07T12:00:00+01:00","end":"2024-04-07T13:00:00+01:00","price":-2.679},{"begin":"2024-04-07T13:00:00+01:00","end":"2024-04-07T14:00:00+01:00","price":-2.006},{"begin":"2024-04-07T14:00:00+01:00","end":"2024-04-07T15:00:00+01:00","price":-0.504},{"begin":"2024-04-07T15:00:00+01:00","end":"2024-04-07T16:00:00+01:00","price":-0.016},{"begin":"2024-04-07T16:00:00+01:00","end":"2024-04-07T17:00:00+01:00","price":0.692},{"begin":"2024-04-07T17:00:00+01:00","end":"2024-04-07T18:00:00+01:00","price":9.146},{"begin":"2024-04-07T18:00:00+01:00","end":"2024-04-07T19:00:00+01:00","price":10.839},{"begin":"2024-04-07T19:00:00+01:00","end":"2024-04-07T20:00:00+01:00","price":10.64},{"begin":"2024-04-07T20:00:00+01:00","end":"2024-04-07T21:00:00+01:00","price":8.995},{"begin":"2024-04-07T21:00:00+01:00","end":"2024-04-07T22:00:00+01:00","price":7.746},{"begin":"2024-04-07T22:00:00+01:00","end":"2024-04-07T23:00:00+01:00","price":7.206}],"next_offset":null,"last_modified":"2024-04-06T10:45:03+00:00"}

This is the resulting time-series from the rule:

2024-04-07 10:50:57.755 [INFO ] [hab.event.ItemTimeSeriesUpdatedEvent] - Item 'PreisTS' updated timeseries [Entry[timestamp=2024-04-07T09:00:00Z, state=-0.325], Entry[timestamp=2024-04-07T10:00:00Z, state=-0.994], Entry[timestamp=2024-04-07T11:00:00Z, state=-2.679], Entry[timestamp=2024-04-07T12:00:00Z, state=-2.006], Entry[timestamp=2024-04-07T13:00:00Z, state=-0.504], Entry[timestamp=2024-04-07T14:00:00Z, state=-0.016], Entry[timestamp=2024-04-07T15:00:00Z, state=0.692], Entry[timestamp=2024-04-07T16:00:00Z, state=9.146], Entry[timestamp=2024-04-07T17:00:00Z, state=10.839], Entry[timestamp=2024-04-07T18:00:00Z, state=10.64], Entry[timestamp=2024-04-07T19:00:00Z, state=8.995], Entry[timestamp=2024-04-07T20:00:00Z, state=7.746], Entry[timestamp=2024-04-07T21:00:00Z, state=7.206]]

This is the related Number item:

2024-04-07 11:00:00.001 [INFO ] [openhab.event.ItemStateChangedEvent ] - Item 'PreisTS' changed from NULL to -0.325

I guess something is (still) wrong with the time offsets:

"begin":"2024-04-07T10:00:00+01:00","end":"2024-04-07T11:00:00+01:00","price":-0.325}

actually means 10AM…11AM my time zone which is CE(S)T.

What do I need to change in the rule to make it fit?

timestamp = ZonedDateTime.parse(element["begin"])
                         .with_zone_same_instant(ZoneId.system_default)

Thanks!

JimT · April 8, 2024, 11:05am

Instead of with_zone_same_instant, use with_zone_same_local, but I think that’s wrong.

Your date command shows that your system is set to UTC, Not CEST. So you need to set the correct time zone on your system and not alter the code. You may need to reboot for the new timezone to take effect on your system and on openhab

Openhab (java) itself needs to be set to the correct time zone also. I don’t remember off the top of my head how to do this.

Another, easier way to achieve it without fixing your system timezone is to replace ZoneId.system_default with something specific to your zone.