How to get the last 5 values from the RRD4J?

[aleksey.kurchavy (Alias Nameless)]

Hi everyone!!! Is there any possibility to get the last 5 latest values from the RRD4J Persistence service. Without linking to dates. Just the last 5.

[jimtng]

There’s no direct method to do that, but you could call getAllStatesSince with a guesstimated/sufficient duration, and simply take the last 5 elements in the returned list.

[mstormi (Markus Storm)]

Sounds like an X-Y problem to me. Better explain what you want to accomplish.

[aleksey.kurchavy (Alias Nameless)]

I’m getting data from the ToF sensor about the sewage tank fullness. Every 10 minutes Tasmota wakes up, sends data touch MQTT and falls asleep again. Sometimes the sensor sends erroneous data. I have marked them on the graph. The best way to filter out such data is a median filter. To do this I need to have a cache of the last N values

[jimtng]

This might help filter that data out of your persistence in the first place
https://github.com/openhab/openhab-addons/pull/17362

Or if you need this now, you could use a script transform as a profile, or write a jruby profile yourself. It’ll just need a few lines of code. I think a “delta” type of filter would work.

[HaKuNa (Remo M)]

I use this to filter faulty values, but changed the filter criteria if the difference is too big.

Filter Profile

This comes up periodically so I figured I’d write a quick tutorial since I had to implement this temporarily in my OH instance to debug a problem. This will probably become a design pattern at some point. Problem Statement Some sensors report really frequently, let’s say every second, and each update has a tiny change. For example, a temperature sensor reports values with changes of 0.01 or smaller with each update. For persistence you can filter changes like these out so they don’t fill up you…

[binderth (Thomas Binder)]

sorry - bit OT for now, but stay with me.

@jimtng

This might help filter that data out of your persistence in the first place

If I understand this correctly, this one does filter out values, not within a certain pattern?
I would like to add to this if possible a variation in percentages instead of absolute values?

Reason I ask is, that I switched (since a long time) to using Node-Red for gathering information and “pre-processing” those and then sending cleared ones to openHAB using the “filter node”.

I also have some devices that are prone to send faulty values from time to time, which could potentially lead to really not expected behaviours of my rules.

So - that was my first thought reading this:

@aleksey.kurchavy

Sometimes the sensor sends erroneous data. I have marked them on the graph.

Use node-red as a pre-processor. Currently many MQTT-values get read from a node-red flow, get checked through a filter (like screenshot above) and are then sent in MQTT topics, which are read by openHAB. If the kind of filtering node-red offers in the filter-node can be done within openHAB, I’m very happy to use it!

see more on node-red RBE node:

node-red-node-rbe

[jimtng]

@binderth

If I understand this correctly, this one does filter out values, not within a certain pattern?
I would like to add to this if possible a variation in percentages instead of absolute values?

Using percentage was not something I had considered, but I think it’s a good idea.

So in this case, perhaps the filter could say “$DELTA_PERCENT < 25”, but I’m open to suggestions for alternative syntax.

I can (or someone else can) implement that in a separate PR. Due to the time taken to wait for a PR to be reviewed, I am reluctant to make changes to the current open PR, especially since I won’t be able to work on this until mid November.

Here’s the topic on this profile State Filter (Range Filter) Profile - #20 by jimtng

[aleksey.kurchavy (Alias Nameless)]

Thank you all! These solutions are redundant. It’s easier to create a data buffer in the metadata of an item

[rlkoshak (Rich Koshak)]

@aleksey.kurchavy

last 5 latest values from the RRD4J Persistence service

@aleksey.kurchavy

Every 10 minutes Tasmota wakes up, sends data touch MQTT and falls asleep again

The way rrd4j works is going to make this not work the way you expect. By default rrd4j saves one value every minute so the last five records will just be what ever the Item’s state happened to be in for the past five minutes whether it’s changed or not. So even if there were a way to get the last five records it’s not going to help with your original problem.

As has been suggested, filtering out the data before it gets to the database will be preferable, or switching to a different persistence database if you want to continue down this path.

@binderth

I would like to add to this if possible a variation in percentages instead of absolute values?

In the mean time, the filter profile @HaKuNa linked to can easily be modified to work on a percentage instead of absolute values. I’m still waiting for UI support but when that comes I’ll be posting this to the marketplace so it can be installed like a rule template.

[aleksey.kurchavy (Alias Nameless)]

Here’s my solution. A median filter with a buffer of size bufferLen. Buffer is stored in metadata.

configuration: {}
triggers:
  - id: "1"
    configuration:
      itemName: septic_sewerVolume
    type: core.ItemStateUpdateTrigger
conditions: []
actions:
  - inputs: {}
    id: "2"
    configuration:
      type: application/javascript
      script: >
        (function() {
          
          const bufferLen = 5;
          const metaDataName = 'buffer';
          
          const newValue = Number(event.itemState); 
          const medianaItem = items.mediana_test;
          let buffer = JSON.parse(medianaItem.getMetadata(metaDataName)?.value || '[]');

          buffer.push(newValue)
          buffer = buffer.slice(-bufferLen)
          
          medianaItem.replaceMetadata(metaDataName,JSON.stringify(buffer))
          
          buffer = buffer.sort((a,b) => a - b);
          
          const mediana = buffer[Math.floor(buffer.length / 2)];
          medianaItem.postUpdate(mediana);
        })() 
    type: script.ScriptAction