Large events.log

I have checked my events.log and it has ca 160k lines per day :astonished:.
I think it is definitely too much, this also cause large persistence db (160MB /day). Question is simple how to reduce it ?
I don’t need updates all events every minute. All date/ stages coming from:
GPIO, Satel/alarm binding, 1wire and Modbus.

1wire is fine, becuase there is (default) option to update to bus values which was updated.
In Satel there is no such option, every 20s - which I set in config data are polling to Openhab, even it is some sensor which change status very rare. Also I can not config longer time (default is 10s) because for other items I will need to wait for reaction

Same problem with GPIO or even worst because I can’t see how to set refresh time, so each items are refresh every few secunds, even with same value.

Also Modbus produce a lot of events.

Any ideas how to reduce it ?
Is it any method to disable events reporting in a log file ?

In /etc/openhab/logback.xml comment out the event bus logger:

        <!--<logger name="runtime.busevents" level="ERROR" additivity="false">
                <appender-ref ref="EVENTFILE" />
                <appender-ref ref="STDOUT" />

This will disable the event.log entirely. The event.log’s purpose is to show everything that gets posted to the event bus. All command and updates get posted there so this is an all or nothing log. If there are some things you need to track in a log, do so through rules after turning off the event.log.

In your .persist file make sure you are using an everyChange strategy and not an everyUpdate strategy. Only persist those Items you need to restoreOnStartup, chart, or access historic data. You can also use MapDB which only saves the most recent state for those Items that you only need restoreOnStartup and you other persistence only for those Items where you need historical data and charting.

Thanks Rich,

In persistence (db4o) I use Everychange + Restore on Startup for all and everyminute for 10 items which I used for charts. I think it is not much.

Problem is with events, if there are many then persistance is also large, becuase of frequent changes.
What do you think 160k events per 24h is not much ?

How to decrease it ?

The only way to change the number of events is to change whatever is producing the events to not generate so many. You’ve already tried to so all you can do from within OH is be more choosy about what does and does not get persisted.

NOTE: you only need to do everyMinute if you are using rrd4j. You don’t need to do it for db4o as far as I know.

I would use MapDB with restoreOnStartup for all and db4o only for those Items you chart or compare a previous value to the current value in rules.

Does anybody knows what is optimum quantity of events and size of persistence for Raspberry Pi2 ?

Currently I have 28k events per 24h, I don’t know is it a much or not.
I use just db4o persistence and it makes ca 250MB file per day. I have limited persistence for such data which I really needs. I use everychange / restore on start-up strategy on all (maybe this is mistake ?) items (+/- 140) and just on 9 with everyminute strategy (I will delete it for tests).
Result of that is I need to wait for a chart at least few minutes :unamused:, as I wrote above when I would delete persistence, charts will work fine, but for first 1-2 days.
I have also noticed, that web GUI (and aps on mobile) has quite large LAG, especially in setpoints - usually it takes few seconds to increase value :astonished:
Maybe it is also caused strange timing, when I have one by one instruction in rule and both of them change item I see even 30sec delay.

When I check memory on Pi it is fine with about 30% usage, procesor workload jumps up to 100% (on core) when I use GUI, if not it is much lower.

It is not easy, because of bindings config. Problem is that refresh time is set once per each binding, when I pull data from alarm system (Satel binding) I have data which I can update once per hour but some of items I must refresh more frequent :(. Same thing with Modbus, 1wire, etc.
For GPIO even I have not found such parameter as refresh time, so I dont based on what Openhab is keep asking it.
Generally I have change timing from default miliseconds to thousands of ms, so I have refresh every few minutes, but again still system has a lag and it influence on cooperation via binding (ie I must wait for change)

I will test it

What for? Is it MabDB faster ?

25k per day is HUGE! Are you sure you are doing every change and not every update?

I’m on my phone and have been traveling since 3 am so I apologize for being terseb and not explaining why.

Use MapDB for restore on startup. It only saves the last value so the db never grows so the size won’t impact performance. But MapDB is worthless for Charting or historic values.

Use rrd4j for charts and historic values of numeric values in rules. Like MapDB, rrd4j never grows and therefore the size of the db will not impact performance.

Only use db4o for those items you want to keep exact values for long time for Charting or analyais. (Months or years).

There is no reason to save every value for every item forever when all you care about is the most recent one to restore on startup. There is no reason to store every value forever for items where you only care about data from the last week or so, it don’t need exact values for older data.

Unless you are really into analyzing your data, you should only have a few items stored to db4o or rrd4j.

I hope you mean it is huge for persistence :slight_smile:
I have cutted most of them, by changing refresh time, so now I have ca 10k per 24h, but I have plan to add more items so this probably will grow up. Also now I am unable to add some rules because information about state is too late. This is big disadvantage of bindings that refresh time is set globally for all items. In only 1wire I have found refresh parameter that I can set per item, so I can set diff refresh times based on importance of item / events

I have reconfigure persistence as you suggest, I agree that there are no need to keep and remember all of events. I have used mapDB as well. Also I have not found persistence option how to control how long particular event will be keep in db, so probably my ex Db40 keeps date since it was initiated.

Now it works fine :slight_smile: I hope for next days it will be no change.
Thanks a lot !

That is correct. Anything saved to db4o is saved forever or until you manually delete it. And I don’t think it is designed to work well with dbs of many gigabytes in size.

That’s interesting problem. However I wouldn’t try to fix it on bindings’ side (like in OneWire binding), but it should be handled on OH side IMO. However in some cases this might be useful to get information about each update. But the question is: should we bother at all? You can disable event logging since this is rather for debuging purposes, and for persistence you have appropriate options (everyChange), the same for rules. All of them have been already mentioned by @rlkoshak.

Regarding Satel binding I have 1sec refresh time configured and this is not a problem at all. The good thing is that Satel gives you the information what has changed since last check, but bad thing is that the data is organized as large bitmaps of all zones/partitions/outputs, so for zone violation you have one bitmap of all zones and you get the info that something has just changed there - so all defined items for zone violation need to be updated. But still - it should not be a problem.

If I will change for 1sec then I will have more then 100k events per day :frowning:. I read about 60 items from Satel. That’s the problem I dont need to update every item such frequently , but there is genereal, one refresh parameter. Satel just give every refresh time info about items stage, if it will update just on event it will be great, but I think it is not possible. So all the time I receive events with same items, even their stage have not been changed.
BTW. Do you have broken pipe error from time to time ? When I restart OH service every 4-7 time I got such error from the binding. After that I am unable to connect automatically so I need to restart Satel. Tech support from Satel told me that there is incorrect reconnection mechanism in OH.

This is the one thing I do not like about the persistence solution. There is no way to persist real data (i.e. not averages like in rrd4j) with a defined time (or size) limit. I am using db4o (for only a few items) because I want to work with proper historic values, however, I only need the history to go back so far in time (let’s say last 24 hours, last week, etc.).

Currently there is no way to do this, and my db4o store keeps growing, :frowning:

Do you know of a way (a tool) to purge old data from db4o? I don’t want to delete the entire store since I then loose data that I care about.

Side note: From the documentation it looks like mapdb supports setting a time to live (max time, max count or max size) of a database. It would have been cool if this was supported in openHAB. I guess the mapdb support that we have now, that is mostly for restore on startup, could simply be implemented as “mapdb with a max count of 1” in this case.

What do you think?

There are some tools that claim to allow you to manually view and edit db4o. I’ve not used them so can’t say whether I’m correct on that.

With rrd4j, keep in mind it only starts to compress the data after a week or so so if you only need exact Vanessa fur such a short period of time rrd4j would probably work.

But I do agree, OH could really use a mechanism to clear out old data in the dbs that just continue to grow. Be it a setting like MapDB, or an action.

I like the idea of an action because you can write rules to find time which Items are cleared out and for how long.

@sherif I am not telling you to switch to 1sec refresh time, but such configuration works well for me. Regarding your problem, why don’t you disable event logging at all? It’s easiest and reasonable solution in your case.

Regarding broken pipe error I haven’t seen antyhing like that on my installation yet and also haven’t heard from other people. Does this happen at start or after some time? In my case it sometimes (over week or so) disconnects on a timeout and the immediately reconnects again.

How to disable event logging ?
Can I disable only Satel’s part, sometimes I use it for troubleshooting, so I need the rest ?
Maybe I will test it, now when I reconfigured persistence I can back to many events, if it will not impact on system stability and speed.
What do you mean by"

The good thing is that Satel gives you the information what has changed since last check
Is it any other method to update items then frequently asked all items to refresh ?

Broken pipe I have received after some Openhab restart or system restarts (during initialization), it looks like follows:

2016-05-23 14:39:43.117 [INFO ] [o.o.b.s.i.protocol.Ethm1Module] - Connecting to ETHM-1 module at
2016-05-23 14:39:43.130 [INFO ] [o.o.b.s.i.protocol.Ethm1Module] - ETHM-1 module connected successfully
2016-05-23 14:39:43.169 [ERROR] [o.o.b.s.i.protocol.SatelModule] - Unexpected exception occurred during writing a message Broken pipe
at Method) ~[na:1.8.0]
at ~[na:1.8.0]
at ~[na:1.8.0]
at org.openhab.binding.satel.internal.protocol.SatelModule.writeMessage( [bundlefile:na]
at org.openhab.binding.satel.internal.protocol.SatelModule.communicationLoop( [bundlefile:na]
at org.openhab.binding.satel.internal.protocol.SatelModule.access$1( [bundlefile:na]
at org.openhab.binding.satel.internal.protocol.SatelModule$CommunicationWatchdog$ [bundlefile:na]
at [na:1.8.0]
2016-05-23 14:39:43.172 [INFO ] [o.o.b.s.i.protocol.Ethm1Module] - Closing connection to ETHM-1 module
2016-05-23 14:39:44.113 [INFO ] [o.o.b.s.i.protocol.Ethm1Module] - Connecting to ETHM-1 module at

and so on, then OH will never connect to Satel without Satel restart.

Comment out EVENTFILE appender ref in “runtime.busevents” logger (logback.xml). But you cannot disable specific events in the log, just enable/disable the log completely.

Regarding refresh: the bindig sends command to check what has changed, there is such thing. But this is for all kind of devices’ status. It is hard to explain if you’re not familiar with the protocol. Basically you for example some device has “violation” status changed, but you don’t know which, so you must update all of items with that status (zone:violation:xxx), which helps somehow but not much.

Regarding broken pipe: it looks like your ethm hangs for some reason. I would upgrade the FW to latest version (unless you already did this). Do you use encrypted connection or plain?
After while I think the reason could be also in disconection. If you restart so frequently it might happen that the module does not close the connection and since it allows only one client it will not allow you to connect again. However this happened only once to me when I was replacing a switch, never in regular use.

Thanks for info !

I understand that binding just refresh all items, but it will be nice if refresh time can be diff for diff items. Then I dont need to refresh so often all items.

Yesterday I got next broken pipe. I have recognized that roller shutters didn’t closed after dark, so I have check a log:

2016-06-06 19:16:25.448 [INFO ] [o.o.b.s.i.protocol.Ethm1Module] - ETHM-1 module connected successfully
2016-06-06 19:16:25.466 [INFO ] [.service.AbstractActiveService] - Exec Refresh Service has been started
2016-06-06 19:16:25.503 [ERROR] [o.o.b.s.i.protocol.SatelModule] - Unexpected exception occurred during writing a message Broken pipe
at Method) ~[na:1.8.0]
at ~[na:1.8.0]
at ~[na:1.8.0]
at org.openhab.binding.satel.internal.protocol.SatelModule.writeMessage( [bundlefile:na]
at org.openhab.binding.satel.internal.protocol.SatelModule.communicationLoop( [bundlefile:na]
at org.openhab.binding.satel.internal.protocol.SatelModule.access$1( [bundlefile:na]
at org.openhab.binding.satel.internal.protocol.SatelModule$CommunicationWatchdog$ [bundlefile:na]
at [na:1.8.0]

It was after PI restart. I can try to update ETHM, but according support it will not help, becuase it wasn’t such fix on this kind of issues. they told me that it looks like OH keeps connection and don’t connect properly. ETHM after connection broke is ready for next connection. It is not so frequently, because I do restarts after some changes, ca once per week.
I dont use encryption for integration.

Hi Marcin, this might be RPi issue only, as none of users I am in touch with has this issue. Let’s move to priv to solve this issue.