Serious 2.5 Release issues. How to troubleshoot?

Did I miss something or has Pushover (action) been changed or is this just yet another unexplained issue on my system… Since yesterday my rules containing pushover send message suddenly no longer works. This is the error I get:

2020-01-06 17:59:00.545 [ERROR] [ntime.internal.engine.RuleEngineImpl] - Rule 'Send besked hvis nogen er ved døren': The name 'sendPushoverMessage' cannot be resolved to an item or type; line 8, column 3, length 86

I havn´t touch the rule at all.
This is the rule given the above error. I have several other rules which uses Pushover, given the same error:

rule "Send besked hvis nogen er ved døren"
when

	Item Node13_AlarmBurglar changed from OFF to ON

  then
    if(kimsPhone.state === OFF)
  sendPushoverMessage(pushoverBuilder("Der er nogen ved hoveddøren!").withSound("none"))

end

All I did yesterday was to update all apt´s available. Since then, the above error.

EDIT - uninstall and reinstall pushover does not change anything. (except it resolve in the same error as install/uninstall other actions/binding like descriped in the first post.

EDIT again - As another frustation act, I went into openhabian-config and ran the update to latest stable again. Though it didn´t install anything new, after OH restarted Pushover works again…

This is really odd!!

If something like that problem with the Pushover problem occurs again, log into the karaf console and run:

bundle:list | grep -i pushover

Obviously, replace the “pushover” with the name of the add-on you are having issues with.This will definitively show whether or not the add-on is installed or not, whether it’s active, and whether you may somehow have two versions installed.

I suspect that the update fixed it because clear the cache is part of the update process, though if there was no update to openHAB from apt there wouldn’t have been even that much.

I can’t explain the behavior or give any further recommendations until/unless it happens again.

Good idea… I just saw that the action was installed in PaperUI, which is why I uninstalled it and installed it again. Unfortunatly that did change anything untill I ran the update from openhabian-config…
I´m not sure the update clear the cache, cause when OH restarted, I saw no sign of everything beeing reinstalled like it used to after a clear cache.

I have a similar problem with the telegram action. The sendTelegramMessage isn’t resolved error’s in log. But a few time after startup same rules works find and send.
From my point of view it seem’s so, that the actions are not ‘started’ when first rules are triggered.

That could be the case. For Telegram though, there is now a v2 Binding that provides the Action now. I suspect the separate Telegram Action would be considered deprecated at this point. Perhaps that would address the issue for that service at the least. Doesn’t help with Pushover though.

I have no better ideas anymore than to try something drastic.

In a thought of this specific issue, and some of my previous issues, could be due to beeing hardware failures, I´ve just bought a Rpi 4B (4gb ram version) I´ll setup from scratch, which will be installed on a new SSD as well, (I know, I cant USB boot on the Rpi 4 yet. I´ll have to live with boot from SD card and move root to USB).

This way, any chance of this beeing a hardware (Rpi/SSD) issue should be eliminated, hopefully!
If the issues still appear, I have four Rpi´s for sale, cheap! :smiley:

If you go down that path try to be deliberate and avoid doing too much all at once. For example, install and migrate stuff for one binding at a time. That will help isolate the problem more easily if it turns out to be a problem with the configuration.

I will… In some situations I´ll have to, cause I´ll have to disable my “faulty” system, and then install the same configuration on the new system one by one anyway for the physical devices like my zwave dong, RS485 dongle etc…

Some good and some bad news from here…

I order to reinstall my system on the Rpi4, I first started adding the bindings one by one… No issues at all. Then I started to add the services, all went well untill I installed the Google Cloud Text-To-Speech (under voice menu). Then the same issue started again. (Netamo going offline and a few seconds after online again).

To make sure it really is caused by the Google TTS, I uninstalled this service, and started to uninstall/reinstall several other things… No issues at all.
The I installed Google TTS again, and again the issue is back as soon as I install or uninstall something from PaperUI, actions, misc, persistens no matter what I install from PaperUI, this issue appear when Google TTS is installed.

So the bad news is, I can reproduce this issue, even on a fresh setup using new hardware.
The good news is, its has to be a combination of the system with the Google TTS somehow.

I guess it´s time to file a issue now.

EDIT - HMMPRF! Dont know why, but I just had a idea… Having the Google TTS uninstalled, I installed Habmin… Then the issue came back :rage: Unistalled Habmin again, and the issue seem gone again.
Guess I need to dig deeper into this one.

This is driven me nuts here… Something is very wrong… :hot_face:

I can reproduce the issue when installing the Rest Docs as well, and without the Rest Docs, the issue seems gone… BUT! I know for sure I could install Rest Docs without this issue, cause I did it last night when I started to reinstall on the new Rpi4. And it didn´t reproduce the issue at that time…

So I´m back to nothing clearly again, except I can reproduce the issue for sure when installing:
Google TTS
Habmin
Rest Docs.

Without these, the issue seems gone.
But I havn´t tried every other actions, bindings, Misc, Persistence, Transformations, UI or Voice. Only some of them.

How the heck do I file an issue from this. And which repo is a issue like this suppose to be filed under?
Any advice @rlkoshak or others??

You say that when ever you “install/uninstall a binding, bindings X, Y, Z all appear to restart. This behavior appears to only occur when bindings A, B, C are installed.”

Let the developers ask for more information if they need it.

Which repo? and which binding do you suggest I use in the subject?

It’s almost certainly a core issue. If not they will tell you. Don’t put a binding in the title, just say “Some bindings” or something generic.

Issue filed… Lets see what happens…

Its kinda sad this is the stable release. It makes it a bit pointless continuing setting up my new Rpi4 for now :cry:

I recall a similar behavior to what you have reported after updating to one of the 2.5 snapshots a while back. Search the forum for the Ephemeris configuration issues. This may be the root of the problem.

There are other posts, but here’s one…

I know Scott. My comments about the emphesis was mainly because of frustrations of going from a unstable to a stable version, and then new issues is beeing introduced.

Some extra info…

I stripped my whole (new) Rpi4 system from everything, leaving only the netamo binding with the API, Indoor and outdoor module (things) added.

With only Netamo binding installed (and the standard UI´s), I can reproduce the issue adding either Google TTS, Habmin or Rest Docs.

To eliminate (or verify) the netamo binding beeing the cause. I uninstalled the netamo things and binding.
Then I added the Expire binding (cause it was affected a as well by the issue).
After that I added Google TTS again… The issue is still there… Which leaves out the netamo binding beeing the cause. Uninstalled Google TTS again.

I then shut down OH, cleared cache (which I have done many times during the whole troubleshooting process).
Went into openhabian-config and deselected the serial ports setting, (cause I had enabled them at frist startup).
Rebooted the system.
When it was ready, I went into PaperUI igen, making sure I only had the Expire binding installed, (and the standard UI´s).
Added Google TTS - Issue is still there (expire shut down, and return back within a few seconds). Uninstalled Google TTS again.
Added Rest Docs - Issue still there (expire shut down, and return back within a few seconds). Uninstalled Rest Docs again.
Added Habmin - Issue still there (expire shut down, and return back within a few seconds). Uninstalled Habmin again.

###############################################################################
###############  openhab  #####################################################
###############################################################################
##        Ip = 10.4.28.221
##   Release = Raspbian GNU/Linux 10 (buster)
##    Kernel = Linux 4.19.57-v7l+
##  Platform = Raspberry Pi 4 Model B Rev 1.1
##    Uptime = 0 day(s). 0:13:1
## CPU Usage = 0.75% avg over 4 cpu(s) (4 core(s) x 1 socket(s))
##  CPU Load = 1m: 0.08, 5m: 0.09, 15m: 0.04
##    Memory = Free: 3.11GB (81%), Used: 0.74GB (19%), Total: 3.86GB
##      Swap = Free: 0.09GB (100%), Used: 0.00GB (0%), Total: 0.09GB
##      Root = Free: 100.88GB (97%), Used: 2.80GB (3%), Total: 109.28GB
##   Updates = 0 apt updates available.
##  Sessions = 1 session(s)
## Processes = 119 running processes of 32768 maximum processes
###############################################################################

              Welcome to            __  _____    ____  _
            ____  ____  ___  ____  / / / /   |  / __ )(_)___ _____
           / __ \/ __ \/ _ \/ __ \/ /_/ / /| | / __  / / __ `/ __ \
          / /_/ / /_/ /  __/ / / / __  / ___ |/ /_/ / / /_/ / / / /
          \____/ .___/\___/_/ /_/_/ /_/_/  |_/_____/_/\__,_/_/ /_/
              /_/
                  openHAB 2.5.0-1 (Release Build)


Have in mind - This issue I first noticed on my main Openhab system which is running Rpi3B+ and OH 2.5 Stable release (upgaded from 2.5M2).
And now I´m able to reproduce the issue on a new setup using a Rpi4B (4GB Ram). (all apt updated, Rpi firmware updated as well on the Rpi4).
Both systems are running with a SSD connected to the USB port.
I have NOT tried with the SD card only.

This should be enough info for someone else to try reproduce the issue.

I hope you’ve added this information to the issue. What would be interesting is to see whether there are bindings that do not show the behavior.

Could you provide an openhab.log from startup?

I did ofcouse :slight_smile:

There are… I could limit the issue to the Habmin, Rest Docs and Google TTS. I did try quite a few others, but there were no issue, if one of these three wasn´t installed. I did not try all of them ofcouse… It would take forever.
The worse part is, it´s not all bindings which are affected by the issue. On my main OH system I can tell for sure it´s netamo, Zwave, Expire and… Hmm one other I dont recall right now. The worse is the zwave ofcouse, cause it can take up to several minutes to return back online.

I really hope this is enough for the developers to spot the cause and get it fixed quickly… Remember, it´s the stable release. They need to get this fixed as fast as possible.

Sure, but it´s really borring… (from a fresh startup a few minutes ago).

2020-01-09 23:47:29.413 [INFO ] [.dashboard.internal.DashboardService] - Started Dashboard at http://10.4.28.221:8080
2020-01-09 23:47:29.416 [INFO ] [.dashboard.internal.DashboardService] - Started Dashboard at https://10.4.28.221:8443
2020-01-09 23:47:29.562 [INFO ] [b.core.service.AbstractActiveService] - Expire Refresh Service has been started
2020-01-09 23:47:30.170 [INFO ] [openhab.ui.paper.internal.PaperUIApp] - Started Paper UI at /paperui
2020-01-09 23:47:30.306 [INFO ] [panel.internal.HABPanelDashboardTile] - Started HABPanel at /habpanel

I quess I should somehow activate some debug logging. I´m not sure how to though?