I am local,
while in the logs I see exceptions and the like
(/var/log/openhab/openhab.log)
the UI doesn’t show me anything all.
What is actually sent to the frontend? Where does it gets its logs from? How can I investigate this?
Those exceptions are DEBUG level, not ERROR or WARN. I’m not sure if that’s relevant or not but it’s worth noting.
In the upper right there is a badge showing the number filtered log entries shown/total number log entries received (I think). Do the numbers match?
If not maybe the debug logs are being filtered?
My understanding is there is a websocket that the log viewer subscribes to. But Chris can answer more fully.
Hey all,
My apologies for the confusion and any frustration. I was implementing unrelated changes last night that required an update to the base image and got a little trigger happy and pushed the removal of frontail as well. That was a mistake.
The forced removal for users on openHAB 4.3 has been removed and will be fixed to display a more prominent warning / prompt. I will also be adding the install routine back with this change as well for any who want to keep it around in the meantime.
As for the docs, one of my projects over Christmas break is to try and finish rewriting and fixing all of the openHABian docs as they are in much need of help.
These changes should be pushed in the next day or two. Sorry for any inconvenience and trouble these changes caused.
9 Likes
This is currently not possible. As I said above, there are changes required to the core to do this.
Currently, the functionality in the core implements a log appender, so it receives logs as they are generated and sends them to the UI. There is no storage of logs in the core, so the UI cannot request old data.
You can always view the log file itself. Personally, I found that frontail didn’t work well for viewing startup logs since it only stored logs for a relatively short time. Of course, this depends on the amount of logging each user has configured, but frontail only shows about 1 minute for me, and then I have to revert to reading the log files in the file system - this is still available now and for me at least, these time critical periods are better with the logs.
So one of the drivers for me to implement the log viewer was I got a new server to run OH on. It’s not an RPi, and I installed everything manually. When I went to install frontail, I looked at the huge list of dependencies and decided I didn’t want to install this and bog down my system.
All other alternatives that I looked at are reasonably heavy - like Rich, I first looked around to try and find something, but even the “lighteight” alternatives were not attractive to me - of course others will have a different view on the benefits, downsides and risks of installing these, but I was not keen.
I’m not sure that’s strictly correct. I believe that effectively the events log is a subset of the logging, and the logging will show all logs that you have enabled.
As above, from what I understand from looking at this a few years back, the core registers a log appender, and this appender receives “all” logs. I think it is configured in the same way as any other log appender defined in the xml file.
Note that I didn’t write this - it was added to the core some time ago. I have plans to add functionality to this to (for example) buffer locally on the core so that when the UI opens it can see old logs, to filter logs on the server, and possibly to “chunk” logs together to improve performance, but at the moment the core implementation is very simple - it just pipes logs receives from the log system to the websocket interface.
I will also add that the idea with the viewer is ultimately to enhance on what was previously available. I think the highlighting is much improved (I don’t think Frontail had the ability to configure anything!), and we can now export as either HTML or CSV so providing logs to maintainers should be easier. This was only submitted as a PR a couple of weeks before 4.3 was released and the maintainers did a great job to review and get this in to 4.3. I agree it still requires some work - especially in the core, but it wasn’t possible to add everything in the short time available and personally for me I still think this is a useful tool.
5 Likes
The only tool that works with large amount of logs also and also at startup is lnav. The docs mention something about SQL
Query Logs Using SQL
Log files are directly used as the backing for SQLite virtual tables. This means you can perform queries on messages without having to load the data into an SQL database.
I think we are both saying the same thing. For most users the logs are the two files whicih is why I mentioned them. A better way to have said it would have been “you can see statements in that screen shot that would normally be logged to both openhab.log and events.log.”
It’s a good point to make that the logging is independent from the files they are saved to and the log viewer is unaware of the log files.
Absolutely, it’s a hugely useful new addition!
lnav doesn’t have a webpage view which is kind of the whole point. It’s a great command line tool but that doesn’t help anyone replace Frontail.
1 Like
Log files do exactly the same thing - they are stored as text files instead of into SQL.
There are a number of good viewers out there that can view, filter and enhance offline logs since this is a pretty standard format across linux systems. However, the idea of the UI is to provide a simple, embedded and useful real-time debug tool to users who may be debugging bindings or rules (etc, etc).
Every tool has advantages and disadvantages, and nearly no tool will do everything everyone wants without some negative impact at least.
Well you mentioned also instead of focusing on just logs to create some kind of web terminal and to me that sounds like a great ideea where lnav can shine and also with it’s SQL capability and json output can still be an great alternative for debug …
I have been using lnav for the past years and it never failed me even with large amount of logs. I just mentioned it because of my experience with it has been solid debugging live startup problem going back into time etc. I even have some utilities to pass some specific commands.
I believe if Rick ideea of making an terminal ui this should be default tool for the logs… Also as a side note frontail was crashing my web browser and there was also a guy here in the forums that made an attempt and once I gave him my logs he said I give up it’s impossible.
2023
70 000 / 24 / 60 / 60 = 0.8 message per second. Wow!
I didn’t do stress tests. But even if js doesn’t break, it’s not convenient to work with that much data. Navigation elements are needed. Maybe filters by day and hour.
It would be interesting to try to parse your logs. Can you send it to me?
There really hasn’t been any comments on this front so I’m still not sure if it’s a good idea. As I said in my initial post, I’ve concerns about exposing the ssh terminal in this way. I’m taking the silence thus far on this to mean most have the same concerns and a special purpose log viewer might be a better choice.
But there’s nothing stopping anyone from installing lnav or WebSSH or whatever if that’s what they want. We are mainly focused on what openHABian should handle.
That’s great - I’m sure it works fine and I am in no way knocking it. My point was that you stated that lnav was the ONLY tool available and my point is that actually log files are doing the same thing.
This also requires additional dependancies. My point is simply that a simple text log with a good front end will do the same as lnav.
My system logging into a text file is much much higher than this. 70000 log entries per day is pretty. low for me 
The zigbee binding in debug mode will potentially generate large logs - I have one here that is 4200 lines over 2 minutes and 3 seconds - that’s an average on 29 milliseconds - or 34 / second, but I’ve seen much higher rates than this. This would equate to around 3 million lines per day and I often run with this level of debugging.
Wouldn’t it be possible to use tailscale for this ? Tailscale also offers ssh in the browser.
1 Like
Yeah…you and I have…uh, slightly different requirements 
I can, of course. I run frontail and OH in separate docker containers, so OH outputs the log files in a fairly accessible shared location. I could also access the console or container shell, and I can manage my containers with portainer so I even have web access to those interfaces. I’ve even got rsyslog setup and running for other things and could just fold OH into that. It’s not that I don’t have options, but at present no other options are integrated into my general flow quite the same way frontail is. The new built-it viewer is close and covers 90+% of that flow so it’s just a matter of taking the time somepoint soon to find the best way to cover those last few %.
Asolutely that could be an option. But that requires a dependency on a third party cloud service. That alone is going to cut out a bunch of users from wanting to use that. Ideally we’d have something completely local.
It’s a great option for those who will want access to their machine remotely. Frontail et al isn’t going to provide that for you.
For me, an ssh web terminal is a different topic. Personally, if it can be secured, I think it would be useful, however it’s in no way useful for logging. Sure - you can view logs in a terminal, but debugging applications requires a lot more functionality - filtering, highlighting and ease of exporting are just some such things that are needed (IMHO!).
As a developer, the only way we can see what users see is through the logs - so we need something that allows users to filter down what is happening - highlight the data and make it available. A simple ssh terminal will not do that - it’s more specialised (although it could of course be in some way built in to a terminal, I think it’s a lot less friendly than a dedicated UI for the majority of users).
3 Likes
Viewing the logs is pretty much all that Frontail provided.
Beyond that, the rest of the functionality would be provided by some tool installed on the host. I use multitail. @stamate_viorel likes lnav. There are several other great command line based log tools out there (a lot more than web based ones). And openHABian can add aliases to the shell to make bringing them up both quick and easy for the users who are not excited about access to the terminal.
But we are really just talking about catching those first few dozen or so lines of log that happened when OH was starting up before the log viewer is usable, right? I’m definitely not looking to replace or reproduce what your log viewer does. Just, as @JustinG said, handling that last few % is what I’m after.
I would expect most of the users who need to get logs to us would be using the log viewer built into MainUI in the long run. I love the copy and export to CSV options! It’s only when that isn’t available or the logs needed happened before the log viewer was able to receive the events that this would be used for mucking about with the terminal.
For those who prefer command line only tools or want to capture a fulle OH reboot’s logs or the like and still have those in the browser I was thinking a terminal could serve that purpose plus give a web based way to administer the machine it’s running on.
And even here, we are only talking about openHABian deployments.
1 Like
That’s what I understood, too (although with my log level it’s rather some 1000 lines 
).
Then again, filter requirements on this are very much different from OH’s in-operations requirement for filtering.
Plus, and I think that’s key, once you got your OH installation working, you will not want to monitor that any longer except under rare error conditions on restarting, right ?
A simple tail -f will do fine in these cases.
Long story short, I think we don’t need another log viewer for that in openHABian at all once the built in viewer will be enhanced as requested above.
Sure - I’m familiar with that.
Or not. I personally use an application on my Mac that parses the log files. It provides filtering, highlighting etc, but it doesn’t run on the host.
Maybe - but maybe it’s also possible for the UI viewer to capture these. I’ve already started to look at this. As I’ve said in a few places (including this thread I think) - the core side needs some work - at the moment it just sends the logs to the UI in real time…
I want to add filtering and buffering. We MAY find that once this buffering is added, that you will get the startup logs. The implementation in the core is just an appender - the “same” as the file appender that writes to the openhab.log file. So, as long as this bundle is started early, we should capture these log entries as well, and with buffering (ie storing the logs locally in the core) they may be available in the UI log viewer…
We’ll need to see…
Again - I pretty much never use the terminal for log viewing. I might very occasionally for a quick look at something, but my goto place are the log files themselves - not using a text viewer (normally!), but using a dedicated application for viewing logs. These are ALWAYS available and require no terminal etc - just a shared folder.
Anyway, I agree - everyone does, and can do, their own thing. That’s fine. The log viewer was a quick initial work to get a good level of functionality for 99% of use cases.
1 Like
It would be awesome if the log viewer can handle it all. That would just leave the openHABian initial installation output (which really hasn’t been part of this discussion).
if you are a developer and are restarting openHAB for development purposes, I really fail to understand why opening a terminal and typing
tail -f /var/log/openhab/openhab.log -f /var/log/openhab/events.log
before your restart is so hard
3 Likes