PI stuck after update

mrteo90 · February 9, 2021, 7:12pm

Platform information:
- Hardware: RPI4 4GB
- OS: Openhabian 1.6.2
- openHAB version: 3.0.0

So, tonight I tried to update openhabian to the newer version (1.6.3).
I SSHed into the pi, launched openhabian-config, it required to be updated so I let it update and then I selected menu option 02 Upgrade System and launched the upgrade.
The openhabian-config tool got closed and I went to bed.
When I woke up this morning I noticed that the SSH connection was closed, so i checked on the web gui, but none of the web-guis was working (checked ports 80, 8080, 9001), they all reported as 404.
So I opened another SSH session to check what was going on.

I immediately noticed something was wrong:
-The first row, where you usually see the IP Address, said that there was a problem retrieving the IP (but I was able to connect via ssh using the usual IP i set for the PI on my DHCP server)
-The openhab version was reported as 3.0.1, so it seemed the upgrade went well, but the uptime indicator showed a few days (shouldn’t the upgrade reboot the pi?).

So, at first I thought that rebooting the pi would be a good idea.
But when I tried I got an error message saying that apt-get process was preventing it to reboot, so that looked like a clear indicator that the upgrade didn’t went through.

So, at first I looked for a /opt/openHABian-install-failed file, but there was only a /opt/openHABian-install-success file which was created when I first installed openhabian.
I then looked for logs in /boot/first-boot.log but of course there were only entries from the day I installed openhabian for the first time.
I then tried to launch the upgrade process again from openhabian-config, but I got an error.

I checked /var/log/apt/term.log and noticed there were a few lines saying that a package (can’t remember which one) needed a confirmation in order to proceed with the installation.
The weird thing is that the first line of the log showed that apt-get was run with the --yes parameter, so it shouldn’t have been asking for a confirmation.

So, at this point I didn’t really know what to do, I didn’t know how to resume the update and so I did something which at this point I believe was a stupid move: sudo reboot.

From that point, I wasn’t able to connect to the PI anymore.
My router keeps showing it as disconnected and of course I get no response if I try to ping it.
I even tried a straight connection to my laptop via ethernet, but nothing.
I ran nmap, just in case it got another ip for some weird reason, but it is not connected.

So, now I’m here asking for some help.
Is there anything I can do besides flashing the sd card with a fresh openhabian image?
I have done a backup before updating, but the backup is on the openhabian partition, so I have to understand how to retrieve it in case the only solution is to start from scratch.
If I mount the SD card on my mac I only see a boot volume. Guess I will need to try on a linux VM.
Any recommendation in order to extract the backup file (and maybe some log files too)?

Thank you

dirkdirk · February 9, 2021, 7:55pm

Hi @mrteo90 ,
I experienced the exact same issue when updating from 3.0.0 to 3.0.1.
Several hours of try and error to get the pi back to life…with no success. Also my raw copy sd (done via config) wasn’t booting into openhabian.
At the end I flashed the sd with the 3.0.1 image and restored my backup which was taken before the update. Luckily I had moved it to my nas.
Unfortunately I can’t help you how the extract the backup from the sd, but I am confident someone knows how to do it and will answer you soon.

Dirk

rlkoshak · February 9, 2021, 8:33pm

This isn’t a Windows machine. Only an update to the kernel requires a reboot and I don’t think openHABian runs the apt update that would update the kernel anyway. So updating the system should almost never cause the machine to reboot.

Or more likely what happened was your ssh session was killed in the middle of the upgrade. When the ssh session was killed, all the processes started in that session were killed to, in this case the apt process. When apt starts it creates a little lock file and I suspect that tis what the error message is telling you. But you can always check to see if it’s running with ps -elf | grep apt. If nothing is returned (except for the grep itself) you know that apt isn’t running but also that it didn’t close down normally either, leaving your system in a slightly corrupted staet.

Please always post the actual error. Error messages almost always provide the clues we need to help. For example, everything I just wrote above is a guess. If I had the error I would know for sure whether or not the above is what happened or not .

Unfortunately yes, that was not the best choice.

You could plug in a monitor and keyboard and reboot the Pi and see what it says. Clearly the networking got hosed.

But ultimately, you’re either about to take a crash course on Linux in an attempt to fix this or you’re going to have to rebuild/restore from backup anyway. I’d cut my losses and rebuild/restore from backup.

A Linux VM or you’ll need to install software on the Mac that can read ext4 file systems. Paragon soft makes one but I’m sure there are others.

mrteo90 · February 9, 2021, 9:49pm

Hello @rlkoshak,

thanks for your prompt reply!

Got it, I don’t know exactly what the upgrade process for openhabian does, so I guessed that a reboot was to be expected. I was wrong.

Yes, I forgot to say that I ran ps and the apt process was still running, but was somehow stuck at that confirmation prompt I mentioned before as was reported in the term.log file.
In the meanwhile I was able to recover the term.log file, but unfortunately it seems it hasn’t been written, because the last entry it’s still from the first install day, so what I saw this morning in the log file it’s no longer there.

I didn’t think I was going to screw up openhabian, so I didn’t wrote down the error. But as far as I can remember, it was something about the apt process being in use.

Linux VM did the trick and I was able to recover the backup. I will now do a fresh install and then import the backup.

Thanks for your help

mrteo90 · February 9, 2021, 11:35pm

So, just finished restoring and…now frontail is not working.

I don’t know if this is related to the backup restoring or not, but I haven’t done anything else…
1.Flashed SD card
2.Loaded the backup file in the tmp folder
3.Set proper options in the conf file
4.Put SD card in, started up PI and as soon as it was connected I SSHed in waiting for the install to complete
5.Once the install was complete, realized that files in tmp folder were deleted, so loaded the backup file again and restored using sudo openhab-cli restore /path/to/file

I tried to re-install frontail from openhabian-config, with no luck
I restarted the service multiple times by sudo systemctl restart frontail.service
I rebooted the PI
Nothing…

By issuing sudo journalctl -xu frontail I can see this error repeating multiple times.
Any suggestion on how to proceed?

-- Logs begin at Tue 2021-02-09 23:32:34 CET, end at Wed 2021-02-10 00:26:13 CET. --

Feb 09 23:46:25 openhabian systemd[1]: Started Frontail openHAB instance, reachable at http://openhabian:9001.

-- Subject: A start job for unit frontail.service has finished successfully

-- Defined-By: systemd

-- Support: https://www.debian.org/support

--

-- A start job for unit frontail.service has finished successfully.

--

-- The job identifier is 109.

Feb 09 23:46:46 openhabian frontail[884]: undefined:21

Feb 09 23:46:46 openhabian frontail[884]: "ModelRepositoryImpl": "background-color: #a6cffd; font-weight: bold"

Feb 09 23:46:46 openhabian frontail[884]: ^

Feb 09 23:46:46 openhabian frontail[884]: SyntaxError: Unexpected string in JSON at position 896

Feb 09 23:46:46 openhabian frontail[884]: at JSON.parse (<anonymous>)

Feb 09 23:46:46 openhabian frontail[884]: at Object.<anonymous> (/usr/lib/node_modules/frontail/index.js:114:30)

Feb 09 23:46:46 openhabian frontail[884]: at Module._compile (internal/modules/cjs/loader.js:1063:30)

Feb 09 23:46:46 openhabian frontail[884]: at Object.Module._extensions..js (internal/modules/cjs/loader.js:1092:10)

Feb 09 23:46:46 openhabian frontail[884]: at Module.load (internal/modules/cjs/loader.js:928:32)

Feb 09 23:46:46 openhabian frontail[884]: at Function.Module._load (internal/modules/cjs/loader.js:769:14)

Feb 09 23:46:46 openhabian frontail[884]: at Module.require (internal/modules/cjs/loader.js:952:19)

Feb 09 23:46:46 openhabian frontail[884]: at require (internal/modules/cjs/helpers.js:88:18)

Feb 09 23:46:46 openhabian frontail[884]: at Object.<anonymous> (/usr/lib/node_modules/frontail/bin/frontail:2:1)

Feb 09 23:46:46 openhabian frontail[884]: at Module._compile (internal/modules/cjs/loader.js:1063:30)

Feb 09 23:46:46 openhabian systemd[1]: **frontail.service: Main process exited, code=exited, status=1/FAILURE**

-- Subject: Unit process exited

-- Defined-By: systemd

-- Support: https://www.debian.org/support

--

-- An ExecStart= process belonging to unit frontail.service has exited.

--

-- The process' exit code is 'exited' and its exit status is 1.

Feb 09 23:46:46 openhabian systemd[1]: **frontail.service: Failed with result 'exit-code'.**

-- Subject: Unit failed

-- Defined-By: systemd

-- Support: https://www.debian.org/support

--

-- The unit frontail.service has entered the 'failed' state with result 'exit-code'.

Feb 09 23:46:47 openhabian systemd[1]: frontail.service: Service RestartSec=100ms expired, scheduling restart.

Feb 09 23:46:47 openhabian systemd[1]: frontail.service: Scheduled restart job, restart counter is at 1.

-- Subject: Automatic restarting of a unit has been scheduled

-- Defined-By: systemd

-- Support: https://www.debian.org/support

--

-- Automatic restarting of the unit frontail.service has been scheduled, as the result for

-- the configured Restart= setting for the unit.

Feb 09 23:46:47 openhabian systemd[1]: Stopped Frontail openHAB instance, reachable at http://openhabian:9001.

-- Subject: A stop job for unit frontail.service has finished

-- Defined-By: systemd

-- Support: https://www.debian.org/support

--

-- A stop job for unit frontail.service has finished.

--

-- The job identifier is 930 and the job result is done.

------[CUTTING REPEATING MESSAGES]------

Feb 10 00:12:20 openhabian systemd[1]: Started Frontail openHAB instance, reachable at http://openhabian:9001.

-- Subject: A start job for unit frontail.service has finished successfully

-- Defined-By: systemd

-- Support: https://www.debian.org/support

--

-- A start job for unit frontail.service has finished successfully.

--

-- The job identifier is 6519.

Feb 10 00:12:22 openhabian frontail[14058]: undefined:21

Feb 10 00:12:22 openhabian frontail[14058]: "ModelRepositoryImpl": "background-color: #a6cffd; font-weight: bold"

Feb 10 00:12:22 openhabian frontail[14058]: ^

Feb 10 00:12:22 openhabian frontail[14058]: SyntaxError: Unexpected string in JSON at position 896

Feb 10 00:12:22 openhabian frontail[14058]: at JSON.parse (<anonymous>)

Feb 10 00:12:22 openhabian frontail[14058]: at Object.<anonymous> (/usr/lib/node_modules/frontail/index.js:114:30)

Feb 10 00:12:22 openhabian frontail[14058]: at Module._compile (internal/modules/cjs/loader.js:1063:30)

Feb 10 00:12:22 openhabian frontail[14058]: at Object.Module._extensions..js (internal/modules/cjs/loader.js:1092:10)

Feb 10 00:12:22 openhabian frontail[14058]: at Module.load (internal/modules/cjs/loader.js:928:32)

Feb 10 00:12:22 openhabian frontail[14058]: at Function.Module._load (internal/modules/cjs/loader.js:769:14)

Feb 10 00:12:22 openhabian frontail[14058]: at Module.require (internal/modules/cjs/loader.js:952:19)

Feb 10 00:12:22 openhabian frontail[14058]: at require (internal/modules/cjs/helpers.js:88:18)

Feb 10 00:12:22 openhabian frontail[14058]: at Object.<anonymous> (/usr/lib/node_modules/frontail/bin/frontail:2:1)

Feb 10 00:12:22 openhabian frontail[14058]: at Module._compile (internal/modules/cjs/loader.js:1063:30)

Feb 10 00:12:22 openhabian systemd[1]: **frontail.service: Main process exited, code=exited, status=1/FAILURE**

-- Subject: Unit process exited

-- Defined-By: systemd

-- Support: https://www.debian.org/support

--

-- An ExecStart= process belonging to unit frontail.service has exited.

--

-- The process' exit code is 'exited' and its exit status is 1.

Feb 10 00:12:22 openhabian systemd[1]: **frontail.service: Failed with result 'exit-code'.**

-- Subject: Unit failed

-- Defined-By: systemd

-- Support: https://www.debian.org/support

--

-- The unit frontail.service has entered the 'failed' state with result 'exit-code'.

Feb 10 00:12:22 openhabian systemd[1]: frontail.service: Service RestartSec=100ms expired, scheduling restart.

Feb 10 00:12:22 openhabian systemd[1]: frontail.service: Scheduled restart job, restart counter is at 5.

-- Subject: Automatic restarting of a unit has been scheduled

-- Defined-By: systemd

-- Support: https://www.debian.org/support

--

-- Automatic restarting of the unit frontail.service has been scheduled, as the result for

-- the configured Restart= setting for the unit.

Feb 10 00:12:22 openhabian systemd[1]: Stopped Frontail openHAB instance, reachable at http://openhabian:9001.

-- Subject: A stop job for unit frontail.service has finished

-- Defined-By: systemd

-- Support: https://www.debian.org/support

--

-- A stop job for unit frontail.service has finished.

--

-- The job identifier is 6582 and the job result is done.

Feb 10 00:12:22 openhabian systemd[1]: **frontail.service: Start request repeated too quickly.**

Feb 10 00:12:22 openhabian systemd[1]: **frontail.service: Failed with result 'exit-code'.**

-- Subject: Unit failed

-- Defined-By: systemd

-- Support: https://www.debian.org/support

--

-- The unit frontail.service has entered the 'failed' state with result 'exit-code'.

Feb 10 00:12:22 openhabian systemd[1]: **Failed to start Frontail openHAB instance, reachable at http://openhabian:9001.**

-- Subject: A start job for unit frontail.service has failed

-- Defined-By: systemd

-- Support: https://www.debian.org/support

--

-- A start job for unit frontail.service has finished with a failure.

--

-- The job identifier is 6582 and the job result is failed.

rlkoshak · February 10, 2021, 3:15pm

I use neither openhabian nor frontail so I can’t help with these errors. A common problem people who upgrade run into is the path to the logs has changed which requires a change to the frontail.service file, which doesn’t seem to always happen during the upgrade.

mrteo90 · February 15, 2021, 9:29am

Just in case anyone encounter this same issue, please see here

hyzteric · February 15, 2021, 3:33pm

Thank Matteo, that did the trick for me!