Yes. I don’t know how you came to use sudo -u, i.e. start as some ordinary user.
In a default openhabian systemd config (/lib/systemd/system/systemd-timesyncd.service
), it’s started as root.
Thanks for the hint. I can’t imagine how the user would have been changed. I didn’t know where the configuration was until you told me.
Regardless, I changed User=systemd-timesync to User=root in the config file and started the unit. It appears to be running as expected.
I received a warning that the file had been changed and I should run the daemon reload, which I did.
openhabian@openhab:/lib/systemd/system $ systemctl start systemd-timesyncd
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to start 'systemd-timesyncd.service'.
Authenticating as: ,,, (openhabian)
Password:
==== AUTHENTICATION COMPLETE ===
Warning: The unit file, source configuration file or drop-ins of systemd-timesyncd.service changed on disk. Run 'systemctl daemon-reload' to reload units.
openhabian@openhab:/lib/systemd/system $ systemctl status systemd-timesyncd
Warning: The unit file, source configuration file or drop-ins of systemd-timesyncd.service changed on disk. Run 'systemctl daemon-reload' to reload units.
● systemd-timesyncd.service - Network Time Synchronization
Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; enabled; vendor preset: enabled)
Drop-In: /lib/systemd/system/systemd-timesyncd.service.d
└─disable-with-time-daemon.conf
Active: active (running) since Sat 2022-12-03 15:32:09 CST; 5s ago
Docs: man:systemd-timesyncd.service(8)
Main PID: 6542 (systemd-timesyn)
Status: "Synchronized to time server for the first time 204.93.207.11:123 (0.debian.pool.ntp.org)."
Tasks: 2 (limit: 4915)
CGroup: /system.slice/systemd-timesyncd.service
└─6542 /lib/systemd/systemd-timesyncd
Dec 03 15:32:09 openhab systemd[1]: Starting Network Time Synchronization...
Dec 03 15:32:09 openhab systemd[1]: Started Network Time Synchronization.
Dec 03 15:32:09 openhab systemd-timesyncd[6542]: Synchronized to time server for the first time 204.93.207.11:123 (0.debian.pool.ntp.org).
openhabian@openhab:/lib/systemd/system $ systemctl status systemd-timesyncd
openhabian@openhab:/lib/systemd/system $ systemctl daemon-reload
==== AUTHENTICATING FOR org.freedesktop.systemd1.reload-daemon ===
Authentication is required to reload the systemd state.
Authenticating as: ,,, (openhabian)
Password:
==== AUTHENTICATION COMPLETE ===
openhabian@openhab:/lib/systemd/system $ B
Sorry I overlooked the User=
line it’s indeed correct to start as systemd-timesyncd, but you’ll have to find and fix where it fails due to permissions.
Eventually use sudo -u systemd-timesyncd strace/lib/systemd/systemd-timesyncd ....|& grep ....
to find out.
Thanks for the additional information. I will do as you suggest and report back.
It wasn’t happy with the changed to root:
Dec 03 15:41:23 openhab systemd-timesyncd[7214]: Failed to chmod or chown /var/lib/systemd/timesync/clock, ignoring: Operation not permitted
Dec 03 15:41:23 openhab systemd-timesyncd[7214]: Failed to change group ID: Operation not permitted
Dec 03 15:41:23 openhab systemd-timesyncd[7214]: Failed to drop privileges: Operation not permitted
Dec 03 15:41:23 openhab systemd[1]: systemd-timesyncd.service: Main process exited, code=exited, status=1/FAILURE
Dec 03 15:41:23 openhab systemd[1]: systemd-timesyncd.service: Failed with result 'exit-code'.
Dec 03 15:41:23 openhab systemd[1]: Failed to start Network Time Synchronization.
Dec 03 15:41:23 openhab systemd[1]: systemd-timesyncd.service: Service has no hold-off time (RestartSec=0), scheduling restart.
Dec 03 15:41:23 openhab systemd[1]: systemd-timesyncd.service: Scheduled restart job, restart counter is at 1.
Dec 03 15:41:23 openhab systemd[1]: Stopped Network Time Synchronization.
Dec 03 15:41:23 openhab systemd[1]: Starting Network Time Synchronization...
Dec 03 15:41:24 openhab systemd-timesyncd[7217]: Failed to chmod or chown /var/lib/systemd/timesync/clock, ignoring: Operation not permitted
Dec 03 15:41:24 openhab systemd-timesyncd[7217]: Failed to change group ID: Operation not permitted
Dec 03 15:41:24 openhab systemd-timesyncd[7217]: Failed to drop privileges: Operation not permitted
Dec 03 15:41:24 openhab systemd[1]: systemd-timesyncd.service: Main process exited, code=exited, status=1/FAILURE
Dec 03 15:41:24 openhab systemd[1]: systemd-timesyncd.service: Failed with result 'exit-code'.
Dec 03 15:41:24 openhab systemd[1]: Failed to start Network Time Synchronization.
Dec 03 15:41:24 openhab systemd[1]: systemd-timesyncd.service: Service has no hold-off time (RestartSec=0), scheduling restart.
Dec 03 15:41:24 openhab systemd[1]: systemd-timesyncd.service: Scheduled restart job, restart counter is at 2.
Dec 03 15:41:24 openhab systemd[1]: Stopped Network Time Synchronization.
Dec 03 15:41:24 openhab systemd[1]: Starting Network Time Synchronization...
Dec 03 15:41:24 openhab systemd-timesyncd[7220]: Failed to chmod or chown /var/lib/systemd/timesync/clock, ignoring: Operation not permitted
Dec 03 15:41:24 openhab systemd-timesyncd[7220]: Failed to change group ID: Operation not permitted
Dec 03 15:41:24 openhab systemd-timesyncd[7220]: Failed to drop privileges: Operation not permitted
Dec 03 15:41:24 openhab systemd[1]: systemd-timesyncd.service: Main process exited, code=exited, status=1/FAILURE
Dec 03 15:41:24 openhab systemd[1]: systemd-timesyncd.service: Failed with result 'exit-code'.
Dec 03 15:41:24 openhab systemd[1]: Failed to start Network Time Synchronization.
Dec 03 15:41:24 openhab systemd[1]: systemd-timesyncd.service: Service has no hold-off time (RestartSec=0), scheduling restart.
Dec 03 15:41:24 openhab systemd[1]: systemd-timesyncd.service: Scheduled restart job, restart counter is at 3.
Dec 03 15:41:24 openhab systemd[1]: Stopped Network Time Synchronization.
Dec 03 15:41:24 openhab systemd[1]: Starting Network Time Synchronization...
Dec 03 15:41:24 openhab systemd-timesyncd[7223]: Failed to chmod or chown /var/lib/systemd/timesync/clock, ignoring: Operation not permitted
Dec 03 15:41:24 openhab systemd-timesyncd[7223]: Failed to change group ID: Operation not permitted
Dec 03 15:41:24 openhab systemd-timesyncd[7223]: Failed to drop privileges: Operation not permitted
Dec 03 15:41:24 openhab systemd[1]: systemd-timesyncd.service: Main process exited, code=exited, status=1/FAILURE
Dec 03 15:41:24 openhab systemd[1]: systemd-timesyncd.service: Failed with result 'exit-code'.
Dec 03 15:41:24 openhab systemd[1]: Failed to start Network Time Synchronization.
Dec 03 15:41:24 openhab systemd[1]: systemd-timesyncd.service: Service has no hold-off time (RestartSec=0), scheduling restart.
Dec 03 15:41:24 openhab systemd[1]: systemd-timesyncd.service: Scheduled restart job, restart counter is at 4.
Dec 03 15:41:24 openhab systemd[1]: Stopped Network Time Synchronization.
Dec 03 15:41:24 openhab systemd[1]: Starting Network Time Synchronization...
Dec 03 15:41:24 openhab systemd-timesyncd[7226]: Failed to chmod or chown /var/lib/systemd/timesync/clock, ignoring: Operation not permitted
Dec 03 15:41:24 openhab systemd-timesyncd[7226]: Failed to change group ID: Operation not permitted
Dec 03 15:41:24 openhab systemd-timesyncd[7226]: Failed to drop privileges: Operation not permitted
Dec 03 15:41:24 openhab systemd[1]: systemd-timesyncd.service: Main process exited, code=exited, status=1/FAILURE
Dec 03 15:41:24 openhab systemd[1]: systemd-timesyncd.service: Failed with result 'exit-code'.
lines 158-230
openhabian@openhab:/lib/systemd/system $ sudo -u systemd-timesync strace /lib/systemd/system/systemd-timesyncd.service
execve("/lib/systemd/system/systemd-timesyncd.service", ["/lib/systemd/system/systemd-time"...], 0xbedcb730 /* 17 vars */) = -1 EACCES (Permission denied)
strace: exec: Permission denied
+++ exited with 1 +++
openhabian@openhab:/lib/systemd/system $
Hopefully this helps
What are the permissions of
here it is
-rw-r--r-- 1 root root 1433 Aug 6 2021 /lib/systemd/system/systemd-timesyncd.service
Thanks for the help.
openhabian@openhab:~ $ ls -l /lib/systemd/system/systemd-timesyncd.service
-rw-r--r-- 1 root root 1433 Dec 3 16:07 /lib/systemd/system/systemd-timesyncd.service
openhabian@openhab:~ $
To my untrained eye, it looks the same as yours.
Openhabian on Pi 4 4GB openHAB 3.4.0 Build #3029
Hi @BigGeorgeTx just found this thread as I am having the exact same issue.
In my case I have found NTP runs fine for awhile, then stops a few days later:
sudo systemctl status systemd-timesyncd.service
[sudo] password for openhabian:
● systemd-timesyncd.service - Network Time Synchronization
Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; enabled; vendor preset: enabled)
Active: failed (Result: start-limit-hit) since Tue 2022-12-13 03:23:15 PST; 1 day 13h ago
Docs: man:systemd-timesyncd.service(8)
Process: 18118 ExecStart=/lib/systemd/systemd-timesyncd (code=exited, status=0/SUCCESS)
Main PID: 18118 (code=exited, status=0/SUCCESS)
Status: "Idle."
CPU: 438ms
Dec 13 03:23:15 openhabian systemd[1]: Starting Network Time Synchronization...
Dec 13 03:23:15 openhabian systemd[1]: Started Network Time Synchronization.
Dec 13 03:23:15 openhabian systemd[1]: Stopping Network Time Synchronization...
Dec 13 03:23:15 openhabian systemd[1]: systemd-timesyncd.service: Succeeded.
Dec 13 03:23:15 openhabian systemd[1]: Stopped Network Time Synchronization.
Dec 13 03:23:15 openhabian systemd[1]: systemd-timesyncd.service: Start request repeated too quickly.
Dec 13 03:23:15 openhabian systemd[1]: systemd-timesyncd.service: Failed with result 'start-limit-hit'.
Dec 13 03:23:15 openhabian systemd[1]: Failed to start Network Time Synchronization.
Then I restart the service and it comes back …
openhabian@openhabian:~ $ sudo systemctl start systemd-timesyncd.service
openhabian@openhabian:~ $ sudo systemctl status systemd-timesyncd.service
● systemd-timesyncd.service - Network Time Synchronization
Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2022-12-14 17:07:26 PST; 5s left
Docs: man:systemd-timesyncd.service(8)
Main PID: 9336 (systemd-timesyn)
Status: "Initial synchronization to time server 38.229.60.9:123 (0.debian.pool.ntp.org)."
Tasks: 2 (limit: 4915)
CPU: 433ms
CGroup: /system.slice/systemd-timesyncd.service
└─9336 /lib/systemd/systemd-timesyncd
Dec 14 17:07:17 openhabian systemd-timesyncd[9336]: Initial synchronization to time server 38.229.60.9:123 (0.debian.pool.ntp.org).
Much of my system has the same output as from the steps you did with Markus.
The next time it fails I’ll gather similar info as you did to compare.
Not sure when this problem started happening, I haven’t rebooted my raspberry for 80 days, although I have restarted Openhab 3-4 times in that period. I often use the openhabian-config
tool and it updates various things (not sure what it updates but stuff from Markus) … I assume those are the config tool updates but maybe system updates too?
What command are you running here so I can replicate it?
Glad to have some company trying to troubleshoot this issue.
The information you are referring to comes from the openHAB UI, not from a command on the openhabian Pi. You can find it at “Help & About” / “Technical Information” / “View Details” / “Copy”.
Ah interesting … never noticed that thanks!
runtimeInfo:
version: 3.4.0
buildString: "Build #3029"
locale: en-US
systemInfo:
configFolder: /etc/openhab
userdataFolder: /var/lib/openhab
logFolder: /var/log/openhab
javaVersion: 11.0.16
javaVendor: Raspbian
osName: Linux
osVersion: 5.15.32-v7l+
osArchitecture: arm
availableProcessors: 4
freeMemory: 310686536
totalMemory: 1021313024
startLevel: 70
bindings:
- androiddebugbridge
- astro
- chromecast
- harmonyhub
- http
- hue
- network
- ntp
- telegram
- tplinksmarthome
- zigbee
- zwave
clientInfo:
device:
ios: false
android: false
androidChrome: false
desktop: true
iphone: false
ipod: false
ipad: false
edge: false
ie: false
firefox: false
macos: true
windows: false
cordova: false
phonegap: false
electron: false
nwjs: false
webView: false
webview: false
standalone: false
os: macos
pixelRatio: 2
prefersColorScheme: light
isSecureContext: false
locationbarVisible: true
menubarVisible: true
navigator:
cookieEnabled: true
deviceMemory: N/A
hardwareConcurrency: 12
language: en-US
languages:
- en-US
- en
onLine: true
platform: MacIntel
screen:
width: 2560
height: 1440
colorDepth: 30
support:
touch: false
pointerEvents: true
observer: true
passiveListener: true
gestures: false
intersectionObserver: true
themeOptions:
dark: light
filled: true
pageTransitionAnimation: default
bars: filled
homeNavbar: default
homeBackground: default
expandableCardAnimation: default
userAgent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36
timestamp: 2022-12-15T03:33:56.763Z
The problem happened again about 12 hours after I restarted it … same symptoms and status.
sudo systemctl status systemd-timesyncd.service
[sudo] password for openhabian:
● systemd-timesyncd.service - Network Time Synchronization
Loaded: loaded (/lib/systemd/system/systemd-timesyncd.service; enabled; vendor preset: enabled)
Active: failed (Result: start-limit-hit) since Thu 2022-12-15 03:09:45 PST; 1 day 10h ago
Docs: man:systemd-timesyncd.service(8)
Process: 13081 ExecStart=/lib/systemd/systemd-timesyncd (code=exited, status=0/SUCCESS)
Main PID: 13081 (code=exited, status=0/SUCCESS)
Status: "Idle."
CPU: 436ms
Dec 15 03:09:44 openhabian systemd[1]: Starting Network Time Synchronization...
Dec 15 03:09:45 openhabian systemd[1]: Started Network Time Synchronization.
Dec 15 03:09:45 openhabian systemd[1]: Stopping Network Time Synchronization...
Dec 15 03:09:45 openhabian systemd[1]: systemd-timesyncd.service: Succeeded.
Dec 15 03:09:45 openhabian systemd[1]: Stopped Network Time Synchronization.
Dec 15 03:09:45 openhabian systemd[1]: systemd-timesyncd.service: Start request repeated too quickly.
Dec 15 03:09:45 openhabian systemd[1]: systemd-timesyncd.service: Failed with result 'start-limit-hit'.
Dec 15 03:09:45 openhabian systemd[1]: Failed to start Network Time Synchronization.
Not sure what information maybe needed … seems to be exactly the same as what George is reporting. interesting that it is failing at roughly the same time (~3am) … probably just a coincidence.
Still having this problem and I have no idea what todo next to troubleshoot.
@Mr-JR does logrotate also fail for you.
openhabian@openhab:~ $ systemctl -a --failed
UNIT LOAD ACTIVE SUB DESCRIPTION
● logrotate.service loaded failed failed Rotate log files
● systemd-timesyncd.service loaded failed failed Network Time Synchronization
LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
2 loaded units listed.
To show all installed unit files use 'systemctl list-unit-files'.
openhabian@openhab:~ $
Happens all the time for me, sometime after systems-time synced fails.
openhabian@openhab:~ $ systemctl status logrotate.service
● logrotate.service - Rotate log files
Loaded: loaded (/lib/systemd/system/logrotate.service; static; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2022-12-20 00:00:01 CST; 21h ago
Docs: man:logrotate(8)
man:logrotate.conf(5)
Process: 26596 ExecStart=/usr/sbin/logrotate /etc/logrotate.conf (code=exited, status=1/FAILURE)
Main PID: 26596 (code=exited, status=1/FAILURE)
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
openhabian@openhab:~ $
Yes, it is also failing. I had just restarted the timesync before running this command.
FYI - I noticed I had installed the NTP Binding. I do not use this and I dont believe it is needed. My NTP was always fine until recently, its possible it happened after installing the NTP binding - not sure. Anyway, I just removed the NTP Binding and will watch to see if NTP still fails.
openhabian@openhabian:~ $ systemctl -a --failed
UNIT LOAD ACTIVE SUB DESCRIPTION
● ifup@wlan0.service loaded failed failed ifup for wlan0
● logrotate.service loaded failed failed Rotate log files
LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
2 loaded units listed.
To show all installed unit files use 'systemctl list-unit-files'.
Here is the status of logrotate … not sure what the effect of this crashing is.
openhabian@openhabian:~ $ sudo systemctl status logrotate.service
● logrotate.service - Rotate log files
Loaded: loaded (/lib/systemd/system/logrotate.service; static)
Active: failed (Result: exit-code) since Tue 2022-12-20 00:00:03 PST; 19h ago
TriggeredBy: ● logrotate.timer
Docs: man:logrotate(8)
man:logrotate.conf(5)
Process: 12208 ExecStart=/usr/sbin/logrotate /etc/logrotate.conf (code=exited, status=1/FAILURE)
Main PID: 12208 (code=exited, status=1/FAILURE)
CPU: 375ms
Dec 20 00:00:03 openhabian systemd[1]: Starting Rotate log files...
Dec 20 00:00:03 openhabian logrotate[12208]: error: skipping "/var/log/exim4/mainlog" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group shoul>
Dec 20 00:00:03 openhabian logrotate[12208]: error: skipping "/var/log/exim4/rejectlog" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group sho>
Dec 20 00:00:03 openhabian logrotate[12208]: error: skipping "/var/log/exim4/paniclog" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group shou>
Dec 20 00:00:03 openhabian systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
Dec 20 00:00:03 openhabian systemd[1]: logrotate.service: Failed with result 'exit-code'.
Dec 20 00:00:03 openhabian systemd[1]: Failed to start Rotate log files.
openhabian@openhabian:~ $
Did removing the NTP binding make a difference?
I have it on my system. I have a vague recollection of having used it for a rule when using OpenHAB. I need to search my rules and see if I’m still using it. (I do that by using the REST API to generate output of all the rules, copying that to a text editor and then searching for an item name. There is probably a better way, but it gets the job done).
Hi George, sorry forgot to follow up … no it didn’t make any difference.
I think what I’m going to do is restart the service manually nightly. Doesn’t seem there is much interest out there to solve this
Doesn’t seem you get the point here… it’s not a systemic software error, it’s your system to have the problem, others don’t have it so cannot analyze/resolve it. It’s up to you to solve it.
And even if it was systemic, there’s no vendor or other anonymous magic that would need to do that for you. openHAB is a community effort.
I have spent hours searching on various websites, even posting on some, trying to address this problem. I’m not asking someone else to solve it, but I would appreciate it if someone could suggest some areas to explore in trying to troubleshoot it. I’m willing to put in the effort, but openHAB/openHABian is my first and only Linux adventure, and I have run out of ideas.
My two setups are both running on Pi 4Bs, on different networks 3,000 km apart. I started with the openhabian distribution on the official Raspberry Pi site, and the only things running on the Pi are openHAB and related applications loaded with openhabian (such as Tailscale).
Any suggestions would be greatly appreciated.
Sorry - I didn’t make my point clear. I was not directing this comment to the Openhab development team as I do understand that it is not an Openhab issue. I meant to imply that there aren’t others beside George and myself who are seeing this and that there isn’t much interest to offer more ideas of how we might narrow down the issue.
In my case, this issue came up in the last few months as it was never an issue before. I have been running the exact same Raspberry Pi image other that what gets updated when I use the Openhab Config tool.
Still trying to troubleshoot this. Using the command suggested with my own guess for what to grep for, I get:
openhabian@openhab:~ $ sudo -u systemd-timesync strace /lib/systemd/systemd-timesyncd |& grep clock_adjtime
[sudo] password for openhabian:
clock_adjtime(CLOCK_REALTIME, 0xbeee92a0) = -1 EPERM (Operation not permitted)
sendmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="PRIORITY=3\nSYSLOG_FACILITY=9\nCOD"..., iov_len=160}, {iov_base="MESSAGE=", iov_len=8}, {iov_base="Failed to call clock_adjtime(): "..., iov_len=55}, {iov_base="\n", iov_len=1}], msg_iovlen=4, msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 224
clock_adjtime(CLOCK_REALTIME, 0xbeee92a0) = -1 EPERM (Operation not permitted)
sendmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="PRIORITY=3\nSYSLOG_FACILITY=9\nCOD"..., iov_len=160}, {iov_base="MESSAGE=", iov_len=8}, {iov_base="Failed to call clock_adjtime(): "..., iov_len=55}, {iov_base="\n", iov_len=1}], msg_iovlen=4, msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 224
clock_adjtime(CLOCK_REALTIME, 0xbeee92a0) = -1 EPERM (Operation not permitted)
sendmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="PRIORITY=3\nSYSLOG_FACILITY=9\nCOD"..., iov_len=160}, {iov_base="MESSAGE=", iov_len=8}, {iov_base="Failed to call clock_adjtime(): "..., iov_len=55}, {iov_base="\n", iov_len=1}], msg_iovlen=4, msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 224
Searching the internet, I found:
EPERM
buf.modes is neither 0 nor ADJ_OFFSET_SS_READ, and the caller does not have sufficient privilege. Under Linux, the CAP_SYS_TIME capability is required.
on clock_adjtime(2) — Arch manual pages, which I think confirms that it is a permission issue. But I’m not sure exactly what it is that needs its permission changed.