For some reason Incus refuses to start, the SystemD service terminates without successfully launching:
jūl 15 12:39:19 pc-158 systemd[1]: Starting incus-startup.service - Incus - Instance startup...
jūl 15 12:39:19 pc-158 systemd[1]: incus-startup.service: Main process exited, code=exited, status=203/EXEC
jūl 15 12:39:19 pc-158 systemd[1]: incus-startup.service: Failed with result 'exit-code'.
jūl 15 12:39:19 pc-158 systemd[1]: Failed to start incus-startup.service - Incus - Instance startup.
I think it might be related to the latest update to Incus (would like to track this down if that is true).
I see that there are a couple of unit files in /usr/lib/systemd/system/:
incus.service
incus.socket
incus-startup.service
incus-user.service
incus-user.socket
incus-workaround.service
But none of them are able to start, journalctl doesn’t reveal much information and I don’t see any daemon logs in /var/log/incus to track down why it’s failing.
sudo systemctl status incus
× incus.service - Incus - Daemon
Loaded: loaded (/usr/lib/systemd/system/incus.service; indirect; preset: disabled)
Drop-In: /usr/lib/systemd/system/service.d
└─10-timeout-abort.conf
Active: failed (Result: exit-code) since Mon 2024-07-15 12:37:13 EEST; 12min ago
TriggeredBy: ● incus.socket
Docs: man:incusd(1)
Process: 10029 ExecStart=/usr/lib/incus/incusd --group incus-admin (code=exited, status=203/EXEC)
Process: 10030 ExecStartPost=/usr/lib/incus/incusd waitready --timeout=600 (code=exited, status=203/EXEC)
Main PID: 10029 (code=exited, status=203/EXEC)
CPU: 4ms
jūl 15 12:37:13 pc-158 systemd[1]: incus.service: Scheduled restart job, restart counter is at 5.
jūl 15 12:37:13 pc-158 systemd[1]: incus.service: Start request repeated too quickly.
jūl 15 12:37:13 pc-158 systemd[1]: incus.service: Failed with result 'exit-code'.
jūl 15 12:37:13 pc-158 systemd[1]: Failed to start incus.service - Incus - Daemon.
jūl 15 12:37:20 pc-158 systemd[1]: incus.service: Start request repeated too quickly.
jūl 15 12:37:20 pc-158 systemd[1]: incus.service: Failed with result 'exit-code'.
jūl 15 12:37:20 pc-158 systemd[1]: Failed to start incus.service - Incus - Daemon.
Perhaps completely unrelated but incus-workaround.service fails because /usr/lib/incus is not present as per ConditionPathExists=/usr/lib/incus
sudo systemctl status incus-workaround -l
○ incus-workaround.service - Workaround swtpm not having the correct label
Loaded: loaded (/usr/lib/systemd/system/incus-workaround.service; enabled; preset: disabled)
Drop-In: /usr/lib/systemd/system/service.d
└─10-timeout-abort.conf
Active: inactive (dead)
Condition: start condition unmet at Mon 2024-07-15 12:40:32 EEST; 13min ago
└─ ConditionPathExists=/usr/lib/incus was not met
jūl 15 12:35:30 pc-158 systemd[1]: incus-workaround.service - Workaround swtpm not having the correct label was skipped because of an unmet condition check (ConditionPathExists=/usr/lib/incus).
jūl 15 12:40:32 pc-158 systemd[1]: incus-workaround.service - Workaround swtpm not having the correct label was skipped because of an unmet condition check (ConditionPathExists=/usr/lib/incus).
Interestingly, wanted to add device-info output but that just failed too:
ujust device-info
Gathering device info...
/run/user/1000/just/just-ShMU6M/device-info: line 120: syntax error near unexpected token `('
/run/user/1000/just/just-ShMU6M/device-info: line 120: `fpaste <(rpm-ostree status) <(fpaste --sysinfo --printonly) <<(flatpak list --columns=application,version,options)'
error: Recipe `device-info` failed with exit code 2
None the less…
rpm-ostree status
State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: no runs since boot
Deployments:
● ostree-image-signed:docker://ghcr.io/ublue-os/bluefin-dx:latest
Digest: sha256:a2bf26818d4c779cc53627018587dd7cee590858f1439f0a1dc2e001f9b000b4
Version: 40.20240715.0 (2024-07-15T04:52:31Z)
LayeredPackages: code screen teamviewer warp-terminal
ostree-image-signed:docker://ghcr.io/ublue-os/bluefin-dx:latest
Digest: sha256:055ad89d548c5fe0c169acb40f77d767433f56e2bc624b24cd09fc922e9491b8
Version: 40.20240712.0 (2024-07-12T05:15:58Z)
LayeredPackages: code screen teamviewer warp-terminal
Seems like a bug, as a workaround, starting the daemon manually works:
sudo INCUS_EDK2_PATH=/usr/share/edk2/ovmf /usr/libexec/incus/incusd --group incus-admin
I’m seeing the same thing, so it’s not just you. Thanks for the workaround, hopefully this is temporary. The workaround isn’t great; incus commands work, but the service and socket obviously still don’t run and keeping an extra.
Noticed an ujust update today got some incus packages updated:
incus 6.3-0.1.fc40 -> 6.3-0.2.fc40
incus-agent 6.3-0.1.fc40 -> 6.3-0.2.fc40
incus-client 6.3-0.1.fc40 -> 6.3-0.2.fc40
incus-selinux 6.3-0.1.fc40 -> 6.3-0.2.fc40
The services don’t crap out anymore:
sudo systemctl status incus.socket
● incus.socket - Incus - Daemon (unix socket)
Loaded: loaded (/usr/lib/systemd/system/incus.socket; enabled; preset: disabled)
Active: active (running) since Wed 2024-07-17 11:33:00 EEST; 1min 5s ago
Triggers: ● incus.service
Listen: /run/incus/unix.socket (Stream)
Tasks: 0 (limit: 18626)
Memory: 0B (peak: 476.0K)
CPU: 2ms
CGroup: /system.slice/incus.socket
jūl 17 11:33:00 pc-158 systemd[1]: Starting incus.socket - Incus - Daemon (unix socket)...
jūl 17 11:33:00 pc-158 systemd[1]: Listening on incus.socket - Incus - Daemon (unix socket).
But there are still some lingering issues as the incus client can’t interact with the Daemon:
incus list
Error: The incus daemon doesn't appear to be started (socket path: /var/lib/incus/unix.socket)
Running the same command with sudo works:
sudo incus list
+---------------+---------+------+------+-----------------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+---------------+---------+------+------+-----------------+-----------+
| gitlab | STOPPED | | | VIRTUAL-MACHINE | 0 |
+---------------+---------+------+------+-----------------+-----------+
| svn-migration | STOPPED | | | CONTAINER | 0 |
+---------------+---------+------+------+-----------------+-----------+
Setting up a variable in ~/.zshrc (or ~/.bashrc if you’re on BASH):
export INCUS_SOCKET="/run/incus/unix.socket"
…gets me closer to a better fix but there are now file access permission errors:
incus list
Error: You don't have the needed permissions to talk to the incus daemon (socket path: /run/incus/unix.socket)
ls -la /run/incus
"/run/incus": Permission denied (os error 13)
ls -la /run | grep incus
drwx------@ - root 17 jūl 11:33 incus
So in the end, to get it working as it should, I had to use the INCUS_SOCKET env variable:
export INCUS_SOCKET="/run/incus/unix.socket"
And chmod ‘/run/incus’:
sudo chmod 755 /run/incus
After which incus finally works normally:
ls -la /run/incus
srwx------@ - root 17 jūl 11:33 seccomp.socket
srw-rw----@ - root 17 jūl 11:33 unix.socket
incus list
+---------------+---------+------+------+-----------------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+---------------+---------+------+------+-----------------+-----------+
| gitlab | STOPPED | | | VIRTUAL-MACHINE | 0 |
+---------------+---------+------+------+-----------------+-----------+
| svn-migration | STOPPED | | | CONTAINER | 0 |
+---------------+---------+------+------+-----------------+-----------+
After reboot the permissions issue comes back but this error seems to have been already adressed by Incus project and should arrive with oackage updates soon: incus changes the mode of /run/incus back to 0700 · Issue #1003 · lxc/incus · GitHub
The bug is upstream in incus itself. I submitted a patch that was accepted yesterday. The daemon does a chmod 0700 on /run/incus on first start, which prevents the client from being able to see the incus socket at /run/incus/unix.socket. If you chmod 0711 /run/incus, then the client can read the directory contents and will connect automatically (without having to set an environment variable)
Here’s a link to the upstream PR that fixes it