Incus daemon won't start

For some reason Incus refuses to start, the SystemD service terminates without successfully launching:

jūl 15 12:39:19 pc-158 systemd[1]: Starting incus-startup.service - Incus - Instance startup...
jūl 15 12:39:19 pc-158 systemd[1]: incus-startup.service: Main process exited, code=exited, status=203/EXEC
jūl 15 12:39:19 pc-158 systemd[1]: incus-startup.service: Failed with result 'exit-code'.
jūl 15 12:39:19 pc-158 systemd[1]: Failed to start incus-startup.service - Incus - Instance startup.

I think it might be related to the latest update to Incus (would like to track this down if that is true).

I see that there are a couple of unit files in /usr/lib/systemd/system/:

incus.service
incus.socket
incus-startup.service
incus-user.service
incus-user.socket
incus-workaround.service

But none of them are able to start, journalctl doesn’t reveal much information and I don’t see any daemon logs in /var/log/incus to track down why it’s failing.

sudo systemctl status incus
× incus.service - Incus - Daemon
     Loaded: loaded (/usr/lib/systemd/system/incus.service; indirect; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: failed (Result: exit-code) since Mon 2024-07-15 12:37:13 EEST; 12min ago
TriggeredBy: ● incus.socket
       Docs: man:incusd(1)
    Process: 10029 ExecStart=/usr/lib/incus/incusd --group incus-admin (code=exited, status=203/EXEC)
    Process: 10030 ExecStartPost=/usr/lib/incus/incusd waitready --timeout=600 (code=exited, status=203/EXEC)
   Main PID: 10029 (code=exited, status=203/EXEC)
        CPU: 4ms

jūl 15 12:37:13 pc-158 systemd[1]: incus.service: Scheduled restart job, restart counter is at 5.
jūl 15 12:37:13 pc-158 systemd[1]: incus.service: Start request repeated too quickly.
jūl 15 12:37:13 pc-158 systemd[1]: incus.service: Failed with result 'exit-code'.
jūl 15 12:37:13 pc-158 systemd[1]: Failed to start incus.service - Incus - Daemon.
jūl 15 12:37:20 pc-158 systemd[1]: incus.service: Start request repeated too quickly.
jūl 15 12:37:20 pc-158 systemd[1]: incus.service: Failed with result 'exit-code'.
jūl 15 12:37:20 pc-158 systemd[1]: Failed to start incus.service - Incus - Daemon.

Perhaps completely unrelated but incus-workaround.service fails because /usr/lib/incus is not present as per ConditionPathExists=/usr/lib/incus

sudo systemctl status incus-workaround -l
○ incus-workaround.service - Workaround swtpm not having the correct label
     Loaded: loaded (/usr/lib/systemd/system/incus-workaround.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: inactive (dead)
  Condition: start condition unmet at Mon 2024-07-15 12:40:32 EEST; 13min ago
             └─ ConditionPathExists=/usr/lib/incus was not met

jūl 15 12:35:30 pc-158 systemd[1]: incus-workaround.service - Workaround swtpm not having the correct label was skipped because of an unmet condition check (ConditionPathExists=/usr/lib/incus).
jūl 15 12:40:32 pc-158 systemd[1]: incus-workaround.service - Workaround swtpm not having the correct label was skipped because of an unmet condition check (ConditionPathExists=/usr/lib/incus).

Interestingly, wanted to add device-info output but that just failed too:

ujust device-info
Gathering device info...
/run/user/1000/just/just-ShMU6M/device-info: line 120: syntax error near unexpected token `('
/run/user/1000/just/just-ShMU6M/device-info: line 120: `fpaste <(rpm-ostree status) <(fpaste --sysinfo --printonly) <<(flatpak list --columns=application,version,options)'
error: Recipe `device-info` failed with exit code 2

None the less…

rpm-ostree status
State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: no runs since boot
Deployments:
● ostree-image-signed:docker://ghcr.io/ublue-os/bluefin-dx:latest
                   Digest: sha256:a2bf26818d4c779cc53627018587dd7cee590858f1439f0a1dc2e001f9b000b4
                  Version: 40.20240715.0 (2024-07-15T04:52:31Z)
          LayeredPackages: code screen teamviewer warp-terminal

  ostree-image-signed:docker://ghcr.io/ublue-os/bluefin-dx:latest
                   Digest: sha256:055ad89d548c5fe0c169acb40f77d767433f56e2bc624b24cd09fc922e9491b8
                  Version: 40.20240712.0 (2024-07-12T05:15:58Z)
          LayeredPackages: code screen teamviewer warp-terminal

Seems like a bug, as a workaround, starting the daemon manually works:
sudo INCUS_EDK2_PATH=/usr/share/edk2/ovmf /usr/libexec/incus/incusd --group incus-admin

I’m seeing the same thing, so it’s not just you. Thanks for the workaround, hopefully this is temporary. The workaround isn’t great; incus commands work, but the service and socket obviously still don’t run and keeping an extra.

Noticed an ujust update today got some incus packages updated:

  incus 6.3-0.1.fc40 -> 6.3-0.2.fc40
  incus-agent 6.3-0.1.fc40 -> 6.3-0.2.fc40
  incus-client 6.3-0.1.fc40 -> 6.3-0.2.fc40
  incus-selinux 6.3-0.1.fc40 -> 6.3-0.2.fc40

The services don’t crap out anymore:

sudo systemctl status incus.socket
● incus.socket - Incus - Daemon (unix socket)
     Loaded: loaded (/usr/lib/systemd/system/incus.socket; enabled; preset: disabled)
     Active: active (running) since Wed 2024-07-17 11:33:00 EEST; 1min 5s ago
   Triggers: ● incus.service
     Listen: /run/incus/unix.socket (Stream)
      Tasks: 0 (limit: 18626)
     Memory: 0B (peak: 476.0K)
        CPU: 2ms
     CGroup: /system.slice/incus.socket

jūl 17 11:33:00 pc-158 systemd[1]: Starting incus.socket - Incus - Daemon (unix socket)...
jūl 17 11:33:00 pc-158 systemd[1]: Listening on incus.socket - Incus - Daemon (unix socket).

But there are still some lingering issues as the incus client can’t interact with the Daemon:

incus list
Error: The incus daemon doesn't appear to be started (socket path: /var/lib/incus/unix.socket)

Running the same command with sudo works:

sudo incus list
+---------------+---------+------+------+-----------------+-----------+
|     NAME      |  STATE  | IPV4 | IPV6 |      TYPE       | SNAPSHOTS |
+---------------+---------+------+------+-----------------+-----------+
| gitlab        | STOPPED |      |      | VIRTUAL-MACHINE | 0         |
+---------------+---------+------+------+-----------------+-----------+
| svn-migration | STOPPED |      |      | CONTAINER       | 0         |
+---------------+---------+------+------+-----------------+-----------+

Setting up a variable in ~/.zshrc (or ~/.bashrc if you’re on BASH):
export INCUS_SOCKET="/run/incus/unix.socket"
…gets me closer to a better fix but there are now file access permission errors:

incus list
Error: You don't have the needed permissions to talk to the incus daemon (socket path: /run/incus/unix.socket)
ls -la /run/incus
"/run/incus": Permission denied (os error 13)
ls -la /run | grep incus
drwx------@    - root           17 jūl 11:33 incus

So in the end, to get it working as it should, I had to use the INCUS_SOCKET env variable:

export INCUS_SOCKET="/run/incus/unix.socket"

And chmod ‘/run/incus’:

sudo chmod 755 /run/incus

After which incus finally works normally:

ls -la /run/incus
srwx------@ - root 17 jūl 11:33 seccomp.socket
srw-rw----@ - root 17 jūl 11:33 unix.socket

incus list
+---------------+---------+------+------+-----------------+-----------+
|     NAME      |  STATE  | IPV4 | IPV6 |      TYPE       | SNAPSHOTS |
+---------------+---------+------+------+-----------------+-----------+
| gitlab        | STOPPED |      |      | VIRTUAL-MACHINE | 0         |
+---------------+---------+------+------+-----------------+-----------+
| svn-migration | STOPPED |      |      | CONTAINER       | 0         |
+---------------+---------+------+------+-----------------+-----------+

After reboot the permissions issue comes back but this error seems to have been already adressed by Incus project and should arrive with oackage updates soon: incus changes the mode of /run/incus back to 0700 · Issue #1003 · lxc/incus · GitHub

The bug is upstream in incus itself. I submitted a patch that was accepted yesterday. The daemon does a chmod 0700 on /run/incus on first start, which prevents the client from being able to see the incus socket at /run/incus/unix.socket. If you chmod 0711 /run/incus, then the client can read the directory contents and will connect automatically (without having to set an environment variable)

Here’s a link to the upstream PR that fixes it

1 Like

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.