Locally built, automatically updating custom bootc image

So, rpm-ostree install is really slow, and it also can’t change the versions of any packages in the base system. Custom images forked from GitHub - ublue-os/image-template: Build your own custom Universal Blue Image! is the right way to do things, but that involves having GitHub set up, and dealing with the hated GitHub Actions.

However, you can just build a bootc image locally, and then switch to it. This is one of the brilliant properties we get when we remember it’s all just containers, it’s only tarballs with layers, caches and configuration.

Begin by putting your Containerfile somewhere, perhaps /etc/system-image/Containerfile. Derive from a base image like ghcr.io/ublue-os/bluefin:stable, or ghcr.io/ublue-os/bazzite:stable, and remember to run bootc container lint at the end to catch things. An example, adding Visual Studio Code to Bazzite:

FROM ghcr.io/ublue-os/bazzite:stable

RUN --mount=type=cache,destination=/var/cache \
    --mount=type=cache,destination=/var/lib/dnf \
    --mount=type=tmpfs,destination=/var/log \
    dnf5 config-manager addrepo --set=baseurl="https://packages.microsoft.com/yumrepos/vscode" --id="vscode" && \
    dnf5 config-manager setopt vscode.enabled=0 && \
    dnf5 config-manager setopt vscode.gpgcheck=0 && \
    dnf5 -y --setopt=vscode.enabled=1 install code

RUN bootc container lint

Podman, all of our favourite container manager, has shipped for a while with a systemd unit generator that turns files that look like systemd units for running containers into real systemd units, best known as Quadlet (a squished Kubelet). If you’ve used Quadlet before you’ve definitely used its .container and .volume unit types, maybe .pod for multiple containers in one context. However, it also supports .image for pulling images into your system or user image storage, and for our purposes the .build type for running a container image build as a systemd service. Write something like this to somewhere like /etc/containers/systemd/system.build:

[Build]
Arch=amd64
ImageTag=localhost/system-image
Pull=newer
SetWorkingDirectory=/etc/system-image
PodmanArgs=--squash

[Service]
ExecStartPost=/usr/bin/bootc switch --quiet --transport=containers-storage localhost/system-image:latest
ExecStartPost=/usr/bin/bootc update --quiet
Nice=0

When you run systemctl daemon-reload, the generator will create a unit called system-build.service that will run podman build as a systemd service. When the image is finished, it will ensure that bootc switches to the localhost/system-image image that is inside root’s container image storage, the one used by Podman, by way of --transport=containers-storage, and then it will immediately update so the newly built image will be rebooted into.

We can now automate this build process by defining a daily timer to run system-build.service. Remember that Quadlet names things other than .containers by appending a hyphen and its type before the .service part, so system.build becomes system-build.service. Timers need to match the name of the service they are going to trigger. In our example system-build.timer would look something like this:

[Timer]
OnCalendar=daily

[Install]
WantedBy=timers.target

Do a systemctl reload, then systemctl enable system-build.timer, and now you have an automatically rebuilding and updating custom image that runs locally and involves no pfaffing with frustrating CI systems.

Finally, on the ublue family, uupd will still be running in the background and trying to update like usual, so drop this in /etc/uupd/config.json. Automatic updates are good and important. Don’t do this unless you’ve verified everything you’ve done from this post up to this point.

{
  "modules": {
    "system": {
      "disable": true
    }
  }
}

Thanks for posting what you did. I build test ones locally myself as well. It is quicker to build locally than on the GitHub runners, which can be slow.

I don’t know why GitHub Actions are hated. Mine work fine. I have both builds using bluebuild (which take way longer to run but are easier to manage) and the traditional method using the uBlue image template.

The reason I don’t like GitHub Actions is because the development loop is terrible. It doesn’t run locally, and the control plane is completely within GitHub itself even if you can self host a runner. To get things working, you have edit the workflow, push the branch, wait for the jobs to work or error out, read the logs in the browser, repeat. It takes so long and there is no debugging available, it’s a closed system. Even if you are using workflows written by someone else, you still have to deal with this when you’re working on the thing that the workflows actually use.

Turns out that when you tag a new image, Podman leaves the old image dangling instead of cleaning it up. Over time this means you have left-over images which, given the size of bootc images, accrues a lot of disk usage over time.

My fix for this was to add an additional post-run command to the build unit that prunes unused bootc images after apply, identified by the label containers.bootc=1.

[Service]
ExecStartPost=/usr/bin/podman image prune --force --filter=label=containers.bootc=1

I had 100GB of images lol

Why don’t you use self-hosted GitHub runners?

Personally, I haven’t tried that in my iterative troubleshooting loop when fixing repos I’ve forked, but I’ve been meaning to.

Edit: I’d imagine the FIRST build would take a bit longer since it would need to pull all the containers a build would use, but after they are stored in local container store, as long as the digest didn’t change it would re-use them (it looks like).

Where the job runs is not the problem with GHA. The tedious development loop, the pile of YAML, and the poor documentation and debuggability, are my problems

Well, GitHub runners are far slower than my local rig, so there is some time saved waiting for it to finish.

Have you tried another local Git server system like Gitea? (I have no idea if that’s any better).