Hello!
I’m interested in using ucore-hci on bare metal on a home server. I’d like to use it to serve a few things like NextCloud and NFS shares, and run a few ad-hoc VMs and containers.
I’ve been using Bluefin for a while and really like it, so am keen to see how uCore works on the server. Installation with ignition is not something I’ve done before but I’m sure I can figure out.
Once installed, am I okay to manage it “stateful server as a pet” style using Cockpit and CLI, or would I be required/encouraged to use ignition and redeploy to make config changes? ( for example if I add another disk or want to juggle LVM )
Can I install with encrypted root (LUKS) and have automatic unlock without TPM, with the key being on the efi or boot partition? (I know this makes the security very weak, but it still gives me the ability to wipe the drive when I want by just erasing the LUKS keys.)
Anything else I should know before using ucore-hci on bare metal..? Would people recommend I use something like Proxmox on bare metal instead and put ucore only in VMs?
I have spent last few weeks with tinkering around CoreOS, uCore and Fedora IoT on local VPSs and bare metal servers and my conclusion is that coreos-installer is not ready and by my impression it’s not even under active development. Most of the commits from the recent months are updates of dependencies.
I have multiple Debian and Fedora servers on combination of mdadm+luks+lvm and I wanted the same with uCore/CoreOS. After many tries I found out that coreos-installer is buggy and it can’t even install bootable system on bare mdadm RAID because EFI partition is empty at the end of the process. I encountered other bugs on the way. One of the problems is that butane lets you generate ignition file that fails so debugging process is painfully time consuming.
Other issues:
Fedora IoT doesn’t have mdadm in its initramfs. Installer lets you install the system on mdadm RAID but the system won’t boot up.
CoreOS has LVM and mdadm in its initramfs but ignition file doesn’t support LVM. (+ the mdadm bug makes it unusable)
Whenever I build my own image based on uCore or CoreOS /boot became unavailable.
Documentation is missing important parts or I can’t figure out why if I define /, /boot and EFI partitions in ignition file nothing is copied to any of them and butane does say everything is ok.
I tried installation via bootc install commands but this way lacks documentation or at least I couldn’t find it so I gave up on a few error messages.
I didn’t have strength for testing LUKS.
My impression from all of this is that CoreOS is designed and developed for cloud as thin layer between virtualisation platform and containers that requires zero maintenance and proper bare metal setup is not their priority. Because uCore depends on its installation process it hits it too regardless how good it is.
I rushed in without properly reading the docs, so it took me longer than it should have to grok the installation process. For anyone else reading the basic process is two steps:
Use coreos-installer in a live environment to blat the basic image and your config to the target disk. At this point a fixed set of partitions are written with a fixed layout and completely ignoring your config. Your config is stuffed somewhere into the blatted image.
On first boot the actual setup happens. This is the point at which your config is read and your requested partitions, users, etc are set up. Depending on what partitioning you configued the system will repartition the disk or reformat from under itself (by first copying data into RAM).
I think it’s important to get this understanding, as when you see config options for wiping filesystems or partition tables you need to remember that this is from the perspective of the first boot. A mistake I made was to set those options to request that partitions be wiped (I thought “This is a clean install so of course I want the filesystems to be wiped!”), then see installation fail as on first boot it was erasing itself.
The “step 1” above will always wipe the installation disk, and so the options for wiping partitions and filesystems really should only be used for your other disks. (Maybe there’s another usecase for them that I’m not aware of.)
Like you, I did find it jarring that I couldn’t manage disks and filesystems in the way that I’m used to. I miss not having LVM and the lack of support for installing to btrfs subvolumes.
I’ve been busy and so far I’ve just been toying with uCore-hci in a VM. When I find the time I do still intend to put it bare metal onto my home server. I’ll be going back to using fixed partitions (efi, boot, root, var) and will not have root and var on the same filesystem under different subvolumes.
I love the Bluefin experience on my laptop, and if uCore can be similar on my home server then I’m happy to lean in to these restrictions.
If you embed your ignition file to the iso of the coreos-installer you skip the intermediate basic system installed on one of the target drives and it will install the final setup at the first boot. It’s also better for debugging.
Do you mean I skip the step of booting into the CoreOS live environment and manually running coreos-installer, I somehow mash my ignition file into the ISO and put that directly onto my target disk? Then on first boot it does the rest?
As cool as that sounds I unfortunately think that I need that extra intermediate step. After the first phase of install and before first boot is when I’m choosing to resize the boot partition.
That is something which ignition cannot do, but I feel is necessary as the default boot partition is teensy (384MiB) and in my tests was already 40% full with just a single deployment. That’s not enough headroom for future growth, or having a pinned image.
Doing it before reboot seems like the best time to manually resize it, as root and other partitions haven’t consumed the rest of the disk yet.
I too am beginning to switch a bare metal home server over to ucore-hci. Have you made progress in getting a custom sized /boot partition working? If so, would you be willing to post the relevant snippet from your butane file?
I’d like to crib from your work ;-).
Also ideally I’d be able to use btrfs for the root FS but I’ll try that afterwards.
Now I’ve got to see if it is possible to use btrfs for the root partition (so I can put /var /sysroot /etc inside of one btrfs partition). I’m not yet sure if this is a good idea though ;-).
From what I’ve determined you can use btrfs for your filesystem(s), but you cannot make use of subvolumes. So for example ignition will let you setup one btrfs filesystem for each of root, var, and home; but will not let you put all three onto subvolumes of a single btrfs filesystem.
You might be able to use subvolumes if you manually arrange things after installation. YMMV.
What I’ve done is just accept this way of working. I have a root btrfs which is a few 10s of GB, then the rest of the disk is a bigger btrfs filesystem just for var.
