Recovering a non-booting system

A topgrade update somehow borked my grub. It boots into the grub command line without showing any boot optins.
Attempting to load the boot .conf files from /loader/entries with the configfile unsuccessful: title, version, options unrecognized by grub as valid commands.
Attempts to mount the encrypted root manually with cryptmount also unsuccessful. It returns invalid passphrase even though it is correct.
Is there any way to somehow chroot into the filesystem and update grub? I would have to mount the root directory from initrd as root on disk is mostly empty, right?

Otherwise I could copy the home and var directories to anotger disk and reinstall the whole system?

First a disclaimer: I am not an expert in this stuff and go out of my way trying to avoid messing with it on a personal basis. But I can imagine how stressed you must be over an unbootable system and can try to brainstorm with you.

I think you’ve got a few different issues going on here. For one, it sounds like your grub.cfg is possibly erroneous and that you believe you need to mount the encrypted root to get to it. Do you not have a /boot/efi partition? Is /boot a dedicated partition? Is it encrypted?

I think having a bad grub.cfg in /boot/efi/EFI/fedora/ (never encrypted) or wherever could cause you to lose menus as you describe. And I also think it might be possible that without that config file telling grub to launch in secure boot mode that you could have problems mounting your encrypted root. But, again, I’m not an expert and I could be wildly off-base.

Is there any way to somehow chroot into the filesystem and update grub? I would have to mount the root directory from initrd as root on disk is mostly empty, right?

If you’ve got a grub command line and an unencrypted /boot/efi you probably have what you need without mounting root just yet. Worst case, you could boot a recovery media or even your install media (where you ctrl+alt+f3(?) away from the setup GUI to a virtual console). For sure, I’d expect any of the mainstream liveCD options to allow you to successfully decrypt your LUKFS.

I could copy the home and var directories to anotger disk and reinstall the whole system?

Directories or partitions? These aren’t also encrypted?

tl/dr: it sounds like the easiest and most likely general-purpose fix is to boot from a liveCD or recoveryCD, mount all your partitions (including encrypted ones), then sudo grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg (assuming some Fedora spin and UEFI w/ grub2).

gl

Thanks for the reply.
I ended up reinstalling the OS, but thankfully was able to keep my home directory and most of my config except for a few Gnome settings. Regenerating the grub config in a chroot environment was more complicated than expected with the Fedora Atomic way of mounting the filesystem. I was stuck with an osprober directory not found error and decided to reinstall.

@j0rge, in the installer one can choose custom partitioning through Blivet-gui; the issue I faced is that the installer insists on creating a new filesystem for the /var mounting point and refuses to use the old btrfs subvolume. Would you be able to explain why that is or if the behavior could somehow be changed in order to maintain flatpaks installed to /var/lib/flatpak during a reinstall? This could be a Fedora decision and I might be asking in the wrong forum.

Yes there are limitations there, it’s one of the reasons we don’t support custom partitioning. (aka there be dragons), see also:

I must’ve misinterpreted what was going on, because it seems like being unable to mount your encrypted partition would preclude a chroot into it. To my mind, having grub fail before it can even read its own config means the problem isn’t atomic-specific.

I was stuck with an osprober directory not found

Did you determine that it was even necessary? I have no idea what kind of crazy rampaging your topgrade scripts were doing to your system, but I have a hard time imagining that it involved replacing your grub config with one that wasn’t setup to boot any OS at all.

Glad you evidently got sorted, though.

Can you elaborate on the limitations, please?

They are in the links in my previous post.

Am I missing something, then, or is it just a single upstream bug that has existed for years and has a relatively simple workaround?

Maybe I’m barking up the wrong tree, but it feels political that people are told they can’t dual boot or manually partition. Because it just feels so antithetical to the spirit of the thing that draws us together to say “you can’t do that thing that’s been commonplace forever” instead of saying “you can do that, you just need to do it this way because of this reason.”