Double-extra s*rewed: Can't login with GUI or TTY, can't rebase, can't change shell

I am trying to implement a fix suggested by @zeroballoons for the black screens I have been experiencing in ublue 39 and 40. Apparently there is a bug in SDM that creates black screens with the fish shell – which I certainly use.

Well, the fix involved changing back to bash, which is ok for me for a while (though weird). But I must have done something wrong and now I can’t log in at all in any way. Here are the details.

After a reboot, I see the normal GUI login window. I enter my credentials, and after a blank screen for a second or two, it brings the login screen back as if I never entered anything. Yikes. CTRL-ALT-F2 brings up the TTY. I enter my credentials, and it also resets and shows me the login prompt again. Yikes.

I try logging from my desktop, and I get the following ominous error:

fork/exec /usr/bin//bin/bash: no such file or directory

I’m not exactly certain what that means, but perhaps I shouldn’t have entered `ujust configure-shell /bin/bash’.

But I don’t know how to recover from this and change the shell again. Sometimes when I reboot I see a menu of pinned images to start from, but not now, it goes straight to the login screen.

Any ideas?

Hey,

Sorry, I made a typo. It’s just ujust configure-shell

I have corrected it in my post.

No worries. But kind of bad when making one mistake in changing shells renders a system unusable…

Did you fix it after I corrected the command?

No. I can’t login even to a TTY now.

You might need to enter rescue or single user mode to fix this. On the grub menu press ‘e’ when your default entry is selected. That will open up a page where you can edit the boot paramters. You want to add single init=/bin/bash to the command line (the line starting with linux) and boot.

This should boot you into a root shell (no graphical UI), from which you can use chsh to edit your user’s shell. Make sure you change your user’s shell and not root’s. Once you’ve made the fix type /sbin/reboot to reboot and boot normally again (*).

I think this ought to work, but I’m not 100% certain how ostree systems manage their /etc, and if the correct /etc will be present in this single user rescue mode.


(*) - I just tested, and neither reboot or poweroff work for me in single user mode, I guess because dbus is off. You may just have to do sync, then exit, then hard reset with the power switch.

1 Like

Thanks for the suggestion. I was able to edit the boot parameters and changed the user shell to /bin/bash. As you expected, I could not reboot, so I had to hard reset.

But now it’s even worse (!). Before, I could see a graphical login screen. Now, all I see is the LUKS unlock gadget, and then it goes directly to a black screen (insert Fred Armisen jail meme here). CTRL-ALT-Fx do nothing. Can’t SSH into it.

Hilariously I ended up in a similar boat about a month ago when I first started using linux. Tried to switch to zsh, switch went bad for some reason. Couldn’t login exactly as you described.

I resolved it by (on a separate machine) downloading fedora 40 workstation iso, using fedora media writer to imprint it on a USB, then using it as a live distro to boot into. I mounted my main drive (having to do some voodoo that I don’t quite recall to manually decrypt the luks encryption), finding the current image, stepping into it, and then directly editing, I believe, /etc/passwd to reset my shell back to /bin/sh. Then just rebooted into my primary system and was good to go again.

That’s the rough outline, anyway. Sorry the specifics are lost to time. Maybe someone more tooled up can provide a more detailed steps, as I really just sort of muddled around until I found it.

If you are able to get through luks we can fix this without going to a live iso.

A live iso can be helpful if you want a gui.

Boot system. Add to grub command (press e to edit) and at end of Linux line, add init=/bin/bash. Press Ctrl+x to then boot.

Your system should boot and drop you to a root shell. You now need to make sure SELinux doesn’t bite you.

mount -t selinuxfs selinuxfs /sys/fs/selinux
/sbin/load_policy

You will need to edit the file /etc/passwd. Change the incorrect shell item to /bin/bash. Ensure you only correct that single entry. There is no chsh to do this automatically.

Run sync and then /sbin/reboot

If you can’t do that. You can do very similar using a live iso. You will mount disk, and you can chroot into the disk.

Wow. Ok.

Yes I have LUKS encryption. How does that change things?

You woud have to decrypt your install with cryptsetup If using a live ISO.

Hi chicagonyc,

I don’t know if it’s still helpful, but the fork/exec /usr/bin//bin/bash error was because the shell was changed to /usr/bin//bin/bash instead of /usr/bin/bash or /bin/bash.

Another cause of inability to log in on Linux that I’ve had often is messing up a shell .profile or any script sourced by the shell, could also be /etc/profile or in /etc/profile.d/

The process is actually not that bad to recover from a LiveUSB. I recommend removing all drives except for the one containing ublue, and boot from a Fedora Workstation 40 LiveUSB. You should have a drive with 3 partitions, if it’s an NVMe SSD it might be something like /dev/nvmen0p1 through /dev/nvmen0p3. Assumming you removed the other drives, as suggested, you can unlock the luks partition like this (If you’re using NVMe):
sudo cryptsetup luksOpen /dev/nvmen0p3 fedora
or sudo cryptsetup luksOpen /dev/sda3 fedora # If using a SATA drive.

You will have to enter your encryption password, then it will create a “decrypted” version of the drive you can access at /dev/mapper/fedora. This will function exactly like a partition that isn’t encrypted, but remember that partitions 1 and 2 are still at /dev/nvme0nX or /dev/sdaX, because they are the EFI partition and the /boot partition respectively, which “can’t” be encrypted (Technically untrue, but reasonably true for most use-cases).

You should only need to mount the decrypted drive though:
sudo mount /dev/mapper/fedora /mnt

Then please run grep 1000 /mnt/etc/passwd and post the result here, or refer to this document for the format it should take: https://linux.die.net/man/5/passwd

Basically the format should be USER:x:1000:1000::/var/home/USER:/bin/bash - Where you replace both instances of USER with your username. You can edit the file with sudo nano /mnt/etc/passwd if it doesn’t match that format, but only edit the like with your username and 1000:1000 in it!

Then you can run: sudo umount /mnt && sudo cryptsetup close fedora

Then reboot systemctl reboot and remove the LiveUSB. Hopefully that will fix your issue, please let me know if not.

Good luck!

  • Dan

EDIT 1: Fixed ordering of instructions.

Sorry for the late reply. Had to finish up some work before coming back to this.

I successfully decrypted the partition as per your suggestion (I think), but I can’t seem to find /mnt/etc/passwd. All I see in /mnt is home, root, var. In root I see boot, dev, home, ostree, proc, root, run, sys, tmp, and var.

BTW this is a BTRFS partition, with a subvolumes for var, home, and root.

In summary, I have executed the following commands:

  • sudo cryptsetup luksOpen /dev/nvme0n1p3 fedora
  • sudo mount -o subvol=root /dev/mapper/fedora /mnt
  • ls /mnt
1 Like

Hi @chiagonyc,

EDIT 4: Ok, I think I’ve figured out the problem now. Please use this method!

Sorry for the mistake in my suggestion. It looks like /etc is part of the ostree (The way Fedora Atomic/Ublue makes its updates “immutable”/atomic). This is normally good, but may make our job harder… Please proceed with these instructions at your own risk!

You may be able to edit /mnt/ostree/boot.?/fedora/*/0/etc/passwd. Only edit one line, or append one line. Best to run this first:

grep ':1000:' /mnt/ostree/boot.?/default/*/0/etc/passwd

and look for lines that don’t match the format I suggest below. This may be multiple files. You should probably only edit only the first one, then see whichever “snapshot” is able to boot (From grub, you’ll probably have to try one at a time). I included the * because that directory is a “random” long hexidecimal hash, so better to not try to type it. I also don’t know how to correlate which hash belongs to which “snapshot”.

First, even though the system is broken, you should back up the file:

ls /mnt/ostree/boot.?/default/*/0/etc/passwd | while read file; do cp "${file}" "${file}.bak"; done

Then the backups will be next to the originals with a .bak suffix.

Also, in the .../etc/passwd file, I mentioned you want a line in the format: USER:x:1000:1000::/var/home/USER:/bin/bash (Don’t edit the other lines!)

nano /mnt/ostree/boot.?/default/*/0/etc/passwd

(You can replace nano with another terminal-based text-editor if you prefer.) Note, each time you save and exit, it will bring you to a new file to edit, probably a total of 5 to edit, looking at the outputs you sent.

Where a better format would be:

USER:x:1000:1000:FULL_NAME:/var/home/USER:/bin/bash

where USER is your Linux username (Hopefully, you know this already! It should also exist as a directory in /mnt/var/home/ often the first word of your full name, all lowercase), and FULL_NAME is your full name, e.g. the full name you want to appear on the lock screen. Again, don’t edit the other lines! If the line doesn’t exist, you’ll have to append it, but for the moment, I’ll assume it does, unless you tell me otherwise! Please be careful here… Make sure this line has exactly 6 of the character : … Also, please make sure you’re only editing the end of the line, if everything else looks correct already!!! (i.e. just the /bin/bash instead of /bin/zsh or /bin/whatnot )

I wouldn’t suggest this in general, as it might mess up your system, depending on how ostree works, but if your system seems completely shot, maybe it’s worth a try… Can someone with more experience than me let us know if this is dangerous to the way ostree works?

Thanks, Good luck!

  • Dan

EDIT 1: Replaced /boot.1 with /boot.? which is more generic.

EDIT 2: I just tried this on an extra snapshot, and it appears to not mess up ostree, but still Your Results May Vary! Please proceed at your own risk! Also please type commands exactly as they appear! I’ve also tried to clean this up somewhat.

EDIT 3: More clarification and command to edit all the files.

EDIT 4: Noticed problem with my assumptions. should be /mnt/ostree/boot.?/default/*/...

Thanks for the researched help. I’m having difficulty locating /etc/passwd in /mnt. It’s not in /mnt/ostree/boot.1/default. When I search for etc/passwd, I get the following locations:

./ostree/deploy/default/deploy/2d04588d7af8c23fe3124cc308d2e01fed745dce883376b855f047c1f60ab491.0/etc/passwd
./ostree/deploy/default/deploy/3e5c39f82cfd6cd7c96c683bbe4aeb004c1641f62bf4bb6bae1eace673684f33.0/etc/passwd
./ostree/deploy/default/deploy/2d04588d7af8c23fe3124cc308d2e01fed745dce883376b855f047c1f60ab491.0/usr/etc/passwd
./ostree/deploy/default/deploy/9c0eab598171385a359ae8640fd054760d3eaba8d0a2b0a479f5a7ea0805a28c.0/etc/passwd
./ostree/deploy/default/deploy/a86b0df3db3b39c1c346f23365864f4daee4e09a0000c60cbf8270952c9073e6.0/etc/passwd
./ostree/deploy/default/deploy/3e5c39f82cfd6cd7c96c683bbe4aeb004c1641f62bf4bb6bae1eace673684f33.0/usr/etc/passwd
./ostree/deploy/default/deploy/9c0eab598171385a359ae8640fd054760d3eaba8d0a2b0a479f5a7ea0805a28c.0/usr/etc/passwd
./ostree/deploy/default/deploy/0d19cb698bf6e93b5bf61ee98231d44a5d88d6b3a2cdc503ff07812b148d1682.0/etc/passwd
./ostree/repo/extensions/rpmostree/private/commit/usr/etc/passwd
./ostree/deploy/default/deploy/0d19cb698bf6e93b5bf61ee98231d44a5d88d6b3a2cdc503ff07812b148d1682.0/usr/etc/passwd
./ostree/deploy/default/deploy/a86b0df3db3b39c1c346f23365864f4daee4e09a0000c60cbf8270952c9073e6.0/usr/etc/passwd

In /mnt/ostree/boot.? I see:

liveuser@localhost-live:~$ ls /mnt/ostree
boot.1  boot.1.1  deploy  repo
liveuser@localhost-live:~$ ls /mnt/ostree/boot.1
default
liveuser@localhost-live:~$ ls /mnt/ostree/boot.1/default
37e32f910c383efc36403fc1b42e89073fb9c19aca7a3924b6d3a45bfc44565b  e6459322a672ca7d0d9bd17b3cd89b4044594fd23c4c7c5242ebd7d2f96fa4f6
3a73017ed4366ea1a02249557f7b990b99dac32e2d4daca4fe230a1d39f675bf  f5b39002afba3cb62c1b25ade952f3df9251f078ba779aa3776ebc718f924f0e
3a8c479a20b1355907f8323f0192eef210e7dc91d31c77003004e3513f15e309
liveuser@localhost-live:~$ ls /mnt/ostree/boot.1.1
default
liveuser@localhost-live:~$ ls /mnt/ostree/boot.1.1/default
37e32f910c383efc36403fc1b42e89073fb9c19aca7a3924b6d3a45bfc44565b  e6459322a672ca7d0d9bd17b3cd89b4044594fd23c4c7c5242ebd7d2f96fa4f6
3a73017ed4366ea1a02249557f7b990b99dac32e2d4daca4fe230a1d39f675bf  f5b39002afba3cb62c1b25ade952f3df9251f078ba779aa3776ebc718f924f0e
3a8c479a20b1355907f8323f0192eef210e7dc91d31c77003004e3513f15e309
1 Like

Hello,

NOTE! - I assumed the above was incorrect, but it was on the right track. Please see my previous post, which I’ve updated again!

NOTE/EDIT: Before you try this, please re-read my last post, and make sure to pay attention to the /*/ wildcard matcher… If that doesn’t work, it means somethings different between Silverblue and Ublue that I wasn’t aware of, if it does, hopefully you’ll be all set!

Sorry, I was adapting those instructions from Fedora Silverblue, so I guess I didn’t realize how differently things were set up for Ublue.

Basically, each of those long directories in the /mnt/ostree/deploy/default/deploy/ directory should all be “snapshots” bootable from grub. Can you access the grub boot menu?

For example, that /ostree/deploy/default/deploy/0d19cb698bf6e93b5bf61ee98231d44a5d88d6b3a2cdc503ff07812b148d1682.0/ should be thought of as the root of that snapshot with the ID: 0d19cb..... I don’t know how to correlate between snapshot IDs and actual grub bootable snapshots, but if you edit each ./ostree/deploy/default/deploy/*/etc/passwd files to make sure it has the proper format (for your user only, Don’t Modify the Other Lines!), and /bin/bash as the last entry in your user line (it will generally have a :1000: in the line somewhere, and should have your username as well).

If your grub boot menu is hidden, I’d modify all of them. You can do this:

This should be copied and pasted! Sorry for the crazy command list!~

sudo -i /bin/bash

read -p "Please enter the name of the editor to use: " EDITOR; export EDITOR; [ -z "$(which $EDITOR)" ] && echo "Editor: $EDITOR not found! Please try again!" || \ ls /mnt/ostree/deploy/default/deploy/*/etc/passwd | while read file; do [ ! -f "${file}.bak" ] && cp "${file}" "${file}.bak"; $EDITOR "${file}"; done && echo "DONE EDITING FILES"

This will backup each file and let you edit each one, one at a time. When it prompts for an editor to use, type whatever terminal based editor the live environment has, and hit enter (e.g. nano, vi, etc as long as it’s on the Live environment you’re using). It’ll keep putting you into that editor for each of those files, until you’ve exited out of the editor for all of them, then it will print DONE EDITING FILES when you quit out of the last file. If you can’t write to those files, please let me know! I’ll look into this further!

Then please run grep :1000: /mnt/ostree/deploy/default/deploy/*/etc/passwd | grep ':/bin/bash$' | wc -l and make sure the number returned is 5 (I believe it should be 5 in your case). If it’s less, please run the above command again, and double-check each file in the list!

If you need a graphical editor, let me know and I can try to change it to work for that as well, I’ll just need to know if the live environment is Xorg or Wayland as well.

I would like to make sure this works correctly on UBlue in a VM for you though, which version of UBlue are/were you on?

Thanks, Good luck,
Dan

EDIT 1: Maybe I misread your problem? Please re-read my last post and note the ? and the /*/ in the filename, that is a wild-card matcher, so it should match all the deployments.

EDIT 2: Realized this was probably wrong. So I’ve striked out the stuff that wasn’t correct. Please refer to my previous post instead (Which I’ve updated!)!

I think … you did it!

I’m now typing from a running Aurora 39 system. Absolutely fantastic! You spent a lot of time on this, and I appreciate it very much.

That being said, changing the shell to fish shouldn’t send the system into a death spiral?!

1 Like

And I updated the system to Aurora 40 which now works, and with Wayland, to boot.

@jedi453 is a credit to this forum!

I will never switch the default shell away from Bash using ujust or chsh. I did discover how to run a custom command in Ptyxis, so I access fish that way.

1 Like

Hi chicagonyc,

Thank you so much! I’m just glad I could help! I’m really happy it worked out!

Good luck! Enjoy Aurora!

  • Dan

P.S. regular backups are your friend! PikaBackup on the Flathub is great for stuff in your home directory, but I also really like Restic which has to be installed via brew or rpm-ostree, but is an amazing command-line tool for full backups if interested!

Hi @chicagonyc ,

I agree that shouldn’t have happened, but there are cases where certain window managers, desktop environments, etc. rely on scripts that can’t be run in certain shells (Which they consider obscure).

Still, you can run fish whenever you want! Just typefish in the terminal to switch or put it in the #! line for scripts! Brew might be the better way to go here though!

Good luck! Thanks for the appreciation!

  • Dan