Important announcement regarding system updates [Action needed]

General
1

THIS IS NO LONGER AN ISSUE, THIS THREAD IS BEING KEPT OPEN FOR INFORMATIONAL PURPOSES

Only necessary if you have an old image from before July 2024

Original text follows:

If you use Bazzite, Bluefin, Aurora, uCore, or any other Universal Blue image (including our toolboxes) then you need to follow the instructions in this announcement in order to ensure that your device is getting updates. We were rotating our cosign keypairs this morning, which is the method that we use to sign our images.

During this process I made a critical error which has resulted in forcing you to take manual steps to migrate to our newly signed images.

  • All existing Universal Blue images BEFORE 2024-07-02 will need to issue the commands below in order to receive future updates, your device will reject future updates unless this action is undertaken.

  • If you are a new user and have installed from an ISO AFTER 2024-07-03 then you do not need to take any action, this only affects existing installations before that date.

  • This incident does not mean that there was a security breach, quite the opposite, in fact. It means the protections and checks we’ve built into the operating system are working and they’ll refuse to accept an update signed by an unknown key.

  • The installation on your device is fine, upgrades just won’t work. But you do need to follow the instructions below in order to get updates.

  • All the Universal Blue images and ISOs have been updated to the new key, we strongly recommend replacing any downloaded ISOs with the new ones to avoid having to do this on new installations.

  • We are working on signing some older images so that you can still have rollback (especially for you Bazzite users on AMD Polaris GPUs) and will post more information as soon as we can.

I deeply apologize for this, I take full personal responsibility as the error was completely mine, from both a technical and process standpoint. I know this shakes the amount of trust we’ve built up over the past three years, so there’s no easy way to say it other than by being transparent about the mistakes.

Instructions

We have a script that will resolve the issue. You are strongly recommended to review the script first prior to running it by inspecting it here. You can either perform those steps by hand or use this following command:

curl -sL https://fix.universal-blue.org/ | sudo bash

Process changes

Process changes enacted to ensure this doesn’t happen again, please reference the following github issue: create update process/solution for cosign key rotation · Issue #600 · ublue-os/main · GitHub

Quick video

Huge thanks to p5, m2, bsherman, antheas, hikariknight, bketelsen, and eyecantcu for mitigating my mistake as best they could. Thank you for your patience and support!

Update 2024-07-01

This is what the error looks like if you try to run an rpm-ostree update command on an affected install:

rpm-ostree update
note: automatic updates (stage) are enabled
Pulling manifest: ostree-image-signed:docker://ghcr.io/ublue-os/bluefin-dx-nvidia:stable
error: Creating importer: Failed to invoke skopeo proxy method OpenImage: remote error: cryptographic signature verification failed: invalid signature when validating ASN.1 encoded signature
75 Likes
2

Thanks for the quick fix!

10 Likes
4

Shit happens.

Thank you for the transparency, the video, and the easy fix.

16 Likes
5

It wouldn’t be truly “cloud native” without certificate rotation screwups! Thanks for fixing this so quickly, I was sure I’d broken something myself.

18 Likes
6

Don’t stress about it too much, you made a mistake and had a resolution in under 24 hours. Nice work!

8 Likes
7

Amazing transparency and feedback. Thanks. I doubt that I would get a honest apology and quick fix from a “distro” the way you did (yeah I am looking at you manjaro).

Thanks Jorge. Shit happens. Get back on the horse. :slightly_smiling_face:

6 Likes
8

I know the feeling :smiley:
Thanks for being transparent and the quick solution.

3 Likes
9

Thank you for your honesty. It’s quite rare these days, and it truly makes me appreciate Bazzite even more :wink:

4 Likes
10

I’m laughing so hard right now, thank you for this.

9 Likes
11

Thank you for your transparency, stuff like this can happen. I’ve notified my friends and family who use Ublue images and they really don’t mind, they’re not going back to Windows just because of a minor issue.

Though maybe there’s a way to reach out to people who don’t check these posts?

4 Likes
12

Thanks for the fix and the explanation, mistakes happen and I’ll get all the machines in the house rocking again.

2 Likes
13

Update script worked perfectly on my Aurora system. No problem at all.

Thank you for creating ublue and staying on top of things! It’s been a great ride.

3 Likes
14

Very glad to see such measures taken after a mistake, it shows again a will to do things the right way :smile:

3 Likes
15

Love the way it is communicated - cheers!

2 Likes
16

The way this is communicated and quickly fixed does not shake my trust in the project in any way. In fact it is encouraging to continue using it. In my case that is currently Bazzite and possibly in the near future Aurora replacing Fedora proper.

2 Likes
17

@j0rge how does a normal Key Rotation work, when everything works out correctly?

And Do you have to update every key Rotation? I have a laptop I barely use but love the idea of the atomic update and being up-to-date.

The script worked perfectly on two Aurora Installs.

1 Like
18

Back up and running with no issues thanks to the script. And hey, accidents happen, don’t sweat over it.

2 Likes
19

Great communication and quick fix. Don’t sweat it and thank you.

1 Like
20

In the future, when we need to do a key rotation, the process should look something like this:

  1. Generate a new keypair, replace the public key in our config with the new public key
  2. Sign our images with both the old private key and the new private key to ensure users have a path forward. Users that have updated shouldn’t be affected by this as they’ll be using the new keypair for validation
  3. Resign a set of older images with the new private key to ensure rollback works for people that updated
  4. Stop signing images with the old private key after a determined period of time inline with how frequently users update
  5. Provide a path forward for any stragglers. I.E., document the last build we signed with the old private key or implement a hidden just script that pulls down the new public key so that the current image can be updated to

This should be seamless for most people and shouldn’t require any manual intervention unless a nonstandard update procedure is used, such as updating once within a period of several months. It’s not perfect, but the process will be significantly better

8 Likes