How to deal with corrupted file objects in ostree?

General
1

Hi there, just a few questions to help me understand working with ostree-based systems a little better.

While dealing with my issues outlined here, I also investigated whether corrupted files in ostree could be the culprit (it turned out to be something else). I ran sudo ostree fsck -a, which did (and still does) yield some corrupted file objects.

Currently I get:

In commits 2275a8fe598a30dd3aea07b8cc3b7df5925c6809338ae1fdc20258a58ef37500, 9b978aa048e4371ea6b9cef368ad8ebb4416ee3f4039b829c2444071d62ab72f: fsck content object c7847cc7c9164ce704d7c17508230693bc3089bcf0ca7c6fdb3b2355ca07addf: Corrupted file object; checksum expected='c7847cc7c9164ce704d7c17508230693bc3089bcf0ca7c6fdb3b2355ca07addf' actual='bf13d8fe63612425b89f526a906f06359a1a71ac71902caef8a7160558d91187'
In commits 6eb7f49e9d4b5886d22603f4cc16e92d82675b854072fcdca8477de919cc8ed2, 30bcf537a4b3e4d82f408efcac46abeec89ae35e6166228b4b0cb61363a5de83, 2275a8fe598a30dd3aea07b8cc3b7df5925c6809338ae1fdc20258a58ef37500, a9fd52e73c899035ccfe3fe7ee077df3769c858aa70b45d04a5442c237d53499, fbd2f3e7993005b43834eba85aba4825aebe71f3816fc609978006db33db01ac, eb69eca691659fe40e8ceb048f1ed6fa8ab4fbc96614f693442161db5651e93f, 8d4d5f2c1061ee2f154caf8d0d2bcc7b01ca7a459750c9a7bb041d83300d7578, 8596e36c1e9801421fe24c4dfbbe879f07ac342bd781baa30d4cd7309964841a, f0d91f5082375d80fbf91de198429438cf07649b546a0894551f619a1f94f5b7, 9bc31623ffd150487b1435091c82c906c1fa33ba5d20365936fa202e775ccef2, b6bd88ad590477e97fae3f1019cf8e915da2eb8f279436976f3d394003e72c92: fsck content object e2b13cdbe0ef6facc26c5938f51bcb18d5fc293d65ab846d19e3ce8c63df38f5: Corrupted file object; checksum expected='e2b13cdbe0ef6facc26c5938f51bcb18d5fc293d65ab846d19e3ce8c63df38f5' actual='933b8b10b69525ffe6c613f3a30e71b2a610432b40f29df5f9c60db06900b02b'
In commits b6bd88ad590477e97fae3f1019cf8e915da2eb8f279436976f3d394003e72c92, a2bca77c7b014f0d6d79412cf3a9302dc5ba50419693ccbfea044e7c8e7c9454: fsck content object 6837237c9e5f97f5a807332eb28dbcc7dd55f8a5896bd0c3399e9c267f07d90e: Corrupted file object; checksum expected='6837237c9e5f97f5a807332eb28dbcc7dd55f8a5896bd0c3399e9c267f07d90e' actual='bf13d8fe63612425b89f526a906f06359a1a71ac71902caef8a7160558d91187'
In commits dcbb05504a77a0bd897caa8beaaea567dcefc55679fe24d51204c83f37542528, 8d4d5f2c1061ee2f154caf8d0d2bcc7b01ca7a459750c9a7bb041d83300d7578: fsck content object 3a2af5e5746347147d9cf3ba216c56734ad1b4e2ec0aa84edc21495710a5e1c0: Corrupted file object; checksum expected='3a2af5e5746347147d9cf3ba216c56734ad1b4e2ec0aa84edc21495710a5e1c0' actual='bf13d8fe63612425b89f526a906f06359a1a71ac71902caef8a7160558d91187'
error: Repository corruption encountered

Now obviously, file corruption does not sound like something desirable :sweat_smile:.
The official Silverblue documentation is very vague about this and just mentions a workaround that doesn’t seem to be applicable to OCI-based Fedora Atomic, according to this ostree GitHub issue.

So my questions are:

  • How critical is ostree corruption in general? Should I just ignore it for the time being as the system is running fine atm?
  • Is there any way to repair corrupted file objects?

Thanks in advance!

1 Like
2

Did you experience this issue more than once? I encountered the same problem a week ago and wasted many hours trying to fix it. Ended up just reinstalling the OS.

Now it’s back. The only lead I have is that it seems to be related to waking from suspend. It’s not just the OSTree files that are getting corrupted, but that’s the majority of them. I don’t even know how this is happening on a volume that’s mounted as read-only.

The system still functions normally, albeit a few corrupted files. However, the deployment is utterly @#$%ed and cannot be recovered. Can’t upgrade. Rolling back doesn’t fix the corrupted files. Will basically have to reinstall the OS again, from what I can tell…

3

Can you see if rpm-ostree cleanup -bm cleans it up?

4

Currently, I still get the same four corrupted objects when running ostree fsck. As I couldn’t really find good instructions on how to fix these files and because my system is still running fine, I didn’t attempt anything new. As far as I understand, a way to fix this issue would be to somehow re-pull the commits, but the workaround outlined here only seems to apply to plain Silverblue, not OCI-based images (as mentioned here and in this Reddit post).

5

Might be worth reporting this issue upstream, not sure on this one.

6

For what it’s worth, it seems like the upgrade to Fedora 41 has re-pulled the damaged commits, so I guess I got lucky… :grin:

Validating refs...
Validating refs in collections...
Enumerating commits...
Verifying content integrity of 243 commit objects...
fsck objects (272890/272890) [=============] 100%
object fsck of 243 commits completed successfully - no errors found.
1 Like