Root disk is full

Hi all,
This morning immediately after booting I got a warning saying the root disk is full. I tried to find the culprit and saw this:
in folder /sysroot/ostree I see 4 folders. I check the content size in each of them:

boot.o not readable cause of being owned by root, also sudo doesn't help here
boot.0.1 16 KB
deploy 35.1 GB
repo 18.2 GB

in /sysroot/ostree/deploy/default I see 3 folders

backing is locked
deploy 17.2 GB
var 17.9 GB

The partition is 40GB and was using a lot less until now. I don’t have real numbers here, but in Dolpin it shows the partition with a bar next to it and it was maybe 1/2 - 2/3 full:

I use Aurora-dx-nvidia, have no layered packages, no containers, just a bunch of flatpacks installed.
Are the numbers I wrote here normal for a system like this or did I find what is causing the 0% free disk space?

Do you have pinned deployments?

No, I only have 2 deployments, the one I use and the backup

This is what I see when using rpm-ostree status:

rpm-ostree status
State: idle
AutomaticUpdates: stage; rpm-ostreed-automatic.timer: no runs since boot
Deployments:
● ostree-image-signed:docker://ghcr.io/ublue-os/aurora-dx-nvidia:stable
                   Digest: sha256:388277fa6e3a38b6b271c9ee8f0e6de553cc0e178a27e6ff48c2c93dfac82578
                  Version: 41.20250209.1 (2025-02-09T06:00:42Z)

  ostree-image-signed:docker://ghcr.io/ublue-os/aurora-dx-nvidia:stable
                   Digest: sha256:3bc10febe593166ec3f2c849a1e07b158a80c4646a468866e3c8ea1c48c85dfe
                  Version: 41.20250126.1 (2025-01-26T06:02:06Z)

I am now using the 09 February deployment which was installed yesterday apparently. I had no problem yesterday.

These are the changelogs from the last update. No idea if this will help.

$ ujust changelogs

  This is an automatically generated changelog for release  stable-20250209.1 .   
                                                                                  
  From previous  stable  version  stable-20250126.1  there have been the following
  changes. One package per new version shown.                                     
                                                                                  
  ### Major packages                                                              
                                                                                  
  Name                           β”‚Version                                         
  ───────────────────────────────┼────────────────────────────────────────────────
  Kernel                         β”‚6.11.8-300 ➑ 6.12.9-200                        
  KDE                            β”‚6.2.5-1                                         
  Mesa                           β”‚24.3.3-1 ➑ 24.3.4-3                            
  Podman                         β”‚5.3.2-1                                         
  Nvidia                         β”‚565.77-1 ➑ 570.86.16-1                         
                                                                                  
  ### Major DX packages                                                           
                                                                                  
  Name                               β”‚Version                                     
  ───────────────────────────────────┼────────────────────────────────────────────
  Incus                              β”‚6.8-1 ➑ 6.9-0.1                            
  Docker                             β”‚27.5.1-1                                    
  Devpod                             β”‚v0.6.8-1                                    
                                                                                  
  ### Commits                                                                     
                                                                                  
  Hash                                   β”‚Subject                                 
  ───────────────────────────────────────┼────────────────────────────────────────
  da1b7a3 https://github.com/ublue-os/au…│feat: add vfio modules to image (#198)  
  238c055 https://github.com/ublue-os/au…│feat: add font downloader and embelish …
  9a7a934 https://github.com/ublue-os/au…│feat(just): add aurora-specific changel…
  3f06229 https://github.com/ublue-os/au…│feat: enable fw-fanctrl support (+ectoo…
  8083d46 https://github.com/ublue-os/au…│feat: change to CachyOS addons repo for…
  8fbdca3 https://github.com/ublue-os/au…│fix: Justfile to avoid collision with f…
  56195c1 https://github.com/ublue-os/au…│fix: make cursor smaller in gtk flatpak…
  0f40f38 https://github.com/ublue-os/au…│feat: add scx-scheds package (#186)     
  91cef18 https://github.com/ublue-os/au…│fix: remove bootc key from topgrade (#1…
  5aab168 https://github.com/ublue-os/au…│fix: remove ublue-update and rely on un…
  2b3e5af https://github.com/ublue-os/au…│fix: update topgrade.toml (#183)        
  d81321e https://github.com/ublue-os/au…│fix: fail build if kf6-kio version skew…
  7e5bad4 https://github.com/ublue-os/au…│fix: make ujust update use direct calls…
  82abec2 https://github.com/ublue-os/au…│Revert "feat: update fastfetch logo to …
  d792d92 https://github.com/ublue-os/au…│feat: update fastfetch logo to new logo…
  b013855 https://github.com/ublue-os/au…│feat: remove adw-gtk3 and inputleap fro…
  f1671d1 https://github.com/ublue-os/au…│fix: update funding (#176)              
  c404580 https://github.com/ublue-os/au…│feat: add KCM for managing updates/reba…
  69ed7fa https://github.com/ublue-os/au…│fix: flatpak comparison in ujust script…
  a7f998d https://github.com/ublue-os/au…│feat: install new flatpaks for current …
  64f98ce https://github.com/ublue-os/au…│feat: add udev rules on just recipe ope…
  c382ca5 https://github.com/ublue-os/au…│Create 99-pcscd.rules (#164)            
  93f9bdc https://github.com/ublue-os/au…│fix: rename tmpfiles.d conf file (#162) 
  2c10e3e https://github.com/ublue-os/au…│feat: add build recipe to rollback stab…
  c9e6748 https://github.com/ublue-os/au…│feat: update stable kernel to 6.12.9 (#…
  5f66f88 https://github.com/ublue-os/au…│feat: Do not duplicate isos anymore and…
  2eafb39 https://github.com/ublue-os/au…│fix: use libheif from Fedora to support…
                                                                                  
  ### All Images                                                                  
                                                                                  
    β”‚Name                     β”‚Previous                  β”‚New                     
  ──┼─────────────────────────┼──────────────────────────┼────────────────────────
  βœ¨β”‚assimp                   β”‚                          β”‚5.3.1-3                 
  βœ¨β”‚fw-ectool                β”‚                          β”‚v0.3.3-3.20221204.git54…
  βœ¨β”‚fw-fanctrl               β”‚                          β”‚0.0.0-8.20241107.gitf49…
  βœ¨β”‚gpg-pubkey               β”‚                          β”‚b99058cd-59130e89       
  βœ¨β”‚jsoncpp                  β”‚                          β”‚1.9.5-8                 
  βœ¨β”‚kcm_ublue                β”‚                          β”‚0.5.4-1                 
  βœ¨β”‚kf6-kio                  β”‚                          β”‚6.10.0-2.switcheroo.0.0…
  βœ¨β”‚openxr-libs              β”‚                          β”‚1.1.43-1                
  βœ¨β”‚poly2tri                 β”‚                          β”‚0.0^20130501hg26242d0aa…
  βœ¨β”‚pugixml                  β”‚                          β”‚1.13-6                  
  βœ¨β”‚scx-scheds               β”‚                          β”‚1.0.9-1                 
  βœ¨β”‚topgrade                 β”‚                          β”‚15.0.0-1                
  πŸ”„β”‚ImageMagick              β”‚7.1.1.41-1                β”‚7.1.1.43-1              
  πŸ”„β”‚accountsservice          β”‚23.13.9-5                 β”‚23.13.9-8               
  πŸ”„β”‚brlapi                   β”‚0.8.5-19                  β”‚0.8.5-22                
  πŸ”„β”‚brltty                   β”‚6.6-19                    β”‚6.6-22                  
  πŸ”„β”‚cmake-filesystem         β”‚3.30.5-1                  β”‚3.30.7-1                
  πŸ”„β”‚cosign                   β”‚2.4.1-1                   β”‚2.4.2-1                 
  πŸ”„β”‚crypto-policies          β”‚20241029-1.git8baf557     β”‚20250124-1.git4d262e7   
  πŸ”„β”‚distrobox                β”‚1.8.0-1                   β”‚1.8.1.2-1               
  πŸ”„β”‚freeipa-client           β”‚4.12.2-6                  β”‚4.12.2-8                
  πŸ”„β”‚fzf                      β”‚0.57.0-1                  β”‚0.58.0-1                
  πŸ”„β”‚glibc                    β”‚2.40-17                   β”‚2.40-21                 
  πŸ”„β”‚gnutls                   β”‚3.8.6-7                   β”‚3.8.8-3                 
  πŸ”„β”‚gum                      β”‚0.15.1-1                  β”‚0.15.2-1                
  πŸ”„β”‚heif-pixbuf-loader       β”‚1.18.2-5                  β”‚1.17.6-2                
  πŸ”„β”‚hwdata                   β”‚0.391-1                   β”‚0.392-1                 
  πŸ”„β”‚ibus-typing-booster      β”‚2.27.10-1                 β”‚2.27.16-1               
  πŸ”„β”‚just                     β”‚1.38.0-1                  β”‚1.39.0-1                
  πŸ”„β”‚jxl-pixbuf-loader        β”‚0.10.3-5                  β”‚0.10.4-1                
  πŸ”„β”‚kernel-tools             β”‚6.12.10-200               β”‚6.12.11-200             
  πŸ”„β”‚kmod-xone                β”‚0.0.git.218.d8c42b2e-1    β”‚1000.0.0.git.284.a89672…
  πŸ”„β”‚less                     β”‚661-2                     β”‚668-1                   
  πŸ”„β”‚libX11                   β”‚1.8.10-2                  β”‚1.8.11-1                
  πŸ”„β”‚libdav1d                 β”‚1.5.0-1                   β”‚1.5.1-1                 
  πŸ”„β”‚libipa_hbac              β”‚2.10.1-1                  β”‚2.10.2-1                
  πŸ”„β”‚libnfsidmap              β”‚2.8.1-5.rc2               β”‚2.8.1-7.rc2             
  πŸ”„β”‚libsoup3                 β”‚3.6.0-1                   β”‚3.6.4-1                 
  πŸ”„β”‚libusb1                  β”‚1.0.27-6                  β”‚1.0.27-9                
  πŸ”„β”‚llvm-libs                β”‚19.1.7-1                  β”‚19.1.7-2                
  πŸ”„β”‚openssh                  β”‚9.9p1-1                   β”‚9.9p1-2                 
  πŸ”„β”‚policycoreutils          β”‚3.7-6                     β”‚3.7-7                   
  πŸ”„β”‚python3-boto3            β”‚1.35.93-1                 β”‚1.36.12-1               
  πŸ”„β”‚python3-cryptography     β”‚43.0.0-3                  β”‚43.0.0-4                
  πŸ”„β”‚python3-packaging        β”‚24.1-2                    β”‚24.2-3                  
  πŸ”„β”‚python3-pydantic         β”‚2.10.5-1                  β”‚2.10.6-1                
  πŸ”„β”‚python3-wrapt            β”‚1.17.0-2                  β”‚1.17.1-1                
  πŸ”„β”‚qt6-filesystem           β”‚6.8.1-4                   β”‚6.8.2-1                 
  πŸ”„β”‚qt6-qtbase               β”‚6.8.1-11                  β”‚6.8.2-2                 
  πŸ”„β”‚selinux-policy           β”‚41.28-1                   β”‚41.32-1                 
  πŸ”„β”‚stress-ng                β”‚0.18.08-1                 β”‚0.18.09-1               
  πŸ”„β”‚tailscale                β”‚1.78.1-1                  β”‚1.80.0-1                
  πŸ”„β”‚tuned                    β”‚2.25.0-0.1.rc1            β”‚2.25.1-1                
  πŸ”„β”‚unbound-anchor           β”‚1.22.0-8                  β”‚1.22.0-11               
  πŸ”„β”‚xorg-x11-server-Xwayland β”‚24.1.4-1                  β”‚24.1.5-1                
  βŒβ”‚add-determinism          β”‚0.3.6-3                   β”‚                        
  βŒβ”‚adw-gtk3-theme           β”‚5.6-1                     β”‚                        
  βŒβ”‚akmod-xone               β”‚0.0.git.251.c7fb361c-1    β”‚                        
  βŒβ”‚akmods                   β”‚0.6.0-8                   β”‚                        
  βŒβ”‚annobin-docs             β”‚12.69-1                   β”‚                        
  βŒβ”‚ansible-srpm-macros      β”‚1-16                      β”‚                        
  βŒβ”‚fakeroot                 β”‚1.36.2-1                  β”‚                        
  βŒβ”‚forge-srpm-macros        β”‚0.4.0-1                   β”‚                        
  βŒβ”‚fpc-srpm-macros          β”‚1.3-13                    β”‚                        
  βŒβ”‚ghc-srpm-macros          β”‚1.9.1-2                   β”‚                        
  βŒβ”‚gnat-srpm-macros         β”‚6-6                       β”‚                        
  βŒβ”‚go-srpm-macros           β”‚3.6.0-5                   β”‚                        
  βŒβ”‚grubby                   β”‚8.40-78                   β”‚                        
  βŒβ”‚input-leap               β”‚3.0.2-1                   β”‚                        
  βŒβ”‚kernel-srpm-macros       β”‚1.0-24                    β”‚                        
  βŒβ”‚kmodtool                 β”‚1.1-11                    β”‚                        
  βŒβ”‚libgit2                  β”‚1.9.0-4                   β”‚                        
  βŒβ”‚llhttp                   β”‚9.2.1-2                   β”‚                        
  βŒβ”‚lua-srpm-macros          β”‚1-14                      β”‚                        
  βŒβ”‚ocaml-srpm-macros        β”‚10-3                      β”‚                        
  βŒβ”‚openblas-srpm-macros     β”‚2-18                      β”‚                        
  βŒβ”‚package-notes-srpm-macrosβ”‚0.5-12                    β”‚                        
  βŒβ”‚perl-srpm-macros         β”‚1-56                      β”‚                        
  βŒβ”‚pyproject-srpm-macros    β”‚1.16.4-1                  β”‚                        
  βŒβ”‚python-srpm-macros       β”‚3.13-3                    β”‚                        
  βŒβ”‚python3-babel            β”‚2.16.0-1                  β”‚                        
  βŒβ”‚python3-click-plugins    β”‚1.1.1-22                  β”‚                        
  βŒβ”‚python3-hid-parser       β”‚0.0.3-9.20210228git7d94740β”‚                        
  βŒβ”‚python3-progressbar2     β”‚3.53.2-13                 β”‚                        
  βŒβ”‚python3-rpmautospec      β”‚0.7.3-1                   β”‚                        
  βŒβ”‚python3-rpmautospec-core β”‚0.1.5-2                   β”‚                        
  βŒβ”‚python3-utils            β”‚3.7.0-6                   β”‚                        
  βŒβ”‚python3-xlib             β”‚0.33-8                    β”‚                        
  βŒβ”‚redhat-rpm-config        β”‚293-1                     β”‚                        
  βŒβ”‚rpmdevtools              β”‚9.6-8                     β”‚                        
  βŒβ”‚rust-srpm-macros         β”‚26.3-3                    β”‚                        
  βŒβ”‚solaar                   β”‚1.1.14-7                  β”‚                        
  βŒβ”‚ublue-update             β”‚1.3.2+26c55f6-1           β”‚                        
  βŒβ”‚zig-srpm-macros          β”‚1-3                       β”‚                        
                                                                                  
  ### Dev Experience Images https://docs.projectbluefin.io/bluefin-dx             
                                                                                  
    β”‚Name            β”‚Previous                  β”‚New                              
  ──┼────────────────┼──────────────────────────┼─────────────────────────────────
  πŸ”„β”‚bpftop          β”‚0.5.1-4                   β”‚0.5.1-8                          
  πŸ”„β”‚clang-libs      β”‚19.1.7-1                  β”‚19.1.7-2                         
  πŸ”„β”‚clevis-pin-tpm2 β”‚0.5.3-7                   β”‚0.5.3-9                          
  πŸ”„β”‚cockpit-bridge  β”‚331-1                     β”‚332-1                            
  πŸ”„β”‚cockpit-machinesβ”‚326-1                     β”‚327-1                            
  πŸ”„β”‚cockpit-ostree  β”‚206-1                     β”‚207-1                            
  πŸ”„β”‚cockpit-podman  β”‚99-1                      β”‚100-1                            
  πŸ”„β”‚code            β”‚1.96.4-1736991167.el8     β”‚1.97.0-1738713467.el8            
  πŸ”„β”‚cowsql          β”‚1.15.6-3                  β”‚1.15.8-0.1                       
  πŸ”„β”‚gnutls-utils    β”‚3.8.6-7                   β”‚3.8.8-3                          
  πŸ”„β”‚guestfs-tools   β”‚1.53.5-1                  β”‚1.53.5-2                         
  πŸ”„β”‚gvisor-tap-vsockβ”‚0.8.1-1                   β”‚0.8.2-1                          
  πŸ”„β”‚hivex-libs      β”‚1.3.24-1                  β”‚1.3.24-2                         
  πŸ”„β”‚kcli            β”‚99.0.0.git.202501231904.d…│99.0.0.git.202502070946.17f4fdd-0
  πŸ”„β”‚libguestfs      β”‚1.55.3-1                  β”‚1.55.3-2                         
  πŸ”„β”‚libnbd          β”‚1.21.6-1                  β”‚1.21.6-2                         
  πŸ”„β”‚nbdkit          β”‚1.40.4-1                  β”‚1.40.4-2                         
  πŸ”„β”‚osinfo-db       β”‚20240701-2                β”‚20250124-1                       
  πŸ”„β”‚podman-tui      β”‚1.3.0-3                   β”‚1.3.1-1                          
  πŸ”„β”‚supermin        β”‚5.3.5-1                   β”‚5.3.5-2                          
  πŸ”„β”‚usbredir        β”‚0.13.0-5                  β”‚0.15.0-1                         
  πŸ”„β”‚virt-v2v        β”‚2.7.4-1                   β”‚2.7.4-2                          
  πŸ”„β”‚xen-libs        β”‚4.19.1-3                  β”‚4.19.1-4                         
                                                                                  
  ### Nvidia Images                                                               
                                                                                  
     β”‚Name                β”‚Previous                   β”‚New                        
  ───┼────────────────────┼───────────────────────────┼───────────────────────────
  πŸ”„ β”‚egl-wayland         β”‚1.1.18~20241210git0c6f823-1β”‚1.1.18~20250114git26ba0e3-2
  πŸ”„ β”‚libva-nvidia-driver β”‚0.0.13^20241108git259b7b7-1β”‚0.0.13^20241108git259b7b7-2
                                                                                  
  ### How to rebase                                                               
                                                                                  
  For current users, type the following to rebase to this version:                
                                                                                  
    # Get Image Name                                                              
    IMAGE_NAME=$(jq -r '.["image-name"]' < /usr/share/ublue-os/image-info.json)   
                                                                                  
    # For this Stream                                                             
    sudo bootc switch --enforce-container-sigpolicy ghcr.io/ublue-                
  os/$IMAGE_NAME:stable                                                           
                                                                                  
    # For this Specific Image:                                                    
    sudo bootc switch --enforce-container-sigpolicy ghcr.io/ublue-                
  os/$IMAGE_NAME:stable-20250209.1                                                
                                                                                  
  ### Documentation                                                               
                                                                                  
  Be sure to read the documentation https://docs.projectbluefin.io/ for more      
  information                                                                     
  on how to use your cloud native system.

What is your partition layout like? Is it a custom partition scheme you have created during install?

40gigs sounds kind of small partition size.

Hi again, and thank you for helping me.
I do have a custom partition scheme, it looks like this:

lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
zram0       252:0    0     8G  0 disk [SWAP]
nvme0n1     259:0    0 953.9G  0 disk 
β”œβ”€nvme0n1p1 259:1    0   600M  0 part /boot/efi
β”œβ”€nvme0n1p2 259:2    0     1G  0 part /boot
β”œβ”€nvme0n1p3 259:3    0    40G  0 part /usr/bin/swtpm
β”‚                                     /var
β”‚                                     /sysroot/ostree/deploy/default/var
β”‚                                     /usr
β”‚                                     /etc
β”‚                                     /
β”‚                                     /sysroot
└─nvme0n1p4 259:4    0 912.3G  0 part /var/home/jan

This has worked for me for a long time, never had problems with it.

In the mean time I re-installed the whole system and kept my home folder as it was, meaning I still have all the personal settings and files.
Now when I look at the root partition I see this:

df -lh /
Filesystem       Size  Used Avail Use% Mounted on
/dev/nvme0n1p3   40G   28G  9.7G  74% /

It is filled more than what I expected to see. I can see if I can get it larger and at the same time reducing the home partition, just to be safe, but there is still the question why I have to do that. The 40 GB was always more than enough and now suddenly the disk was full and after the re-install it is 3/4 full.
What is different now? Any idea?

This is how it looks now:

❯ df -lh /
Filesystem      Size  Used Avail Use% Mounted on
/dev/nvme0n1p3   59G   28G   28G  50% /

Let’ see how long I can use this setup before I run into problems again. I have used Aurora for 3-4 months now and it was always good, suddenly today the root disk was completely full. Why? I have no idea.

Well, thank you @inffy for your help.

Btrfs supports subvolumes so a separate partition isn’t fully necessary between root and home.

Generally speaking when testing I usually allocate 64 GB per system which should support up to 3 images without concern.

The reason for my manual setup is I don’t trust btrfs and the automated partitioning Fedora uses.
A few years ago I was using openSUSE Tumbleweed on btrfs and during a holiday where I had little access to the internet my disk ran full and the whole system crashed. Thanks to Starbucks and their free wifi I could download an ISO file and install the system again, but since / and /home were on the same part of the disk my home folder was gone.
Now I have a separate home folder and can re-install in 15 minutes, including installing the Flatpaks I use. Feels way better.

1 Like

You realize you can do the same with a subvolume?

No, I don’t. After the disaster with the crashing computer when the disk was full, I never looked at btrfs again. I still use ext4 because it has always served me well.