[Bug Report] Bazzite does not handle OOM properly

Bazzite
1

Hello, I do work in blender and of course as standard sometimes I push things a bit to far and run out of memory. I have 32gbs of ddr5. Although this same issue has been affecting me on 32gbs of ddr4 to, Not that the generation/ram speed matters in this instance, just wanted to let people know that is has been affecting me for a while.

image
image1920Ɨ546 140 KB

However unlike on windows(As far back as the early versions of windows NT workstation and Windows XP) or on MacOS. when Atomic Fedora(At least if im aware. This might just be a bazzite thing but I will take the doubt and say that it might also happen on Base Fedora Atomic as well)/Bazzite runs out of memory the entire system goes down, everything freezes and im forced to force restart(flick the psu switch) the computer losing my work, Typically this happens sometimes almost everyday for me and it affects my work. Im here to report that OOM is not working on bazzite or whatever is happening when the system runs out of memory is not enough to prevent the entire system breaking. I mean this nicely and just wish for bazzite to improve of course

2

With just 2 blender instances open im already using a staggering 24gbs of ram
(This is everything open on my computer right now in total)

image
image1920Ɨ1079 273 KB

With just only 6 gbs available in ram. Of course i understand that what gnome or most services do is not bazzites fault and I canā€™t blame bazzite for taking up all my ram.
But still thatā€™s LITTLE wiggle room. If i were to launch a third blender instance out of nowhere there is a good chance that for a second I could go OOM and the entire system would go down. Which is why I bring up this problem.

3

image
image1567Ɨ1094 41.7 KB

4

We ship systemd-oomd with the same config that Fedora uses.

5

huh.. well now im really confused. Because yeah your right it does use that.

image
image907Ɨ380 47.2 KB

Tell you what. Ill record you a video showing what happens on OOM for me(This has happened on both AM4 and AM5 for me). (It just happened again, ran out of memory and the entire system froze and would not come back, had to pull the power plug again)

6

Yeah no I can comfirm. When bazzite runs out of memory it just freezes and never comes back. OOM does not appear to save the system at all. Even if it just a LITTLE bit reaches ā€œOOMā€ territory the entire system goes down.

7

I have had a few OOM events running Wan 2 models in comfy-ui with ROCM. I will say sometimes oomd will kill the process causing the OOM but not always. I have had a few OOMs that the system would crawl to a halt. It was never clear to me when it would happen and it wasnā€™t consistent.

I have a feeling that the default config for oomd is not aggressive enough which allows some system hangs to happen. I havenā€™t had a chance to look at the logs myself about why this is.

8

please im begging. fix itā€¦

I cant take these crashes anymore

9

A few weeks ago I did have a OOM event un one of mi PCs (AMD Ryzen 5, 32 GB DDR 5), in which I left open Firefox with many tabs, and it did slowly use the free memory until used almost all, and the OOM killed Firefox. I know this because a day or two later, when powering on the TV attached to it with HDMI, I saw a message telling me that Firefox was closed because the system was out of memory, and everything keep working ok.

So, this is to say that I think that the systemd-oomd is working when memory is appropiated slowly by a program, but may me the problem is when it is appropiated faster, not giving oomd time to kill the process.

At the same time, I donā€™t know if a more aggressive oomd setting could result in false positives, killing programs erroneouslyā€¦

10

Here are instructions on how to manually switch from swap on zram to disk based swap with zswap enabled. This may help resolve some memory issues, as it allows data to be moved out of memory completely (freeing it for another use), rather than compressing it but keeping it in memory (what zram does). This worked for me on Bluefin, and I assume Bazzite is similar enough for this to work there too.

Step 1, switch off zram:

Create a file /etc/systemd/zram-generator.conf which overrides the system zram settings. The file can be empty. In mine Iā€™ve put a comment to myself:

# This empty file overrides `/usr/lib/systemd/zram-generator.conf`.

Step 2, create a swapfile:

Assuming the system is using btrfs (this is the default on Bluefin, so I assume Bazzite is the same). Change the size to whatever makes sense for you.

sudo btrfs filesystem mkswapfile --size 8G /var/swapfile

Add the swapfile to /etc/fstab. The entry for me looks like this:

/var/swapfile    none    swap    pri=0    0 0

Step 3, enable zswap (optional):

sudo rpm-ostree kargs --append='zswap.enabled=1'

Step 4, reboot.

Step 5, confirm settings:

The command cat /proc/swaps can confirm the active swap. It should now show only your swapfile.

Example:

> cat /proc/swaps
Filename				Type		Size		Used		Priority
/var/swapfile                           file		8388604		0		0

If you enabled zswap, check it is enabled:

> sudo dmesg | grep -F zswap
[    0.000000] Command line: BOOT_IMAGE=(hd1,gpt5)/ostree/default-2ee8f203d4c5b97fd5fc04daf879a9dad7dc1c405c8aba2b6ddbda907ce9816e/vmlinuz-6.18.13-200.fc43.x86_64 rd.luks.uuid=luks-0c755fd5-99d6-4578-bb46-328eb72fd038 rd.lvm.lv=vg0/root rhgb quiet root=UUID=8e069d8e-b937-4ad1-a692-cd921ee55f6f rootflags=subvol=ub_root rw ostree=/ostree/boot.0/default/2ee8f203d4c5b97fd5fc04daf879a9dad7dc1c405c8aba2b6ddbda907ce9816e/0 rd.luks.options=discard zswap.enabled=1
[    0.048097] Kernel command line: BOOT_IMAGE=(hd1,gpt5)/ostree/default-2ee8f203d4c5b97fd5fc04daf879a9dad7dc1c405c8aba2b6ddbda907ce9816e/vmlinuz-6.18.13-200.fc43.x86_64 rd.luks.uuid=luks-0c755fd5-99d6-4578-bb46-328eb72fd038 rd.lvm.lv=vg0/root rhgb quiet root=UUID=8e069d8e-b937-4ad1-a692-cd921ee55f6f rootflags=subvol=ub_root rw ostree=/ostree/boot.0/default/2ee8f203d4c5b97fd5fc04daf879a9dad7dc1c405c8aba2b6ddbda907ce9816e/0 rd.luks.options=discard zswap.enabled=1
[    1.116601] zswap: loaded using pool lzo

I am not well enough informed of the tradeoffs to have opinions on whether the Bazzite defaults should be changed but I know that Chris Down recommends disk swap with zswap instead of zram (ref).

11

My experience has been that it eventually kills something and recovers, but this takes a LONG time. Encountering this frequently is very frustrating. I have layered earlyoom which is supposed to be much quicker about killing an application before the system freezes up entirely and this has been a better experience on my end. I would prefer a more graceful swapping to disk that resulted in slower performance than killing memory-using applications.

12

The Bazzite docs also have a page on using disk swap: Using Swap for Additional Memory or Hibernation - Bazzite Documentation

Itā€™s probably better to follow those instructions than mine.

13

Can confirm the same behavior, noted here: Overwatch 2 Freezing System - #4 by Baz

I ended up just buying two more 8GB ram sticks and upgrading from 16GB to 32GB of ram. Not a great solution but now I donā€™t run out any more so issue mitigated lol. Never had the issue on this PC with Windows but something in the Linux stack seems to handle OOM issues less gracefully.

14

Wait thatā€™s not enabled by default???

15

I turn zram off and switch to zswap on all of my Bazzite setups, Im sure there is a reason for using zram, but it just seams odd to me swapping ram to ram to save ram? kind of like a snake eating its own tail.

This is what I do:

rpm-ostree kargs --append-if-missing=zswap.enabled=1
rpm-ostree kargs --append-if-missing=zswap.max_pool_percent=25
rpm-ostree kargs --append-if-missing=zswap.compressor=lz4
rpm-ostree kargs --append-if-missing=systemd.zram=0
sudo btrfs subvolume create /var/swap
sudo semanage fcontext -a -t var_t /var/swap
sudo restorecon /var/swap
sudo btrfs filesystem mkswapfile --size 16G /var/swap/swapfile
sudo semanage fcontext -a -t swapfile_t /var/swap/swapfile
sudo restorecon /var/swap/swapfile
sudo sed -i ā€˜//var/swap/swapfile/dā€™ /etc/fstab
sudo bash -c ā€˜echo /var/swap/swapfile none swap defaults,nofail 0 0 >> /etc/fstabā€™
sudo echo ā€˜vm.swappiness = 180ā€™ | sudo tee /etc/sysctl.d/99-swappiness.conf
rpm-ostree initramfs --enable --arg=ā€“add-drivers --arg=lz4
systemctl reboot

Just replace the ā€œ16Gā€ with the size you want your swapfile to be, and the lz4 compression has the least overhead on the CPU, I wrote this to help optimise systems like the BC-250 that have very limited shred ram and were suffering with OOM issues and hard lockin in a number of games, mainly crap unoptimised games based on the UE5 engine, but thats another discussionā€¦

If you would like to change the swapfile to a different size later on then you can run this:

sudo swapoff /var/swap/swapfile
sudo rm -f /var/swap/swapfile
sudo btrfs subvolume delete /var/swap
sudo btrfs subvolume create /var/swap
sudo semanage fcontext -a -t var_t /var/swap
sudo restorecon /var/swap 2>/dev/null
sudo btrfs filesystem mkswapfile --size 32G /var/swap/swapfile
sudo semanage fcontext -a -t swapfile_t /var/swap/swapfile
sudo restorecon /var/swap/swapfile
sudo sed -i ā€˜//var/swap/swapfile/dā€™ /etc/fstab
sudo bash -c ā€˜echo /var/swap/swapfile none swap defaults,nofail 0 0 >> /etc/fstabā€™
sudo echo ā€˜vm.swappiness = 180ā€™ | sudo tee /etc/sysctl.d/99-swappiness.conf
rpm-ostree initramfs --enable --arg=ā€“add-drivers --arg=lz4
systemctl reboot

Change the ā€œ32Gā€ to whaty ever size you would like.

After you reboot, run this to check if everything worked:

swapon --show

16

Too many issues: Bluetooth, Wiā€‘Fi, missing modules/drivers in the system and moreā€¦
But everything works under version 43.
For now Iā€™ll stay on 43, but if after 6 months they donā€™t fix most of the issues, Iā€™m moving to the newest Fedora ā€” everything works there.

17

If your using the Deck version, then yes we are still waiting for a major Kernel update, I also found that certain Mediatek Wifi/BT chipsets are not working right, Intel and Realtek are all functioning perfectly though since about 6.15.

Fedora KDE is fantastic but not as good for gaming IMO, or for media playback support, not out of the box anyway.

Maybe give CachyOS a go, as its on the very latest Kernel, its no where near as polished as Bazzite, but its still quite easy to install and setup, and ā€œshouldā€ fix most of the hardware issues, and in some cases has a small bump in gaming performance on lower-end hardware.

I have three identical Dell Laptops one is on each of the above, except the Bazzite one is on 44, and I prefer to use that one as my daily driver, the OS is just more complete and usable as portable daily driver, Fedora often has bugs, and CachyOS just takes more steps to do anything on it, including updating.

18

The worst part for me was that my AM4 system broke and while at microcenter replacing it im a shy nervous guy and was easily presuaded into getting a AM5 system.. I have a whole bunch of am4 sticks of ramā€¦ but well.. only 32gbs of ddr5 ramā€¦ and I canā€™t just upgradeā€¦ have you seen the price of ddr5 ram..

I know that running out of ram constantly on 32gbs is a first world problem, and it is. But the philosophy of ā€œUnused ram is wasted ramā€ really has infested modern linux and you would be surprised how much I justā€¦ run outta ramā€¦

Until I sell my organs to the cartel for money im pretty much stuck with 32gbs

19

Iā€™ve used Fedora prior to discovering Universal Blue - problems with OOM handling like this is something that drove me away from Fedora to CachyOS at one point. Unfortunately when Fedora runs out of memory ā€¦ things just halt. I donā€™t think this particular issue is tied to Bazzite or can be easily fixed.