Ollama on Bazzite with Nvidia GPU support

Bazzite
1

Hi!

In the last few days I was trying to run Ollama on Bazzite, ran into many problems, but have been successful in the end running it in a podman container. I’m sharing the following guide I wrote so it can help other people.

Would you be interested in adding it to the wiki? This might be the right place (there is already a guide for Ollama in docker, but Bazzite doesn’t have docker installed).

I would be happy to hear any feedback (including language corrections since English is not my first language).

This guide shows how to setup Ollama on Bazzite including Nvidia GPU support.

I’ve ended up using the container (docker) version of Ollama because:

  • Although the default direct installation script seemed to work for me at first, the systemd ollama.service wasn’t able to start after a PC reboot because it wanted to create a file in a read-only location. Can be maybe fixed by messing around with the install/run paths, but I haven’t tried it.
  • Installing Ollama via brew (brew install ollama, brew services start ollama) might be the easiest way for running models on CPU, but at the time of writing (Feb 2025) the brew version doesn’t seem to have Nvidia GPU support.

1. Generating CDI specification

To generate the Container Device Interface (CDI) specification, follow the podman docs (or Nvidia docs). Thanks to CDI, devices can be accessed from within a container.

2. Running Ollama container

Now, you should be able to run the Ollama docker container in podman including Nvidia GPU support:

podman run --device nvidia.com/gpu=all --security-opt=label=disable -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

3. Using Ollama as a systemd service

To enable Ollama as a systemd service, create the file ~/.config/containers/systemd/ollama.container with the content below (this is a Quadlet unit; other locations are also possible):

[Unit]
Description=The ollama container
# After=local-fs.target

[Container]
ContainerName=ollama
Image=ollama/ollama
AddDevice=nvidia.com/gpu=all
SecurityLabelDisable=true
PublishPort=11434:11434
Volume=ollama:/root/.ollama:z

# Remove if you don't want autostart
[Install]
WantedBy=default.target

And then run

systemctl --user daemon-reload
systemctl --user start ollama.service
2 Likes
5

(Sorry for the deleted messages. Originaly as a new user I couldn’t post more than two links in a post so I split the guide into multiple replies, but I’ve been promoted since so the whole guide is back in the main thread post.)

6

Hi,

Thanks for documenting this.

There used to be an ollama container in a just file recipe but it was removed in favour of Alpaca - see “Has ujust ollama-web been removed?”.

You can still use homebrew for a quick install without a container brew install ollama and even install a user service:

❯ brew services start ollama
Created symlink '/home/myuser/.config/systemd/user/default.target.wants/homebrew.ollama.service' → '/home/myuser/.config/systemd/user/homebrew.ollama.service'.

● homebrew.ollama.service - Homebrew generated unit for ollama
     Loaded: loaded (/home/myuser/.config/systemd/user/homebrew.ollama.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/user/service.d
             └─10-timeout-abort.conf
     Active: active (running) since Sun 2025-02-23 22:06:27 GMT; 1min 8s ago

I still think it would be useful to have an “Advanced” section for the container approach, and even @j0rge suggested (see Has ujust ollama-web been removed? - #33 by j0rge) raising a PR against that wiki to add the instructions. So go ahead and do it.

7

Thanks for the reply!

As said before, if you want API (Alpaca doesn’t expose it) and use Nvidia GPU (Ollama in brew doesn’t support it), you want the Ollama container.

Apparently there is a pull request trying to achieve something similar so I’ll link my guide there.

1 Like
8

So, I’ve just ended up using the docker container again, and connecting Alpaca to it. For some reason Alpaca refuses to use my Nvidia GPU and falls back to CPU…

9

Can you describe your issue more?

Do you run the container in docker or podman?

Also, do you still have the brew version installed? If so, are you sure you are connecting to the container, not the brew version, which doesn’t support Nvidia GPU right now? (You can for example try changing the port for the container)

10

So, I have uninstalled the brew version completely.

Both your instructions (the user container from docker.io/ollama/ollama, running with podman) and also the “ramalama” container created by the documentation linked work fine: the Nvidia GPU is detected and used.

The problem is the ollama that is embedded in the Alpaca flatpak: You don’t need any container running with Alpaca (nor homebrew) because it include ollama with Nvidia support inside the flatpak (the AMD support is an extra flatpak). For whatever reason this has stopped working. I have made a detailed post in Alpaca github discussions describing the issue.

I’m now using your instructions to run the podman container with ollama/ollama (where the 4060 works fine), and then in Alpaca I have defined an external (non-managed) instance so that I can connect to it and work with acceleration (rather than have Alpaca launch its managed one that does not work).

11

Well, I’m glad that my instructions still work for you.

I’m thinking that running Ollama as a system service might be the best way if you want to try different UIs like Alpaca, Open WebUI, using it inside VSCode for code completion etc. - everything can connect to the one system service.

12

Update: I think only the third step might be sufficient on Bazzite. If it doesn’t work, try the first two too.

Also, if someone who understands podman is reading this, could you please check the proposed additons to the documentation? There is some more discussion about what to put into the configuration file.