Machine Learning

Bluefin
1

Machine Learning

Note: We want Bluefin and Aurora to be the best developer environment for AI enthusiasts, help wanted!

Local AI

Ollama allows the running of open-source large language models, such as Llama 3, locally. It bundles model weights, configuration, and data into a single package, defined by a Modelfile, and optimizes setup and configuration details, including GPU usage.

Bluefin-dx supports the installation of Ollama in different ways, for example by using the following ujust commands:

  • ujust ollama installs the CLI-version as a container.
  • ujust ollama-web installs Open Web UI & Ollama as a container. During the installation process, there is the choice to install either a GPU or CPU-enabled version. Additionally, installation through Homebrew (brew install ollama) is required.

systemd does not autostart the containers; instead, the user needs to activate the script manually by using systemctl --user start ollama or systemctl --user start ollama-web. The systemd scripts are saved under: ~/.config/containers/systemd. The scripts are:

  • ollama.container - which starts the CLI under port: 11434
  • ollama-web.container - which starts the Open Web UI under port: 8080 (http://localhost:11434)
  • ollama.network, the network name is set as “ollama”

To cross-check if the containers are launched correctly, you can use podman ps --all.

Running the ollama open-webui

ujust ollama-web will set up ollama with the webui as a service unit in your user space:

  • Status: systemctl status --user ollama-web
  • Restart: systemctl restart --user ollama-web

You can also start, stop, and disable and enable the service unit. Open localhost:8080 in your browser, then make yourself an account:

image
image1336Ă—1239 67.1 KB

Running ollama

ujust ollama to get started! This will setup an ollama service unit in your user space. Use this one if you just want ollama without the web ui:

  • Status: systemctl status --user ollama
  • Restart: systemctl restart --user ollama

You can also start, stop, and disable and enable the service unit.

Desktop integration

Install alpaca to use a native desktop application.

image
image1255Ă—936 74.5 KB

Se the server as http://localhost:11434 in the settings to connect to your local setup:

image
image1255Ă—936 64.6 KB

Terminal Integration

We recommend using mods to use the provided ollama instance as an interface to your command line:

image
image1379Ă—713 134 KB

  • Install mods: brew install mods
  • Open the config file for mods: mods --settings
  • Modify the config file with the model you’ve pulled with ollama:

image
image991Ă—123 26.8 KB

and then make sure the ollama section in the config file matches the same thing:

image
image700Ă—235 30.4 KB

Save the file and then try it:

image
image1347Ă—613 165 KB

Pytorch

Bluefin and Aurora include a NGC container that includes the latest stable PyTorch from Nvidia on top of Ubuntu LTS. It includes python3, pip3 , pandas, matplotlib, and scikit-learn.

The additional pip packages are commonly used but not comprehensive. Only pip is used and mamba or conda are not currently tested.

Pre-requisites

You must already be using bluefin-dx-nvidia as it’s meant for those GPUs and has nvidia container toolkit support. If you want to test before downloading the larger nvidia container run nvidia-smi
inside a regular Ubuntu box (distrobox create --nvidia -i ubuntu) to check that everything works correctly.

To create the box use the following command. Note that this container is large (20+ GB!):

ujust distrobox-mlbox

To enter the working environment:

distrobox enter mlbox

Then the init-hooks will run once. After which, you should be able to run:

nvidia-smi

To check if GPUs are seen and enter the python repo run:

import torch;torch.cuda.is_available()

Various tests can be run to test a transformers inference or training job, or to git clone a pytorch benchmarks repo and run single or multi gpu commands: E.g. to test multi-gpu setup on two 3090s:

git clone https://github.com/aime-team/pytorch-benchmarks.git
cd pytorch-benchmarks
python3 main.py --num_gpus 2 --compile --model bert-large-uncased --data_name squad --global_batch_size 24

On other operating systems, use this .ini file and run:

distrobox assemble create --file /path/to/your/mlbox.ini

Doc credits: @axelquack

3 Likes
2

Many thanks for your work.

I’m just going to test the init file.

I’m running dx-nvidia but I’m not seeing the ujust distrobox-mlbox command.

3

Looks like we misplaced it while reorganizing things, I filed this so we fix it, thanks for the feedback! (Re)Surface the mlbox · Issue #1014 · ublue-os/bluefin · GitHub

(I’ll post back when we fix this)

2 Likes
4

Thanks!

I also have been looking for the pre-init hook with no luck, I think it’s also missing.

I dont know why but if i install it without the hook nvidia-smi doesn’t work. Which is weird, because I have an ubuntu distrobox working well with the nvidia drivers.

5

I’ve created mlbox and it seems to be a podman container.

❯ podman container ls | grep mlbox
dd361651282f  nvcr.io/nvidia/pytorch:23.07-py3  --verbose --name ...  11 hours ago  Up 6 hours                            mlbox

I want to connect to the container using Visual Studio and this is currently not supported with podman, so I forced docker:

❯ DBX_CONTAINER_MANAGER=docker distrobox assemble create --file pytorch-nvidia.in

❯ DBX_CONTAINER_MANAGER=docker distrobox list
ID           | NAME                 | STATUS             | IMAGE                         
1dd9595a10be | mlbox                | Up 14 minutes      | nvcr.io/nvidia/pytorch:23.07-py3

❯ DBX_CONTAINER_MANAGER=docker distrobox enter mlbox

I’m rather new to podman/toolbox/distrobox (have only used docker in the past) so I’m sure others probably know better, but just putting this here in case others are also “getting started” with this.

I was able to attach to mlbox with VS Code and will try out pytorch.

6

It appears this is not “lost”. I found it in:

> cat /etc/distrobox/apps.ini
# Add more distros and then use:
# distrobox assemble create --replace --file /etc/distrobox/apps.ini --name containername
# This will replace your distroboxes in place
# Tip: Use additional_packages as declarative config!
# See https://distrobox.privatedns.org/usage/distrobox-assemble/
[mlbox]
image=nvcr.io/nvidia/pytorch:23.08-py3
additional_packages="nano git htop"
init_hooks="pip3 install huggingface_hub tokenizers transformers accelerate datasets wandb peft bitsandbytes fastcore fastprogress watermark torchmetrics deepspeed"
pre-init-hooks="/init_script.sh"
nvidia=true
pull=true
root=false
replace=false

[obs-studio-portable]
image=ghcr.io/ublue-os/obs-studio-portable
nvidia=true
exported_apps=obs
entry=false

I am also able to install it with?:

# optiona: if you want to use docker instead of podman
> export DBX_CONTAINER_MANAGER=docker

> ujust setup-distrobox-app
# Select "mlbox" from the menu

The only “concerning” thing is a message complaining about the pre-init-hook specified?:

WARNING: This will assemble and replace the container mlbox
with the one defined in /etc/distrobox/apps.ini.
/tmp/tmp.dzOlp8dPse: line 4: pre-init-hooks=/init_script.sh: No such file or directory
 - Creating mlbox...
...

Other than that, the container seems to be created successfully…

I have created it both ways (once in my laptop and once in my desktop).

7

Hello everyone,

I am working on a container with ROCm support. I’ve managed to get it working with PyTorch, but there are some quirks I need help with my mlbox-rocm.ini file:

[mlbox-rocm]
image=rocm/dev-ubuntu-22.04
additional_packages="git"
#pre-init-hooks="/init_script.sh"
nvidia=true
pull=false
root=false
# Init hooks will fail so commenting them out, see below

Now, where is the the problems:

  1. There is not enough space in the /tmp file-system by default to install pytorch with ROCm support
  2. I need multiple init hooks, as I need to run two pip install commands using different index sites.

For (1) the problem is simple, by default Bluefin-DX has an 8GB temp file system:

> df -h /tmp
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           7.8G  107M  7.7G   2% /tmp

As a result, once the container is created, I install it by overriding TMPDIR with:

> TMPDIR="/home/myuser/tmp" pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.0

So my first question is: is it possible to override the temporary directory in the manifest instructions so that the user’s home is used?

Now regarding the second problem, as you can see you need to specify --index-url https://download.pytorch.org/whl/rocm6.0 to get the correct pytorch for ROCm. Therefore I need to run two separate commands, one is the one immediately above, the second would pull in the extra packages afterword. Something like:

# This hook must run first to install PyTorch for ROCm. Note that it must also override TMPDIR
init_hoos_01="TMPDIR=<what_can_i_use?> pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.0"

# This hook can run second, it will resolve pytorch as already-installed and only add the extras
 init_hooks_02="pip3 install huggingface_hub tokenizers transformers accelerate datasets peft bitsandbytes"

I have created the container and ran the two commands manually, and PyTorch works fine with my Radeon 6800.

8

Ollama allows the running of open-source large language models, such as Llama 3, locally. It bundles model weights, configuration, and data into a single package, defined by a Modelfile, and optimizes setup and configuration details, including GPU usage.

Bluefin-dx supports the installation of Ollama in different ways, for example by using the following ujust commands:

  • ujust ollama installs the CLI-version as a container.
  • ujust ollama-web installs Open Web UI & Ollama as a container. During the installation process, there is the choice to install either a GPU or CPU-enabled version.

Additionally, installation through Homebrew (brew install ollama) is required.

systemd does not autostart the containers; instead, the user needs to activate the script manually by using systemctl --user start ollama or systemctl --user start ollama-web.

The systemd scripts are saved under: ~/.config/containers/systemd. The scripts are:

  • ollama.container - which starts the CLI under port: 11434
  • ollama-web.container - which starts the Open Web UI under port: 8080 (http://localhost:11434)
  • ollama.network, the network name is set as “ollama”

To cross-check if the containers are launched correctly, you can use podman ps --all.

1 Like
9

Hello! I have an AMD gpu and when I run ollama --version I see the following output:

ollama version is 0.1.45
Warning: client version is 0.1.48

I assume this is because of a mismatch between the brew ollama cli and the container. For most models, this doesn’t cause an issue. But for gemma2, support was added in 0.1.48, so I can’t use the model with my GPU at the moment.

At first I thought this may be due to the ROCm version lagging behind, but it seems that there is a 0.1.48 image for ROCm on Docker Hub.

So with that in mind, I was wondering: is there a straightforward way to update the ollama container, or at least uninstall it? Looking at the justfile:

    if [  ! -f ~/.config/containers/systemd/ollama.container ]; then
        mkdir -p ~/.config/containers/systemd
        echo "${QUADLET}" > ~/.config/containers/systemd/ollama.container
    else
        echo "Ollama container already exists, skipping..."
    fi

It seems like if any ollama container exists already, then the new one won’t be installed, even if there has been an update. Although I may be misunderstanding the logic, since I’m not super familiar with either just or systemd containers.

EDIT: I was able to resolve this with podman pull docker.io/ollama/ollama:rocm and then podman restart ollama. I still think it would be nice to be able to do this in ujust, but I suppose managing with podman directly works.

10

Yeah we should add a pull there, mind filing an issue in the bluefin repo? Thanks!

1 Like