Machine Learning

Machine Learning

Note: We want Bluefin and Aurora to be the best developer environment for AI enthusiasts, help wanted!

Local AI

Ollama allows the running of open-source large language models, such as Llama 3, locally. It bundles model weights, configuration, and data into a single package, defined by a Modelfile, and optimizes setup and configuration details, including GPU usage.

Bluefin-dx supports the installation of Ollama in different ways, for example by using the following ujust commands:

  • ujust ollama installs the CLI-version as a container.
  • ujust ollama-web installs Open Web UI & Ollama as a container. During the installation process, there is the choice to install either a GPU or CPU-enabled version. Additionally, installation through Homebrew (brew install ollama) is required.

systemd does not autostart the containers; instead, the user needs to activate the script manually by using systemctl --user start ollama or systemctl --user start ollama-web. The systemd scripts are saved under: ~/.config/containers/systemd. The scripts are:

  • ollama.container - which starts the CLI under port: 11434
  • ollama-web.container - which starts the Open Web UI under port: 8080 (http://localhost:11434)
  • ollama.network, the network name is set as “ollama”

To cross-check if the containers are launched correctly, you can use podman ps --all.

Running the ollama open-webui

ujust ollama-web will set up ollama with the webui as a service unit in your user space:

  • Status: systemctl status --user ollama-web
  • Restart: systemctl restart --user ollama-web

You can also start, stop, and disable and enable the service unit. Open localhost:8080 in your browser, then make yourself an account:

Running ollama

ujust ollama to get started! This will setup an ollama service unit in your user space. Use this one if you just want ollama without the web ui:

  • Status: systemctl status --user ollama
  • Restart: systemctl restart --user ollama

You can also start, stop, and disable and enable the service unit.

Desktop integration

Install alpaca to use a native desktop application.

Se the server as http://localhost:11434 in the settings to connect to your local setup:

Pytorch

Bluefin and Aurora include a NGC container that includes the latest stable PyTorch from Nvidia on top of Ubuntu LTS. It includes python3, pip3 , pandas, matplotlib, and scikit-learn.

The additional pip packages are commonly used but not comprehensive. Only pip is used and mamba or conda are not currently tested.

Pre-requisites

You must already be using bluefin-dx-nvidia as it’s meant for those GPUs and has nvidia container toolkit support. If you want to test before downloading the larger nvidia container run nvidia-smi
inside a regular Ubuntu box (distrobox create --nvidia -i ubuntu) to check that everything works correctly.

To create the box use the following command. Note that this container is large (20+ GB!):

ujust distrobox-mlbox

To enter the working environment:

distrobox enter mlbox

Then the init-hooks will run once. After which, you should be able to run:

nvidia-smi

To check if GPUs are seen and enter the python repo run:

import torch;torch.cuda.is_available()

Various tests can be run to test a transformers inference or training job, or to git clone a pytorch benchmarks repo and run single or multi gpu commands: E.g. to test multi-gpu setup on two 3090s:

git clone https://github.com/aime-team/pytorch-benchmarks.git
cd pytorch-benchmarks
python3 main.py --num_gpus 2 --compile --model bert-large-uncased --data_name squad --global_batch_size 24

On other operating systems, use this .ini file and run:

distrobox assemble create --file /path/to/your/mlbox.ini

Doc credits: @axelquack

3 Likes

Many thanks for your work.

I’m just going to test the init file.

I’m running dx-nvidia but I’m not seeing the ujust distrobox-mlbox command.

Looks like we misplaced it while reorganizing things, I filed this so we fix it, thanks for the feedback! (Re)Surface the mlbox · Issue #1014 · ublue-os/bluefin · GitHub

(I’ll post back when we fix this)

2 Likes

Thanks!

I also have been looking for the pre-init hook with no luck, I think it’s also missing.

I dont know why but if i install it without the hook nvidia-smi doesn’t work. Which is weird, because I have an ubuntu distrobox working well with the nvidia drivers.

I’ve created mlbox and it seems to be a podman container.

❯ podman container ls | grep mlbox
dd361651282f  nvcr.io/nvidia/pytorch:23.07-py3  --verbose --name ...  11 hours ago  Up 6 hours                            mlbox

I want to connect to the container using Visual Studio and this is currently not supported with podman, so I forced docker:

❯ DBX_CONTAINER_MANAGER=docker distrobox assemble create --file pytorch-nvidia.in

❯ DBX_CONTAINER_MANAGER=docker distrobox list
ID           | NAME                 | STATUS             | IMAGE                         
1dd9595a10be | mlbox                | Up 14 minutes      | nvcr.io/nvidia/pytorch:23.07-py3

❯ DBX_CONTAINER_MANAGER=docker distrobox enter mlbox

I’m rather new to podman/toolbox/distrobox (have only used docker in the past) so I’m sure others probably know better, but just putting this here in case others are also “getting started” with this.

I was able to attach to mlbox with VS Code and will try out pytorch.

It appears this is not “lost”. I found it in:

> cat /etc/distrobox/apps.ini
# Add more distros and then use:
# distrobox assemble create --replace --file /etc/distrobox/apps.ini --name containername
# This will replace your distroboxes in place
# Tip: Use additional_packages as declarative config!
# See https://distrobox.privatedns.org/usage/distrobox-assemble/
[mlbox]
image=nvcr.io/nvidia/pytorch:23.08-py3
additional_packages="nano git htop"
init_hooks="pip3 install huggingface_hub tokenizers transformers accelerate datasets wandb peft bitsandbytes fastcore fastprogress watermark torchmetrics deepspeed"
pre-init-hooks="/init_script.sh"
nvidia=true
pull=true
root=false
replace=false

[obs-studio-portable]
image=ghcr.io/ublue-os/obs-studio-portable
nvidia=true
exported_apps=obs
entry=false

I am also able to install it with?:

# optiona: if you want to use docker instead of podman
> export DBX_CONTAINER_MANAGER=docker

> ujust setup-distrobox-app
# Select "mlbox" from the menu

The only “concerning” thing is a message complaining about the pre-init-hook specified?:

WARNING: This will assemble and replace the container mlbox
with the one defined in /etc/distrobox/apps.ini.
/tmp/tmp.dzOlp8dPse: line 4: pre-init-hooks=/init_script.sh: No such file or directory
 - Creating mlbox...
...

Other than that, the container seems to be created successfully…

I have created it both ways (once in my laptop and once in my desktop).

Hello everyone,

I am working on a container with ROCm support. I’ve managed to get it working with PyTorch, but there are some quirks I need help with my mlbox-rocm.ini file:

[mlbox-rocm]
image=rocm/dev-ubuntu-22.04
additional_packages="git"
#pre-init-hooks="/init_script.sh"
nvidia=true
pull=false
root=false
# Init hooks will fail so commenting them out, see below

Now, where is the the problems:

  1. There is not enough space in the /tmp file-system by default to install pytorch with ROCm support
  2. I need multiple init hooks, as I need to run two pip install commands using different index sites.

For (1) the problem is simple, by default Bluefin-DX has an 8GB temp file system:

> df -h /tmp
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           7.8G  107M  7.7G   2% /tmp

As a result, once the container is created, I install it by overriding TMPDIR with:

> TMPDIR="/home/myuser/tmp" pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.0

So my first question is: is it possible to override the temporary directory in the manifest instructions so that the user’s home is used?

Now regarding the second problem, as you can see you need to specify --index-url https://download.pytorch.org/whl/rocm6.0 to get the correct pytorch for ROCm. Therefore I need to run two separate commands, one is the one immediately above, the second would pull in the extra packages afterword. Something like:

# This hook must run first to install PyTorch for ROCm. Note that it must also override TMPDIR
init_hoos_01="TMPDIR=<what_can_i_use?> pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.0"

# This hook can run second, it will resolve pytorch as already-installed and only add the extras
 init_hooks_02="pip3 install huggingface_hub tokenizers transformers accelerate datasets peft bitsandbytes"

I have created the container and ran the two commands manually, and PyTorch works fine with my Radeon 6800.

Ollama allows the running of open-source large language models, such as Llama 3, locally. It bundles model weights, configuration, and data into a single package, defined by a Modelfile, and optimizes setup and configuration details, including GPU usage.

Bluefin-dx supports the installation of Ollama in different ways, for example by using the following ujust commands:

  • ujust ollama installs the CLI-version as a container.
  • ujust ollama-web installs Open Web UI & Ollama as a container. During the installation process, there is the choice to install either a GPU or CPU-enabled version.

Additionally, installation through Homebrew (brew install ollama) is required.

systemd does not autostart the containers; instead, the user needs to activate the script manually by using systemctl --user start ollama or systemctl --user start ollama-web.

The systemd scripts are saved under: ~/.config/containers/systemd. The scripts are:

  • ollama.container - which starts the CLI under port: 11434
  • ollama-web.container - which starts the Open Web UI under port: 8080 (http://localhost:11434)
  • ollama.network, the network name is set as “ollama”

To cross-check if the containers are launched correctly, you can use podman ps --all.

1 Like