Should we really recommend vscode + podman as supported?

fdr · December 10, 2023, 12:39pm

It took me some time to actually make devcontainers and podman work together.
The default commands from inside vscode don’t work properly, as there are extra settings needed (per-devcontainer) work properly when using podman (and also not nicely documented anywhere), specially under selinux (like fedora).

bluefin dev-ex main page states that vscode comes prepared to work with podman, but that seems misleading.

Would not be better to direct people to install docker if they intent to use devcontainers and vscode in bluefin, at least until devcontainers works “by default” with podman ?

j0rge · December 10, 2023, 4:41pm

Yeah, we can recommend whatever works best. What was involved in getting it to work with podman? I switched over a while ago so I haven’t done a clean install to check that in a while.

fdr · December 10, 2023, 5:01pm

the lines I had to add to my devcontainer.json where:

"runArgs": [
		"--userns=keep-id",
		"--security-opt=label=disable"
	  ],
"containerEnv": {
		"HOME": "/home/vscode"
	},

this for the default rust devcontainer image.

The default “try devcontainer sample” fails because of the lack of docker.socket .

The default file generated by the default “add devcontainer conf file” will fail because of some root user mismatch. if we run as root , then we get permission denied on the mounted workspace and the files are not accessible.

The “new devcontainer” command fails with the same problem

setting “keep-id” and forcing the devcontainer to use the non root user from the image solved the “boot without root” problem, but the workspace is still not acessible because of selinux.

All of that I found after jumping through several half solved github . There are other possibilities to solve the problem that involve mounting the workspace manually inside your devcontainer, using the “Z” (which will relabel the files in your workspace), forcing everything to run as root , …

The experience to understand (and solve) is confusing and quite frustrating. It’s easy to do the wrong thing if we don’t understand the pieces involved, and is not that hard to end with a scenario that “kinda works”, but will reloabel your files, or change their permission

I’m don’t believe a regular “devcontainer.json” from a public project will work inside podman without change.

I would not recommend that as a “paved road” for someone that just wanna "have things works"™ .

Docker seems to be the “no friction” option for now and it would be better to recommend that instead of podman.

Edit: adding some references where I found those configurations

github.com/microsoft/vscode-remote-release

Podman Usage Improvements

opened 06:44PM - 25 May 22 UTC

nlvw

containers feature-request

Improve default handling of both user Home directories and the use of Podman. This post is mostly to document Podman specific configuration for devcontainer.json to get it working in a friendly fashion. The feature request part is more of a plea to improve the default support so a lot of this custom config is unnecessary. 1. Add built-in support for mounting user home directory. 1.1 Mounting the users home directory is important as it solves the git + ssh issue as your keys and git config will be available inside the container (not everyone uses an ssh agent). It also exposes other user configs such as git configuration, bash configuration, and other tooling. The end result is a better default environment to work out of. 2. Podman should be run unprivileged and as the current user by default. 3. When using Podman selinux needs to be detected and handled without needing specific flags in devcontainer.json. 3.1. 'Z'/'z' should be avoided in bind/volume mounts as it changes the selinux context on files permanently which can break regular access to those files. For instance if you were to mount `$HOME/.ssh` you would break key based ssh authentication for that user until the context is manually reset. settings.json ```json { "terminal.integrated.defaultProfile.linux": "bash", "remote.containers.dockerPath": "podman", "remote.containers.dockerComposePath": "podman-compose" } ``` devcontainer.json (podman specific settings) ```json { "workspaceMount": "", "workspaceFolder": "${localWorkspaceFolder}", "runArgs": [ // run container as current user "--userns=keep-id", // disable selinux isolation that breaks bind mounts "--security-opt=label=disable", // mount user home directory for things like git, ssh, and other configs "--volume=${env:HOME}:${env:HOME}", // ensure project directory is mounted incase it exists outside the home directory "--volume=${localWorkspaceFolder}:${localWorkspaceFolder}", // isolate the .vscode-server folder so you don't overwrite settings from remote ssh vscode "--volume=${localWorkspaceFolder}/.cache/vscode-server:${env:HOME}/.vscode-server" ], "containerEnv": { // ensure users home directory is the same inside the container as it is outside "HOME": "${env:HOME}" } } ``` Relates to: Remote - Containers

github.com/microsoft/vscode-remote-release

Support SELinux enabled systems

opened 06:25PM - 30 Aug 19 UTC

langdon

containers feature-request upstream

Environment - VSCode Version: 1.37.1 - Local OS Version: Fedora 30 - Remote O…S Version: fedora 30, debian 10 (really the python 3 default container) - Remote Extension/Connection Type: Docker Steps to Reproduce: 1. use default open-folder in container 2. choose python 3 container 3. docker exec in to the generated container 4. ls -l /workspaces/name-of-your-project 5. permission denied You can see the same problem in the normal GUI interface but it is less obvious what is going on. You also have the same issue if you use a custom container and (probably) any other container. Basically, as far as I can tell, the bind mount of the user's devel dir into ```/workspaces``` is not using the ```z``` or ```Z``` flag that let's it work well with SELinux. I think this will be particularly a problem as you can't set that flag using the new ```--mount``` option (see: Differences between “--mount” and “--volume” on https://docs.docker.com/engine/reference/commandline/service_create/#add-bind-mounts-or-volumes) at all. There is a workaround that I note here for anyone running into this issue but probably not something the tool should do automatically. You can chcon your devel directory to be modifiable by docker. For example: ```chcon -Rt svirt_sandbox_file_t /full/path/to/your/code``` then reattach your devel dir to the container (probably using rebuild).

j0rge · December 10, 2023, 5:23pm

Ok filed this as an issue!

bigpod · December 10, 2023, 5:42pm

From my understanding selinux part is problematic on docker as well not just podman that is because devcomtainers dont integrate with selinux, in my opinion this should be solved at a system level not by swapping podman for docker which from my understanding does nothing for this issue and just makes worse experiance with uswr needing to be added to right group and such

fdr · December 10, 2023, 6:03pm

It does not seem to be the case.
I did enable docker here, removed all my ‘extra’ flags… Everything just works™. All vs code commands, generated files, samples, etc… All out of the box without “special” arguments

SFaulken · December 11, 2023, 3:05pm

Granted, it’s not apples to apples, but I’ve been using vscode with podman just fine for months, if not as much as a year, on openSUSE Kalpa and distrobox. Seems like if distrobox can make it work, it’s not something that can’t be done.

Is how it’s done over there (granted, that’s not “integrated” and ready to run, but it seems like it’s something that could be adapted and shipped within bluefin/ublue if somebody were interested in doing it.

fdr · December 11, 2023, 6:50pm

yes, it can be done. it’s just that it does not work “out of the box” with devcontainers and podman (which is not the same as using just remote containers as indicated in the link).

Daniel_Walsh · December 11, 2023, 7:31pm

Most likely your dockerd is not running with --selinux-enabled so Docker is not doing SELinux stuff.

Daniel_Walsh · December 11, 2023, 7:33pm

These fields could be set in a containers.conf file, which can be dropped in /etc/containers/containers.conf.d directory at install time.

fdr · December 15, 2023, 9:55pm

Most likely your dockerd is not running with --selinux-enabled so Docker is not doing SELinux stuff.

Not sure, I’m running with the default flags set by from “just docker” , so, its whatever a new user would have.

I added the flags the containers.conf. That “solves” the selinux problem, but the problem regarding workspace mapping and remote user persists and I believe can’t be solved on a general user config.

I only managed to make it work using a handcrafted Dockerfile (as I did before), instead of the regular files created by vscode (which would be what a regular user would get if he was colaborating with other users).

j0rge · December 15, 2023, 10:22pm

Well that’s a start, we can fix the just docker snippets to do that.

fdr · December 16, 2023, 10:57am

There is nothing wrong with docker. It works out of the box without any problems after we remove the settings from vscode to use podman and enable it with just docker.
The problem is the combination vscode + devcontainers (extension) + podman …

I believe I poorly choose my works for the title, I should have said vscode + devcontainers + podman …
I quote Bluefin-dx page here:

Visual Studio Code

[Visual Studio Code] is included on the image to facilitate local development. It comes configured for usage with devcontainers and Podman via a small [default configuration file ].

[Dev Containers Documentation ] - you can skip most of the installation instructions and go directly to [the tutorial ]

[Dev Containers Specification ]

[Beginner’s Series to: Dev Containers ]/watch?v=b1RavPr_878) - great introductory tutorial from the [VS Code YouTube channel ]

That is the main problem I see: vscode + devcontainers + podman is not working out of the box, and I believe it’s not possible to make it work out of the box (currently)… A user can make it work, but the docs give the wrong impression about what a user will get out of the box currently.

Malix · May 30, 2024, 9:09pm

I think it’s possible to edit the title after the post has been created

BTW @fdr,
Is the error you get while using VSCode command palette this one :

?

If so, that would explain why I got trouble setting it up.

geoffrey_smith · May 31, 2024, 4:29am

A lot of issues with devcontainers is that the extensions and features often make giant assumptions about your environment. You are also assuming to use docker or containerd and not OCI complaint images. Docker is now more or less compliant with Podman we need to accept it. It’s like X is better than JS there’s no reason to change. The problem is bad dev container files.

Look at this one from gVisor (Google hypervisor) probably generated by bazel:

github.com

google/gvisor/blob/04e7902bb551d0cde41c23893a3850c2c321e017/.devcontainer.json

{
    "dockerFile": "images/default/Dockerfile",
    "overrideCommand": true,
    "mounts": ["source=/var/run/docker.sock,target=/var/run/docker-host.sock,type=bind"],
    "runArgs": ["--cap-add=SYS_PTRACE", "--security-opt", "seccomp=unconfined"],
    "extensions": [
        "bazelbuild.vscode-bazel"
    ]
}

They have three ways to make the project like make, bazel or I am guessing this file. Podman might catch up but it’s industry standard now for Docker.

Malix · May 31, 2024, 6:20am

So what should be the default minimal addition to .devcontainer/devcontainer.json file to make Podman run on SELinux ?

So far I’ve seen 3 versions, but I don’t know which is the best, and what some options really does and why do they work

1

"runArgs": [
	"--userns=keep-id"
],
"containerEnv": {
	"HOME": "/home/node"
}

2

"runArgs": [
	"--userns=keep-id:uid=1000,gid=1000"
],
"containerUser": "vscode",
"updateRemoteUserUID": true,
"containerEnv": {
	"HOME": "/home/vscode"
},

3

"runArgs": [
	"--userns=keep-id",
	"--security-opt=label=disable"
],
"containerEnv": {
	"HOME": "/home/vscode"
},

Malix · May 31, 2024, 6:29am

Support SELinux enabled systems

opened 06:25PM - 30 Aug 19 UTC

langdon

containers feature-request upstream

Environment - VSCode Version: 1.37.1 - Local OS Version: Fedora 30 - Remote O…S Version: fedora 30, debian 10 (really the python 3 default container) - Remote Extension/Connection Type: Docker Steps to Reproduce: 1. use default open-folder in container 2. choose python 3 container 3. docker exec in to the generated container 4. ls -l /workspaces/name-of-your-project 5. permission denied You can see the same problem in the normal GUI interface but it is less obvious what is going on. You also have the same issue if you use a custom container and (probably) any other container. Basically, as far as I can tell, the bind mount of the user's devel dir into ```/workspaces``` is not using the ```z``` or ```Z``` flag that let's it work well with SELinux. I think this will be particularly a problem as you can't set that flag using the new ```--mount``` option (see: Differences between “--mount” and “--volume” on https://docs.docker.com/engine/reference/commandline/service_create/#add-bind-mounts-or-volumes) at all. There is a workaround that I note here for anyone running into this issue but probably not something the tool should do automatically. You can chcon your devel directory to be modifiable by docker. For example: ```chcon -Rt svirt_sandbox_file_t /full/path/to/your/code``` then reattach your devel dir to the container (probably using rebuild).

github.com/microsoft/vscode-remote-release

Podman Usage Improvements

opened 06:44PM - 25 May 22 UTC

nlvw

containers feature-request podman

Improve default handling of both user Home directories and the use of Podman. This post is mostly to document Podman specific configuration for devcontainer.json to get it working in a friendly fashion. The feature request part is more of a plea to improve the default support so a lot of this custom config is unnecessary. 1. Add built-in support for mounting user home directory. 1.1 Mounting the users home directory is important as it solves the git + ssh issue as your keys and git config will be available inside the container (not everyone uses an ssh agent). It also exposes other user configs such as git configuration, bash configuration, and other tooling. The end result is a better default environment to work out of. 2. Podman should be run unprivileged and as the current user by default. 3. When using Podman selinux needs to be detected and handled without needing specific flags in devcontainer.json. 3.1. 'Z'/'z' should be avoided in bind/volume mounts as it changes the selinux context on files permanently which can break regular access to those files. For instance if you were to mount `$HOME/.ssh` you would break key based ssh authentication for that user until the context is manually reset. settings.json ```json { "terminal.integrated.defaultProfile.linux": "bash", "remote.containers.dockerPath": "podman", "remote.containers.dockerComposePath": "podman-compose" } ``` devcontainer.json (podman specific settings) ```json { "workspaceMount": "", "workspaceFolder": "${localWorkspaceFolder}", "runArgs": [ // run container as current user "--userns=keep-id", // disable selinux isolation that breaks bind mounts "--security-opt=label=disable", // mount user home directory for things like git, ssh, and other configs "--volume=${env:HOME}:${env:HOME}", // ensure project directory is mounted incase it exists outside the home directory "--volume=${localWorkspaceFolder}:${localWorkspaceFolder}", // isolate the .vscode-server folder so you don't overwrite settings from remote ssh vscode "--volume=${localWorkspaceFolder}/.cache/vscode-server:${env:HOME}/.vscode-server" ], "containerEnv": { // ensure users home directory is the same inside the container as it is outside "HOME": "${env:HOME}" } } ``` Relates to: Remote - Containers

fdr · May 31, 2024, 12:08pm

It’s been some time already, but I believe that that was error .

Malix · June 2, 2024, 12:30am

@j0rge, to fix this issue, instead of recommending Docker,

Would you consider introducing gVisor or Quark ?

j0rge · June 2, 2024, 1:08am

I’m confused, what issue are you trying to solve?

Topic		Replies	Views
Devcontainer Setup Bluefin and Aurora developer-experience	8	841	June 4, 2024
Community Guide: Setting up devpod with podman General developer-experience	22	871	September 27, 2024
How to setup VScode for Project Bluefin Bluefin and Aurora developer-experience	6	252	September 12, 2024
Devcontainer workflow question Bluefin and Aurora developer-experience	7	396	December 9, 2023
Error vsCode with Devpod Bluefin and Aurora developer-experience	6	456	September 9, 2024

Visual Studio Code

Related topics