Bluefin Enterprise Woes

Bluefin
1

I am part of a small development team who are all using Bluefin in an enterprise setting. Compared to our experience on Ubuntu LTS it has been great, except for one glaring issue.

Every month or two, there is an update that completely bricks our ability to use our domain logins. As our work requires us to access network shares this hampers us greatly. rpm-ostree rollback is a very useful tool, however, it is not a sustainable solution.

Originally we had to move from GTS to Latest as the SSSD module was completely broken on GTS, however, this ended up making problems appear more often with our machines given the faster SSSD updates.

  • Would it be possible to have the SSSD version pinned, or a slower update schedule for the GTS?
  • Is anyone else using Bluefin in a professional setting with domain logins?
  • Does anyone with Bluefin or enterprise login experience have any helpful advice?

What work is necessary to establish the stability of enterprise logins? If it’s lighter development work I’d be willing to spend a few hours a week to assist.

3 Likes
2

You can pin any image using

sudo ostree admin pin X

X being the image number you wish to pin. Images start at zero. To unpin an image you use

sudo ostree admin pin --unpin X.

3

Enterprise logins have been a pain point.

Pinning certain versions of sssd is something we could easily do. File an issue here with the version you know works here: Sign in to GitHub · GitHub and we can do that, if you need it asap then add that to the ticket. :smiley:

I haven’t had access to this stuff in years, I’m happy to defer to your team for advice and input. Happy to add any packages you may need or configs that are necessary. Maybe we can at least go into 2025 with less regressions for ya. :smile:

1 Like
4

I’ve already pinned the image, however this is not a sustainable option as eventually I’d want the other system packages to be updated as well. I am currently doing this I would just rather prefer that there be a fix eventually.

On that note however if would be nice if pinning and setting default branches were ujust commands.

1 Like
5

Thanks! One of those posts was actually made by one of my coworkers :smile:.
The underlying issue seems to be that sssd itself is moving away from assigned root permissions to the files, and creating a new sssd group to assign permissions with. The transition seems to have a lot of bumps along the way.

We’re testing with both GTS and Stable/Latest at the moment to find a working configuration. Right now it seems that sssd packages at 2.10.0 function just fine, 2.10.1 packages are what’s causing the issue. We’ll put in an issue to see if stable can be pinned at 2.10.0 and latest can continue updating so that we can test when a new sssd configuration begins working again.

6

Ok pin is merging now and will be in the stable-daily tag and migrate over to stable at the end of the week. I’ve created the enterprise-sso label in github, so feel free to file and tag, thanks for helping out!

1 Like
7

Background

Alright some updates after investigating this for another day. We were able to get domain logins working on SSSD version 2.10.1 with help from the SSSD folks.
So it really comes down to two bugs:

Ostree bug

SSSD 2.10.0 required higher capabilities from its binaries. While these capabilities were later reduced in 2.10.1, an ostree bug prevented these xattrs from being extended to the binaries. This results in a mismatch between the capabilities defined in the 2.10.1 sssd system file and the binaries.
This was resolved after an update, but for ostree version 2024.10. Bluefin still uses 2024.9.

To resolve the issue the user must run the following in an offline account.
sudo systemctl edit sssd and input:

[Service]
CapabilityBoundingSet= CAP_SETGID CAP_SETUID CAP_DAC_READ_SEARCH CAP_CHOWN CAP_DAC_OVERRIDE

Restart the computer.

Note: After Bluefin updates to 2024.10 this work around will have to be removed

SSSD Bug

2.10.0 uses sssd:sssd for its directory permissions, 2.10.1 uses root:sssd for its directories. The permissions for the directory were not updated for 2.10.1* and thus the service was unable to access the sssd directory which rendered it unable to start. This should be patched by now, but until it makes it’s way into Bluefin, this can be resolved by running the following:

sudo /bin/find /etc/sssd -type d -exec /bin/chmod g+x {} \;

Restart the computer.

*Note from the SSSD Devs: – this specific issue is with rpm-ostree based systems. More traditional package based system had this covered properly in a spec-file. But this is a known “trap” that rpm-ostree can’t propagate /etc and /var updates of underlying packages. And this is the main reason of all those (really undesirable) gimmicks with chmod/chown in sssd.service…
Sorry for this poor experience. It’s not that we do not test our updates at all, but there is some acknowledged lack of rpm-ostree based systems testing.

Conclusion

Although package versions maybe changed, the permissions for specific file directories may carry over from previous configurations. This means that a new machine booting into 2.10.1 will encounter no issues, however a machine carrying over legacy permissions from 2.10.0 will fail to start the SSSD service.

Based on our testing this will persist even if the machine regresses from 2.10.1 to 2.10.0 in a different build (daily-stable with 2.10.0 still failed to start SSSD). This means that keeping the package version pinned at 2.10.0 is unlikely to be helpful unless a user hasn’t already moved to 2.10.1 at some point.

Recommendations

  • Unpin 2.10.0 from the images
  • Once sssd 2.10.2 and 2024.10 ostree are available those may be some good ones to pin to prevent future issues.

Both latest and stable-daily (and eventually stable) are pulling it. For us to upgrade now we require an image that will provide 2.10.1.

We have a strong interest in the proposed Cent OS stream GTS as a long term solution, it seems that may alleviate many of the issues for enterprise use.

We will provide updates on the forum and issues board for any other SSSD things we run across.

tl;dr

If you have ostree version 2024.9 and SSSD version 2.10.1 run:

sudo /bin/find /etc/sssd -type d -exec /bin/chmod g+x {} \;
sudo systemctl edit sssd

Follow the instructions and input the following:

[Service]
CapabilityBoundingSet= CAP_SETGID CAP_SETUID CAP_DAC_READ_SEARCH CAP_CHOWN CAP_DAC_OVERRIDE

Restart your machine.

2 Likes
8

This topic was automatically closed 24 hours after the last reply. New replies are no longer allowed.