What does this mean for the security of this project as it relies on github actions to create its images. Is it just a manner of waiting for an upstream patch?
Hey there!
Thanks for the post! I was one of the maintainers involved in incident response here, and can share a few details.
Firstly, we are using GitHub Actions, and were making use of one of those compromised actions inside our ublue-os/packages repository. That being the tj-actions/changed-files action, though thankfully the workflows where we use the compromised action do not have access to any credentials, so were able to avoid any credentials or secrets from leaking.
On 15th March, just a couple hours after Step Security posted their write-up on the compromise, the Universal Blue maintainers performed a discovery on all workflows within the org and across multiple upstream projects to determine where these compromised actions were in use. We found a single instance inside the ublue-os/packages repository, used to decide which RPM spec files we want to build during a Pull Request.
Now we had an inventory of all locations where this action was being used, we swiftly disabled the affected workflows in GitHub Actions to prevent any more logs from being generated. This means 30 minutes after being aware of the issue, we had put temporary measures in place to prevent further log generation.
We then started looking through all logs generated by the affected repository dating back to when the tj-actions/changed-files action was compromised, and determined none of our secrets were outputted to the build logs, and therefore Universal Blue wasn’t impacted by the supply chain attack.
90 minutes after we were made aware of the incident, we had prevented the compromised code from running again, and ensured none of our secrets were compromised. We then started looking at how do we restore functionality to the ublue-os/packages repository, since this is a vital part of our infrastructure. Thankfully, Step Security announced they had decided to maintain their own (secure) fork of the tj-actions/changed-files action which they made available for the community. We promptly switched to this version and resumed workflows without issue.
This incident has reinforced the importance of supply-chain security, and are actively making changes across the org to protect us from a similar supply chain compromise in the future. The most important of which is pinning our GitHub Actions to their Git hashes, rather than relying on Git tags. Since Git hashes are immutable, they cannot be modified (*with some exceptions) whereas anybody is able to point a Git tag at another commit, branch or even forked repository. This allows us to be aware of each and every change to the actions we depend on, and Renovate now automates these updates for us to help keep us up-to-date.
We’ve learned a lot from this event and are continuing to refine our security practices to mitigate future risks.
12 Likes
Stellar response!
Thanks to everyone involved!
1 Like