GitLab’s built in continuous integration (CI) tools are some of the best in the industry. Onyx Point has been leading an effort to improve GitLab’s CI security. Continue reading to learn more about how Onyx Point has implemented more secure job access controls in high-performance computing infrastructures.
GitLab CI runners allow users to define a suite of tests for their code repositories that can be configured to run automatically. Runners can be installed on any infrastructure that can connect to the GitLab instance and can be configured to be shared across the entire instance or locked to a specific repository. Runners can be configured out of the box to leverage one of an impressive number of virtualization tools, including Docker and Kubernetes. Runners can also be configured to run shell commands if you want to run the tests without virtualization.
When a shell runner is registered on a piece of infrastructure it can be configured to use either a system account (
gitlab-runner by default) or to run as a privileged user in case that test requires additional access.
One of the side-effects of the shared nature of shell-based runners is that it can be difficult to guarantee security. If CI jobs started on behalf of different users are picked up and run by the same shell-based runner, the logs for those jobs will be placed into shared directories. Further, there is not really a good way to limit the amount of access a specific repo, user, or job has to the filesystem on the infrastructure that the shell-based runner is installed on. It is possible to limit access to the system account that the shell-based runner runs as, but it’s impossible to make those permissions granular based on who started the job or what the job is attempting to do.
SetUID Runner Extension to GitLab Runners
Recently the US Department of Energy identified the need for user-based granular control over the amount of access granted to specific CI jobs on their high performance computing infrastructure and was referred to Onyx Point as a potential resource. Onyx Point and several DoE supercomputing laboratories worked together to define specific requirements and the concept of a “SetUID Runner” was created. An Onyx Point development team began work on an implementation.
The goals for the first phase of the project were pretty straightforward, CI jobs should be secured on a per-user basis.
- The CI system, leveraging shell-based GitLab runners, should have some concept of which user a specific CI job is running on behalf of
- Jobs that are run on behalf of a specific user should only have access to the underlying infrastructure equal to the the access granted to that user
- Artifacts and data that are cloned onto underlying infrastructure as part of a CI job that is running on behalf of a specific user should be isolated from the artifacts and data that ‘belong’ to other users’ jobs.
With those goals in mind, we set out to gather information on the specific needs of the individual DoE labs.
As is the case with most enterprise systems, User Account Management (UAM) is handled by an external system that interfaces with specific tools to provide access across the infrastructure. Usually the UAM system is LDAP or in the case of a Microsoft Windows infrastructure, Active Directory. This brings us to our first assumption: If the UAM system allows a user with access to running CI jobs to log into GitLab then that user will also have an account on the underlying infrastructure upon which the GitLab Runner is installed. We can make this assumption because both the GitLab Server and the GitLab Runner systems are managed by the same UAM system.
This assumption immediately lightens the implementation burden significantly. Without this assumption, we have to interface with an external system that pairs GitLab users to system accounts, methods for providing secure tokens to users on the GitLab front end, and add network code to perform validation of those tokens each time a CI job starts. Because all of the DoE labs already leverage UAM systems to grant access to subsystems like runner infrastructure and the GitLab server front-end, we can rely on the existing authentication systems to do most of that validation.
The easiest way to run bash code with the same level of access as a specific user account is to run the bash code as that user. This makes our implementation a simple series of steps:
- At the start of a CI job get information about the user that the job is running on behalf of from the GitLab front-end.
- Ensure that a corresponding user account exists on the underlying CI system and that the user is allowed to run CI jobs.
- Create a location on the filesystem that the specified user has access to for the artifacts and data required to complete the job. The location must be isolated from other users’ jobs.
- Fork the shell-based runner process such that the child process belongs to the specified user.
- Execute the contents of the CI job as the user specified in the location specified. If the job attempts to access locations on the filesystem that it doesn’t have access to or if commands are run that the specified user would not be allowed to run, the job should fail gracefully.
These steps represent the key pieces of the SetUID Runner’s functionality. Once the basic functionality was in place, further conversations with the would-be end users led to the identification of additional features that would greatly enhance user productivity.
The additional features included:
- The ability to set, on a per-runner basis, whitelists and blacklists for both users and groups.
- The ability to create custom file system build paths, and to control ownership of the directories created to support those paths. The goal was to ensure that each user had access to only their own builds, but still allow sites to control access for groups of users who might need to access the other builds within their group.
Keep an eye out for a second post on SetUID Runners covering a deeper dive into the specifics of the implementation of both the SetUID Runners and these additional features. You can also view the whole repository here.
Conclusion and Future Plans
SetUID Runners are the first step in a plan to improve security and implement new functionality for the DoE’s HPC facilities. The next step is to integrate SetUID Runners with existing HPC job scheduling tools, such as Platform Load Sharing Facility (LSF) or SLURM Workload Manager.
Development is a collaborative effort headed by a development team at Onyx Point, LLC. with teams at GitLab and the DoE facilities. Development is public, and the final product is intended to be an Open Source addition to the GitLab code base.
Onyx Point is dedicated to supporting open standards, reducing
vendor lock-in, and contributing to the open source community.
We have partnered with GitLab to help extend modern development
practices to more organizations by providing professional
services, training, and custom development.
to learn more about how we can help you.
At Onyx Point, our engineers focus on Security, System
Administration, Automation, Dataflow, and DevOps consulting for
government and commercial clients. We offer professional
services for Puppet, RedHat, SIMP, NiFi, GitLab, and the other
solutions in place that keep your systems running securely and
efficiently. We offer Open Source Software support and
Engineering and Consulting services through GSA IT Schedule 70.
As Open Source contributors and advocates, we encourage the use
of FOSS products in Government as part of an overarching IT
Efficiencies plan to reduce ongoing IT expenditures attributed
to software licensing. Our support and contributions to Open
Source, are just one of our many guiding principles
- Customer First.
- Security in All We Do.
- Pursue Innovation with Integrity.
- Communicate Openly and Respectfully.
- Offer Your Talents, and Appreciate the Talents of Others