SetUID Runners Implementation

Back to Listing

Clayton Mentzer

Hanover, MD, 24 July 2018


This post follows up on the previous SetUID Runners article by taking a deeper look at code and rationale for specific features. In the previous post we outlined our goals and process for the first phase of ongoing work to improve security and functionality of GitLab CI Runners at the Department of Energy’s (DoE) High Performance Computing (HPC) labs. If you haven’t seen it, you can read it here

To recap, our development team had several primary goals, as well as additional features that were identified later in the development process. The main goals of the project can be summed up in the following steps:

  1. At the start of a CI job, get information about the user that the job is running on behalf of from the GitLab front-end.
  2. Ensure that a corresponding user account exists on the underlying CI system, and that the user is allowed to run CI jobs.
  3. Create a location on the filesystem that the specified user has access to for the artifacts and data required to complete the job. The location must be isolated from other users’ jobs.
  4. Fork the shell-based runner process such that the child process belongs to the specified user.
  5. Execute the contents of the CI job as the user specified, in the location specified. If the job attempts to access locations on the filesystem that it doesn’t have access to, or if commands are run that the specified user would not be allowed to run, fail gracefully.

These steps represent the key pieces of the SetUID Runner’s functionality. Once the basic functionality was in place, further conversations with the would-be end users led to the identification of a few other quality of life features that we were able to add in relatively easily.

The additional features included:

  • The ability to set, on a per-runner basis, whitelists and blacklists for both users and groups.
  • The ability to create custom file system build paths, and to control ownership of the directories created to support those paths. The goal here was to ensure that each user had access to only their own builds, but also to allow sites to grant access to groups of users who might want to access the builds of other users in their group.

In the following pages we’ll outline the steps taken at a high level, with specific references to functions and structures in the actual code. Since the entire project is open-source, each code block will also have a link to the corresponding lines of code in the open-source repository that we’re working from.

Step 1: Get information about the user the job should run as.

This is pretty trivial, because the GitLab server includes information about the state of the front-end, including the logged-in user, in the job specification that is sent to the Runner that picks up the job.

Once we have the logged in user’s name, it’s a simple matter of using GoLang’s os library to get the user’s UID and the GIDs of the groups that the user belongs to.

[
  // Ensure the specified user exits, and has permissions to run jobs.
  // manage whitelist / blacklist functionality
  func (s *executor) ValidateUser(options common.ExecutorPrepareOptions) error {
    variables := options.Build.GetAllVariables()
    gitlabUser := variables.Get("GITLAB_USER_LOGIN")

    // Set the UID/GID of the logged in user
    usr, usrErr := user.Lookup(gitlabUser)
    if usrErr != nil {
      return fmt.Errorf("golang was unable to perform a lookup on the logged in user %s: %s", gitlabUser, usrErr)
    }
  }
...
]

View on GitLab

Step 2: Ensure that the user is allowed to run CI jobs.

As part of the SetUID runner implementation, we added configuration keys to /etc/gitlab-runner/config.toml file that allows for additional configuration of runners. These configuration keys included ones that allow sites to specify whitelists and blacklists for both specific users, and users that are members of specific groups. The documentation outlines our rationale for the order of precedence for whitelists and blacklists, which is as follows:

  • Users in the “user whitelist” will always be able to run CI jobs. Full stop.
  • Users in the “user blacklist” will not be allowed to run CI jobs, unless they are also on the user whitelist.
  • Users who belong to any group that is specified in the “groups blacklist” will not be allowed to run jobs, unless they are also specified in the user whitelist.
  • Users who belong to any group in the “groups whitelist” will be allowed to run jobs, but only if they are not also included in either of the blacklists. Further, if a groups whitelist is specified and a given user is not a member of any group on that whitelist, that user will not be permitted to run CI jobs (again, barring the case that that user is explicitly included in the user whitelist).

The rationale for ordering the precedence of the lists this way is that some sites may only want to specify a groups whitelist, and nothing else. In that case the desired functionality is that only users that are members of groups included in the groups whitelist can run CI jobs. The groups blacklist is higher precedence than the groups whitelist simply for security reasons. If a user belongs to multiple groups, one of which is on the whitelist and one of which is on the blacklist, we deny them access rather than permit.

Implementation of the user whitelists is trivial, just check to see if a username is a member of a list of strings:


  // For reference, s is the shell object we are operating on, and
  // setting the values of the ValidatedUser... members of the struct
  // is later used to fork the process as that user.

  // If the user is in the user whitelist, we always run the job
  if contains(setuidUserWhitelist, gitlabUser) {
    s.ValidatedUserUID = uid
    s.ValidatedUserGID = gid
    s.ValidatedUser = gitlabUser
    s.UserHomeDir = usr.HomeDir
    return nil

    // If the user is not in the whitelist, check if the user is in the user
    // blacklist. If it is, we will deny always
  } else if contains(setuidUserBlacklist, gitlabUser) {
    return fmt.Errorf("the logged in user, %s, is not in the user whitelist and is in the user blacklist", gitlabUser)
  }
...

View on GitLab

For reference, the contains function simply checks if a string is a member of a list of strings:


func contains(list []string, item string) bool {
  if len(list) == 0 {
    // the list is empty
    return false
  }
  for _, x := range list {
    if x == item {
      return true
    }
  }
  return false
}
...

View on GitLab

Assuming the user is not in either of the user lists, we move on the group whitelist and blacklist. Implementation of the group whitelist and blacklist is a simple nested for loop, where we check each group that a user is a member of for equality against the whitelist or blacklist.

   // If the user is not in either user list, check the group blacklist.
   // If the user is in the group blacklist, we deny
 } else if inGroupBlacklist, sharedBlacklistedGroups = CheckGroupBlacklist(setuidGroupBlacklist, groups); inGroupBlacklist {
   return fmt.Errorf("the logged in user, %s, is not on the user whitelist and is a member of the following groups that are on the group
s blacklist, and is not allowed to run CI jobs: %s", gitlabUser, sharedBlacklistedGroups)

   // If the user is on no blacklists and is in any group on the groups
   // whitelist, they are allowed to run CI jobs.
 } else if inGroupWhitelist = CheckGroupWhitelist(setuidGroupWhitelist, groups); inGroupWhitelist {
   s.ValidatedUserUID = uid
   s.ValidatedUserGID = gid
   s.ValidatedUser = gitlabUser
   s.UserHomeDir = usr.HomeDir
   return nil

   // If there isnt a group whitelist at all, and we haven't failed yet, then
   // the user is allowed to run jobs.
 } else if len(setuidGroupWhitelist) == 0 {
   s.ValidatedUserUID = uid
   s.ValidatedUserGID = gid
   s.ValidatedUser = gitlabUser
   s.UserHomeDir = usr.HomeDir
   return nil

   // If the user is not in ANY lists AND there is a groups whitelist
   // defined, that user is not allowed to run jobs.
 } else if len(setuidGroupWhitelist) > 0 && !inGroupWhitelist {
   return fmt.Errorf("a group whitelist exists, but the user %s is not a member of any groups that are on that whitelist, and is not allowed to run CI jobs", gitlabUser)
...

View on GitLab

Step 3: Ensure there exists a location on the filesystem for job artifacts and data

There are several options available in /etc/gitlab-runner/config.toml that sites can use to specify a location for build artifacts. The most common use case is the one where a shared builds directory is specified, and each user has an access-restricted subdir for their own builds.

The process for creating the directory structure is fairly straightforward, but because we have to check if a directory exists and then create it if it does not for each layer of the directory tree, it’s too much code to show here. What is worth showing is the function that actually creates a single directory:

// given a string path, owernship, and mode -> create the directory with the given
// owner, group, and mode. If it already exists, do nothing
func (s *executor) CreateDirectory(dir string, owner, group int, mode os.FileMode) error {
  if _, err := os.Stat(dir); os.IsNotExist(err) {
    // path/to/file does not exist
    fmt.Printf("directory %s does not exist, creating it..\n", dir)
    buildErr := os.MkdirAll(dir, mode)

    if buildErr != nil {
      fmt.Printf("CreateDirectory failed, os.mkdirall returned err %s\n", buildErr)
      return buildErr
    }

    ownershipErr := s.EnsureOwnership(dir, owner, group)
    if ownershipErr != nil {
      fmt.Printf("ownership modification failed, s.ensureOwnership returned err %s\n", ownershipErr)
      return ownershipErr
    }
    return nil
  } else {
    // path to file exists already
    return nil
  }
}

func (s *executor) EnsureOwnership(dir string, uid, gid int) error {
  fmt.Printf("Setting ownership of %v to %v %v\n", dir, uid, gid)
  chownErr := os.Chown(dir, uid, gid)
  if chownErr != nil {
    return chownErr
  }
  return nil
}

View on GitLab

Steps 4 and 5. Fork the runner process, and run the CI job in the child process

The process of forking the runner process and creating a process as the desired user leverages GoLang’s os/exec library. Effectively, the process builds out a shell object, which contains a number of environment variables and meta-information, including the actual commands to run, their arguments, a number of environment variables, and the UID of a user to run the command as. After we validate that the user exists on the system, is allowed to run jobs, and that the directory structure is appropriate for the job to leverage, we just need to set the correct environment variables to ensure that the shell commands run as the desired user.

The PrepareSetUID function is responsible for updating the appropriate structures with user metadata:

func (s *executor) prepareSetUID(options common.ExecutorPrepareOptions, mapping func(string) string) error {
  user_valid_err := s.ValidateUser(options)

  if user_valid_err != nil {
    return fmt.Errorf("User provided for SetUID Runner is not valid, error code %v", user_valid_err)
  }

  s.PrepareSetUIDDirectories(options, mapping)
  s.Shell().User = s.ValidatedUser

  return nil
}

View on GitLab

Finally, after several more steps, the completed structures are passed into the final run function, which kicks off the child process and monitors the running job. Parts of the function have been removed for clarity:

func (s *executor) Run(cmd common.ExecutorCommand) error {
  c := exec.Command(s.BuildShell.Command, s.BuildShell.Arguments...)

...

  helpers.SetProcessGroup(c)
  defer helpers.KillProcessGroup(c)

  // Fill process environment variables
  c.Env = append(os.Environ(), s.BuildShell.Environment...)

...

    scriptFile := filepath.Join(scriptDir, "script."+s.BuildShell.Extension)
    err = ioutil.WriteFile(scriptFile, []byte(cmd.Script), 0700)
    if err != nil {
      return err
    }

...

    c.Args = append(c.Args, scriptFile)

...

  // Start a process
  err := c.Start()
}

View on GitLab

Next Steps

SetUID Runners are the first step in a plan to improve security and implement new functionality for the DoE’s HPC facilities. The next step is to integrate SetUID Runners with existing HPC job scheduling tools, such as Platform Load Sharing Facility (Usually just LSF) or SL Once the next phase of the project nears completion, there will likely be a set of similar posts regarding the ongoing development process. If you’re interested in designing systems to interface with HPC facilities, keep an eye out for both the high-level overview and the technical deep dive. nt team at Onyx Point, INC., which is in frequent contact with teams at GitLab and the DoE facilities. Development is public, and the final product is intended to be an open-source addition to the GitLab code base.

Onyx Point is dedicated to supporting open standards, reducing vendor lock-in, and contributing to the open source community. We have partnered with GitLab to help extend modern development practices to more organizations by providing professional services, training, and custom development.

Click here to learn more about how we can help you.

At Onyx Point, our engineers focus on Security, System Administration, Automation, Dataflow, and DevOps consulting for government and commercial clients. We offer professional services for Puppet, RedHat, SIMP, NiFi, GitLab, and the other solutions in place that keep your systems running securely and efficiently. We offer Open Source Software support and Engineering and Consulting services through GSA IT Schedule 70. As Open Source contributors and advocates, we encourage the use of FOSS products in Government as part of an overarching IT Efficiencies plan to reduce ongoing IT expenditures attributed to software licensing. Our support and contributions to Open Source, are just one of our many guiding principles

  • Customer First.
  • Security in All We Do.
  • Pursue Innovation with Integrity.
  • Communicate Openly and Respectfully.
  • Offer Your Talents, and Appreciate the Talents of Others

programming, gitlab, open-source, DoE, security

Share this story

We work with these Technologies + Partners

puppet
gitlab
simp
beaker
redhat
AFCEA
GitHub
FOSSFeb