Slurm Integration

This chapter will cover how UCloud/IM for Slurm integrates with Slurm itself. The chapter will cover the internals of how the software will interact with the local Slurm system. The chapter will also discuss how various integrations can ensure automatic management of accounts and quotas.

Introduction

UCloud/IM interacts with the Slurm system through the normal Slurm commands. UCloud/IM does not currently utilize the JSON or REST functionality. As a result, UCloud/IM simply requires a somewhat recent version of Slurm. Any version above 20.02 is guaranteed to work, but older versions might also work.

As covered in the Architecture and Networking chapter, these commands (when responding to a user request) will be run as the corresponding local user. Thus, if Donna#1234 has a local identity of donna01 then all Slurm commands run on her behalf will be run as the donna01 user.

The diagram below summarizes the significant steps involved in the submission of a job.

1. The end-user creates a job specification. This step is typically done through the UCloud user-interface where the user can select from an application catalog. Once selected, the user will be selected with a variety of input parameters. These input parameters can be specific to the software, while others are related to how it should be scheduled.
2. The job specification is submitted to UCloud/Core. Once the specification has been created, a network call is made to UCloud/Core. This call is authenticated as the user’s UCloud identity. Along with the request is also information about which project it should be executed as.
3. UCloud/Core validates the request and forwards it to the relevant provider. UCloud/Core will validate, authenticate and authorize all parts of the request. Once done, it will find the appropriate service provider based on the request and forward the message.
4. The UCloud/IM (User) instance corresponding to the end-user submits the job to Slurm. The details on how this submission takes place is covered by this chapter along with the following chapters.

Machine Types and Slices (Products)

As covered in the User and Project Management chapter, users must first apply and be approved to use any service provider. Their application contain one or more resource allocations for a category of products which the service provider exposes. A product category or, as we will refer to them in this context, a machine type is the unit for which you can receive a resource allocation. Machine types bundle together machines that have similar hardware. For example, if a system have three types of nodes in the HPC system then at least three different machine types are required. A machine type defines several machine slices, also known as a product in the rest of UCloud. Such a slice describes the concrete specification of the slice which includes the number of cores, the amount of RAM and the number of GPUs connected to it.

As seen in the Installation chapter, UCloud/IM can bootstrap the configuration which includes the definition of one or more machine types. The result of this is a section in the config.yml configuration file. In slurm.machines you are able to define as many machine types as you wish. Note that it is possible to have different machine types pointing at the same Slurm partition, as long as it points to a different quality-of-service (QoS). In UCloud/IM for Slurm a machine type corresponds roughly to a (partition, qos) pair. As we will see in the next section, machine types and machine slices play a critical role in how Slurm jobs are submitted to the queue.

In the diagram below we show how the YAML configuration maps to our previous diagram. For the full list of options, see the reference chapter.

slurm:
  machines:
    CPU:
      payment:
        type: Resource
        unit: Cpu
        interval: Minutely

      cpuModel: "AMD EPYC 7742"
      memoryModel: "Model name"

      nameSuffix: Cpu

      cpu: [ 16, 32, 64, 128 ]
      memory: [ 128, 256, 512, 1024 ]

      partition: normal
      qos: normal

Job Submission

UCloud/IM uses the following procedure to submit a job to Slurm:

1. Determine the primary drive belonging to the project. Currently this is the first drive for the project, found by using the defined driveLocator as described in Filesystem Integration.
2. In the primary drive, create the job folder at ucloud-jobs/${UCloudJobId}.
3. In the job folder, create the submission script job.sh.
4. Generate the sbatch preamble and add it to the submission script.
5. Generate the application start-up code using the job specification and referenced application
6. Submit the job using:

   $ sbatch "${PrimaryDrive}/ucloud-jobs/${UCloudJobId}/job.sh"
   412512
   

In this section, we will discuss step 4 of how UCloud/IM selects the submission parameters for the sbatch preamble. In the coming chapters, we will also be discussing step 5, which is how UCloud/IM determines how to run the user’s job.

The diagram below highlights the important steps in submitting a job.

The table below summarizes which submission parameters UCloud/IM will include. Note that the integration module does not support all the option of Slurm, but keep in mind that the integration module does not disallow the direct use of Slurm. Advanced users should still be encouraged to use Slurm directly through the HPC frontend.

Choosing a Slurm Account

Slurm accounts are selected according to the project and the machine type specified for a given job. Irrespective of the account management integration method employed, UCloud/IM must be configured to identify the appropriate account. The account mapping process is defined within the configuration, allowing for the selection between invoking a script or employing the simpler pattern method. Configuration is managed through the slurm.accountManagement.accountMapper section by setting the type property accordingly. Regardless of the chosen method, the same variables are accessible for both the pattern and the script.

slurm:
  accountManagement:
    accountMapper:
      type: Pattern
      users: "#{localUsername}_#{partition}"
      projects: "#{localGroupName}_#{partition}"

slurm:
  accountManagement:
    accountMapper:
      type: Script
      script: "/opt/ucloud/scripts/account-mapper"

{
  /* string */
  "account": ""
}

Variables

If the configured account mapper is ever changed, then this change will only have an effect on new Slurm accounts. The account mapper is evaluated once per project + machine type combination once they receive a resource allocation for this machine type. The result of the account mapper is saved in the internal database for future use. It is possible to list mappings and delete faulty mappings from the command line. Deleting a mapping will automatically trigger the account mapper and resource allocation update trigger to re-run immediately.

# Listing Slurm mappings based on a regex
$ sudo ucloud slurm accounts list --regex ".*_faulty_cpu.*"
| Type    | Project title      | Local ID             | Category | Account             |
|--------------------------------------------------------------------------------------|
| User    | Donna#1234         | donna02              | CPU      | donna02_faulty_cpu  |

# Removing a faulty mapping
# This will NOT delete the Slurm account
$ sudo ucloud slurm accounts rm --account donna02_faulty_cpu
OK

$ sudo ucloud slurm accounts list --regex ".*_faulty_cpu.*"
| Type    | Project title      | Local ID             | Category | Account             |
|--------------------------------------------------------------------------------------|
| No results                                                                           |

Keeping Track of Jobs

UCloud/IM (Server) is responsible for keeping track of all relevant jobs in the Slurm cluster. It will do this by periodically pulling information about all jobs in the relevant partitions of the Slurm cluster. For each job, it will produce an internal update report. If this update report indicates that UCloud/Core should be notified about a change, then it will send either an update or register a new job. New jobs are registered in case of jobs which are submitted outside UCloud. UCloud/IM will keep an internal mapping between Slurm jobs and UCloud jobs, this allows it to know which UCloud job is to be updated.

TODO The integration module should keep track of reservations and generate a warning to the user if their job is set to expire before the node reservation starts. This is similar to how a warning should be generated if a K8s node is cordoned.

Example: Dealing with a Known Job

In this example, we will look at how UCloud/IM (Server) deals with the monitoring of a job which it knows. Jobs submitted through UCloud are always immediately known since a mapping is created when the job is submitted. Jobs which are submitted outside UCloud become known after their initial registration.

In this case, UCloud/IM is looking at the job with Slurm ID 141232. Slurm is reporting that this job is running. The job report shows that a mapping has already been established with the corresponding UCloud job with ID 5124313. The report also states that a state transition has occurred from IN_QUEUE to RUNNING. This trigger a message to UCloud which lets it know that the job is now running. The end-user(s) owning this job will automatically receive a notification about the state change.

Example: Dealing with an Unknown Job (Submitted outside UCloud)

UCloud/IM is not only capable of handling jobs that have been submitted from UCloud, it can also handle jobs that are submitted directly to Slurm. In such a case, the job update report will not be able to find a corresponding UCloud job. This triggers a separate process of finding the appropriate UCloud owner, machine slice and other scheduling parameters. The owner is found by using the database of already invoked account mappers. This is used in combination with the partition and QoS to determine the correct project and machine slice.

If no owner could be found, then no job is registered in UCloud. UCloud/IM will make no attempts to monitor or manage such a job.

Assuming that an owner is found, then a new job is created and registered with UCloud/Core. UCloud/Core will respond with the ID of the new job, which is stored internally in UCloud/IM. This establishes a mapping which means that it is now considered a “known” job.

Integrations

Automatic

This integration will automatically manage Slurm accounts, configure quotas and fairshare. It will achieve this using the standard Slurm mechanisms and will use only the normal command-line tools, as shown in the breakdown section. In order for this to be done, the UCloud service user must have the correct privileges. The setup of these privileges are covered by the Installation chapter as these are already a requirement.

As described in the rest of this chapter, the automatic integration will aim to create exactly one account per (project, machine type) resource allocation that exists in the system. The integration will also ensure that an association is created between the members of the project and the associated Slurm accounts. Members are automatically added and removed, in a similar fashion to how they can be synchronized to Unix groups.

This integration is configured in the following way:

  slurm:
    accountManagement:
      accounting: # Usage and quota management
        type: Automatic

Configuring Partitions and Accounting

This integration depends on the accounting functionality of Slurm. You should generally follow the Slurm documentation for setup. Below we have included a checklist of items that you must ensure have been configured, as the integration depends on these.

1. Confirm that accounting is enabled

You should confirm that the following properties are set. Note that the values likely need to be different for any specific system.

The priority/multifactor plugin must be enabled, otherwise accounting and quotas will not work correctly. You should also make sure that Slurm does not decay or reset usage, because this is all handled by UCloud/IM.

# /etc/slurm/slurm.conf

AccountingStorageType=accounting_storage/slurmdbd
AccountingStorageHost=slurmdbd
AccountingStoragePort=6819

PriorityType=priority/multifactor
PriorityDecayHalfLife=0
PriorityUsageResetPeriod=NONE

2. Configure accounting to enforce the correct properties

You should confirm that the accounting is actually enforced. The property MUST include the options listed here, but MAY include more.

# /etc/slurm/slurm.conf

AccountingStorageEnforce=associations,limits,qos,safe

3. Configure tracking of resource consumption

Slurm will by default track CPU and memory usage. However, it does track GPUs. If the system needs to support a machine type with an accounting unit of GPU hours, then this must be turned on with:

# /etc/slurm/slurm.conf

AccountingStorageTRES=gres/gpu

4. Configure partition(s) to only allow the relevant quality-of-service(s)

This step will ensure that only users with a resource allocation to a partition can actually submit to said partition. The system should generally be configured in such a way that every partition has exactly one corresponding quality-of-service. The only exception to this should be for scavenger queues, in which case two quality-of-services are needed for one partition.

The AllowQoS property must be set on the partition. This can contain a comma-separated value to allow multiple QoS to access a given partition.

# /etc/slurm/slurm.conf

# Example with no scavenger queue
PartitionName=cpu Nodes=c[1-2] AllowQoS=normal

# Example with a scavenger queue
PartitionName=cpu Nodes=c[1-2] AllowQoS=normal,scavenger

A quality-of-service can be configured with the following command:

$ sacctmgr add qos $qosName
 Adding QOS(s)
  $qosName
 Settings
  Description    = $qosName
Would you like to commit changes? (You have 30 seconds to decide)
(N/y): Y

5. Configure the partition’s billing to match the UCloud/IM configuration

This integration uses grptresmins billing for resource tracking. The billing weights are configurable in Slurm and should match the billing configured in the UCloud/IM configuration. The machine type billing configuration looks like this:

slurm:
  machines:
    CPU:
      payment:
        type: Resource
        interval: Minutely

        # One of the following lines will be active
        unit: "Cpu"
        # unit: "Gpu"
        # unit: "Memory"

Depending on the unit used, the billing should be configured differently. The table below shows how to configure the billing for each unit.

PartitionName=mypartition Nodes=c[1-2] AllowQoS=normal TRESBillingWeights="CPU=1.0"

PartitionName=mypartition Nodes=c[1-2] AllowQoS=normal TRESBillingWeights="GRES/gpu=1.0"

PartitionName=mypartition Nodes=c[1-2] AllowQoS=normal TRESBillingWeights="Mem=1.0G"

Breakdown of operations executed by UCloud/IM

$ sacctmgr -nP show account $accountName

$ sacctmgr -i create account $accountName qos=$qos defaultqos=$qos

$ sacctmgr -i modify account $accountName set \
    maxjobs=$maxJobs \
    grptresmins=billing=$resourceMinutes \
    fairshare=$fairshare

$ sacctmgr -i add user $localUsername account=$accountName

$ sacctmgr -i remove user $localUsername account=$accountName

$ sshare -Pho account,user,grptresraw

Scripted

This integration is another script integration. Script integrations allow you to fully customize all aspects of Slurm account and quota management. It is entirely up to you to create these scripts. All scripts will be invoked with a single argument, which is a path to a JSON file. The contents of the JSON file will depend on the script, see below for details. All scripts are expected to return the response as a JSON object on stdout. Detailed error messages and debug information should be printed to stderr. In case of errors a non-zero exit code should be returned.

In order to configure this integration, you must first configure it:

slurm:
  accountManagement:
    accounting:
      type: Scripted
      onQuotaUpdated: /opt/ucloud/scripts/onQuotaUpdated
      onProjectUpdated: /opt/ucloud/scripts/onProjectUpdated
      onUserDisconnected: /opt/ucloud/scripts/onUserDisconnected
      onUsageReporting: /opt/ucloud/scripts/onUsageReporting

Note that the scripts pointed to in the configuration file is your choice. You can point it to any script of your choosing. The script does not have to reside in any specific folder. The script must be readable and executable by the UCloud/IM (Server) user. They do not have to be invoked by any of the user instances.

Script: onQuotaUpdated

The output of the account mapper, explained earlier in this chapter, are provided to the onQuotaUpdated script. This script is invoked every time a relevant update happens to a resource allocation. This includes the first time that the resource allocation is created.

The script is expected to perform the following actions:

• It must ensure that the Slurm account is created
• It should ensure that the Slurm account is capable of submitting jobs with the relevant QoS and partition
• It should ensure that no further usage is allowed if locked = true
• It should ensure that further usage is allowed if locked = false
• It should update billing and fairshare systems appropriately
• It should not attempt to add any users to the Slurm account

{
  /* object */
  "account": {
    /* string */
    "accountName": "",
    /* string */
    "partition": "",
    /* string */
    "qos": "",
    /* string */
    "categoryName": ""
  },
  /* object */
  "quotaUpdate": {
    /* uint64 */
    "combinedQuotaInResourceMinutes": 0,
    /* bool */
    "locked": false
  }
}

{
  "account": {
    "accountName": "donna01_cpu",
    "partition": "cpu",
    "qos": "normal",
    "categoryName": "CPU"
  },
  "quotaUpdate": {
    "combinedQuotaInResourceMinutes": 60000,
    "locked": false
  }
}

Script: onProjectUpdated

The output of the account mapper, explained earlier in this chapter, are provided to the onProjectUpdated script. This information is combined with information about users added to a project and removed from a project.

The script is expected to perform the following actions:

• It must ensure that the users added to the project are added to the corresponding account
• It must ensure that the users removed from the project are removed from the corresponding account
• It should attempt to verify that the project members are correctly synchronized with the account

{
  /* object */
  "account": {
    /* string */
    "accountName": "",
    /* string */
    "partition": "",
    /* string */
    "qos": "",
    /* string */
    "categoryName": ""
  },
  /* object[] */
  "allMembers": [
    {
      /* uint32 */
      "uid": 0,
      /* string */
      "ucloudUsername": "",
      /* "PI" | "ADMIN" | "USER" */
      "role": "USER"
    }
  ],
  /* object[] */
  "membersAddedToProject": [
    {
      /* uint32 */
      "uid": 0,
      /* string */
      "ucloudUsername": "",
      /* "PI" | "ADMIN" | "USER" */
      "role": "USER"
    }
  ],
  /* object[] */
  "membersRemovedFromProject": [
    {
      /* uint32 */
      "uid": 0,
      /* string */
      "ucloudUsername": ""
    }
  ]
}

{
  "account": {
    "accountName": "donna01_cpu",
    "partition": "cpu",
    "qos": "normal",
    "categoryName": "CPU"
  },
  "allMembers": [
    {
      "uid": 41235122,
      "ucloudUsername": "Donna#1234",
      "role": "USER"
    }
  ],
  "membersAddedToProject": [
    {
      "uid": 41235122,
      "ucloudUsername": "Donna#1234",
      "role": "USER"
    }
  ],
  "membersRemovedFromProject": []
}

Script: onUserDisconnected

TODO

Script: onUsageReporting

The onUsageReporting script is invoked periodically. The script should return a list of accounts and their associated usage in “resource”-minutes. Where “resource” matches the accounting unit specified on the machine type’s configuration. For example, if a machine type is configured in GPU-minutes/GPU-hours then it should be returned in GPU-minutes. Similarly, Core-minutes/Core-hours should return Core-minutes. Note that it is always returned in minutes regardless of the accounting unit used.

{
  /* object[] */
  "usage": [
    {
      /* string */
      "account": "",
      /* uint64 */
      "usageInResourceMinutes": 0
    }
  ]
}

{
  "usage": [
    {
      "account": "donna01_cpu",
      "usageInResourceMinutes": 5128
    }
  ]
}