Managing Security in a Lustre File System

This chapter describes security features of the Lustre file system and includes the following sections:

• Using ACLs
• Using Root Squash
• Isolating Clients to a Sub-directory Tree
• Checking SELinux Policy Enforced by Lustre Clients
• Encrypting files and directories
• Configuring Kerberos (KRB) Security

Using ACLs

An access control list (ACL), is a set of data that informs an operating system about permissions or access rights that each user or group has to specific system objects, such as directories or files. Each object has a unique security attribute that identifies users who have access to it. The ACL lists each object and user access privileges such as read, write or execute.

How ACLs Work

Implementing ACLs varies between operating systems. Systems that support the Portable Operating System Interface (POSIX) family of standards share a simple yet powerful file system permission model, which should be well-known to the Linux/UNIX administrator. ACLs add finer-grained permissions to this model, allowing for more complicated permission schemes. For a detailed explanation of ACLs on a Linux operating system, refer to the SUSE Labs article Posix Access Control Lists on Linux.

We have implemented ACLs according to this model. The Lustre software works with the standard Linux ACL tools, setfacl, getfacl, and the historical chacl, normally installed with the ACL package.

Using ACLs with the Lustre Software

POSIX Access Control Lists (ACLs) can be used with the Lustre software. An ACL consists of file entries representing permissions based on standard POSIX file system object permissions that define three classes of user (owner, group and other). Each class is associated with a set of permissions [read (r), write (w) and execute (x)].

• Owner class permissions define access privileges of the file owner.
• Group class permissions define access privileges of the owning group.
• Other class permissions define access privileges of all users not in the owner or group class.

The ls -l command displays the owner, group, and other class permissions in the first column of its output (for example, -rw-r- -- for a regular file with read and write access for the owner class, read access for the group class, and no access for others).

Minimal ACLs have three entries. Extended ACLs have more than the three entries. Extended ACLs also contain a mask entry and may contain any number of named user and named group entries.

To check ACLs on the MDS, check that the acl connect flag is listed (default since Lustre 1.8):

# lctl get_param -n mdc.home-MDT0000-mdc-*.connect_flags | grep acl

ACLs are enabled by default on a Lustre file system, and are controlled on a system-wide basis; either all clients enable ACLs or none do. Activating ACLs is controlled by MDS mount options acl/noacl to enable or disable ACLs, respectively. You do not need to change the client configuration, and the acl string will not appear in the client mount options in /etc/mtab.

If ACLs are not enabled on the MDS, then any attempts to reference an ACL on a client return an Operation not supported error.

Examples

These examples are taken directly from the POSIX paper referenced above. ACLs on a Lustre file system work exactly like ACLs on any Linux file system. They are manipulated with the standard tools in the standard manner. Below, we create a directory and allow a specific user access.

[phil@client lustre]$ umask 027
[phil@client lustre]$ mkdir rain
[phil@client lustre]$ ls -ld rain
drwxr-x---  2 phil dev 4096 Feb 20 06:50 rain
[phil@client lustre]$ getfacl rain
# file: rain
# owner: phil
# group: dev
user::rwx
group::r-x
other::---

[phil@client lustre]$ setfacl -m user:chirag:rwx rain
[phil@client lustre]$ ls -ld rain
drwxrwx---+ 2 phil dev 4096 Feb 20 06:50 rain
[phil@client lustre]$ getfacl --omit-header rain
user::rwx
user:chirag:rwx
group::r-x
mask::rwx
other::---

Using Root Squash

Root squash is a security feature which restricts super-user access rights to a Lustre file system. Without the root squash feature enabled, Lustre file system users on untrusted clients could access or modify files owned by root on the file system, including deleting them. Using the root squash feature restricts file access/modifications as the root user. Note, however, that this does not prevent users from accessing files owned by other users.

The root squash feature works by re-mapping the user ID (UID) and group ID (GID) of the root user to a UID and GID specified by the system administrator. The preferred way to configure root squash is via nodemaps and the admin property. Nodemaps allow root squash on a per-client basis. With UID maps, the clients can even have a local root UID without actually having root access to the filesystem itself.

Please refer to explanations about the admin property in the chapter dedicated to Nodemaps, in Managing the Properties.

Isolating Clients to a Sub-directory Tree

Namespace isolation is the Lustre implementation of the generic concept of multi-tenancy, which aims at providing separated namespaces from a single filesystem. Lustre namespace isolation enables different populations of users on the same file system beyond normal Unix permissions/ACLs, even when users on the clients may have root access. Those tenants share the same file system, but they are isolated from each other, i.e., they cannot access or even see each other’s files, and they are not aware that they are sharing common file system resources.

Lustre namespace isolation leverages the Fileset feature (Fileset Feature) to restrict clients to subdirectories of the filesystem rather than the root directory. To achieve namespace isolation, the subdirectory mount, which presents to tenants only their own fileset, has to be imposed to the clients. To that extent, we make use of the nodemap feature (Mapping UIDs and GIDs with Nodemap). We group all clients used by a tenant under a common nodemap entry, and we assign to this nodemap entry the fileset to which the tenant is restricted.

Identifying Clients

Enforcing multi-tenancy on Lustre relies on the ability to properly identify the client nodes used by a tenant, and trust those identities. This can be achieved by having physical hardware and/or network security, so that client nodes have well-known NIDs. It is also possible to make use of strong authentication with Kerberos or Shared-Secret Key (see Configuring Shared-Secret Key (SSK) Security). Kerberos prevents NID spoofing, as every client needs its own credentials, based on its NID, in order to connect to the servers. Shared-Secret Key also prevents tenant impersonation, because keys can be linked to a specific nodemap. See Role of Nodemap in SSK for detailed explanations.

Configuring Namespace Isolation

Namespace isolation on Lustre can be achieved by setting one fileset property (or multiple filesets with Lustre 2.17 or later) on a nodemap entry. All clients belonging to this nodemap entry will automatically mount this (primary) fileset instead of the root directory. For example:

mgs# lctl nodemap_set_fileset --name tenant1 --fileset '/dir1'

So all clients matching the tenant1 nodemap are presented the fileset /dir1 when mounting. This means these clients are doing an implicit subdirectory mount on the subdirectory /dir1.

To delete the fileset parameter, just set it to an empty string:

mgs# lctl nodemap_set_fileset --name tenant1 --fileset ''

Using lctl nodemap_set_fileset automatically distributes the fileset to all servers and makes isolation permanent. As of Lustre 2.18, lctl nodemap_set_fileset is deprecated in favor of lctl nodemap_fileset_{add,del,modify}. Note that lctl nodemap_set_fileset can only set the primary fileset and cannot set it as read-only.

Multiple Filesets

Previously, nodemaps supported a single fileset per nodemap. As of Lustre 2.17, it is possible to define multiple filesets per nodemap. The previous single fileset is called a primary fileset, offering similar behavior and acting as the default subdirectory when clients (associated with the nodemap) mount the filesystem. Additional alternate filesets can be defined, which offer improved flexibility and allow clients to mount additional subdirectories of the filesystem.

Each nodemap supports a single primary fileset and up to 255 alternate filesets. It is possible to only define alternate filesets without a primary fileset. In this case, any subdirectory must be explicitly set at client mount time. Therefore, alternate filesets never enforce a default subdirectory mountpoint for clients. Each fileset can further be set as read-only, preventing clients from mounting the subdirectory with write access, i.e., the clients implicitly mount the filesystem in mode=ro if the read-only fileset applies to the current mount subdirectory.

Similar to other nodemap commands, multiple fileset commands must be run on the MGS. The nodemap configuration is then asychonously propagated to the other server targets and persisted.

Managing Multiple Filesets

The following commands are used to manage multiple filesets on a nodemap. Please refer to the respective man page of each command for more details and examples. A fileset is added to the nodemap with the lctl nodemap_fileset_add command:

lctl nodemap_fileset_add --name NODEMAP_NAME --fileset SUBDIRECTORY [--alt] [--ro]

• --name specifies the nodemap that this fileset should be associated with (mandatory).
• --fileset specifies the subdirectory to restrict the clients to. The fileset must begin with ‘/’ (mandatory).
• --alt specifies that the fileset is an alternate fileset. If not specified, the fileset is the primary fileset. Multiple primary filesets and duplicate fileset definitions across fileset types are not allowed.
• --ro specifies that the fileset is read-only. If not specified, the fileset is read-write.

A fileset is deleted from the nodemap with the lctl nodemap_fileset_del command:

lctl nodemap_fileset_del --name NODEMAP_NAME [--all] [--fileset SUBDIRECTORY]

• --name specifies the nodemap that one or all fileset should be deleted from (mandatory).
• --all specifies that all filesets should be deleted from the nodemap. This option cannot be used with --fileset.
• --fileset specifies the fileset to delete from the nodemap. This option cannot be used with --all. Either --all or --fileset must be specified.

Filesets can be updated in-place with the lctl nodemap_fileset_modify command:

lctl nodemap_fileset_modify --name NODEMAP_NAME --fileset SUBDIRECTORY \
  [--rename|-r NEW_SUBDIRECTORY] [--primary|-p] [--alt|-a] [--rw] [--ro]

• --name specifies the nodemap containing the fileset to modify (mandatory).
• --fileset specifies the path of the fileset to modify (mandatory).
• --rename changes the fileset’s path to the specified new subdirectory. The new path must begin with a slash (‘/’) and cannot yet exist as a fileset of that type on the same nodemap.
• --primary converts the fileset to a primary fileset. The flag has no effect if the fileset is already primary. Note that --primary can only convert an alternate fileset to primary if there is currently no primary fileset set on the nodemap. This option cannot be used with --alt.
• --alt converts the fileset to an alternate fileset. The flag has no effect if the fileset is already alternate. This option cannot be used with --primary.
• --rw sets the fileset to read-write access mode. The flag has no effect if the fileset is already set to read-write. This option cannot be used with --ro.
• --ro sets the fileset to read-only access mode. The flag has no effect if the fileset is already set to read-only. This option cannot be used with --rw.

How Multiple Filesets affect Client Mounting

Filesets can be used to provide namespace isolation within the Lustre filesystem by explicitly restricting clients of a nodemap to specific subdirectories when mounting the client. The restriction depends on whether the fileset is designated as primary, alternate, or read-only. On mounting the Lustre client, the following rules apply:

1. If the nodemap is inactive or no filesets are defined, no subdirectory restrictions are applied.
2. If the primary fileset is set and no subdirectory is presented when mounting the Lustre client, the primary fileset’s subdirectory is used as the filesystem root directory.
3. If any defined fileset (primary or alternate) matches the presented mounted subdirectory exactly or as a prefix, the subdirectory mount is used as the filesystem root directory.
4. If the fileset matches in 3, the presented mounting subdirectory is appended to the fileset’s subdirectory.

If alternate filesets are defined but no primary fileset is set, only 3 applies.

Checking SELinux Policy Enforced by Lustre Clients

SELinux provides a mechanism in Linux for supporting Mandatory Access Control (MAC) policies. When a MAC policy is enforced, the operating system’s (OS) kernel defines application rights, firewalling applications from compromising the entire system. Regular users do not have the ability to override the policy.

One purpose of SELinux is to protect the OS from privilege escalation. To that extent, SELinux defines confined and unconfined domains for processes and users. Each process, user, file is assigned a security context, and rules define the allowed operations by processes and users on files.

Another purpose of SELinux can be to protect data sensitivity, thanks to Multi-Level Security (MLS). MLS works on top of SELinux, by defining the concept of security levels in addition to domains. Each process, user and file is assigned a security level, and the model states that processes and users can read the same or lower security level, but can only write to their own or higher security level.

From a file system perspective, the security context of files must be stored permanently. Lustre makes use of the security.selinux extended attributes on files to hold this information. Lustre supports SELinux on the client side. All you have to do to have MAC and MLS on Lustre is to enforce the appropriate SELinux policy (as provided by the Linux distribution) on all Lustre clients. No SELinux is required on Lustre servers.

Because Lustre is a distributed file system, the specificity when using MLS is that Lustre really needs to make sure data is always accessed by nodes with the SELinux MLS policy properly enforced. Otherwise, data is not protected. This means Lustre has to check that SELinux is properly enforced on client side, with the right, unaltered policy. And if SELinux is not enforced as expected on a client, the server denies its access to Lustre.

Determining SELinux Policy Info

A string that represents the SELinux Status info will be used by servers as a reference, to check if clients are enforcing SELinux properly. This reference string can be obtained on a client node known to enforce the right SELinux policy, by calling the l_getsepol command line utility:

client# l_getsepol
SELinux status info: 1:mls:31:40afb76d077c441b69af58cccaaa2ca63641ed6e21b0a887dc21a684f508b78f

The string describing the SELinux policy has the following syntax:

mode:name:version:hash

where:

• mode is a digit telling if SELinux is in Permissive mode (0) or Enforcing mode (1)
• name is the name of the SELinux policy
• version is the version of the SELinux policy
• hash is the computed hash of the binary representation of the policy, as exported in /etc/selinux/name/policy/policy. version

Enforcing SELinux Policy Check

SELinux policy check can be enforced by setting the sepol parameter on a nodemap entry. All clients belonging to this nodemap entry must enforce the SELinux policy described by this parameter, otherwise they are denied access to the Lustre file system. For example:

mgs# lctl nodemap_set_sepol --name restricted
     --sepol '1:mls:31:40afb76d077c441b69af58cccaaa2ca63641ed6e21b0a887dc21a684f508b78f'

So all clients matching the restricted nodemap must enforce the SELinux policy which description matches 1:mls:31:40afb76d077c441b69af58cccaaa2ca63641ed6e21b0a887dc21a684f508b78f. If not, they will get Permission Denied when trying to mount or access files on the Lustre file system.

To delete the sepol parameter, just set it to an empty string:

mgs# lctl nodemap_set_sepol --name restricted --sepol ''

See Mapping UIDs and GIDs with Nodemap for more details about the Nodemap feature.

Making SELinux Policy Check Permanent

In order to make SELinux Policy check permanent, the sepol parameter on the nodemap has to be set with lctl set_param with the -P option.

mgs# lctl set_param nodemap.restricted.sepol=1:mls:31:40afb76d077c441b69af58cccaaa2ca63641ed6e21b0a887dc21a684f508b78f
mgs# lctl set_param -P nodemap.restricted.sepol=1:mls:31:40afb76d077c441b69af58cccaaa2ca63641ed6e21b0a887dc21a684f508b78f

This way the sepol parameter will be stored in the Lustre config logs, letting the servers retrieve the information after a restart.

Sending SELinux Status Info from Clients

In order for Lustre clients to send their SELinux status information, in case SELinux is enabled locally, the send_sepol ptlrpc kernel module’s parameter has to be set to a non-zero value. send_sepol accepts various values:

• 0: do not send SELinux policy info;
• -1: fetch SELinux policy info for every request;
• N > 0: only fetch SELinux policy info every N seconds. Use N = 2^31-1 to have SELinux policy info fetched only at mount time.

Clients that are part of a nodemap on which sepol is defined must send SELinux status info. And the SELinux policy they enforce must match the representation stored into the nodemap. Otherwise they will be denied access to the Lustre file system.

Encrypting files and directories

The purpose that client-side encryption wants to serve is to be able to provide a special directory for each user, to safely store sensitive files. The goals are to protect data in transit between clients and servers, and protect data at rest.

This feature is implemented directly at the Lustre client level. Lustre client-side encryption relies on kernel fscrypt. fscrypt is a library which filesystems can hook into to support transparent encryption of files and directories. As a consequence, the key points described below are extracted from fscrypt documentation.

For full details, please refer to documentation available with the Lustre sources, under the Documentation/client_side_encryption directory.

Client-side encryption access semantics

Only Lustre clients need access to encryption master keys. Keys are added to the filesystem-level encryption keyring on the Lustre client.

• With the key With the encryption key, encrypted regular files, directories, and symlinks behave very similarly to their unencrypted counterparts — after all, the encryption is intended to be transparent. However, astute users may notice some differences in behavior:
  • Unencrypted files, or files encrypted with a different encryption policy (i.e. different key, modes, or flags), cannot be renamed or linked into an encrypted directory. However, encrypted files can be renamed within an encrypted directory, or into an unencrypted directory.

    Note

    “moving” an unencrypted file into an encrypted directory, e.g. with the mv program, is implemented in userspace by a copy followed by a delete. Be aware the original unencrypted data may remain recoverable from free space on the disk; it is best to keep all files encrypted from the very beginning.

  • On Lustre, Direct I/O is supported for encrypted files.
  • The fallocate() operations FALLOC_FL_COLLAPSE_RANGE, FALLOC_FL_INSERT_RANGE, and FALLOC_FL_ZERO_RANGE are not supported on encrypted files and will fail with EOPNOTSUPP.
  • DAX (Direct Access) is not supported on encrypted files.
  • The st_size of an encrypted symlink will not necessarily give the length of the symlink target as required by POSIX. It will actually give the length of the ciphertext, which will be slightly longer than the plaintext due to NUL-padding and an extra 2-byte overhead.
  • The maximum length of an encrypted symlink is 2 bytes shorter than the maximum length of an unencrypted symlink.
  • mmap is supported. This is possible because the pagecache for an encrypted file contains the plaintext, not the ciphertext.
• Without the key Some filesystem operations may be performed on encrypted regular files, directories, and symlinks even before their encryption key has been added, or after their encryption key has been removed:
  • File metadata may be read, e.g. using stat().
  • Directories may be listed, in which case the filenames will be listed in an encoded form derived from their ciphertext. The algorithm is subject to change but it is guaranteed that the presented filenames will be no longer than NAME_MAX bytes, will not contain the / or \0 characters, and will uniquely identify directory entries. The . and .. directory entries are special. They are always present and are not encrypted or encoded.
  • Files may be deleted. That is, nondirectory files may be deleted with unlink() as usual, and empty directories may be deleted with rmdir() as usual. Therefore, rm and rm -r will work as expected.
  • Symlink targets may be read and followed, but they will be presented in encrypted form, similar to filenames in directories. Hence, they are unlikely to point to anywhere useful. Without the key, regular files cannot be opened or truncated. Attempts to do so will fail with ENOKEY. This implies that any regular file operations that require a file descriptor, such as read(), write(), mmap(), fallocate(), and ioctl(), are also forbidden. Also without the key, files of any type (including directories) cannot be created or linked into an encrypted directory, nor can a name in an encrypted directory be the source or target of a rename, nor can an O_TMPFILE temporary file be created in an encrypted directory. All such operations will fail with ENOKEY. It is not currently possible to backup and restore encrypted files without the encryption key. This would require special APIs which have not yet been implemented.
• Encryption policy enforcement After an encryption policy has been set on a directory, all regular files, directories, and symbolic links created in that directory (recursively) will inherit that encryption policy. Special files — that is, named pipes, device nodes, and UNIX domain sockets — will not be encrypted. Except for those special files, it is forbidden to have unencrypted files, or files encrypted with a different encryption policy, in an encrypted directory tree.

Client-side encryption key hierarchy

Each encrypted directory tree is protected by a master key.

To “unlock” an encrypted directory tree, userspace must provide the appropriate master key. There can be any number of master keys, each of which protects any number of directory trees on any number of filesystems.

Client-side encryption modes and usage

fscrypt allows one encryption mode to be specified for file contents and one encryption mode to be specified for filenames. Different directory trees are permitted to use different encryption modes. Currently, the following pairs of encryption modes are supported:

• AES-256-XTS for contents and AES-256-CTS-CBC for filenames
• AES-128-CBC for contents and AES-128-CTS-CBC for filenames

If unsure, you should use the (AES-256-XTS, AES-256-CTS-CBC) pair.

mgs# lctl set_param -P llite.*.enable_filename_encryption=1

Client-side encryption threat model

• Offline attacks For the Lustre case, block devices are Lustre targets attached to the Lustre servers. Manipulating the filesystem offline means accessing the filesystem on these targets while Lustre is offline. Provided that a strong encryption key is chosen, fscrypt protects the confidentiality of file contents in the event of a single point-in-time permanent offline compromise of the block device content. Lustre client-side encryption does not protect the confidentiality of metadata, e.g. file names, file sizes, file permissions, file timestamps, and extended attributes. Also, the existence and location of holes (unallocated blocks which logically contain all zeroes) in files is not protected.
• Online attacks
  • On Lustre client After an encryption key has been added, fscrypt does not hide the plaintext file contents or filenames from other users on the same node. Instead, existing access control mechanisms such as file mode bits, POSIX ACLs, LSMs, or namespaces should be used for this purpose. For the Lustre case, it means plaintext file contents or filenames are not hidden from other users on the same Lustre client. An attacker who compromises the system enough to read from arbitrary memory, e.g. by exploiting a kernel security vulnerability, can compromise all encryption keys that are currently in use. However, fscrypt allows encryption keys to be removed from the kernel, which may protect them from later compromise. Key removal can be carried out by non-root users. In more detail, the key removal will wipe the master encryption key from kernel memory. Moreover, it will try to evict all cached inodes which had been “unlocked” using the key, thereby wiping their per-file keys and making them once again appear “locked”, i.e. in ciphertext or encrypted form.
  • On Lustre server An attacker on a Lustre server who compromises the system enough to read arbitrary memory, e.g. by exploiting a kernel security vulnerability, cannot compromise Lustre files content. Indeed, encryption keys are not forwarded to the Lustre servers, and servers do not carry out decryption or encryption. Moreover, bulk RPCs received by servers contain encrypted data, which is written as-is to the underlying filesystem.

Manage encryption on directories

By default, Lustre client-side encryption is enabled, letting users define encryption policies on a per-directory basis.

fscrypt userspace tool can be used to manage encryption policies. See https://github.com/google/fscrypt for comprehensive explanations. Below are examples on how to use this tool with Lustre. If not told otherwise, commands must be run on Lustre client side.

• Two preliminary steps are required before actually deciding which directories to encrypt, and this is the only functionality which requires root privileges. Administrator has to run:

  # fscrypt setup
  Customizing passphrase hashing difficulty for this system...
  Created global config file at "/etc/fscrypt.conf".
  Metadata directories created at "/.fscrypt".

  This first command has to be run on all clients that want to use encryption, as it sets up global fscrypt parameters outside of Lustre.

  # fscrypt setup /mnt/lustre
  Metadata directories created at "/mnt/lustre/.fscrypt"

  This second command has to be run on just one Lustre client.

  Note

  The file /etc/fscrypt.conf can be edited. It is strongly recommended to set policy_version to 2, so that fscrypt wipes files from memory when the encryption key is removed.

• Now a regular user is able to select a directory to encrypt:

  $ fscrypt encrypt /mnt/lustre/vault
  The following protector sources are available:
  1 - Your login passphrase (pam_passphrase)
  2 - A custom passphrase (custom_passphrase)
  3 - A raw 256-bit key (raw_key)
  Enter the source number for the new protector [2 - custom_passphrase]: 2
  Enter a name for the new protector: shield
  Enter custom passphrase for protector "shield":
  Confirm passphrase:
  "/mnt/lustre/vault" is now encrypted, unlocked, and ready for use.

  Starting from here, all files and directories created under /mnt/lustre/vault will be encrypted, according to the policy defined at the previsous step.

  Note

  The encryption policy is inherited by all subdirectories. It is not possible to change the policy for a subdirectory.

• Another user can decide to encrypt a different directory with its own protector:

  $ fscrypt encrypt /mnt/lustre/private
  Should we create a new protector? [y/N] Y
  The following protector sources are available:
  1 - Your login passphrase (pam_passphrase)
  2 - A custom passphrase (custom_passphrase)
  3 - A raw 256-bit key (raw_key)
  Enter the source number for the new protector [2 - custom_passphrase]: 2
  Enter a name for the new protector: armor
  Enter custom passphrase for protector "armor":
  Confirm passphrase:
  "/mnt/lustre/private" is now encrypted, unlocked, and ready for use.

• Users can decide to lock an encrypted directory at any time:

  $ fscrypt lock /mnt/lustre/vault
  "/mnt/lustre/vault" is now locked.

  This action prevents access to encrypted content, and by removing the key from memory, it also wipes files from memory if they are not still open.
• Users regain access to the encrypted directory with the command:

  $ fscrypt unlock /mnt/lustre/vault
  Enter custom passphrase for protector "shield":
  "/mnt/lustre/vault" is now unlocked and ready for use.

• Actually, fscrypt does not give direct access to master keys, but to protectors that are used to encrypt them. This mechanism gives the ability to change a passphrase:

  $ fscrypt status /mnt/lustre
  lustre filesystem "/mnt/lustre" has 2 protectors and 2 policies
  
  PROTECTOR         LINKED  DESCRIPTION
  deacab807bf0e788  No      custom protector "shield"
  e691ae7a1990fc2a  No      custom protector "armor"
  
  POLICY                            UNLOCKED  PROTECTORS
  52b2b5aff0e59d8e0d58f962e715862e  No        deacab807bf0e788
  374e8944e4294b527e50363d86fc9411  No        e691ae7a1990fc2a
  
  $ fscrypt metadata change-passphrase --protector=/mnt/lustre:deacab807bf0e788
  Enter old custom passphrase for protector "shield":
  Enter new custom passphrase for protector "shield":
  Confirm passphrase:
  Passphrase for protector deacab807bf0e788 successfully changed.

  It makes also possible to have multiple protectors for the same policy. This is really useful when several users share an encrypted directory, because it avoids the need to share any secret between them.

  $ fscrypt status /mnt/lustre/vault
  "/mnt/lustre/vault" is encrypted with fscrypt.
  
  Policy:   52b2b5aff0e59d8e0d58f962e715862e
  Options:  padding:32 contents:AES_256_XTS filenames:AES_256_CTS policy_version:2
  Unlocked: No
  
  Protected with 1 protector:
  PROTECTOR         LINKED  DESCRIPTION
  deacab807bf0e788  No      custom protector "shield"
  
  $ fscrypt metadata create protector /mnt/lustre
  Create new protector on "/mnt/lustre" [Y/n] Y
  The following protector sources are available:
  1 - Your login passphrase (pam_passphrase)
  2 - A custom passphrase (custom_passphrase)
  3 - A raw 256-bit key (raw_key)
  Enter the source number for the new protector [2 - custom_passphrase]: 2
  Enter a name for the new protector: bunker
  Enter custom passphrase for protector "bunker":
  Confirm passphrase:
  Protector f3cc1b5cf9b8f41c created on filesystem "/mnt/lustre".
  
  $ fscrypt metadata add-protector-to-policy
            --protector=/mnt/lustre:f3cc1b5cf9b8f41c
            --policy=/mnt/lustre:52b2b5aff0e59d8e0d58f962e715862e
  WARNING: All files using this policy will be accessible with this protector!!
  Protect policy 52b2b5aff0e59d8e0d58f962e715862e with protector f3cc1b5cf9b8f41c? [Y/n] Y
  Enter custom passphrase for protector "bunker":
  Enter custom passphrase for protector "shield":
  Protector f3cc1b5cf9b8f41c now protecting policy 52b2b5aff0e59d8e0d58f962e715862e.
  
  $ fscrypt status /mnt/lustre/vault
  "/mnt/lustre/vault" is encrypted with fscrypt.
  
  Policy:   52b2b5aff0e59d8e0d58f962e715862e
  Options:  padding:32 contents:AES_256_XTS filenames:AES_256_CTS policy_version:2
  Unlocked: No
  
  Protected with 2 protectors:
  PROTECTOR         LINKED  DESCRIPTION
  deacab807bf0e788  No      custom protector "shield"
  f3cc1b5cf9b8f41c  No      custom protector "bunker"

Configuring Kerberos (KRB) Security

This chapter describes how to use Kerberos with Lustre.

What Is Kerberos?

Kerberos is a mechanism for authenticating all entities (such as users and servers) on an “unsafe” network. Each of these entities, known as “principals”, negotiate a runtime key with the Kerberos server. This key enables principals to verify that messages from the Kerberos server are authentic. By trusting the Kerberos server, users and services can authenticate one another.

Setting up Lustre with Kerberos can provide advanced security protections for the Lustre network. Broadly, Kerberos offers three types of benefit:

• Allows Lustre connection peers (MDS, OSS and clients) to authenticate one another.
• Protects the integrity of PTLRPC messages from being modified during network transfer.
• Protects the privacy of the PTLRPC message from being eavesdropped during network transfer.

Kerberos uses the “kernel keyring” client upcall mechanism.

Security Flavor

A security flavor is a string to describe what kind authentication and data transformation be performed upon a PTLRPC connection. It covers both RPC message and BULK data.

The supported flavors are described in following table:

Kerberos Setup

Distribution

We only support MIT Kerberos 5, from version 1.3.

For environmental requirements in general, and clock synchronization in particular, please refer to section Environmental Requirements.

Principals Configuration

• Configure client nodes:
  • For each client node, create a lustre_root principal and generate keytab.

    kadmin> addprinc -randkey lustre_root/client_host.domain@REALM

    kadmin> ktadd lustre_root/client_host.domain@REALM

  • Install the keytab on the client node.
• Configure MGS nodes:
  • For each MGS node, create a lustre_mgs principal and generate keytab.

    kadmin> addprinc -randkey lustre_mgs/mgs_host.domain@REALM

    kadmin> ktadd lustre_mds/mgs_host.domain@REALM

  • Install the keytab on the MGS nodes.
• Configure MDS nodes:
  • For each MDS node, create a lustre_mds principal and generate keytab.

    kadmin> addprinc -randkey lustre_mds/mds_host.domain@REALM

    kadmin> ktadd lustre_mds/mds_host.domain@REALM

  • Install the keytab on the MDS nodes.
• Configure OSS nodes:
  • For each OSS node, create a lustre_oss principal and generate keytab.

    kadmin> addprinc -randkey lustre_oss/oss_host.domain@REALM

    kadmin> ktadd lustre_oss/oss_host.domain@REALM

  • Install the keytab on the client node.

Networking

On networks for which name resolution to IP address is possible, like TCP or InfiniBand, the names used in the principals must be the ones that resolve to the IP addresses used by the Lustre NIDs.

If you are using a network which is NOT TCP or InfiniBand (e.g. PTL4LND), you need to have a /etc/lustre/nid2hostname script on each node, which purpose is to translate NID into hostname. Following is a possible example for PTL4LND:

#!/bin/bash
set -x

# convert a NID for a LND to a hostname

# called with thre arguments: lnd netid nid
#   $lnd is the string "PTL4LND", etc.
#   $netid is the network identifier in hex string format
#   $nid is the NID in hex format
# output the corresponding hostname,
# or error message leaded by a '@' for error logging.

lnd=$1
netid=$2
# convert hex NID number to decimal
nid=$((0x$3))

case $lnd in
    PTL4LND)   # simply add 'node' at the beginning
        echo "node$nid"
        ;;
    *)
	echo "@unknown LND: $lnd"
        ;;
esac

Required packages

Every node should have following packages installed:

• krb5-workstation
• krb5-libs
• keyutils
• keyutils-libs

On the node used to build Lustre with GSS support, following packages should be installed:

• krb5-devel
• keyutils-libs-devel

Build Lustre

Enable GSS at configuration time:

./configure --enable-gss --other-options

Running

GSS Daemons

Make sure to start the daemon process lsvcgssd on each server node (MGS, MDS and OSS) before starting Lustre. The command syntax is:

lsvcgssd [-f] [-v] [-g] [-m] [-o] -k

• -f: run in foreground, instead of as daemon
• -v: increase verbosity by 1. For example, to set the verbose level to 3, run ‘lsvcgssd -vvv’. Verbose logging can help you make sure Kerberos is set up correctly.
• -g: service MGS
• -m: service MDS
• -o: service OSS
• -k: enable kerberos support

Setting Security Flavors

Security flavors can be set by defining sptlrpc rules on the MGS. These rules are persistent, and are in the form: <spec>=<flavor>

• To add a rule:

  mgs> lctl conf_param <spec>=<flavor>

  If there is an existing rule on , it will be overwritten.
• To delete a rule:

  mgs> lctl conf_param -d <spec>

• To list existing rules:

  msg> lctl get_param mgs.MGS.live.<fs-name> | grep "srpc.flavor"

Rules Syntax & Examples

The general syntax is: <target>.srpc.flavor.<network>[.<direction>]=flavor

• <target> can be filesystem name, or specific MDT/OST device name. For example testfs, testfs-MDT0000, testfs-OST0001.
• <network> is the LNet network name, for example tcp0, o2ib0, or default to not filter on LNet network.
• <direction> can be one of cli2mdt, cli2ost, mdt2mdt, mdt2ost. Direction is optional.

Examples:

• Apply krb5i on ALL connections for file system testfs:

mgs> lctl conf_param testfs.srpc.flavor.default=krb5i

• Nodes in network tcp0 use krb5p; all other nodes use null.

mgs> lctl conf_param testfs.srpc.flavor.tcp0=krb5p
mgs> lctl conf_param testfs.srpc.flavor.default=null

• Nodes in network tcp0 use krb5p; nodes in o2ib0 use krb5n; among other nodes, clients use krb5i to MDT/OST, MDTs use null to other MDTs, MDTs use krb5a to OSTs.

mgs> lctl conf_param testfs.srpc.flavor.tcp0=krb5p
mgs> lctl conf_param testfs.srpc.flavor.o2ib0=krb5n
mgs> lctl conf_param testfs.srpc.flavor.default.cli2mdt=krb5i
mgs> lctl conf_param testfs.srpc.flavor.default.cli2ost=krb5i
mgs> lctl conf_param testfs.srpc.flavor.default.mdt2mdt=null
mgs> lctl conf_param testfs.srpc.flavor.default.mdt2ost=krb5a

Regular Users Authentication

On client nodes, non-root users need to issue kinit before accessing Lustre, just like other Kerberized applications.

• Required by kerberos, the user’s principal (username@REALM) should be added to the KDC.
• Client and MDT nodes should have the same user database used for name and uid/gid translation.

Regular users can destroy the established security contexts before logging out, by issuing:

lfs flushctx -k -r <mount point>

Here -k is to destroy the on-disk Kerberos credential cache, similar to kdestroy, and -r is to reap the revoked keys from the keyring when flushing the GSS context. Otherwise it only destroys established contexts in kernel memory.

Secure MGS connection

Each node can specify which flavor to use to connect to the MGS, by using the mgssec=flavor mount option. Once a flavor is chosen, it cannot be changed until re-mount.

Because a Lustre node only has one connection to the MGS, if there is more than one target or client on the node, they necessarily use the same security flavor to the MGS, being the one enforced when the first connection to the MGS was established.

By default, the MGS accepts RPCs with any flavor. But it is possible to configure the MGS to only accept a given flavor. The syntax is identical to what is explained in paragraph Rules Syntax & Examples, but with special target _mgs:

mgs> lctl conf_param _mgs.srpc.flavor.<network>=<flavor>