Lustre Operations

Once you have the Lustre file system up and running, you can use the procedures in this section to perform these basic Lustre administration tasks.

Mounting by Label

The file system name is limited to 8 characters. We have encoded the file system and target information in the disk label, so you can mount by label. This allows system administrators to move disks around without worrying about issues such as SCSI disk reordering or getting the /dev/device wrong for a shared target. Soon, file system naming will be made as fail-safe as possible. Currently, Linux disk labels are limited to 16 characters. To identify the target within the file system, 8 characters are reserved, leaving 8 characters for the file system name:

fsname-MDT0000 or
fsname-OST0a19

To mount by label, use this command:

mount -t lustre -L file_system_label /mount_point

This is an example of mount-by-label:

mds# mount -t lustre -L testfs-MDT0000 /mnt/mdt

Although the file system name is internally limited to 8 characters, you can mount the clients at any mount point, so file system users are not subjected to short names. Here is an example:

client# mount -t lustre mds0@tcp0:/short /dev/long_mountpoint_name

Starting Lustre

On the first start of a Lustre file system, the components must be started in the following order:

1. Mount the MGT.

   Note

   If a combined MGT/MDT is present, Lustre will correctly mount the MGT and MDT automatically.

2. Mount the MDT.

   Note

   Mount all MDTs if multiple MDTs are present.

3. Mount the OST(s).
4. Mount the client(s).

Mounting a Server

Starting a Lustre server is straightforward and only involves the mount command. Lustre servers can be added to /etc/fstab:

mount -t lustre

The mount command generates output similar to this:

/dev/sda1 on /mnt/test/mdt type lustre (rw)
/dev/sda2 on /mnt/test/ost0 type lustre (rw)
192.168.0.21@tcp:/testfs on /mnt/testfs type lustre (rw)

In this example, the MDT, an OST (ost0) and file system (testfs) are mounted.

LABEL=testfs-MDT0000 /mnt/test/mdt lustre defaults,_netdev,noauto 0 0
LABEL=testfs-OST0000 /mnt/test/ost0 lustre defaults,_netdev,noauto 0 0

In general, it is wise to specify noauto and let your high-availability (HA) package manage when to mount the device. If you are not using failover, make sure that networking has been started before mounting a Lustre server. If you are running Red Hat Enterprise Linux, SUSE Linux Enterprise Server, Debian operating system (and perhaps others), use the _netdev flag to ensure that these disks are mounted after the network is up, unless you are using systemd 232 or greater, which recognize lustre as a network filesystem. If you are using lnet.service, use x-systemd.requires=lnet.service regardless of systemd version.

We are mounting by disk label here. The label of a device can be read with e2label. The label of a newly-formatted Lustre server may end in FFFF if the --index option is not specified to mkfs.lustre, meaning that it has yet to be assigned. The assignment takes place when the server is first started, and the disk label is updated. It is recommended that the --index option always be used, which will also ensure that the label is set at format time.

Stopping the Filesystem

A complete Lustre filesystem shutdown occurs by unmounting all clients and servers in the order shown below. Please note that unmounting a block device causes the Lustre software to be shut down on that node.

1. Unmount the clients On each client node, unmount the filesystem on that client using the umount command: umount -a -t lustre The example below shows the unmount of the testfs filesystem on a client node:

[root@client1 ~]# mount -t lustre
XXX.XXX.0.11@tcp:/testfs on /mnt/testfs type lustre (rw,lazystatfs)

[root@client1 ~]# umount -a -t lustre
[154523.177714] Lustre: Unmounted testfs-client

2. Unmount the MDT and MGT On the MGS and MDS node(s), run the umount command: umount -a -t lustre The example below shows the unmount of the MDT and MGT for the testfs filesystem on a combined MGS/MDS:

[root@mds1 ~]# mount -t lustre
/dev/sda on /mnt/mgt type lustre (ro)
/dev/sdb on /mnt/mdt type lustre (ro)

[root@mds1 ~]# umount -a -t lustre
[155263.566230] Lustre: Failing over testfs-MDT0000
[155263.775355] Lustre: server umount testfs-MDT0000 complete
[155269.843862] Lustre: server umount MGS complete

For a seperate MGS and MDS, the same command is used, first on the MDS and then followed by the MGS. 3. Unmount all the OSTs On each OSS node, use the umount command: umount -a -t lustre The example below shows the unmount of all OSTs for the testfs filesystem on server OSS1:

[root@oss1 ~]# mount |grep lustre
/dev/sda on /mnt/ost0 type lustre (ro)
/dev/sdb on /mnt/ost1 type lustre (ro)
/dev/sdc on /mnt/ost2 type lustre (ro)

[root@oss1 ~]# umount -a -t lustre
Lustre: Failing over testfs-OST0002
Lustre: server umount testfs-OST0002 complete

For unmount command syntax for a single OST, MDT, or MGT target please refer to Unmounting a Specific Target on a Server

Unmounting a Specific Target on a Server

To stop a Lustre OST, MDT, or MGT , use the umount */mount_point* command.

The example below stops an OST, ost0, on mount point /mnt/ost0 for the testfs filesystem:

[root@oss1 ~]# umount /mnt/ost0
Lustre: Failing over testfs-OST0000
Lustre: server umount testfs-OST0000 complete

Gracefully stopping a server with the umount command preserves the state of the connected clients. The next time the server is started, it waits for clients to reconnect, and then goes through the recovery procedure.

If the force ( -f) flag is used, then the server evicts all clients and stops WITHOUT recovery. Upon restart, the server does not wait for recovery. Any currently connected clients receive I/O errors until they reconnect.

Specifying Failout/Failover Mode for OSTs

In a Lustre file system, an OST that has become unreachable because it fails, is taken off the network, or is unmounted can be handled in one of two ways:

• In failout mode, Lustre clients immediately receive errors (EIOs) after a timeout, instead of waiting for the OST to recover.
• In failover mode, Lustre clients wait for the OST to recover.

By default, the Lustre file system uses failover mode for OSTs. To specify failout mode instead, use the --param="failover.mode=failout" option as shown below (entered on one line):

oss# mkfs.lustre --fsname=fsname --mgsnode=mgs_NID \
        --param=failover.mode=failout --ost --index=ost_index /dev/ost_block_device

In the example below, failout mode is specified for the OSTs on the MGS mds0 in the file system testfs(entered on one line).

oss# mkfs.lustre --fsname=testfs --mgsnode=mds0 --param=failover.mode=failout \
      --ost --index=3 /dev/sdb

# tunefs.lustre --param failover.mode=failout /dev/ost_device

Handling Degraded OST RAID Arrays

Lustre includes functionality that notifies Lustre if an external RAID array has degraded performance (resulting in reduced overall file system performance), either because a disk has failed and not been replaced, or because a disk was replaced and is undergoing a rebuild. To avoid a global performance slowdown due to a degraded OST, the MDS can avoid the OST for new object allocation if it is notified of the degraded state.

A parameter for each OST, called degraded, specifies whether the OST is running in degraded mode or not.

To mark the OST as degraded, use:

oss# lctl set_param obdfilter.{OST_name}.degraded=1

To mark that the OST is back in normal operation, use:

oss# lctl set_param obdfilter.{OST_name}.degraded=0

To determine if OSTs are currently in degraded mode, use:

oss# lctl get_param obdfilter.*.degraded

If the OST is remounted due to a reboot or other condition, the flag resets to 0.

It is recommended that this be implemented by an automated script that monitors the status of individual RAID devices, such as MD-RAID’s mdadm(8) command with the --monitor option to mark an affected device degraded or restored.

Running Multiple Lustre File Systems

Lustre supports multiple file systems provided the combination of NID:fsname is unique. Each file system must be allocated a unique name during creation with the --fsname parameter. Unique names for file systems are enforced if a single MGS is present. If multiple MGSs are present (for example if you have an MGS on every MDS) the administrator is responsible for ensuring file system names are unique. A single MGS and unique file system names provides a single point of administration and allows commands to be issued against the file system even if it is not mounted.

Lustre supports multiple file systems on a single MGS. With a single MGS fsnames are guaranteed to be unique. Lustre also allows multiple MGSs to co-exist. For example, multiple MGSs will be necessary if multiple file systems on different Lustre software versions are to be concurrently available. With multiple MGSs additional care must be taken to ensure file system names are unique. Each file system should have a unique fsname among all systems that may interoperate in the future.

By default, the mkfs.lustre command creates a file system named lustre. To specify a different file system name (limited to 8 characters) at format time, use the --fsname option:

oss# mkfs.lustre --fsname=file_system_name

To mount a client on the file system, run:

client# mount -t lustre mgsnode:/new_fsname /mount_point

For example, to mount a client on file system foo at mount point /mnt/foo, run:

client# mount -t lustre mgsnode:/foo /mnt/foo

A Lustre installation with two file systems ( foo and bar) could look like this, where the MGS node is mgsnode@tcp0 and the mount points are /mnt/foo and /mnt/bar.

mgsnode# mkfs.lustre --mgs /dev/sda
mdtfoonode# mkfs.lustre --fsname=foo --mgsnode=mgsnode@tcp0 --mdt --index=0
/dev/sdb
ossfoonode# mkfs.lustre --fsname=foo --mgsnode=mgsnode@tcp0 --ost --index=0
/dev/sda
ossfoonode# mkfs.lustre --fsname=foo --mgsnode=mgsnode@tcp0 --ost --index=1
/dev/sdb
mdtbarnode# mkfs.lustre --fsname=bar --mgsnode=mgsnode@tcp0 --mdt --index=0
/dev/sda
ossbarnode# mkfs.lustre --fsname=bar --mgsnode=mgsnode@tcp0 --ost --index=0
/dev/sdc
ossbarnode# mkfs.lustre --fsname=bar --mgsnode=mgsnode@tcp0 --ost --index=1
/dev/sdd

To mount a client on file system foo at mount point /mnt/foo, run:

client# mount -t lustre mgsnode@tcp0:/foo /mnt/foo

To mount a client on file system bar at mount point /mnt/bar, run:

client# mount -t lustre mgsnode@tcp0:/bar /mnt/bar

Creating a sub-directory on a specific MDT

It is possible to create individual directories, along with its files and sub-directories, to be stored on specific MDTs. To create a sub-directory on a given MDT use the command:

client$ lfs mkdir -i mdt_index /mount_point/remote_dir

This command will allocate the sub-directory remote_dir onto the MDT with index mdt_index. For more information on adding additional MDTs and mdt_index see addmdtindex.

mgs# lctl set_param -P mdt.fsname-MDT*.enable_remote_dir=1

mgs# lctl set_param -P mdt.scratch-*.enable_remote_dir=1

mds# lctl get_param mdt.*.enable_remote_dir

With Lustre software version 2.8, a new tunable is available to allow users with a specific group ID to create and delete remote and striped directories. This tunable is enable_remote_dir_gid. For example, setting this parameter to the ‘wheel’ or ‘admin’ group ID allows users with that GID to create and delete remote and striped directories. Setting this parameter to -1 on MDT0000 to permanently allow any non-root users create and delete remote and striped directories. On the MGS execute the following command:

mgs# lctl set_param -P mdt.fsname-*.enable_remote_dir_gid=-1

For the Lustre filesystem ‘scratch’, the commands expands to:

mgs# lctl set_param -P mdt.scratch-*.enable_remote_dir_gid=-1

The change can be verified by executing the following command on every MDS:

mds# lctl get_param mdt.*.enable_remote_dir_gid

Creating a directory striped across multiple MDTs

The Lustre 2.8 DNE feature enables files in a single large directory to be distributed across multiple MDTs (a striped directory), if there are mutliple MDTs added to the filesystem, see Adding a New MDT to a Lustre File System. The result is that metadata requests for files in a single large striped directory are serviced by multiple MDTs and metadata service load is distributed over all the MDTs that service a given directory. By distributing metadata service load over multiple MDTs, performance of very large directories can be improved beyond the limit of one MDT. Normally, all files in a directory must be created on a single MDT.

This command to stripe a directory over mdt_count MDTs is:

client$ lfs mkdir -c mdt_count /mount_point/new_directory

The striped directory feature is most useful for distributing a single large directory (50k entries or more) across multiple MDTs. This should be used with discretion since creating and removing striped directories incurs more overhead than non-striped directories.

Directory creation by space/inode usage

If the starting MDT is not specified when creating a new directory, this directory and its stripes will be distributed on MDTs by space usage. For example the following will create a new directory on an MDT preferring one that has less space usage:

client$ lfs mkdir -c 1 -i -1 dir1

Alternatively, if a default directory stripe is set on a directory, the subsequent use of mkdir for subdirectories in dir1 will have the same effect:

client$ lfs setdirstripe -D -c 1 -i -1 dir1

The policy is:

• If free inodes/blocks on all MDT are almost the same, i.e. max_inodes_avail * 84% < min_inodes_avail and max_blocks_avail * 84% < min_blocks_avail, then choose MDT roundrobin.
• Otherwise, create more subdirectories on MDTs with more free inodes/blocks.

Sometime there are many MDTs. But it is not always desirable to stripe a directory across all MDTs, even if the directory default stripe_count=-1 (unlimited). In this case, the per-filesystem tunable parameter lod.*.max_mdt_stripecount can be used to limit the actual stripe count of directory to fewer than the full MDT count. If lod.*.max_mdt_stripecount is not 0, and the directory stripe_count=-1, the real directory stripe count will be the minimum of the number of MDTs and max_mdt_stripecount. If lod.*.max_mdt_stripecount=0, or an explicit stripe count is given for the directory, it is ignored.

To set max_mdt_stripecount, on all MDSes of file system, run:

mgs# lctl set_param -P lod.$fsname-MDTxxxx-mdtlov.max_mdt_stripecount=<N>

To check max_mdt_stripecount, run:

mds# lctl get_param lod.$fsname-MDTxxxx-mdtlov.max_mdt_stripecount

To reset max_mdt_stripecount, run:

mgs# lctl set_param -P -d lod.$fsname-MDTxxxx-mdtlov.max_mdt_stripecount

Filesystem-wide default directory striping

Similar to file objects allocation, the directory objects are allocated on MDTs by a round-robin algorithm or a weighted algorithm. For the top three level of directories from the root of the filesystem, if the amount of free inodes and blocks is well balanced (i.e., by default, when the free inodes and blocks across MDTs differ by less than 5%), the round-robin algorithm is used to select the next MDT on which a directory is to be created.

If the directory is more than three levels below the root directory, or MDTs are not balanced, then the weighted algorithm is used to randomly select an MDT with more free inodes and blocks.

To avoid creating unnecessary remote directories, if the MDT where its parent directory is located is not too full (the free inodes and blocks of the parent MDT is not more than 5% full than average of all MDTs), this directory will be created on parent MDT.

If administrator wants to change this default filesystem-wide directory striping, run the following command to limit this striping to the top level below the root directory:

client$ lfs setdirstripe -D -i -1 -c 1 --max-inherit 0 <mountpoint>

To revert to the pre-2.15 behavior of all directories being created only on MDT0000 by default (deleting this striping won’t work because it will be recreated if missing):

client$ lfs setdirstripe -D -i 0 -c 1 --max-inherit 0 <mountpoint>

Default Dir Stripe Policy

If default dir stripe policy is set to a directory, it will be applied to sub directories created later. For example:

$ mkdir testdir1
$ lfs setdirstripe testdir1 -D -c 2
$ lfs getdirstripe testdir1 -D
lmv_stripe_count: 2 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 3 lmv_max_inherit_rr: 0
$ mkdir dir1/subdir1
$ lfs getdirstripe testdir1/subdir1
lmv_stripe_count: 2 lmv_stripe_offset: 0 lmv_hash_type: crush
mdtidx       FID[seq:oid:ver]
     0       [0x200000400:0x2:0x0]
     1       [0x240000401:0x2:0x0]

Default dir stripe can be inherited by sub directory. This behavior is controlled by lmv_max_inherit parameter. If lmv_max_inherit is 0 or 1, sub directory stops to inherit default dir stripe policy. Or sub directory decreases its parent’s lmv_max_inherit and uses it as its own lmv_max_inherit. -1 is special because it means unlimited. For example:

$ lfs getdirstripe testdir1/subdir1 -D
lmv_stripe_count: 2 lmv_stripe_offset: -1 lmv_hash_type: none lmv_max_inherit: 2 lmv_max_inherit_rr: 0

lmv_max_inherit can be set explicitly with --max-inherit option in lfs setdirstripe -D command. If the max-inherit value is not specified, the default value is -1 when stripe_count is 0 or 1. For other values of stripe_count, the default value is 3.

Setting and Retrieving Lustre Parameters

Several options are available for setting parameters in Lustre:

• When creating a file system, use mkfs.lustre. See Setting Tunable Parameters with mkfs.lustrebelow.
• When a server is stopped, use tunefs.lustre. See Setting Parameters with tunefs.lustrebelow.
• When the file system is running, use lctl to set or retrieve Lustre parameters. See Setting Parameters with lctland Reporting Current Parameter Valuesbelow.

Setting Tunable Parameters with mkfs.lustre

When the file system is first formatted, parameters can simply be added as a --param option to the mkfs.lustre command. For example:

mds# mkfs.lustre --mdt --param="sys.timeout=50" /dev/sda

For more details about creating a file system,see Configuring a Lustre File System. For more details about mkfs.lustre, see System Configuration Utilities.

Setting Parameters with tunefs.lustre

If a server (OSS or MDS) is stopped, parameters can be added to an existing file system using the --param option to the tunefs.lustre command. For example:

oss# tunefs.lustre --param=failover.node=192.168.0.13@tcp0 /dev/sda

With tunefs.lustre, parameters are additive– new parameters are specified in addition to old parameters, they do not replace them. To erase all old tunefs.lustre parameters and just use newly-specified parameters, run:

mds# tunefs.lustre --erase-params --param=new_parameters

The tunefs.lustre command can be used to set any parameter settable via lctl conf_param and that has its own OBD device, so it can be specified as *obdname|fsname*. *obdtype*. *proc_file_name*= *value*. For example:

mds# tunefs.lustre --param mdt.identity_upcall=NONE /dev/sda1

For more details about tunefs.lustre, see System Configuration Utilities.

Setting Parameters with lctl

When the file system is running, the lctl command can be used to set parameters (temporary or permanent) and report current parameter values. Temporary parameters are active as long as the server or client is not shut down. Permanent parameters live through server and client reboots.

For more details about the lctl command, see the examples in the sections below and System Configuration Utilities.

Setting Temporary Parameters

Use lctl set_param to set temporary parameters on the node where it is run. These parameters internally map to corresponding items in the kernel /proc/{fs,sys}/{lnet,lustre} and /sys/{fs,kernel/debug}/lustre virtual filesystems. However, since the mapping between a particular parameter name and the underlying virtual pathname may change, it is not recommended to access the virtual pathname directly. The lctl set_param command uses this syntax:

# lctl set_param [-n] [-P] obdtype.obdname.proc_file_name=value

For example:

# lctl set_param osc.*.max_dirty_mb=1024
osc.myth-OST0000-osc.max_dirty_mb=32
osc.myth-OST0001-osc.max_dirty_mb=32
osc.myth-OST0002-osc.max_dirty_mb=32
osc.myth-OST0003-osc.max_dirty_mb=32
osc.myth-OST0004-osc.max_dirty_mb=32

Setting Permanent Parameters

Use lctl set_param -P or lctl conf_param command to set permanent parameters. In general, the set_param -P command is preferred for new parameters, as this isolates the parameter settings from the MDT and OST device configuration, and is consistent with the common lctl get_param and lctl set_param commands. The lctl conf_param command was previously used to specify settable parameter, with the following syntax (the same as the mkfs.lustre and tunefs.lustre commands):

obdname|fsname.obdtype.proc_file_name=value)

Here are a few examples of lctl conf_param commands:

mgs# lctl conf_param testfs-MDT0000.sys.timeout=40
mgs# lctl conf_param testfs-MDT0000.mdt.identity_upcall=NONE
mgs# lctl conf_param testfs.llite.max_read_ahead_mb=16
mgs# lctl conf_param testfs-OST0000.osc.max_dirty_mb=29.15
mgs# lctl conf_param testfs-OST0000.ost.client_cache_seconds=15
mgs# lctl conf_param testfs.sys.timeout=40

Setting Permanent Parameters with lctl set_param -P

The lctl set_param -P command can also set parameters permanently using the same syntax as lctl set_param and lctl get_param commands. Permanent parameter settings must be issued on the MGS. The given parameter is set on every host using lctl upcall. The lctl set_param command uses the following syntax:

lctl set_param -P obdtype.obdname.proc_file_name=value

For example:

mgs# lctl set_param -P timeout=40
mgs# lctl set_param -P mdt.testfs-MDT*.identity_upcall=NONE
mgs# lctl set_param -P llite.testfs-*.max_read_ahead_mb=16
mgs# lctl set_param -P osc.testfs-OST*.max_dirty_mb=29.15
mgs# lctl set_param -P ost.testfs-OST*.client_cache_seconds=15

Use the -P -d option to delete permanent parameters. Syntax:

lctl set_param -P -d obdtype.obdname.parameter_name

For example:

mgs# lctl set_param -P -d osc.*.max_dirty_mb

Listing Persistent Parameters

To list tunable parameters stored in the params log file by lctl set_param -P and applied to nodes at mount, run the lctl --device MGS llog_print params command on the MGS. For example:

mgs# lctl --device MGS llog_print params
- { index: 2, event: set_param, device: general, parameter: osc.*.max_dirty_mb, value: 1024 }

Listing All Tunable Parameters

To list Lustre or LNet parameters that are available to set, use the lctl list_param command. For example:

lctl list_param [-FR] obdtype.obdname

The following arguments are available for the lctl list_param command.

-F Add ’ /‘, ’ @’ or ’ =’ for directories, symlinks and writeable files, respectively

-R Recursively lists all parameters under the specified path

For example:

oss# lctl list_param obdfilter.lustre-OST0000

Reporting Current Parameter Values

To report current Lustre parameter values, use the lctl get_param command with this syntax:

lctl get_param [-n] obdtype.obdname.proc_file_name

This example reports data on RPC service times.

oss# lctl get_param -n ost.*.ost_io.timeouts
service : cur 1 worst 30 (at 1257150393, 85d23h58m54s ago) 1 1 1 1

This example reports the amount of space this client has reserved for writeback cache with each OST:

client# lctl get_param osc.*.cur_grant_bytes
osc.myth-OST0000-osc-ffff8800376bdc00.cur_grant_bytes=2097152
osc.myth-OST0001-osc-ffff8800376bdc00.cur_grant_bytes=33890304
osc.myth-OST0002-osc-ffff8800376bdc00.cur_grant_bytes=35418112
osc.myth-OST0003-osc-ffff8800376bdc00.cur_grant_bytes=2097152
osc.myth-OST0004-osc-ffff8800376bdc00.cur_grant_bytes=33808384

Specifying NIDs and Failover

If a node has multiple network interfaces, it may have multiple NIDs, which must all be identified so other nodes can choose the NID that is appropriate for their network interfaces. Typically, NIDs are specified in a list delimited by commas ( ,). However, when failover nodes are specified, the NIDs are delimited by a colon ( :) or by repeating a keyword such as --mgsnode= or --servicenode=).

To display the NIDs of all servers in networks configured to work with the Lustre file system, run (while LNet is running):

# lctl list_nids

In the example below, mds0 and mds1 are configured as a combined MGS/MDT failover pair and oss0 and oss1 are configured as an OST failover pair. The Ethernet address for mds0 is 192.168.10.1, and for mds1 is 192.168.10.2. The Ethernet addresses for oss0 and oss1 are 192.168.10.20 and 192.168.10.21 respectively.

mds0# mkfs.lustre --fsname=testfs --mdt --mgs \
        --servicenode=192.168.10.2@tcp0 \
        --servicenode=192.168.10.1@tcp0 /dev/sda1
mds0# mount -t lustre /dev/sda1 /mnt/test/mdt
oss0# mkfs.lustre --fsname=testfs --servicenode=192.168.10.20@tcp0 \
        --servicenode=192.168.10.21 --ost --index=0 \
        --mgsnode=192.168.10.1@tcp0 --mgsnode=192.168.10.2@tcp0 \
        /dev/sdb
oss0# mount -t lustre /dev/sdb /mnt/test/ost0
client# mount -t lustre 192.168.10.1@tcp0:192.168.10.2@tcp0:/testfs \
        /mnt/testfs
mds0# umount /mnt/mdt
mds1# mount -t lustre /dev/sda1 /mnt/test/mdt
mds1# lctl get_param mdt.testfs-MDT0000.recovery_status

Where multiple NIDs are specified separated by commas (for example, 10.67.73.200@tcp,192.168.10.1@tcp), the two NIDs refer to the same host, and the Lustre software chooses the best one for communication. When a pair of NIDs is separated by a colon (for example, 10.67.73.200@tcp:10.67.73.201@tcp), the two NIDs refer to two different hosts and are treated as a failover pair (the Lustre software tries the first one, and if that fails, it tries the second one.)

Two options to mkfs.lustre can be used to specify failover nodes. The --servicenode option is used to specify all service NIDs, including those for primary nodes and failover nodes. When the --servicenode option is used, the first service node to load the target device becomes the primary service node, while nodes corresponding to the other specified NIDs become failover locations for the target device. An older option, --failnode, specifies just the NIDs of failover nodes. For more information about the --servicenode and --failnode options, see Configuring Failover in a Lustre File System.

Erasing a File System

If you want to erase a file system and permanently delete all the data in the file system, run this command on your targets:

# mkfs.lustre --reformat

If you are using a separate MGS and want to keep other file systems defined on that MGS, then set the writeconf flag on the MDT for that file system. The writeconf flag causes the configuration logs to be erased; they are regenerated the next time the servers start.

To set the writeconf flag on the MDT:

1. Unmount all clients/servers using this file system, run:

client# umount /mnt/lustre

2. Permanently erase the file system and, presumably, replace it with another file system, run:

mgs# mkfs.lustre --reformat --fsname spfs --mgs --mdt --index=0 /dev/mdsdev

3. If you have a separate MGS (that you do not want to reformat), then add the --writeconf flag to mkfs.lustre on the MDT, run:

mgs# mkfs.lustre --reformat --writeconf --fsname spfs --mgsnode=mgs_nid \
       --mdt --index=0 /dev/mds_device

Reclaiming Reserved Disk Space

All current Lustre installations run the ldiskfs file system internally on service nodes. By default, ldiskfs reserves 5% of the disk space to avoid file system fragmentation. In order to reclaim this space, run the following command on your OSS for each OST in the file system:

# tune2fs [-m reserved_blocks_percent] /dev/ostdev

You do not need to shut down Lustre before running this command or restart it afterwards.

Replacing an Existing OST or MDT

To copy the contents of an existing OST to a new OST (or an old MDT to a new MDT), follow the process for either OST/MDT backups in Backing Up and Restoring an MDT or OST (ldiskfs Device Level)Backing Up an OST or MDT (Backend File System Level). For more information on removing a MDT, see Removing an MDT from the File System.

Identifying To Which Lustre File an OST Object Belongs

Use this procedure to identify the file containing a given object on a given OST.

1. On the OST (as root), run debugfs to display the file identifier ( FID) of the file associated with the object. For example, if the object is 34976 on /dev/lustre/ost_test2, the debug command is:

# debugfs -c -R "stat /O/0/d$((34976 % 32))/34976" /dev/lustre/ost_test2

The command output is:

debugfs 1.45.6.wc1 (20-Mar-2020)
/dev/lustre/ost_test2: catastrophic mode - not reading inode or group bitmaps
Inode: 352365   Type: regular    Mode:  0666   Flags: 0x80000
Generation: 2393149953    Version: 0x0000002a:00005f81
User:  1000   Group:  1000   Size: 260096
File ACL: 0    Directory ACL: 0
Links: 1   Blockcount: 512
Fragment:  Address: 0    Number: 0    Size: 0
ctime: 0x4a216b48:00000000 -- Sat May 30 13:22:16 2009
atime: 0x4a216b48:00000000 -- Sat May 30 13:22:16 2009
mtime: 0x4a216b48:00000000 -- Sat May 30 13:22:16 2009
crtime: 0x4a216b3c:975870dc -- Sat May 30 13:22:04 2009
Size of extra inode fields: 24
Extended attributes stored in inode body:
  fid = "b9 da 24 00 00 00 00 00 6a fa 0d 3f 01 00 00 00 eb 5b 0b 00 00 00 0000
00 00 00 00 00 00 00 00 " (32)
  fid: objid=34976 seq=0 parent=[0x200000400:0x122:0x0] stripe=1
EXTENTS:
(0-64):4620544-4620607

2. The parent FID will be of the form [0x200000400:0x122:0x0] and can be resolved directly using the command lfs fid2path [0x200000404:0x122:0x0] /mnt/lustre on any Lustre client, and the process is complete.
3. In cases of an upgraded 1.x inode (if the first part of the FID is below 0x200000400), the MDT inode number is 0x24dab9 and generation 0x3f0dfa6a and the pathname can also be resolved using debugfs.
4. On the MDS (as root), use debugfs to find the file associated with the inode:

# debugfs -c -R "ncheck 0x24dab9" /dev/lustre/mdt_test
debugfs 1.42.3.wc3 (15-Aug-2012)
/dev/lustre/mdt_test: catastrophic mode - not reading inode or group bitmaps
Inode      Pathname
2415289    /ROOT/brian-laptop-guest/clients/client11/~dmtmp/PWRPNT/ZD16.BMP

The command lists the inode and pathname associated with the object.