Troubleshooting Recovery

This chapter describes what to do if something goes wrong during recovery. It describes:

• Recovering from Errors or Corruption on a Backing ldiskfs File System
• Recovering from Corruption in the Lustre File System
• Recovering from an Unavailable OST
• Checking the file system with LFSCK

Recovering from Errors or Corruption on a Backing ldiskfs File System

When an OSS, MDS, or MGS server crash occurs, it is not necessary to run e2fsck on the file system. ldiskfs journaling ensures that the file system remains consistent over a system crash. The backing file systems are never accessed directly from the client, so client crashes are not relevant for server file system consistency.

The only time it is REQUIRED that e2fsck be run on a device is when an event causes problems that ldiskfs journaling is unable to handle, such as a hardware device failure or I/O error. If the ldiskfs kernel code detects corruption on the disk, it mounts the file system as read-only to prevent further corruption, but still allows read access to the device. This appears as error “-30” ( EROFS) in the syslogs on the server, e.g.:

Dec 29 14:11:32 mookie kernel: LDISKFS-fs error (device sdz):
            ldiskfs_lookup: unlinked inode 5384166 in dir #145170469
Dec 29 14:11:32 mookie kernel: Remounting filesystem read-only

In such a situation, it is normally required that e2fsck only be run on the bad device before placing the device back into service.

In the vast majority of cases, the Lustre software can cope with any inconsistencies found on the disk and between other devices in the file system.

For problem analysis, it is strongly recommended that e2fsck be run under a logger, like script, to record all of the output and changes that are made to the file system in case this information is needed later.

If time permits, it is also a good idea to first run e2fsck in non-fixing mode (-n option) to assess the type and extent of damage to the file system. The drawback is that in this mode, e2fsck does not recover the file system journal, so there may appear to be file system corruption when none really exists.

To address concern about whether corruption is real or only due to the journal not being replayed, you can briefly mount and unmount the ldiskfs file system directly on the node with the Lustre file system stopped, using a command similar to:

mount -t ldiskfs /dev/{ostdev} /mnt/ost; umount /mnt/ost

This causes the journal to be recovered.

The e2fsck utility works well when fixing file system corruption (better than similar file system recovery tools and a primary reason why ldiskfs was chosen over other file systems). However, it is often useful to identify the type of damage that has occurred so an ldiskfs expert can make intelligent decisions about what needs fixing, in place of e2fsck.

root# {stop lustre services for this device, if running}
root# script /tmp/e2fsck.sda
Script started, file is /tmp/e2fsck.sda
root# mount -t ldiskfs /dev/sda /mnt/ost
root# umount /mnt/ost
root# e2fsck -fn /dev/sda   # don't fix file system, just check for corruption
:
[e2fsck output]
:
root# e2fsck -fp /dev/sda   # fix errors with prudent answers (usually yes)

Recovering from Corruption in the Lustre File System

In cases where an ldiskfs MDT or OST becomes corrupt, you need to run e2fsck to ensure local filesystem consistency, then use LFSCK to run a distributed check on the file system to resolve any inconsistencies between the MDTs and OSTs, or among MDTs.

1. Stop the Lustre file system.
2. Run e2fsck -f on the individual MDT/OST that had problems to fix any local file system damage. We recommend running e2fsck under script, to create a log of changes made to the file system in case it is needed later. After e2fsck is run, bring up the file system, if necessary, to reduce the outage window.

Working with Orphaned Objects

The simplest problem to resolve is that of orphaned objects. When the LFSCK layout check is run, these objects are linked to new files and put into .lustre/lost+found/MDT*xxxx* in the Lustre file system (where MDTxxxx is the index of the MDT on which the orphan was found), where they can be examined and saved or deleted as necessary.

With Lustre version 2.7 and later, LFSCK will identify and process orphan objects found on MDTs as well.

Recovering from an Unavailable OST

One problem encountered in a Lustre file system environment is when an OST becomes unavailable due to a network partition, OSS node crash, etc. When this happens, the OST’s clients pause and wait for the OST to become available again, either on the primary OSS or a failover OSS. When the OST comes back online, the Lustre file system starts a recovery process to enable clients to reconnect to the OST. Lustre servers put a limit on the time they will wait in recovery for clients to reconnect.

During recovery, clients reconnect and replay their requests serially, in the same order they were done originally. Until a client receives a confirmation that a given transaction has been written to stable storage, the client holds on to the transaction, in case it needs to be replayed. Periodically, a progress message prints to the log, stating how_many/expected clients have reconnected. If the recovery is aborted, this log shows how many clients managed to reconnect. When all clients have completed recovery, or if the recovery timeout is reached, the recovery period ends and the OST resumes normal request processing.

If some clients fail to replay their requests during the recovery period, this will not stop the recovery from completing. You may have a situation where the OST recovers, but some clients are not able to participate in recovery (e.g. network problems or client failure), so they are evicted and their requests are not replayed. This would result in any operations on the evicted clients failing, including in-progress writes, which would cause cached writes to be lost. This is a normal outcome; the recovery cannot wait indefinitely, or the file system would be hung any time a client failed. The lost transactions are an unfortunate result of the recovery process.

Checking the file system with LFSCK

LFSCK is an administrative tool for checking and repair of the attributes specific to a mounted Lustre file system. It is similar in concept to an offline fsck repair tool for a local filesystem, but LFSCK is implemented to run as part of the Lustre file system while the file system is mounted and in use. This allows consistency checking and repair of Lustre-specific metadata without unnecessary downtime, and can be run on the largest Lustre file systems with minimal impact to normal operations.

LFSCK can verify and repair the Object Index (OI) table that is used internally to map Lustre File Identifiers (FIDs) to MDT internal ldiskfs inode numbers, in an internal table called the OI Table. An OI Scrub traverses the OI table and makes corrections where necessary. An OI Scrub is required after restoring from a file-level MDT backup ( Backing Up and Restoring an MDT or OST (ldiskfs Device Level)), or in case the OI Table is otherwise corrupted. Later phases of LFSCK will add further checks to the Lustre distributed file system state. LFSCK namespace scanning can verify and repair the directory FID-in-dirent and LinkEA consistency.

In Lustre software release 2.6, LFSCK layout scanning can verify and repair MDT-OST file layout inconsistencies. File layout inconsistencies between MDT-objects and OST-objects that are checked and corrected include dangling reference, unreferenced OST-objects, mismatched references and multiple references.

In Lustre software release 2.7, LFSCK layout scanning is enhanced to support verify and repair inconsistencies between multiple MDTs.

Control and monitoring of LFSCK is through LFSCK and the lctl get_param command. LFSCK supports three types of interface: switch interface, status interface, and adjustment interface. These interfaces are detailed below.

LFSCK switch interface

Manually Starting LFSCK

Description

LFSCK can be started after the MDT is mounted using the lctl lfsck_start command.

Usage

lctl lfsck_start <-M | --device [MDT,OST]_device> \
                    [-A | --all] \
                    [-c | --create_ostobj on | off] \
                    [-C | --create_mdtobj on | off] \
                    [-d | --delay_create_ostobj on | off] \
                    [-e | --error {continue | abort}] \
                    [-h | --help] \
                    [-n | --dryrun on | off] \
                    [-o | --orphan] \
                    [-r | --reset] \
                    [-s | --speed ops_per_sec_limit] \
                    [-t | --type check_type[,check_type...]] \
                    [-w | --window_size size]

Options

The various lfsck_start options are listed and described below. For a complete list of available options, type lctl lfsck_start -h.

Manually Stopping LFSCK

Description

To stop LFSCK when the MDT is mounted, use the lctl lfsck_stop command.

Usage

lctl lfsck_stop <-M | --device [MDT,OST]_device> \
                    [-A | --all] \
                    [-h | --help]

Options

The various lfsck_stop options are listed and described below. For a complete list of available options, type lctl lfsck_stop -h.

Check the LFSCK global status

Description

Check the LFSCK global status via a single lctl lfsck_query command on the MDS.

Usage

lctl lfsck_query <-M | --device MDT_device> \
                    [-h | --help] \
                    [-t | --type lfsck_type[,lfsck_type...]] \
                    [-w | --wait]

Options

The various lfsck_query options are listed and described below. For a complete list of available options, type lctl lfsck_query -h.

LFSCK status interface

LFSCK status of OI Scrub via procfs

Description

For each LFSCK component there is a dedicated procfs interface to trace the corresponding LFSCK component status. For OI Scrub, the interface is the OSD layer procfs interface, named oi_scrub. To display OI Scrub status, the standard lctl get_param command is used as shown in the usage below.

Usage

lctl get_param -n osd-ldiskfs.FSNAME-[MDT_target|OST_target].oi_scrub

Output

LFSCK status of namespace via procfs

Description

The namespace component is responsible for checks described in Checking the file system with LFSCK. The procfs interface for this component is in the MDD layer, named lfsck_namespace. To show the status of this component, lctl get_param should be used as described in the usage below.

The LFSCK namespace status output refers to phase 1 and phase 2. Phase 1 is when the LFSCK main engine, which runs on each MDT, linearly scans its local device, guaranteeing that all local objects are checked. However, there are certain cases in which LFSCK cannot know whether an object is consistent or cannot repair an inconsistency until the phase 1 scanning is completed. During phase 2 of the namespace check, objects with multiple hard-links, objects with remote parents, and other objects which couldn’t be verified during phase 1 will be checked.

Usage

lctl get_param -n mdd. FSNAME-MDT_target.lfsck_namespace

Output

LFSCK status of layout via procfs

Description

The layout component is responsible for checking and repairing MDT-OST inconsistency. The procfs interface for this component is in the MDD layer, named lfsck_layout, and in the OBD layer, named lfsck_layout. To show the status of this component lctl get_param should be used as described in the usage below.

The LFSCK layout status output refers to phase 1 and phase 2. Phase 1 is when the LFSCK main engine, which runs on each MDT/OST, linearly scans its local device, guaranteeing that all local objects are checked. During phase 1 of layout LFSCK, the OST-objects which are not referenced by any MDT-object are recorded in a bitmap. During phase 2 of the layout check, the OST-objects in the bitmap will be re-scanned to check whether they are really orphan objects.

Usage

lctl get_param -n mdd.
FSNAME-
MDT_target.lfsck_layout
lctl get_param -n obdfilter.
FSNAME-
OST_target.lfsck_layout

Output

LFSCK adjustment interface

Rate control

Description

The LFSCK upper speed limit can be changed using lctl set_param as shown in the usage below.

Usage

lctl set_param mdd.${FSNAME}-${MDT_target}.lfsck_speed_limit=
N
lctl set_param obdfilter.${FSNAME}-${OST_target}.lfsck_speed_limit=
N

Values

Auto scrub

Description

The auto_scrub parameter controls whether OI scrub will be triggered when an inconsistency is detected during OI lookup. It can be set as described in the usage and values sections below.

There is also a noscrub mount option (see mount.lustre) which can be used to disable automatic OI scrub upon detection of a file-level backup at mount time. If the noscrub mount option is specified, auto_scrub will also be disabled, so OI scrub will not be triggered when an OI inconsistency is detected. Auto scrub can be renabled after the mount using the command shown in the usage. Manually starting LFSCK after mounting provides finer control over the starting conditions.

Usage

lctl set_param osd_ldiskfs.${FSNAME}-${MDT_target}.auto_scrub=N

where N is an integer as described below.

Values