Glossary
A
: Access control list. An extended attribute associated with a file that contains enhanced authorization directives.
: A manual configuration change to mark an OST as unavailable, so that operations intended for that OST fail immediately with an I/O error instead of waiting indefinitely for OST recovery to complete
C
: An RPC made by the lock server on an OST or MDT to another system, usually a client, to indicate that the lock is now granted.
: An llog file used in a node, or retrieved from a management server over the network with configuration instructions for the Lustre file system at startup time.
: A lock held by every node in the cluster to control configuration changes. When the configuration is changed on the MGS, it revokes this lock from all nodes. When the nodes receive the blocking callback, they quiesce their traffic, cancel and re-enqueue the lock and wait until it is granted again. They can then fetch the configuration updates and resume normal operation.
D
: An extended attribute on the filesystem root directory that describes the default stripe count, stripe size, and layout pattern used for new files created in a file system. This can be amended by using a default file layout on a directory or a per-file layout.
: A mechanism that can be used during read and write system calls to avoid memory cache overhead for large I/O requests. It bypasses the data copy between application and kernel memory, and avoids buffering the data in the client memory.
: An extended attribute that describes the default file layout used for new files created within that directory. This is also inherited by new subdirectories created in that directory at creation time.
Distributed Namespace Environment (DNE)
: A collection of metadata targets serving a single file system namespace. Without DNE, Lustre file systems are limited to a single metadata target for the entire name space. Without the ability to distribute metadata load over multiple targets, Lustre file system performance may be limited. The DNE functionality has two types of scalability. Remote Directories (DNE1) allows sub-directories to be serviced by an independent MDT(s), increasing aggregate metadata capacity and performance for independent sub-trees of the filesystem. This also allows performance isolation of workloads running in a specific sub-directory on one MDT from workloads on other MDTs. In Lustre 2.8 and later Striped Directories (DNE2) allows a single directory to be serviced by multiple MDTs.
E
: Extended attribute (also xattr). A small amount of metadata stored on a file that can be retrieved by a name associated with a particular inode. A Lustre file system uses EAs to store striping information (indicating the location of file data on OSTs). Examples of extended attributes are ACLs, striping information, and the FID of the file.
: The process of removing a client’s state from the server if the client is unresponsive to server requests after a timeout or if server recovery fails. If a client is still running, it is required to flush the cache associated with the server when it becomes aware that it has been evicted.
: The state held by a server for a client that is sufficient to transparently recover all in-flight operations when a single failure occurs.
Extent
: A range of contiguous bytes or blocks in a file that are addressed by a {start, length} tuple instead of individual block numbers.
: An LDLM lock used by the OSC to protect an extent in an OST object for concurrent control of read/write, file size acquisition, and truncation operations.
F
: The failover process in which the default active server regains control from the backup server that had taken control of the service.
: An OST that is not expected to recover if it fails to answer client requests. A failout OST can be administratively failed, thereby enabling clients to return errors when accessing data on the failed OST without making additional network requests or waiting for OST recovery to complete.
: The process by which a standby computer server system takes over for an active computer server after a failure of the active node. Typically, the standby computer server gains exclusive access to a shared storage device between the two servers.
: Lustre File Identifier. A 128-bit file system-unique identifier for a file or object in the file system. The FID structure contains a unique 64-bit sequence number (see FLDB), a 32-bit object ID (OID), and a 32-bit version number. The sequence number is unique across all Lustre targets (OSTs and MDTs).
: A group of files that are defined through a directory that represents the start point of a file system.
: FID Location Database. This database maps a sequence of FIDs to a specific target (MDT or OST), which manages the objects within the sequence. The FLDB is cached by all clients and servers in the file system, but is typically only modified when new servers are added to the file system.
: Group of I/O RPCs initiated by the OSC that are concurrently queued or processed at the OST. Increasing the number of RPCs in flight for high latency networks can increase throughput and reduce visible latency at the client.
G
: An RPC made by an OST or MDT to another system (usually a client) to indicate that a held extent lock should be surrendered. If the system is using the lock, then the system should return the object size and timestamps in the reply to the glimpse callback instead of cancelling the lock. Glimpses are introduced to optimize the acquisition of file attributes without introducing contention on an active lock.
I
: The state held by the client for each target that it is connected to. It holds server NIDs, connection state, and uncommitted RPCs needed to fully recover a transaction sequence after a server failure and restart.
: A special Lustre file system locking operation in the Linux kernel. An intent lock combines a request for a lock with the full information to perform the operation(s) for which the lock was requested. This offers the server the option of granting the lock or performing the operation and informing the client of the operation result without granting a lock. The use of intent locks enables metadata operations (even complicated ones) to be implemented with a single RPC from the client to the server.
L
: A fatal error condition detected by the software that halts execution of the kernel thread to avoid potential further corruption of the system state. It is printed to the console log and triggers a dump of the internal debug log. The system must be rebooted to clear this state.
: Lustre Distributed Lock Manager.
: The Lustre file system command-line utility that allows end users to interact with Lustre software features, such as setting or checking file striping or per-target free space. For more details, see lfs.
: Lustre file system check. A distributed version of a disk file system checker. Normally, lfsck does not need to be run, except when file systems are damaged by events such as multiple disk failures and cannot be recovered using file system journal recovery.
: Lustre lite. This term is in use inside code and in module names for code that is related to the Linux client VFS interface.
: Lustre log. An efficient log data structure used internally by the file system for storing configuration and distributed transaction records. An llog is suitable for rapid transactional appends of records and cheap cancellation of records through a bitmap.
: Lustre log catalog. An llog with records that each point at an llog. Catalogs were introduced to give llogs increased scalability. llogs have an originator which writes records and a replicator which cancels records when the records are no longer needed.
: Logical Metadata Volume. A module that implements a DNE client-side abstraction device. It allows a client to work with many MDTs without changes to the llite module. The LMV code forwards requests to the correct MDT based on name or directory striping information and merges replies into a single result to pass back to the higher llite layer that connects the Lustre file system with Linux VFS, supports VFS semantics, and complies with POSIX interface specifications.
: Lustre network driver. A code module that enables LNet support over particular transports, such as TCP and various kinds of InfiniBand networks.
: Lustre networking. A message passing network protocol capable of running and routing through various physical layers. LNet forms the underpinning of LNETrpc.
: A module that makes lock RPCs to a lock server and handles revocations from the server.
: A service that is co-located with a storage target that manages locks on certain objects. It also issues lock callback requests, calls while servicing or, for objects that are already locked, completes lock requests.
: Logical Object Volume. The object storage analog of a logical volume in a block device volume management system, such as LVM or EVMS. The LOV is primarily used to present a collection of OSTs as a single device to the MDT and client file system drivers.
: An operating instance with a mounted Lustre file system.
: A file in the Lustre file system. The implementation of a Lustre file is through an inode on a metadata server that contains references to one or more objects on an OSSs.
M
: Multi-block allocator. Functionality in ldiskfs that enables the ldiskfs file system to allocate multiple blocks with a single request to the block allocator.
: MetaData Client. A Lustre client component that sends metadata requests via RPC over LNet to the metadata target (MDT).
: Metadata Disk Device. Lustre server component that interfaces with the underlying object storage device to manage the Lustre file system namespace (directories, file ownership, attributes).
: MetaData Server. The server node that is hosting the metadata target (MDT).
: MetaData Target. A storage device containing the file system namespace that is made available over the network to a client. It stores filenames, attributes, and the layout of OST objects that store the file data.
: The metadata target storing the file system root directory, as well as some core services such as quota tables. Multiple metadata targets are possible in the same file system. MDT0000 must be available for the file system to be accessible.
: Management Service. A software module that manages the startup configuration and changes to the configuration. Also, the server node on which this system runs.
N
: Network IDentifier. Encodes the type, network number, and network address of a network interface on a node for use by the Lustre file system.
: Node affinity describes the property of process threads to be affine to specific cores on a NUMA system. Without node affinity, an operating system scheduler may move threads across processors in a sub-optimal way that significantly reduces performance of the system overall.
: Network Request Scheduler. A subcomponent of the PtlRPC layer, which specifies the order in which RPCs are handled by servers. This allows optimizing large numbers of incoming requests for disk access patterns, fairness between clients, and other administrator-selected policies.
: Non-Uniform Memory Access. Describes a multi-processing architecture where the time taken to access specific memory differs depending on memory location relative to a given processor. Typically machines with multiple sockets are NUMA architectures.
O
: Object-Based Device. The generic term for components in the Lustre software stack that can be configured on the client or server. Examples include MDC, OSC, LOV, LMV, MDT, and OST.
: Module that can implement the Lustre object or metadata APIs. Examples of OBD types include the LOV, OSC and OSD.
: Refers to a storage-device API or protocol involving storage objects and offsets therein, rather than addressing storage by individual blocks.
: A cache of open file handles. This is a performance enhancement for NFS and files that are opened repeatedly on a client.
: OST objects to which no Lustre file points. Orphan objects can arise from crashes and are automatically removed by an llog recovery between the MDT and OST. When a client deletes a file, the MDT unlinks it from the namespace. If this is the last link, it will atomically add the OST objects into a per-OST llog(in case a crash occurrs) and then wait until the MDT unlink commits to disk. At this point, it is safe to destroy the OST objects. Once the OST object destroy is committed, the MDT llog records can be cancelled.
: Object Storage Client. The client module communicating to an OST mounted on an OSS.
: Object Storage Device. A generic, industry term for storage devices with a more extended interface than block-oriented devices such as disks. For the Lustre file system, this name is used to describe a software module that implements an object storage API in the kernel. It is also used to refer to an instance of an object storage device created by that driver. The OSD device is layered on a file system, either ldiskfs (ext4) or ZFS, with methods that implement create, destroy and other I/O operations on file inodes.
: Object Storage Server. A server OBD that provides network access between the client and local OSTs.
: Object Storage Target. An OSD made accessible through a network protocol. Typically, an OST is associated with a unique OSD which, in turn is associated with a formatted disk file system on the server containing the data objects.
: Using wide striping for a file that allocates multiple objects in a file to each OST. This allows the number of stripes to exceed the number of OSTs, and can improve scalability for IO and locking.
P
: A locking protocol in the Linux VFS layer that allows for directory operations performed in parallel.
: OST pools allows the administrator to associate a name with an arbitrary subset of OSTs in a Lustre cluster. A group of OSTs can be combined into a named pool with unique access permissions and stripe characteristics.
: A service address on an LNet NID that binds requests to a specific request service, such as an MDS, MGS, OSS, or LDLM. Services may listen on multiple portals to ensure that high priority messages are not queued behind many slow requests on another portal.
: An RPC protocol layered on LNet. This protocol deals with stateful servers and has exactly-once semantics and built in support for recovery.
R
: The process that re-establishes the connection state when a client that was previously connected to a server reconnects after the server restarts.
: A remote directory describes a feature of Lustre DNE where a subdirectory may be stored on a different MDT than the parent directory. This is sometimes referred to as DNE1.
: The concept of re-executing a server request after the server has lost information in its memory caches and shut down. The replay requests are retained by clients until the server(s) have confirmed that the data is persistent on disk. Only requests for which a client received a reply and were assigned a transaction number by the server are replayed. Requests that did not get a reply are resent.
: An RPC request sent from a client to a server that has not had a reply from the server. This might happen if the request was lost on the way to the server, if the reply was lost on the way back from the server, or if the server crashes before or after processing the request. During server RPC recovery processing, resent requests are processed after replayed requests, and use the client RPC XID to determine if the resent RPC request was already executed on the server.
: Also called a “blocking callback”. An RPC request made by the lock server (typically for an OST or MDT) to a lock client to revoke a granted DLM lock.
: A mechanism whereby the identity of a root user on a client system is mapped to a different identity on the server to avoid root users on clients from accessing or modifying root-owned files on the servers. This does not prevent root users on the client from assuming the identity of a non-root user, so should not be considered a robust security mechanism. Typically, for management purposes, at least one client system should not be subject to root squash.
: LNet routing between different networks and LNDs.
: Remote Procedure Call. A network encoding of a request, often sent from a client to a server to perform a particular action.
S
: A contiguous, logical extent of a Lustre file written to a single OST. Used synonymously with a single OST data object that makes up part of a file visible to user applications.
: A striped directory is when metadata for files in a given directory are distributed evenly over multiple MDTs. Striped directories are only available in Lustre software version 2.8 or later. A user can create a striped directory to increase metadata performance of very large directories (e.g. 1M+ entries) by distributing the metadata requests in a single directory over two or more MDTs. This is sometimes called DNE2.
: The maximum number of bytes that will be written to an OST object in a RAID0-striped Lustre file before the next object in the file’s layout is used, when writing data sequentially to a file. Once a full stripe has been written to each of the objects in the layout, the first object will be written to again in round-robin fashion.
: The number of OSTs holding objects for a RAID0-striped Lustre file.
T
: Checksum format defined by the T10 SCSI committee to store data checksums together with the data on supporting storage devices.
: Token Bucket Filter. NRS policy that assigns tokens to each rule proportional to the priority of RPCs submitted that match the rule. When the tokens for a rule have all been consumed within a time period, any remaining RPCs matching that rule must wait until the next time period to be processed.
W
: Strategy of using many OSTs to store stripes of a single file. This obtains maximum bandwidth access to a single file through parallel utilization of many OSTs. For more information about wide striping, see Lustre Striping Internals.