Beginners Guide to Global File System 2 (GFS2)

Global File System 2 (GFS2)

Global File System 2 (GFS2) is a cluster file system interfacing directly with the kernel VFS layer. This means that the same file system can be mounted and used by multiple cluster nodes simultaneously, while still providing a full regular file system, including features such as support for POSIX ACLs, extended attributes, and quotas.

To accomplish this, every node accessing a GFS2 file system uses the cluster infrastructure provided by Corosync and Pacemaker to provide services such as fencing and locking. Each cluster node mounting a GFS2 file system will use a separate journal. If a node fails, one of the other nodes in the cluster will replay the journal for the failed node after the failed node has been fenced. To prevent race conditions between two nodes when accessing the file system, GFS2 uses the Distributed Lock Manager (DLM) to coordinate locks on files and directories.

When running in a pure 64-bit environment, a GFS2 file system can theoretically scale up to 8 EiB. However, the maximum GFS2 file system size currently supported by Red Hat is 100 TiB. For most deployments, having multiple smaller file systems makes more sense than a single large file system. On a larger file system, running a fsck.gfs2 command will take longer, and use more memory. A full restore of a file system from backup will also take longer.

Red Hat offers the GFS2 file system and CLVM as part of the Resilient Storage Add-on for Red Hat Enterprise Linux Server. The Resilient Storage Add-on includes the High-Availability Add-on as part of the subscription.

SELinux and GFS2

GFS2 supports extended attributes (xattrs) and can store file labels used by Security Enhanced Linux (SELinux) just like XFS and ext4. However, using SELinux with GFS2 is complicated by the fact that updates to SELinux file labels on a GFS2 file system are currently not cluster coherent.

This means that if one node changes the SELinux context of a file on a GFS2 file system, other cluster nodes that have that file system mounted may continue using the old context on that file indefinitely. This is somewhat tricky to resolve, and the issue is currently being tracked at bugzilla.redhat.com as bug #437984.

Due to this issue, it may be beneficial to not write SELinux labels to individual files on a GFS2 file system. Due to the way in which GFS2 stores file xattrs, updating those labels may result in a performance penalty specific to GFS2. This is likely to be most noticeable with workloads that involve many small files. One approach that may help to work around this while still using SELinux is to take advantage of the mount option context= to set the context of all files on that GFS2 file system to a particular value when mounted. This will avoid xattr lookups and writes. Alternatively, other steps might be taken to ensure that file labels are not changed on the file system while it is mounted by multiple nodes.

If SELinux labels are not changed on files, then SELinux in enforcing mode will otherwise function normally with GFS2 file systems, and Red Hat does test GFS2 file systems and the entire cluster stack with SELinux enforcing on.