• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
  • Skip to footer navigation

The Geek Diary

  • OS
    • Linux
    • CentOS/RHEL
    • Solaris
    • Oracle Linux
    • VCS
  • Interview Questions
  • Database
    • oracle
    • oracle 12c
    • ASM
    • mysql
    • MariaDB
  • DevOps
    • Docker
    • Shell Scripting
  • Big Data
    • Hadoop
    • Cloudera
    • Hortonworks HDP

How to check and repair ACFS with FSCK

by admin

The Problem

You noticed many inconsistencies in different asm metadata views, asmcmd and os outputs that are associated with the ACFS file system. Database shows corruption in RMAN or from dbverify. Database uses ACFS. ALTER DISKGROUP CHECK REPAIR comes back clean but you still see errors or check for corruption returns back positive, eg:

$ /sbin/acfsutil info fs -o iscorrupt [MOUNT_POINT]
acfsutil info fs: ACFS-03037: not an ACFS file system
$ /sbin/acfsutil info fs -o iscorrupt [MOUNT_POINT]
1

Also output from “/sbin/acfsutil info fs” shows Corrupt under flags. For example:

$ /sbin/acfsutil info fs

[MOUNT_POINT]
   ACFS Version: 18.3.0.0.0
   on-disk version: 47.0
   compatible.advm: 18.3.0.0.0
   ACFS compatibility: 18.3.0.0.0
   flags: MountPoint,Available,Corrupt,AutoResizeEnabled

The Solution

You should use FSCK to do the following checks and try to repair ACFS.

The amount of time for FSCK to finish depends on the filesystem and on the amount of files to check. With the verbose option, fsck has to generate this information for all directories and files. This can impact performance significantly, especially for file systems with numerous files.
Fsck processing:

Fsck attempts to validate all of a file system’s metadata. Part of this process includes validation of metadata block free lists. The free block itself has to be checked to ensure that is it the correct type of metadata block and the free list has to be checked to ensure that there are no missing entries and that there are no loops in the list. Since files can be deleted in any order, these free blocks can be located anywhere on the volume. Processing and validating these lists can produce lots of random I/O due to the random nature of file deletion. If fsck takes too long it is ok to interrupt it given it’s running in check mode.

fsck checks and repairs an existing Oracle ACFS. This command can only be run on a dismounted file system. root privileges are required to run fsck. The Oracle ACFS driver must be loaded for fsck to work. To confirm or discard any consistency problem on the affected ACFS filesystem, run FSCK as follows:

1. Dismount the ACFS filesystem on each node as follow (as root user) and keep them dismounted to avoid corruption. Otherwise will get warning:

fsck.acfs: ACFS-00511: /dev/asm/ is mounted on at least one node of the cluster.
#  /usr/sbin/umountall -F acfs

Or

# /usr/sbin/umount [MOUNT_POINT]

Where,
MOUNT_POINT – is your ACFS filesystem.

2. Then, execute FSCK on the associated ADVM volume as follow (from each node if it RAC):

On Linux

# script /tmp/fsck_[node name].txt
# /sbin/fsck -a -y -t acfs /dev/asm/[VOLUME_NAME]
# exit

On Solaris

# script /tmp/fsck_[node name].txt
# /usr/sbin/fsck -F acfs -y -o a,v /dev/asm/[VOLUME_NAME]
# exit

On IBM-AIX

# /usr/sbin/fsck -V acfs -y -o a,v /dev/asm/[VOLUME_NAME]

Where,
/dev/asm/[VOLUME_NAME – is your ADVM volume.

IMPORTANT: By default, fsck only checks for and reports any errors. In check mode fsck can be cancelled if it is taking a long time. The -a flag must be specified to instruct fsck to repair errors in the file system. If check mode completed in a reasonable amount of time, and if it reported problems, run fsck in repair mode. In repair mode fsck cannot be interrupted without risk of leaving the file system in a worse state (loss of data depending on the nature of the corruption).

# /sbin/fsck -a -y -t acfs /dev/asm/[VOLUME_NAME]

3. Then mount back the ACFS filesystem on each node if it is RAC. It is always better to manually mount after fsck repairs.

# /usr/sbin/mount -v acfs /dev/asm/[VOLUME_NAME] [MOUNT_POINT]

Where,
/dev/asm/[VOLUME_NAME] – is your ADVM volume.
MOUNT_POINT – is your ACFS filesystem.

Or

# /usr/sbin/mount -v acfs -o all none none

Example:
– Locate correct volume with ‘/sbin/acfsutil info fs‘ and execute fsck.

[HOSTNAME]:(+ASM2)/opt/oracle > /sbin/acfsutil info fs
/acfs1
  ACFS Version: 11.2.0.4.0
  flags:        MountPoint,Available,Corrupt
  mount time:   Fri Jun 10 17:29:14 2016
  volumes:      1
  total size:   xxxxxxxxxxxx
  total free:   xxxxxxxxxxxx
  primary volume: /dev/asm/[VOLUME_NAME2]

Example of an clean output:

Script command is started on Fri May 18 10:26:18 EDT 2016.

[/dev/asm]# /usr/sbin/fsck -V acfs -y -o a,v /dev/asm/[VOLUME_NAME3]

version                   = 11.2.0.3.0
fsck: temporary directory '/usr/tmp'
fsck: current directory '/dev/asm'
*****************************
********** Pass: 1 **********
*****************************
fsck: file system check starting for volume: /dev/asm/[VOLUME_NAME3]
Oracle ASM Cluster File System (ACFS) On-Disk Structure Version: 39.0
Volume indicates the on-disk version is 39.0
Volume indicates the on-disk version is 39.0
ACFS file system created at: Tue Apr 24 22:37:46 2012
checking primary file system
fsck: Volume_Log recovery on node 3 not needed

fsck: checking File_Entry_Table entry: 28 (0x1c) at disk offset: 83968 (0x14800)
fsck: checking File_Entry_Table entry: 29 (0x1d) at disk offset: 84480 (0x14a00)
fsck: checking File_Entry_Table entry: 30 (0x1e) at disk offset: 84992 (0x14c00)
fsck: checking File_Entry_Table entry: 31 (0x1f) at disk offset: 85504 (0x14e00)

fsck: check for unprocessed File_Entry_Table entries complete
      Files checked in primary file system: 100%

       Checking if any files are orphaned...

Phase 1 Orphan check...

Phase 2 Orphan check...

       0 orphans found

Checker completed with no errors.

[/dev/asm]# exit

Script command is complete on Fri May 18 10:27:14 EDT 2012.

Example of an output when fsck is fixing corruption:

fsck.acfs: file system check starting for volume: /dev/asm/[VOLUME_NAME4]
Oracle ASM Cluster File System (ACFS) On-Disk Structure Version: 39.0
Volume indicates the on-disk version is 39.0
Volume indicates the on-disk version is 39.0
ACFS file system created at: Fri Jun  3 15:18:10 2011
checking primary file system
fsck.acfs: Volume_Log recovery on node 1 not needed
fsck.acfs: Volume_Log recovery on node 2 not needed

fsck.acfs: ACFS-07592: metadata structure has incorrect header for:
ACFS Internal Structure: [ACFS Free Block]
    file identifier: 1280 (0x500)    disk offset: 24006656 (0x16e5000)
Parent Structure: [ACFS Local Free List]
    file identifier: 33 (0x21)    disk offset: 193024 (0x2f200)

                   Value Found              Expected Value
                   ------------------------ ------------------------
  File Identifier: 1280 (0x500)                  1280 (0x500)
  Struct_Type:     0x0f500004                    0x0f500007  
                   (OFS_FILE_ENTRY)              (OFS_FETA_FREE_BLOCK)
  Endian_Format:   Little Endian                 Little Endian
  OSCreatedOn:     Linux                         Linux
  Struct_Version:  1                             1
  FileSystemID:    1315928241 (0x4e6f78b1)       1315928241 (0x4e6f78b1)
  CheckSum:        1731346914 (0x673241e2)       1731346914 (0x673241e2)

fsck.acfs: checking directory:
for file:
ACFS Internal File: [ACFS Root Directory]
    file identifier: 2 (0x2)    disk offset: 70656 (0x11400)
fsck.acfs: checking file '/.ACFS'
fsck.acfs: checking directory:
for file:
File: '/.ACFS'
    file identifier: 7 (0x7)    disk offset: 73216 (0x11e00)
Parent Directory: [ACFS Root Directory]
    file identifier: 2 (0x2)    disk offset: 70656 (0x11400)
fsck.acfs: checking file '/.ACFS/.fileid'
fsck.acfs: checking directory:

.............

fsck.acfs: checking file '/home/[PATH]/dirchk/REP_UR.cpr'
fsck.acfs: checking file '/home/[PATH]/dirchk/REP_UR.cps'
fsck.acfs: checking file '/home/[PATH]/dirchk/EXT_AL.cps'
fsck.acfs: checking file '/home/[PATH]/dirdat'
fsck.acfs: checking directory:
for file:
File: '/home/[PATH]/dirdat'
    file identifier: 45 (0x2d)    disk offset: 199168 (0x30a00)
Parent Directory: /home/[PATH]
    file identifier: 43 (0x2b)    disk offset: 198144 (0x30600)
fsck.acfs: checking file '/home/[PATH]/dirdef'
fsck.acfs: checking directory:
for file:
File: '/home/[PATH]/dirdef'
    file identifier: 46 (0x2e)    disk offset: 199680 (0x30c00)
Parent Directory: /home/[PATH]
    file identifier: 43 (0x2b)    disk offset: 198144 (0x30600)
fsck.acfs: checking file '/home/[PATH]/dirpcs'
fsck.acfs: checking directory:
for file:
File: '/home/[PATH]/dirpcs'
    file identifier: 47 (0x2f)    disk offset: 200192 (0x30e00)
Parent Directory: /home/[PATH]
    file identifier: 43 (0x2b)    disk offset: 198144 (0x30600)
fsck.acfs: checking file '/home/[PATH]/dirpcs/MGR.pcm'
fsck.acfs: checking file '/home/[PATH]/macro'
fsck.acfs: checking directory:

...........

fsck.acfs: ACFS-07711: orphan metadata structure (type: 0x0f500007  (OFS_FETA_FREE_BLOCK)) found for file identifier: 2142 (0x85e) at disk offset: 24448000 (0x1750c00)

fsck.acfs: ACFS-07711: orphan metadata structure (type: 0x0f500007  (OFS_FETA_FREE_BLOCK)) found for file identifier: 2143 (0x85f) at disk offset: 24448512 (0x1750e00)

fsck.acfs: ACFS-07711: orphan metadata structure (type: 0x0f500007  (OFS_FETA_FREE_BLOCK)) found for file identifier: 2339 (0x923) at disk offset: 61203968 (0x3a5e600)

    159 orphans found

fixing file system

problems processed: 5%
problems processed: 10%
problems processed: 15%
problems processed: 20%

.............

problems processed: 90%
problems processed: 95%
problems processed: 100%
........

fsck.acfs: check for unprocessed File_Entry_Table entries complete
       Files checked in primary file system: 55%
       Files checked in primary file system: 60%
       Files checked in primary file system: 65%
       Files checked in primary file system: 70%
       Files checked in primary file system: 75%
       Files checked in primary file system: 80%
       Files checked in primary file system: 85%
       Files checked in primary file system: 90%
       Files checked in primary file system: 95%
       Files checked in primary file system: 100%

   Checking if any files are orphaned...

Phase 1 Orphan check...

Phase 2 Orphan check...

    0 orphans found

fsck.acfs: Checker/Fixer completed with the following results:
    File System Errors:  168
    Fixed:               168
    Not Fixed:           0

By default, fsck only checks for and reports any errors. The -o a option must be specified to instruct fsck to fix errors in the file system. In a few cases, fsck prompts for questions before proceeding to check a file system. These cases include:

  • If fsck detects that another fsck is in progress on the file system.
  • If fsck detects that the Oracle ACFS driver is not loaded.
  • If the file system does not appear to be Oracle ACFS.

In checking mode, fsck also prompts if there are transaction logs that have not been processed completely due to an incomplete shutdown. To run in a non-interactive mode, include either the -y or -n options to answer yes or no to any questions.

Filed Under: ASM, oracle

Some more articles you might also be interested in …

  1. How To Convert A Partitioned Table To A Non-Partitioned Table Using DataPump In Oracle 11g and 12c
  2. How to rename a Pluggable Database, along with the respective directories, in Oracle 12c
  3. How to Check the Environment Variables for an Oracle Process
  4. What is the difference between PRIMARY Key and UNIQUE Key
  5. How to Connect After ORA-20 has Occurred
  6. How to create restore points for PDB and perform flashback at PDB level
  7. How to install and configure Oracle ASMLIB on Native Linux multipath mapper devices
  8. Beginners Guide to Monitoring SQL Statements with Real-Time SQL Monitoring in Oracle Database
  9. How to Rename or Move Datafiles and Logfiles in Oracle Database
  10. How to Enable Fast-Start Failover in Oracle Data Guard

You May Also Like

Primary Sidebar

Recent Posts

  • “az storage blob” Command Examples (Manage blob storage containers and objects in Azure)
  • “az storage account” Command Examples (Manage storage accounts in Azure)
  • “az sshkey” Command Examples (Manage ssh public keys with virtual machines)
  • “az redis” Command Examples

© 2023 · The Geek Diary

  • Archives
  • Contact Us
  • Copyright