• Skip to primary navigation
  • Skip to main content
  • Skip to primary sidebar
  • Skip to footer navigation

The Geek Diary

  • OS
    • Linux
    • CentOS/RHEL
    • Solaris
    • Oracle Linux
    • VCS
  • Interview Questions
  • Database
    • oracle
    • oracle 12c
    • ASM
    • mysql
    • MariaDB
  • DevOps
    • Docker
    • Shell Scripting
  • Big Data
    • Hadoop
    • Cloudera
    • Hortonworks HDP

How to check and repair ACFS with FSCK

by admin

The Problem

You noticed many inconsistencies in different asm metadata views, asmcmd and os outputs that are associated with the ACFS file system. Database shows corruption in RMAN or from dbverify. Database uses ACFS. ALTER DISKGROUP CHECK REPAIR comes back clean but you still see errors or check for corruption returns back positive, eg:

$ /sbin/acfsutil info fs -o iscorrupt [MOUNT_POINT]
acfsutil info fs: ACFS-03037: not an ACFS file system
$ /sbin/acfsutil info fs -o iscorrupt [MOUNT_POINT]
1

Also output from “/sbin/acfsutil info fs” shows Corrupt under flags. For example:

$ /sbin/acfsutil info fs

[MOUNT_POINT]
   ACFS Version: 18.3.0.0.0
   on-disk version: 47.0
   compatible.advm: 18.3.0.0.0
   ACFS compatibility: 18.3.0.0.0
   flags: MountPoint,Available,Corrupt,AutoResizeEnabled

The Solution

You should use FSCK to do the following checks and try to repair ACFS.

The amount of time for FSCK to finish depends on the filesystem and on the amount of files to check. With the verbose option, fsck has to generate this information for all directories and files. This can impact performance significantly, especially for file systems with numerous files.
Fsck processing:

Fsck attempts to validate all of a file system’s metadata. Part of this process includes validation of metadata block free lists. The free block itself has to be checked to ensure that is it the correct type of metadata block and the free list has to be checked to ensure that there are no missing entries and that there are no loops in the list. Since files can be deleted in any order, these free blocks can be located anywhere on the volume. Processing and validating these lists can produce lots of random I/O due to the random nature of file deletion. If fsck takes too long it is ok to interrupt it given it’s running in check mode.

fsck checks and repairs an existing Oracle ACFS. This command can only be run on a dismounted file system. root privileges are required to run fsck. The Oracle ACFS driver must be loaded for fsck to work. To confirm or discard any consistency problem on the affected ACFS filesystem, run FSCK as follows:

1. Dismount the ACFS filesystem on each node as follow (as root user) and keep them dismounted to avoid corruption. Otherwise will get warning:

fsck.acfs: ACFS-00511: /dev/asm/ is mounted on at least one node of the cluster.
#  /usr/sbin/umountall -F acfs

Or

# /usr/sbin/umount [MOUNT_POINT]

Where,
MOUNT_POINT – is your ACFS filesystem.

2. Then, execute FSCK on the associated ADVM volume as follow (from each node if it RAC):

On Linux

# script /tmp/fsck_[node name].txt
# /sbin/fsck -a -y -t acfs /dev/asm/[VOLUME_NAME]
# exit

On Solaris

# script /tmp/fsck_[node name].txt
# /usr/sbin/fsck -F acfs -y -o a,v /dev/asm/[VOLUME_NAME]
# exit

On IBM-AIX

# /usr/sbin/fsck -V acfs -y -o a,v /dev/asm/[VOLUME_NAME]

Where,
/dev/asm/[VOLUME_NAME – is your ADVM volume.

IMPORTANT: By default, fsck only checks for and reports any errors. In check mode fsck can be cancelled if it is taking a long time. The -a flag must be specified to instruct fsck to repair errors in the file system. If check mode completed in a reasonable amount of time, and if it reported problems, run fsck in repair mode. In repair mode fsck cannot be interrupted without risk of leaving the file system in a worse state (loss of data depending on the nature of the corruption).

# /sbin/fsck -a -y -t acfs /dev/asm/[VOLUME_NAME]

3. Then mount back the ACFS filesystem on each node if it is RAC. It is always better to manually mount after fsck repairs.

# /usr/sbin/mount -v acfs /dev/asm/[VOLUME_NAME] [MOUNT_POINT]

Where,
/dev/asm/[VOLUME_NAME] – is your ADVM volume.
MOUNT_POINT – is your ACFS filesystem.

Or

# /usr/sbin/mount -v acfs -o all none none

Example:
– Locate correct volume with ‘/sbin/acfsutil info fs‘ and execute fsck.

[HOSTNAME]:(+ASM2)/opt/oracle > /sbin/acfsutil info fs
/acfs1
  ACFS Version: 11.2.0.4.0
  flags:        MountPoint,Available,Corrupt
  mount time:   Fri Jun 10 17:29:14 2016
  volumes:      1
  total size:   xxxxxxxxxxxx
  total free:   xxxxxxxxxxxx
  primary volume: /dev/asm/[VOLUME_NAME2]

Example of an clean output:

Script command is started on Fri May 18 10:26:18 EDT 2016.

[/dev/asm]# /usr/sbin/fsck -V acfs -y -o a,v /dev/asm/[VOLUME_NAME3]

version                   = 11.2.0.3.0
fsck: temporary directory '/usr/tmp'
fsck: current directory '/dev/asm'
*****************************
********** Pass: 1 **********
*****************************
fsck: file system check starting for volume: /dev/asm/[VOLUME_NAME3]
Oracle ASM Cluster File System (ACFS) On-Disk Structure Version: 39.0
Volume indicates the on-disk version is 39.0
Volume indicates the on-disk version is 39.0
ACFS file system created at: Tue Apr 24 22:37:46 2012
checking primary file system
fsck: Volume_Log recovery on node 3 not needed

fsck: checking File_Entry_Table entry: 28 (0x1c) at disk offset: 83968 (0x14800)
fsck: checking File_Entry_Table entry: 29 (0x1d) at disk offset: 84480 (0x14a00)
fsck: checking File_Entry_Table entry: 30 (0x1e) at disk offset: 84992 (0x14c00)
fsck: checking File_Entry_Table entry: 31 (0x1f) at disk offset: 85504 (0x14e00)

fsck: check for unprocessed File_Entry_Table entries complete
      Files checked in primary file system: 100%

       Checking if any files are orphaned...

Phase 1 Orphan check...

Phase 2 Orphan check...

       0 orphans found

Checker completed with no errors.

[/dev/asm]# exit

Script command is complete on Fri May 18 10:27:14 EDT 2012.

Example of an output when fsck is fixing corruption:

fsck.acfs: file system check starting for volume: /dev/asm/[VOLUME_NAME4]
Oracle ASM Cluster File System (ACFS) On-Disk Structure Version: 39.0
Volume indicates the on-disk version is 39.0
Volume indicates the on-disk version is 39.0
ACFS file system created at: Fri Jun  3 15:18:10 2011
checking primary file system
fsck.acfs: Volume_Log recovery on node 1 not needed
fsck.acfs: Volume_Log recovery on node 2 not needed

fsck.acfs: ACFS-07592: metadata structure has incorrect header for:
ACFS Internal Structure: [ACFS Free Block]
    file identifier: 1280 (0x500)    disk offset: 24006656 (0x16e5000)
Parent Structure: [ACFS Local Free List]
    file identifier: 33 (0x21)    disk offset: 193024 (0x2f200)

                   Value Found              Expected Value
                   ------------------------ ------------------------
  File Identifier: 1280 (0x500)                  1280 (0x500)
  Struct_Type:     0x0f500004                    0x0f500007  
                   (OFS_FILE_ENTRY)              (OFS_FETA_FREE_BLOCK)
  Endian_Format:   Little Endian                 Little Endian
  OSCreatedOn:     Linux                         Linux
  Struct_Version:  1                             1
  FileSystemID:    1315928241 (0x4e6f78b1)       1315928241 (0x4e6f78b1)
  CheckSum:        1731346914 (0x673241e2)       1731346914 (0x673241e2)

fsck.acfs: checking directory:
for file:
ACFS Internal File: [ACFS Root Directory]
    file identifier: 2 (0x2)    disk offset: 70656 (0x11400)
fsck.acfs: checking file '/.ACFS'
fsck.acfs: checking directory:
for file:
File: '/.ACFS'
    file identifier: 7 (0x7)    disk offset: 73216 (0x11e00)
Parent Directory: [ACFS Root Directory]
    file identifier: 2 (0x2)    disk offset: 70656 (0x11400)
fsck.acfs: checking file '/.ACFS/.fileid'
fsck.acfs: checking directory:

.............

fsck.acfs: checking file '/home/[PATH]/dirchk/REP_UR.cpr'
fsck.acfs: checking file '/home/[PATH]/dirchk/REP_UR.cps'
fsck.acfs: checking file '/home/[PATH]/dirchk/EXT_AL.cps'
fsck.acfs: checking file '/home/[PATH]/dirdat'
fsck.acfs: checking directory:
for file:
File: '/home/[PATH]/dirdat'
    file identifier: 45 (0x2d)    disk offset: 199168 (0x30a00)
Parent Directory: /home/[PATH]
    file identifier: 43 (0x2b)    disk offset: 198144 (0x30600)
fsck.acfs: checking file '/home/[PATH]/dirdef'
fsck.acfs: checking directory:
for file:
File: '/home/[PATH]/dirdef'
    file identifier: 46 (0x2e)    disk offset: 199680 (0x30c00)
Parent Directory: /home/[PATH]
    file identifier: 43 (0x2b)    disk offset: 198144 (0x30600)
fsck.acfs: checking file '/home/[PATH]/dirpcs'
fsck.acfs: checking directory:
for file:
File: '/home/[PATH]/dirpcs'
    file identifier: 47 (0x2f)    disk offset: 200192 (0x30e00)
Parent Directory: /home/[PATH]
    file identifier: 43 (0x2b)    disk offset: 198144 (0x30600)
fsck.acfs: checking file '/home/[PATH]/dirpcs/MGR.pcm'
fsck.acfs: checking file '/home/[PATH]/macro'
fsck.acfs: checking directory:

...........

fsck.acfs: ACFS-07711: orphan metadata structure (type: 0x0f500007  (OFS_FETA_FREE_BLOCK)) found for file identifier: 2142 (0x85e) at disk offset: 24448000 (0x1750c00)

fsck.acfs: ACFS-07711: orphan metadata structure (type: 0x0f500007  (OFS_FETA_FREE_BLOCK)) found for file identifier: 2143 (0x85f) at disk offset: 24448512 (0x1750e00)

fsck.acfs: ACFS-07711: orphan metadata structure (type: 0x0f500007  (OFS_FETA_FREE_BLOCK)) found for file identifier: 2339 (0x923) at disk offset: 61203968 (0x3a5e600)

    159 orphans found

fixing file system

problems processed: 5%
problems processed: 10%
problems processed: 15%
problems processed: 20%

.............

problems processed: 90%
problems processed: 95%
problems processed: 100%
........

fsck.acfs: check for unprocessed File_Entry_Table entries complete
       Files checked in primary file system: 55%
       Files checked in primary file system: 60%
       Files checked in primary file system: 65%
       Files checked in primary file system: 70%
       Files checked in primary file system: 75%
       Files checked in primary file system: 80%
       Files checked in primary file system: 85%
       Files checked in primary file system: 90%
       Files checked in primary file system: 95%
       Files checked in primary file system: 100%

   Checking if any files are orphaned...

Phase 1 Orphan check...

Phase 2 Orphan check...

    0 orphans found

fsck.acfs: Checker/Fixer completed with the following results:
    File System Errors:  168
    Fixed:               168
    Not Fixed:           0

By default, fsck only checks for and reports any errors. The -o a option must be specified to instruct fsck to fix errors in the file system. In a few cases, fsck prompts for questions before proceeding to check a file system. These cases include:

  • If fsck detects that another fsck is in progress on the file system.
  • If fsck detects that the Oracle ACFS driver is not loaded.
  • If the file system does not appear to be Oracle ACFS.

In checking mode, fsck also prompts if there are transaction logs that have not been processed completely due to an incomplete shutdown. To run in a non-interactive mode, include either the -y or -n options to answer yes or no to any questions.

Filed Under: ASM, oracle

Some more articles you might also be interested in …

  1. What is a FACTLESS FACT TABLE? Where we use Factless Fact
  2. How to Install and configure OSWatcher Black Box (OSWbb)
  3. Oracle SQL : Script To Convert Hexadecimal Input Into a Decimal Value
  4. Oracle OS watcher (OSWatcher) – Understanding oswmpstat
  5. How to check the status of OMS and Agent in OEM 13cR2
  6. RMAN ‘Duplicate From Active Database’ Feature in Oracle 11g
  7. Oracle Interview Questions – Flash Recovery Area
  8. SQL Script to Monitor Usage of Indexes
  9. How to split BCV and open oracle ASM database
  10. Oracle RMAN 12c – New Features

You May Also Like

Primary Sidebar

Recent Posts

  • ncat Command Examples in Linux
  • ncat: command not found
  • nautilus Command Examples in Linux
  • namei: command not found

© 2023 · The Geek Diary

  • Archives
  • Contact Us
  • Copyright