Recently, we encountered a critical issue in one of our Oracle Restart environments (version 19.26 RU) where the SUNDNELLY database suddenly stopped working and all the ASM diskgroups were dismounted.

We had two diskgroups configured:

• DATA
• RECO

Problem Summary

Upon initial investigation, ASM appeared to be running fine. However, when attempting to manually mount the DATA diskgroup, we encountered the following error:

ERROR: /* ASMCMD cguid:f7d77 nodename:<NODE> */ALTER DISKGROUP DATA MOUNT
2025-04-10T23:20:31.352392+05:30
SQL> /* ASMCMD cguid:f7d77e7c8ea6df7 nodename:<NODE> */ALTER DISKGROUP DATA MOUNT
NOTE: cache registered group DATA 1/0x5D701EFD
NOTE: cache began mount (first) of group DATA 1/0x5D701EFD
ERROR: no read quorum in group: required 2, found 0 disks ◄▬▬▬▬▬▬
NOTE: cache dismounting (clean) group 1/0x5D701EFD (DATA)
NOTE: messaging CKPT to quiesce pins Unix process pid: 29162, image: oracle@<NODE> (TNS V1-V3)
NOTE: dbwr not being msg'd to dismount
NOTE: LGWR not being messaged to dismount
NOTE: cache dismounted group 1/0x5D701EFD (DATA)
NOTE: cache ending mount (fail) of group DATA number=1 incarn=0x5d701efd
NOTE: cache deleting context for group DATA 1/0x5d701efd
GMON dismounting group 1 at 20 for pid 26, osid 29162
ERROR: diskgroup DATA was not mounted ◄▬▬▬▬▬▬
ORA-15032: not all alterations performed
ORA-15017: diskgroup "DATA" cannot be mounted ◄▬▬▬▬▬▬
ORA-15040: diskgroup is incomplete ◄▬▬▬▬▬▬

Error Breakdown

• ORA-15032: not all alterations performed
• ORA-15017: diskgroup “DATA” cannot be mounted
• ORA-15040: diskgroup is incomplete

Root Cause Analysis

We checked the disk availability using blkid:

root@<NODE># blkid | grep oracleasm
/dev/sdp: TYPE="oracleasm"
/dev/sdm: TYPE="oracleasm"
/dev/sdr: TYPE="oracleasm"
/dev/sds: TYPE="oracleasm"

Although the root user could see all Oracle ASM disks, the grid user (responsible for ASM) could not access them.

On further inspection, we identified that ownership and group permissions were incorrect. All ASM disks were owned by root:disk, which prevented the grid user from accessing them.

Solution

We corrected the ownership of the ASM disks:

# Change owner and group for all Oracle ASM disks
chown oracle:oinstall /dev/sd{m,p,r,s}

Once ownership was updated, the grid user was able to access the disks, and we successfully mounted the diskgroups:

sqlplus / as sysasm
SQL> ALTER DISKGROUP DATA MOUNT;
SQL> ALTER DISKGROUP RECO MOUNT;

After correcting ownership, the diskgroups were mounted successfully, and the database resumed normal operations.

Key Takeaways

• Always ensure ASM disks have proper ownership and group permissions (oracle:oinstall or as per your environment setup).
• Oracle Restart environments are sensitive to disk access permissions, even if ASM appears to be running normally.
• It’s a good practice to verify disk visibility from both root and grid users.