Automatic Storage Management - Midwest Oracle Users Group

Automatic Storage 

Management 

(ASM)

Speaker Introduction 

Michael Messina 

Senior Managing Consultant Rolta, TUSC Infrastructure 

Services 

Working with Oracle Approximately 17 years 

Background includes Performance Tuning, High Availability 

and Disaster Recovery 

Oracle Database OCP 

Oracle ACE 

Mike.Messina@roltasolutions.com 

www.tusc.com / www.roltasolutions.com

Contents 

Automatic Storage Management (ASM) 

Overview 

ASM Structure / Components 

Benefits of ASM 

Any Issues? 

Infrastructure 

Storage Options 

Redundancy 

External, Normal, High 

Multiple Paths to Storage 

Database Storage 

File Storage (ASM Clustered File System)

Contents 

Volume Management 

Comparison to other Volume Managers 

Use with Other Volume Managers 

How ASM can replace other Logical Volume Managers 

ASM in a High Availability Environment 

Fail Groups 

Real Application Clusters (RAC) Database 

DataGuard 

Prepare Volumes for ASM 

Requirements 

ASMLib for Linux 

Create disks for ASM

Contents 

Installing ASM 

Software and OS Requirements for Oracle Enterprise Linux 

Installation Basics 

ASM Command Line Utility 

What is the ASM Command Line Utility 

ASMCMD Commands

Contents 

Working with ASM Disk Groups 

Discovering Disks 

Mounting/Un-mounting Disk Groups 

Create ASM Disk Groups 

Add Disk/Space to ASM Disk Group 

Remove Disk/Space from ASM Disk Group 

Change ASM Disk Group Compatibility 

Rename Disk Group 

Managing ASM File Access Control for Disk Groups 

Drop ASM Disk Group

Contents 

ASM Cluster File System 

Overview 

Oracle ACFS Driver Model 

Oracle ACFS Mount Model and Namespace 

Oracle ACFS Mount Registry 

Oracle ACFS Snapshots 

Oracle ACFS and Backup and Restore 

Oracle ACFS Integration with ASM 

Loading ACFS Drivers / Modules 

Create/Manage ACFS File System

AUTOMATIC STORAGE 

MANAGEMENT (ASM)

Overview 

A Volume Manager for Databases 

A File System for Databases 

11gR2 a File System 

11gR2 a Cluster File System


ASM Instance 

ASM Disk Groups 

ASM Disks 

ASM Files 

Control files 

Data files, temporary data files, and data file copies 

SPFILEs 

Online redo logs, archive logs, and Flashback logs 

RMAN backups 

Disaster recovery configurations 

Change tracking bitmaps 

Data Pump dumpsets


September 12 / Slide 11 / Title of Document

Benefits of ASM 

Dynamically Add and Remove Disk/Space while database is running 

Works with Oracle Managed Files (OMF) 

ASM can coexist with other storage types 

ASM Can Improve I/O Performance 

When in a Cluster ASM Supports Rolling Upgrades 

ASM Reduces Management and Administrative Overhead 

ASM can support storage for multiple Databases and Database Versions 

Flexible Redundancy Options 

Oracle Enterprise Manager Integration 


Any Issues? 

ASM Requires additional Home location for single instance 

implementations, meaning additional space for install is required. 

For Normal or High Redundancy Disk Groups the disk for the failgroup(s) 

are required when the disk group is created. 

RMAN is required to backup databases using ASM. 

ASM does not provide any power management features, all power 

management features are controlled outside the ASM technology stack. 


INFRASTRUCTURE 


Storage Options 

ASM works with virtually any storage type 

Direct Attached Storage 

Network Attached Storage 

Storage Area Networks 

Solid State Disk 

Storage regardless of the type is presented to the server as 

a RAW device(s) 

Operating System on the Server Controls disk use 


Redundancy 

External 

Disk hardware controls the Redundancy 

Normal 

1 disk mirror or failgroup 

High 

2 disk mirrors or failgroups 


Redundancy 

Multipathing 

provide failover by using redundant physical paths between server 

and the storage subsystem. 

If one or more components fails, then applications can still access 

data on the storage 

Multipathing used with Storage Area Network (SAN), Host 

Bus Adapter, interface cable, or host port on a multiported 

storage array. 


Database Storage 

Disks assigned to ASM Disk Groups 

ASM Instance manages database storage to disk groups 

Database datafiles, tempfiles, controlfiles, spfiles and redo 

logs stored in ASM Disk Groups 

Database can store files in one or more Disk Groups 


File Storage (ASM Clustered File System) 

ASM Clustered File System (ACFS) 

11gR2 added the capability for ASM to act as a file system for 

all operating system files with ACFS 

Can act as a local filesystem or a cluster filesystem 

Works natively at the OS layer to act just like any other 

filesystem 

ASM Creation Assistant can be used to create ACFS 

filesystems 

ASM command line utility can be used to create and manage 

ACFS file systems (asmcmd) 


VOLUME MANAGEMENT 


Comparison to other Volume Managers 

ASM Requires RMAN for Database Backups while other 

volume managers will allow non-RMAN backups. 

ASM is being taught with all Oracle Database Training and it 

Oracle’s Direction for the future. Other volume managers 

may require additional training to utilize effectively. 

ASM Cluster File System adds capability to utilize ASM as a 

regular native OS filesystem. 

ASM is a no-cost cluster file system option, 3 rd party volume 

managers require additional license purchase and support 

costs. 

ASM typically provides better performance

Use with Other Volume Managers 

ASM co-exists with other cluster file system or native OS file 

systems. 

Mix database and non-database files on different filesystems 

Use ASM for Database files and non-database files 

Use OCFS2, Veritas, ext3, etc. database and non database files 

Use ASM, OCFS2, Veritas, etc. for database and non-database files 

Move database and non-database files between different 

filesystems including ASM.

How ASM can replace other Logical Volume Managers 

11gR2 adds ASM Cluster File System expanding the capability 

of ASM to be used as a native OS filesystem. 

Create ASM file systems/volumes for databases 

Create ACFS file systems which use ASM for storage for nondatabase 

files. 

ACFS provides full volume management for virtually all file 

types. 

ASM with ACFS however can not replace the native OS 

volume manager for root and boot volumes and ASM/ACFS 

was not designed to support root or boot volumes for the 

operating system but to supplement and add additional 

capabilities

ASM IN A HIGH AVAILABILITY 

ENVIRONMENT

Fail Groups 

FailGroups are utilized in NORMAL and HIGH Redundancy 

Disk Groups. 

FailGroups provide Disk Group storage protection using 

mirrors within Disk Groups 

FailGroups can be located on multiple storage systems 

protecting against storage system failure 

Failgroups can be located at a separate site based on 

network and latency conditions.

Real Application Clusters (RAC) Database 

ASM was designed to provide a Highly Available Clustered 

Storage for Real Application Clusters 

Provide highest availability for RAC 

External storage Redundancy using RAID 10, RAID 5, RAID 6 to 

protect against individual disk failures 

ASM FailGroups to protect against Storage System Failures, storage 

systems can be located at separate sites depending on network 

capability and latency. 

Multiple Nodes in cluster provide protection against node 

failure.

DataGuard 

Data Guard Provides for long distance cross site failover 

Uses ASM effectively 

ASM can simplify the building and maintaining Data Guard 

Standby database 

RAC capable standbys from RAC primary or single instance 

standby from RAC primary. 

Closes the maximum availability and scalability gap that RAC 

and ASM alone can not provide.

PREPARE VOLUMES FOR ASM

Requirements 

For Linux ASMLib utility is not required, however 

recommended and considered best practice. ** Not support 

in Red Hat 6 for ASMLib going forward based on most 

recent information 

For Linux ASMLib should be installed and configured 

RAW volumes presented to the host/operating System 

RAW volumes partitioned 

Permissions and ownership of volumes for oracle and dba or 

ASM Admin group 

Make permissions and ownership of volumes persistent 

across host restarts

ASMLib for Linux 

Check Installation 

rpm –qa | grep oracleasm 

Install ASMLib 

http://www.oracle.com/technology/software/tech/linux/asmlib/rhel5.html 

rpm -i oracleasm-support-2.1.3-1.el5.i386.rpm 

rpm -i oracleasm-2.6.18-128.1.6.el5-2.0.5-1.el5.i686.rpm 

rpm -i oracleasmlib-2.0.4-1.el5.i386.rpm 

rpm -i oracleasm-2.6.18-128.1.6.el5debug-2.0.5-1.el5.i686.rpm 

rpm -i oracleasm-2.6.18-128.1.6.el5-debuginfo-2.0.5-1.el5.i686.rpm 

rpm -i oracleasm-2.6.18-128.1.6.el5PAE-2.0.5-1.el5.i686.rpm 

rpm -i oracleasm-2.6.18-128.1.6.el5xen-2.0.5-1.el5.i686.rpm 

Configure ASMLib 

/etc/init.d/oracleasm configure

Create disks for ASM 

Partition RAW volumes to be used for ASM 

fdisk /dev/sda 

Change Group at min and permissions on volume and 

partition 

chmod 660 /dev/sda 

chmod 660 /dev/sda1 

chown oracle:dba /dev/sda 

chown oracle:dba /dev/sda1 

Place Group and permission changes in /etc/rc.d/rc.local 

Initialize partitions on devices into ASM Disks 

/etc/init.d/oracleasm createdisk ASMDSK1 /dev/sda1

INSTALLING ASM

Software and OS Requirements for Oracle Enterprise 

Linux 

OS Requirements Enterprise Linux 

At least 1GB of free available memory on server 

At least 1.8GB of free swap 

Oracle user 

Oracle dba group 

Ensure the following exist in the /etc/security/limits.conf 

* hard nproc 16384 

* hard nofile 65536 

Ensure that the limits are configured to set for new sessions by 

ensuring the following are in the /etc/pam.d/system-auth file. 

session required pam_limits.so 

session required pam_unix.so 



Linux 

Ensure that the following are included in the .bash_profile for the 

oracle user 

ulimit -n 65536 

ulimit -u 16384 

Ensure oracle primary group dba 

$ usermod –g dba oracle 

set the following in the /etc/sysctl.conf 

kernel.msgmnb = 65536 

kernel.msgmax = 65536 

kernel.shmmax = 4294967295 

kernel.shmall = 268435456 

kernel.sem=250 32000 100 128 

net.ipv4.ip_local_port_range = 1024 65000 < 11gR1 

net.ipv4.ip_local_port_range = 9000 65500 >= 11gR2 

net.core.rmem_default=4194304 

net.core.wmem_default=262144 

net.core.rmem_max=4194304 

net.core.wmem_max=1048576 

fs.file-max = 6815744 

fs.aio-max-nr = 1048576 



Linux 

CD3 

rpm -Uvh libaio-0 

rpm -Uvh libaio-devel-0.3.105-2.i386.rpm 

CD4 

rpm -Uvh elfutils-libelf-0 

rpm -Uvh elfutils-libelf-devel-0.91.1-3.i386.rpm 

CD1 

rpm -Uvh unixODBC-2.2.11-1.RHEL4.1.i386.rpm 


Installation Basics 

Ensure OS compatibility for the ASM version to be installed 

Ensure all OS pre-reqs are satisfied 

For ASM the latest ASM version is recommended to be used 

even for lower versions of the database software 

Obtain the latest ASM Software currently 11.2 and is included 

in the Grid Infrastructure Software Installation 

The Oracle Universal Installer greatly simplifies the 

installation process for the Database and Grid Infrastructure 

software.

ASM COMMAND LINE UTILITY

What is the ASM Command Line Utility 

Command line interface for management of ASM disk 

groups and ACFS volumes 

Run from Grid Infrastructure home 

/bin/asmcmd

ASMCMD Commands 

Instance Management 

iostat Displays statistics for mounted devices. 

lsct Lists information about current ASM clients. 

lsop Lists the current operations on a disk group, database, or 

ASM instance. 

lspwusr List the users from an Oracle PWFILE file. 

orapwusr Add, drop, or change an Oracle PWFILE user. 

shutdown Shuts down an ASM instance. 

spcopy Copies a SPFILE. 

startup Starts up an ASM instance.


File Access 

chgrp Changes the group of a file. 

chmod Changes permissions of a file. 

chown Changes the owner of a file. 

groups List the user groups that a user belongs to. 

grpmod Adds or removes users from an existing user group. 

lsgrp Lists ASM access control list (ACL) groups. 

lsusr Lists users in a disk group. 

mkgrp Creates a new user group. 

mkusr Adds a user to disk group. 

passwd Changes the password of a user in a disk group. 

rmgrp Deletes a user group. 

rmusr Deletes a user from a disk group.


File Management 

cd Changes the current directory to the specified directory. 

du Displays the total disk space occupied by ASM files in the 

specified ASM directory and all of its subdirectories, recursively. 

find Lists the paths of all occurrences of the specified name (with 

wildcards) under the specified directory. 

ls Lists the contents of an ASM directory, the attributes of the 

specified file, or the names and attributes of all disk groups. 

lsof Lists the open files. 

mkalias Creates an alias for system-generated filenames. 

mkdir Creates ASM directories. 

pwd Displays the path of the current ASM directory. 

rm Deletes the specified ASM files or directories. 

rmalias Deletes the specified alias, retaining the file alias points to.


Disk Group Management 

chdg Changes a disk group (add, drop, or rebalance). 

chkdg Checks or repairs a disk or failure group. 

cp Enables you to copy files between ASM disk groups on a 

local instance and remote instances. 

dgdrop Drops a disk group. 

lsattr Lists the attribute and attribute settings of a disk group. 

lsdg Lists disk groups and their information. 

lsdsk Lists disks visible to ASM. 

lsgrp Lists ASM access control list (ACL) groups. 

md_backup Creates a backup of all of the mounted disk groups. 

md_restore Restores disk groups from a backup. 

mkdg Creates a disk group. 

mount Mounts a disk group.


Disk Group Management (cont.) 

offline Offline a disk or a failure group that belongs to a disk 

group. 

online Onlines a disk or a failure group that belongs to a disk 

group. 

rebal Rebalances a disk group. 

remap Repairs a range of physical blocks on a disk. 

setattr Sets attributes in an ASM disk group. 

umount Dismounts a disk group.


ACFS – ASM Clustered File System 

volcreate Create a volume 

voldelete Delete a volume 

voldisable Disable a volume. 

volenable Enable a Volume 

volinfo Volume Information, 

volresize Resize a volume 

volset Change existing attribute 

volstat Report volume I/O statistics.

WORKING WITH ASM DISK 

GROUPS 


Discovering Disks 

You can discover disk groups using ASM lib or the ASM 

instance. 

ASMLib 

/etc/init.d/oracleasm scandisks 

/etc/init.d/oracleasm listdisks 

ASM Instance 

SQL> select name, path from v$asm_disk ;


Set ORACLE_HOME to Grid Infrastructure installation 

location 

SET ORACLE_SID to ASM instance name typically +ASM 

A disk group can be created using SQL Plus or the ASM 

command line utility.


External Redundancy 

SQL Plus 

SQL> create diskgroup UNDO external redundancy disk 'ORCL:UNDO1' ; 

asmcmd 

Create diskgroup_config.xml 

•


Normal Redundancy 

SQL Plus 

SQL> create diskgroup test_failgroup normal redundancy 

FAILGROUP fg1 DISK ‘ORCL:ASMDSK1' NAME ASMDSK1 

FAILGROUP fg2 DISK ‘ORCL:ASMDSK2' NAME ASMDSK2 ; 

asmcmd 


• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

• 

ASMCMD> mkdg diskgroup_config.xml


High Redundancy 

SQL Plus 

SQL> create diskgroup test_failgroup high redundancy 

FAILGROUP fg1 DISK '/dev/oracleasm/disks/ASMDSK1' NAME ASMDSK1 

FAILGROUP fg2 DISK '/dev/oracleasm/disks/ASMDSK2' NAME ASMDSK2 

FAILGROUP fg3 DISK '/dev/oracleasm/disks/ASMDSK2' NAME ASMDSK3 ;


asmcmd 


•

Mounting/Un-mounting Disk Groups 

Mounting and Un-mounting Disk Groups can be done via 

SQL Plus or the asmcmd utility 

Mount 

SQL Plus 

• SQL> alter diskgroup undo mount ; 

asmcmd 

• asmcmd> mount undo ; 

Un-mount 

SQL Plus 

• SQL> alter diskgroup undo dismount ; 

asmcmd 

• asmcmd> umount undo ;

Add Disk/Space to ASM Disk Group 

Space is added to a disk group by the addition of ASM 

disks/volumes. 

SQL Plus 

SQL> alter diskgroup test add disk ‘ORCL:ASMDSK5‘

Remove Disk/Space from ASM Disk Group 

Space is removed from an ASM Disk Group by removing 

disks from the diskgroup. 

SQL Plus 

SQL> alter diskgroup test_failgroup drop disk ASMDSK2 ;

Change ASM Disk Group Compatibility 

The Disk Group Compatibility can be change after the disk 

group is created provided it does not exceed compatibility in 

the environment being used. The COMPATIBLE.ASM and 

COMPATIBLE.RDBMS dictate the instance and RDBMS 

capability instance parameter settings. 

COMPATIBLE.ASM COMPATIBLE.RDBMS ASM Instance RDBMS Instance 

10.1 10.1 >=10.1 >=10.1 

11.1 10.1 >=11.1 >=10.1 

11.2 11.1 >=11.2 >=11.1 

11.2 11.2 >=11.2 >=11.2 

Disk Group can only be changed upward (so be careful) 

SQL Plus 

SQL> ALTER DISKGROUP data SET ATTRIBUTE 'compatible.asm' = '11.1';

Rename Disk Group 

Starting will 11g Release 2 a disk group can be renamed using 

the renamedg command line utility 

asmcmd 

renamedg dgname=olddiskgroup newdgname=newdiskgroup 

** Note: If Disk Group Contains Database Files the database files 

will need to be relocated. This will require off lining the 

tablespaces that have datafiles on the diskgroup or shutting 

the database down prior to renaming the diskgroup. Then 

once diskgroup is rename relocating all files with alter 

database rename command for new Disk Group Name.


Managing file and directory is done using the asmcmd 

command line utility 

Managing access in asm is much like managing file and 

directory access in unix operating systems. 

You have users, groups, directory and file permissions 

controlled through the use of commands 

mkusr 

mkgrp 

rmgrp 

rmusr 

chown 

chgrp 

chmod


mkusr – Create New User 

asmcmd> mkusr dg1 myuser ; 

mkgrp – Create New Group 

asmcmd> mkgrp dg1 mygroup ; 

grpmod – Add/Remove Users for Group 

-a Add User -d Remove User 

grpmod –a dg1 mygroup myuser 

grpmod –d dg1 mygroup myuser 

rmgrp – Remove Group 

asmcmd> rmgrp dg1 mygroup ; 

rmusr – Remove User 

asmcmd> rmusr dg1 myuser ;


chown 

ASMCMD> chown db1 +data/orcl/datafile/* 

ASMCMD> chown –R db1 +data/orcl 

ASMCMD> chown db1:backup_user +data/controlfile.f 

chgrp 

asmcmd> chgrp backup_users +data/controlfile.f ; 

asmcmd> chgrp –R backup_users +data/* ; 

chmod 

ASMCMD> chmod 644 +data/controlfile.f 

ASMCMD> chmod go+rw +data/controlfile.f 

ASMCMD> chmod ug-w +data/orcl/datafile/* 

ASMCMD> chmod –R 600 +data/orcl

Drop ASM Disk Group 

A disk group can be dropped using either SQL Plus or the 

asmcmd command line utility 

SQL Plus 

DROP DISKGROUP dg1 ; 

** If Disk Group has files then use the INCLUDING CONTENTS clause 

SQL> DROP DISKGROUP disk_group_1 INCLUDING CONTENTS; 

asmcmd 

asmcmd> dgdrop dg1 ; 

** If Disk Group has files then use the –r option 

asmcmd> dgdrop –r dg1 ;

ASM CLUSTER FILE SYSTEM

Overview 

Requires installation of 11g Release 2 (11.2) Grid Infrastructure Software 

ACFS executes as a UNIX (POSIX and X/OPEN compatible) file system for 

Linux and UNIX and ACFS works as a Windows Filesystem on Windows. 

Applications and OS commands work natively against ACFS 

Exabyte capable files and file system capacities on 64 bit platforms 

Peer to peer, multi-node shared filesystem with direct, coherent and 

cache I/O to ASM Storage. 

Works as a Clustered or non-clustered filesystem 

Oracle ACFS are provided with “single node file system coherency” for 

single host or when distributed across cluster members. 

Oracle ACFS file system can be used for an Oracle Database Home file 

system.

Oracle ACFS Driver Model 

Oracle ACFS is installed as a dynamically loadable Operating 

System filesystem driver. 

Driver is implemented as a Virtual File System (VFS) 

Integrated with the native operating system

Oracle ACFS Mount Model and Namespace 

Hierarchical filesystem 

Files 

Sub-directories 

Tree-structured Namespace 

Files are the leaf nodes 

Single-filesystem naming model 

Not designed to be a root filesystem or bootable.

Oracle ACFS Mount Registry 

Multi-platform mount registration facility 

Automatically mount file systems on all nodes in a cluster 

that are registered 

Clustered and non-clustered filesystems are registered in the 

Mount Registry, most useful for clusters really 

Clustered see acfsutil command 

Non-clustered see srvctl add filesystem command 

Creates the mount point if it does not exist 

Registry mount actions will automatically mount associated 

ASM disk groups

Oracle ACFS Snapshots 

Point in time copy of an ACFS filesystem 

Works very similar to SAN Volume or Flash Copies 

Snapshots are immediately available for use after creation 

Always online while the file system is mounted 

Snapshot storage is maintained within the file system 

Snapshots are administered with the acfsutil snap 

commands 

63 snapshot views supported for each file system

Oracle ACFS and Backup and Restore 

Uses same backup capabilities as other filesystems for the 

host operating system environment. 

ACFS snapshots can be dynamically created and used to 

present a consistent view of an active file system for a 

backup 

NOTE: Windows backup applications that depend upon 

reparse points or the Windows Volume Shadow Copy 

Service (VSS) may not be supported.

Oracle ACFS Integration with ASM 

ACFS requires ASM for storage 

Always integrated with ASM 

Configured with a traditional device file 

ACFS filesystem inherits ASM storage management features 

like balanced distribution, mirroring, dynamic resizing, etc. 

Driver established connection with ASM instance 

Always unmount ACFS filesystems prior to ASM instance 

shutdown or prior t0 unmounting a disk group to avoid the 

filesystem being put into an error state.

Loading ACFS Drivers / Modules 

To utilize ACFS the ACFS drivers and modules must be 

loaded into the operating system 

For Grid Infrastructure installations in a cluster the drivers and 

modules are loaded automatically 

Single instance Grid Infrastructure installations the ACFS 

Drivers/Modules must be loaded Manaually 

Load ACFS Drivers/Modules Manually 

Log into the host operating system as root 

/bin/acfsload start 

Place the load command into /etc/rc.d/rc.local for the load to be 

persistent across node restarts

Create/Manage ACFS File System 

Must first have a Disk Group for the ACFS file system to use 

Disk Group Compatibility must be 11.2.0 or higher 

SQL> select group_number, name, compatibility, 

database_compatibility from v$asm_diskgroup; 

alter diskgroup avm SET ATTRIBUTE 'compatible.advm' = '11.2.0.0' 

Create an ACFS File System through ASMCA or asmcmd 

Create using asmcmd – ASM command line utility 

volcreate -d diskgroupa -s 10G volume 

Get ACFS volume information 

volinfo –a dg1 

Enable volumes in a Disk Group 

All - volenable –a dg1 

Volume – volenable –d dg1

Create/Manage ACFS File System 

Resize a ACFS volume in a Disk Group 

volresize -d -s 2G [ -f ] 

ACFS Volume Statistics 

volstat -d 

Disable ACFS volume 

voldisable -d 

Remove an ACFS volume 

voldelete -d

Thank You 

Thank you!

Automatic Storage Management - Midwest Oracle Users Group

Create successful ePaper yourself

Delete template?

Save as template?