A Quick Reference for Oracle Database 10g RAC on Linux and Unix Platforms

A Quick Reference for Oracle Database 10g RAC on Linux and Unix Platforms
Introduction
This paper will discuss and demonstrate some of the basic RAC management commands you might use to manage your Oracle 10g RAC components. The material presented will be applicable to most if not all Linux and Unix platforms. It will not cover RAC on the Microsoft Windows operating system. We will begin with the basics of checking out a RAC system to identify whether or not the appropriate services and resources are running or not. Then we will go through the basic startup and shutdown commands for all services and resources. Lastly, we will go over the startup of individual resources and checking the status of each step as we progress. The shutdown of individual resources can be done by reversing the individual startup order.
About our Environment

The RAC environment that we will reference throughout this paper is composed of two nodes running Red Hat Enterprise Linux 4 ES with a shared disk, and has two ASM instances named ASM1 and ASM2, two database instances named orcl1 and orcl2, and a service named RAC that is used for transparent application failover (TAF) and load balancing.

Overview of Basic RAC Management Commands
The commands we will use are listed below. Remember that this document is a quick reference, and not an exhaustive list of all commands for managing your RAC environment.
Cluster Related Commands
crs_stat -t Shows HA resource status (hard to read)
crsstat Ouptut of crs_stat -t formatted nicely
ps -ef|grep d.bin crsd.bin evmd.bin ocssd.bin
crsctl check crs CSS,CRS,EVM appears healthy
crsctl stop crs Stop crs and all other services
crsctl disable crs* Prevents CRS from starting on reboot
crsctl enable crs* Enables CRS start on reboot
crs_stop -all Stops all registered resources
crs_start -all Starts all registered resources

* These commands update the file /etc/oracle/scls_scr//root/crsstart which contains the string “enable” or “disable” as appropriate.
Database Related Commands
srvctl start instance -d -i Starts an instance
srvctl start database -d Starts all instances
srvctl stop database -d Stops all instances, closes database
srvctl stop instance -d -i Stops an instance
srvctl start service -d -s Starts a service
srvctl stop service -d -s Stops a service
srvctl status service -d Checks status of a service
srvctl status instance -d -i Checks an individual instance
srvctl status database -d Checks status of all instances
srvctl start nodeapps -n Starts gsd, vip, listener, and ons
srvctl stop nodeapps -n Stops gsd, vip and listener

Keep in mind that some resources will not start unless other resources are already online. We will now look at the general dependency list in greater detail.
There are three main background processes you can see when doing a ps –ef|grep d.bin. They are normally started by init during the operating system boot process. They can be started and stopped manually by issuing the command /etc/init.d/init.crs {start|stop|enable|disable}
1. /etc/rc.d/init.d/init.evmd
2. /etc/rc.d/init.d/init.cssd
3. /etc/rc.d/init.d/init.crsd
Once the above processes are running, they will automatically start the following services in the following order if they are enabled. This list assumes you are using ASM and have a service set up for TAF/load balancing.
1. The nodeapps (gsd, VIP, ons, listener) are brought online.
2. The ASM instances are brought online.
3. The database instances are brought online.
4. Any defined services are brought online.

Basic RAC Management Commands
Now that we know the dependency tree and have some commands at our disposal, let’s have a look at them one at a time, starting with the cluster commands and processes.
crs_stat -t
This command shows us the status of each registered resource in the cluster. I generally avoid this command because its output is hard to read since the names are truncated as you can see in the sample output below. You can download a helpful script called crsstat from http://www.dbspecialists.com/specialists/specialist2007-05.html to make it easy on your eyes.
[oracle@green ~]$ crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....SM1.asm application ONLINE ONLINE green
ora....EN.lsnr application ONLINE ONLINE green
ora.green.gsd application ONLINE ONLINE green
ora.green.ons application ONLINE ONLINE green
ora.green.vip application ONLINE ONLINE green
ora.....RAC.cs application ONLINE ONLINE red
ora....cl1.srv application ONLINE ONLINE green
ora....cl2.srv application ONLINE ONLINE red
ora.orcl.db application ONLINE ONLINE red
ora....l1.inst application ONLINE ONLINE green
ora....l2.inst application ONLINE ONLINE red
ora....SM2.asm application ONLINE ONLINE red
ora....ED.lsnr application ONLINE ONLINE red
ora.red.gsd application ONLINE ONLINE red
ora.red.ons application ONLINE ONLINE red
ora.red.vip application ONLINE ONLINE red
oracle@green ~]$
crsstat
The output of this script is much better. You can learn more about this script and download it at http://www.dbspecialists.com/specialists/specialist2007-05.html.
[oracle@green ~]$ crsstat
HA Resource Target State
----------- ------ -----
ora.green.ASM1.asm ONLINE ONLINE on green
ora.green.LISTENER_GREEN.lsnr ONLINE ONLINE on green
ora.green.gsd ONLINE ONLINE on green
ora.green.ons ONLINE ONLINE on green
ora.green.vip ONLINE ONLINE on green
ora.orcl.RAC.cs ONLINE ONLINE on red
ora.orcl.RAC.orcl1.srv ONLINE ONLINE on green
ora.orcl.RAC.orcl2.srv ONLINE ONLINE on red
ora.orcl.db ONLINE ONLINE on red
ora.orcl.orcl1.inst ONLINE ONLINE on green
ora.orcl.orcl2.inst ONLINE ONLINE on red
ora.red.ASM2.asm ONLINE ONLINE on red
ora.red.LISTENER_RED.lsnr ONLINE ONLINE on red
ora.red.gsd ONLINE ONLINE on red
ora.red.ons ONLINE ONLINE on red
ora.red.vip ONLINE ONLINE on red