Build Your Own Oracle RAC 10g Release 2 Cluster On Linux and ISCSI

Build Your Own Oracle RAC 10g Release 2 Cluster on
Linux and iSCSI

by Jeffrey Hunter
Learn how to set up and configure an Oracle RAC 10g Release 2 development cluster for
less than US$2,500.
For development and testing only; production deployments will not be supported!
Updated November 2006
Contents
1. Introduction
2. Oracle RAC 10g Overview
3. Shared-Storage Overview
4. iSCSI Technology
5. Hardware & Costs
6. Install the Linux Operating System
7. Network Configuration
8. Install Openfiler
9. Configure iSCSI Volumes using Openfiler
10. Configure iSCSI Volumes on Oracle RAC Nodes
11. Create "oracle" User and Directories
12. Configure the Linux Servers for Oracle
13. Configure the hangcheck-timer Kernel Module
14. Configure RAC Nodes for Remote Access
15. All Startup Commands for Both Oracle RAC Nodes
16. Install & Configure Oracle Cluster File System (OCFS2)
17. Install & Configure Automatic Storage Management (ASMLib 2.0)
18. Download Oracle 10g RAC Software
19. Pre-Installation Tasks for Oracle10g Release 2
20. Install Oracle 10g Clusterware Software
21. Install Oracle 10g Database Software
22. Install Oracle 10g Companion CD Software
23. Create TNS Listener Process
24. Create the Oracle Cluster Database
25. Verify TNS Networking Files
26. Create / Alter Tablespaces
27. Verify the RAC Cluster & Database Configuration
28. Starting / Stopping the Cluster
29. Transparent Application Failover - (TAF)
30. Troubleshooting
31. Conclusion
32. Acknowledgements
Downloads for this guide:

CentOS Enterprise Linux 4.4 or Red Hat Enterprise Linux 4
Oracle Cluster File System Release 2 - (1.2.3-1)
Oracle Cluster File System Releaase 2 Tools - (1.2.1-1)
Oracle Database 10g Release 2 EE, Clusterware, Companion CD - (10.2.0.1.0)
Openfiler 2.1 x86 (Final)
ASMLib 2.0 Library and Tools - (2.0.3-1) - Tools / Libraries
ASMLib 2.0 Driver - (2.6.9-42.EL / 2.0.3-1) - Single Processor / SMP / Hugemem
1. Introduction
One of the most efficient ways to become familiar with Oracle Real Application Clusters
(RAC) 10g technology is to have access to an actual Oracle RAC 10g cluster. There's no
better way to understand its benefitsincluding fault tolerance, security, load balancing,
and scalabilitythan to experience them directly.
Unfortunately, for many shops, the price of the hardware required for a typical production
RAC configuration makes this goal impossible. A small two-node cluster can cost from
US$10,000 to well over US$20,000. That cost would not even include the heart of a
production RAC environmenttypically a storage area networkwhich can start at
US$10,000.
For those who want to become familiar with Oracle RAC 10g without a major cash
outlay, this guide provides a low-cost alternative to configuring an Oracle RAC 10g
Release 2 system using commercial off-the-shelf components and downloadable software
at an estimated cost of US$2,200 to US$2,500. The system will consist of a dual node
cluster (each with a single processor), both running Linux (CentOS 4.4 or Red Hat
Enterprise Linux 4 Update 4), Oracle10g Release 2, OCFS2, and ASMLib 2.0. All shared
disk storage for Oracle RAC will be based on iSCSI using a Network Storage Server;
namely Openfiler Release 2.1. (Of course, you could also consider building a virtual
cluster on a VMware Virtual Machine, but the experience won't quite be the same!)
Note that future releases of this article will be based on Oracle's Enterprise Linux in place
of CentOS and Red Hat Linux. Enterprise Linux will provide the same if not better
stability and will already include the OCFS2 and ASMLib software packages. To learn
more about Enterprise Linux, please visit http://www.oracle.com/linux. Although this
configuration has not been tested with Enterprise Linux, it should work with minor
modifications.
Powered by rPath Linux, Openfiler is a free browser-based network storage management
utility that delivers file-based Network Attached Storage (NAS) and block-based Storage
Area Networking (SAN) in a single framework. Openfiler supports CIFS, NFS,
HTTP/DAV, FTP, however, we will only be making use of its iSCSI capabilities to
implement an inexpensive SAN for the shared storage components required by Oracle10g
RAC. A 500GB external hard drive will be connected to the network storage server
(sometimes referred to in this article as the Openfiler server) via its USB 2.0 interface.
The Openfiler server will be configured to use this disk for iSCSI based storage and will
be used in our Oracle10g RAC configuration to store the shared files required by Oracle
Clusterware as well as all Oracle ASM volumes.
Note: This article is provided for educational purposes only, so the setup is kept simple to
demonstrate ideas and concepts. For example, the disk mirroring was setup on one
physical disk only, while in practice that should be done on at least two physical drives.
This is not the only way to build a low-cost Oracle RAC 10g system. I have worked on
other solutions that utilize an implementation based on SCSI for the shared storage
component. In some cases, SCSI will cost more than the implementation described in this
article where an inexpensive SCSI configuration will consist of:
SCSI Controller:Two SCSI controllers priced from $20 (Adaptec AHA-2940UW)

to $220 (Adaptec 39320A-R) each
SCSI Enclosure: $70 - (Inclose 1 Bay 3.5" U320 SCSI Case)
SCSI Hard Drive: $140 - (36GB 15K 68p U320 SCSI Hard Drive)
SCSI Cables: Two SCSI cables priced at $20 each - (3ft External HD68 to HD68
U320 Cable)
Keep in mind that some motherboards may already include built-in SCSI controllers.
The previous Oracle9i and Oracle 10g Release 1 guides used raw partitions for storing
files on shared storage, but here we will make use of the Oracle Cluster File System
Release 2 (OCFS2) and Oracle Automatic Storage Management (ASM) feature. The two
Oracle RAC nodes will be configured as follows:
Oracle Database Files

File System /
Volume
RAC Node Name Instance Name Database Name $ORACLE_BASE
Manager for
DB Files
linux1 orcl1 orcl /u01/app/oracle ASM
linux2 orcl2 orcl /u01/app/oracle ASM

Oracle Clusterware Shared Files
File Type File Name iSCSI Volume Name Mount Point File System
Oracle Cluster Registry /u02/oradata/orcl/OCRFile crs /u02/oradata/orcl OCFS2
CRS Voting Disk /u02/oradata/orcl/CSSFile crs /u02/oradata/orcl OCFS2
Note that with Oracle Database 10g Release 2 (10.2), Cluster Ready Services, or CRS, is
now called Oracle Clusterware.
Starting with Oracle Database 10g Release 2 (10.2), Oracle Clusterware should be
installed in a separate Oracle Clusterware home directory which is non-release specific.
This is a change to the Optimal Flexible Architecture (OFA) rules. You should not install
Oracle Clusterware in a release-specific Oracle home mount point,
(/u01/app/oracle/product/10.2.0/... for example), as succeeding versions of
Oracle Clusterware will overwrite the Oracle Clusterware installation in the same path.
Also, If Oracle Clusterware 10g Release 2 (10.2) detects an existing Oracle Cluster
Ready Services installation, then it overwrites the existing installation in the same path.
The Oracle Clusterware software will be installed to /u01/app/oracle/product/crs on

both of the nodes that make up the RAC cluster. However, the Clusterware software
requires that two of its files, the "Oracle Cluster Registry (OCR)" file and the "Voting
Disk" file be shared with both nodes in the cluster. These two files will be installed on
shared storage using Oracle's Cluster File System, Release 2 (OCFS2). It is possible (but
not recommended by Oracle) to use RAW devices for these files, however, it is not
possible to use ASM for these two Clusterware files.
The Oracle10g Release 2 Database software will be installed into a separate Oracle
Home; namely /u01/app/oracle/product/10.2.0/db_1 on both of the nodes that
make up the RAC cluster. All of the Oracle physical database files (data, online redo logs,
control files, archived redo logs) will be installed to shared volumes being managed by
Automatic Storage Management (ASM). (The Oracle database files can just as easily be
stored on OCFS2. Using ASM, however, makes the article that much more interesting!)
Note: This article is only designed to work as documented with absolutely no

substitutions. The only exception here is the choice of vendor hardware (i.e. machines,
networking equipment, and external hard drive). Ensure that the hardware you purchase
from the vendor is supported on Red Hat Linux 4.
If you are looking for an example that takes advantage of Oracle RAC 10g Release 2 with
RHEL 4 using FireWire, click here.
If you are looking for an example that takes advantage of Oracle RAC 10g Release 1 with
RHEL 3, click here.
For the previously published Oracle9i RAC version of this guide, click here.
2. Oracle RAC 10g Overview
Oracle RAC, introduced with Oracle9i, is the successor to Oracle Parallel Server (OPS).
RAC allows multiple instances to access the same database (storage) simultaneously.
RAC provides fault tolerance, load balancing, and performance benefits by allowing the
system to scale out, and at the same time since all nodes access the same database, the
failure of one instance will not cause the loss of access to the database.
At the heart of Oracle10g RAC is a shared disk subsystem. All nodes in the cluster must
be able to access all of the data, redo log files, control files and parameter files for all
nodes in the cluster. The data disks must be globally available in order to allow all nodes
to access the database. Each node has its own redo log, control files, and UNDO
tablespace, but the other nodes must be able to access them in order to recover that node
in the event of a system failure.
The biggest difference between Oracle RAC and OPS is the addition of Cache Fusion.
With OPS a request for data from one node to another required the data to be written to
disk first, then the requesting node can read that data. With cache fusion, data is passed
along a high-speed interconnect using a sophisticated locking algorithm.
Not all clustering solutions use shared storage. Some vendors use an approach known as
a "Federated Cluster", in which data is spread across several machines rather than shared
by all. With Oracle10g RAC, however, multiple nodes use the same set of disks for
storing data. With Oracle10g RAC, the data files, redo log files, control files, and
archived log files reside on shared storage on raw-disk devices, a NAS, ASM, or on a
clustered file system. Oracle's approach to clustering leverages the collective processing
power of all the nodes in the cluster and at the same time provides failover security.
Pre-configured Oracle10g RAC solutions are available from vendors such as Dell, IBM
and HP for production environments. This article, however, focuses on putting together
your own Oracle10g RAC environment for development and testing by using Linux
servers and a low cost shared disk solution; iSCSI.
For more background about Oracle RAC, visit the Oracle RAC Product Center on OTN.
3. Shared-Storage Overview
Today, fibre channel is one of the most popular solutions for shared storage. As
mentioned earlier, fibre channel is a high-speed serial-transfer interface that is used to
connect systems and storage devices in either point-to-point (FC-P2P), arbitrated loop
(FC-AL), or switched topologies (FC-SW). Protocols supported by Fibre Channel include
SCSI and IP. Fibre channel configurations can support as many as 127 nodes and have a
throughput of up to 2.12 gigabits per second in each direction, and 4.25 Gbps is expected.
Fibre channel, however, is very expensive. Just the fibre channel switch alone can start at
around US$1,000. This does not even include the fibre channel storage array and high-
end drives, which can reach prices of about US$300 for a 36GB drive. A typical fibre
channel setup which includes fibre channel cards for the servers is roughly US$10,000,
which does not include the cost of the servers that make up the cluster.
A less expensive alternative to fibre channel is SCSI. SCSI technology provides

acceptable performance for shared storage, but for administrators and developers who are
used to GPL-based Linux prices, even SCSI can come in over budget, at around
US$2,000 to US$5,000 for a two-node cluster.
Another popular solution is the Sun NFS (Network File System) found on a NAS. It can
be used for shared storage but only if you are using a network appliance or something
similar. Specifically, you need servers that guarantee direct I/O over NFS, TCP as the
transport protocol, and read/write block sizes of 32K.
The shared storage that will be used for this article is based on iSCSI technology using a
network storage server installed with Openfiler. This solution offers a low-cost alternative
to fibre channel for testing and educational purposes, but given the low-end hardware
being used, it should not be used in a production environment.
4. iSCSI Technology
For many years, the only technology that existed for building a network based storage
solution was a Fibre Channel Storage Area Network (FC SAN). Based on an earlier set of
ANSI protocols called Fiber Distributed Data Interface (FDDI), Fibre Channel was
developed to move SCSI commands over a storage network.
Several of the advantages to FC SAN include greater performance, increased disk

utilization, improved availability, better scalability, and most important to us support
for server clustering! Still today, however, FC SANs suffer from three major
disadvantages. The first is price. While the costs involved in building a FC SAN have
come down in recent years, the cost of entry still remains prohibitive for small companies
with limited IT budgets. The second is incompatible hardware components. Since its
adoption, many product manufacturers have interpreted the Fibre Channel specifications
differently from each other which has resulted in scores of interconnect problems. When
purchasing Fibre Channel components from a common manufacturer, this is usually not a
problem. The third disadvantage is the fact that a Fibre Channel network is not Ethernet!
It requires a separate network technology along with a second set of skill sets that need to
exist with the datacenter staff.
With the popularity of Gigabit Ethernet and the demand for lower cost, Fibre Channel has
recently been given a run for its money by iSCSI-based storage systems. Today, iSCSI
SANs remain the leading competitor to FC SANs.
Ratified on February 11th, 2003 by the Internet Engineering Task Force (IETF), the
Internet Small Computer System Interface, better known as iSCSI, is an Internet Protocol
(IP)-based storage networking standard for establishing and managing connections
between IP-based storage devices, hosts, and clients. iSCSI is a data transport protocol
defined in the SCSI-3 specifications framework and is similar to Fibre Channel in that it
is responsible for carrying block-level data over a storage network. Block-level
communication means that data is transferred between the host and the client in chunks
called blocks. Database servers depend on this type of communication (as opposed to the
file level communication used by most NAS systems) in order to work properly. Like a
FC SAN, an iSCSI SAN should be a separate physical network devoted entirely to
storage, however, its components can be much the same as in a typical IP network
(LAN).
While iSCSI has a promising future, many of its early critics were quick to point out
some of its inherent shortcomings with regards to performance. The beauty of iSCSI is its
ability to utilize an already familiar IP network as its transport mechanism. The TCP/IP
protocol, however, is very complex and CPU intensive. With iSCSI, most of the
processing of the data (both TCP and iSCSI) is handled in software and is much slower
than Fibre Channel which is handled completely in hardware. The overhead incurred in
mapping every SCSI command onto an equivalent iSCSI transaction is excessive. For
many the solution is to do away with iSCSI software initiators and invest in specialized
cards that can offload TCP/IP and iSCSI processing from a server's CPU. These
specialized cards are sometimes referred to as an iSCSI Host Bus Adaptor (HBA) or a
TCP Offload Engine (TOE) card. Also consider that 10-Gigabit Ethernet is a reality
today!
As with any new technology, iSCSI comes with its own set of acronyms and terminology.
For the purpose of this article, it is only important to understand the difference between
an iSCSI initiator and an iSCSI target.
iSCSI Initiator. Basically, an iSCSI initiator is a client device that connects and initiates
requests to some service offered by a server (in this case an iSCSI target). The iSCSI
initiator software will need to exist on each of the Oracle RAC nodes (linux1 and
linux2).
An iSCSI initiator can be implemented using either software or hardware. Software
iSCSI initiators are available for most major operating system platforms. For this article,
we will be using the free Linux iscsi-sfnet software driver found in the iscsi-
initiator-utils RPM developed as part of the Linux-iSCSI Project. The iSCSI
software initiator is generally used with a standard network interface card (NIC) a
Gigabit Ethernet card in most cases. A hardware initiator is an iSCSI HBA (or a TCP
Offload Engine (TOE) card), which is basically just a specialized Ethernet card with a
SCSI ASIC on-board to offload all the work (TCP and SCSI commands) from the system
CPU. iSCSI HBAs are available from a number of vendors, including Adaptec,
Alacritech, Intel, and QLogic.
iSCSI Target. An iSCSI target is the "server" component of an iSCSI network. This is
typically the storage device that contains the information you want and answers requests
from the initiator(s). For the purpose of this article, the node openfiler1 will be the
iSCSI target.
So with all of this talk about iSCSI, does this mean the death of Fibre Channel anytime
soon? Probably not. Fibre Channel has clearly demonstrated its capabilities over the years
with its capacity for extremely high speeds, flexibility, and robust reliability. Customers
who have strict requirements for high performance storage, large complex connectivity,
and mission critical reliability will undoubtedly continue to choose Fibre Channel.
Before closing out this section, I thought it would be appropriate to present the following
chart that shows speed comparisons of the various types of disk interfaces and network
technologies. For each interface, I provide the maximum transfer rates in kilobits (kb),
kilobytes (KB), megabits (Mb), megabytes (MB), and gigabits (Gb) per second with
some of the more common ones highlighted in grey.
Speed
Disk Interface / Network
Kb KB Mb MB Gb
Serial 115 14.375 0.115 0.014
Parallel (standard) 920 115 0.92 0.115
10Base-T Ethernet 10 1.25
IEEE 802.11b wireless Wi-Fi (2.4 GHz band) 11 1.375
USB 1.1 12 1.5
Parallel (ECP/EPP) 24 3
SCSI-1 40 5
IEEE 802.11g wireless WLAN (2.4 GHz band) 54 6.75
SCSI-2 (Fast SCSI / Fast Narrow SCSI) 80 10
100Base-T Ethernet (Fast Ethernet) 100 12.5
ATA/100 (parallel) 100 12.5
IDE 133.6 16.7
Fast Wide SCSI (Wide SCSI) 160 20
Ultra SCSI (SCSI-3 / Fast-20 / Ultra Narrow) 160 20
Ultra IDE 264 33
Wide Ultra SCSI (Fast Wide 20) 320 40
Ultra2 SCSI 320 40
FireWire 400 - (IEEE1394a) 400 50
USB 2.0 480 60
Wide Ultra2 SCSI 640 80
Ultra3 SCSI 640 80
FireWire 800 - (IEEE1394b) 800 100
Gigabit Ethernet 1000 125 1
Serial ATA I - (SATA I) 1200 150 1.2
Wide Ultra3 SCSI 1280 160 1.28
Ultra160 SCSI 1280 160 1.28
Serial ATA II - (SATA II) 2400 300 2.4
Ultra320 SCSI 2560 320 2.56
FC-AL Fibre Channel 3200 400 3.2
Serial ATA III - (SATA III) 4800 600 4.8
10G Ethernet (IEEE 802.3ae) 10000 1250 10
5. Hardware & Costs
The hardware used to build our example Oracle10g RAC environment consists of three
Linux servers (two Oracle RAC nodes and one Network Storage Server) and components
that can be purchased at many local computer stores or over the Internet.
Oracle RAC Node 1 - (linux1)
Dimension 2400 Series US$620

Intel(R) Pentium(R) 4 Processor at 2.80GHz
1GB DDR SDRAM (at 333MHz)
40GB 7200 RPM Internal Hard Drive
Integrated Intel 3D AGP Graphics
Integrated 10/100 Ethernet
CDROM (48X Max Variable)
3.5" Floppy
No monitor (Already had one)
USB Mouse and Keyboard
1 - Ethernet LAN Card

Used for RAC interconnect to linux2 and Openfiler networked storage.
Each Linux server for Oracle RAC should contain two NIC adapters. The Dell
Dimension includes an integrated 10/100 Ethernet adapter that will be used to
connect to the public network. The second NIC adapter will be used for the
private network (RAC interconnect and Openfiler networked storage). Select
the appropriate NIC adapter that is compatible with the maximum data
transmission speed of the network switch to be used for the private network.
10/100 Ethernet
Linksys 10/100 Mpbs - (LNE100TX)
Gigabit Ethernet
NETGEAR 10/100/1000Mbps PCI Network Adapter - (GA311) US$20
Dimension 2400 Series

Intel(R) Pentium(R) 4 Processor at 2.80GHz
Integrated Intel 3D AGP Graphics
3.5" Floppy
No monitor (already had one)
USB Mouse and Keyboard US$620
1 - Ethernet LAN Card US$20

Used for RAC interconnect to linux1 and Openfiler networked storage.
Each Linux server for Oracle RAC should contain two NIC adapters. The Dell
Dimension includes an integrated 10/100 Ethernet adapter that will be used to
connect to the public network. The second NIC adapter will be used for the
private network (RAC interconnect and Openfiler networked storage). Select
the appropriate NIC adapter that is compatible with the maximum data
transmission speed of the network switch to be used for the private network.
10/100 Ethernet
Gigabit Ethernet
NETGEAR 10/100/1000Mbps PCI Network Adapter - (GA311)
Network Storage Server - (openfiler1)
Clone / Pentium 4
Intel(R) Pentium(R) 4 CPU 1.80GHz
NVIDIA GeForce FX 5200 / AGP Graphics
4 x Integrated USB 2.0 Ports
3.5" Floppy
No monitor (Already had one)
USB Mouse and Keyboard US$500
1 - Ethernet LAN Card

Used for networked storage on the private network.
The Network Storage Server (Openfiler server) should contain two NIC
adapters. The Clone / Pentium 4 machine included an integrated 10/100
Ethernet adapter that will be used to connect to the public network. The second
NIC adapter will be used for the private network (Openfiler networked
storage). Select the appropriate NIC adapter that is compatible with the
maximum data transmission speed of the network switch to be used for the
private network.
10/100 Ethernet
Gigabit Ethernet
NETGEAR 10/100/1000Mbps PCI Network Adapter - (GA311) US$20
Miscellaneous Components
Storage Device(s) - External Hard Drive US$320

For the database storage I used a single external Maxtor OneTouch III
(500GB) drive which was connected to the Openfiler server via its USB 2.0
interface. The Openfiler server will be configured to use this disk for iSCSI
based storage and will be used in our Oracle10g RAC configuration to store
the shared files required by Oracle Clusterware as well as all Oracle ASM
volumes. Note that any type of hard disk (internal or external) should work as
long as it can be recognized by the network storage server and has adequate
space.
Maxtor OneTouch III - 500GB FireWire 800/FireWire 400/USB 2.0
Drive - (F01W500)
Here are the details about the disk that I purchased for this test:
Vendor: Maxtor
Model: OneTouch III
Mfg. Part No. or KIT No.: F01W500
Capacity: 500 GB
Cache Buffer: 16 MB
Rotational Speed (rpm): 7200 RPM
Interface Transfer Rate:
FireWire 800 - 800 Mbits/sec
FireWire 400 - 400 Mbits/sec
USB 2.0 - 480 Mbits/sec
"Combo" Interface:
IEEE 1394b - FireWire 800
IEEE 1394a - FireWire 400
USB 2.0 / USB 1.1
1 - Ethernet Switch
Used for the interconnect between linux1-priv and linux2-priv. This switch
will also be used for network storage traffic for Openfiler. In an effort to keep
costs down, I used a 10/100 Mb Ethernet switch (and 10/100 Mb Ethernet
cards) for the private network. A Gigabit Ethernet switch (and 1Gb Ethernet
cards), however, is highly recommended.
10/100 Ethernet
Linksys EtherFast 10/100 5-port Ethernet Switch - (EZXS55W)
Gigabit Ethernet
D-Link 8-port 10/100/1000 Desktop Switch - (DGS-2208) US$25
6 - Network Cables
Category 5e patch cable - (Connect linux1 to public network)
Category 5e patch cable - (Connect linux2 to public network)
Category 5e patch cable - (Connect openfiler1 to public network)
Category 5e patch cable - (Connect linux1 to interconnect ethernet
switch) US$5
Category 5e patch cable - (Connect linux2 to interconnect ethernet US$5
switch) US$5
US$5
Category 5e patch cable - (Connect openfiler1 to interconnect ethernet US$5
switch) US$5
Total US$2,175
We are about to start the installation process. Now that we have talked about the
hardware that will be used in this example, let's take a conceptual look at what the
environment would look like (click on the graphic below to view larger image):
Figure 1 Architecture
As we start to go into the details of the installation, it should be noted that most of the
tasks within this document will need to be performed on both Oracle RAC nodes (linux1
and linux2). I will indicate at the beginning of each section whether or not the task(s)
should be performed on both Oracle RAC nodes or on the network storage server
(openfiler1).
6. Install the Linux Operating System
Perform the following installation on both Oracle RAC nodes in the cluster!
This section provides a summary of the screens used to install the Linux operating
system. This guide is designed to work with the Red Hat Enterprise Linux 4 AS/ES
(RHEL4) operating environment. As an alternative, and what I used for this article, is
CentOS 4.4: a free and stable version of the RHEL4 operating environment.
For more detailed installation instructions, it is possible to use the manuals from Red Hat
Linux. I would suggest, however, that the instructions I have provided below be used for
this configuration.
Before installing the Linux operating system on both nodes, you should have the two NIC
interfaces (cards) installed.
Download the following ISO images for CentOS 4.4:
CentOS-4.4-i386-bin1of4.iso (622 MB)

If you are downloading the above ISO files to a MS Windows machine, there are many
options for burning these images (ISO files) to a CD. You may already be familiar with
and have the proper software to burn images to CD. If you are not familiar with this
process and do not have the required software to burn images to CD, here are just two (of
many) software packages that can be used:
UltraISO
Magic ISO Maker
After downloading and burning the CentOS images (ISO files) to CD, insert CentOS
Disk #1 into the first server (linux1 in this example), power it on, and answer the
installation screen prompts as noted below. After completing the Linux installation on the
first node, perform the same Linux installation on the second node while substituting the
node name linux1 for linux2 and the different IP addresses where appropriate.
Boot Screen
The first screen is the CentOS Enterprise Linux boot screen. At the boot: prompt, hit
[Enter] to start the installation process.
Media Test
When asked to test the CD media, tab over to [Skip] and hit [Enter]. If there were any
errors, the media burning software would have warned us. After several seconds, the
installer should then detect the video card, monitor, and mouse. The installer then goes
into GUI mode.
Welcome to CentOS Enterprise Linux

At the welcome screen, click [Next] to continue.
Language / Keyboard Selection

The next two screens prompt you for the Language and Keyboard settings. Make the
appropriate selections for your configuration.
Installation Type
Choose the [Custom] option and click [Next] to continue.
Disk Partitioning Setup

Select [Automatically partition] and click [Next] continue.
If there were a previous installation of Linux on this machine, the next screen will ask if
you want to "remove" or "keep" old partitions. Select the option to [Remove all partitions
on this system]. Also, ensure that the [hda] drive is selected for this installation. I also
keep the checkbox [Review (and modify if needed) the partitions created] selected. Click
[Next] to continue.
You will then be prompted with a dialog window asking if you really want to remove all
partitions. Click [Yes] to acknowledge this warning.
Partitioning
The installer will then allow you to view (and modify if needed) the disk partitions it
automatically selected. In almost all cases, the installer will choose 100MB for /boot,
double the amount of RAM for swap, and the rest going to the root (/) partition. I like to
have a minimum of 1GB for swap. For the purpose of this install, I will accept all
automatically preferred sizes. (Including 2GB for swap since I have 1GB of RAM
installed.)
Starting with RHEL 4, the installer will create the same disk configuration as just noted
but will create them using the Logical Volume Manager (LVM). For example, it will
partition the first hard drive (/dev/hda for my configuration) into two partitionsone for
the /boot partition (/dev/hda1) and the remainder of the disk dedicate to a LVM named
VolGroup00 (/dev/hda2). The LVM Volume Group (VolGroup00) is then partitioned into
two LVM partitions - one for the root filesystem (/) and another for swap. I basically
check that it created at least 1GB of swap. Since I have 1GB of RAM installed, the
installer created 2GB of swap. Saying that, I just accept the default disk layout.
Boot Loader Configuration

The installer will use the GRUB boot loader by default. To use the GRUB boot loader,
accept all default values and click [Next] to continue.
Network Configuration
I made sure to install both NIC interfaces (cards) in each of the Linux machines before
starting the operating system installation. This screen should have successfully detected
each of the network devices.
First, make sure that each of the network devices are checked to [Active on boot]. The
installer may choose to not activate eth1.
Second, [Edit] both eth0 and eth1 as follows. You may choose to use different IP
addresses for both eth0 and eth1 and that is OK. If possible, try to put eth1 (the
interconnect) on a different subnet than eth0 (the public network):
eth0:
- Check off the option to [Configure using DHCP]
- Leave the [Activate on boot] checked
- IP Address: 192.168.1.100
- Netmask: 255.255.255.0
eth1:
- IP Address: 192.168.2.100
- Netmask: 255.255.255.0
Continue by setting your hostname manually. I used "linux1" for the first node and
"linux2" for the second one. Finish this dialog off by supplying your gateway and DNS
servers.
Firewall
On this screen, make sure to select [No firewall] and click [Next] to continue. You may
be prompted with a warning dialog about not setting the firewall. If this occurs, simply
hit [Proceed] to continue.
Additional Language Support/Time Zone

The next two screens allow you to select additional language support and time zone
information. In almost all cases, you can accept the defaults.
Set Root Password

Select a root password and click [Next] to continue.
Package Group Selection

Scroll down to the bottom of this screen and select [Everything] under the
"Miscellaneous" section. Click [Next] to continue.
Please note that the installation of Oracle does not require all Linux packages to be
installed. My decision to install all packages was for the sake of brevity. Please see
section Section 19 ("Pre-Installation Tasks for Oracle10g Release 2") for a more detailed
look at the critical packages required for a successful Oracle installation.
Note that with some RHEL4 distributions, you will not get the "Package Group
Selection" screen by default. There, you are asked to simply "Install default software
packages" or "Customize software packages to be installed". Select the option to
"Customize software packages to be installed" and click [Next] to continue. This will
then bring up the "Package Group Selection" screen. Now, scroll down to the bottom of
this screen and select [Everything] under the "Miscellaneous" section. Click [Next] to
continue.
About to Install
This screen is basically a confirmation screen. Click [Continue] to start the installation.
During the installation process, you will be asked to switch disks to Disk #2, Disk #3, and
then Disk #4.
Note that with CentOS 4.4, the installer will ask to switch to Disk #2, Disk #3, Disk #4,
Disk #1, and then back to Disk #4.
Graphical Interface (X) Configuration

With most RHEL4 distributions (not the case with CentOS 4.4), when the installation is
complete, the installer will attempt to detect your video hardware. Ensure that the
installer has detected and selected the correct video hardware (graphics card and monitor)
to properly use the X Windows server. You will continue with the X configuration in the
next serveral screens.
Congratulations
And that's it. You have successfully installed CentOS Enterprise Linux on the first node
(linux1). The installer will eject the CD from the CD-ROM drive. Take out the CD and
click [Reboot] to reboot the system.
When the system boots into Linux for the first time, it will prompt you with another
Welcome screen. The following wizard allows you to configure the date and time, add
any additional users, testing the sound card, and to install any additional CDs. The only
screen I care about is the time and date (and if you are using CentOS 4.x, the
monitor/display settings). As for the others, simply run through them as there is nothing
additional that needs to be installed (at this point anyways!). If everything was successful,
you should now be presented with the login screen.
Perform the same installation on the second node

After completing the Linux installation on the first node, repeat the above steps for the
second node (linux2). When configuring the machine name and networking, ensure to
configure the proper values. For my installation, this is what I configured for linux2:
installer will choose not to activate eth1.
Second, [Edit] both eth0 and eth1 as follows:
eth0:
- IP Address: 192.168.1.101
- Netmask: 255.255.255.0
eth1:
- IP Address: 192.168.2.101
- Netmask: 255.255.255.0
Continue by setting your hostname manually. I used "linux2" for the second node. Finish
this dialog off by supplying your gateway and DNS servers.
7. Network Configuration
Perform the following network configuration on both Oracle RAC nodes in the cluster!
Note: Although we configured several of the network settings during the Linux
installation, it is important to not skip this section as it contains critical steps that are
required for the RAC environment.
Introduction to Network Settings
During the Linux O/S install we already configured the IP address and host name for both
of the Oracle RAC nodes. We now need to configure the /etc/hosts file as well as
adjusting several of the network settings for the interconnect.
Both of the Oracle RAC nodes should have one static IP address for the public network
and one static IP address for the private cluster interconnect. Do not use DHCP naming
for the public IP address or the interconnects; you need static IP addresses! The private
interconnect should only be used by Oracle to transfer Cluster Manager and Cache
Fusion related data along with data for the network storage server (Openfiler). Although
it is possible to use the public network for the interconnect, this not recommended as it
may cause degraded database performance (reducing the amount of bandwidth for Cache
Fusion and Cluster Manager traffic). For a production RAC implementation, the
interconnect should be at least gigabit (or more) and only be used by Oracle as well as
having the network storage server (Openfiler) on a separate gigabit network.
Configuring Public and Private Network
In our two node example, we need to configure the network on both Oracle RAC nodes
for access to the public network as well as their private interconnect.
The easiest way to configure network settings in Red Hat Linux is with the program
"Network Configuration". This application can be started from the command-line as the
"root" user account as follows:
# su -
# /usr/bin/system-config-network &
Using the Network Configuration application, you need to configure both NIC devices as
well as the /etc/hosts file. Both of these tasks can be completed using the Network
Configuration GUI. Notice that the /etc/hosts settings are the same for both nodes.
Our example configuration will use the following settings:

Device IP Address Subnet Gateway Purpose
Connects linux1 to the public
eth0 192.168.1.100 255.255.255.0 192.168.1.1
network
Connects linux1 (interconnect) to
eth1 192.168.2.100 255.255.255.0
linux2 (linux2-priv)
/etc/hosts
127.0.0.1 localhost loopback
# Public Network - (eth0)

192.168.1.100 linux1
192.168.1.101 linux2
# Private Interconnect - (eth1)

192.168.2.100 linux1-priv
192.168.2.101 linux2-priv
# Public Virtual IP (VIP) addresses for - (eth0)

192.168.1.200 linux1-vip
192.168.1.201 linux2-vip

Device IP Address Subnet Gateway Purpose
Connects linux2 to the public
eth0 192.168.1.101 255.255.255.0 192.168.1.1
network
Connects linux2 (interconnect) to
eth1 192.168.2.101 255.255.255.0
linux1 (linux1-priv)
/etc/hosts
127.0.0.1 localhost loopback

192.168.1.100 linux1
192.168.1.101 linux2

192.168.2.100 linux1-priv
192.168.2.101 linux2-priv

192.168.1.200 linux1-vip
192.168.1.201 linux2-vip
Note that the virtual IP addresses only need to be defined in the /etc/hosts file (or your
DNS) for both Oracle RAC nodes. The public virtual IP addresses will be configured
automatically by Oracle when you run the Oracle Universal Installer, which starts
Oracle's Virtual Internet Protocol Configuration Assistant (VIPCA). All virtual IP
addresses will be activated when the srvctl start nodeapps -n <node_name>
command is run. This is the Host Name/IP Address that will be configured in the client(s)
tnsnames.ora file (more details later).
In the screenshots below, only Oracle RAC Node 1 (linux1) is shown. Be sure to make all
the proper network settings to both Oracle RAC nodes.
Figure 2 Network Configuration Screen, Node 1 (linux1)

Figure 3 Ethernet Device Screen, eth0 (linux1)
Figure 4 Ethernet Device Screen, eth1 (linux1)
Figure 5: Network Configuration Screen, /etc/hosts (linux1)
Once the network is configured, you can use the ifconfig command to verify everything
is working. The following example is from linux1:
$ /sbin/ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:0D:56:FC:39:EC
inet addr:192.168.1.100 Bcast:192.168.1.255 Mask:255.255.255.0
inet6 addr: fe80::20d:56ff:fefc:39ec/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:835 errors:0 dropped:0 overruns:0 frame:0
TX packets:1983 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:705714 (689.1 KiB) TX bytes:176892 (172.7 KiB)
Interrupt:3
eth1 Link encap:Ethernet HWaddr 00:0C:41:E8:05:37
inet addr:192.168.2.100 Bcast:192.168.2.255 Mask:255.255.255.0
inet6 addr: fe80::20c:41ff:fee8:537/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX bytes:0 (0.0 b) TX bytes:546 (546.0 b)
Interrupt:11 Base address:0xe400
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX bytes:8276758 (7.8 MiB) TX bytes:8276758 (7.8 MiB)
sit0 Link encap:IPv6-in-IPv4
NOARP MTU:1480 Metric:1
RX bytes:0 (0.0 b) TX bytes:0 (0.0 b)
About Virtual IP
Why is there a Virtual IP (VIP) in 10g? Why does it just return a dead connection when
its primary node fails?
It's all about availability of the application. When a node fails, the VIP associated with it
is supposed to be automatically failed over to some other node. When this occurs, two
things happen.
1. The new node re-arps the world indicating a new MAC address for the address.
For directly connected clients, this usually causes them to see errors on their
connections to the old address.
2. Subsequent packets sent to the VIP go to the new node, which will send error RST
packets back to the clients. This results in the clients getting errors immediately.
This means that when the client issues SQL to the node that is now down, or traverses the
address list while connecting, rather than waiting on a very long TCP/IP time-out (~10
minutes), the client receives a TCP reset. In the case of SQL, this is ORA-3113. In the case
of connect, the next address in tnsnames is used.
Going one step further is making use of Transparent Application Failover (TAF). With
TAF successfully configured, it is possible to completely avoid ORA-3113 errors
alltogether! TAF will be discussed in more detail in Section 29 ("Transparent Application
Failover - (TAF)").
Without using VIPs, clients connected to a node that died will often wait a 10-minute
TCP timeout period before getting an error. As a result, you don't really have a good HA
solution without using VIPs (Source - Metalink Note 220970.1).
Confirm the RAC Node Name is Not Listed in Loopback Address
Ensure that the node names (linux1 or linux2) are not included for the loopback
address in the /etc/hosts file. If the machine name is listed in the in the loopback
address entry as below:
127.0.0.1 linux1 localhost.localdomain localhost

it will need to be removed as shown below:
127.0.0.1 localhost.localdomain localhost
If the RAC node name is listed for the loopback address, you will receive the following
error during the RAC installation:
ORA-00603: ORACLE server session terminated by fatal error

or
ORA-29702: error occurred in Cluster Group Service operation
Adjusting Network Settings
With Oracle 9.2.0.1 and later, Oracle makes use of UDP as the default protocol on Linux
for inter-process communication (IPC), such as Cache Fusion and Cluster Manager buffer
transfers between instances within the RAC cluster.
Oracle strongly suggests to adjust the default and maximum send buffer size (SO_SNDBUF
socket option) to 256KB, and the default and maximum receive buffer size
(SO_RCVBUF socket option) to 256KB.
The receive buffers are used by TCP and UDP to hold received data until it is read by the
application. The receive buffer cannot overflow because the peer is not allowed to send
data beyond the buffer size window. This means that datagrams will be discarded if they
don't fit in the socket receive buffer, potentially causing the sender to overwhelm the
receiver.
The default and maximum window size can be changed in the /proc file system without
reboot:
# su - root
# sysctl -w net.core.rmem_default=262144
net.core.rmem_default = 262144
# sysctl -w net.core.wmem_default=262144
net.core.wmem_default = 262144
# sysctl -w net.core.rmem_max=262144
net.core.rmem_max = 262144
# sysctl -w net.core.wmem_max=262144
net.core.wmem_max = 262144
The above commands made the changes to the already running OS. You should now
make the above changes permanent (for each reboot) by adding the following lines to the
/etc/sysctl.conf file for both nodes in your RAC cluster:
# Default setting in bytes of the socket receive buffer

net.core.rmem_default=262144
# Default setting in bytes of the socket send buffer

net.core.wmem_default=262144
# Maximum socket receive buffer size which may be set by using

# the SO_RCVBUF socket option
net.core.rmem_max=262144
# Maximum socket send buffer size which may be set by using

# the SO_SNDBUF socket option
net.core.wmem_max=262144
Check and turn off UDP ICMP rejections
During the Linux installation process, I indicated to not configure the firewall option. By
default the option to configure a firewall is selected by the installer. This has burned me
several times so I like to do a double-check that the firewall option is not configured and
to ensure udp ICMP filtering is turned off.
If UDP ICMP is blocked or rejected by the firewall, the Oracle Clusterware software will
crash after several minutes of running. When the Oracle Clusterware process fails, you
will have something similar to the following in the <machine_name>_evmocr.log file:
08/29/2005 22:17:19
oac_init:2: Could not connect to server, clsc retcode = 9
08/29/2005 22:17:19
a_init:12!: Client init unsuccessful : [32]
ibctx:1:ERROR: INVALID FORMAT
proprinit:problem reading the bootblock or superbloc 22
When experiencing this type of error, the solution was to remove the udp ICMP (iptables)
rejection rule - or to simply have the firewall option turned off. The Oracle Clusterware
software will then start to operate normally and not crash. The following commands
should be executed as the root user account:
1. Check to ensure that the firewall option is turned off. If the firewall option is
stopped (like it is in my example below) you do not have to proceed with the
following steps.
# /etc/rc.d/init.d/iptables status
Firewall is stopped.
2. If the firewall option is operating you will need to first manually disable UDP
ICMP rejections:
# /etc/rc.d/init.d/iptables stop
Flushing firewall rules: [ OK ]
Setting chains to policy ACCEPT: filter [ OK ]
Unloading iptables modules: [ OK ]
3. Then, to turn UDP ICMP rejections off for next server reboot (which should
always be turned off):
# chkconfig iptables off
8. Install Openfiler
Perform the following installation on the network storage server (openfiler1)!
With the network configured on both Oracle RAC nodes, the next step is to install the
Openfiler software to the network storage server (openfiler1). Later in this article, the
network storage server will be configured as an iSCSI storage device for all Oracle10g
RAC shared storage requirements.
Powered by rPath Linux, Openfiler is a free browser-based network storage management

utility that delivers file-based Network Attached Storage (NAS) and block-based Storage
Area Networking (SAN) in a single framework. The entire software stack interfaces with
open source applications such as Apache, Samba, LVM2, ext3, Linux NFS and iSCSI
Enterprise Target. Openfiler combines these ubiquitous technologies into a small, easy to
manage solution fronted by a powerful web-based management interface.
Openfiler supports CIFS, NFS, HTTP/DAV, and FTP, however, we will only be making
use of its iSCSI capabilities to implement an inexpensive SAN for the shared storage
components required by Oracle10g RAC. A 500GB external hard drive will be connected
to the Openfiler server via its USB 2.0 interface. The Openfiler server will be configured
to use this disk for iSCSI based storage and will be used in our Oracle10g RAC
configuration to store the shared files required by Oracle Clusterware as well as all
Oracle ASM volumes.
To learn more about Openfiler, please visit their website at http://www.openfiler.com/
Download Openfiler
Use the links (below) to download Openfiler 2.1 x86 (Final). After downloading
Openfiler, you will then need to burn the ISO image to CD.
Openfiler
o openfiler-2.1-x86-disc1.iso (302 MB)
If you are downloading the above ISO file to a MS Windows machine, there are many
options for burning these images (ISO files) to a CD. You may already be familiar with
and have the proper software to burn images to CD. If you are not familiar with this
process and do not have the required software to burn images to CD, here are just two (of
many) software packages that can be used:
UltraISO
Magic ISO Maker
Install Openfiler
This section provides a summary of the screens used to install the Openfiler software. For
the purpose of this article, I opted to install Openfiler with all default options. The only
manual change required was for configuring the local network settings.
Once the install has completed, the server will reboot to make sure all required
components, services and drivers are started and recognized. After the reboot, the
external Maxtor hard drive should be discovered by the Openfiler server as the device
/dev/sda.
For more detailed installation instructions, please visit http://www.openfiler.com/docs/. I

would suggest, however, that the instructions I have provided below be used for this
Oracle10g RAC configuration.
Before installing the Openfiler software to the network storage server, you should have
both NIC interfaces (cards) installed and any external hard drives connected and turned
on.
After downloading and burning the Openfiler ISO image (ISO file) to CD, insert the CD
into the network storage server (openfiler1 in this example), power it on, and answer
the installation screen prompts as noted below.
Boot Screen
The first screen is the Openfiler boot screen. At the boot: prompt, hit [Enter] to start the
installation process.
Media Test
When asked to test the CD media, tab over to [Skip] and hit [Enter]. If there were any
errors, the media burning software would have warned us. After several seconds, the
installer should then detect the video card, monitor, and mouse. The installer then goes
into GUI mode.
Welcome to Openfiler NAS/SAN Appliance

At the welcome screen, click [Next] to continue.
Keyboard Configuration
The next screen prompts you for the Keyboard settings. Make the appropriate selection
for your configuration.
Disk Partitioning Setup

The next screen asks whether to perform disk partitioning using "Automatic Partitioning"
or "Manual Partitioning with Disk Druid". You can choose either method here, although
the official Openfiler documentation suggests to use Manual Partitioning. Since the
internal hard drive I will be using for this install is small and only going to be used to
store the Openfiler software (I will not be using any space on the internal 40GB hard
drive for iSCSI storage), I opted to use "Automatic Partitioning".
Select [Automatically partition] and click [Next] continue.
If there were a previous installation of Linux on this machine, the next screen will ask if
you want to "remove" or "keep" old partitions. Select the option to [Remove all partitions
on this system]. Also, ensure that ONLY the [hda] drive is selected for this installation. I
also keep the checkbox [Review (and modify if needed) the partitions created] selected.
Click [Next] to continue.
You will then be prompted with a dialog window asking if you really want to remove all
partitions. Click [Yes] to acknowledge this warning.
Partitioning
The installer will then allow you to view (and modify if needed) the disk partitions it
automatically selected for /dev/hda. In almost all cases, the installer will choose 100MB
for /boot, double the amount of RAM for swap, and the rest going to the root (/) partition.
I like to have a minimum of 1GB for swap. For the purpose of this install, I will accept all
automatically preferred sizes. (Including 2GB for swap since I have 1GB of RAM
installed.)
Network Configuration
I made sure to install both NIC interfaces (cards) in the network storage server before
starting the Openfiler installation. This screen should have successfully detected each of
the network devices.
installer may choose to not activate eth1 by default.
Second, [Edit] both eth0 and eth1 as follows. You may choose to use different IP
addresses for both eth0 and eth1 and that is OK. You must, however, configure eth1 (the
storage network) to be on the same subnet you configured for eth1 on linux1 and linux2:
eth0:
- IP Address: 192.168.1.195
- Netmask: 255.255.255.0
eth1:
- IP Address: 192.168.2.195
- Netmask: 255.255.255.0
Continue by setting your hostname manually. I used a hostname of "openfiler1". Finish

this dialog off by supplying your gateway and DNS servers.
Time Zone Selection

The next screen allows you to configure your time zone information. Make the
appropriate selection for your location.
Set Root Password
Select a root password and click [Next] to continue.
About to Install
This screen is basically a confirmation screen. Click [Next] to start the installation.
Congratulations
And that's it. You have successfully installed Openfiler on the network storage server. The
installer will eject the CD from the CD-ROM drive. Take out the CD and click [Reboot]
to reboot the system.
If everything was successful after the reboot, you should now be presented with a text
login screen and the URL to use for administering the Openfiler server.
9. Configure iSCSI Volumes using Openfiler
Perform the following configuration tasks on the network storage server (openfiler1)!
Openfiler administration is performed using the Openfiler Storage Control Center a

browser based tool over an https connection on port 446. For example:
https://openfiler1:446/
From the Openfiler Storage Control Center home page, login as an administrator. The
default administration login credentials for Openfiler are:
Username: openfiler
Password: password
The first page the administrator sees is the [Accounts] / [Authentication] screen.
Configuring user accounts and groups is not necessary for this article and will therefore
not be discussed.
To use the Openfiler as an iSCSI storage server, we have to perform three major tasks; set
up iSCSI services, configure network access, and create physical storage.
Services
To control services, we use the Openfiler Storage Control Center and navigate to
[Services] / [Enable/Disable]:
Figure 6 Enable iSCSI Openfiler Service
To enable the iSCSI service, click on 'Enable' under the 'iSCSI target' service name. After
that, the 'iSCSI target' status should change to 'Enabled'.
The ietd program implements the user level part of iSCSI Enterprise Target software for
building an iSCSI storage system on Linux. With the iSCSI target enabled, we should be
able to SSH into the Openfiler server and see the iscsi-target service running:
[root@openfiler1 ~]# service iscsi-target status

ietd (pid 3784) is running...
Network Access Restriction
The next step is to configure network access in Openfiler so both Oracle RAC nodes
(linux1 and linux2) have permissions to our iSCSI volumes through the storage
(private) network. (iSCSI volumes will be created in the next section!)
Again, this task can be completed using the Openfiler Storage Control Center by
navigating to [General] / [Local Networks]. The Local Networks screen allows an
administrator to setup networks and/or hosts that will be allowed to access resources
exported by the Openfiler appliance. For the purpose of this article, we will want to add
both Oracle RAC nodes individually rather than allowing the entire 192.168.2.0 network
have access to Openfiler resources.
When entering each of the Oracle RAC nodes, note that the 'Name' field is just a logical
name used for reference only. As a convention when entering nodes, I simply use the
node name defined for that IP address. Next, when entering the actual node in the
'Network/Host' field, always use its IP address even though its host name may already be
defined in your /etc/hosts file or DNS. Lastly, when entering actual hosts in our Class
C network, use a subnet mask of 255.255.255.255.
It is important to remember that you will be entering the IP address of the private
network (eth1) for each of the RAC nodes in the cluster.
The following image shows the results of adding both Oracle RAC nodes:
Figure 7 Configure Openfiler Host Access for Oracle RAC Nodes
Physical Storage
In this section, we will be creating the five iSCSI volumes to be used as shared storage by
both of the Oracle RAC nodes in the cluster. This involves multiple steps that will be
performed on the external USB hard drive connected to the Openfiler server.
Storage devices like internal IDE/SATA/SCSI disks, external USB or FireWire drives, or
any other storage can be connected to the Openfiler server, and served to the clients.
Once these devices are discovered at the OS level, Openfiler Storage Control Center can
be used to set up and manage all that storage.
In our case, we have a 500GB external USB hard drive for our storage needs. On the
Openfiler server this drive is seen as /dev/sda (external Maxtor OneTouch III drive). To
see this and to start the process of creating our iSCSI volumes, navigate to [Volumes] /
[Physical Storage Mgmt.] from the Openfiler Storage Control Center:
Figure 8 Openfiler Physical Storage
Partitioning the Physical Disk
The first step we will perform is to create a single primary partition on the /dev/sda
external USB hard drive. By clicking on the /dev/sda link, we are presented with the
options to 'Edit' or 'Create' a partition. Since we will be creating a single primary partition
that spans the entire disk, most of the options can be left to their default setting where the
only modification would be to change the 'Partition Type' from 'Extended partition' to
'Physical volume'. Here are the values I specified to create the primary partition on
/dev/sda:
Mode: Primary
Partition Type: Physical volume
Starting Cylinder: 1
Ending Cylinder: 60801
The size now shows 465.76 GB. To accept that we click on the Create button. This results
in a new partition (/dev/sda1) on our external hard drive:
Figure 9 Partition the Physical Volume
Volume Group Management
The next step is to create a Volume Group. We will be creating a single volume group
named rac1 that contains the newly created primary partition.
From the Openfiler Storage Control Center, navigate to [Volumes] / [Volume Group
Mgmt.]. There we would see any existing volume groups, or none as in our case. Using
the Volume Group Management screen, enter the name of the new volume group (rac1),
click on the checkbox in front of /dev/sda1 to select that partition, and finally click on
the 'Add volume group' button. After that we are presented with the list that now shows
our newly created volume group named "rac1":
Figure 10 New Volume Group Created
Logical Volumes
We can now create the five logical volumes in the newly created volume group (rac1).
From the Openfiler Storage Control Center, navigate to [Volumes] / [Create New
Volume]. There we will see the newly created volume group (rac1) along with its block
storage statistics. Also available at the bottom of this screen is the option to create a new
volume in the selected volume group. Use this screen to create the following five logical
(iSCSI) volumes. After creating each logical volume, the application will point you to the
"List of Existing Volumes" screen. You will then need to click back to the "Create New
Volume" tab to create the next logical volume until all five iSCSI volumes are created:
iSCSI / Logical Volumes

Volume Name Volume Description Required Space (MB) Filesystem Type
crs Oracle Clusterware 2,048 iSCSI
asm1 Oracle ASM Volume 1 118,720 iSCSI
In effect we have created five iSCSI disks that can now be presented to iSCSI clients
(linux1 and linux2) on the network. The "List of Existing Volumes" screen should look
as follows:
Figure 11 New Logical (iSCSI) Volumes
Grant Access Rights to New Logical Volumes
Before an iSCSI client can have access to the newly created iSCSI volumes, it needs to
be granted the appropriate permissions. Awhile back, we configured Openfiler with two
hosts (the Oracle RAC nodes) that can be configured with access rights to resources. We
now need to grant both of the Oracle RAC nodes access to each of the newly created
iSCSI volumes.
From the Openfiler Storage Control Center, navigate to [Volumes] / [List of Existing
Volumes]. This will present the screen shown in the previous section. For each of the
logical volumes, click on the 'Edit' link (under the Properties column). This will bring up
the 'Edit properties' screen for that volume. Scroll to the bottom of this screen, change
both hosts from 'Deny' to 'Allow' and click the 'Update' button:
Figure 12 Grant Host Access to Logical (iSCSI) Volumes
Perform this task for all five logical volumes.
Make iSCSI Targets Available to Clients
Every time a new logical volume is added, we need to restart the associated service on
the Openfiler server. In our case we created iSCSI logical volumes, so we have to restart
the iSCSI target (iscsi-target) service. This will make the new iSCSI targets available to
all clients on the network who have privileges to access them.
To restart the iSCSI target service, use the Openfiler Storage Control Center and navigate
to [Services] / [Enable/Disable]. The iSCSI target service should already be enabled
(several sections back). If so, disable the service then enable it again. (See Figure 6)
The same task can be achieved through an SSH session on the Openfiler server:
[root@openfiler1 ~]# service iscsi-target restart

Stopping iSCSI target service: [ OK ]
Starting iSCSI target service: [ OK ]
10. Configure iSCSI Volumes on Oracle RAC Nodes

Configure the iSCSI initiator on both Oracle RAC nodes in the cluster! Creating
partitions, however, should only be executed on one of nodes in the RAC cluster.
An iSCSI client can be any system (Linux, Unix, MS Windows, Apple Mac, etc.) for
which iSCSI support (a driver) is available. In our case, the clients are two Linux servers,
(linux1 and linux2), running Red Hat 4.
In this section we will be configuring the iSCSI initiator on both of the Oracle RAC
nodes. This involves configuring the /etc/iscsi.conf file on both of the Oracle RAC
nodes with the name of the network storage server (openfiler1) so they can discover the
iSCSI volumes created in the previous section. We then go through the arduous task of
mapping the iSCSI target names discovered from Openfiler to the local SCSI device
name on one of the nodes namely linux1 (the node where we will be partioning the
iSCSI volumes from). This is often considered a lengthy task but only needs to be
performed in this section and when formatting the iSCSI volumes with the Oracle Cluster
File System (OCFS2) and Automatic Storage Management (ASM). Knowing the local
SCSI device name and which iSCSI target it maps to is required in order to know which
volume (device) is to be used for OCFS2 and which volumes belong to ASM. Note that
every time one of the Oracle RAC nodes is rebooted, the mappings may be different. For
example, the iSCSI target name "iqn.2006-01.com.openfiler:rac1.crs" may have
been discovered as /dev/sdd on linux1 during the process of configuring the volumes
(as it was for me when writing this section!). After rebooting this node, however,
"iqn.2006-01.com.openfiler:rac1.crs" may get discovered as /dev/sde. This will
not be a problem since all disks will be labeled either by OCFS2 or ASM (later in this
article). When either of these services attempt to mount a volume, it will do so using their
label and not using their local SCSI device name.
iSCSI (initiator) service
On each of the Oracle RAC nodes, we have to make sure the iSCSI (initiator) service is
up and running. If not installed as part of the operating system setup, the iscsi-initiator-
utils RPM (i.e. iscsi-initiator-utils-4.0.3.0-4.i386.rpm) should be downloaded and
installed on each of the Oracle RAC nodes.
To determine if this package is installed, perform the following on both Oracle RAC
nodes:
# rpm -qa | grep iscsi

iscsi-initiator-utils-4.0.3.0-4
If not installed, the iscsi-initiator-utils RPM package can be found on disk 3 of 4 of the
RHEL4 Update 4 distribution or downloaded from one of the Internet RPM resources.
Use the following command to install the iscsi-initiator-utils RPM package if not present:
# rpm -Uvh iscsi-initiator-utils-4.0.3.0-4.i386.rpm

warning: iscsi-initiator-utils-4.0.3.0-4.i386.rpm:
V3 DSA signature: NOKEY, key ID 443e1821
Preparing... ###########################################
[100%]
1:iscsi-initiator-utils ###########################################
[100%]
After verifying that the iscsi-initiator-utils RPM is installed, the only configuration step
required on the Oracle RAC nodes (iSCSI client) is to specify the network storage server
(iSCSI server) in the /etc/iscsi.conf file. Edit the /etc/iscsi.conf file and include
an entry for DiscoveryAddress which specifies the hostname of the Openfiler network
storage server. In our case that was:
...
DiscoveryAddress=openfiler1-priv
...
After making that change to the /etc/iscsi.conf file on both Oracle RAC nodes, we
can start (or restart) the iscsi initiator service on both nodes:
# service iscsi restart

Searching for iscsi-based multipath maps
Found 0 maps
Stopping iscsid: iscsid not running
Checking iscsi config: [ OK ]

Loading iscsi driver: [ OK ]
Starting iscsid: [ OK ]
We should also configure the iSCSI service to be active across machine reboots for both
Oracle RAC nodes. The Linux command chkconfig can be used to achieve that as
follows:
# chkconfig --level 345 iscsi on
Discovering iSCSI Targets
Although the iSCSI initiator service has been configured and is running on both of the
Oracle RAC nodes, the discovery instructions in this section only need to be run from the
node we will be partitioning and labeling volumes from; namely linux1.
When the Openfiler server publishes available iSCSI targets (that happens when the iscsi-
target service gets started/restarted on the Openfiler server), or when the iSCSI initiator
service is started/restarted on the client, configured clients get the message that new
iSCSI disks are now available. We would see something like this in the client's
/var/log/messages file:
...
Oct 10 19:55:39 linux1 iscsi: iscsid startup succeeded
Oct 10 19:55:39 linux1 iscsid[3073]: Connected to Discovery Address
192.168.2.195
Oct 10 19:55:39 linux1 kernel: iscsi-sfnet:host1: Session established
Oct 10 19:55:39 linux1 kernel: scsi1 : SFNet iSCSI driver
Oct 10 19:55:39 linux1 kernel: Vendor: Openfile Model: Virtual disk
Rev: 0
Oct 10 19:55:39 linux1 kernel: Type: Direct-Access
ANSI SCSI revision: 04
Rev: 0
Oct 10 19:55:39 linux1 kernel: SCSI device sda: 243138560 512-byte hdwr
sectors (124487 MB)
Rev: 0
Rev: 0
Oct 10 19:55:39 linux1 kernel: SCSI device sda: drive cache: write
through
Rev: 0
Oct 10 19:55:39 linux1 kernel: SCSI device sda: 243138560 512-byte hdwr
sectors (124487 MB)
Oct 10 19:55:39 linux1 kernel: SCSI device sda: drive cache: write
through
Oct 10 19:55:39 linux1 kernel: sda: unknown partition table
Oct 10 19:55:39 linux1 kernel: Attached scsi disk sda at scsi1, channel
0, id 0, lun 0
Oct 10 19:55:39 linux1 kernel: SCSI device sdb: 243138560 512-byte hdwr
sectors (124487 MB)
Oct 10 19:55:39 linux1 kernel: SCSI device sdb: drive cache: write
through
Oct 10 19:55:39 linux1 kernel: SCSI device sdb: 243138560 512-byte hdwr
sectors (124487 MB)
Oct 10 19:55:39 linux1 scsi.agent[3222]: disk at
/devices/platform/host0/target0:0:0/0:0:0:0
Oct 10 19:55:39 linux1 kernel: SCSI device sdb: drive cache: write
through
Oct 10 19:55:40 linux1 kernel: sdb: unknown partition table
Oct 10 19:55:40 linux1 kernel: Attached scsi disk sdb at scsi0, channel
0, id 0, lun 0
Oct 10 19:55:40 linux1 kernel: SCSI device sdc: 243138560 512-byte hdwr
sectors (124487 MB)
Oct 10 19:55:40 linux1 kernel: SCSI device sdc: drive cache: write
through
Oct 10 19:55:40 linux1 kernel: SCSI device sdc: 243138560 512-byte hdwr
sectors (124487 MB)
Oct 10 19:55:40 linux1 kernel: SCSI device sdc: drive cache: write
through
Oct 10 19:55:40 linux1 kernel: sdc: unknown partition table
Oct 10 19:55:40 linux1 kernel: Attached scsi disk sdc at scsi3, channel
0, id 0, lun 0
Oct 10 19:55:40 linux1 kernel: SCSI device sdd: 4194304 512-byte hdwr
sectors (2147 MB)
Oct 10 19:55:40 linux1 kernel: SCSI device sdd: drive cache: write
through
Oct 10 19:55:40 linux1 kernel: SCSI device sdd: 4194304 512-byte hdwr
sectors (2147 MB)
Oct 10 19:55:40 linux1 kernel: SCSI device sdd: drive cache: write
through
Oct 10 19:55:40 linux1 kernel: sdd: unknown partition table
Oct 10 19:55:40 linux1 kernel: Attached scsi disk sdd at scsi4, channel
0, id 0, lun 0
Oct 10 19:55:40 linux1 kernel: SCSI device sde: 243138560 512-byte hdwr
sectors (124487 MB)
Oct 10 19:55:40 linux1 kernel: SCSI device sde: drive cache: write
through
Oct 10 19:55:40 linux1 kernel: SCSI device sde: 243138560 512-byte hdwr
sectors (124487 MB)
Oct 10 19:55:40 linux1 kernel: SCSI device sde: drive cache: write
through
Oct 10 19:55:40 linux1 kernel: sde: unknown partition table
Oct 10 19:55:40 linux1 kernel: Attached scsi disk sde at scsi2, channel
0, id 0, lun 0
Oct 10 19:56:09 linux1 portmap: portmap startup succeeded
...
The above entries show that the client (linux1) was able to establish the iSCSI sessions
with the iSCSI storage server (openfiler1-priv at 192.168.2.195).
We also see how the local SCSI device names map to iSCSI targets' host IDs and LUNs:
Attached scsi disk sda at scsi1, channel 0, id 0, lun 0

Attached scsi disk sdb at scsi0, channel 0, id 0, lun 0
Attached scsi disk sdc at scsi3, channel 0, id 0, lun 0
Attached scsi disk sdd at scsi4, channel 0, id 0, lun 0
Attached scsi disk sde at scsi2, channel 0, id 0, lun 0
Another way to determine how local SCSI device names map to iSCSI targets' host IDs
and LUNs is with the dmesg command:
# dmesg | sort | grep '^Attached scsi disk'
Attached scsi disk sda at scsi1, channel 0, id 0, lun 0
Attached scsi disk sdb at scsi0, channel 0, id 0, lun 0
Attached scsi disk sdc at scsi3, channel 0, id 0, lun 0
Attached scsi disk sdd at scsi4, channel 0, id 0, lun 0
Attached scsi disk sde at scsi2, channel 0, id 0, lun 0
We now have to work out the mapping of iSCSI target names to local SCSI IDs (which
gets displayed as HOST ID below), by running the iscsi-ls command on the client
(linux1):
# iscsi-ls
*********************************************************************
SFNet iSCSI Driver Version ...4:0.1.11-3(02-May-2006)
*********************************************************************
TARGET NAME : iqn.2006-01.com.openfiler:rac1.asm4
TARGET ALIAS :
HOST ID : 0
BUS ID : 0
TARGET ID : 0
TARGET ADDRESS : 192.168.2.195:3260,1
SESSION STATUS : ESTABLISHED AT Tue Oct 10 19:55:40 EDT 2006
SESSION ID : ISID 00023d000001 TSIH c00
*********************************************************************
TARGET ALIAS :
HOST ID : 1
BUS ID : 0
TARGET ID : 0
TARGET ADDRESS : 192.168.2.195:3260,1
SESSION ID : ISID 00023d000001 TSIH b00
*********************************************************************
TARGET ALIAS :
HOST ID : 2
BUS ID : 0
TARGET ID : 0
TARGET ADDRESS : 192.168.2.195:3260,1
SESSION ID : ISID 00023d000001 TSIH d00
*********************************************************************
TARGET ALIAS :
HOST ID : 3
BUS ID : 0
TARGET ID : 0
TARGET ADDRESS : 192.168.2.195:3260,1
SESSION ID : ISID 00023d000001 TSIH e00
*********************************************************************
TARGET NAME : iqn.2006-01.com.openfiler:rac1.crs
TARGET ALIAS :
HOST ID : 4
BUS ID : 0
TARGET ID : 0
TARGET ADDRESS : 192.168.2.195:3260,1
SESSION ID : ISID 00023d000001 TSIH f00
*********************************************************************
Using the mapping information from local SCSI ID to the iSCSI targets' host IDs / LUNs
along with the iSCSI targets' name to SCSI ID, we can then generate a full mapping from
iSCSI target name to local SCSI device name for the host linux1:
iSCSI Target Name to local SCSI Device Name

iSCSI Target Name Host / SCSI ID SCSI Device Name
iqn.2006-01.com.openfiler:rac1.asm4 0 /dev/sdb
iqn.2006-01.com.openfiler:rac1.asm3 1 /dev/sda
iqn.2006-01.com.openfiler:rac1.asm2 2 /dev/sde
iqn.2006-01.com.openfiler:rac1.asm1 3 /dev/sdc
iqn.2006-01.com.openfiler:rac1.crs 4 /dev/sdd
Create Partitions on iSCSI Volumes
The next step is to create a single primary partition on each of the iSCSI volumes that
spans the entire size of the volume. As mentioned earlier in this article, I will be using
Oracle's Cluster File System, Release 2 (OCFS2) to store the two files to be shared for
Oracle's Clusterware software. We will then be using Automatic Storage Management
(ASM) to create four ASM volumes; two for all physical database files (data/index files,
online redo log files, and control files) and two for the Flash Recovery Area (RMAN
backups and archived redo log files).
The following table lists the five iSCSI volumes and what file systems they will support:
Oracle Shared Drive Configuration

File
iSCSI Target Mount ASM Diskgroup
System Size File Types
(short) Name Point Name
Type
/ Oracle Cluster Registry
2
OCFS2 crs u02/orad (OCR) File - (~100 MB)
GB
ata/orcl Voting Disk - (~20MB)
118 ORCL:V
ASM asm1 +ORCL_DATA1 Oracle Database Files
GB OL1
ASM asm2 118 ORCL:V +ORCL_DATA1 Oracle Database Files
GB OL2
118 ORCL:V +FLASH_RECO Oracle Flash Recovery
ASM asm3
GB OL3 VERY_AREA Area
118 ORCL:V +FLASH_RECO Oracle Flash Recovery
ASM asm4
GB OL4 VERY_AREA Area
474
Total
GB
As shown in the table above, we will need to create a single Linux primary partition on
each of the five iSCSI volumes. The fdisk command is used in Linux for creating (and
removing) partitions. For each of the five iSCSI volumes, you can use the default values
when creating the primary partition as the default action is to use the entire disk. You can
safely ignore any warnings that may indicate the device does not contain a valid DOS
partition (or Sun, SGI or OSF disklabel).
For the purpose of this example, I will be running the fdisk command from linux1 to
create a single primary partition for each of the local SCSI devices identified in the
previous section:
/dev/sda
/dev/sdb
/dev/sdc
/dev/sdd
/dev/sde
Please note that creating the partition on each of the iSCSI volumes must only be run
from one of the nodes in the Oracle RAC cluster!
# ---------------------------------------
# fdisk /dev/sda
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-15134, default 1): 1
Last cylinder or +size or +sizeM or +sizeK (1-15134, default 15134):
15134
Command (m for help): p
Disk /dev/sda: 124.4 GB, 124486942720 bytes

255 heads, 63 sectors/track, 15134 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 1 15134 121563823+ 83 Linux
Command (m for help): w

The partition table has been altered!
Calling ioctl() to re-read partition table.

Syncing disks.
# ---------------------------------------
# fdisk /dev/sdb
Command action
e extended
p
15134
Disk /dev/sdb: 124.4 GB, 124486942720 bytes


/dev/sdb1 1 15134 121563823+ 83 Linux


Syncing disks.
# ---------------------------------------
# fdisk /dev/sdc
Command action
e extended
p
15134
Disk /dev/sdc: 124.4 GB, 124486942720 bytes


/dev/sdc1 1 15134 121563823+ 83 Linux


Syncing disks.
# ---------------------------------------
# fdisk /dev/sdd
Command action
e extended
p
Last cylinder or +size or +sizeM or +sizeK (1-1009, default 1009): 1009
Disk /dev/sdd: 2147 MB, 2147483648 bytes


/dev/sdd1 1 1009 2095662 83 Linux


Syncing disks.
# ---------------------------------------
# fdisk /dev/sde
Command action
e extended
p
15134
Disk /dev/sde: 124.4 GB, 124486942720 bytes


/dev/sde1 1 15134 121563823+ 83 Linux

Syncing disks.
# ---------------------------------------
After creating all required partitions, you should now inform the kernel of the partition
changes using the following command as the "root" user account from both of the
Oracle RAC nodes in the cluster:
# partprobe
# fdisk -l
Disk /dev/hda: 40.0 GB, 40000000000 bytes


/dev/hda1 * 1 13 104391 83 Linux
/dev/hda2 14 4863 38957625 8e Linux LVM
Disk /dev/sda: 124.4 GB, 124486942720 bytes


/dev/sda1 1 15134 121563823+ 83 Linux
Disk /dev/sdb: 124.4 GB, 124486942720 bytes


/dev/sdb1 1 15134 121563823+ 83 Linux
Disk /dev/sdc: 124.4 GB, 124486942720 bytes


/dev/sdc1 1 15134 121563823+ 83 Linux
Disk /dev/sdd: 2147 MB, 2147483648 bytes


/dev/sdd1 1 1009 2095662 83 Linux
Disk /dev/sde: 124.4 GB, 124486942720 bytes

/dev/sde1 1 15134 121563823+ 83 Linux
11. Create "oracle" User and Directories
Perform the following tasks on both Oracle RAC nodes in the cluster!
You will be using OCFS2 to store the files required to be shared for the Oracle
Clusterware software. When using OCFS2, the UID of the UNIX user oracle and GID
of the UNIX group dba must be the same on both of the Oracle RAC nodes in the cluster.
If either the UID or GID are different, the files on the OCFS2 file system may show up as
"unowned" or may even be owned by a different user. For this article, I will use 175 for
the oracle UID and 115 for the dba GID.
Create Group and User for Oracle
Let's continue our example by creating the Unix dba group and oracle user account
along with all appropriate directories.
# mkdir -p /u01/app
# groupadd -g 115 dba
# groupadd -g 116 oinstall
# useradd -u 175 -g 115 -d /u01/app/oracle -s /bin/bash -c "Oracle
Software Owner" -p oracle oracle
# chown -R oracle:dba /u01
# passwd oracle
# su - oracle
Note: When you are setting the Oracle environment variables for each Oracle RAC node,
ensure to assign each RAC node a unique Oracle SID! For this example, I used:
linux1 : ORACLE_SID=orcl1
linux2 : ORACLE_SID=orcl2
Create Login Script for oracle User Account
After creating the "oracle" UNIX userid on both nodes, ensure that the environment is
setup correctly by using the following .bash_profile:
....................................
# .bash_profile
# Get the aliases and functions

if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
alias ls="ls -FA"

# User specific environment and startup programs
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/10.2.0/db_1
export ORA_CRS_HOME=$ORACLE_BASE/product/crs
export ORACLE_PATH=$ORACLE_BASE/common/oracle/sql:.:
$ORACLE_HOME/rdbms/admin
# Each RAC node must have a unique ORACLE_SID. (i.e. orcl1, orcl2,...)
export ORACLE_SID=orcl1
export PATH=.:${PATH}:$HOME/bin:$ORACLE_HOME/bin
export PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin
export PATH=${PATH}:$ORACLE_BASE/common/oracle/bin
export ORACLE_TERM=xterm
export TNS_ADMIN=$ORACLE_HOME/network/admin
export ORA_NLS10=$ORACLE_HOME/nls/data
export LD_LIBRARY_PATH=$ORACLE_HOME/lib
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:$ORACLE_HOME/oracm/lib
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
export CLASSPATH=$ORACLE_HOME/JRE
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/jlib
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/rdbms/jlib
export CLASSPATH=${CLASSPATH}:$ORACLE_HOME/network/jlib
export THREADS_FLAG=native
export TEMP=/tmp
export TMPDIR=/tmp
....................................
Create Mount Point for OCFS2 / Clusterware
Finally, create the mount point for the OCFS2 filesystem that will be used to store the
two Oracle Clusterware shared files. These commands will need to be run as the "root"
user account:
$ su -
# mkdir -p /u02/oradata/orcl
# chown -R oracle:dba /u02
12. Configure the Linux Servers for Oracle
Perform the following configuration procedures on both Oracle RAC nodes in the
cluster!
The kernel parameters discussed in this section will need to be defined on both Oracle
RAC nodes in the cluster every time the machine is booted. This section provides very
detailed information about setting those kernel parameters required for Oracle.
Instructions for placing them in a startup script (/etc/sysctl.conf) are included in
Section 15 ("All Startup Commands for Both Oracle RAC Nodes").
Overview
This section focuses on configuring both Oracle RAC Linux servers - getting each one
prepared for the Oracle10g RAC installation. This includes verifying enough swap space,
setting shared memory and semaphores, setting the maximum amount of file handles,
setting the IP local port range, setting shell limits for the oracle user, activating all kernel
parameters for the system, and finally how to verify the correct date and time for both
nodes in the cluster.
Throughout this section you will notice that there are several different ways to configure
(set) these parameters. For the purpose of this article, I will be making all changes
permanent (through reboots) by placing all commands in the /etc/sysctl.conf file.
Swap Space Considerations
Installing Oracle10g Release 2 requires a minimum of 512MB of memory. (Note:

An inadequate amount of swap during the installation will cause the Oracle
Universal Installer to either "hang" or "die")
To check the amount of memory you have, type:
# cat /proc/meminfo | grep MemTotal

MemTotal: 1034180 kB
To check the amount of swap you have allocated, type:
# cat /proc/meminfo | grep SwapTotal

SwapTotal: 2031608 kB
If you have less than 512MB of memory (between your RAM and SWAP), you
can add temporary swap space by creating a temporary swap file. This way you
do not have to use a raw device or even more drastic, rebuild your system.
As root, make a file that will act as additional swap space, let's say about 300MB:
# dd if=/dev/zero of=tempswap bs=1k count=300000
Now we should change the file permissions:
# chmod 600 tempswap
Finally we format the "partition" as swap and add it to the swap space:
# mke2fs tempswap
# mkswap tempswap
# swapon tempswap
Setting Shared Memory
Shared memory allows processes to access common structures and data by placing them
in a shared memory segment. This is the fastest form of inter-process communications
(IPC) available, mainly due to the fact that no kernel involvement occurs when data is
being passed between the processes. Data does not need to be copied between processes.
Oracle makes use of shared memory for its Shared Global Area (SGA) which is an area
of memory that is shared by all Oracle backup and foreground processes. Adequate sizing
of the SGA is critical to Oracle performance because it is responsible for holding the
database buffer cache, shared SQL, access paths, and so much more.
To determine all shared memory limits, use the following:
# ipcs -lm
------ Shared Memory Limits --------
max number of segments = 4096
max seg size (kbytes) = 32768
max total shared memory (kbytes) = 8388608
min seg size (bytes) = 1
Setting SHMMAX
The SHMMAX parameters defines the maximum size (in bytes) for a shared memory
segment. The Oracle SGA is comprised of shared memory and it is possible that
incorrectly setting SHMMAX could limit the size of the SGA. When setting SHMMAX, keep in
mind that the size of the SGA should fit within one shared memory segment. An
inadequate SHMMAX setting could result in the following:
ORA-27123: unable to attach to shared memory segment

You can determine the value of SHMMAX by performing the following:
# cat /proc/sys/kernel/shmmax
33554432
The default value for SHMMAX is 32MB. This size is often too small to configure the
Oracle SGA. I generally set the SHMMAX parameter to 2GB using the following methods:
You can alter the default setting for SHMMAX without rebooting the machine
by making the changes directly to the /proc file system
(/proc/sys/kernel/shmmax) by using the following command:
# sysctl -w kernel.shmmax=2147483648
You should then make this change permanent by inserting the kernel
parameter in the /etc/sysctl.conf startup file:
# echo "kernel.shmmax=2147483648" >> /etc/sysctl.conf
Setting SHMMNI
We now look at the SHMMNI parameters. This kernel parameter is used to set the
maximum number of shared memory segments system wide. The default value for this
parameter is 4096.
You can determine the value of SHMMNI by performing the following:

# cat /proc/sys/kernel/shmmni
4096
The default setting for SHMMNI should be adequate for your Oracle RAC 10g Release 2
installation.
Setting SHMALL
Finally, we look at the SHMALL shared memory kernel parameter. This parameter controls
the total amount of shared memory (in pages) that can be used at one time on the system.
In short, the value of this parameter should always be at least:
ceil(SHMMAX/PAGE_SIZE)
The default size of SHMALL is 2097152 and can be queried using the following command:
# cat /proc/sys/kernel/shmall
2097152
The default setting for SHMALL should be adequate for our Oracle RAC 10g Release 2
installation.
(Note: The page size in Red Hat Linux on the i386 platform is 4,096 bytes. You can,
however, use bigpages which supports the configuration of larger memory page sizes.)
Setting Semaphores
Now that you have configured your shared memory settings, it is time to configure your
semaphores. The best way to describe a "semaphore" is as a counter that is used to
provide synchronization between processes (or threads within a process) for shared
resources like shared memory. Semaphore sets are supported in UNIX System V where
each one is a counting semaphore. When an application requests semaphores, it does so
using "sets."
To determine all semaphore limits, use the following:
# ipcs -ls
------ Semaphore Limits --------
max number of arrays = 128
max semaphores per array = 250
max semaphores system wide = 32000
max ops per semop call = 32
semaphore max value = 32767
You can also use the following command:
# cat /proc/sys/kernel/sem
250 32000 32 128
Setting SEMMSL
The SEMMSL kernel parameter is used to control the maximum number of semaphores per
semaphore set.
Oracle recommends setting SEMMSL to the largest PROCESS instance parameter setting in
the init.ora file for all databases on the Linux system plus 10. Also, Oracle
recommends setting the SEMMSL to a value of no less than 100.
Setting SEMMNI
The SEMMNI kernel parameter is used to control the maximum number of semaphore sets
in the entire Linux system. Oracle recommends setting the SEMMNI to a value of no less
than 100.
Setting SEMMNS
The SEMMNS kernel parameter is used to control the maximum number of semaphores (not
semaphore sets) in the entire Linux system.
Oracle recommends setting the SEMMNS to the sum of the PROCESSES instance parameter
setting for each database on the system, adding the largest PROCESSES twice, and then
finally adding 10 for each Oracle database on the system.
Use the following calculation to determine the maximum number of semaphores that can
be allocated on a Linux system. It will be the lesser of:
SEMMNS -or- (SEMMSL * SEMMNI)
Setting SEMOPM
The SEMOPM kernel parameter is used to control the number of semaphore operations that
can be performed per semop system call.
The semop system call (function) provides the ability to do operations for multiple
semaphores with one semop system call. A semaphore set can have the maximum number
of SEMMSL semaphores per semaphore set and is therefore recommended to set SEMOPM
equal to SEMMSL.
Oracle recommends setting the SEMOPM to a value of no less than 100.
Setting Semaphore Kernel Parameters
Finally, we see how to set all semaphore parameters using several methods. In the
following, the only parameter I care about changing (raising) is SEMOPM. All other default
settings should be sufficient for our example installation.
You can alter the default setting for all semaphore settings without
rebooting the machine by making the changes directly to the /proc file
system (/proc/sys/kernel/sem) by using the following command:
# sysctl -w kernel.sem="250 32000 100 128"
You should then make this change permanent by inserting the kernel
parameter in the /etc/sysctl.conf startup file:
# echo "kernel.sem=250 32000 100 128" >> /etc/sysctl.conf
Setting File Handles
When configuring our Red Hat Linux server, it is critical to ensure that the maximum
number of file handles is sufficiently large. The setting for file handles denotes the
number of open files that you can have on the Linux system.
Use the following command to determine the maximum number of file handles for the
entire system:
# cat /proc/sys/fs/file-max
102563
Oracle recommends that the file handles for the entire system be set to at least 65536.
You can alter the default setting for the maximum number of file handles without
rebooting the machine by making the changes directly to the /proc file system
(/proc/sys/fs/file-max) using the following:
# sysctl -w fs.file-max=65536
You should then make this change permanent by inserting the kernel parameter in
the /etc/sysctl.conf startup file:
# echo "fs.file-max=65536" >> /etc/sysctl.conf
You can query the current usage of file handles by using the following:
# cat /proc/sys/fs/file-nr
825 0 65536
The file-nr file displays three parameters: total allocated file handles, currently used file
handles, and maximum file handles that can be allocated.
(Note: If you need to increase the value in /proc/sys/fs/file-max, then make sure that the
ulimit is set properly. Usually for 2.4.20 it is set to unlimited. Verify the ulimit setting
my issuing the ulimit command:
# ulimit
unlimited
Setting IP Local Port Range
Configure the system to allow a local port range of 1024 through 65000.
Use the following command to determine the value of ip_local_port_range:
# cat /proc/sys/net/ipv4/ip_local_port_range
32768 61000
The default value for ip_local_port_range is ports 32768 through 61000. Oracle
recommends a local port range of 1024 to 65000.
You can alter the default setting for the local port range without rebooting the
machine by making the changes directly to the /proc file system
(/proc/sys/net/ipv4/ip_local_port_range) by using the following
command:
# sysctl -w net.ipv4.ip_local_port_range="1024 65000"

You should then make this change permanent by inserting the kernel parameter in
the /etc/sysctl.conf startup file:
# echo "net.ipv4.ip_local_port_range = 1024 65000" >>

/etc/sysctl.conf
Setting Shell Limits for the oracle User
To improve the performance of the software on Linux systems, Oracle recommends you
increase the following shell limits for the oracle user:
Shell Limit Item in limits.conf Hard Limit
Maximum number of open file descriptors nofile 65536
Maximum number of processes available to a single user nproc 16384
To make these changes, run the following as root:
cat >> /etc/security/limits.conf <<EOF
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
EOF
cat >> /etc/pam.d/login <<EOF
session required /lib/security/pam_limits.so
EOF
Update the default shell startup file for the "oracle" UNIX account.
For the Bourne, Bash, or Korn shell, add the following lines to the /etc/profile
file by running the following command:
cat >> /etc/profile <<EOF

if [ \$USER = "oracle" ]; then
if [ \$SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
umask 022
fi
EOF
For the C shell (csh or tcsh), add the following lines to the /etc/csh.login file
by running the following command:
cat >> /etc/csh.login <<EOF
if ( \$USER == "oracle" ) then
limit maxproc 16384
limit descriptors 65536
endif
EOF
Activating All Kernel Parameters for the System
At this point, we have covered all of the required Linux kernel parameters needed for a
successful Oracle installation and configuration. Within each section above, we
configured the Linux system to persist each of the kernel parameters on system startup by
placing them all in the /etc/sysctl.conf file.
We could reboot at this point to ensure all of these parameters are set in the kernel or we
could simply "run" the /etc/sysctl.conf file by running the following command as
root. Perform this on each node of the cluster!
# sysctl -p
net.ipv4.ip_forward = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.default.accept_source_route = 0
kernel.sysrq = 0
kernel.core_uses_pid = 1
net.core.rmem_default = 262144
net.core.wmem_default = 262144
net.core.rmem_max = 262144
net.core.wmem_max = 262144
kernel.shmmax = 2147483648
kernel.sem = 250 32000 100 128
fs.file-max = 65536
net.ipv4.ip_local_port_range = 1024 65000
Setting the Correct Date and Time on All Cluster Nodes
During the installation of Oracle Clusterware, the Database, and the Companion CD, the
Oracle Universal Installer (OUI) first installs the software to the local node running the
installer (i.e. linux1). The software is then copied remotely to all of the remaining nodes
in the cluster (i.e. linux2). During the remote copy process, the OUI will execute the
UNIX "tar" command on each of the remote nodes to extract the files that were archived
and copied over. If the date and time on the node performing the install is greater than
that of the node it is copying to, the OUI will throw an error from the "tar" command
indicating it is attempting to extract files stamped with a time in the future:
Error while copying directory
/u01/app/oracle/product/crs with exclude file list 'null' to nodes
'linux2'.
[PRKC-1002 : All the submitted commands did not execute successfully]
---------------------------------------------
linux2:
/bin/tar: ./bin/lsnodes: time stamp 2006-09-13 09:21:34 is 735 s in
the future
/bin/tar: ./bin/olsnodes: time stamp 2006-09-13 09:21:34 is 735 s in
the future
...(more errors on this node)
Please note that although this would seem like a severe error from the OUI, it can safely
be disregarded as a warning. The "tar" command DOES actually extract the files;
however, when you perform a listing of the files (using ls -l) on the remote node, they
will be missing the time field until the time on the server is greater than the timestamp of
the file.
Before starting any of the above noted installations, ensure that each member node of the
cluster is set as closely as possible to the same date and time. Oracle strongly
recommends using the Network Time Protocol feature of most operating systems for this
purpose, with both Oracle RAC nodes using the same reference Network Time Protocol
server.
Accessing a Network Time Protocol server, however, may not always be an option. In
this case, when manually setting the date and time for the nodes in the cluster, ensure that
the date and time of the node you are performing the software installations from (linux1)
is less than all other nodes in the cluster (linux2). I generally use a 20 second difference
as shown in the following example:
Setting the date and time from linux1:
# date -s "9/13/2006 23:00:00"
Setting the date and time from linux2:
# date -s "9/13/2006 23:00:20"
The two-node RAC configuration described in this article does not make use of a
Network Time Protocol server.
13. Configure the hangcheck-timer Kernel Module
cluster!
Oracle9i Release 1 (9.0.1) and Oracle9i Release 2 ( 9.2.0.1) used a userspace watchdog
daemon called watchdogd to monitor the health of the cluster and to restart a RAC node
in case of a failure. Starting with Oracle9i Release 2 (9.2.0.2) (and still available in
Oracle 10g Release 2), the watchdog daemon has been deprecated by a Linux kernel
module named hangcheck-timer which addresses availability and reliability problems
much better. The hang-check timer is loaded into the Linux kernel and checks if the
system hangs. It will set a timer and check the timer after a certain amount of time. There
is a configurable threshold to hang-check that, if exceeded will reboot the machine.
Although the hangcheck-timer module is not required for Oracle Clusterware (Cluster
Manager) operation, it is highly recommended by Oracle.
The hangcheck-timer.ko Module
The hangcheck-timer module uses a kernel-based timer that periodically checks the
system task scheduler to catch delays in order to determine the health of the system. If the
system hangs or pauses, the timer resets the node. The hangcheck-timer module uses the
Time Stamp Counter (TSC) CPU register, which is incremented at each clock signal. The
TCS offers much more accurate time measurements because this register is updated by
the hardware automatically.
Much more information about the hangcheck-timer project can be found here.
Installing the hangcheck-timer.ko Module
The hangcheck-timer was normally shipped only by Oracle, however, this module is now
included with Red Hat Linux AS starting with kernel versions 2.4.9-e.12 and higher. The
hangcheck-timer should already be included. Use the following to ensure that you have
the module included:
# find /lib/modules -name "hangcheck-timer.ko"

/lib/modules/2.6.9-42.EL/kernel/drivers/char/hangcheck-timer.ko
In the above output, we care about the hangcheck timer object (hangcheck-timer.ko) in
the /lib/modules/2.6.9-42.EL/kernel/drivers/char directory.
Configuring and Loading the hangcheck-timer Module
There are two key parameters to the hangcheck-timer module:
hangcheck-tick: This parameter defines the period of time between checks of

system health. The default value is 60 seconds; Oracle recommends setting it to
30 seconds.
hangcheck-margin: This parameter defines the maximum hang delay that should
be tolerated before hangcheck-timer resets the RAC node. It defines the margin of
error in seconds. The default value is 180 seconds; Oracle recommends setting it
to 180 seconds.
Note: The two hangcheck-timer module parameters indicate how long a RAC node
must hang before it will reset the system. A node reset will occur when the following is
true:
system hang time > (hangcheck_tick + hangcheck_margin)
Configuring Hangcheck Kernel Module Parameters
Each time the hangcheck-timer kernel module is loaded (manually or by Oracle), it needs
to know what value to use for each of the two parameters we just discussed: (hangcheck-
tick and hangcheck-margin). These values need to be available after each reboot of the
Linux server. To do that, make an entry with the correct values to the
/etc/modprobe.conf file as follows:
# su -
# echo "options hangcheck-timer hangcheck_tick=30 hangcheck_margin=180"
>> /etc/modprobe.conf
Each time the hangcheck-timer kernel module gets loaded, it will use the values defined
by the entry I made in the /etc/modprobe.conf file.
Manually Loading the Hangcheck Kernel Module for Testing
Oracle is responsible for loading the hangcheck-timer kernel module when required. For
that reason, it is not required to perform a modprobe or insmod of the hangcheck-timer
kernel module in any of the startup files (i.e. /etc/rc.local).
It is only out of pure habit that I continue to include a modprobe of the hangcheck-timer
kernel module in the /etc/rc.local file. Someday I will get over it, but realize that it
does not hurt to include a modprobe of the hangcheck-timer kernel module during startup.
So to keep myself sane and able to sleep at night, I always configure the loading of the
hangcheck-timer kernel module on each startup as follows:
# echo "/sbin/modprobe hangcheck-timer" >> /etc/rc.local
(Note: You don't have to manually load the hangcheck-timer kernel module using
modprobe or insmod after each reboot. The hangcheck-timer module will be loaded by
Oracle automatically when needed.)
Now, to test the hangcheck-timer kernel module to verify it is picking up the correct
parameters we defined in the /etc/modprobe.conf file, use the modprobe command.
Although you could load the hangcheck-timer kernel module by passing it the
appropriate parameters (e.g. insmod hangcheck-timer hangcheck_tick=30
hangcheck_margin=180), we want to verify that it is picking up the options we set in the
/etc/modprobe.conf file.
To manually load the hangcheck-timer kernel module and verify it is using the correct
values defined in the /etc/modprobe.conf file, run the following command:
# su -
# modprobe hangcheck-timer
# grep Hangcheck /var/log/messages | tail -2
Oct 10 19:56:24 linux1 kernel: Hangcheck: starting hangcheck timer 0.9.0
(tick is 30 seconds, margin is 180 seconds).
Oct 10 19:56:24 linux1 kernel: Hangcheck: Using monotonic_clock().
14. Configure RAC Nodes for Remote Access
cluster!
Before you can install and use Oracle Real Application clusters, you must configure
either secure shell (SSH) or remote shell (RSH) for the "oracle" UNIX user account on
all cluster nodes. The goal here is to setup user equivalence for the "oracle" UNIX user
account. User equivalence enables the "oracle" UNIX user account to access all other
nodes in the cluster (running commands and copying files) without the need for a
password. This can be configured using either SSH or RSH where SSH is the preferred
method. Oracle added support in 10g Release 1 for using the SSH tool suite for setting up
user equivalence. Before Oracle10g, user equivalence had to be configured using remote
shell.
Note that if the Oracle Universal Installer in 10g does not detect the presence of the
secure shell tools (ssh and scp), it will attempt to use the remote shell tools instead (rsh
and rcp).
So, why do we have to setup user equivalence? Installing Oracle Clusterware and the
Oracle Database software is only performed from one node in a RAC cluster. When
running the Oracle Universal Installer (OUI) on that particular node, it will use the ssh
and scp commands (or rsh and rcp commands if using remote shell) to run remote
commands on and copy files (the Oracle software) to all other nodes within the RAC
cluster. The "oracle" UNIX user account on the node running the OUI (runInstaller)
must be trusted by all other nodes in your RAC cluster. This means that you must be able
to run the secure shell commands (ssh or scp) or the remote shell commands (rsh and
rcp) on the Linux server you will be running the OUI from against all other Linux
servers in the cluster without being prompted for a password.
Note that the use of secure shell or remote shell is not required for normal RAC
operation. This configuration, however, must to be enabled for RAC and patchset
installations as well as creating the clustered database.
The first step is to decide which method of remote access to use - secure shell or remote
shell. Both of them have their pros and cons. Remote shell, for example, is extremely
easy to setup and configure. It takes fewer steps to construct and is always available in
the terminal session when logging on to the trusted node (the node you will be
performing the install from). The connection to the remote nodes, however, is not secure
during the installation and any patching process. Secure shell on the other hand does
provide a secure connection when installing and patching but does require a greater
number of steps. It also needs to be enabled in the terminal session each time the oracle
user logs in to the trusted node. The official Oracle documentation only describes the
steps for setting up secure shell and is considered the preferred method.
Both methods for configuring user equivalence are described in the following two
sections:
Using the Secure Shell Method
Using the Remote Shell Method
Using the Secure Shell Method

This section describes how to configure OpenSSH version 3.
To determine if SSH is installed and running, enter the following command:
# pgrep sshd
2808
If SSH is running, then the response to this command is a list of process ID number(s).
Please run this command on both Oracle RAC nodes in the cluster to verify the SSH
daemons are installed and running!
To find out more about SSH, refer to the man page:
# man ssh
Creating RSA and DSA Keys on Both Oracle RAC Nodes
The first step in configuring SSH is to create RSA and DSA key pairs on both Oracle
RAC nodes in the cluster. The command to do this will create a public and private key for
both RSA and DSA (for a total of four keys per node). The content of the RSA and DSA
public keys will then need to be copied into an authorized key file which is then
distributed to both Oracle RAC nodes in the cluster.
Use the following steps to create the RSA and DSA key pairs. Please note that these steps
will need to be completed on both Oracle RAC nodes in the cluster:
1. Logon as the "oracle" UNIX user account.
# su - oracle
2. If necessary, create the .ssh directory in the "oracle" user's home directory and
set the correct permissions on it:
$ mkdir -p ~/.ssh
$ chmod 700 ~/.ssh
3. Enter the following command to generate an RSA key pair (public and private
key) for version 3 of the SSH protocol:
$ /usr/bin/ssh-keygen -t rsa
At the prompts:
o Accept the default location for the key files.

o Enter and confirm a pass phrase. This should be different from the
"oracle" UNIX user account password however it is not a requirement.
This command will write the public key to the ~/.ssh/id_rsa.pub file and the
private key to the ~/.ssh/id_rsa file. Note that you should never distribute the
private key to anyone!
4. Enter the following command to generate a DSA key pair (public and private key)
for version 3 of the SSH protocol:
$ /usr/bin/ssh-keygen -t dsa
At the prompts:
o Accept the default location for the key files.

o Enter and confirm a pass phrase. This should be different from the
"oracle" UNIX user account password however it is not a requirement.
This command will write the public key to the ~/.ssh/id_dsa.pub file and the
private key to the ~/.ssh/id_dsa file. Note that you should never distribute the
private key to anyone!
5. Repeat the above steps for both Oracle RAC nodes in the cluster.
Now that both Oracle RAC nodes contain a public and private key for both RSA and
DSA, you will need to create an authorized key file on one of the nodes. An authorized
key file is nothing more than a single file that contains a copy of everyone's (every
node's) RSA and DSA public key. Once the authorized key file contains all of the public
keys, it is then distributed to all other nodes in the cluster.
Complete the following steps on one of the nodes in the cluster to create and then
distribute the authorized key file. For the purpose of this article, I am using linux1:
1. First, determine if an authorized key file already exists on the node

(~/.ssh/authorized_keys). In most cases this will not exist since this article
assumes you are working with a new install. If the file doesn't exist, create it now:
$ touch ~/.ssh/authorized_keys
$ cd ~/.ssh
$ ls -l *.pub
-rw-r--r-- 1 oracle dba 603 Aug 31 23:40 id_dsa.pub
-rw-r--r-- 1 oracle dba 223 Aug 31 23:36 id_rsa.pub
The listing above should show the id_rsa.pub and id_dsa.pub public keys
created in the previous section.
2. In this step, use SSH to copy the content of the ~/.ssh/id_rsa.pub and
~/.ssh/id_dsa.pub public key from both Oracle RAC nodes in the cluster to the
authorized key file just created (~/.ssh/authorized_keys). Again, this will be
done from linux1. You will be prompted for the "oracle" UNIX user account
password for both Oracle RAC nodes accessed. Notice that when using SSH to
access the node you are on (linux1), the first time prompts for the "oracle" UNIX
user account password. The second attempt at accessing this node will prompt for
the pass phrase used to unlock the private key. For any of the remaining nodes, it
will always ask for the "oracle" UNIX user account password.
The following example is being run from linux1 and assumes a two-node cluster,
with nodes linux1 and linux2:
$ ssh linux1 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

The authenticity of host 'linux1 (192.168.1.100)' can't be
established.
RSA key fingerprint is
61:8a:f9:9e:28:a2:b7:d3:70:8d:dc:76:ca:d9:23:43.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'linux1,192.168.1.100' (RSA) to the
list of known hosts.
oracle@linux1's password: xxxxx
$ ssh linux1 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
Enter passphrase for key '/u01/app/oracle/.ssh/id_rsa': xxxxx
$ ssh linux2 cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
established.
84:2b:bd:eb:31:2c:23:36:55:c2:ee:54:d2:23:6a:e4.
Warning: Permanently added 'linux2,192.168.1.101' (RSA) to the
list of known hosts.
$ ssh linux2 cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
Note: The first time you use SSH to connect to a node from a particular system,
you may see a message similar to the following:

established.
61:8a:f9:9e:28:a2:b7:d3:70:8d:dc:76:ca:d9:23:43.
Enter yes at the prompt to continue. You should not see this message again when
you connect from this system to the same node.
3. At this point, we have the content of the RSA and DSA public keys from every
node in the cluster in the authorized key file (~/.ssh/authorized_keys) on
linux1. We now need to copy it to the remaining nodes in the cluster. In our two-
node cluster example, the only remaining node is linux2. Use the scp command
to copy the authorized key file to all remaining nodes in the cluster:
$ scp ~/.ssh/authorized_keys linux2:.ssh/authorized_keys

authorized_keys 100% 1652 1.6KB/s 00:00
4. Change the permission of the authorized key file for both Oracle RAC nodes in
the cluster by logging into the node and running the following:
$ chmod 600 ~/.ssh/authorized_keys

5. At this point, if you use ssh to log in to or run a command on another node, you
are prompted for the pass phrase that you specified when you created the DSA
key. For example, test the following from linux1:
$ ssh linux1 hostname

linux1
linux2
Note: If you see any other messages or text, apart from the host name, then the
Oracle installation can fail. Make any changes required to ensure that only the
host name is displayed when you enter these commands. You should ensure that
any part of a login script(s) that generate any output, or ask any questions, are
modified so that they act only when the shell is an interactive shell.
Enabling SSH User Equivalency for the Current Shell Session

When running the OUI, it will need to run the secure shell tool commands (ssh and scp)
without being prompted for a pass phrase. Even though SSH is configured on both Oracle
RAC nodes in the cluster, using the secure shell tool commands will still prompt for a
pass phrase. Before running the OUI, you need to enable user equivalence for the
terminal session you plan to run the OUI from. For the purpose of this article, all Oracle
installations will be performed from linux1.
User equivalence will need to be enabled on any new terminal shell session before
attempting to run the OUI. If you log out and log back in to the node you will be
performing the Oracle installation from, you must enable user equivalence for the
terminal shell session as this is not done by default.
To enable user equivalence for the current terminal shell session, perform the following
steps:
1. Logon to the node where you want to run the OUI from (linux1) as the "oracle"
UNIX user account.
# su - oracle
2. Enter the following commands:
$ exec /usr/bin/ssh-agent $SHELL

$ /usr/bin/ssh-add
Enter passphrase for /u01/app/oracle/.ssh/id_rsa: xxxxx
Identity added: /u01/app/oracle/.ssh/id_rsa
(/u01/app/oracle/.ssh/id_rsa)
Identity added: /u01/app/oracle/.ssh/id_dsa
(/u01/app/oracle/.ssh/id_dsa)
At the prompts, enter the pass phrase for each key that you generated.
3. If SSH is configured correctly, you will be able to use the ssh and scp commands
without being prompted for a password or pass phrase from this terminal session:
$ ssh linux1 "date;hostname"

Wed Sep 13 17:35:30 EDT 2006
linux1
Wed Sep 13 17:36:14 EDT 2006
linux2
Note: The commands above should display the date set on both Oracle RAC
nodes along with its hostname. If any of the nodes prompt for a password or pass
phrase then verify that the ~/.ssh/authorized_keys file on that node contains
the correct public keys. Also, if you see any other messages or text, apart from the
date and hostname, then the Oracle installation can fail. Make any changes
required to ensure that only the date is displayed when you enter these commands.
You should ensure that any part of a login script(s) that generate any output, or
ask any questions, are modified so that they act only when the shell is an
interactive shell.
4. The Oracle Universal Installer is a GUI interface and requires the use of an X
Server. From the terminal session enabled for user equivalence (the node you will
be performing the Oracle installations from), set the environment variable
DISPLAY to a valid X Windows display:
Bourne, Korn, and Bash shells:
$ DISPLAY=<Any X-Windows Host>:0

$ export DISPLAY
C shell:
$ setenv DISPLAY <Any X-Windows Host>:0
After setting the DISPLAY variable to a valid X Windows display, you should
perform another test of the current terminal session to ensure that X11 forwarding
is not enabled:

linux1
linux2
Note: If you are using a remote client to connect to the node performing the
installation, and you see a message similar to: "Warning: No xauth data;
using fake authentication data for X11 forwarding." then this means
that your authorized keys file is configured correctly; however, your SSH
configuration has X11 forwarding enabled. For example:
$ export DISPLAY=melody:0
Warning: No xauth data; using fake authentication data for X11
forwarding.
linux2
Note that having X11 Forwarding enabled will cause the Oracle installation to
fail. To correct this problem, create a user-level SSH client configuration file for
the "oracle" UNIX user account that disables X11 Forwarding:
o Using a text editor, edit or create the file ~/.ssh/config
o Make sure that the ForwardX11 attribute is set to no. For example, insert
the following into the ~/.ssh/config file:
Host *
ForwardX11 no
5. You must run the Oracle Universal Installer from this terminal session or
remember to repeat the steps to enable user equivalence (steps 2, 3, and 4 from
this section) before you start the Oracle Universal Installer from a different
terminal session.
Remove any stty Commands

When installing the Oracle software, any hidden files on the system (i.e. .bashrc,
.cshrc, .profile) will cause the installation process to fail if they contain stty
commands.
To avoid this problem, you must modify these files to suppress all output on STDERR as in
the following examples:
Bourne, Bash, or Korn shell:
if [ -t 0 ]; then
stty intr ^C
fi
C shell:
test -t 0
if ($status == 0) then
stty intr ^C
endif
Note: If there are hidden files that contain stty commands that are loaded by the remote
shell, then OUI indicates an error and stops the installation.
Using the Remote Shell Method

The services provided by remote shell are disabled by default on most Linux systems.
This section describes the tasks required for enabling and configuring user equivalence
for use by the Oracle Universal Installer when commands should be run and files copied
to the remote nodes in the cluster using the remote shell tools. The goal is to enable the
Oracle Universal Installer to use rsh and rcp to run commands and copy files to a remote
node without being prompted for a password. Please note that using the remote shell
method for configuring user equivalence is not secure.
The rsh daemon validates users using the /etc/hosts.equiv file or the .rhosts file
found in the user's (oracle's) home directory.
First, let's make sure that we have the rsh RPMs installed on both Oracle RAC nodes in
the cluster:
# rpm -q rsh rsh-server
rsh-0.17-25.4
rsh-server-0.17-25.4
From the above, we can see that we have the rsh and rsh-server installed. Were rsh
not installed, we would run the following command from the CD where the RPM is
located:
# su -
# rpm -ivh rsh-0.17-25.4.i386.rpm rsh-server-0.17-25.4.i386.rpm
To enable the "rsh" and "rlogin" services, the "disable" attribute in the
/etc/xinetd.d/rsh file must be set to "no" and xinetd must be reloaded. This can be
done by running the following commands on all nodes in the cluster:
# su -
# chkconfig rsh on
# chkconfig rlogin on
# service xinetd reload
Reloading configuration: [ OK ]
To allow the "oracle" UNIX user account to be trusted among the RAC nodes, create the
/etc/hosts.equiv file on all nodes in the cluster:
# su -
# touch /etc/hosts.equiv
# chmod 600 /etc/hosts.equiv
# chown root.root /etc/hosts.equiv
Now add all RAC nodes to the /etc/hosts.equiv file similar to the following example
for both Oracle RAC nodes in the cluster:
# cat /etc/hosts.equiv
+linux1 oracle
+linux2 oracle
+linux1-priv oracle
+linux2-priv oracle
Note: In the above example, the second field permits only the oracle user account to run
rsh commands on the specified nodes. For security reasons, the /etc/hosts.equiv file
should be owned by root and the permissions should be set to 600. In fact, some systems
will only honor the content of this file if the owner of this file is root and the permissions
are set to 600.
Before attempting to test your rsh command, ensure that you are using the correct
version of rsh. By default, Red Hat Linux puts /usr/kerberos/sbin at the head of the
$PATH variable. This will cause the Kerberos version of rsh to be executed.
I will typically rename the Kerberos version of rsh so that the normal rsh command will
be used. Use the following:
# su -
# which rsh
/usr/kerberos/bin/rsh
# mv /usr/kerberos/bin/rsh /usr/kerberos/bin/rsh.original
# mv /usr/kerberos/bin/rcp /usr/kerberos/bin/rcp.original
# mv /usr/kerberos/bin/rlogin /usr/kerberos/bin/rlogin.original
# which rsh
/usr/bin/rsh
You should now test your connections and run the rsh command from the node that will
be performing the Oracle Clusterware and 10g RAC installation. I will be using the node
linux1 to perform all installs so this is where I will run the following commands from:
# su - oracle
$ rsh linux1 ls -l /etc/hosts.equiv
-rw------- 1 root root 68 Sep 27 23:37 /etc/hosts.equiv
$ rsh linux1-priv ls -l /etc/hosts.equiv
$ rsh linux2 ls -l /etc/hosts.equiv
$ rsh linux2-priv ls -l /etc/hosts.equiv
Unlike when using secure shell, no other actions or commands are needed to enable user
equivalence using the remote shell. User equivalence will be enabled for the "oracle"
UNIX user account after successfully logging in to a terminal session.
15. All Startup Commands for Both Oracle RAC Nodes
Verify that the following startup commands are included on both of the Oracle RAC
nodes in the cluster!
Up to this point, we have talked in great detail about the parameters and resources that
need to be configured on both nodes in the Oracle10g RAC configuration. This section
will take a deep breath and recap those parameters, commands, and entries (in previous
sections of this document) that need to happen on both Oracle RAC nodes when the
machine is booted.
For each of the startup files below, entries in gray should be included in each startup file.
/etc/modprobe.conf
(All parameters and values to be used by kernel modules.)
alias eth0 b44

alias eth1 tulip
alias snd-card-0 snd-intel8x0
options snd-card-0 index=0
alias usb-controller ehci-hcd
alias usb-controller1 uhci-hcd
options hangcheck-timer hangcheck_tick=30 hangcheck_margin=180
/etc/sysctl.conf
(We wanted to adjust the default and maximum send buffer size as well as the default and
maximum receive buffer size for the interconnect. This file also contains those
parameters responsible for configuring shared memory, semaphores, file handles, and
local IP range for use by the Oracle instance.)
# Kernel sysctl configuration file for Red Hat Linux

#
# For binary values, 0 is disabled, 1 is enabled. See sysctl(8) and
# sysctl.conf(5) for more details.
# Controls IP packet forwarding

net.ipv4.ip_forward = 0
# Controls source route verification

net.ipv4.conf.default.rp_filter = 1
# Controls the System Request debugging functionality of the kernel

kernel.sysrq = 0
# Controls whether core dumps will append the PID to the core filename.
# Useful for debugging multi-threaded applications.
kernel.core_uses_pid = 1
# Default setting in bytes of the socket receive buffer

net.core.rmem_default=262144
# Default setting in bytes of the socket send buffer

net.core.wmem_default=262144
# Maximum socket receive buffer size which may be set by using

# the SO_RCVBUF socket option
net.core.rmem_max=262144
# Maximum socket send buffer size which may be set by using

# the SO_SNDBUF socket option
net.core.wmem_max=262144
# +---------------------------------------------------------+
# | SHARED MEMORY |
# +---------------------------------------------------------+
kernel.shmmax=2147483648
# +---------------------------------------------------------+
# | SEMAPHORES |
# | ---------- |
# | |
# | SEMMSL_value SEMMNS_value SEMOPM_value SEMMNI_value |
# | |
# +---------------------------------------------------------+
kernel.sem=250 32000 100 128
# +---------------------------------------------------------+
# | FILE HANDLES |
# ----------------------------------------------------------+
fs.file-max=65536
# +---------------------------------------------------------+
# | LOCAL IP RANGE |
# ----------------------------------------------------------+
net.ipv4.ip_local_port_range=1024 65000
/etc/hosts
(All machine/IP entries for nodes in our RAC cluster.)
# Do not remove the following line, or various programs

# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
192.168.1.100 linux1
192.168.1.101 linux2
192.168.2.100 linux1-priv
192.168.2.101 linux2-priv
192.168.1.200 linux1-vip
192.168.1.201 linux2-vip
192.168.1.106 melody
192.168.1.102 alex
192.168.1.105 bartman
/etc/hosts.equiv
(The /etc/hosts.equiv file is only required when using the remote shell method to establish
remote access and user equivalency. Allow logins to both Oracle RAC nodes as the
oracle user account without the need for a password when using the remote shell method
for enabling user equivalency.)
+linux1 oracle
+linux2 oracle
+linux1-priv oracle
+linux2-priv oracle
/etc/rc.local
(Loading the hangcheck-timer kernel module.)
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.
touch /var/lock/subsys/local
# +---------------------------------------------------------+
# | HANGCHECK TIMER |
# | (I do not believe this is required, but doesn't hurt) |
# ----------------------------------------------------------+
/sbin/modprobe hangcheck-timer
16. Install & Configure Oracle Cluster File System (OCFS2)
Most of the configuration procedures in this section should be performed on both Oracle
RAC nodes in the cluster! Creating the OCFS2 filesystem, however, should only be
executed on one of nodes in the RAC cluster.
It is now time to install the Oracle Cluster File System, Release 2 (OCFS2). OCFS2,
developed by Oracle Corporation, is a Cluster File System which allows all nodes in a
cluster to concurrently access a device via the standard file system interface. This allows
for easy management of applications that need to run across a cluster.
OCFS (Release 1) was released in December 2002 to enable Oracle Real Application
Cluster (RAC) users to run the clustered database without having to deal with RAW
devices. The file system was designed to store database related files, such as data files,
control files, redo logs, archive logs, etc. OCFS2 is the next generation of the Oracle
Cluster File System. It has been designed to be a general purpose cluster file system.
With it, one can store not only database related files on a shared disk, but also store
Oracle binaries and configuration files (shared Oracle Home) making management of
RAC even easier.
In this guide, you will be using the latest release of OCFS2 (OCFS2 Release 1.2.3-1 at
the time of this writing) to store the two files that are required to be shared by the Oracle
Clusterware software. Along with these two files, you will also be using this space to
store the shared SPFILE for all Oracle RAC instances.
See this page for more information on OCFS2 (including Installation Notes) for Linux.
Download OCFS2
First, download the latest OCFS2 distribution. The OCFS2 distribution comprises of two
sets of RPMs; namely, the kernel module and the tools. The latest kernel module is
available for download from http://oss.oracle.com/projects/ocfs2/files/ and the tools from
http://oss.oracle.com/projects/ocfs2-tools/files/.
Download the appropriate RPMs starting with the latest OCFS2 kernel module (the
driver). With CentOS 4.4 Enterprise Linux, I am using kernel release 2.6.9-42.EL. The
appropriate OCFS2 kernel module was found in the latest release of OCFS2 at the time of
this writing (OCFS2 Release 1.2.3-1). The available OCFS2 kernel modules for Linux
kernel 2.6.9-42.EL are listed below. Always download the latest OCFS2 kernel module
that matches the distribution, platform, kernel version and the kernel flavor (smp,
hugemem, psmp, etc).
ocfs2-2.6.9-42.EL-1.2.3-1.i686.rpm - (for single processor)

or
ocfs2-2.6.9-42.ELsmp-1.2.3-1.i686.rpm - (for multiple processors)
or
ocfs2-2.6.9-42.ELhugemem-1.2.3-1.i686.rpm - (for hugemem)
For the tools, simply match the platform and distribution. You should download the
OCFS2 tools as well as the OCFS2 console applications.
ocfs2-tools-1.2.1-1.i386.rpm - (OCFS2 tools)

ocfs2console-1.2.1-1.i386.rpm - (OCFS2 console)
The OCFS2 Console is optional but highly recommended. The ocfs2console

application requires e2fsprogs, glib2 2.2.3 or later, vte 0.11.10 or later, pygtk2 (EL4) or
python-gtk (SLES9) 1.99.16 or later, python 2.3 or later and ocfs2-tools.
If you were curious as to which OCFS2 driver release you need, use the OCFS2 release
that matches your kernel version. To determine your kernel release:
$ uname -a
Linux linux1 2.6.9-42.EL #1 Sat Aug 12 09:17:58 CDT 2006 i686 i686 i386
GNU/Linux
In the absence of the string "smp" after the string "EL", we are running a single processor
(Uniprocessor) machine. If the string "smp" were to appear, then you would be running
on a multi-processor machine.
Install OCFS2
I will be installing the OCFS2 files onto two single-processor machines. The installation
process is simply a matter of running the following command on both Oracle RAC nodes
in the cluster as the root user account:
$ su -
# rpm -Uvh ocfs2-2.6.9-42.EL-1.2.3-1.i686.rpm \
ocfs2console-1.2.1-1.i386.rpm \
ocfs2-tools-1.2.1-1.i386.rpm
Preparing... ###########################################
[100%]
1:ocfs2-tools ###########################################
[ 33%]
2:ocfs2-2.6.9-42.EL ###########################################
[ 67%]
3:ocfs2console ###########################################
[100%]
Disable SELinux (RHEL4 U2 and higher)
Users of RHEL4 U2 and higher (CentOS 4.4 is based on RHEL4 U4) are advised that
OCFS2 currently does not work with SELinux enabled. If you are using RHEL4 U2 or
higher (which includes us since we are using CentOS 4.4) you will need to disable
SELinux (using tool system-config-securitylevel) to get the O2CB service to
execute.
To disable SELinux, run the "Security Level Configuration" GUI utility:
# /usr/bin/system-config-securitylevel &
This will bring up the following screen:
Figure 13 Security Level Configuration Opening Screen
Now, click the SELinux tab and check off the "Enabled" checkbox. After clicking on
[OK], you will be presented with a warning dialog. Simply acknowledge this warning by
clicking "Yes". Your screen should now look like the following after disabling the
SELinux option:
Figure 14 SELinux Disabled
After making this change on both nodes in the cluster, both Oracle RAC nodes will need
to be rebooted to implement the change. SELinux must be disabled before you can
continue with configuring OCFS2!
# init 6
Configure OCFS2
The next step is to generate and configure the /etc/ocfs2/cluster.conf file on both
Oracle RAC nodes in the cluster. The easiest way to accomplish this is to run the GUI
tool ocfs2console. In this section, we will not only create and configure the
/etc/ocfs2/cluster.conf file using ocfs2console, but will also create and start the
cluster stack O2CB. When the /etc/ocfs2/cluster.conf file is not present, (as will be
the case in our example), the ocfs2console tool will create this file along with a new
cluster stack service (O2CB) with a default cluster name of ocfs2. This will need to be
done on both Oracle RAC nodes in the cluster as the root user account:
$ su -
# ocfs2console &
This will bring up the GUI as shown below:
Figure 15 ocfs2console GUI
Using the ocfs2console GUI tool, perform the following steps:
1. Select [Cluster] -> [Configure Nodes...]. This will start the OCFS2
Cluster Stack (Figure 16) and bring up the "Node Configuration" dialog.
2. On the "Node Configuration" dialog, click the [Add] button.
o This will bring up the "Add Node" dialog.
o In the "Add Node" dialog, enter the Host name and IP address for the first
node in the cluster. Leave the IP Port set to its default value of 7777. In
my example, I added both nodes using linux1 / 192.168.1.100 for the
first node and linux2 / 192.168.1.101 for the second node.
o Click [Apply] on the "Node Configuration" dialog - All nodes should now
be "Active" as shown in Figure 17.
o Click [Close] on the "Node Configuration" dialog.
3. After verifying all values are correct, exit the application using [File] ->
[Quit]. This needs to be performed on both Oracle RAC nodes in the cluster.
Figure 16. Starting the OCFS2 Cluster Stack
The following dialog show the OCFS2 settings I used for the node linux1 and linux2:
Figure 17 Configuring Nodes for OCFS2
After exiting the ocfs2console, you will have a /etc/ocfs2/cluster.conf similar to

the following. This process needs to be completed on both Oracle RAC nodes in the
cluster and the OCFS2 configuration file should be exactly the same for all of the nodes:
node:
ip_port = 7777
ip_address = 192.168.1.100
number = 0
name = linux1
cluster = ocfs2
node:
ip_port = 7777
ip_address = 192.168.1.101
number = 1
name = linux2
cluster = ocfs2
cluster:
node_count = 2
name = ocfs2
O2CB Cluster Service
Before we can do anything with OCFS2 like formatting or mounting the file system, we
need to first have OCFS2's cluster stack, O2CB, running (which it will be as a result of
the configuration process performed above). The stack includes the following services:
NM: Node Manager that keep track of all the nodes in the cluster.conf
HB: Heart beat service that issues up/down notifications when nodes join or leave
the cluster
TCP: Handles communication between the nodes
DLM: Distributed lock manager that keeps track of all locks, its owners and
status
CONFIGFS: User space driven configuration file system mounted at /config
DLMFS: User space interface to the kernel space DLM
All of the above cluster services have been packaged in the o2cb system service
(/etc/init.d/o2cb). Here is a short listing of some of the more useful commands and
options for the o2cb system service.
Note: The following commands are for demonstration purposes only and should not be
run when installing and configuring OCFS2!
/etc/init.d/o2cb status
Module "configfs": Not loaded

Filesystem "configfs": Not mounted
Module "ocfs2_nodemanager": Not loaded
Module "ocfs2_dlm": Not loaded
Module "ocfs2_dlmfs": Not loaded
Filesystem "ocfs2_dlmfs": Not mounted
Note that with this example, all of the services are not loaded. I did an "unload"
right before executing the "status" option. If you were to check the status of the
o2cb service immediately after configuring OCFS2 using ocfs2console utility,
they would all be loaded.
/etc/init.d/o2cb load
Loading module "configfs": OK

Mounting configfs filesystem at /config: OK
Loading module "ocfs2_nodemanager": OK
Loading module "ocfs2_dlm": OK
Loading module "ocfs2_dlmfs": OK
Mounting ocfs2_dlmfs filesystem at /dlm: OK
Loads all OCFS2 modules.

/etc/init.d/o2cb online ocfs2
Starting cluster ocfs2: OK
The above command will online the cluster we created, ocfs2.
/etc/init.d/o2cb offline ocfs2
Unmounting ocfs2_dlmfs filesystem: OK

Unloading module "ocfs2_dlmfs": OK
Unmounting configfs filesystem: OK
Unloading module "configfs": OK
The above command will offline the cluster we created, ocfs2.
/etc/init.d/o2cb unload
Cleaning heartbeat on ocfs2: OK

Stopping cluster ocfs2: OK
The above command will unload all OCFS2 modules.
Configure O2CB to Start on Boot
You now need to configure the on-boot properties of the OC2B driver so that the cluster
stack services will start on each boot. All the tasks within this section will need to be
performed on both nodes in the cluster.
NOTE: With releases of OCFS2 prior to 1.2.1, a bug existed where the driver would not
get loaded on each boot even after configuring the on-boot properties to do so. This bug
was fixed in release 1.2.1 of OCFS2 and does not need to be addressed in this article. If
however you are using a release of OCFS2 prior to 1.2.1, please see the Troubleshooting
section for a workaround to this bug.
Set the on-boot properties as follows:
# /etc/init.d/o2cb offline ocfs2

# /etc/init.d/o2cb unload
# /etc/init.d/o2cb configure
Configuring the O2CB driver.
This will configure the on-boot properties of the O2CB driver.
The following questions will determine whether the driver is loaded on

boot. The current values will be shown in brackets ('[]'). Hitting
<ENTER> without typing an answer will keep that current value. Ctrl-C
will abort.
Load O2CB driver on boot (y/n) [n]: y
Cluster to start on boot (Enter "none" to clear) [ocfs2]: ocfs2
Writing O2CB configuration: OK
Format the OCFS2 Filesystem
Note: Unlike the other tasks in this section, creating the OCFS2 file system should only
be executed on one of nodes in the RAC cluster. I will be executing all commands in this
section from linux1 only.
We can now start to make use of the iSCSI volume we partitioned for OCFS2 in the
section "Create Partitions on iSCSI Volumes".
It is extremely important to note that at this point in the article you may have rebooted
linux1 several times after partitioning the iSCSI volume to be used for OCFS2 (e.g.
/dev/sdd1). This means that the mapping of iSCSI target names discovered from
Openfiler to the local SCSI device name on linux1 may be different. Please repeat the
procedures documented in the section "Discovering iSCSI Targets" to determine if the
iSCSI target names have been discovered as a different local SCSI device name on
linux1 if you have rebooted.
For example, when creating the primary partition on the iSCSI volume to be used for
OCFS2, I documented that the iSCSI target name "iqn.2006-
01.com.openfiler:rac1.crs" mapped to the local SCSI device name /dev/sdd. Then,
earlier in this section, I had to reboot both nodes when disabling SELinux. After working
through the procedures documented in the section "Discovering iSCSI Targets", I
determined that "iqn.2006-01.com.openfiler:rac1.crs" was now mapped to the
local SCSI device name /dev/sde. This means I will be creating the OCFS2 file system
on partition /dev/sde1 in this section! Please note that the local SCSI device name will
most likely be different on your machine.
If the O2CB cluster is offline, start it. The format operation needs the cluster to be online,
as it needs to ensure that the volume is not mounted on some node in the cluster.
Earlier in this document, we created the directory /u02/oradata/orcl under the section
Create Mount Point for OCFS2/Clusterware. This section contains the commands to
create and mount the file system to be used for the Cluster Manager -
/u02/oradata/orcl.
Note that it is possible to create and mount the OCFS2 file system using either the GUI
tool ocfs2console or the command-line tool mkfs.ocfs2. From the ocfs2console
utility, use the menu [Tasks] - [Format].
See the instructions below on how to create the OCFS2 file system using the command-
line tool mkfs.ocfs2.
To create the file system, we can use the Oracle executable mkfs.ocfs2. For the purpose
of this example, I run the following command only from linux1 as the root user account
using the local SCSI device name mapped to the iSCSI volume for crs /dev/sde1.
Also note that I specified a label named "oracrsfiles" which will be referred to when
mounting or un-mounting the volume:
$ su -
# mkfs.ocfs2 -b 4K -C 32K -N 4 -L oracrsfiles /dev/sde1
mkfs.ocfs2 1.2.1
Filesystem label=oracrsfiles
Block size=4096 (bits=12)
Cluster size=32768 (bits=15)
Volume size=2145943552 (65489 clusters) (523912 blocks)
3 cluster groups (tail covers 977 clusters, rest cover 32256 clusters)
Journal size=67108864
Initial number of node slots: 4
Creating bitmaps: done
Initializing superblock: done
Writing system files: done
Writing superblock: done
Formatting Journals: done
Writing lost+found: done
mkfs.ocfs2 successful
Mount the OCFS2 Filesystem
Now that the file system is created, we can mount it. Let's first do it using the command-
line, then I'll show how to include it in the /etc/fstab to have it mount on each boot.
Note: Mounting the file system will need to be performed on both nodes in the Oracle
RAC cluster as the root user account using the OCFS2 label oracrsfiles!
First, here is how to manually mount the OCFS2 file system from the command-line.
Remember that this needs to be performed as the root user account:
$ su -
# mount -t ocfs2 -o datavolume,nointr -L "oracrsfiles" /u02/oradata/orcl
If the mount was successful, you will simply get your prompt back. We should, however,
run the following checks to ensure the file system is mounted correctly.
Let's use the mount command to ensure that the new file system is really mounted. This
should be performed on both nodes in the RAC cluster:
# mount
/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
usbfs on /proc/bus/usb type usbfs (rw)
/dev/hda1 on /boot type ext3 (rw)
none on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
cartman:SHARE2 on /cartman type nfs (rw,addr=192.168.1.120)
configfs on /config type configfs (rw)
ocfs2_dlmfs on /dlm type ocfs2_dlmfs (rw)
/dev/sde1 on /u02/oradata/orcl type ocfs2
(rw,_netdev,datavolume,nointr,heartbeat=local)
Note: Please take note of the datavolume option I am using to mount the new file
system. Oracle database users must mount any volume that will contain the Voting Disk
file, Cluster Registry (OCR), Data files, Redo logs, Archive logs and Control files with
the datavolume mount option so as to ensure that the Oracle processes open the files
with the o_direct flag. The nointr option ensures that the I/O's are not interrupted by
signals.
Any other type of volume, including an Oracle home (which I will not be using for this
article), should not be mounted with this mount option.
Why does it take so much time to mount the volume? It takes around 5 seconds for a
volume to mount. It does so as to let the heartbeat thread stabilize. In a later release,
Oracle plans to add support for a global heartbeat, which will make most mounts instant.
Configure OCFS2 to Mount Automatically at Startup
Let's take a look at what you've have done so far. You downloaded and installed the
OCFS2, which will be used to store the files needed by Cluster Manager files. After
going through the install, you loaded the OCFS2 module into the kernel and then
formatted the clustered file system. Finally, you mounted the newly created file system
using the OCFS2 label "oracrsfiles". This section walks through the steps responsible
for mounting the new OCFS2 file system each time the machine(s) are booted using its
label.
Start by adding the following line to the /etc/fstab file on both Oracle RAC nodes in
the cluster:
LABEL=oracrsfiles /u02/oradata/orcl ocfs2 _netdev,datavolume,nointr

0 0
Notice the "_netdev" option for mounting this file system. The _netdev mount option is
a must for OCFS2 volumes. This mount option indicates that the volume is to be mounted
after the network is started and dismounted before the network is shutdown.
Now, let's make sure that the ocfs2.ko kernel module is being loaded and that the file
system will be mounted during the boot process.
If you have been following along with the examples in this article, the actions to load the
kernel module and mount the OCFS2 file system should already be enabled. However,
you should still check those options by running the following on both Oracle RAC nodes
in the cluster as the root user account:
$ su -
# chkconfig --list o2cb
o2cb 0:off 1:off 2:on 3:on 4:on 5:on 6:off
The flags that I have marked in bold should be set to "on".
Check Permissions on New OCFS2 Filesystem
Use the ls command to check ownership. The permissions should be set to 0775 with
owner "oracle" and group "dba".
Let's first check the permissions:
# ls -ld /u02/oradata/orcl
drwxr-xr-x 3 root root 4096 Sep 29 12:11 /u02/oradata/orcl
As you can see from the listing above, the oracle user account (and the dba group) will
not be able to write to this directory. Let's fix that:
# chown oracle.dba /u02/oradata/orcl
# chmod 775 /u02/oradata/orcl
Let's now go back and re-check that the permissions are correct for both Oracle RAC
nodes in the cluster:
# ls -ld /u02/oradata/orcl
drwxrwxr-x 3 oracle dba 4096 Sep 29 12:11 /u02/oradata/orcl
Adjust the O2CB Heartbeat Threshold
This is a very important section when configuring OCFS2 for use by Oracle
Clusterware's two shared files. With previous versions of this article, (using FireWire as
opposed to iSCSI for the shared storage), I was able to install and configure OCFS2,
format the new volume, and finally install Oracle Clusterware (with its two required
shared files; the voting disk and OCR file), located on the new OCFS2 volume. While I
was able to install Oracle Clusterware and see the shared drive using FireWire, however,
I was receiving many lock-ups and hanging after about 15 minutes when the Clusterware
software was running on both nodes. It always varied on which node would hang (either
linux1 or linux2 in my example). It also didn't matter whether there was a high I/O load
or none at all for it to crash (hang).
After looking through the trace files for OCFS2, it was apparent that access to the voting
disk was too slow (exceeding the O2CB heartbeat threshold) and causing the Oracle
Clusterware software (and the node) to crash.
The solution I used was to simply increase the O2CB heartbeat threshold from its default
setting of 7, to 61. Some setups may even require a higher setting. This is a configurable
parameter that is used to compute the time it takes for a node to "fence" itself.
Whether using FireWire or iSCSI for the shared storage, let's keep in mind that the
configuration I am creating is a rather low-end setup that is being used for educational
purposes only. This is by no means a high-end setup and may be susceptible to bogus
timeouts. It is therefore highly recommended to use the tasks in this section to increase
the O2CB heartbeat threshold to account for the slow access to our shared drive. I should
note that I was able to create the configuration documented in this article (using iSCSI)
without the need to increase the O2CB heartbeat threshold. I do, however, like to keep
this section included in the article in for those who may experience the same situation
given their hardware selection.
First, let's see how to determine what the O2CB heartbeat threshold is currently set to.
This can be done by querying the /proc file system as follows:
# cat /proc/fs/ocfs2_nodemanager/hb_dead_threshold
7
The value is 7, but what does this value represent? Well, it is used in the formula below to
determine the fence time (in seconds):
[fence time in seconds] = (O2CB_HEARTBEAT_THRESHOLD - 1) * 2
So, with a O2CB heartbeat threshold of 7, you would have a fence time of:
(7 - 1) * 2 = 12 seconds
You may need a much larger threshold (120 seconds to be exact) if you are using a slow
network and external disk. For 120 seconds, you will want a
O2CB_HEARTBEAT_THRESHOLD of 61 as shown below:
(61 - 1) * 2 = 120 seconds
Let's see now how to increase the O2CB heartbeat threshold from 7 to 61. This will need
to be performed on both Oracle RAC nodes in the cluster. You first need to modify the
file /etc/sysconfig/o2cb and set O2CB_HEARTBEAT_THRESHOLD to 61:
# O2CB_ENABELED: 'true' means to load the driver on boot.

O2CB_ENABLED=true
# O2CB_BOOTCLUSTER: If not empty, the name of a cluster to start.

O2CB_BOOTCLUSTER=ocfs2
# O2CB_HEARTBEAT_THRESHOLD: Iterations before a node is considered dead.

O2CB_HEARTBEAT_THRESHOLD=61
After modifying the file /etc/sysconfig/o2cb, you need to alter the o2cb
configuration. Again, this should be performed on both Oracle RAC nodes in the cluster.
# umount /u02/oradata/orcl/
Load O2CB driver on boot (y/n) [y]: y

You can now check again to make sure the settings took place in for the o2cb cluster
stack:
61
It is important to note that the value of 61 I used for the O2CB heartbeat threshold will
not work for all configurations. In some cases, the O2CB heartbeat threshold value had to
be increased to as high as 601 in order to prevent OCFS2 from panicking the kernel.
Reboot Both Nodes
Before starting the next section, this would be a good place to reboot both of the nodes in
the RAC cluster. When the machines come up, ensure that the cluster stack services are
being loaded and the new OCFS2 file system is being mounted:
# mount
/dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)
none on /proc type proc (rw)
none on /sys type sysfs (rw)
none on /dev/pts type devpts (rw,gid=5,mode=620)
usbfs on /proc/bus/usb type usbfs (rw)
/dev/hda1 on /boot type ext3 (rw)
none on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
configfs on /config type configfs (rw)
ocfs2_dlmfs on /dlm type ocfs2_dlmfs (rw)
cartman:SHARE2 on /cartman type nfs (rw,addr=192.168.1.120)
/dev/sdd1 on /u02/oradata/orcl type ocfs2
(rw,_netdev,datavolume,nointr,heartbeat=local)
If you modified the O2CB heartbeat threshold, you should verify that it is set correctly::
61
How to Determine OCFS2 Version

To determine which version of OCFS2 is running, use:
# cat /proc/fs/ocfs2/version
OCFS2 1.2.3 Thu Aug 10 18:16:03 PDT 2006 (build
6b798aaadf626d3b137c3952809b2f38)
17. Install & Configure Automatic Storage Management (ASMLib 2.0)
Most of the installation and configuration procedures should be performed on both of the
Oracle RAC nodes in the cluster! Creating the ASM disks, however, will only need to be
performed on a single node within the cluster.
In this section, we will configure Automatic Storage Management (ASM) to be used as

the file system / volume manager for all Oracle physical database files (data, online redo
logs, control files, archived redo logs) and a Flash Recovery Area.
ASM was introduced in Oracle10g Release 1 and is used to alleviate the DBA from
having to manage individual files and drives. ASM is built into the Oracle kernel and
provides the DBA with a way to manage thousands of disk drives 24x7 for both single
and clustered instances of Oracle. All of the files and directories to be used for Oracle
will be contained in a disk group. ASM automatically performs load balancing in parallel
across all available disk drives to prevent hot spots and maximize performance, even with
rapidly changing data usage patterns.
There are two different methods to configure ASM on Linux:
ASM with ASMLib I/O: This method creates all Oracle database files on raw
block devices managed by ASM using ASMLib calls. RAW devices are not
required with this method as ASMLib works with block devices.
ASM with Standard Linux I/O: This method creates all Oracle database files on
raw character devices managed by ASM using standard Linux I/O system calls.
You will be required to create RAW devices for all disk partitions used by ASM.
In this article, I will be using the "ASM with ASMLib I/O" method. Oracle states (in
Metalink Note 275315.1) that "ASMLib was provided to enable ASM I/O to Linux disks
without the limitations of the standard UNIX I/O API". I plan on performing several tests
in the future to identify the performance gains in using ASMLib. Those performance
metrics and testing details are out of scope of this article and therefore will not be
discussed.
We start this section by first downloading the ASMLib drivers (ASMLib Release 2.0)
specific to our Linux kernel. We will then install and configure the ASMLib 2.0 drivers
while finishing off the section with a demonstration of how to create the ASM disks.
If you would like to learn more about Oracle ASMLib 2.0, visit
http://www.oracle.com/technology/tech/linux/asmlib/
Download the ASMLib 2.0 Packages
We start this section by downloading the latest ASMLib 2.0 libraries and the driver from
OTN. At the time of this writing, the latest release of the ASMLib driver was 2.0.3-1.
Like the Oracle Cluster File System, we need to download the version for the Linux
kernel and number of processors on the machine. We are using kernel 2.6.9-42.EL #1
while the machines I am using are both single processor machines:
# uname -a
Linux linux1 2.6.9-42.EL #1 Sat Aug 12 09:17:58 CDT 2006 i686 i686 i386
GNU/Linux
Oracle ASMLib Downloads for Red Hat Enterprise Linux 4 AS
oracleasm-2.6.9-42.EL-2.0.3-1.i686.rpm - (for single processor)
-OR-
oracleasm-2.6.9-42.ELsmp-2.0.3-1.i686.rpm - (for multiple processors)
-OR-
oracleasm-2.6.9-42.ELhugemem-2.0.3-1.i686.rpm - (for hugemem)
You will also need to download the following ASMLib tools:
oracleasmlib-2.0.2-1.i386.rpm - (Userspace library)

oracleasm-support-2.0.3-1.i386.rpm - (Driver support files)
Install ASMLib 2.0 Packages
This installation needs to be performed on on both nodes in the RAC cluster as the root
user account:
$ su -
# rpm -Uvh oracleasm-2.6.9-42.EL-2.0.3-1.i686.rpm \
oracleasmlib-2.0.2-1.i386.rpm \
oracleasm-support-2.0.3-1.i386.rpm
Preparing... ###########################################
[100%]
1:oracleasm-support ###########################################
[ 33%]
2:oracleasm-2.6.9-42.EL ###########################################
[ 67%]
3:oracleasmlib ###########################################
[100%]
Configure and Loading the ASMLib 2.0 Packages
Now that you downloaded and installed the ASMLib Packages for Linux, you need to
configure and load the ASM kernel module. This task needs to be run on both Oracle
RAC nodes as root:
$ su -
# /etc/init.d/oracleasm configure
Configuring the Oracle ASM library driver.
This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting <ENTER> without typing an
answer will keep that current value. Ctrl-C will abort.
Default user to own the driver interface []: oracle

Default group to own the driver interface []: dba
Start Oracle ASM library driver on boot (y/n) [n]: y
Fix permissions of Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: [ OK ]
Creating /dev/oracleasm mount point: [ OK ]
Loading module "oracleasm": [ OK ]
Mounting ASMlib driver filesystem: [ OK ]
Scanning system for ASM disks: [ OK ]
Create ASM Disks for Oracle
Creating the ASM disks only needs to be done on one node in the RAC cluster as the
root user account. I will be running these commands on linux1. On the other Oracle
RAC node, you will need to perform a scandisk to recognize the new volumes. When that
is complete, you should then run the oracleasm listdisks command on both Oracle
RAC nodes to verify that all ASM disks were created and available.
In the section "Create Partitions on iSCSI Volumes", we configured (partitioned) four

iSCSI volumes to be used by ASM. ASM will be used for storing Oracle database files
like online redo logs, database files, control files, archived redo log files, and the flash
recovery area.
It is extremely important to note that at this point in the article you may have rebooted
linux1 several times after partitioning the iSCSI volumes to be used for ASM (e.g.
/dev/sda, /dev/sdb, /dev/sdc, and /dev/sde). This means that the mapping of iSCSI
target names discovered from Openfiler to the local SCSI device name on linux1 may be
different. Please repeat the procedures documented in the section "Discovering iSCSI
Targets" to determine if the iSCSI target names for all four ASM volumes have been
discovered as a different local SCSI device name on linux1 if you have rebooted.
For example, I rebooted both Oracle RAC nodes after configuring OCFS2 (previous
section). My ASM iSCSI target name mappings for linux1 did change and are now:
iSCSI Target Name to local SCSI Device Name - (ASM)

iSCSI Target Name Host / SCSI ID SCSI Device Name
iqn.2006-01.com.openfiler:rac1.asm4 0 /dev/sda
iqn.2006-01.com.openfiler:rac1.asm3 1 /dev/sdb
iqn.2006-01.com.openfiler:rac1.asm2 2 /dev/sde
iqn.2006-01.com.openfiler:rac1.asm1 3 /dev/sdc
Use the above iSCSI target names to map which local SCSI device name to use when
creating the ASM disks.
Note: If you are repeating this article using the same hardware (actually, the same shared
logical drives), you may get a failure when attempting to create the ASM disks. If you do
receive a failure, try listing all ASM disks using:
# /etc/init.d/oracleasm listdisks
VOL1
VOL2
VOL3
VOL4
As you can see, the results show that I have four ASM volumes already defined. If you
have the four volumes already defined from a previous run, go ahead and remove them
using the following commands. After removing the previously created volumes, use the
"oracleasm createdisk" commands (below) to create the new volumes.
# /etc/init.d/oracleasm deletedisk VOL1
Removing ASM disk "VOL1" [ OK ]
To create the ASM disks using the iSCSI target names to local SCSI device name
mappings (above), type the following:
$ su -
# /etc/init.d/oracleasm createdisk VOL1 /dev/sdc1
Marking disk "/dev/sdc1" as an ASM disk [ OK ]
# /etc/init.d/oracleasm createdisk VOL2 /dev/sde1

Marking disk "/dev/sde1" as an ASM disk [ OK ]
# /etc/init.d/oracleasm createdisk VOL3 /dev/sdb1

Marking disk "/dev/sdb1" as an ASM disk [ OK ]
# /etc/init.d/oracleasm createdisk VOL4 /dev/sda1

Marking disk "/dev/sda1" as an ASM disk [ OK ]
On all other nodes in the RAC cluster, you must perform a scandisk to recognize the new
volumes:
# /etc/init.d/oracleasm scandisks
Scanning system for ASM disks [ OK ]
We can now test that the ASM disks were successfully created by using the following
command on both nodes in the RAC cluster as the root user account:
# /etc/init.d/oracleasm listdisks
VOL1
VOL2
VOL3
VOL4
18. Download Oracle RAC 10g Software
The following download procedures only need to be performed on one node in the
cluster!
The next logical step is to install Oracle Clusterware Release 2 (10.2.0.1.0), Oracle
Database 10g Release 2 (10.2.0.1.0), and finally the Oracle Database 10g Companion CD
Release 2 (10.2.0.1.0) for Linux x86 software. However, you must first download and
extract the required Oracle software packages from OTN.
You will be downloading and extracting the required software from Oracle to only one of
the Linux nodes in the clusternamely, linux1. You will perform all installs from this
machine. The Oracle installer will copy the required software packages to all other nodes
in the RAC configuration we set up in Section 14 (Configure RAC Nodes for Remote
Access).
Login to the node that you will be performing all of the Oracle installations from
(linux1) as the "oracle" user account. In this example, you will be downloading the
required Oracle software to linux1 and saving them to /u01/app/oracle/orainstall.
Downloading and Extracting the Software
First, download the Oracle Clusterware Release 2 (10.2.0.1.0), Oracle Database 10g
Release 2 (10.2.0.1.0), and Oracle Database 10g Companion CD Release 2 (10.2.0.1.0)
software for Linux x86. All downloads are available from the same page.
As the oracle user account, extract the three packages you downloaded to a temporary
directory. In this example, we will use /u01/app/oracle/orainstall.
Extract the Oracle Clusterware package as follows:
# su - oracle
$ cd ~oracle/orainstall
$ unzip 10201_clusterware_linux32.zip
Then extract the Oracle Database Software:

$ unzip 10201_database_linux32.zip
Finally, extract the Oracle Companion CD Software:

$ unzip 10201_companion_linux32.zip
19. Pre-Installation Tasks for Oracle10g Release 2
Perform the following checks on both Oracle RAC nodes in the cluster!
When installing the Linux O/S (CentOS Enterprise Linux or Red Hat Enterprise Linux 4),
you should verify that all required RPMs for Oracle are installed. If you followed the
instructions I used for installing Linux, you would have installed Everything, in which
case you will have all of the required RPM packages. However, if you performed another
installation type (i.e. "Advanced Server), you may have some packages missing and will
need to install them. All of the required RPMs are on the Linux CDs/ISOs.
The next pre-installation step is to run the Cluster Verification Utility (CVU). CVU is a
command-line utility provided on the Oracle Clusterware installation media. It is
responsible for performing various system checks to assist you with confirming the
Oracle RAC nodes are properly configured for Oracle Clusterware and Oracle Real
Application Clusters installation. The CVU only needs to be run from the node you will
be performing the Oracle installations from (linux1 in this article).
Check Required RPMs
The following packages (keeping in mind that the version number for your Linux
distribution may vary slightly) must be installed:
binutils-2.15.92.0.2-21
compat-db-4.1.25-9
compat-gcc-32-3.2.3-47.3
compat-gcc-32-c++-3.2.3-47.3
compat-libstdc++-33-3.2.3-47.3
compat-libgcc-296-2.96-132.7.2
control-center-2.8.0-12.rhel4.5
cpp-3.4.6-3
gcc-3.4.6-3
gcc-c++-3.4.6-3
glibc-2.3.4-2.25
glibc-common-2.3.4-2.25
glibc-devel-2.3.4-2.25
glibc-headers-2.3.4-2.25
glibc-kernheaders-2.4-9.1.98.EL
gnome-libs-1.4.1.2.90-44.1
libaio-0.3.105-2
libstdc++-3.4.6-3
libstdc++-devel-3.4.6-3
make-3.80-6.EL4
openmotif-2.2.3-10.RHEL4.5
openmotif21-2.1.30-11.RHEL4.6
pdksh-5.2.14-30.3
setarch-1.6-1
sysstat-5.0.5-11.rhel4
xscreensaver-4.18-5.rhel4.11
Note that the openmotif RPM packages are only required to install Oracle demos. This
article does not cover the installation of Oracle demos.
To query package information (gcc and glibc-devel for example), use the "rpm -q
<PackageName> [,<PackageName>]" command as follows:
# rpm -q gcc glibc-devel

gcc-3.4.6-3
glibc-devel-2.3.4-2.25
If you need to install any of the above packages, use "rpm -Uvh <PackageName.rpm>".
For example, to install the GCC gcc-3.4.6-3 package, use:
# rpm -Uvh gcc-3.4.6-3.i386.rpm
Prerequisites for Using Cluster Verification Utility
JDK 1.4.2
You must have JDK 1.4.2 installed on your system before you can run CVU. If you do
not have JDK 1.4.2 installed on your system, and you attempt to run CVU, you will
receive an error message similar to the following:
ERROR. Either CV_JDKHOME environment variable should be set

or /stagepath/cluvfy/jrepack.zip should exist.
If you do not have JDK 1.4.2 installed, then download it from the Sun Web site, and use
the Sun instructions to install it. JDK 1.4.2 is available as a download from the following
Web site: http://www.sun.com/java.
If you do have JDK 1.4.2 installed, then you must define the user environment variable
CV_JDKHOME for the path to the JDK. For example, if JDK 1.4.2 is installed in
/usr/local/j2re1.4.2_08, then log in as the user that you plan to use to run CVU, and
enter the following commands:
CV_JDKHOME=/usr/local/j2re1.4.2_08
export CV_JDKHOME
Install cvuqdisk RPM (RHEL Users Only)
The second pre-requisite for running the CVU is for Red Hat Linux users. If you are
using Red Hat Linux, then you must download and install the Red Hat operating system
package cvuqdisk to both of the Oracle RAC nodes in the cluster. This means you will
need to install the cvuqdisk RPM to both linux1 and linux2. Without cvuqdisk, CVU
will be unable to discover shared disks, and you will receive the error message "Package
cvuqdisk not installed" when you run CVU.
The cvuqdisk RPM can be found on the Oracle Clusterware installation media in the rpm
directory. For the purpose of this article, the Oracle Clusterware media was extracted to
the /u01/app/oracle/orainstall/clusterware directory on linux1. Note that before
installing the cvuqdisk RPM, we need to set an environment variable named
CVUQDISK_GRP to point to the group that will own the cvuqdisk utility. The default group
is oinstall which is not the group we are using for the oracle UNIX user account in
this article. Since we are using the dba group, we will need to set CVUQDISK_GRP=dba
before attempting to install the cvuqdisk RPM.
Locate and copy the cvuqdisk RPM from linux1 to linux2 then perform the following
steps on both Oracle RAC nodes to install:
$ su -
# cd /u01/app/oracle/orainstall/clusterware/rpm
# CVUQDISK_GRP=dba; export CVUQDISK_GRP
# rpm -iv cvuqdisk-1.0.1-1.rpm

Preparing packages for installation...
cvuqdisk-1.0.1-1
# ls -l /usr/sbin/cvuqdisk
-rwsr-x--- 1 root dba 4168 Jun 2 2005 /usr/sbin/cvuqdisk
Verify Remote Access / User Equivalence
The CVU should be run from linux1 the node we will be performing all of the Oracle
installations from. Before running CVU, login as the oracle user account and verify
remote access / user equivalence is configured to all nodes in the cluster. When using the
secure shell method, user equivalence will need to be enabled for the terminal shell
session before attempting to run the CVU. To enable user equivalence for the current
terminal shell session, perform the following steps remembering to enter the pass phrase
for each key that you generated when prompted:
# su - oracle
$ /usr/bin/ssh-add
When using the remote shell method, user equivalence is generally defined in the
/etc/hosts.equiv file for the oracle user account and is enabled on all new terminal
shell sessions.
Checking Pre-Installation Tasks for CRS with CVU
Once all prerequisites for using CVU have been met, we can start by checking that all
pre-installation tasks for Oracle Clusterware (CRS) are completed by executing the
following command as the "oracle" UNIX user account (with user equivalence enabled)
from linux1:
$ cd /u01/app/oracle/orainstall/clusterware/cluvfy
$ ./runcluvfy.sh stage -pre crsinst -n linux1,linux2 -verbose
Review the CVU report. Note that there are several errors you may ignore in this report.
The first error is with regards to membership of the user "oracle" in group "oinstall" [as
Primary]. For the purpose of this article, the "oracle" user account will only be assigned
to the "dba" group so this error can be safely ignored.
The second error is with regards to finding a suitable set of interfaces for VIPs. This is a
bug documented in Metalink Note 338924.1:
Suitable interfaces for the private interconnect on subnet

"192.168.2.0":
linux2 eth1:192.168.2.101
linux1 eth1:192.168.2.100
ERROR:
Could not find a suitable set of interfaces for VIPs.
Result: Node connectivity check failed.

As documented in the note, this error can be safely ignored.
The last set of errors that can be ignored deal with specific RPM package versions that do
not exist in RHEL4 Update 4. For example:
compat-gcc-7.3-2.96.128
compat-gcc-c++-7.3-2.96.128
compat-libstdc++-7.3-2.96.128
compat-libstdc++-devel-7.3-2.96.128
While these specific packages are listed as missing in the CVU report, please ensure that
the correct versions of the compat-* packages are installed on both of the Oracle RAC
nodes in the cluster. For example, in RHEL4 Update 4, these would be:
compat-gcc-32-3.2.3-47.3
compat-gcc-32-c++-3.2.3-47.3
compat-libstdc++-33-3.2.3-47.3
Checking the Hardware and Operating System Setup with CVU
The next CVU check to run will verify the hardware and operating system setup. Again,
run the following as the "oracle" UNIX user account from linux1:
$ ./runcluvfy.sh stage -post hwos -n linux1,linux2 -verbose
Review the CVU report. As with the previous check (pre-installation tasks for CRS), the
check for finding a suitable set of interfaces for VIPs will fail and can be safely ignored.
Also note that the check for shared storage accessibility will fail.
Checking shared storage accessibility...
WARNING:
Unable to determine the sharedness of /dev/sde on nodes:
linux2,linux2,linux2,linux2,linux2,linux1,linux1,linux1,linux1,linux1
Shared storage check failed on nodes "linux2,linux1".
This too can be safely ignored. While we know the disks are visible and shared from both
of our Oracle RAC nodes in the cluster, the check itself fails. Several reasons for this
have been documented. The first came from Metalink indicating that cluvfy currently
does not work with devices other than SCSI devices. This would include devices like
EMC PowerPath and volume groups like those from Openfiler. At the time of this
writing, no workaround exists other than to use manual methods for detecting shared
devices. Another reason for this error was documented by Bane Radulovic at Oracle
Corporation. His research shows that CVU calls smartclt on Linux, and the problem is
that smartclt does not return the serial number from our iSCSI devices. For example, a
check against /dev/sde shows:
# /usr/sbin/smartctl -i /dev/sde
smartctl version 5.33 [i686-redhat-linux-gnu] Copyright (C) 2002-4 Bruce
Allen
Home page is http://smartmontools.sourceforge.net/
Device: Openfile Virtual disk Version: 0

Serial number:
Device type: disk
Local Time is: Wed Oct 25 23:41:20 2006 EDT
Device supports SMART and is Disabled
Temperature Warning Disabled or Not Supported
At the time of this writing, it is unknown if the Openfiler developers have plans to fix
this.
20. Install Oracle 10g Clusterware Software
Perform the following installation procedures from only one of the Oracle RAC nodes in
the cluster (linux1)! The Oracle Clusterware software will be installed to both of Oracle
RAC nodes in the cluster by the Oracle Universal Installer.
You are now ready to install the "cluster" part of the environment - the Oracle
Clusterware. In a previous section, you downloaded and extracted the install files for
Oracle Clusterware to linux1 in the directory
/u01/app/oracle/orainstall/clusterware. This is the only node from which you
need to perform the install.
During the installation of Oracle Clusterware, you will be asked for the nodes involved
and to configure in the RAC cluster. Once the actual installation starts, it will copy the
required software to all nodes using the remote access we configured in the section
Section 14 ("Configure RAC Nodes for Remote Access").
So, what exactly is the Oracle Clusterware responsible for? It contains all of the cluster
and database configuration metadata along with several system management features for
RAC. It allows the DBA to register and invite an Oracle instance (or instances) to the
cluster. During normal operation, Oracle Clusterware will send messages (via a special
ping operation) to all nodes configured in the cluster, often called the "heartbeat." If the
heartbeat fails for any of the nodes, it checks with the Oracle Clusterware configuration
files (on the shared disk) to distinguish between a real node failure and a network failure.
After installing Oracle Clusterware, the Oracle Universal Installer (OUI) used to install
the Oracle 10g database software (next section) will automatically recognize these nodes.
Like the Oracle Clusterware install you will be performing in this section, the Oracle
Database 10g software only needs to be run from one node. The OUI will copy the
software packages to all nodes configured in the RAC cluster.
Oracle Clusterware Shared Files
The two shared files (actually file groups) used by Oracle Clusterware will be stored on
the Oracle Cluster File System, Release 2 (OFCS2) we created earlier. The two shared
Oracle Clusterware file groups are:
Oracle Cluster Registry (OCR)

o File 1 : /u02/oradata/orcl/OCRFile
o File 2 : /u02/oradata/orcl/OCRFile_mirror
o Size : (2 * 100MB) = 200M
CRS Voting Disk
o File 1 : /u02/oradata/orcl/CSSFile
o File 2 : /u02/oradata/orcl/CSSFile_mirror1
o File 3 : /u02/oradata/orcl/CSSFile_mirror2
o Size : (3 * 20MB) = 60MB
Note: It is not possible to use Automatic Storage Management (ASM) for the two shared
Oracle Clusterware files: Oracle Cluster Registry (OCR) or the CRS Voting Disk files.
The problem is that these files need to be in place and accessible BEFORE any Oracle
instances can be started. For ASM to be available, the ASM instance would need to be
run first.
Also note that the two shared files could be stored on the OCFS2, shared RAW devices,
or another vendor's clustered file system.
Verifying Terminal Shell Environment
Before starting the Oracle Universal Installer, you should first verify you are logged onto
the server you will be running the installer from (i.e. linux1) then run the xhost
command as root from the console to allow X Server connections. Next, login as the
oracle user account. If you are using a remote client to connect to the node performing
the installation (SSH / Telnet to linux1 from a workstation configured with an X Server),
you will need to set the DISPLAY variable to point to your local workstation. Finally,
verify remote access / user equivalence to all nodes in the cluster:
Verify Server and Enable X Server Access
# hostname
linux1
# xhost +
access control disabled, clients can connect from any host
Login as the oracle User Account and Set DISPLAY (if necessary)
# su - oracle
$ # IF YOU ARE USING A REMOTE CLIENT TO CONNECT TO THE
$ # NODE PERFORMING THE INSTALL
$ DISPLAY=<your local workstation>:0.0
$ export DISPLAY
Verify you are able to run the Secure Shell commands (ssh or scp) or the Remote Shell
commands (rsh and rcp) on the Linux server you will be running the Oracle Universal
Installer from against all other Linux servers in the cluster without being prompted for a
password.
When using the secure shell method, user equivalence will need to be enabled on any
new terminal shell session before attempting to run the OUI. To enable user equivalence
for the current terminal shell session, perform the following steps remembering to enter
the pass phrase for each key that you generated when prompted:
$ /usr/bin/ssh-add
Wed Sep 13 18:27:17 EDT 2006
linux1
Wed Sep 13 18:28:00 EDT 2006
linux2
shell sessions:
$ rsh linux1 "date;hostname"

Tue Sep 12 12:37:17 EDT 2006
linux1
Tue Sep 12 12:38:00 EDT 2006
linux2
Installing Cluster Ready Services
Perform the following tasks to install the Oracle Clusterware:
$ cd ~oracle
$ /u01/app/oracle/orainstall/clusterware/runInstaller -ignoreSysPrereqs
Screen Name Response

Welcome Screen Click Next
Specify
Accept the default values:
Inventory
Inventory directory: /u01/app/oracle/oraInventory
directory and
Operating System group name: dba
credentials
Set the Name and Path for the ORACLE_HOME (actually the $ORA_CRS_HOME that
Specify Home I will be using in this article) as follows:
Details Name: OraCrs10g_home
Path: /u01/app/oracle/product/crs
The installer will run through a series of checks to determine if the node meets the
Product-Specific minimum requirements for installing and configuring the Oracle Clusterware software. If
Prerequisite any of the checks fail, you will need to manually verify the check that failed by clicking
Checks on the checkbox. For my installation, all checks passed with no problems.
Click Next to continue.
Cluster Name: crs
Public Node Name Private Node Name Virtual Node Name
Specify Cluster
Configuration linux1 linux1-priv linux1-vip
linux2 linux2-priv linux2-vip
Interface Name Subnet Interface Type

Specify Network
eth0 192.168.1.0 Public
Interface Usage
eth1 192.168.2.0 Private
Starting with Oracle Database 10g Release 2 (10.2) with RAC, Oracle Clusterware
provides for the creation of a mirrored OCR file, enhancing cluster reliability. For the
Specify OCR purpose of this example, I did choose to mirror the OCR file by keeping the default
Location option of "Normal Redundancy":
Specify OCR Location: /u02/oradata/orcl/OCRFile
Specify OCR Mirror Location: /u02/oradata/orcl/OCRFile_mirror
Starting with Oracle Database 10g Release 2 (10.2) with RAC, CSS has been modified
to allow you to configure CSS with multiple voting disks. In Release 1 (10.1), you could
configure only one voting disk. By enabling multiple voting disk configuration, the
redundant voting disks allow you to configure a RAC database with multiple voting
disks on independent shared physical disks. This option facilitates the use of the iSCSI
Specify Voting network protocol, and other Network Attached Storage (NAS) storage solutions. Note
Disk Location that to take advantage of the benefits of multiple voting disks, you must configure at
least three voting disks. For the purpose of this example, I did choose to mirror the
voting disk by keeping the default option of "Normal Redundancy":
Voting Disk Location: /u02/oradata/orcl/CSSFile
Additional Voting Disk 1 Location: /u02/oradata/orcl/CSSFile_mirror1
Additional Voting Disk 2 Location: /u02/oradata/orcl/CSSFile_mirror2
Summary Click Install to start the installation!
Execute After the installation has completed, you will be prompted to run the orainstRoot.sh
Configuration and root.sh script. Open a new console window on both Oracle RAC nodes in the
Scripts cluster, (starting with the node you are performing the install from), as the "root" user
account.
Navigate to the /u01/app/oracle/oraInventory directory and run orainstRoot.sh

ON ALL NODES in the RAC cluster.
Within the same new console window on both Oracle RAC nodes in the cluster, (starting
with the node you are performing the install from), stay logged in as the "root" user
account.
Navigate to the /u01/app/oracle/product/crs directory and locate the root.sh file
for each node in the cluster - (starting with the node you are performing the install from).
Run the root.sh file ON ALL NODES in the RAC cluster ONE AT A TIME.
You will receive several warnings while running the root.sh script on all nodes. These
warnings can be safely ignored.
The root.sh may take awhile to run. When running the root.sh on the last node, you
will receive a critical error and the output should look like:
...
Expecting the CRS daemons to be up within 600 seconds.
CSS is active on these nodes.
linux1
linux2
CSS is active on all nodes.
Waiting for the Oracle CRSD and EVMD to start
Oracle CRS stack installed and running under init(1M)
Running vipca(silent) for configuring nodeapps
The given interface(s), "eth0" is not public. Public interfaces should
be used to configure virtual IPs.
This issue is specific to Oracle 10.2.0.1 (noted in Metalink article 338924.1) and needs
to be resolved before continuing. The easiest workaround is to re-run vipca (GUI)
manually as root from the last node in which the error occurred. Please keep in mind that
vipca is a GUI and will need to set your DISPLAY variable accordingly to your X server:
# $ORA_CRS_HOME/bin/vipca
When the "VIP Configuration Assistant" appears, this is how I answered the screen
prompts:
Welcome: Click Next

Network interfaces: Select both interfaces - eth0 and eth1
Virtual IPs for cluster notes:
Node Name: linux1
IP Alias Name: linux1-vip
IP Address: 192.168.1.200
Subnet Mask: 255.255.255.0
Node Name: linux2

IP Alias Name: linux2-vip
IP Address: 192.168.1.201
Subnet Mask: 255.255.255.0
Summary: Click Finish

Configuration Assistant Progress Dialog: Click OK after configuration is complete.
Configuration Results: Click Exit
Go back to the OUI and acknowledge the "Execute Configuration scripts" dialog
window.
End of
At the end of the installation, exit from the OUI.
installation
Verify Oracle Clusterware Installation
After the installation of Oracle Clusterware, we can run through several tests to verify the
install was successful. Run the following commands on both nodes in the RAC cluster.
Check cluster nodes
$ /u01/app/oracle/product/crs/bin/olsnodes -n
linux1 1
linux2 2
Check Oracle Clusterware Auto-Start Scripts
$ ls -l /etc/init.d/init.*
-r-xr-xr-x 1 root root 1951 Oct 4 14:21 /etc/init.d/init.crs*
-r-xr-xr-x 1 root root 4714 Oct 4 14:21 /etc/init.d/init.crsd*
-r-xr-xr-x 1 root root 35394 Oct 4 14:21 /etc/init.d/init.cssd*
-r-xr-xr-x 1 root root 3190 Oct 4 14:21 /etc/init.d/init.evmd*
21. Install Oracle Database 10g Software
the cluster (linux1)! The Oracle Database software will be installed to both of Oracle
RAC nodes in the cluster by the Oracle Universal Installer.
After successfully installing the Oracle Clusterware software, the next step is to install
the Oracle Database 10g Release 2 (10.2.0.1.0) with RAC.
For the purpose of this example, you will forgo the "Create Database" option when
installing the software. You will, instead, create the database using the Database Creation
Assistant (DBCA) after the install.
Like the Oracle Clusterware install (previous section), the Oracle10g database software
only needs to be run from one node. The OUI will copy the software packages to all
nodes configured in the RAC cluster.

As discussed in the previous section, (Install Oracle10g Clusterware Software), the
terminal shell environment needs to be configured for remote access and user equivalence
to all nodes in the cluster before running the Oracle Universal Installer. Note that you can
utilize the same terminal shell session used in the previous section which in this case, you
do not have to take any of the actions described below with regards to setting up remote
access and the DISPLAY variable:
# su - oracle
$ export DISPLAY
password.

$ /usr/bin/ssh-add
Wed Sep 13 18:27:17 EDT 2006
linux1
Wed Sep 13 18:28:00 EDT 2006
linux2
shell sessions:

Tue Sep 12 12:37:17 EDT 2006
linux1
Tue Sep 12 12:38:00 EDT 2006
linux2
Run the Oracle Cluster Verification Utility
Before installing the Oracle Database Software, we should run the following database
pre-installation check using the Cluster Verification Utility (CVU).
Note: Instructions for configuring CVU can be found in the section "Prerequisites for
Using Cluster Verification Utility discussed earlier in this article.
$ ./runcluvfy.sh stage -pre dbinst -n linux1,linux2 -r 10gR2 -verbose
Review the CVU report. Note that this report will contain the same errors we received
when checking pre-installation tasks for CRS failure to find a suitable set of interfaces
for VIPs and the failure to find specific RPM packages that do not exist in RHEL4
Update. These two errors can be safely ignored.
Install Oracle Database 10g Release 2 Software
Install the Oracle Database 10g Release 2 software with the following:
$ cd ~oracle
$ /u01/app/oracle/orainstall/database/runInstaller -ignoreSysPrereqs

Select Installation Type I selected the Enterprise Edition option.
Set the Name and Path for the ORACLE_HOME as
follows:
Specify Home Details
Name: OraDb10g_home1
Path: /u01/app/oracle/product/10.2.0/db_1
Specify Hardware Cluster Select the Cluster Installation option then select all nodes
Installation Mode available. Click Select All to select all servers: linux1
and linux2.
If the installation stops here and the status of any of the

RAC nodes is "Node not reachable", perform the
following checks:
Ensure Oracle Clusterware is running on the node

in question.
Ensure you are table to reach the node in question
from the node you are performing the installation
from.
The installer will run through a series of checks to
determine if the node meets the minimum requirements
for installing and configuring the Oracle database
Product-Specific Prerequisite Checks software. If any of the checks fail, you will need to
manually verify the check that failed by clicking on the
checkbox.
Click Next to continue.
Select the option to "Install database software only."
Select Database Configuration Remember that we will create the clustered database as a
separate step using DBCA.
Summary Click on Install to start the installation!
After the installation has completed, you will be
prompted to run the root.sh script. It is important to
keep in mind that the root.sh script will need to be run on
all nodes in the RAC cluster one at a time starting with
the node you are running the database installation from.
First, open a new console window on the node you are

installing the Oracle 10g database software from as the
Root Script Window - Run root.sh root user account. For me, this was "linux1".
Navigate to the
/u01/app/oracle/product/10.2.0/db_1 directory and
run root.sh.
After running the root.sh script on all nodes in the

cluster, go back to the OUI and acknowledge the
"Execute Configuration scripts" dialog window.
End of installation At the end of the installation, exit from the OUI.
22. Install Oracle 10g Companion CD Software
the cluster (linux1)! The Oracle10g Companion CD software will be installed to both of
Oracle RAC nodes in the cluster by the Oracle Universal Installer.
After successfully installing the Oracle Database software, the next step is to install the
Oracle 10g Release 2 Companion CD software (10.2.0.1.0).
Please keep in mind that this is an optional step. For the purpose of this guide, my testing
database will often make use of the Java Virtual Machine (Java VM) and Oracle
interMedia and therefore will require the installation of the Oracle Database 10g
Companion CD. The type of installation to perform will be the Oracle Database 10g
Products installation type.
This installation type includes the Natively Compiled Java Libraries (NCOMP) files to
improve Java performance. If you do not install the NCOMP files, the ORA-
29558:JAccelerator (NCOMP) not installed error occurs when a database that uses
Java VM is upgraded to the patch release.
As discussed in the previous section, (Install Oracle10g Database Software), the terminal
shell environment needs to be configured for remote access and user equivalence to all
nodes in the cluster before running the Oracle Universal Installer. Note that you can
utilize the same terminal shell session used in the previous section which in this case, you
do not have to take any of the actions described below with regards to setting up remote
access and the DISPLAY variable:
# su - oracle
$ export DISPLAY
password.

$ /usr/bin/ssh-add
Wed Sep 13 18:27:17 EDT 2006
linux1
Wed Sep 13 18:28:00 EDT 2006
linux2
shell sessions:

Tue Sep 12 12:37:17 EDT 2006
linux1
Tue Sep 12 12:38:00 EDT 2006
linux2
Install Oracle10g Companion CD Software
Install the Oracle10g Companion CD software with the following:
$ cd ~oracle
$ /u01/app/oracle/orainstall/companion/runInstaller -ignoreSysPrereqs

Select the "Oracle Database 10g Products 10.2.0.1.0"
Select a Product to Install
option.
Set the destination for the ORACLE_HOME Name and
Path to that of the previous Oracle10g Database software
Specify Home Details install as follows:
Name: OraDb10g_home1
Path: /u01/app/oracle/product/10.2.0/db_1
Specify Hardware Cluster The Cluster Installation option will be selected along
Installation Mode with all of the available nodes in the cluster by default.
Stay with these default options and click Next to
continue.
If the installation stops here and the status of any of the

RAC nodes is "Node not reachable", perform the
following checks:
Ensure Oracle Clusterware is running on the node

in question.
Ensure you are table to reach the node in question

from the node you are performing the installation
from.
The installer will run through a series of checks to
determine if the node meets the minimum requirements
for installing and configuring the Companion CD
Software. If any of the checks fail, you will need to
Product-Specific Prerequisite Checks
manually verify the check that failed by clicking on the
checkbox. For my installation, all checks passed with no
problems.
Click on Next to continue.
On the Summary screen, click Install to start the
Summary
installation!
End of installation At the end of the installation, exit from the OUI.
23. Create TNS Listener Process
Perform the following configuration procedures from only one of the Oracle RAC nodes
in the cluster (linux1)! The Network Configuration Assistant (NETCA) will setup the TNS
listener in a clustered configuration on both of Oracle RAC nodes in the cluster..
The DBCA requires the Oracle TNS Listener process to be configured and running on all
nodes in the RAC cluster before it can create the clustered database.
The process of creating the TNS listener only needs to be performed on one node in the
cluster. All changes will be made and replicated to all nodes in the cluster. On one of the
nodes (I will be using linux1) bring up the NETCA and run through the process of
creating a new TNS listener process and also configure the node for local access.
As discussed in the previous section, (Install Oracle10g Companion CD Software), the

terminal shell environment needs to be configured for remote access and user equivalence
to all nodes in the cluster before running the Network Configuration Assistant (NETCA).
Note that you can utilize the same terminal shell session used in the previous section
which in this case, you do not have to take any of the actions described below with
regards to setting up remote access and the DISPLAY variable:
# su - oracle
$ export DISPLAY
password.

$ /usr/bin/ssh-add
Wed Sep 13 18:27:17 EDT 2006
linux1
Wed Sep 13 18:28:00 EDT 2006
linux2
shell sessions:

Tue Sep 12 12:37:17 EDT 2006
linux1
Tue Sep 12 12:38:00 EDT 2006
linux2
Run the Network Configuration Assistant
To start the NETCA, run the following:
$ netca &
The following table walks you through the process of creating a new Oracle listener for
our RAC environment.
Select the Type of Oracle
Net Services Select Cluster Configuration
Configuration
Select the nodes to
Select all of the nodes: linux1 and linux2.
configure
Type of Configuration Select Listener configuration.
The following screens are now like any other normal listener
configuration. You can simply accept the default parameters for the
next six screens:
What do you want to do: Add
Listener Configuration - Listener name: LISTENER
Next 6 Screens Selected protocols: TCP
Port number: 1521
Configure another listener: No
Listener configuration complete! [ Next ]
You will be returned to this Welcome (Type of Configuration) Screen.
Type of Configuration Select Naming Methods configuration.
The following screens are:
Naming Methods Selected Naming Methods: Local Naming
Configuration Naming Methods configuration complete! [ Next ]
You will be returned to this Welcome (Type of Configuration) Screen.
Type of Configuration Click Finish to exit the NETCA.
The Oracle TNS listener process should now be running on all nodes in the RAC cluster:
$ hostname
linux1
$ ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print
$9}'
LISTENER_LINUX1
=====================
$ hostname
linux2
$ ps -ef | grep lsnr | grep -v 'grep' | grep -v 'ocfs' | awk '{print
$9}'
LISTENER_LINUX2
24. Create the Oracle Cluster Database

The database creation process should only be performed from one of the Oracle RAC
nodes in the cluster (linux1)!
We will use the DBCA to create the clustered database.
Before executing the DBCA, make sure that $ORACLE_HOME and $PATH are set
appropriately for the $ORACLE_BASE/product/10.2.0/db_1 environment.
You should also verify that all services we have installed up to this point (Oracle TNS
listener, Oracle Clusterware processes, etc.) are running before attempting to start the
clustered database creation process.
As discussed in the previous section, (Create TNS Listener Process), the terminal shell
environment needs to be configured for remote access and user equivalence to all nodes
in the cluster before running the Database Configuration Assistant (DBCA). Note that
you can utilize the same terminal shell session used in the previous section which in this
case, you do not have to take any of the actions described below with regards to setting
up remote access and the DISPLAY variable:
# su - oracle
$ export DISPLAY
password.

$ /usr/bin/ssh-add
Wed Sep 13 18:27:17 EDT 2006
linux1
Wed Sep 13 18:28:00 EDT 2006
linux2
shell sessions:

Tue Sep 12 12:37:17 EDT 2006
linux1
Tue Sep 12 12:38:00 EDT 2006
linux2
Run the Oracle Cluster Verification Utility
Before creating the Oracle clustered database, we should run the following database
configuration check using the Cluster Verification Utility (CVU).
Note: Instructions for configuring CVU can be found in the section "Prerequisites for
Using Cluster Verification Utility discussed earlier in this article.
$ ./runcluvfy.sh stage -pre dbcfg -n linux1,linux2 -d ${ORACLE_HOME}
-verbose
Review the CVU report. Note that this report will contain the same error we received
when checking pre-installation tasks for CRS failure to find a suitable set of interfaces
for VIPs. This error can be safely ignored.
Create the Clustered Database
To start the database creation process, run the following:
$ dbca &

Welcome
Select "Oracle Real Application Clusters database."
Screen
Operations Select Create a Database.
Node Selection Click on the Select All button to select all servers: linux1 and linux2.
Database Select Custom Database.
Templates
Select:
Global Database Name: orcl.idevelopment.info
Database
SID Prefix: orcl
Identification
I used idevelopment.info for the database domain. You may use any domain.
Keep in mind that this domain does not have to be a valid DNS domain.
Management Leave the default options here, which is to "Configure the Database with
Option Enterprise Manager / Use Database Control for Database Management."
Database I selected to Use the Same Password for All Accounts. Enter the password
Credentials (twice) and make sure the password does not start with a digit number.
Storage
For this guide, we will select to use Automatic Storage Management (ASM).
Options
Supply the SYS password to use for the new ASM instance.
Also, starting with Oracle10g Release 2, the ASM instance server parameter file
(SPFILE) needs to be on a shared disk. You will need to modify the default entry
for "Create server parameter file (SPFILE)" to reside on the OCFS2 partition as
follows: /u02/oradata/orcl/dbs/spfile+ASM.ora. All other options can stay
Create ASM
at their defaults.
Instance
You will then be prompted with a dialog box asking if you want to create and
start the ASM instance. Select the OK button to acknowledge this dialog.
The OUI will now create and start the ASM instance on all nodes in the RAC
cluster.
ASM Disk To start, click the Create New button. This will bring up the "Create Disk
Groups Group" window with the four volumes we configured earlier using ASMLib.
If the volumes we created earlier in this article do not show up in the "Select
Member Disks" window: (ORCL:VOL1, ORCL:VOL2, ORCL:VOL3, and
ORCL:VOL4) then click on the "Change Disk Discovery Path" button and input
"ORCL:VOL*".
For the first "Disk Group Name", I used the string "ORCL_DATA1". Select the
first two ASM volumes (ORCL:VOL1 and ORCL:VOL2) in the "Select Member
Disks" window. Keep the "Redundancy" setting to "Normal".
After verifying all values in this window are correct, click the [OK] button. This
will present the "ASM Disk Group Creation" dialog. When the ASM Disk Group
Creation process is finished, you will be returned to the "ASM Disk Groups"
windows.
Click the Create New button again. For the second "Disk Group Name", I used
the string "FLASH_RECOVERY_AREA". Select the last two ASM volumes
(ORCL:VOL3 and ORCL:VOL4) in the "Select Member Disks" window. Keep
the "Redundancy" setting to "Normal".
After verifying all values in this window are correct, click the [OK] button. This
will present the "ASM Disk Group Creation" dialog.
When the ASM Disk Group Creation process is finished, you will be returned to
the "ASM Disk Groups" window with two disk groups created and selected.
Select only one of the disk groups by using the checkbox next to the newly
created Disk Group Name "ORCL_DATA1" (ensure that the disk group for
"FLASH_RECOVERY_AREA" is not selected) and click [Next] to continue.
Database File I selected to use the default, which is to use Oracle Managed Files:
Locations Database Area: +ORCL_DATA1
Check the option for "Specify Flash Recovery Area".
For the Flash Recovery Area, click the [Browse] button and select the disk group
Recovery
name "+FLASH_RECOVERY_AREA".
Configuration
My disk group has a size of about 118GB. I used a Flash Recovery Area Size of
117GB (117760 MB).
I left all of the Database Components (and destination tablespaces) set to their
Database
default value, although it is perfectly OK to select the Sample Schemas. This
Content
option is available since we installed the Oracle Companion CD software.
For this test configuration, click Add, and enter orcltest as the "Service
Database
Name." Leave both instances set to Preferred and for the "TAF Policy" select
Services
"Basic".
Initialization Change any parameters for your environment. I left them all at their default
Parameters settings.
Database Change any parameters for your environment. I left them all at their default
Storage settings.
Keep the default option Create Database selected and click Finish to start the
Creation
database creation process.
Options
Click OK on the "Summary" screen.
At the end of the database creation, exit from the DBCA.
End of When exiting the DBCA, another dialog will come up indicating that it is
Database starting all Oracle instances and HA service "orcltest". This may take several
Creation minutes to complete. When finished, all windows and dialog boxes will
disappear.
When the DBCA has completed, you will have a fully functional Oracle RAC cluster
running!
Create the orcltest Service
During the creation of the Oracle clustered database, you added a service named
orcltest that will be used to connect to the database with TAF enabled. During several
of my installs, the service was added to the tnsnames.ora, but was never updated as a
service for each Oracle instance.
Use the following to verify the orcltest service was successfully added:
SQL> show parameter service
NAME TYPE VALUE

-------------------- ----------- --------------------------------
service_names string orcl.idevelopment.info, orcltest
If the only service defined was for orcl.idevelopment.info, then you will need to
manually add the service to both instances:
SQL> show parameter service
NAME TYPE VALUE

-------------------- ----------- --------------------------
service_names string orcl.idevelopment.info
SQL> alter system set service_names =

2 'orcl.idevelopment.info, orcltest.idevelopment.info' scope=both;
25. Verify TNS Networking Files
Ensure that the TNS networking files are configured on both Oracle RAC nodes in the
cluster!
listener.ora
We already covered how to create a TNS listener configuration file (listener.ora) for a
clustered environment in Section 23. The listener.ora file should be properly
configured and no modifications should be needed.
For clarity, I have included a copy of the listener.ora file from my node linux1 in this
guide's support files. I've also included a copy of my tnsnames.ora file that was
configured by Oracle and can be used for testing the Transparent Application Failover
(TAF). This file should already be configured on both Oracle RAC nodes in the cluster.
You can include any of these entries on other client machines that need access to the
clustered database.
Connecting to Clustered Database From an External Client
This is an optional step, but I like to perform it in order to verify my TNS files are
configured correctly. Use another machine (i.e. a Windows machine connected to the
network) that has Oracle installed (either 9i or 10g) and add the TNS entries (in the
tnsnames.ora) from either of the nodes in the cluster that were created for the clustered
database.
Then try to connect to the clustered database using all available service names defined in
the tnsnames.ora file:
C:\> sqlplus system/manager@orcl2

C:\> sqlplus system/manager@orcl1
C:\> sqlplus system/manager@orcltest
C:\> sqlplus system/manager@orcl
26. Create / Alter Tablespaces
When creating the clustered database, we left all tablespaces set to their default size. If
you are using a large drive for the shared storage, you may want to make a sizable testing
database.
Below are several optional SQL commands for modifying and creating all tablespaces for
the test database. Please keep in mind that the database file names (OMF files) used in
this example may differ from what Oracle creates for your environment. The following
query can be used to determine the file names for your environment:
SQL> select tablespace_name, file_name

2 from dba_data_files
3 union
4 select tablespace_name, file_name
5 from dba_temp_files;
TABLESPACE_NAME FILE_NAME
--------------- --------------------------------------------------
EXAMPLE +ORCL_DATA1/orcl/datafile/example.257.570913311
INDX +ORCL_DATA1/orcl/datafile/indx.270.570920045
SYSAUX +ORCL_DATA1/orcl/datafile/sysaux.260.570913287
SYSTEM +ORCL_DATA1/orcl/datafile/system.262.570913215
TEMP +ORCL_DATA1/orcl/tempfile/temp.258.570913303
UNDOTBS1 +ORCL_DATA1/orcl/datafile/undotbs1.261.570913263
UNDOTBS2 +ORCL_DATA1/orcl/datafile/undotbs2.265.570913331
USERS +ORCL_DATA1/orcl/datafile/users.264.570913355
$ sqlplus "/ as sysdba"
SQL> create user scott identified by tiger default tablespace users;
SQL> grant dba, resource, connect to scott;
SQL> alter database datafile

'+ORCL_DATA1/orcl/datafile/users.264.570913355' resize 1024m;
SQL> alter tablespace users add datafile '+ORCL_DATA1' size 1024m
autoextend off;
SQL> create tablespace indx datafile '+ORCL_DATA1' size 1024m

2 autoextend on next 50m maxsize unlimited
3 extent management local autoallocate
4 segment space management auto;

'+ORCL_DATA1/orcl/datafile/system.262.570913215' resize 800m;

'+ORCL_DATA1/orcl/datafile/sysaux.260.570913287' resize 500m;
SQL> alter tablespace undotbs1 add datafile '+ORCL_DATA1' size 1024m

2 autoextend on next 50m maxsize 2048m;
SQL> alter tablespace undotbs2 add datafile '+ORCL_DATA1' size 1024m

2 autoextend on next 50m maxsize 2048m;
SQL> alter database tempfile

'+ORCL_DATA1/orcl/tempfile/temp.258.570913303' resize 1024m;
Here is a snapshot of the tablespaces I have defined for my test database environment:
Status Tablespace Name TS Type Ext. Mgt. Seg. Mgt.
Tablespace Size Used (in bytes) Pct. Used
--------- --------------- ------------ ---------- ---------
------------------ ------------------ ---------
ONLINE UNDOTBS1 UNDO LOCAL MANUAL
1,283,457,024 85,065,728 7
ONLINE SYSAUX PERMANENT LOCAL AUTO
524,288,000 275,906,560 53
ONLINE USERS PERMANENT LOCAL AUTO
2,147,483,648 131,072 0
ONLINE SYSTEM PERMANENT LOCAL MANUAL
838,860,800 500,301,824 60
ONLINE EXAMPLE PERMANENT LOCAL AUTO
157,286,400 83,820,544 53
ONLINE INDX PERMANENT LOCAL AUTO
1,073,741,824 65,536 0
ONLINE UNDOTBS2 UNDO LOCAL MANUAL
1,283,457,024 3,801,088 0
ONLINE TEMP TEMPORARY LOCAL MANUAL
1,073,741,824 27,262,976 3
------------------ ------------------ ---------

avg
22
sum
8,382,316,544 976,355,328
8 rows selected.
27. Verify the RAC Cluster & Database Configuration
The following RAC verification checks should be performed on both Oracle RAC nodes
in the cluster! For this article, however, I will only be performing checks from linux1.
This section provides several srvctl commands and SQL queries you can use to validate
your Oracle RAC 10g configuration.
There are five node-level tasks defined for SRVCTL:
Adding and deleting node-level applications

Setting and unsetting the environment for node-level applications
Administering node applications
Administering ASM instances
Starting and stopping a group of programs that includes virtual IP addresses,
listeners, Oracle Notification Services, and Oracle Enterprise Manager agents (for
maintenance purposes).
Status of all instances and services
$ srvctl status database -d orcl

Instance orcl1 is running on node linux1
Status of a single instance
$ srvctl status instance -d orcl -i orcl2

Status of a named service globally across the database
$ srvctl status service -d orcl -s orcltest

Service orcltest is running on instance(s) orcl2, orcl1
Status of node applications on a particular node
$ srvctl status nodeapps -n linux1

VIP is running on node: linux1
GSD is running on node: linux1
Listener is running on node: linux1
ONS daemon is running on node: linux1
Status of an ASM instance

$ srvctl status asm -n linux1
ASM instance +ASM1 is running on node linux1.
List all configured databases
$ srvctl config database

orcl
Display configuration for our RAC database
$ srvctl config database -d orcl

linux1 orcl1 /u01/app/oracle/product/10.2.0/db_1
linux2 orcl2 /u01/app/oracle/product/10.2.0/db_1
Display all services for the specified cluster database
$ srvctl config service -d orcl

orcltest PREF: orcl2 orcl1 AVAIL:
Display the configuration for node applications - (VIP, GSD, ONS, Listener)
$ srvctl config nodeapps -n linux1 -a -g -s -l

VIP exists.: /linux1-vip/192.168.1.200/255.255.255.0/eth0:eth1
GSD exists.
ONS daemon exists.
Listener exists.
Display the configuration for the ASM instance(s)
$ srvctl config asm -n linux1

+ASM1 /u01/app/oracle/product/10.2.0/db_1
All running instances in the cluster
SELECT
inst_id
, instance_number inst_no
, instance_name inst_name
, parallel
, status
, database_status db_status
, active_state state
, host_name host
FROM gv$instance
ORDER BY inst_id;
INST_ID INST_NO INST_NAME PAR STATUS DB_STATUS STATE HOST

-------- -------- ---------- --- ------- ------------ --------- -------
1 1 orcl1 YES OPEN ACTIVE NORMAL linux1
2 2 orcl2 YES OPEN ACTIVE NORMAL linux2
All data files which are in the disk group

select name from v$datafile
union
select member from v$logfile
union
select name from v$controlfile
union
select name from v$tempfile;
NAME
-------------------------------------------
+FLASH_RECOVERY_AREA/orcl/controlfile/current.258.570913191
+FLASH_RECOVERY_AREA/orcl/onlinelog/group_1.257.570913201
+ORCL_DATA1/orcl/controlfile/current.259.570913189
+ORCL_DATA1/orcl/datafile/example.257.570913311
+ORCL_DATA1/orcl/datafile/indx.270.570920045
+ORCL_DATA1/orcl/datafile/sysaux.260.570913287
+ORCL_DATA1/orcl/datafile/system.262.570913215
+ORCL_DATA1/orcl/datafile/undotbs1.261.570913263
+ORCL_DATA1/orcl/datafile/users.264.570913355
+ORCL_DATA1/orcl/datafile/users.269.570919829
+ORCL_DATA1/orcl/onlinelog/group_1.256.570913195
+ORCL_DATA1/orcl/tempfile/temp.258.570913303
21 rows selected.
All ASM disk that belong to the 'ORCL_DATA1' disk group
SELECT path
FROM v$asm_disk
WHERE group_number IN (select group_number
from v$asm_diskgroup
where name = 'ORCL_DATA1');
PATH
----------------------------------
ORCL:VOL1
ORCL:VOL2
28. Starting / Stopping the Cluster
At this point, we've installed and configured Oracle RAC 10g entirely and have a fully
functional clustered database.
After all the work done up to this point, you may well ask, "OK, so how do I start and
stop services?" If you have followed the instructions in this guide, all servicesincluding
Oracle Clusterware, all Oracle instances, Enterprise Manager Database Console, and so
onshould start automatically on each reboot of the Linux nodes.
There are times, however, when you might want to shut down a node and manually start
it back up. Or you may find that Enterprise Manager is not running and need to start it.
This section provides the commands (using SRVCTL) responsible for starting and stopping
the cluster environment.
Ensure that you are logged in as the oracle UNIX user. We will run all commands in this
section from linux1:
# su - oracle
$ hostname
linux1
Stopping the Oracle RAC 10g Environment
The first step is to stop the Oracle instance. When the instance (and related services) is
down, then bring down the ASM instance. Finally, shut down the node applications
(Virtual IP, GSD, TNS Listener, and ONS).
$ export ORACLE_SID=orcl1
$ emctl stop dbconsole
$ srvctl stop instance -d orcl -i orcl1
$ srvctl stop asm -n linux1
$ srvctl stop nodeapps -n linux1
Starting the Oracle RAC 10g Environment
The first step is to start the node applications (Virtual IP, GSD, TNS Listener, and ONS).
When the node applications are successfully started, then bring up the ASM instance.
Finally, bring up the Oracle instance (and related services) and the Enterprise Manager
Database console.
$ export ORACLE_SID=orcl1
$ srvctl start nodeapps -n linux1
$ srvctl start asm -n linux1
$ srvctl start instance -d orcl -i orcl1
$ emctl start dbconsole
Start/Stop All Instances with SRVCTL
Start/stop all the instances and their enabled services. I have included this step just for fun
as a way to bring down all instances!
$ srvctl start database -d orcl

$ srvctl stop database -d orcl
29. Transparent Application Failover (TAF)
It is not uncommon for businesses to demand 99.99% (or even 99.999%) availability for
their enterprise applications. Think about what it would take to ensure a downtime of no
more than .5 hours or even no downtime during the year. To answer many of these high-
availability requirements, businesses are investing in mechanisms that provide for
automatic failover when one participating system fails. When considering the availability
of the Oracle database, Oracle RAC 10g provides a superior solution with its advanced
failover mechanisms. Oracle RAC 10g includes the required components that all work
within a clustered configuration responsible for providing continuous availability; when
one of the participating systems fail within the cluster, the users are automatically
migrated to the other available systems.
A major component of Oracle RAC 10g that is responsible for failover processing is the
Transparent Application Failover (TAF) option. All database connections (and processes)
that lose connections are reconnected to another node within the cluster. The failover is
completely transparent to the user.
This final section provides a short demonstration on how TAF works in Oracle RAC 10g.
Please note that a complete discussion of failover in Oracle RAC 10g would require an
article in itself; my intention here is to present only a brief overview.
One important note is that TAF happens automatically within the OCI libraries. Thus
your application (client) code does not need to change in order to take advantage of TAF.
Certain configuration steps, however, will need to be done on the Oracle TNS file
tnsnames.ora. (Keep in mind that as of this writing, the Java thin client will not be able to
participate in TAF because it never reads tnsnames.ora.)
Setup the tnsnames.ora File
Before demonstrating TAF, we need to verify that a valid entry exists in the tnsnames.ora
file on a non-RAC client machine (if you have a Windows machine lying around). Ensure
that you have the Oracle RDBMS software installed. (Actually, you only need a client
install of the Oracle software.)
During the creation of the clustered database in this guide, we created a new service that
will be used for testing TAF named ORCLTEST. It provides all the necessary
configuration parameters for load balancing and failover. You can copy the contents of
this entry to the %ORACLE_HOME%\network\admin\tnsnames.ora file on the client
machine (my Windows laptop is being used in this example) in order to connect to the
new Oracle clustered database:
...
ORCLTEST =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = linux1-vip)(PORT = 1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = linux2-vip)(PORT = 1521))
(LOAD_BALANCE = yes)
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = orcltest.idevelopment.info)
(FAILOVER_MODE =
(TYPE = SELECT)
(METHOD = BASIC)
(RETRIES = 180)
(DELAY = 5)
)
)
)
...
SQL Query to Check the Session's Failover Information
The following SQL query can be used to check a session's failover type, failover method,
and if a failover has occurred. We will be using this query throughout this example.
COLUMN instance_name FORMAT a13

COLUMN host_name FORMAT a9
COLUMN failover_method FORMAT a15
COLUMN failed_over FORMAT a11
SELECT
instance_name
, host_name
, NULL AS failover_type
, NULL AS failover_method
, NULL AS failed_over
FROM v$instance
UNION
SELECT
NULL
, NULL
, failover_type
, failover_method
, failed_over
FROM v$session
WHERE username = 'SYSTEM';
TAF Demo
From a Windows machine (or other non-RAC client machine), login to the clustered
database using the orcltest service as the SYSTEM user:
C:\> sqlplus system/manager@orcltest

SELECT
instance_name
, host_name
FROM v$instance
UNION
SELECT
NULL
, NULL
, failover_type
, failover_method
, failed_over
FROM v$session
INSTANCE_NAME HOST_NAME FAILOVER_TYPE FAILOVER_METHOD FAILED_OVER

------------- --------- ------------- --------------- -----------
orcl1 linux1
SELECT BASIC NO
DO NOT log out of the above SQL*Plus session!
Now that we have run the query (above), we should now shutdown the instance orcl1 on
linux1 using the abort option. To perform this operation, you can use the srvctl
command-line utility as follows:
# su - oracle
$ srvctl stop instance -d orcl -i orcl1 -o abort

Instance orcl1 is not running on node linux1
Now let's go back to our SQL session and rerun the SQL statement in the buffer:
SELECT
instance_name
, host_name
FROM v$instance
UNION
SELECT
NULL
, NULL
, failover_type
, failover_method
, failed_over
FROM v$session
INSTANCE_NAME HOST_NAME FAILOVER_TYPE FAILOVER_METHOD FAILED_OVER

------------- --------- ------------- --------------- -----------
orcl2 linux2
SELECT BASIC YES
SQL> exit
From the above demonstration, we can see that the above session has now been failed
over to instance orcl2 on linux2.
30. Troubleshooting
OCFS2 - Configure O2CB to Start on Boot
With the releases of OCFS2 prior to 1.2.1, there is a bug that exists where the driver does
not get loaded on each boot even after configuring the on-boot properties to do so. After
attempting to configure the on-boot properties to start on each boot according to the
official OCFS2 documentation, you will still get the following error on each boot:
...
Mounting other filesystems:
mount.ocfs2: Unable to access cluster service
Cannot initialize cluster mount.ocfs2:

Unable to access cluster service Cannot initialize cluster [FAILED]
...
Red Hat changed the way the service is registered between chkconfig-1.3.11.2-1 and
chkconfig-1.3.13.2-1. The O2CB script used to work with the former.
Before attempting to configure the on-boot properties:
REMOVE the following lines in /etc/init.d/o2cb

### BEGIN INIT INFO
# Provides: o2cb
# Required-Start:
# Should-Start:
# Required-Stop:
# Default-Start: 2 3 5
# Default-Stop:
# Description: Load O2CB cluster services at system boot.
### END INIT INFO
Re-register the o2cb service.
# chkconfig --del o2cb
# chkconfig --add o2cb
# chkconfig --list o2cb
o2cb 0:off 1:off 2:on 3:on 4:on 5:on
6:off

# ll /etc/rc3.d/*o2cb*
lrwxrwxrwx 1 root root 14 Sep 29 11:56 /etc/rc3.d/S24o2cb
-> ../init.d/o2cb
The service should be S24o2cb in the default runlevel.
After resolving the bug I listed above, we can now continue to set the on-boot properties
as follows:
# /etc/init.d/o2cb offline ocfs2
Configuring the O2CB driver.
This will configure the on-boot properties of the O2CB driver.

The following questions will determine whether the driver is loaded on
boot. The current values will be shown in brackets ('[]'). Hitting
without typing an answer will keep that current value. Ctrl-C
will abort.
Load O2CB driver on boot (y/n) [n]: y

OCFS2 - Adjusting the O2CB Heartbeat Threshold
With previous versions of this article, (using FireWire as opposed to iSCSI for the shared
storage), I was able to install and configure OCFS2, format the new volume, and finally
install Oracle Clusterware (with its two required shared files; the voting disk and OCR
file), located on the new OCFS2 volume. While I was able to install Oracle Clusterware
and see the shared drive using FireWire, however, I was receiving many lock-ups and
hanging after about 15 minutes when the Clusterware software was running on both
nodes. It always varied on which node would hang (either linux1 or linux2 in my
example). It also didn't matter whether there was a high I/O load or none at all for it to
crash (hang).
After looking through the trace files for OCFS2, it was apparent that access to the voting
disk was too slow (exceeding the O2CB heartbeat threshold) and causing the Oracle
Clusterware software (and the node) to crash.
The solution I used was to simply increase the O2CB heartbeat threshold from its default
setting of 7, to 61. Some setups may even require a higher setting. This is a configurable
parameter that is used to compute the time it takes for a node to "fence" itself. The tasks
required to increase the O2CB heartbeat threshold are fully documented in the section
"Adjust the O2CB Heartbeat Threshold" of this article.
Oracle Clusterware (10g RAC 10.1.0.3 and higher) - CSS Timeout Computation
This section pertains to both Oracle10g R1 (10.1.0.3 and higher) and Oracle10g R2 (the
release of Oracle this article is based on). Please note that I did not need to perform any
of the steps documented in this section for this article to work. It is included for
documentation purposes only in the rare case the symptoms described do occur on your
RAC system. For a more detailed and current overview of CSS timeout computation in
RAC 10g, please see Metalink note 294430.1.
In some cases, after the Oracle Clusterware software is installed, you will need to modify
the CSS timeout value for Clusterware. This is not true, however, for Oracle10g R2 and
higher. Oracle10g R2 users should instead apply patch 4896338 and NOT modify the
default misscount value. For 10.1.0.3 and higher, the CSS timeout is computed differently
than with 10.1.0.2. Several problems have been documented as a result of the CSS
daemon timing out starting with Oracle 10.1.0.3 on the Linux platform (including IA32,
IA64, and x86-64). This has been a big problem for me in the past, especially during
database creation (DBCA) and using FireWire as the shared storage. During the database
creation process, for example, it was not uncommon for the database creation process to
fail with the error: ORA-03113: end-of-file on communication channel. The key
error was reported in the log file $ORA_CRS_HOME/css/log/ocssd1.log as:
clssnmDiskPingMonitorThread: voting device access hanging (45010

miliseconds)
The problem was essentially slow disks and the default value for CSS misscount. The
CSS misscount value is the number of heartbeats missed before CSS evicts a node. CSS
uses this number to calculate the time after which an I/O operation to the voting disk
should be considered timed out and thus terminating itself to prevent split brain
conditions. The default value for CSS misscount on Linux for Oracle 10.1.0.2 and higher
is 60. The formula for calculating the timeout value (in seconds), however, did change
from release 10.1.0.2 to 10.1.0.3.
With 10.1.0.2, the timeout value was calculated as follows:
time_in_secs > CSS misscount, then EXIT

With the default value of 60, for example, the timeout period would be 60 seconds.
Starting with 10.1.0.3, the formula was changed to:

disktimeout_in_secs = MAX((3 * CSS misscount)/4, CSS misscount - 15)
Again, using the default CSS misscount value of 60, this would result in a timeout of 45
seconds.
This change was motivated to allow for a faster cluster reconfiguration in case of node
failure. With the default CSS misscount value of 60 in 10.1.0.2, we would have to wait at
least 60 seconds for a timeout, where this same default value of 60 can be shaved by 15
seconds to 45 seconds starting with 10.1.0.3.
OK, so why all the talk about CSS misscount? As I mentioned earlier, I would often have
the database creation process fail (or other high I/O loads to the system) from the Oracle
Clusterware crashing. The high I/O would cause lengthy timeouts for CSS while
attempting to query the voting disk. When the calculated timeout was exceeded, Oracle
Clusterware crashed. This was common as the network and external drive I was using
were not the fastest. The slower the access is to the shared drive, the more often this will
occur.
Well, the good news is that we can modify the CSS misscount value from its default
value of 60 (for Linux) to allow for lengthier timeouts. For slower drives, I have been
able to get away with a CSS misscount value of 360. Although I haven't been able to
verify this, I believe the CSS misscount can be set as large as 600.
Please note that Oracle support does not recommend modifying the misscount value.
They always recommend that you apply the latest patchset which changes the CSS
behaviour (not evicting the node from the cluster due to I/O to voting disk taking more
than misscount seconds unless it is during the initial cluster formation or slightly before
reconfiguration).
It is also worth noting that with Oracle10g R2, there is no need to increase the misscount
at all. Starting with 10.2.0.1 + Bug 4896338, CSS will not evict the node from the cluster
due to (IOT) I/O to voting disk taking more than misscount seconds unless it is during the
initial cluster formation or slightly before reconfiguration. So if we have a N number of
nodes in a cluster and one of the nodes takes more than misscount seconds to access the
voting disk, the node will not be evicted as long as the access to the voting disk is
completed within disktimeout seconds. Consequently with this patch, there is no need to
increase the misscount at all.
So, if you are using 10g R1, how do we modify the default value for CSS misscount?
Well, there are several ways. The easiest way is to modify the
$ORA_CRS_HOME/install/rootconfig script after installing Oracle Clusterware but
BEFORE running the root.sh script on both of the Oracle RAC nodes in the cluster. For
example, to modify the entry for CSS misscount from 60 to 360, modify the file
$ORA_CRS_HOME/install/rootconfig as follows (on both of the Oracle RAC nodes in
the cluster). Change the following entry that can be found on line 356:
CLSCFG_MISCNT="-misscount 60"
to
CLSCFG_MISCNT="-misscount 360"
If Oracle Clusterware is already installed, you can still modify the CSS misscount value
using the $ORA_CRS_HOME/bin/crsctl command. First, you should query the current
value for CSS misscount:
# $ORA_CRS_HOME/bin/crsctl get css misscount

360
If you get back a value of 60, you can modify it to 360 as follows:
Start only one node in the cluster. For my example, I would shutdown
linux2 and startup only linux1.
From the one node (linux1), login as the root user account and type:
$ORA_CRS_HOME/bin/crsctl set css misscount 360

Reboot the single node (linux1).
Start all other nodes in the cluster.
31. Conclusion
Oracle10g RAC allows the DBA to configure a database solution with superior fault
tolerance and load balancing. For those DBAs, however, that want to become more
familiar with the features and benefits of Oracle10g RAC will find the costs of
configuring even a small RAC cluster costing in the range of US$15,000 to US$20,000.
This article has hopefully given you an economical solution to setting up and configuring
an inexpensive Oracle10g Release 2 RAC Cluster using CentOS 4 Linux (or Red Hat
Enterprise Linux 4) and iSCSI technology. The RAC solution presented in this article can
be put together for around US$2,200 and will provide the DBA with a fully functional
Oracle10g Release 2 RAC cluster. While the hardware used for this article should be
stable enough for educational purposes, it should never be considered for a production
environment.
32. Acknowledgements
An article of this magnitude and complexity is generally not the work of one person
alone. Although I was able to author and successfully demonstrate the validity of the
components that make up this configuration, there are several other individuals that
deserve credit in making this article a success.
First, I would like to thank Bane Radulovic from the Server BDE Team at Oracle
Corporation. Bane not only introduced me to Openfiler, but shared with me his
experience and knowledge of the product and how to best utilize it for Oracle10g RAC.
His research and hard work made the task of configuring Openfiler seamless. Bane was
also involved with hardware recommendations and testing.
I would next like to thank Werner Puschitz for his outstanding work on "Installing
Oracle Database 10g with Real Application Cluster (RAC) on Red Hat Enterprise Linux
Advanced Server 3". This article, along with several others of his, provided information
on Oracle10g RAC that could not be found in any other Oracle documentation. Without
his hard work and research into issues like configuring and installing the hangcheck-timer
kernel module, properly configuring UNIX shared memory, and configuring ASMLib,
this article may have never come to fruition. If you are interested in examining technical
articles on Linux internals and in-depth Oracle configurations written by Werner
Puschitz, please visit his excellent website at www.puschitz.com.
Also, thanks to Tzvika Lemel for his comments and suggestions on using Oracle's Cluster
Verification Utility (CVU).
Lastly, I would like to express my appreciation to the following vendors for generously
supplying the hardware for this article; Maxtor, D-Link, SIIG, and LaCie.
Jeffrey Hunter [www.idevelopment.info] graduated from Stanislaus State University in

Turlock, California, with a Bachelor's degree in Computer Science and has been a senior
DBA and software engineer for over 12 years. He is an Oracle Certified Professional,
Java Development Certified Professional, an Oracle ACE, Author, and currently works as
a Senior Database Administrator for The DBA Zone, Inc.. Jeff's work includes advanced
performance tuning, Java and PL/SQL programming, capacity planning, database
security, and physical / logical database design in a UNIX, Linux, and Windows NT
environment. Jeff's other interests include mathematical encryption theory, programming
language processors (compilers and interpreters) in Java and C, LDAP, writing web-
based database administration tools, and of course Linux.

Build Your Own Oracle RAC 10g Release 2 Cluster On Linux and ISCSI

Cargado por

Información del documento

Título original

Derechos de autor

Formatos disponibles

Compartir este documento

Compartir o incrustar documentos

Opciones para compartir

¿Le pareció útil este documento?

¿Este contenido es inapropiado?

Copyright:

Formatos disponibles

Build Your Own Oracle RAC 10g Release 2 Cluster On Linux and ISCSI

Cargado por

Copyright:

Formatos disponibles

Build Your Own Oracle RAC 10g Release 2 Cluster on

Linux and iSCSI

Updated November 2006

Downloads for this guide:

SCSI Controller:Two SCSI controllers priced from $20 (Adaptec AHA-2940UW)

Oracle Database Files

linux2 orcl2 orcl /u01/app/oracle ASM

CRS Voting Disk /u02/oradata/orcl/CSSFile crs /u02/oradata/orcl OCFS2

The Oracle Clusterware software will be installed to /u01/app/oracle/product/crs on

Note: This article is only designed to work as documented with absolutely no

2. Oracle RAC 10g Overview

A less expensive alternative to fibre channel is SCSI. SCSI technology provides

Several of the advantages to FC SAN include greater performance, increased disk

5. Hardware & Costs

Oracle RAC Node 1 - (linux1)

Dimension 2400 Series US$620

1 - Ethernet LAN Card

Oracle RAC Node 2 - (linux2)

Dimension 2400 Series

1 - Ethernet LAN Card US$20

Network Storage Server - (openfiler1)

1 - Ethernet LAN Card

Storage Device(s) - External Hard Drive US$320

6. Install the Linux Operating System

Download the following ISO images for CentOS 4.4:

CentOS-4.4-i386-bin1of4.iso (622 MB)

Welcome to CentOS Enterprise Linux

Language / Keyboard Selection

Disk Partitioning Setup

Boot Loader Configuration

Additional Language Support/Time Zone

Set Root Password

Package Group Selection

Graphical Interface (X) Configuration

Perform the same installation on the second node

Second, [Edit] both eth0 and eth1 as follows:

Introduction to Network Settings

Configuring Public and Private Network

Our example configuration will use the following settings:

Oracle RAC Node 1 - (linux1)

# Public Network - (eth0)

# Private Interconnect - (eth1)

# Public Virtual IP (VIP) addresses for - (eth0)

Oracle RAC Node 2 - (linux2)

# Public Network - (eth0)

# Private Interconnect - (eth1)

# Public Virtual IP (VIP) addresses for - (eth0)

Figure 2 Network Configuration Screen, Node 1 (linux1)

Confirm the RAC Node Name is Not Listed in Loopback Address

127.0.0.1 linux1 localhost.localdomain localhost

ORA-00603: ORACLE server session terminated by fatal error

Adjusting Network Settings

# Default setting in bytes of the socket receive buffer

# Default setting in bytes of the socket send buffer

# Maximum socket receive buffer size which may be set by using

# Maximum socket send buffer size which may be set by using

Check and turn off UDP ICMP rejections

# chkconfig iptables off

Powered by rPath Linux, Openfiler is a free browser-based network storage management

To learn more about Openfiler, please visit their website at http://www.openfiler.com/

For more detailed installation instructions, please visit http://www.openfiler.com/docs/. I

Welcome to Openfiler NAS/SAN Appliance

Disk Partitioning Setup