Oracle RAC 11gR2

RAC Joins the Cloud

RAC Grows Up
 RAC 11gR2 (aka 12gR1) adds a number of features that
transforms RAC clusters into database clouds
 Many features are geared towards enterprises but there are
new features that any organization can take advantage of
 With today’s high power x86 and 11gR2 RAC it is
entirely possible to replace many silo clusters with fewer
11gR2 “clouds”
 Many 11gR1 and older RAC commands are depreciated
11gR2 RAC New Features
 Brand New Installer
 Clusterware is now Grid Infrastructure
 ASM is now part of the Grid Infrastructure installation
 Voting and OCR disk are ASM diskgroup
 Raw devices are only supported for upgraded clusters
 Future Linux release may drop support for raw devices
 Plug and Play For Clusters
 Cluster is now a DNS service
 DHCP for server VIPs
 New tools are introduced old tools retired
RAC 11gR2 Install
Non Linear Installer
Grid Plug and Play
SSH Keys
OCR/Voting Disks
OCR/Voting Disks
 OCR/Voting disks can be ASM Diskgroup
 Raw devices are only supported for upgraded clusters
 Other options are NFS and certified clustered filesystems
 3 disks for normal redundancy, 5 disks for high
IPMI
IPMI (cont)
 Intelligent Platform Management Interface
 Defines a cross vendor standard to monitor and manage
hardware
 Dell, HP, IBM, Intel, AMD and dozens more
 IPMI in relation to RAC allows other nodes to manage the
server regardless of the OS state
ASM Roles
Fix Common Problems
 Installer can create a script that can fix some problems
 Won’t install missing packages
Finish Install
 After GUI finished is will ask to run the command
oraInrootinst.sh and root.sh
Cluster Startup Sequence
 Init starts Oracle High Availability Service Deamon
(OHASD)
 OHASD starts
 cssdagent – starts cssd
 orarootagent – manages all root owned ohasd processes
 oraagent – manages all oracle owned ohsad resources
 cssdmonintor – monitors cssd and node health
 OHASD rootagent starts
 crsd – main deamon for managing cluster resources
 ctssd – time synchronization
 Diskmon
 ACFS – ASM cluster file system drivers
Cluster Startup Sequence
 Oraagent starts – level 2
 mdnsd – used for DNS lookups
 gipcd – inter-process and inter-node communication
 gpnpd – Grid Plug & Play deamon
 EVMD – event monitor
 ASM – resource for monitoring ASM instance
 CRSD – level 3
 orarootagent
 oraagent
 CRSD rootagent – level 4
 SCAN VIP(s) - Single Client Access Name
 Node VIPs – one per node
 ACFS Registry – For mounting ASM Cluster File System
 GNS VIP - optional
SCAN
 SIMPLE CLIENT ACCESS NAME
 A single DNS entry that represents the cluster has a whole
 Clients no longer need to know the actual servers that are in a
cluster
 THIN connections or connections not using a VIP are better
protected against a node failure
 If using GNS SCAN must be in the domain managed by GNS
 DNS needs to be able to delegate resolution to GNS
 If not using GNS DNS should be configured to round robin up
to 3 IPs for the SCAN name. Recommend use 11.2 client
 If using GNS 3 IPs will be requested from DHCP
SCAN Listener
 Listener that is dependent on a SCAN VIP
 If the SCAN VIP moves to a different node in the cluster the
listener will move with the VIP
 Depending on the number of nodes in a cluster a single node
may have more than one SCAN VIP and listener
 remote_listener parameter should be set to a SCAN VIP for
11gR2 databases
 Simplify RAC Dataguard connections for clients
 Two new sqlnet parameters
 CONNECT_TIME = timeout in seconds
 RETRY_COUNT = number of tries
 In a single TNS entry enter both primary and standby servers SCAN address.
If the connection is made to standby it will failover to primary.
SCAN Load Balancing
 Client receives IP from DNS or GNS that resolves to
SCAN VIP
 SCAN Listener bound to SCAN VIP redirects client to
local listener that has the least load
 Client connects to the local listener on the node VIP
 Client needs to be able to connect to SCAN VIP and to the
local node VIP and be able to resolve all names
SCAN
 SCAN Listeners Overview
Grid - Plug And Play
 Greatly simplifies the adding/removing and connecting to
a cluster
 Basically run cluster verify script and then addNode.sh
 Configures cluster listener and ASM
 No longer use static IPs for VIPs
GNS
 Grid Naming Service
 Handles name resolution for the cluster
 Requires DHCP for VIP addresses!
 DNS needs to delegate resolution for the cluster to GNS
 Use the GNS VIP not server or scan VIP
 Bind Example
 prodcluster.db.oracle.com NS gns.prodcluster.db.oracle.com
 gns.prodcluster.db.oracle.com. 10.90.51.20
 When a client connects the cluster will provide the IP address
 As RAC servers are added DHCP assigns a VIP address, GNS is
updated with the new server’s VIP address and now can route connects
to the new server.
 No client connection strings are never changed
 Remove a node is basically reversing the process
Client Connections With GNS
ASM Changes
 ASM part of grid infrastructure – one less HOME to
manage
 New GUI management tool – asmca
 Replaces DBCA for ASM management
 Create diskgroups, volumes, ACFS mounts
 Enhanced controls to manage ASM attributes
 SPFILE is stored in ASM diskgroup
 Cluster reads the diskgroup before ASM starts
 ASMCMD is greatly enhanced and mostly replaces sqlplus
 ASM roles can play a large role in managing ASM
 Diskgroups can be cluster resources
ASM Changes (cont)
 New O/S aware cluster file system (ACFS)
 ASM disks can be used to create filesystem mounted by O/S
 Only supported on Redhat/Oracle Linux, other O/S support coming in
the future
 If a file can natively support ASM then the file is not supported on
ACFS!
 No data, control or redo files
 DirectIO is not available
 Take advantage of ASM performance over other filesystems
 ACFS is able to do snapshots
 Still able to access ASM diskgroups with FTP/HTTP via
XMLDB
 Copy command can copy out and in of ASM (11.1.0.7)
ASM Changes (cont)
 Intelligent placement of data
 ASM is able to put hot datafiles on the outer edge of the disk
 Can provide a 50% increase in I/O performance
 Compatible parameters for both ASM & RDBMS 11.2
 Work best with the geometry of the disks are know
 JOBD – uses disks sectors for data placment
 LUN – logical sectoring number which may not have any relation to
the physical layout of the lun on the disk
 Diskgroup should be 50% full for full benefit
 alter diskgroup data add template datafile_hot attributes (hot
mirrorhot);
 ascmd lstmpl –l –G data
ASM – Access Control Lists
 Set permissions at the ASM file level
 Set permissions for users and groups
 User is the oracle database software owner
 Only available for Linux and Unix
 Not so much for security as for separation of duties
 Files created in ASM are owned by DBUSER
 Create a separate OSDBA group for each database using a
separate ORACLE_HOME. Need different groups for
OSASM and OSDBA for ASM
 Compatible 11.2 diskgroup both for ASM and RDBMS
ASM – Access Control Lists (cont)
 Disk attributes
 ACCESS_CONTROL.ENABLED = TRUE
 ACCESS_CONTROL.MASK has to be set
 Mask values
 6 removes all
 2 removes write
 0 removes nothing
 Mask of 026 sets values of 640
 read-write for owner, read for group and nothing for everyone else
 asmcmd> setattr –d data access_control_enabled true
 sql> alter diskgroup data set attribute ‘access_control.enabled’
= true;
ASM Permissions
 ALTER DISKGROUP ADD USERGROUP … WITH MEMBER
 ALTER DISKGROUP DROP USERGROUP
 ALTER DISKGROUP MODIFY USERGROUP ADD MEMBER
 ALTER DISKGROUP MODIFY USERGROUP DROP MEMBER
 ALTER DISKGROUP ADD USER
 ALTER DISKGROUP DROP USER
 ALTER DISKGROUP SET PERMISSION
ALTER DISKGROUP SET OWNERSHIP
 SELECT * FROM V$ASM_USER
 SELECT * FROM V$ASM_USERGROUP
 ALTER DISKGROUP ADD USERGROUP myfiles WITH
MEMBER bill;
ASM Roles
 OS groups control rights in ASM
 OSDBA – can access ASM files and set ACLs
 ASMOPER – can start/stop ASM
 OSASM – full ASM control
 Allow people to administer ASM without having sysdba
rights to the databases
ASM Commandline Tool
ASM Commandline Tool (cont)
 Create diskgroups from ascmd
 Same syntax as sqlplus
 Create diskgroup mydata external redundancy
 DISK ‘ORCL:DISK1’ NAME mydata1, ‘ORCL:DISK2’ NAME
mydata2;
 Create diskgroup from XML via asmcmd
 asmcmd> chkdg
ASM Commandline Tool (cont)
 lsdg – list diskgroup information
 Very similar to select * from v$asm_diskgroup
 lsdsk – list disk information
 Same information found in select * from v$asm_disk
Cluster Registry & Voting Disks
 Raw devices only supported for upgraded clusters
 Can move OCR and Voting disks to ASM diskgroup after
upgrade
 ocrconfig add <diskgroup name>
 ocrcheck
 ocrconfig delete <raw device>
 crsctl query css votedisk
 crsctl add css votedisk <diskgroup name>
 crsctl delete css votedisk <raw device>
Cluster Management
 crsctl in the grid infrastructure home manages almost
every cluster command
 crs_* scripts are depreicated
Depreciated Commands
 crs_stat
 crs_register
 crs_unregister
 crs_start
 crs_stop
 crs_getperm
 crs_profile
 crs_relocate
 crs_setperm
 crsctl check crsd
 crsctl check cssd
 crsctl check evmd
 crsctl debug log
 crsctl set css votedisk
 crsctl start resources
 crsctl stop resources
Server Pools
 Create “mini” clusters inside of larger clusters for policy
based databases
 Two default pools Free/Generic
 Free pool contains servers not assigned to another pool
 Generic pool is for running pre 11.2 databases and non policy
managed databases
 Specify min/max nodes and priority
 Cluster manages the members of the pool based on load, availability,
priority and min/max requirements
 Cluster will move servers from free and lower priority pools to meet
the needs of the server pool
 Can create ACL on server pools for role based management
Vote/OCR Disk Management
 Raw is supported for upgraded clusters but it is possible to
migrate to ASM disks after 11.2 cluster upgrade
 Move vote disks to ASM
 crsctl query css votedisk
 crsctl replace votedisk +asm_disk_group
 Move OCR to ASM
 ocrconfig -add +new_disk_group
 ocrconfig -delete old_storage_location
Troubleshooting Cluster Problems
 Golden Rule – Ensure time is synced across cluster
 Help with comparing log files
 Ensures nodes aren’t evicted due to time skew
 If NTP is not used Oracle will use Cluste Time Synchronization
service
 Recommend to have cluster down if changing the time
 Logs are stored under GRID_HOME/log/hostname
 Main alert log
 Separate directory for each cluster process
 Diagnostic Script
 GRID_HOME/bin/diagcollection.pl
 Collects cluster and system logs for debugging
Troubleshooting (cont)
 Modify time before node is evicted
 Default is 30 seconds
 crsctl set css miscount 45
 On busy systems logs may not flush before node reboots
 Crsctl set css diagwait 13 – force
 Not set by default
 Requires cluster outage to set
Adding A Node
 Much easier in 11gR2
 Setup ssh keys for new host
 Run cluster verify tool
 cluvfy stage –pre crsint –n <new host> <-fixup>
 Fixes any problems
 cluvfy stage –pre nodeadd –n <new host>
 $ORACLE_HOME/oui/bin/addNode.sh –silent
“CLUSTER_NEW_NODES=<new host>”
“CLUSTER_NEW_VIRTUAL_HOSTNAMES=<new host
vip>”
Instance Caging
 Puts a cap on the number of CPUs a instance will use
 Set the cpu_count parameter
 Enable resource manager with CPU polices
Oracle Joins The Cloud
 With 11gR2 it is now possible to create few larger clusters
while being able to maintain performance, security and
reassure customers concerns
 Increased utilization rates
 Lower administration costs
 Happy Management