8QLW2EMHFWLYHV

,QWKLVVHFWLRQ\RXZLOOILQGRXWDERXW
2YHUYLHZRI+$&03(6
'LIIHUHQFHVEHWZHHQ+$&03(6DQG+$&03FODVVLF
:KDW,V+$&03(6"

+$&03(QKDQFHG
6FDODELOLW\
HACMP ES
6FDODEOHKLJKDYDLODELOLW\
FOXVWHUIRU56
H[WHQGHGIXQFWLRQDOLW\
FRPSDUHGZLWKFODVVLF+$&03
XSWRQRGHVLQRQH+$&03
FOXVWHU
EDVHGRQ,%0+LJK$YDLODELOLW\
,QIUDVWUXFWXUH56&7
LQFOXGHVDOOIHDWXUHVRIFODVVLF
+$&03
'LIIHUHQFHV%HWZHHQ&ODVVLFDQG(6

+$&03(6XVHV56&7
8VHU'HILQHG(YHQWVEDVHGRQ56&7
$SSOLFDWLRQ0RQLWRULQJ
5HFRYHU\IURP5HVRXUFH*URXS$FTXLVLWLRQ)DLOXUH
'\QDPLF1RGH3ROLF\
6HOHFWLYH)DOORYHU
3OXJLQV
5HOLDEOH6FDODEOH&OXVWHU7HFKQRORJ\
56&7

(0$3,

(YHQW0DQDJHPHQW

*URXS
HACMP
6HUYLFHV ES

5HVRXUFH0RQLWRUV 7RSRORJ\
6HUYLFHV

6: +:
56&7LQ63

3HUVSHFWLYHV

30DQ

373(
(0$3,

37; (YHQW0DQDJHPHQW

*URXS
HACMP
6HUYLFHV ES

5HVRXUFH0RQLWRUV 7RSRORJ\
6HUYLFHV

6: +:
1HZ+$&03&OXVWHU0DQDJHU

+$&03(6&OXVWHU0DQDJHU
)XQFWLRQDOLW\RIFODVVLF+$&03&OXVWHU0DQDJHUVSOLWLQWR
IRXUPRGXOHV
5HFRYHU\&RPPDQG

5HFRYHU\ 5HFRYHU\&RPPDQG
BB

3URJUDPV BB
BB

+$&03IRU$,; +$&03(6&OXVWHU
&OXVWHU0DQDJHU 0DQDJHU

*URXS6HUYLFHV(6

7RSRORJ\6HUYLFHV(6
8VHU'HILQHG(YHQW'HWHFWLRQ

+$&03(6XVHV56&7HYHQWGHWHFWLRQ
8VHUGHILQHGEDUULHUSRLQWV
8VHUGHILQHGHYHQWV

&RQGLWLRQ
+$&03(6
(YHQW (YHQW
&OXVWHU
0DQDJHPHQW 6XEVFULEHU 0DQDJHU

5HVRXUFHYDULDEOHV

5HVRXUFH
0RQLWRU 5HFRYHU\3URJUDP
&38ORDG
2EVHUYH 3DJLQJVSDFH
WXQDEOHUDWH )LOHV\VWHP
$SSO
3URFHVV
'DWDEDVH
/RDG $,;  $FWLRQ
 )XOO HJWDNHRYHU
6\QFKURQL]DWLRQRI5HFRYHU\3URJUDPV

(DFK&OXVWHU0DQDJHUFRQWUROVQRGH 5HFRYHU\3URJUDPV

V\QFKURQL]DWLRQXVLQJEDUULHUSRLQWV
([DPSOH1RGHMRLQLQJ
5HFRYHU\ 1RGH 1RGH 1RGH 1RGH
1RGHVHW FRPPDQG 5XQQLQJ 5XQQLQJ -RLQLQJ 5XQQLQJ
QRGHBXS QRGHBXS QRGHBXS EDUULHU QRGHBXS

,QRWKHU UF 
QRGHV UF 
EDUULHU EDUULHU
UF 
6\QF EDUULHU
,QHYHQW QRGHBXS EDUULHU EDUULHU QRGHBXS EDUULHU
QRGHV UF 
6\QF EDUULHU
QRGHBXS QRGHBXS QRGHBXS QRGHBXS QRGHBXS
LQDOO BFRPSOHWH BFRPSOHWH BFRPSOHWH BFRPSOHWH BFRPSOHWH
QRGHV
6\QF &EDUULHU &EDUULHU &EDUULHU &EDUULHU
([DPSOHRID8VHU'HILQHG(YHQW

(YHQWH[DPSOH
)LOHV\VWHPWPSIXOO
5HVRXUFHYDULDEOHQDPH,%03663DL[RV)6WRWXVHG

 ,QVWDQFHYHFWRU1RGH1XP 

 3UHGLFDWH;!

 5HDUPSUHGLFDWH;

(YHQW (YHQW
5HDUPHYHQW
RSWLRQDO
([DPSOHRID8VHU'HILQHG(YHQW

(YHQWH[DPSOH
$SSOLFDWLRQSURFHVVGLVDSSHDUHG
5HVRXUFHYDULDEOHQDPH,%036633URJ[SFRXQW
6WUXFWXUHG%\WH6WULQJ
;# ;# ;#
&XUUHQWQXPEHU 3UHYLRXVQXPEHU /LVWRISURFHVV
RISURFHVVHV RISURFHVVHV ,'V

,QVWDQFHYHFWRU
1RGH1XP 3URJ1DPH XSU8VHU1DPH URRW
3UHGLFDWH;#  ;# 
5HDUPSUHGLFDWH;#!
# ps -ef | grep upr
root 7218 5624 0 Jul 18 - 0:00 /usr/sbin/upr6
root 39276 20298 1 10:52:01 pts/8 0:00 grep upr
&RQILJXULQJD8'(

Add Custom User Defined Event

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

[Entry Fields]
* Event name []
* Recovery program path []
* Resource variable name []
* Instance vector []
* Predicate []
Rearm predicate []

To get to this new panel, smit hacmp: Cluster Custom

Modification: Define Custom User Defined Event: Add Custom
User Defined Event
These are the same fields formerly configured in the
rules.hacmprd file.
The Recovery program specifies the executable file(s) that will
be run when the event takes place.
The other parameters are described in RSCT documentation.
$Q([DPSOH8'(

This make-believe UDE logs users out when the number of

processes grows too large. It runs a command "logout_usrs."
Add Custom User Defined Event

Type or select values in entry fields.

Press Enter AFTER making all desired changes.
The variable is
[Entry Fields] IBM.PSSP.aixos.
* Event name [dump_users]
* Recovery program path <dm/logout_usrs] Proc.swpque,
* Resource variable name <os.Proc.swpque] which measures the
* Instance vector [NodeNum=*]
* Predicate [X > 50] number of
Rearm predicate [] processes waiting
to be swapped in.

The instance vector indicates that we will measure this variable

on all the nodes, and take action when the condition is met on
any of them.
The predicate indicates that we will take action when the number
of waiting processes exceeds 50.
$SSOLFDWLRQ0RQLWRULQJ

SAP HACMP ES Cluster

Manager

run_clappmond

clappmond

Process Custom
Monitoring Monitoring
RSCT - Event User-defined Monitoring
Management Method

Application Application
3URFHVV0RQLWRU6XFFHVVIXO5HVWDUW
Increase Failure Count
Stabilization Interval
Set Failure Count to 0

Failure Count

Monitor

Monitor
Event

Event
Restart
3 Method Continue
Monitoring

Monitor
Restart

Event
2 Method
Monitor

Restart
Event

1 Method

Restart Interval
Start
Death
Event

0
3URFHVV0RQLWRU8QVXFFHVVIXO5HVWDUW
Increase Failure Count
Stabilization Interval
Set Failure Count to 0 Failure
Action
Failure Count

Monitor
Event
Restart
3 Method

Monitor
Restart

Event
2 Method
Monitor

Restart
Event

1 Method

Restart Interval
Start
Death
Event

0
&XVWRP0RQLWRU6XFFHVVIXO5HVWDUW
Increase Failure Count
Stabilization Interval
Set Failure Count to 0
Monitor Interval

Failure
Count

Monitor

Monitor

Monitor
Restart

Event

Event

Event
3 Method

Monitor
Restart

Event
2 Method
Monitor

Restart
Event

1 Method

Restart Interval
Start
Death
Event

0
&XVWRP0RQLWRU8QVXFFHVVIXO5HVWDUW
Increase Failure Count
Stabilization Interval
Set Failure Count to 0
Monitor Interval

Failure Action
Failure
Count

Monitor
Restart

Event
3 Method

Monitor
Restart

Event
2 Method
Monitor

Restart
Event

1 Method

Restart Interval
Start
Death
Event

0
!1RGH&OXVWHU6XSSRUW
Recovery from Resource Group Acquisition Failure
RG1 RG2 cascading RG1: [A B C D] RG2 RG1
* * **
A B C D 1 A B C D
node_down A
disk disk
Node A fails, RG1 must move. B Node B's standby adapter fails. The
R G1 cluster does not fail.
move
_
r g 2
RG2 RG1 RG2 RG1
* * * *
A B C D 3 A B C D

rg_move RG1 C
disk disk
Node C cannot connect to the RG1 comes online on node D.
disk. The cluster does not fail.
!1RGH&OXVWHU6XSSRUW
Creating a Node Priority Policy
Configure Dynamic Node Priority Policies

Move cursor to desired item and press Enter.

Add a Dynamic Node Priority Policy

Change/Show a Dynamic Node Priority Policy
Remove a Dynamic Node Priority Policy

Add a Dynamic Node Priority Policy

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

[Entry Fields]
* Dynamic Node Priority Policy Name []
Dynamic Node Priority Policy Description []
* Resource Variable +
* Condition largest +

Every NPP needs a name. The resource variable must be selected

Description is optional. from a menu (next slide).
Condition is "largest" or "smallest."
6HOHFWLQJD5HVRXUFH9DULDEOH

lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk
x Resource Variable x
x x
x Move cursor to desired item and press Enter. x
x x
x [TOP] x
x IBM.PSSP.aixos.PagSp.totalfree Available virtual memory paging space x
x IBM.PSSP.aixos.Disk.busy Fraction of time disks are busy x
x IBM.PSSP.aixos.CPU.glidle Available processor time x
x IBM.PSSP.aixos.CPU.glkern x
x IBM.PSSP.aixos.CPU.gluser x
x IBM.PSSP.aixos.CPU.glwait x
x IBM.PSSP.aixos.cpu.idle x
x IBM.PSSP.aixos.cpu.kern x
x IBM.PSSP.aixos.cpu.user x
x IBM.PSSP.aixos.cpu.wait x
x IBM.PSSP.aixos.Disk.rblk x
x [MORE...67] x
mqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj

If you configure a NPP on a node that does not have a running cluster
manager, only the first three variables are listed along with their descriptions.
If a cluster manager is running, then the utility queries RSCT for all the
variables that are currently available and that are not state variables. In this
way, users have access to all the variables available.
&RQILJXULQJD5HVRXUFH*URXSZLWK'13

Change/Show Resources/Attributes for a Resource Group

Type or select values in entry fields.

Press Enter AFTER making all desired changes.

[TOP] [Entry Fields]

Resource Group Name cas1
Node Relationship cascading
Participating Node Names / Default Node Priority kenny stan er>

Dynamic Node Priority [] +

Service IP label [kenny_trsvc] +

F1=Help F2=Refresh F3=Cancel F4=List

F5=Reset F6=Command F7=Edit F8=Image
F9=Shell F10=Exit Enter=Do

The Dynamic Node Priority field is new in the HAES SMIT panel. If it is
blank, then HAES will use the default, predefined priority.
The administrator can select a NPP from the menu (recommended) or
type in a NPP name directly. The name of the NPP is checked only at
verification.
6HOHFWLYH)DOORYHU2SHUDWLRQ
cascading RG1: [A B C D]
cascading RG3: [Q A B C D]
RG1 RG3 RG2 RG2 RG1 RG3

A B C D A B C D

The lookahead feature

disk ensures that RG1 and RG3 disk
end up on different nodes.

RG1 RG2 RG2 RG1

A B C D A B C D

RG3 RG3
When the adapters fail on one
network, the resource group on
that network moves.
3OXJLQV

In 4.4.1, plugins are provided to help configure the following

services:
DNS
DHCP
Print services
The plugins add the server application and an application
monitor to an existing resource group.
Process Monitoring via provided plugin scripts
Users must create their own scripts/programs for Custom
Monitoring
an example custom monitor script is provided for the DNS
monitor.
,QVWDOOLQJWKH3OXJLQV

Filesets are located on HACMP/ES install media under the

cluster.es.plugins package, and can be installed with SMIT or
installp.
Alternatively, files can be downloaded in tar format from
http://www.ibm.com/servers/eserver/pseries/solutions/ha/ha_plugins.html
Customer can choose which plugins to install.
The plugins work with HACMP/ES 4.4 and 4.4.1.
Fileset names are:
cluster.es.plugins.dns
cluster.es.plugins.dhcp
cluster.es.plugins.printserver
Application plugins install into /usr/es/sbin/cluster/plugins/<plugin
name>
:KDW3OXJLQV*LYH<RX

The resource group now contains an application server to start

and stop the relevant daemons.
The resource group also contains the shared volume group
and filesystem needed by the daemons.
An application monitor is configured to watch the daemons.
When the resource group comes online, the daemon(s) will be
activated.
If a daemon should fail, the application monitor will detect it.
the daemon will be restarted or the resource group will move to
another node.
Note that the limitations of application monitoring still exist!
only one monitored application per resource group.
the monitored application can be included in at most one resource
group.
&KHFNSRLQW

 :KDWLVWKHPD[LPXPQXPEHURIQRGHVDQ+$&03(6
FOXVWHUFDQFRQVLVWRI"

 7) +$&03(6FDQRQO\EHXVHGLQWKH63
7)
+$&03(6UHTXLUHV56&OXVWHU7HFKQRORJ\
56&7

7) +$*(2RQO\ZRUNVZLWK+$&03(6

6XPPDU\

<RXVKRXOGQRZEHDEOHWR
'HVFULEHWKHEDVLFVRIWKHDUFKLWHFWXUHRQZKLFK
+$&03(6LVEDVHG
'HVFULEHWKHGLIIHUHQFHVEHWZHHQ+$&03(6DQG
+$&03FODVVLF