Progress Database Administration

DB-13: OpenEdge VLDB
Dan Foreman
OpenEdge VLDB
(Very Large DataBases)
DataBases)
Dan Foreman
BravePoint
danf@prodb.com
Introduction - Dan Foreman

Progress User since 1984
Author of:
Progress Database Administration Guide
Progress Performance Tuning Guide
V10 Database Administration Jumpstart
Virtual System Tables
Pro Dump & Load
ProMonitor
2
Audience Survey
Progress Version
Progress V8
Progress V9
OpenEdge R10.0*
OpenEdge R10.1A
OpenEdge R10.1B
3
Progress Exchange 2007

10-13 June, Phoenix, AZ, USA

Dan Foreman
Audience Survey
Largest Single Database Size
> 1tb
> 500gb
> 250gb
> 100gb
Everyone else can leave the room
4
Agenda
Definition of VLDB
Common Characteristics of VLDB
Growth Rates and Capacity Planning
Top Challenges for VLDB Customers
Wish List
Questions
Conclusion
5
Definition of VLDB
Minimum of 100gb
Single Database (not a set)
For this presentation I didn
didnt
differentiate between allocated space
and High Water Mark
OpenEdge DB only (no Oracle allowed)
If the site was not a BravePoint
customer, they needed to be willing to
share various statistics
6


Dan Foreman
Note about Single DB Requirement

Two V9 sites previously had much
larger DBs but split their VLDB into
multiple DBs for performance reasons
Reads per Second beyond a certain
point would not improve for a single
DB regardless of spin or B values
The speed limit is because certain
actions need to be serialized - for
example, adjusting the LRU chain when
reading a block
7
Units of Measure
kb
mb
gb
tb
pb
eb
=
=
=
=
=
=
Kilobytes
Megabytes (1000kb)
Gigabytes (1000mb)
Terabytes (1000gb)
Petabytes (1000tb)
Exabytes (1000pb)
8
Progress History DB Size LImits

V8
64gb
256gb
1k DB Block Size
8k DB Block Size
V9
Maximum Areas: 1,000 (some are reserved)
Area Size: 1k Blk Size & 256 RPB = 8gb
Area Size: 8k Blk Size & 1 RPB = 16tb
995 Areas * 16tb = 15,920tb = 16pb
9


Dan Foreman
Progress History DB Size Limits

OpenEdge 10
32,000 Areas (starting in OE 10.1B)
Max File Size of 1tb * 1024 Extents = 1pb
1pb * 32000 Areas approx 32eb
10
Some Progress Limits - Fragments

A record can be split into two or more
fragments
Each fragment has a ROWID address
V9 and V10.0* to V10.1A
2 billion fragments per Area
OpenEdge 10.1B
9,223,372,036,854,775,807 (9 quintillion
for 1k DB block & 256 RPB)
A mere 1 or 2.5 quintillion for 4k/8k DB
block sizes
11
Database Sizes
Site
Version
HWM
Allocated
Anonymous 9.1E02
1.7 TB
2.0 TB
AHM
9.1E0412
743GB
941GB
Wachovia
9.1E0409
359GB
400GB
Broder
10.1A0205 290GB
339GB
Toll (OZ)
10.0B0520 123GB
183GB
12


Dan Foreman
Largest Table
Site
Records
Size
Anonymous 825 million 139gb
Same
Table?
yes
Wachovia
202 million 39gb
yes
AHM
431 million 67gb
no
Broder
Toll (OZ)
81 million
147m
no
no
31gb
18gb
13
Progress Version
Three were on V9.1E*
One was R10.1A SP02 05
One was R10.0B SP05 20
All sites were 6464-bit Progress except
the Anonymous site due to a
disagreement
disagreement about licensing
14
Server Demographics
Site
Server
Anon
IBM P590 AIX 5.2
AHM
Fujitsu
M850
Wachovia Sun
V1280
Broder
IBM P590
Toll
OS
RAM
CPU
64gb
32
Solaris 10 64gb
16
Solaris
2.8
32gb
AIX 5.3
32gb
16
Sun V890 Solaris 10 32gb
8
15


Dan Foreman
Server Demographics
Where is Windows?
One site was migrating from Tru64 to
Linux the day this presentation was
due
One of our HP/UX customers
submitted but they left out too many
details to be included in this
presentation
16
Disk Array Demographics

IBM DS8300; 8tb
IBM DS4800; 8tb; 100 disks
EMC DMX; 1tb; 128 disks
EMC CX700 unknown (under control of
corporate administrators)
HP XP1024 (Hitachi)
Sun StorEdge 9990 (aka HDS 9990)
12 GB Cache
Disks are 146GB/15K FC
Total about 3TB
17
Disk Array Demographics

Bottom line: No JBOD is being used
in the VLDB world
18


Dan Foreman
Concurrent Database Connections

Anon:
472 (355 AppSrv;

AppSrv; 115 W/S)
4203 actual users
AHM:
2999 (.lic
(.lic))
Toll:
1864 (.lic
(.lic))
Broder:
1163 (.lic
Broder:
(.lic))
Wachovia: 389 (.lic
(.lic))
19
Monitoring Tools
ProMonitor
Fathom Management
Homegrown
ProTop
20
Daily Growth Rates

Site
Approximate Growth per Day
AHM
600mb
Toll
280mb
Wachovia
250mb
Broder
TBA
21


Dan Foreman
Capacity Planning
No site had a formal capacity planning
process
22
Capacity Planning Tools - CPU

sar
nmon (AIX)
Adrian Performance Monitor (Solaris)
User
Users Scream
Scream
23
Capacity Planning Tools - RAM

vmstat
nmon (AIX)
24


Dan Foreman
Capacity Planning Tools - Disk

iostat
SAN Vendor
Vendors tools
25
Capacity Planning Tools DB Growth

Area Status (_areastatus
(_areastatus)) Reports
dbanalys + Excel
Fathom Management
ProMonitor
26
Number of Dedicated DBAs

Lowest:
Highest:
.3 (i.e. 30% of 1 person)

2
27


Dan Foreman
Backup Methods
probkup online
probkup online to disk
Full each Sunday
Incremental for the rest of the week
proquiet + Snap Copy

Shutdown, Snap Copy, Restart
Customer was uncomfortable with hot
backup
28
Backups - Wachovia
Full B/U once a week (5hrs)
Two Incrementals Daily (3hrs each)
These are a backstop to the occasional
AI glitch
Warm Spare restarted weekly

AI files are shipped hourly
29
Database Replication
Hot: Fathom Replication (2)
Warm: After Imaging (2)
Cool: Restore from Snap Copy (2)
30

10

Dan Foreman
Replication Issues
Getting Fathom Replication to
integrate smoothly with Veritas
Cluster Server
Long Redo Phase Bug
31
Maintenance Windows
Anonymous
Every Sunday 0300-0400
Everything else is negotiated
Toll
Weekly DB restart at 0400 on Sunday
morning for 15 minutes
Every 6-8 weeks for 6 hours (only if there
is a valid reason which is normally DB
or Application Software Upgrade)
32
Maintenance Windows
AHM
Once a week 2200-0200
Quarterly for Dump/Load
Wachovia
Weekdays 1700-2300
Weekends by negotiation
Broder
5 minutes every night
33

11

Dan Foreman
Tuning
The recommendation of 10000 * (# of
CPUs) for setting spin does not scale
well
Generally spin should be under 50000
34
Anonymous
Tuning
-n 5000
-bibufs 1000 -aibufs 1500
-aistall
-Mn 70 -Mi 3 -Ma 10 -Mpb 50
-L 1500000
-B 160000 (* 8k = 1.28gb)
-t -T /s430/temp/batch
-spin 4000
-tablerangesize 800 -indexrangesize 2000
-semsets 32
-directio
-pinshm
-napmax 500
-Bpmax 40000
35
Tuning
AHM Startup
-B 2000000 (* 8k blocks = 16gb)
-spin 25000
-L 100000
-N tcp -S ntunifipc2
-Mn 1000 -Mi 1 -Ma 50 -Mpb 2 -n 3000
-Mxs 70
-aistall bistall
-tablerangesize 525 -indexrangesize 1000
-semsets 128
-ServerType Both
-DBService replserv -pica 8192
36

12

Dan Foreman
Tuning
Broder
-S liv1
-L 400000
-B 200000
-c 100
-spin 30000
-n 2000
-bibufs 25
-aibufs 25
-directio
-semsets 5
-tablerangesize 700
-Mn 21 -Mi 2 -Mpb 10
-indexrangesize 1600
37
Wachovia
Tuning
-B 2883584
-bithold 2048
-aistall
-n
500
-N
TCP
-L
30000
-Ma 6
-Mn 300
-Mpb 50
-spin 160000
-semsets 30
-Bpmax 2048
-Mxs 65536
-indexrangesize 2500
-tablerangesize 1000
-SQLTempDisk 4000000 -SQLTempPgSize 20
38
Tuning
Toll
-B 600000
-L 1000000
-n 2200
-spin 160000
-aibufs 200 -bibufs 120
-aistall
-bistall
-bithold 8000
-tablerangesize 700
-indexrangesize 700
-Mn 200
-Mi 5 -Ma 10
-semsets 10
-pinshm
-napmax 500
-directio
39

13

Dan Foreman
Storage Areas
Anonymous
560 Storage Areas (Table/Index)
No fixed extents
Large files enabled (of course)
8k DB Block Size
RPB
256 for smaller Areas
64 for large Areas
1 for index Areas
40
Dump/Load
Broder:
Broder:
AHM:
Wachovia:
Anonymous:
Annual Pro D&L

Quarterly Binary D&L
Are you kidding?
kidding?
D&L by Area or None
41
Dump/Load - AHM
Typical outage duration is 2424-36 hours
Do not dump the entire database. Find
the worst offenders by table
(fragmentation or scatter, or a
combination), and then full dump and
load the storage areas that contain
the majority of these worst offenders
The intention is to select a subset of
the DB that can be completed within
the outage window
42

14

Dan Foreman
Dump/Load - AHM
Fathom Replication needs to be
resync
resyncd which means a full backup
and restore to a separate machine
With a 743gb DB, that's a 12 hour
backup, and a 1414-16 hour restore
43
Dump/Load - AHM
While loading should be faster than
dumping, the limit of only one load
per storage area to do an index build
inline (build indexes), or do multiple
loads with a full index rebuild after the
loads complete, makes loading the
critical factor in the outage window
Only an area that can be fully dumped
and loaded in the time frame can be
processed
44
Top Challenges
24 hours is not enough
enough
Physical Redo Phase in AI roll
forward. It could take seconds,
minutes or several hours.
hours.
Who else has this problem?
Client Performance visibility/tracking

It's impossible to add indexes to
some tables
tables
45

15

Dan Foreman
Top Challenges
Knowing more about what a program
is doing would be HUGE help. The
last X number of DB (access)
statements executed would be nice.
Like the SQL query plan, but hopefully
more comprehensible!
comprehensible!
46
Wish List
What program is a Client running (#1)
Backup by Area
Anonymous probkup is 14-16 hours
Table Partitioning (AKA Horizontal

Partitions)
Online SQL Permissions Changes
Online dump/load
Change DB parameters on line
47
Summary
Don
Dont be afraid of OpenEdge VLDBs
especially with OpenEdge 10.1B
VLDB on Windows might be a lonely
place
48

16

Dan Foreman
Questions
49
Conclusion
Thank you for coming!
Don
Dont forget your evaluations (both
good and bad)
50

17

Progress Database Administration

Cargado por

Información del documento

Título original

Derechos de autor

Formatos disponibles

Compartir este documento

Compartir o incrustar documentos

Opciones para compartir

¿Le pareció útil este documento?

¿Este contenido es inapropiado?

Copyright:

Formatos disponibles

Progress Database Administration

Cargado por

Copyright:

Formatos disponibles

DB-13: OpenEdge VLDB

DB-13: OpenEdge VLDB

Introduction - Dan Foreman

Progress Exchange 2007

DB-13: OpenEdge VLDB

Progress Exchange 2007

DB-13: OpenEdge VLDB

Note about Single DB Requirement

Progress History DB Size LImits

Progress Exchange 2007

DB-13: OpenEdge VLDB

Progress History DB Size Limits

Some Progress Limits - Fragments

Progress Exchange 2007

DB-13: OpenEdge VLDB

Anonymous 825 million 139gb

202 million 39gb

431 million 67gb

IBM P590 AIX 5.2

Sun V890 Solaris 10 32gb

DB-13: OpenEdge VLDB

Progress Exchange 2007

DB-13: OpenEdge VLDB

Disk Array Demographics

Disk Array Demographics

Progress Exchange 2007

DB-13: OpenEdge VLDB

Concurrent Database Connections

472 (355 AppSrv;

Daily Growth Rates

Approximate Growth per Day

Progress Exchange 2007

DB-13: OpenEdge VLDB

Capacity Planning Tools - CPU

Capacity Planning Tools - RAM

Progress Exchange 2007

DB-13: OpenEdge VLDB

Capacity Planning Tools - Disk

Capacity Planning Tools DB Growth

Number of Dedicated DBAs

.3 (i.e. 30% of 1 person)

Progress Exchange 2007

DB-13: OpenEdge VLDB

proquiet + Snap Copy

Warm Spare restarted weekly

Progress Exchange 2007

DB-13: OpenEdge VLDB

Progress Exchange 2007

DB-13: OpenEdge VLDB

DB-13: OpenEdge VLDB

Progress Exchange 2007

DB-13: OpenEdge VLDB

Progress Exchange 2007

DB-13: OpenEdge VLDB

Annual Pro D&L

Progress Exchange 2007

DB-13: OpenEdge VLDB

Client Performance visibility/tracking

Progress Exchange 2007

DB-13: OpenEdge VLDB

Table Partitioning (AKA Horizontal

Progress Exchange 2007

DB-13: OpenEdge VLDB