Migrating to Oracle 8i in the Real World: A Senior DBAs

E-business Experience
Roger Schrag
Database Specialists, Inc.
Abstract
Have you seen the Oracle demo where a member of the audience is called up on stage to click the mouse button that
launches an Oracle 8 migration? While salespeople tell you how easy it is to migrate your entire enterprise to the
latest Oracle release, a live migration takes place before your very eyes !f you have been an Oracle "#$ in the real
world, you might be %ust a little bit skeptical Rigorous planning and testing are critical in order to minimi&e risk
and down time, and the actual migration process itself is complicated and error'prone !n this presentation we will
look at the real life story of how a high'profile, high'traffic internet e'business migrated all of its databases from
Oracle () to Oracle 8i release 8*+ and 8*, on Solaris -his company was getting ,. million hits per day, and
had database tables over /+ 0b in si&e -his very technical session will cover the strategy used, the steps followed,
and the pitfalls encountered 1S2 -here was more to it than one mouse click
ntroduction
!n *33, a certain dot'com company developed a family of popular web'based services, using Oracle () databases
running on Solaris on the backend to manage membership, e'commerce transactions, and customer data -he dot'
com became successful by all accounts, and by mid *333 their web site was topping ,. million hits per day !4ll
refer to this company as 5$cme6 from here on out in order to respect their privacy
$cme brought me on board in October of *333 to migrate all databases across their organi&ation from Oracle ()7
to Oracle 8i release 8*+ 8nterprise 8dition9five production databases and a do&en development databases in all
-he databases were to be migrated one at a time in order to make the change as gradual as possible
! developed a detailed plan for the pro%ect, along with precise specifications as to how each database was to be
migrated in a test environment, validated, and then migrated for real -he three smallest production databases and
their corresponding development environments were to be migrated first, using the e:port;import method -he two
larger production databases and their matched development databases were to be migrated last, using the command'
line migration utility
<nfortunately, a bug in Oracle 8i release 8*+ went undetected in the testing environment and $cme4s live web site
e:perienced instability after one of the production databases was migrated to Oracle 8i $cme immediately
slammed the brakes on the enterprise'wide migration effort <ltimately we went to Oracle 8i release 8*, to get
stability $s of the writing of this paper, only the three smaller production databases at $cme have been migrated to
Oracle 8i $ lack of confidence in Oracle 8i has caused $cme to lower the priority on completing the migration
effort
!n the ne:t two sections of this paper ! will outline the high level migration plan ! developed for $cme, and ! will
walk through a detailed checklist you might use to migrate your databases !n the last section, !4ll share a laundry
list of pitfalls and problems we encountered at $cme =rom this paper ! hope you will get ideas for how to plan
your migration, and perhaps you will benefit from $cme4s e:periences when dodging problems along the way
1age *
! need to point out that technology is a fast'moving target When $cme began migrating to Oracle 8i, the current
release was 8*+ 1artway through the pro%ect, 8*, became available 1erhaps an even later release will be
available by the time you read this paper 1lease keep in mind that all of the information presented here is
applicable to Oracle 8i release 8*+ 8nterprise 8dition on Sun S1$R> Solaris Some things might be different
with future releases or other platforms
Ac!es Migration Strateg"
$cme needed to minimi&e the risk of disruption caused by the migration, even if this meant dragging out the pro%ect
over a period of months $cme wanted to migrate production databases one at a time, allowing a few weeks
between each "evelopment databases would be migrated immediately after the production database that they
mirrored #efore a production database would be migrated, it would be copied to a test server where the migration
procedure could be validated and all affected applications could be regression tested against Oracle 8i
Having settled on a gradual approach, the ne:t step was to choose the order in which to migrate the production
databases and the migration method to use for each ! chose to migrate the smallest databases first because in a
catastrophic failure situation, smaller databases are faster to recover from a backup -he smallest databases were
small enough that an up'to'date backup could be kept available uncompressed on local disk, should a recovery be
necessary -he larger databases did not allow this lu:ury, so ! wanted to start small to build up $cme4s confidence
in the stability of Oracle 8i and the migration process
$s is the case at many dot'com companies, most of the Oracle databases at $cme had originally been set up by
developers pinch'hitting for "#$s Some of the databases had problems, such as a suboptimal character set or a
small block si&e, that could only be fi:ed by rebuilding the database Other problems, such as severe fragmentation
and poor segment tablespace assignments, could only be fi:ed by taking tablespaces offline and reorgani&ing them
! saw the migration pro%ect as an opportunity to correct many of the mistakes that had been made in the past #y
using the e:port;import method to migrate the smaller databases to Oracle 8i, ! could switch these databases to a
more appropriate character set and block si&e, as well as institute a uniform e:tent si&ing approach to eliminate free
space fragmentation -he e:port;import method also has the added feature that the migration could be aborted
without having to restore the original database from a backup
-he two larger production databases were too large for the e:port;import migration method to be feasible ?-he
down time would have been measured in days@ ! planned to benchmark the migration process using the
e:port;import method for the three smaller databases !f the reAuired down time for each database was acceptable
to $cme, then ! would proceed with the e:port;import method for these databases Otherwise, !4d use one of the
other migration methods
$side from the e:port;import method, ! considered using the command'line migration utility 5mig6 and the "ata
Bigration $ssistant 0<! tool Having used the "atabase >onfiguration $ssistant 0<! tool that comes with Oracle
8i, ! immediately ruled out the "ata Bigration $ssistant as a viable option ! did not feel comfortable betting
$cme4s business on a 0<! tool that might crash, suppress error messages ! should be aware of, or otherwise do the
wrong thing
#y being clever and preprocessing as much work as possible, ! found that ! could migrate each of the smaller
databases using the e:port;import method with %ust under one hour of down time $cme was able to live with this
timeframe
So, $cme4s migration strategy from a ).,... foot view was to migrate production databases one at a time smallest
to largest, the smallest ones by the e:port;import method and the largest ones by the command'line migration
utility
1age /
#he Migration Steps
!n this section we will look at the steps reAuired to migrate an entire enterprise to Oracle 8i =irst we4ll look at the
phases of the migration9the high level steps !n the latter part of this section we4ll walk through detailed steps for
how you migrate individual databases to Oracle 8i We4ll look at the steps using both the e:port;import method and
the command'line migration utility -he information presented here is based on my e:perience migrating $cme4s
databases to Oracle 8i
$hases o% the Migration
-he $cme migration consisted of the following phases2
* -horoughly document the migration plan, including detailed database migration steps
/ >opy one production database to a test environment and migrate, following the documented steps
) Regression test all applications against the migrated database
7 Bigrate the production database
+ Resync development and test copies of the production database
, Bonitor the system for a few weeks for problems caused by the migration
( Repeat the above steps for each production database
Bigrating a database to Oracle 8i is complicated and involves many steps that must be done e:actly right !
developed thorough documentation for $cme detailing how to carry out the migration of each database ! validated
the documentation by migrating a copy of each database in a test environment, following the documentation to the
letter !f something was stated incorrectly or if ! determined that a step was missing from the plan, ! updated the
documentation immediately
Bigrating a copy of the database in a test environment allowed me to practice the migration, determine how much
down time would be reAuired, and avoid nasty surprises when migrating the live database -his also allowed the
Auality assurance group to test the applications that interact with the database in order to make sure everything
would work the same after the database had been migrated to Oracle 8i
$s soon as a production database was migrated to Oracle 8i, ! resynchroni&ed all development and test copies of
that database with production and thereby brought them up to Oracle 8i -his ensured that applications were being
coded and tested against the same version of Oracle used in production
! planned to take a three week breather between production database migrations -his gave $cme time to watch for
stability and compatibility problems, and it gave me time to prepare for the ne:t migration Since ! was also the
lead production support "#$ for $cme, the hiatus between migrations allowed me to catch up on my other duties
Migration &sing the Export'!port Method
-o migrate an Oracle () database to Oracle 8i using the e:port;import method, you first follow these preparation
steps2
* !nstall the Oracle 8i software on the database server Cou should develop a procedure for Oracle software
installation and follow it rigorously in order to standardi&e the Oracle setup on all of your database servers
Cou don4t need to have Oracle create a starter database for you at this point !f you choose to, however, take
precaution to ensure that Oracle does not start up an Oracle 8i Det8 listener on the same port as your e:isting
SEFGDet listener
1age )
/ <se the "atabase >onfiguration $ssistant to create scripts for creating a new database Cou could %ust go with
the starter database if you wish, but ! found the starter database to be poorly configured ! found it more
practical to edit the scripts created by the "atabase >onfiguration $ssistant once and then use these scripts
repeatedly for each database ! needed to create -his leads to less tedium and more consistency across
databases Here are some of the configurations you4ll need to edit2
Set the database name and global name appropriately
Set an appropriate database block si&e
Set the desired character set and national character set
Si&e and configure the rollback segments and temporary tablespace appropriately
Si&e and relocate the online redo logs optimally
Si&e and relocate the standard tablespaces optimally
>reate the application tablespaces
Set the passwords and temporary tablespace designations for all default users
) Run the scripts to create your new Oracle 8i database !f you will be creating the Oracle 8i database on the
same database server where the current Oracle () database resides, then give it a different name from the
e:isting Oracle () database $dd an entry to the oratab file for this new database
7 >opy the parameter files from your Oracle () database to the new Oracle 8i database 8dit as reAuired to
account for different path names, different instance and database names, and the fact that certain parameters
have changed between Oracle () and 8i Here are some parameters that are obsolete in Oracle 8i that you
should watch for2
distributedHlockHtimeout
%obHAueueHkeepHconnections
seAuenceHcacheHentries
seAuenceHcacheHhashHbuckets
snapshotHrefreshHinterval
snapshotHrefreshHkeepHconnections
snapshotHrefreshHprocesses
unlimitedHrollbackHsegments
+ !f you use tnsnamesora files, then copy an up'to'date tnsnamesora file to the network;admin directory in the
new Oracle 8i home Replace the tnsnamesora files in other Oracle homes with a link to the file in the Oracle 8i
home -his keeps all tnsnamesora files in sync with each other $lso, add a temporary entry for the new Oracle
8i database
, >onfigure the Oracle 8i Det8 listener to listen for all databases on the server Shut down the SEFGDet listener
and start up the Oracle 8i Det8 listener >onfirm connectivity to all databases on the server
( !f you have customi&ed the environment scripts oraenv and coraenv, then copy these customi&ations to the bin
directory in the Oracle 8i home !f you have e:ecutables linked with Oracle () libraries that will need to access
an Oracle 8i database from an Oracle 8i home, then ensure that your OR$HDFS)/ environment variable is set
to point to the DFS data files area of your Oracle () home ?We4ll talk more about this in a later section of the
paper@
8 Cour database server now has multiple versions of Oracle software installed on it !t is important to use oraenv
and coraenv9and not simply hard code environment variables9in order to ensure that all necessary
environment variables are set correctly when switching between Oracle homes
1age 7
3 -est connectivity to your new Oracle 8i database Cou should be able to connect to it without using networking
?by setting OR$>F8HS!"@ and also through Det8 from the Oracle 8i home and through SEFGDet from the
Oracle () home
*. 1repare a script to take a direct path e:port of the Oracle () database Cou want this to run as Auickly as
possible, so you may wish to compare performance writing to different physical disk devices, and you may
wish to e:periment with writing directly into a compress or g&ip process via a named pipe Dote the elapsed
time and disk space reAuired for the e:port
** !f you are unhappy with the tablespace assignments or e:tent si&es of e:isting ob%ects in your Oracle ()
database, then you can reorgani&e your database while migrating it !f you wish to do this, then prepare a
SEFG1lus script to run on the Oracle 8i database that will precreate tables and the users that own them Cou
should not precreate inde:es or constraints, as these will dramatically slow down the import process and lead to
suboptimal inde:es !nstead, you should let the inde:es and constraints import into the wrong place and use the
$F-8R !D"8I R8#<!F" statement later to improve them
*/ 1repare a script to import the Oracle () e:port file into the Oracle 8i database <nless your database is very
small, you should use the commitJy import option !f you will be precreating tables to improve placement or
e:tent si&es, be sure to specify the ignoreJy option as well
*) Run the Oracle () e:port script, Oracle 8i ob%ect creation script ?if you prepared one@, and the Oracle 8i
import script Dote the e:ecution time of each -roubleshoot any errors that occur !f error messages do occur
and are e:pected, then document them so that you will know to e:pect them when performing the live
migration
*7 !f you have access to a tool that can compare two database schemas, then run a comparison between your
Oracle () and Oracle 8i databases to make sure no ob%ects were lost in the e:port;import process
*+ 8:amine the system, role, and ob%ect'level privileges of your database users to make sure they were preserved
correctly when importing into the Oracle 8i database -his is especially important if you manually created users
in your Oracle 8i database so that you could precreate tables before importing
*, !f you are using a character set other than <S($S>!!, look at your data in the Oracle 8i database to make sure
no accidental character set translation has occurred !f you have multi'byte or 8'bit characters in your data,
then Auery them from the Oracle 8i database to make sure they were not garbled in the e:port;import process
*( Cou are now ready to regression test your applications against the Oracle 8i database !deally, you should
perform an e:haustive test of every module in your application $t the very least, it is imperative that you test a
sample of application modules that covers every application technology that must access the database -est
your backup procedures and monitoring infrastructure against the Oracle 8i database as well "o not move
beyond this step until all testing is complete and any discovered issues have been fully resolved
*8 !f the database to be migrated to Oracle 8i is involved in replication in any way, you will need to evaluate how
the migration will impact the replication process and your other databases =or e:ample, if the database you are
migrating contains masters for fast'refresh snapshots based on ROW!"s, then the migration will invalidate the
snapshot logs and you will need to perform a complete refresh ?!f the database containing the snapshot is an
Oracle 8. or 8i database, you may wish to switch to primary key snapshots at this time@
*3 Review the benchmark times and finali&e the plan for the live cutover
/. Scrub the Oracle 8i database clean by dropping all application users and their schema contents
/* !f you chose to precreate tables in order to relocate them or change their e:tent si&es, then run the script you
prepared to do so
1age +
// 1repare a script that will rename the Oracle 8i database and all of its data files and online redo logs so that their
path names match the eventual name of the database Cou will use this script during the live cutover when you
rename the Oracle 8i database to have the same name that the Oracle () database used to have -he easiest
way to prepare this script is to perform an $F-8R "$-$#$S8 #$>K<1 >OD-ROF=!F8 -O -R$>8 on
the Oracle 8i database and edit the script Oracle generates for you
/) Cou may wish to temporarily set the sortHareaHsi&e on the Oracle 8i instance to a very large number in order to
speed up inde: creation during the live cutover
$t this point you are prepared to migrate the Oracle () database to Oracle 8i 5for real6 -he steps for the live
cutover are as follows2
* Shut down all applications that access the Oracle () database
/ <se your previously created and tested scripts to e:port the Oracle () database and import the data into the
Oracle 8i database Review the output from both the e:port and import processes9there should be no error
messages other than those documented during the test process
) Right after launching the import, shut down the Oracle () database with normal priority and rename all of the
directories that hold its control files, online redo logs, archived redo logs, and data files $lso rename the Oracle
() database4s admin area ?where the background dump destination, user dump destination, etc are@
7 While the import is running, edit your tnsnamesora file to remove the temporary entry you added to reference
the Oracle 8i database
+ $fter the import is well underway, connect to the Oracle 8i database from SEFG1lus as an application user
?not as SCS-8B@ !f you get a product profile error, then rerun the pupbldsAl script as the SCS-8B user
-his script is located in the sAlplus;admin directory under the Oracle 8i home
, $fter the import is complete and you have checked for errors, rename the Oracle 8i database and instance to
take on the name previously used by the Oracle () database -his is done by shutting down the Oracle 8i
database with normal priority, renaming all of the directories used by the Oracle 8i database to take on the new
name, changing the OR$>F8HS!" in your environment, and applying the database rename script you created
during the preparation phase Bake sure to fi: the link to the Oracle 8i parameter file in the dbs directory of the
Oracle 8i home
( Switch the Oracle 8i database to archivelog mode, if appropriate $lso, return the sortHareaHsi&e to its normal
setting if you had temporarily increased it
8 8dit both the oratab file on the database server and the listenerora file in the Oracle 8i network;admin directory
to show that your database is now using the Oracle 8i home instead of the Oracle () home $lso, delete the
temporary entry you had created in each file during the preparation phase
3 Restart the Oracle 8i Det8 listener and confirm that you can access the database via Det8 both locally on the
database server and on clients or application servers elsewhere on the network
*. $t this point the migrated database is available for application use However, there are still a few housekeeping
details to take care of
** !f the migrated database is involved in replication, then carry out the plan you devised during the preparation
phase in order to resynchroni&e data between the migrated database and other databases in the enterprise
*/ !f during the preparation phase you determined that the migration would break any of your backup procedures
or monitoring infrastructure, then carry out the plan you devised to fi: these processes
1age ,
Migration &sing the (o!!and-line &tilit" )!ig*
-o migrate an Oracle () database to Oracle 8i using mig, you first follow these preparation steps2
* 1repare a test database server that has Oracle () software and a working, current copy of the Oracle ()
database to be migrated -he test database server should be set up as similarly as possible to the production
database server >reate the copy of the Oracle () database by restoring a hot or cold backup onto the test
database server
/ 1erform some Auick application tests to validate the copy of the Oracle () database on the test server
) !nstall the Oracle 8i software on the test server Cou should develop a procedure for Oracle software
installation and follow it rigorously in order to standardi&e the Oracle setup on all of your database servers
Cou don4t need to have Oracle create a starter database, but there should be no harm done if you do
7 8nsure that the SCS-8B tablespace on the Oracle () database has enough free space Oracle 8i reAuires a lot
more space for the data dictionary than Oracle () did -he Oracle 8i migration guide states that the SCS-8B
tablespace should be at least two'thirds empty so that it can accommodate a temporary copy of the data
dictionary that is twice the si&e of the current data dictionary !n reality you may need even more space than
that !f you plan to use Oracle 8i bells and whistles like LServer, your SCS-8B tablespace should be at least
/.. Bb in si&e to be safe
+ Bake sure that none of the data files are in backup mode or need recovery Cou can check this by Auerying
vMbackup and vMrecoverHfile ?-he first should show all data files have status DO- $>-!N8, and the second
should show no rows@ Dote that the SCS-8B and rollback segment tablespaces must be online, but that the
migration guide says other tablespaces may be offline for the migration9as long as they were taken offline
with normal priority
, 8nsure that the SCS-8B rollback segment has no O1-!B$F setting and large enough D8I- and
B$I8I-8D-S settings to allow it to grow to a few Bb in si&e Cou can check these settings with the
following Auery2
SELECT A.next_extent, A.max_extents, B.optsize
FROM SYS.dba_rollback_segs A, v$rollstat B
WHERE A.segment_name = 'SYSTEM'
AND B.usn = A.segment_id;
( Nerify that the SCS-8B userOs default tablespace is what you want it to be ?typically -OOFS or <S8RS@ and
that the default storage for that tablespace is what you want it to be $t the end of migration, many new ob%ects
will be created in the SCS-8B schema using the default tablespace for the SCS-8B user and the default
storage for that tablespace Cou may want to temporarily set the default storage for the SCS-8B userOs default
tablespace to be the same as the SCS-8B tablespace, but this is up to you
8 8nsure that the SCS-8B userOs default tablespace has sufficient space available to hold many new schema
ob%ects 8:pect a few do&en tables, inde:es, FO#s, and FO# inde:es to be created in the SCS-8B schema
using the SCS-8B user4s default tablespace and that tablespace4s default storage
3 -hroughout the migration process you will need to bounce your environment between the Oracle () and Oracle
8i homes 1repare a script that sets your environment to each Oracle home -he script must take care of setting
OR$>F8HHOB8, 1$-H, OR$HDFS)) ?if you use a character set other than <S($S>!!@, and
F"HF!#R$RCH1$-H !t is very important that when you switch from one environment to the other, the bin
directory from the old Oracle home does not remain on your path
1age (
*. Fog in as the Oracle software owner and set your environment to point to the Oracle 8i home <se the migprep
utility to copy Oracle 8i migration e:ecutables to your Oracle () home -his operation runs Auickly, and
reAuires appro:imately *+ Bb of space in your Oracle () home -he command line is as follows2
migprep <8i home> <7.3 home>
** Set your environment to point to the Oracle () home and set OR$HDFS)) to point to migrate;nls;admin;data
under the Oracle () home-hen shut down the database with normal priority
*/ Cou may wish to run the migration utility in 5check only6 mode to verify that the SCS-8B tablespace has
enough free space in it -o do this, make sure your environment is pointing to the Oracle () home and run2
mig CHECK_ONLY=TRUE
-his will spew a bunch of SEF on the screen, with an estimate of SCS-8B space reAuired at the end mig
might e:it with a non'&ero e:it code On <ni: systems this usually indicates an error condition, but should be
ignored here -he mig utility might also leave the database open !f so, shut it down again with normal priority
*) With the environment still pointing to the Oracle () home, run the migration utility for real with the command2
mig DBNAME=<name> NEW_DBNAME=<name> PFILE=\"<path>\" SPOOL=\"<path>\"
Dote that the backslashes before the Auotes are reAuired -he path provided for the 1=!F8 parameter should be
the full path and filename of the parameter file used by the Oracle () instance to open the database -his
operation takes only a few minutes, and generates a file called convS!"dbf in the dbs directory under the
Oracle () home $ huge amount of output is written to the screen, but all of it appears to be captured in the
spool file as well
*7 "o not open the database again using the Oracle () home !f you do, you4ll need to run the mig utility again to
generate a new convS!"dbf file
*+ Review the spool file generated by the mig utility -here should be no errors Cou can perform case'insensitive
searches for 5ora6 and 5err6 to help you scan the file
*, Cou should take a cold backup of the database at this time -his will allow you to restore the database and try
again if the migration fails further down the line ?"ata files for offline tablespaces should not need to be
restored from the backup, because the migration process should not affect them@
*( Set your environment to point to the Oracle 8i home 8dit the oratab file to show that the database now has
Oracle 8i as its home
*8 Rename the e:isting database control files -he migration process will create new control files with the same
names as the old control files #y renaming the old control files first, you will not lose them when the new
control files are created
*3 >opy the convS!"dbf file created by the mig utility from the dbs directory in the Oracle () home to the dbs
directory in the Oracle 8i home
/. >reate a symbolic link from the dbs directory of the Oracle 8i home to the parameter file for the database
/* 8dit the parameter file and set compatibility to 8*+ or 8*, ?depending on which release of Oracle 8i you are
migrating to@ -emporarily set %obHAueueHprocesses to &ero
// !f there are any Auestion marks in the parameter files, replace them with the full path of the Oracle 8i home
?Remember to check the included parameter file if the ifile parameter is set in the main parameter file@
1age 8
/) >omment out all parameters in the parameter files that are obsolete in Oracle 8i Here are some to watch for2
distributedHlockHtimeout
%obHAueueHkeepHconnections
seAuenceHcacheHentries
seAuenceHcacheHhashHbuckets
snapshotHrefreshHinterval
snapshotHrefreshHkeepHconnections
snapshotHrefreshHprocesses
unlimitedHrollbackHsegments
/7 >onfirm that all environment variables point to the Oracle 8i home 0o to the rdbms;admin directory in the
Oracle 8i home and run the following commands in Server Banager ?svrmgrl@
SPOOL <path>
CONNECT INTERNAL
STARTUP NOMOUNT
!f you get errors when trying to start the instance, then e:amine the parameter files again for parameters that
are obsolete in Oracle 8i
/+ >ontinuing on in the same Server Banager session, migrate the database to Oracle 8i with the following
commands2
ALTER DATABASE CONVERT;
ALTER DATABASE OPEN RESETLOGS;
SET ECHO ON
@u0703040.sql
@utlrp.sql
SHUTDOWN NORMAL
SPOOL OFF
EXIT
-he two $F-8R "$-$#$S8 commands should only take a few seconds -he u(.).7.sAl script rebuilds the
data dictionary catalog views and could take roughly half an hour to run -he utlrpsAl script simply recompiles
all invalid stored 1F;SEF ob%ects -his is handy because the migration process invalidates everything
/, Review the spool output from Server Banager for errors -his log file could be over *. Bb in si&e, so it is not
practical to read it line by line Cou might want to search for lines that begin with 5OR$6 in order to look for
error messages Cou should make sure there is a reason for every error message -his can be difficult %ust
because of the sheer number of messages Cou might be tempted to take it on faith that the migration was
successful, but at least a Auick review of the spool file is strongly recommended Here are some observations2
OR$'.*7)/ and OR$'.*7)7 errors are harmless9a synonym to be dropped did not e:ist
Running the u.(.).7.sAl script a second time will lead to a rash of errors from operations that cannot be
performed more than once -he following are not a cause for concern2
a@ OR$'....* when inserting seed data that already e:ists into dictionary tables
b@ OR$'..3++ when adding tables and inde:es to the data dictionary again
c@ OR$'.*7). when adding columns that have already been added to dictionary tables
d@ OR$'.*77/ when making a column not null that is already not null
e@ OR$'.*7+* when making a column nullable that is already nullable
f@ OR$'.*3*8 when dropping the already'dropped B!0R$-8 user
g@ OR$'.*3/* when adding a database role that has already been added
h@ OR$'/3+)( when calling dbmsHrmininstall to create ob%ects that have already been created
1age 3
8rrors may occur when migrating an Oracle () database that did not have the $dvanced Replication
Option installed to Oracle 8i 8nterprise 8dition
! encountered OR$'.*()* ?circular view definitions@ errors regarding vMrecoveryHservers and
vMrecoveryHtransactions ! never did find out the cause or ramifications of these errors
/( !f you changed the %obHAueueHprocesses parameter in the parameter file earlier, then put it back to its regular
setting now
/8 !f you changed the SCS-8B userOs default tablespace or the default storage for this tablespace earlier, then
you may now change it back
/3 !f you changed the D8I-, B$I8I-8D-S, or O1-!B$F settings on the SCS-8B rollback segment earlier,
you may wish to change them back Cou may find that during migration B$I8I-8D-S for the SCS-8B
rollback segment was set to unlimited, and this could cause you problems when you try to alter the rollback
segment Cou can correct this by changing B$I8I-8D-S to a smaller number such as */*
). >heck both the oratab file on the server and the listenerora file in the network;admin directory of the Oracle 8i
home to make sure that your database is shown as now using the Oracle 8i home instead of the Oracle ()
home Restart the Oracle 8i Det8 listener if you make any changes to the listenerora file
)* !f you want to use a new Oracle 8i option such as LServer on your migrated database, then you must run the
"atabase >onfiguration $ssistant to create the necessary database ob%ects to make the option usable -he basic
steps to do so are as follows2
Set your environment to point to the Oracle 8i home and launch the "atabase >onfiguration $ssistant with
the dbassist command
>hoose 5Bodify a database6 and click De:t
Select the instance from the list and click De:t
>hoose whether you want the instance to use "edicated Server Bode or Shared Server Bode ?multi'
threaded server@ and click De:t
>hoose any options you would like to install, such as !nterBedia or LServer, and click =inish
Cou may be asked to confirm the location of the instance parameter file >lick OK !f you get a Lava
e:ception about not being able to find a backup copy of the parameter file, ignore it and click OK
)/ Run the utlconstsAl script in the rdbms;admin directory of the Oracle 8i home to see if any check constraints in
the migrated database perform date manipulations that are not tolerated by Oracle 8iOs more stringent rules
)) !f the database contains bitmap inde:es, then you should check for ones that have taken on an <D<S$#F8
status during the migration and rebuild them Cou can check for these inde:es with this Auery2
SELECT index_name, index_type, table_owner, status
FROM dba_indexes
WHERE index_type = 'BITMAP'
AND status = 'UNUSABLE';
)7 >heck your database for new users created during the migration or installation of new options, such as
O<-FD and >-ISCS >hange the passwords for these users appropriately
)+ Cou are now ready to regression test your applications against the Oracle 8i database !deally, you should
perform an e:haustive test of every module in your application $t the very least, it is imperative that you test a
sample of application modules that covers every application technology that must access the database -est
your backup procedures and monitoring infrastructure against the Oracle 8i database as well "o not move
beyond this step until all testing is complete and any discovered issues have been fully resolved
1age *.
), Repeat steps ) through *. above on the database server where the production database to be migrated resides
-his will prepare your database server for the live cutover
$t this point you are prepared to migrate the Oracle () database to Oracle 8i 5for real6 -o perform the live
cutover, continue on from step ** above on the database server where the production database to be migrated
resides
Migration $it%alls Encountered Along the Wa"
-he enterprise'wide Oracle 8i migration at $cme went reasonably well, but it was anything but smooth We
discovered plenty of 5gotchas6 and unpleasantries along the way -hankfully, most of the problems were discovered
during the planning and testing phase of the migration and we were able to resolve them before they could impact
production
<nfortunately, one serious problem did sneak through testing and was only discovered after mission'critical
production databases had been migrated to Oracle 8i ?release 8*+ 8nterprise 8dition on Sun S1$R> Solaris, to be
precise@ $ trivial !DS8R- statement in a stored procedure appeared to tip off a memory leak of sorts within the
S0$ of the database instance
8ach time the !DS8R- statement ran9and it ran a few times per second9the sharable memory in the shared SEF
area attributed to the !DS8R- statement would grow by a few bytes -his would continue on until the !DS8R-
statement had consumed over +. Bb of memory in the shared SEF area 0radually Oracle would start spending
more and more >1< time managing ob%ects in the shared SEF area because less and less space was available for
other statements $t a certain point the database would become unusable because the most basic Aueries would
hang for several minutes waiting on a 5library cache pin6 wait event while Oracle tried to make space in the shared
SEF area
>alls to Oracle Support were useless $fter analy&ing $cme4s bstat;estat reports and trace files for si: days, an
Oracle support analyst sent me email suggesting that we ?*@ make sure sAlHtrace is not turned on for the entire
instance, and ?/@ consider adding an inde: to the table to speed up !DS8R- statements
! kid you not
So $cme e:perienced everybody4s worst upgrade nightmare -hey migrated to Oracle 8i and a production system
that was stable on Oracle () suddenly became unstable Oracle Support did not stand behind the product, and
$cme was left with little choice but to restart Oracle instances every few days Seeing no other option, !
recommended that $cme upgrade to Oracle 8i release 8*, as soon as it became available -his indeed fi:ed the
problem
!n the remainder of this section !4ll point out, in no particular order, some of the migration difficulties ! discovered
during the planning and testing phases at $cme and how ! resolved them
+,S ssues
Oracle changed the format for its DFS data files between version () and 8. of the Oracle client -his means that
if you have an application that was linked with Oracle () libraries, you may have trouble running the application
in an Oracle 8. or Oracle 8i home !f you use the <S($S>!! character set in your database and on your client, this
issue will probably not apply to you
However, if you use any other character set in the database or on the client, you will likely encounter an OR$'
*/(.+ error when you run your application linked with Oracle () libraries from an Oracle 8. or Oracle 8i home
-here are several ways to deal with this problem
1age **
Cou could choose to retain an Oracle () home and have your applications run from this environment, connecting
to your Oracle 8i database via SEFGDet and Det8 -his is probably the easiest solution, but it reAuires that you
keep older versions of Oracle software around and it prevents your applications from leveraging new Oracle
features !f your applications are third'party tools, this approach might be your only option
!f you have the source code to your applications, you can relink them with the Oracle 8i libraries -his will break
the dependency on Oracle () DFS files right away !deally you would enhance your applications to leverage new
Oracle 8i features and O>! calls, but this could be done gradually over time
!f your plan is to ultimately relink all of your applications with Oracle 8i libraries but you are not able to do this
immediately, you can run your applications from the Oracle 8i home, but retain the Oracle () DFS data files and
set the OR$HDFS)/ environment variable to point to the Oracle () DFS files -his tactic allows you to work
from an Oracle 8i home without breaking your older applications -his allows you to relink or enhance your
applications gradually over time
nteroperabilit" ssues
One of the nice features of the Optimal =le:ible $rchitecture ?O=$@ is that if you install Oracle software on your
database server in an O=$ compliant manner, it is easy to run multiple databases on different versions of Oracle !f
you have two Oracle () databases on one server, for e:ample, you have the option of migrating them to Oracle 8i
one at a time
<nfortunately, at $cme ! discovered a few hiccups when trying to have Oracle 8i interoperate with other versions
of Oracle on the same database server
-he dbstart script that comes with both Oracle 8i release 8*+ and 8*, has a bug in it that will cause it to skip
Oracle () databases instead of starting them !f you had a database server with an Oracle (/ database, an Oracle
() database, an Oracle 8. database, and an Oracle 8i database, you would find that Oracle 8i4s dbstart script
would start all of the databases e:cept for the Oracle () database
#asically, there is a case statement in the dbstart shell script that decides whether to use sAldba, svrmgrl, or sAlplus
to start the database !f there is no sAldba e:ecutable present in the bin directory of the Oracle home, dbstart
invokes svrmgrl and looks at the version of 1F;SEF installed !f the version is () or 8., then svrmgrl is used to
start the database !f the version is 8*, then sAlplus is used to start the database !f the version is anything else, then
the database is not started On Oracle () databases, the 1F;SEF version shows as /) ?OopsP@
$t $cme we also discovered a Det8 connectivity problem between Oracle 8i release 8*+ and Oracle 8i release
8*, -ypically when you have multiple versions of Oracle installed on one database server, you run the Det8
listener from the Oracle home of the newest Oracle version !f you run an Oracle 8i release 8*, Det8 listener,
however, you may find that you cannot connect to Oracle 8i release 8*+ databases via Det8
Oracle has documented this behavior in technical support bulletin 3+)38* dated Lanuary /., /... -he work'
around is simple2 "on4t set the F"HF!#R$RCH1$-H environment variable when starting the Det8 listener9
even though the documentation tells you to set F"HF!#R$RCH1$-H to the lib directory under the Oracle home
Execution $lan Stabilit"
Oracle >orporation is constantly refining the algorithms used by the Auery optimi&er, and so it should not be
surprising to find that e:ecution plans on SEF statements might change after migrating a database to Oracle 8i
While some e:ecution plans might change for the better, it is possible that some will change for the worse -his is
one of the reasons that e:haustive application testing before migrating a production database to Oracle 8i is
important
1age */
$t $cme four of the five production databases run with cost based optimi&ation We found that one application
suffered significant performance degradation at the hands of the Oracle 8i cost based optimi&er On Oracle () the
application took 87 minutes to complete, but on Oracle 8i the e:ecution time shot up to 77 hours We nudged the
Oracle 8i optimi&er down the right path by embedding optimi&er hints in the main Auery
Sill" Bugs and Anno"ances
Oracle 8i has a few bugs that are more embarrassing for proponents of Oracle technology than real production
problems =or e:ample, when you perform a 5typical6 Oracle 8i installation, the global name of your starter
database will be 5%ava8usoraclecom6 -his is in spite of the fact that the installer specifically asked you what you
would like the global name for the database to be -his can be fi:ed Auickly with an $F-8R "$-$#$S8
R8D$B8 0FO#$FHD$B8 command
$ potentially more irritating bug occurs when you operate an Oracle 8i database in archivelog mode $pparently a
developer at Oracle added a nice feature to allow "#$s to trace and tune archiver operation -his would be really
great, e:cept that you can4t turn it off Oracle has recogni&ed this as bug number 33)3*7, and it is fi:ed in Oracle
8i release 8*, !f your database generates lots of archived redo logs and you will be migrating to Oracle 8i release
8*+, then you4d better make sure you have lots of disk space free for trace files and a fat alert log
!n the documentation bug department, the vMoption dynamic performance view can be confusing or misleading
vMoption lists Oracle options ?such as partitioning and spatial@ and whether or not they are available What the
documentation doesn4t tell you is that vMoption is only showing you whether or not the necessary Oracle software to
support a given option has been installed in the Oracle home that was used to start the instance9vMoption does not
address whether the necessary schemas and ob%ects have been created in the database to enable the option $fter
migrating an Oracle () database to Oracle 8i, for e:ample, you may see that you have options such as spatial and
Lava available when in fact these options will not work against the database Cou need to run the "atabase
>onfiguration $ssistant to create the necessary database ob%ects before these options can be used
(onclusion
!t is definitely possible to migrate all of the Oracle databases across your enterprise to Oracle 8i ?Of course it isP
Fots of companies have done it@ However, there are many issues to consider and many land mines that you want to
avoid $s when making any significant change to a comple: system, the more planning and testing you perform
beforehand, the better your chance of success
! cannot emphasi&e enough the importance of planning your Oracle 8i migration, testing the plan thoroughly, and
sticking rigidly to the plan when performing the live cutover -his approach helps ensure success in three ways2
=irst, you4ll make sure you know how to perform the migration before you actually touch a production database
Second, you4ll discover Oracle 8i bugs or incompatibilities before they affect your live systems $nd third, all of
the practice will help you carry out the live cutover with as little down time as possible
! hope that the e:periences and tips !4ve shared here will help you plan and carry out your Oracle 8i migration
smoothly and cleanly 0ood luckP
About the Author
Roger Schrag has been an Oracle "#$ and application architect for over twelve years He started out at Oracle
>orporation on the Oracle =inancials development team and moved into the roles of production "#$ and database
architect at various companies in the San =rancisco #ay $rea Roger is a freAuent speaker at Oracle OpenWorld
and the !O<0 FiveP conferences He is also vice'president of the Dorthern >alifornia Oracle <sers 0roup !n *33+,
Roger founded "atabase Specialists, !nc, ?http2;;wwwdbspecialistscom@ a consulting firm speciali&ing in business
solutions based on Oracle technology !n addition to consulting, the company offers fle:ible solutions including
1age *)
part'time "#$ support and "atabase R: ?http2;;wwwdbspecialistscom;databaseHr:html@, a web'based
monitoring and alert notification service for Oracle databases !n /..*, the San =rancisco #usiness -imes named
"atabase Specialists one of the -op *+. =astest'0rowing 1rivate >ompanies in the #ay $rea
1age *7