Documentos de Académico
Documentos de Profesional
Documentos de Cultura
WIN-CENOP-TSM Integrated Tape Rotation Date Implemented: 08/07/2006 Date Revised: 09/09/2008
Rev. 2
Suggestions or feedback ctrl + click here
Location(s) this Work Instruction Applies to: This document applies to all SunGard Availability Services locations where data backups occur. Owner (Department): Tape Services Audience: All SunGard Data Center Operations and Tape Services Definitions: TSM= Tivoli Storage Manager Tape Ops= Tape Operations DRM= Disaster Recovery Manager IDC= Internet Data Center PLCI= Pending Logical Check In Other Documentation required: N/A.
Summary: This work instruction outlines the steps required to do daily offsite tape rotations for SunGard Availability Services data centers. Purpose: To provide a standard for tape rotations in all SunGard Availability Services data centers. Records Matrix:
Records Type Completed via Remedy Ticket Responsibility Tape Services Location Managed Services Portal Access All SunGard Staff Filed by: Retention N/A Disposition Close of Remedy Ticket
Page 1 of 21
DETAILS
There are two distinct Tape Rotation processes at SunGard Availability Services data centers. There is a process for the SunGard Availability Services data centers where the Storage Portal is used to track vaulted tapes and a process for SunGard Availability Services data centers where TSM DRM is used to track vaulted tapes. This document will provide work instructions for each distinct process in two sections. At some SunGard Availability Services data centers both processes may be utilized. Tape rotation for SunGard Availability Services sites consists of daily maintenance of the TSM server(s), libraries and Storage Portal. Before beginning the rotation for your assigned location, complete the following to ensure the data center is in good order and a new rotation can be started without any incomplete portions of any previous rotation.
Page 2 of 21
Check for tapes that may have their access mode set to the incorrect status. o Are there any tapes in unavailable status? There may be more than one TSM server for a data center, check all servers for the location. Use q vol access=unavailable to gather this information.
If you find tapes in an access mode of unavailable change the access to the appropriate access mode. o Use q libvol * <volume name> to determine if the tape is in a library. Any tape found in a library that has an access mode of unavailable should be in the access mode of READWRITE. Tapes not in a library should have their access mode set to either READONLY or OFFSITE. Tapes that belong to a Storage Pool that has TAPEPOOL_OFFSITE in its name should be set to READONLY and tapes that belong to a Storage Pool that has COPYPOOL_OFFSITE in its name should be set to OFFSITE. To determine which storage pool a tape belongs to, use q vol <volume name> f=d to list the details of any tape volume. This will give you the name of the Storage Pool tape volumes belong to. o Are there any COPYPOOL tapes in READONLY status? Use q vol stg=*copy* access=readonly All COPYPOOL tapes that are out of the library for vaulting need to be in access mode OFFSITE Use UPDATE VOL <volume name of Copy Pool tape> ACCESS=OFFSITE.
Are there any tapes in the mail-slot doors of the library or libraries? Use the appropriate remote management software to view the operations panel display Are there any errors displayed on the Library(s)? Use the appropriate remote management software to view the operations panel display Are there enough scratch tapes in the library or libraries? Execute the run scratch command on the TSM server. Use this information on how many scratch tapes are available when doing a tape rotation. Check the Storage Portal to see if any tapes were left in Library-PLCI status, Central Operations ->Portal Management >Volumes, if so reconcile. In most cases the tapes left in Library-PLCI status from a previous rotation will be in the library. Use q libvol * <volume name> to verify if a tape in Library-PLCI status is in the library. Check the Managed Services Portal/Remedy ticket queue to see if any outstanding tickets for previous rotations exist. If so check to see if the rotation is in fact complete. Once the rotation is confirmed complete, close the outstanding ticket. From the TSM server execute a run scratch command to get a current status of the amount of scratch volumes.
Page 3 of 21
Check for cleaning tapes in the library and verify the remaining number of cleanings for the cleaning tape. For further instructions on why we need cleaning tapes and what to do next reference the below LiveLink path. https://livelink.sungard.com/livelink/livelink.exe/5194900/PRO%2DCENOP%2DTAPE %2DTape_Drive_Cleaning_Procedure.doc?func=doc.fetch&nodeId=5194900&viewType=1
Before starting the new rotation a very important step is to check to see if any tapes were already initialized into a previous rotation by having a backup set assigned in the Storage Portal. In the Storage Portal go to, Central Operations ->Batch Processes ->Prepare tapes for removal from library
Recall any expired offsite tapes from vaulting. Go to Central Operations ->Batch Processes ->Recall expired offsite pools from vaulting. From the drop down menu at the top of the page select View All, and then click on the Submit button. This will generate a list of Offsite Pool Tapes that have expired and need to be recalled back from the offsite vaulting vendor. By submitting the list that is generated an automated email for each tape will be sent to the associated data center to notify them it is time to recall the tapes listed in the email they receive. One email is generated for each tape recalled.
Completing these core Health Check tasks is key for maintaining a functional and stable environment and is part of the overall routine associated with tape rotation.
Page 4 of 21
If the preceding pre-rotation health check tasks have been completed then it is okay to proceed with the new tape rotation for the assigned data center.
Page 5 of 21
A. Tape Rotation Checking Tapes Out for Vaulting Generate a daily checklist/template for the specific data centers and date of the tape rotation. This is done by selecting the corresponding checklist for the data center from this location in LiveLink: https://livelink.sungard.com/livelink/livelink.exe? func=ll&objId=320840&objAction=browse&sort=name&viewType=1 Use the Storage Portal to generate the list of tapes to be rotated This list is generated by selecting the appropriate IDC from Central Operations ->Batch Processes ->Reconcile TSM/internal portal backup sets:
The key to this first step is verifying that the list of tapes is accurate. There are some basic steps to take in order to make sure the tapes listed on the TSM/Reconcile list are ready for rotation out of the tape library.
Page 6 of 21
Use q mount to see if any of the tapes on the list are currently mounted by the TSM server. If you find any tape(s) on the list displayed in the results of the q mount command, uncheck the tape(s) from the list of tapes in the portal. Look at the % Utilized field for the list of tapes in the portal, if any tapes have % utilized of 0%, uncheck the tape(s). Look at the Volume Status field for the list of tapes in the portal, if any tape(s) show a status other than Library, verify the tape(s) are physically in the library, using the q libvol command, and then change the status of the tape(s) in the Storage Portal. To change the Volume Status of a tape go to Central Operations ->Portal Management >Volumes, locate the tape(s) in the list of volumes and double click the volume id to open up the details for the tape(s). Uncheck any tape(s) that appear on the list but is not present in the library, reconcile the tape(s) after you complete the tape rotation. Submit the list of tapes you have verified on the list. This will advance the desired tapes to rotate offsite to the next logical step in the Storage Portal. Use the Storage Portal to generate the TSM server commands necessary to prepare the desired tapes for offsite vaulting. Central Operations ->Batch Processes ->Generate TSM scripts for IDC tape rotation This function will generate the TSM server commands needed to update access status for those tapes which require it and it will generate the commands needed to checkout the tapes from the library from which they reside.
Copy and paste the UPDATE, CHECKOUT and MOVE DRMEDIA commands generated by the Storage Portal into the checklist/template you created at the beginning of the tape rotation process. This will allow you to organize your commands and ready them for execution. Execute the UPDATE, CHECKOUT and MOVE DRMEDIA commands on the TSM server(s).
Checks and balances are important at every step. To verify the removal of the tapes to the door use the remote management software for the library you are using. It is advisable to have the TSM activity console open when doing a rotation for monitoring TSM messages. There are times where the TSM server says that a CHECKOUT
Proprietary and Confidential All rights reserved by SunGard
Page 7 of 21
command was successful and yet the tapes did not move to the library door. Using the remote management software for the library is the most reliable way to see if all the CHECKOUT commands succeeded. Once the tapes have been successfully updated and checked out of the library the next step is to once again move the tapes along in the Storage Portal. Central Operations ->Batch Processes ->Prepare tapes for removal from library (This will move the tapes being rotated to the next logical step in the Storage Portal where the tapes go to a list of tapes for the IDC/data center operations staff to view when they are doing their part in the tape rotation)
View the list of tapes the IDC will see, this is the next check and balance step to complete. IDC Operations ->Tape Operations ->Send tapes to offsite vaulting
Create a Help Desk ticket for the appropriate data center for processing of the tape rotation request. Use SunGard Availability Services Portal (Remedy), to open a New Request and delegate to the data center responsible for the tape rotation. There is a generic HD ticket template at the bottom of the checklist/template created at the beginning of the tape rotation process.
Page 8 of 21
Verify that Managed Services Portal (Remedy) ticket was completed properly. Read all comments as there may be something out of the ordinary that the data center has noticed and documented. Check in any scratch tapes that are being added to the library or libraries. The command to check in scratch tapes is not generated by the Storage Portal as the commands were for rotating tapes out of the library. This LABEL LIBVOL command will check in all tapes in the door as make them scratch tapes in the library. As with the commands used to rotate tapes out of the library, use the TSM server to execute this command. The library name is only an example.
LABEL LIBVOL PIT01SSTAPLBY01 LABELSOURCE=BARCODE CHECKIN=SCRATCH OVERWRITE=YES SEARCH=BULK
The LABEL LIBVOL command will require interaction with the TSM server to complete the requested checkout(s). After the LABEL LIBVOL command is executed, the TSM server will display in the activity console this output: ANR8373I 036: Fill the bulk entry/exit port of library PDX01SSTAPLBY04 with all LTO volumes to be processed within 60 minute(s); issue 'REPLY' along with the request ID when ready. The number 036 that follows the ANR number is the request ID to use with the REPLY command which is executed after the LABEL LIBVOL command is executed. On both the TSM command console and the TSM activity console you will see ANR8499I Command accepted, if the REPLY command was successfully executed. Verify check in of volumes for the library. As with verifying the checking out of tapes, use the remote management software to verify that the tapes were entered into the library. Or use the q actlog search=ANR8810 command to verify which tapes were checked into the library. The last step in the Storage Portal is to logically check in the tapes so that they are in the status of Library. When tapes are made ready by the data center to check in to the library as scratch, the Volume Status for each scratch tape will change to Library-PLCI in the Storage Portal. Library-PLCI stands for Pending Logical Check In. To logically check in tapes in the Storage Portal go to Central Operations ->Portal Management ->Volumes. The tapes checked into the library will be listed with Current Status of Library-PCLI. To the far right of the tapes listed in the Storage portal as Library-PLCI, there will be a small grey square icon. To check the Library-PLCI tapes in and change its Current Status to Library, click on the small grey icon.
Page 9 of 21
In the Managed Services Portal (Remedy) complete and close the HD tape rotation ticket.
Page 10 of 21
It is advisable to print out the list you see for accuracy and the information on the printed list of tapes to be sent offsite will be needed later for sending the tapes to the vaulting vendor.
Remove the ejected tapes from the library that reside in the library door. Use the printed list of tapes to be sent offsite for accuracy of tapes to be removed and sent to the vaulting vendor. NOTE: Not all tapes removed from the library may be sent to an offsite Vaulting Vendor. Some tapes may, at the discretion of Tape Operations, be removed to onsite storage. These tapes will have a designation of 'IDCVault'. The Remedy rotation ticket will identify any tapes designated for IDC-Vault status.
Scan the removed tapes out of the portal using the provided bar code scanner. If for some reason the scanner is not operating properly, the tape volume numbers can be entered manually in the portal. At the top right of the Send tapes to offsite vaulting page in the Storage portal, there is a text field labeled Volume Label where the tape volume number can be typed. After entering the tape volume number manually hit Enter or Return on your keyboard to submit the volume. This will change the Volume Status of tapes being sent offsite. The new status will be Vault. This the appropriate status for any tape listed in the Storage Portal that is at an offsite Vaulting Vendor.
Prepare the removed list of tapes for offsite vaulting. Each data center may have a different process to accomplish this portion of the tape rotation. Please follow your local procedure for preparing tapes for the vaulting vendor. The printed list of tapes to be sent offsite will have the needed information for the vaulting vendor. Add the desired amount of scratch tapes to the library (this will be part of the Remedy Ticket details.) To identify which tapes are scratch tapes use the Storage Portal. Go to IDC Operations ->Tape Operations ->Add scratch tapes to library
Page 11 of 21
Select the appropriate IDC and library from the pull down menus. All available 'IDC-Scratch' tapes will be listed on this page. Select the required number of tapes from this list. Once you have identified which tapes to use as scratch tapes, use the bar cod scanner to scan the tapes into the portal. Again if for some reason the bar code scanner is not operational the tapes can be entered into the portal manually going into the library. To enter the tapes manually use the text field at the top right portion of the IDC Operations ->Tape Operations ->Add scratch tapes to library page.
Place the IDC scratch tapes into the door of the library. Make sure all tapes are fully inserted into the door slots and the library door is completely closed. If tapes are not completely inserted or the library door is not fully closed the library may not be able to read the barcodes on the new tapes, or the library itself may be damaged. Update the Managed Services Portal Remedy Ticket with the state of the rotation work and progress. If the data center portion of the Tape Rotation is complete, delegate the ticket back to Tape Operations.
Page 12 of 21
NOTE: The IDC will have to recall tapes for Tape Services when an offsite tape is require for fulfilling a restore request. When tapes are recalled from offsite vaulting before they are due back based on their retention expiration date, the tapes should show up in the portal on a report called Notify Vaulting source of tape status changes. To check this report in the Storage Portal go to IDC Operations -> Tape Operations -> Notify Vaulting source of tape status changes. From this page select any tapes that are listed that have been recalled. This report should also be checked daily to make sure no tapes have been inadvertently left behind from a previous recall of a tape from offsite before its retention expiration date. Typically the vaulting vendor is contacted and instructed to return tapes required for a restore then the tapes are submitted in the Notify Vaulting source of tape status changes report.
Page 13 of 21
Methodology:
Where DRM tape rotations are done, the SunGard data center operations staff is primarily involved in the day-to-day activities. They are guided by the outcome of DRM processes on each of the TSM servers. In case they come across any issues, they are supposed to get in touch with the On-Call person of the Tape Operations group immediately. The concerned person in Tape Operations team then works with the data center personnel to resolve the issue. Tape rotation for TSM DRM SunGard sites consists of Daily DRM notification Emails to data center operations, Tape Operations and Off-site vendor Daily maintenance of the TSM server(s) and associated libraries. Making sure that the list of tapes mentioned in the DRM emails are exchanged between Onsite and Off-site on daily basis. Maintaining the Scratch count on all servers at around 30-40 per night, as you perform rotations you will see what is acceptable for each server.
Each day the TSM Server sends three emails of interest through the DRM Process. The first email comes out usually before 7AM local time to the server. The other two emails are sent out when the daily maintenance is complete. This can vary highly from server to server but it is scheduled to be completed before 4PM local time.
Page 14 of 21
If the tape is listed, run the following command: move drm VOLSER wherest=mo tostate=vault remove=bulk
If the tape is not listed in the DR Media list, see if it is listed under q libvol LIBRARY VOLSER
Page 15 of 21
If it is not listed in TSM at all as being inside the library eject it using the library GUI.
If you dont have your RSA passkeys setup you will be prompted for your OS password on each server. The only two lines of concern regarding tape rotation are scratch count and empty slots.
Page 16 of 21
Adding Scratch: To get a count of scratch tapes, use either the Server Health script or the run scratches command. If a server has less then 30 scratch the procedure to add scratch is as follows: ALTSM1 - ARLIB1 (3494) Monitored by the data center - ARLIB2-LTO (3584) Monitored by the data center ALTSM2 (HCL) - TSMLIB3 (3584) Monitored by Tape Ops and/or CSD CRTSM1 - CRLIB1 (3584) Monitored by the data center, limited. CVTSM1 (Varian) - CVLIB1 (3584) Monitored by the data center, reliable. CRTSM2 (HCL) - TSMLIB3 (3584) Monitored by Tape Ops, call the data center for assistance. WDTSM1 - WDLIB1 (3584) Monitored by the data center, limited. PHTSM1 (Library manager for PHTSM1, PHTSM2, PHTSMPMI) - PHLIB1 (3494) Monitored by Tape Ops - PHLIB2-LTO (3584) Monitored by Tape Ops PHTSM2 Handled by PHTSM1 PHTSMPMI Handled by PHTSM1 PHTSM3 (Library manager for PHTSM3, PHTSM4) - PHL00SSTAPLBY03 (Adic I2000, LTO2) Monitored by Tape Ops - PHL00SSTAPLBY04 (Adic I2000, LTO3) Monitored by Tape Ops PHTSM4 Handled by PHTSM3 SDTSMPU (Library manager for SDTSMP1) - SDLIB1-LTO (3584) Monitored by Tape Ops SDTSMP1 Handled by SDTSMPU MBTSMPU (Library Manager for MBTSMP1) - MBLIB1-LTO (3584) Monitored by Tape Ops MBTSMP1 Handled by MBTSMPU If any data center needs scratch, the first step is to look at your Vault Pick / All Inventory list and get an idea of how many scratch are available. Next, open the library GUI and ensure the I/O door is empty, if it is not, be sure to ask the data center operator to remove those tapes prior to adding scratch. Next, call the data center where the TSM server is located and state that you are from tape ops and need scratch added to the library. They will tell you how many they have and insert as many as they have/or as many as they can, into the I/O door of the library. If you are using tapes from the vault pick list, pull up the library GUI and look at the list of tapes they inserted into the I/O door. On each tape in the I/O door, run the TSM script moveonsitescratch on the server who previously owned the tapes as follows run moveonsitescratch VOLSER Note: In environments where there is a library manager present, you will run this script against the server who previously owned the tapes, but you will run the label/checkin command on the library manager itself.
Proprietary and Confidential All rights reserved by SunGard
Page 17 of 21
If you are unsure of the status of a tape, want to know who owned it last, or are having problems checking it in you can also run the script showmedia against the volume (run showmedia VOLSER) It is very informative. Once all volumes are deleted and converted to scratch we can check them in: checkin libvol LIBRARY search=bulk checkl=barcode status=scratch
After you run this command dont forget to reply to the open request (q req / rep XXX) Ideally, we shouldnt have any scratch lying around in an data center as long as there are open slots in the library so do this as many times as you feel necessary until the scratch count is at a comfortable/acceptable level. Labelling new tapes: If a data center does not have any scratch available from offsite, they will usually have new unused tapes around. Call the data center and if they say they have no tapes from offsite, request that new tapes are added. After you confirm there are tapes in the I/O Door run the following commands: label libvol LIBRARY search=bulk labels=barcode checkin=scratch
After you run this command dont forget to reply to the open request (q req / rep XXX) Converting existing tapes: In the event that there are no offsite tapes and no new tapes, and you need scratch tapes, the first thing to do is to run the script qvols against the server and see what primary tape pools can be reclaimed: run qvols If a storage pool with the name TAPE in it is listed under the 90-100% reclaimable section, lower the reclaim level. upd stg STORAGEPOOL reclaim=80
You can also try to delete some of the volhist: del volhist todate=-7
Last buy not least you can look for tapes listed as private with a null last_use field and manually update them to scratch.
Proprietary and Confidential All rights reserved by SunGard
Page 18 of 21
q libvol
Look for tapes listed as private with a blank entry under the Last_Use column and run: update libvol LIBRARY VOLSER status=scratch
If these steps for finding scratch tapes fail, you may need to escalate to avoid further issues because of the lack of available scratch tapes in the library.
Page 19 of 21
Troubleshooting FAQ
Q: Why did I get the error, ANR2212I UPDATE <VOLUME>: No volumes updated, from the TSM server after running my UPDATE command during tape rotation? A: TSM will give this output if the tape volume is mounted and in use by the TSM server.
Q: Why did I get the error? ANR8433E CHECKOUT LIBVOLUME: Volume RD1091L2 is not present in library, when checking out tapes from a library during tape rotation? A: The tape listed in the CHECKOUT command is not currently in the library you attempted to eject it from. In most cases if the tape is supposed to go offsite for vaulting it is located in another library which is also part of the data center.
Q: Why did I get the error, ANR8819E Unable to read the barcode label(s) in library RDU01SSTAPLBY02, in the activity log/activity console when attempting to check in scratch volumes? A: This will occur when the LABEL LIBVOL command is executed with no tapes in the library door.
Q: Why do I see a different Volume Status for a tape in the TSM Reconcile report than in the Volumes report? A: This may be caused by a duplicate entry in the Storage Portal for the tape volume you are looking at.
Q: Why did a tape volume appear in the Prepare tapes for removal from library list fail to appear on the Send tapes to offsite vaulting list? A: Sometimes the Storage Portal will not move a tape that has been submitted from the Prepare tapes for removal from library list. In this case go to the Volumes page and select the yellow icon for the volume in question. This will do a manual add of the volume to the Send tapes to offsite vaulting list.
Page 20 of 21
REVISION HISTORY
Revision 0
Date
Initiator
1 2 3
Minor corrections to syntax on page 2 in details section. Added information about Cleaning Tapes on Page 3
Page 21 of 21