Wednesday, 25 February 2009

ORA-27211: Failed to load Media Management Library

Introduction:
We had interesting but painfull issue for Oracle backups which are done with Veritas Netbackup. There was activity of patching on Netbackup. And after the patching there was issues with Oracle Backups. All the Oracle backups were failing with following error messages....

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of allocate command on ch1 channel at 02/25/2009 13:41:11
ORA-19554: error allocating device, device type: SBT_TAPE, device name:
ORA-27211: Failed to load Media Management Library
Additional information: 2

Troubleshooting:

We tried different steps to resolve the error. Following are the steps which we did.

1- Checked with the "sbttest" command, to see whether Netbackup is doing successfull test backups. and we got the following errors.

$>sbttest test

The sbt function pointers are loaded from oracle.static library.libobk.so could not be loaded. Check that it is installed.

2- We tested with simple rman commands to allocate a tape channel and the release. and following was the error messages.

RMAN> run {
2> allocate channel ch1 type 'SBT_TAPE';
3> release channel ch1;
4> }

using target database controlfile instead of recovery catalog
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of allocate command on ch1 channel at 02/25/2009 13:40:43
ORA-19554: error allocating device, device type: SBT_TAPE, device name:
ORA-27211: Failed to load Media Management Library
Additional information: 2

3- The above error messages were showing that MML is not able to load, and that libobk.so was being used as static from Oracle binaries.

We explicitly used the Netbackup /usr/openv/netbackup/bin/libobk.so64.1, inside the RMAN script, but we got the same error as of step 2. Following were the error messages

RMAN> run {
2> allocate channel ch1 type 'SBT_TAPE' PARMS='SBT_LIBRARY=/usr/openv/netbackup/bin/libobk.so64.1';
3> release channel ch1;
4> }

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of allocate command on ch1 channel at 02/25/2009 13:41:11
ORA-19554: error allocating device, device type: SBT_TAPE, device name:
ORA-27211: Failed to load Media Management Library
Additional information: 2

The above all steps showed that there was strange kind of problem with Netbackup. As we were not able to allocate one channel with "sbt_tape" type, and all the times MML was not able to be loaded.

We also check in the udump of the Oracle Database server, and following were the error messages in udump trace file.

*** SESSION ID:(47.58746) 2009-02-25 13:40:43.872
SKGFQ OSD: Error in function sbtinit on line 2384
SKGFQ OSD: Look for SBT Trace messages in file /usr/oracle/home/oracle/admin/ORA/u
dump1/sbtio.log
SBT Initialize failed for oracle.static
*** 2009-02-25 13:41:11.006
Failed to load SBT library /usr/openv/netbackup/bin/libobk.so64.1

The above error messages of udump trace file, was clearly showing that the libobk.so64.1 was not able to load. And also we found valid symbolic link from $ORALCE_HOME/lib/libobk.so to /usr/openv/netbackup/bin/libobk.so64.1

Resolution:

After trying every way, finally our system admin did one thing and got resolved the issue. What he did, he renamed the /usr/openv/netbackup directory and again pushed the netbackup client from Netbackup Server, and we again tested the Oracle Backups and it was all working. Following is the simple allocation of channel of 'sbt_tape'



RMAN> run {
2> allocate channel ch1 type 'SBT_TAPE';
3> release channel ch1;
4> }

using target database controlfile instead of recovery catalog
allocated channel: ch1
channel ch1: sid=35 devtype=SBT_TAPE
channel ch1: VERITAS NetBackup for Oracle - Release 6.0 (2005090703)

released channel: ch1

RMAN>

4 comments:

Anonymous said...

This is client version differents. The client was upgraded but de oracle cliente no. For example cliente MP7 is not compatible with oracle cliente MP6. Both the packages must be upgraded at same time.

Anonymous said...

This has been a useful article... Thank u!! :)

Abdul Rasheed said...

Nice work Mehmood! I cam across this from the twitter post. Although this is an old post,it is still relevant.

Disclaimer: I work for Symantec. Comments are my own.

Etti said...

We just encountered the same problem for an RMANbackup,solution was: (servername for safety replaced with x characters)
orapiqmg@xxxxx:/piqmg_u01/app/oracle/admin/PIQMG10/rman > ls -la /piqmg_u01/app/oracle/product/11.2.0.3/lib |grep libobk
lrwxrwxrwx 1 orapiqmg dbapiqmg 36 May 2 08:33 libobk.so -> /usr/openv/netbackup/bin/libobk.so64
orapiqmg@xxxxxx:/piqmg_u01/app/oracle/product/11.2.0.3/lib > rm libobk.so
orapiqmg@xxxxx:/piqmg_u01/app/oracle/product/11.2.0.3/lib > ln -s /usr/openv/netbackup/bin/libobk.so64.1 libobk.so
orapiqmg@xxxxxx:/piqmg_u01/app/oracle/product/11.2.0.3/lib > ls -la libobk.so
lrwxrwxrwx 1 orapiqmg dbapiqmg 38 May 8 02:24 libobk.so -> /usr/openv/netbackup/bin/libobk.so64.1
orapiqmg@xxxxx:/piqmg_u01/app/oracle/product/11.2.0.3/lib >