EM for Active-Passive environments

A DBA wrote to me and said he has some issues in seeing the targets in a 2 node active
passive cluster using SUN cluster. My advice to him was as follows:

Refer to the Oracle documention chapter "Configuring Oracle Enterprise Manager for Active and Passive Environments"

http://download.oracle.com/docs/cd/B16240_01/doc/em.102/e10954/actpass_env.htm#sthref274

As per this, the EM agent must be deployed to each physical node as well as to the logical IP address. You have obviously not done that. Setting up in an active-passive environment is more involved as compared to setting up in an active-active cluster (Oracle RAC!)

Reconfiguring Database Control using EMCA

Someone asked me a question. He wanted to restore a database to a new host. The database included the Enterprise Manager Repository. Using RMAN, he restored the database to the new host, but was unable to start Enterprise Manager on the new host.

My answer:

I presume you are talking of Enterprise Manager Database Control and its repository, and not Grid Control.

Since you have restored the database to a new server, you must manually configure database control using EMCA. This is because available ports etc may be different on different servers and Oracle cant use the same settings, also the Database control configuration files are actually sitting outside the db in the oracle home. So when you copy the database, you are not copying the home and that info is lost, so you must reconfigure database control on the new host.

Please refer to the Oracle documentation at :
http://download-uk.oracle.com/docs/cd/B283...ca.htm#BABDEHCE

This will tell you how to use EMCA to config database control. You may need to deconfig first to clear up any existing repository.

Clone Support Files

One Important note from the 10.2.0.5 patch Release notes:

As part of the upgrade of a Grid Control installation from 10.2.x.x to 10.2.0.3 or higher, you must download and install Clone Support Files from My Oracle Support and install the files onto each OMS as part of a post-patch configuration task to enable clone support for Oracle Tech stack components.

This step can often be overlooked and will result in issues for the cloning and provisioning subsystems.

More attempts to release the oci.dll lock

As a follow-up to the oci.dll locking issue and the use of unlocker that I explained

in an earlier blog post, when I conducted a session on "Patching with Grid Control",

a DBA suggested there was a Windows fix avaiable for the oci.dll locking issue.

I found Metalink Note 232827.1 which suggested that the svchost process

has the oci.dll locked on the machine. The note asked to shutdown Windows

services such as the Com+ Event System, MSDTC, and IIS. After

those services are shutdown the machine has to be rebooted in order for the

svchost process to release the oci.dll. The note suggested that these services

lock the oci.dll even though the user may not be using them.

I followed these steps, but it didnt make a difference. The ocii.dll

stayed locked even after the listeners were stopped.

The DBA then suggested to set the following registry key in Windows:

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer]

AlwaysUnloadDLL=1

I already had this registry key set. so this was not the solution. I had to

reply on the previous unlocker strategy (see the previous blog post).

A further bit of reading suggested that Windows keeps dlls in memory on

purpose for a certain period of time even after the dll is not is use. This is

for performance reasons.

I do wonder why Unix has no such issues and the performance is still ok

in unix without such "hold things in memory for some indefinite time"

tactics. We never have such locking issues on unix.

A side note on this issue: when I removed and reloaded the registry key

"AlwaysUnloadDLL=1" into the registry, from then on whenever I tried to

start the Grid Control Management service, it failed to load. I remembered

reading somewhere on the internet that this registry key may be causing

a Java load issue, so I deleted the "AlwaysUnloadDLL=1" key straight

away, and the Grid Control management service started loading fine.

Whew!

Windows - you can love it, you can hate it.

POC findings: Offline Patching of Previous CPUs and Modifying Reports

We were just going through a POC for Enterprise Manager Grid Control recently in a large client, and had some interesting results I like to share with everyone.

This particular client will not connect the Grid Control management service (OMS) to the internet for security reasons. This means the Refresh from Metalink job has to be run manually.

This has to be done after downloading certain xml files from Metalink (which contain the Patch Analysis metadata ie. which patches are available and what bugs they fix) and uploading these files, as per the following "Grid Automation Deployment Procedures" document on the Oracle OTN website:

http://www.oracle.com/technology/products/oem/pdf/grid-automation-deployment-procedures.pdf

After this the patch advisory on the Grid Control Home page rightly suggested that for a 9.2.0.1 home, the 9.2.0.8 patch as well as the October CPU had to be applied. Since the xml files had been downloaded manually from Metalink in December, the October CPU was the latest and therefore recommended to be applied.

The client wanted the earlier July CPU to be recommended instead. They said they rarely apply the latest patch but instead apply the earlier CPU patch. Now, the only way to do this would be to download the xml files from Metalink each month and store them on separate directories somwhere, but only upload the xml files to Grid Control as per the CPU you would like to be recommended by the Patch Advisory. For eg, if you want the Jul CPU to be recommended, then only upload the Jul or August or Sep xmls.

They also wanted a report to show all the patches applied on all their databases, showing each database one by one. The report as suggested was not fully available off-the-shelf in the reporting facility in Grid Control. The closest we come to this, is in this report:

Reports..Oracle Home Patch Advisories..Applied Interim Patches

This shows the patch, bugs fixed, installation time, Host, Home directory and Platform. It doesn’t show the databases, the reason being that the patch is actually at the Oracle Home level and not at the database level. This report may be appropriate for their needs, however if they wish to add in the database name, it is possible to modify the report by creating a copy andthen modifying the Sql statement. Oracle has mgmt$ views from which the info may be extracted.

Patching issue in Grid Control and the FIX

Oracle Support recently advised us to apply a number of patches to our Grid Control 

installation, as per a Metalink Note 427577.1 (OMS Patches required for setting up 

Provisioning and Patching in 10.2.0.3 GC / 10.2.0.4 GC).  The main purpose was to 

improve the Provisioning and Patching facilities in Grid Control.

However, we had an issue with applying one particular patch 7214697 from this list. 

This was giving an error towards the end in the opatch apply. The error occurred 

when PARdeploying prereqs.par :

C:\OracleHomes\oms10g\bin\PARDeploy -action deploy -parFile 

C:\OracleHomes\oms10g\sysman\prov\paf\prereqs.par

0 [main] DEBUG oracle.sysman.eml.swlib.DBEntity - SWLIB getRepositoryFormatForRevision

...

...

78 [main] DEBUG oracle.sysman.eml.swlib.DBEntity - SWLIB getDocuments

78 [main] DEBUG oracle.sysman.eml.swlib.DBEntity - SWLIB getRepositoryFormatForParameters

125 [main] DEBUG oracle.sysman.eml.swlib.DBEntity - SWLIB error=An entity with

the specified name already exists

Even though there was an error during the patch application, this patch showed up in the 

"opatch lsinventory". But then, when Grid Control was started, we were not even 

able to enter the patching screen.  As soon as we went to the "Deployments" tab 

and clicked on "Patching through Deployment Procedures", we got :

"Internal Error has occured. Check the log file for details."

We got exactly the same error when we clicked on "Deployment Procedures",

"Patch Procedures" etc.

The patch, even when rollbacked, failed with exactly the same error.

It disappeared from the inventory, but the internal error was still present

in the patching screens in Grid Control.

Finally, the fix to this issue turned out to be a very simple one:

cd %OMS_ORACLE_HOME%\patches\p7214697_10204_GENERIC\7214697\custom\scripts 

Edit post.bat and change the line: 

%ORACLE_HOME%\bin\PARDeploy -action deploy -parFile %ORACLE_HOME%\sysman\prov\paf\prereqs.par 

to 

%ORACLE_HOME%\bin\PARDeploy -action -force deploy -parFile %ORACLE_HOME%\sysman\prov\paf\prereqs.par

 

This prevents the error when applying the patch, by forcing the PARDeploy of prereqs.par. 

After the patch is successfully applied without any errors, the Grid Control patch deployment procedures work ok.

By the way this seems to be a Windows Grid Control issue. 

How to resolve a Windows locking issue that prevents patching via Grid Control

We ran into a peculiar issue when patching an 11g Windows database using Grid Control, and I want to describe the peculiar way we used to resolve this issue in Windows.

As a background, Grid Control enables the DBA to patch an Oracle Database using the Provisioning Pack. Oracle supplies deployment procedures that can be executed out of the box, or copied and modified, to patch a single database, or a RAC database, and so on. The deployment procedure, when it reaches the step of actually patching the database, ultimately uses opatch at the database server level. 

Now, the problem was, that when trying to patch an 11g database on Windows using Grid control,  opatch was failing since the "oci.dll" file in the 11g db home was locked. 

We first used a third party utility ProcessExplorer.zip to find the file, and this found it was locked by windows process svchost (Service Host). (There are multiple svchost processes running. To see which copy of svchost is running what service, use "tasklist /svc" in a command window.)

As per a hint found in Google search, we set the windows service "Distributed Link Tracking  Client" to manual and stopped this service. This made no difference and the file stayed locked.  As per another hint found in Google search, we renamed oci file to oci.bak then killed the scvhost process using process explorer. The scvhost process restarts automatically but cant find the oci file. But this effected the working of Grid Control (because the listner cannot really work without oci.dll being used) so do not use this renaming technique.

We then found that when the listener process is stopped, and scvhost is killed, scvhost restarts without the lock. So the issue obviously happens because during the patch process, when the listener is stopped, the lock is not released by scvhost. This was the main cause of the issue.

We then edited a custom deployment procedure and enabled the extra step to stop all Windows services in that home. Normally this step is not enabled. But this had no effect on the error.

The solution that ultimately worked was that we used unlocker (another third-party utility) to unlock oci.dll just after the databases and listeners were stopped by the deployment procedure. Make sure you exit process manager before doing the unlocking, because process manager itself locks other dlls. So the steps are:

1. Start the patching process.
   
2. Dont start the third-party process manager at all. 
   
3. In windows explorer go to the 11g Oracle Home/bin and find oci.dll. 
   
4. Wait till the grid control patching reaches the step of stopping databases.
   
5. Right click on oci.dll and unlock the lock of scvhost on this file, using third-party unlocker. 
   
6. Patching then continues successfully since the file is not locked any more.