How to resolve a Windows locking issue that prevents patching via Grid Control

We ran into a peculiar issue when patching an 11g Windows database using Grid Control, and I want to describe the peculiar way we used to resolve this issue in Windows.

As a background, Grid Control enables the DBA to patch an Oracle Database using the Provisioning Pack. Oracle supplies deployment procedures that can be executed out of the box, or copied and modified, to patch a single database, or a RAC database, and so on. The deployment procedure, when it reaches the step of actually patching the database, ultimately uses opatch at the database server level. 

Now, the problem was, that when trying to patch an 11g database on Windows using Grid control,  opatch was failing since the "oci.dll" file in the 11g db home was locked. 

We first used a third party utility ProcessExplorer.zip to find the file, and this found it was locked by windows process svchost (Service Host). (There are multiple svchost processes running. To see which copy of svchost is running what service, use "tasklist /svc" in a command window.)

As per a hint found in Google search, we set the windows service "Distributed Link Tracking  Client" to manual and stopped this service. This made no difference and the file stayed locked.  As per another hint found in Google search, we renamed oci file to oci.bak then killed the scvhost process using process explorer. The scvhost process restarts automatically but cant find the oci file. But this effected the working of Grid Control (because the listner cannot really work without oci.dll being used) so do not use this renaming technique.

We then found that when the listener process is stopped, and scvhost is killed, scvhost restarts without the lock. So the issue obviously happens because during the patch process, when the listener is stopped, the lock is not released by scvhost. This was the main cause of the issue.

We then edited a custom deployment procedure and enabled the extra step to stop all Windows services in that home. Normally this step is not enabled. But this had no effect on the error.

The solution that ultimately worked was that we used unlocker (another third-party utility) to unlock oci.dll just after the databases and listeners were stopped by the deployment procedure. Make sure you exit process manager before doing the unlocking, because process manager itself locks other dlls. So the steps are:

1. Start the patching process.
   
2. Dont start the third-party process manager at all. 
   
3. In windows explorer go to the 11g Oracle Home/bin and find oci.dll. 
   
4. Wait till the grid control patching reaches the step of stopping databases.
   
5. Right click on oci.dll and unlock the lock of scvhost on this file, using third-party unlocker. 
   
6. Patching then continues successfully since the file is not locked any more.