Review of new Middleware Management with Grid Control book

I was recently requested to review a new book "Middleware Management with Oracle Enterprise Manager Grid Control 10g R5" by Debu Panda and Arvind Maheshwari . I received a review e-copy of the book and was pleasantly surprised when I went through the book, it is a very good introduction for all the people who are working with Oracle Fusion Middleware and want to use Oracle Enterprise Manager Grid Control to manage most of the stack.

After a brief explanation of installing Grid Control, including monitoring basics, the book delves into monitoring and managing Oracle WebLogic Server, Oracle Application Server, Oracle Forms and Reports Services and Applications. It then also looks at the monitoring of BPEL process manager and BPEL processes, the monitoring of the Oracle Service Bus (OSB) and OSB services, managing the Identity Manager Suite, the Coherence Cluster, and also non-Oracle middleware like Apache, Tomcat, JBoss, IBM Websphere, and Microsoft middleware.

The book also has a chapter on the new Application Diagnostics for Java (AD4J) product and Composite Application Monitor and Modeler, and how to use these products to diagnose Java applications, and monitor and diagnose composite applications respectively.

As a surprise bonus, the book has a very interesting chapter on Building your own Monitoring Plug-in. The Sun Java Web server is used as an example, and the steps to build a simple monitoring plug-in for this web server are outlined. This is followed by another good chapter on Best Practices for Managing Middleware Components using Enterprise Manager.

All in all, a very useful book for Middleware administrators who are using Oracle Enterprise Manager Grid Control, or for those who want to know how Enterprise Manager is handling the monitoring and management of such products. Well done and a good practical guide.

1.1.0 version of EM widgets

Rejeeb Mohammed, User Experience Designer, EM User Experience Team writes:

We've released the 1.1.0 version of EM widgets.This build has few new features (including the much-requested option to customize database targets in the High-Load Databases widget) and a number of bug fixes.To upgrade to the newer version either use the auto update feature given in the widgets or reinstall the widget from OTN:

http://www.oracle.com/technology/products/oem/widgets/index.html

Thanks,
Rejeeb
User Experience Designer
EM User Experience Team

Most Popular OTN articles in 2009 - two of my articles on Enterprise Manager

Fellow Readers,

Great News. The Oracle Technology Network (OTN) has just published
the list of Most Popular OTN articles in 2009 and two of my articles
on Enterprise Manager are in the list.

http://blogs.oracle.com/otn/2009/11/the_most_popular_articles_and.html

The OTN is the world's largest community of Developers, DBAs and Architects.
The two articles of mine in the top 10 are:

Oracle RMAN Backups: Pushing the "Easy" Button
Oracle Enterprise Manager Grid Control Architecture for Very Large Sites

Thanks for the tremendous support of the readers on this topic. My aim is to make Grid Control popular among the DBA and IT community, it really is a fantastic product that makes our lives easier.

Regards,

Porus.

Final 2009 seminar on Oracle Performance Options, Products and Enterprise Manager Packs

This Friday, I conducted another interesting seminar which was S&I System's last Oracle seminar for the year. The topic was Oracle Performance Options, Products and Enterprise Manager Packs.

These were the topics I covered. The clients were introduced to many new Oracle concepts for the first time.

• Partitioning Option, and improvements in 11gR2
• Parallel Query Option, and improvements in 11gR2
• Advanced Compression Option (11g), and Hybrid Columnar Compression in 11gR2
• Active Data Guard (11g)
• In-memory Database Cache (Times-Ten), and it's Grid in 11gR2
• Enterprise Manager Diagnostics and Tuning Packs
  

The following Enterprise Manager packs were also demonstrated in live demos:

• Service Level Management Pack
• Configuration Management Pack
• Diagnostics and Tuning Packs
• Remedy Connector
  

I also demo'ed the new Enterprise Manager feature of SQL Monitoring of long-running SQL in 11g databases, which suitably impressed the attendees.

A number of clients immediately expressed interest in using the Service Level Management and Configuration Management packs for their hundreds of database targets, and discussed Grid Control architecture for such large systems.

The 2009 list of successful Oracle seminars conducted by S&I Systems can be seen on:

http://www.si-asia.com/marketing/oracleworkshops2009/

Security and Grid Control

A Senior Infrastructure Architect wrote:

"I am coming across distinct resistance towards using OEM Grid from those concerned with security around listener ports and data content - on a colocated management infrastructure (same data centre as the given databases)."

My reponse to this was:

"Any large corporate's security team does have quesions about Grid Control, but this is mainly due to lack of information. Grid Control is fully secure.

Firstly you can have the console protected by a security certificate (https), you can also have the traffic between the agent and the management service in https instead of http, and also it has two levels of security - first, the Grid control admin security (you have to create separate admins for target groups) and then the database login itself. The sysman password should be withheld from most users of Grid control.

If the security department wants tighter security, please recommend Oracle's Advanced Security Option which allow sql net encryption of all sql traffic and data encryption in the database. Regarding listener ports, you can password protect the listeners. Firewall needs to be opened up for certain other ports of Grid control.

We were using Grid Control in a large corporate site with more than 700 databases and found no issues.

Its a great product so do try to convince your security team. The benefits to the company are immense, and it also improves the life of the DBA teams - it leaves them with more quality time to do their senior DBA stuff, like architecture etc, instead of worrying about scripts for RMAN backups and the setup of Dataguard, or applying patches on multiple databases, which are just some examples of the many things automated by Grid Control. "

EM Desktop Widgets released

Rejeeb Mohammed, User Experience Designer in the EM User Experience Team has sent the following information:

I am so pleased to let you know that, EM grid control team has released the
first set of desktop widgets in oracle open world. EM Desktop Widgets are
lightweight internet applications ( built in Adobe's AIR environment) that
provide persistent desktop access to key Enterprise Manager monitoring and
diagnostic information.

There are 3 widgets available in the initial
release:

Target Search & Monitoring : This widget provides rapid
access to EM targets through quick and easy target searching. Also you can mark
targets as your favorites and monitor their status in real time.

High
Load DB : Provides a summary of the top 5 databases ordered by the Average
Active Sessions performance or load metric.

Service Level and
Monitoring: This widget provides a summary view of a set of services including
the service status and availability, and a status of SLA’s associated with the
service including any alerts.

I encourage you to download the widgets
from http://www.oracle.com/technology/products/oem/widgets/index.html,
and use them.

We absolutely need your feedback to determine what's good
and what's not with the current widgets. We will use this information to
determine the direction of future widgets.

Oracle Open World 2009

I visited Oracle Open World 2009 on invitation by Oracle. I had a press card due to this Enterprise Manager blog that I maintain. I met a number of fellow Oracle ACE directors such as Arup Nanda, Steven Feuerstein and so on.

I had a good look at the new Exadata V2 OLTP database machine from Oracle - the first machine to achieve over 1 million IOs (8K) per second in a single rack. Exciting stuff.

Customer Webcast: Mass Patching of Oracle Databases with Enterprise Manager

This is an interesting webcast from Oracle for Customers:

Mass Patching of Oracle Databases with Enterprise Manager

Date: Wednesday 7 October 2009
Time: 08:00 a.m. to 09:00 a.m. (PST8PDT)

Description:
http://conference.oracle.com/imtapp/app/conf_enrollment.uix?mID=159112088&mConfKey=grid

Please register for this.

Third Workshop on Grid Provisioning and Patch Automation

I just conducted a successful third workshop in the Oracle Office at Singapore
on Grid Provisioning and Patch Automation. There were a lot of attendees.
Many who could not attend are asking for repeat seminars.

In the workshop, I introduced the clients to sophisticated Oracle Grid
technology, including RAC, Weblogic server clusters, the Coherence
in-memory data grid, ASM, Oracle Enterprise Linux and Oracle VM.
I interested a lot of attendees in Oracle VM pools with Enterprise Linux
as a future platform for the Grid. I explained the various Enterprise
Manager Packs, especially the Configuration Management Pack and
the Provisioning and Patch Automation Pack.

Using an Amazon installation of Oracle Enterprise Manager
Grid Control Release 5 with the Provisioning and Patch
Automation pack - thanks to one of my friends who loaned me
this Amazon installation - I was able to successfully demonstrate
patching of twelve 10g databases with 5 patches, showing the
power of Grid Automation. The clients were suitably impressed
by the technology capabilities of Grid Control. At the same time
I told them about Oracle’s support of the Cloud.

I also successfully demonstrated a RAC Rolling Patch upgrade
of a two node RAC cluster, again using the Provisioning and
Patch Automation Pack on Oracle demonstration servers.

The DBA attendees asked for:

Ability to backup the db before the patch is applied – this feature is coming
Ability to rollback patches using Grid Control – this feature is coming
Ability to use PBRUN etc for security – which is already there
Ability to do offline download of patches – already there
Ability to keep a log of all changes done via the Grid Control console
Ability to do in Grid Control 10g what can be done in Database Control 11g eg. REPLAY

I have given this feedback to Oracle development.

Future workshops can be seen on:
http://www.si-asia.com/marketing/oracleworkshops2009/

Interesting questions on Grid Control Patching Automation

In a recent Grid Control Webinar, I heard some interesting questions
by clients that were answered by the Grid Control patching automation
expert in Oracle, Hari, who also writes on this EM Blog. The configuration
question was answered by another Grid expert, Shahid.

I am sharing these answers with you.

Question: Can rollback of patches be done using Grid Control?

Answer: As a part of the procedure, there are validate steps if the patch is
rightly applied or not. Rollback is not available Out-of-box for now, but
you could insert the step into the framework using the metalink note
info - :577557.1

Question: May I know that how frequently Grid Control gathers
configuration changes?

Answer: It is upto you. The interval is configurable. e.g. in the compliance
scenario you would want to gather changes at real time while for standard
application configuration it may be done once or twice a day.

Question: How do I get the details on how to use sudo etc. for patching?

Answer: See the Administrator's guide for provisioning and patching.

Some more feedback on the article

Some more feedback on the article:

Another DBA wrote: "Porus, you are the man ... would love to go through your articles. Digging my heels back in Oracle after a while and I intend to go through your articles while in the process. Great Stuff. Great Work."

Another Team leader said:

"I went through some of your papers and they are really amazing I must say. I like the article on Grid Control Architecture and RMAN very much, as we often get this questions from the customers but do not know where to look for. This really gives a very good insight."

Thanks guys for your appreciation of the articles. I wrote them specifically to convince customers that the time is right to use Grid Control. I am happy that you and other readers like them.

Patch a Thousand Databases, Using Oracle Enterprise Manager Grid Control

I am happy to announce that a brand-new article of mine has been published on the Oracle Technology Network (OTN) on 3rd Sept 2009. The article is titled :

"Patch a Thousand Databases, Using Oracle Enterprise Manager Grid Control".

You can read the full article with all the illustrations at the following URL:

http://www.oracle.com/technology/pub/articles/havewala-patch1000.html

The article will allow you to "Get a tour of the automated patching capabilities now available to Oracle DBAs", especially the fantastic new Deployment Procedures.

Please read and tell your friends about it. I am sure you will find it very interesting.

I have received many good comments about this article, one of them on a LINKEDIN.COM Oracle discussion group was from a DBA Team Leader:

"Great article. Very comprehensive with screenshots and covers every details
in patching thru the Grid Control. A must read for every DBA. Thanks."

Enterprise Manager Blog assists Akshaya Patra

The Enterprise Manager Blog, as a part of its social responsibilities, has decided to
assist a worthwhile charity doing good work in India - "Akshaya Patra" which is
feeding thousands of school children in India everyday with a noonday meal.
This in turn will help their education. Akshaya Patra is well known for doing
a great job in India in this regard.

The charity is currently trying to achieve a target in Global Giving, and we
are hosting their widget at the side of this blog. Please give what you can to
help achieve their target. The direct link is also below.

http://www.globalgiving.com/projects/sponsor-school-meals-of-200-000-children-in-india/?RF=widget_small3304

The Blog normally talks only about Oracle Enterprise Manager and related matters, but
we made an exception in this case, since it is a chance to do a lot of young children some good.

Second Workshop showed Grid Control creating and managing Dataguard

We just conducted a highly successful second workshop in
the Oracle Office at Singapore. A number of large banks attended
besides other large corporates in Singapore. We had Solution
Architects, Lead Engineers, Database Engineers, Database
Managers , Asst Vice Presidents, IT Directors, and EVPs.

We explained the disaster recovery options from Oracle, introducing
the attendees to powerful Oracle products such as Normal Dataguard,
Active Dataguard, RAC, Streams, Oracle Transparent Gateways,
Replication, Oracle Data Integrator (ODI) and the newly acquired
Oracle Golden Gate software.

The Enterprise Manager Grid Control interface was used to demonstrate
the management and performance monitoring of the Data Guard configuration,
as well as the command line interface of Data Guard “dgmgrl” for the switchover.
The power of Enterprise Manager Grid Control was emphasized for setting
up and managing Oracle Data Guard, as also the Provisioning Pack
to clone Oracle software and databases.

The attendees were excited about the Oracle technology explained
and showcased, and mentioned it was an “excellent” or “great” seminar.

Future workshops can be ssen on:
http://www.si-asia.com/marketing/oracleworkshops2009/

Revisiting Patch Management - Patch Set Updates

First of the PSU (more like a beta) was released on Jul 17th for Oracle DB 10.2.0.4 across specific platforms of Unix.   Oracle will release PSUs every quarter (starting with July 2009) along with quarterly CPUs across Databases and Middleware.  PSUs will be a superset of CPUs and will have additional recommended patches from Oracle. These set of patches are tested in cohesion.

First patch in the series is patch number : 8576156  applicable on 10.2.0.4 Database. The application of this patch can be identified from the inventory by running – opatch lsinventory. 

PSU provides all high value, recommended patches for the Oracle Software. Based on the Patch Management cycle, factor in PSU for the future cycles of patch management.  Suggest once you take up the PSU track,stay on the course. Without which, resolving conflicts, getting specific MLRs will require significant efforts.As the readme of the PSU note reads.. “Once a PSU has been installed, the recommended way to get future security content is to apply subsequent PSUs. Reverting from PSU back to CPU, while possible, would require significant effort, and so is not advised” 

Application of PSU can be automated through Deployment procedures. Refer to My Oracle Support note: 864316.1

For the future (October 2009 and onwards) you will get Out-of-Box support for the PSU across various versions.

 

Related Links:

854428.1 - Intro to Patch Set Updates (PSU)

Oracle RAC with Enterprise Manager Grid Control workshop

We just conducted an extremely successful Oracle RAC with Enterprise Manager Grid Control workshop at my company S&I Systems Singapore, jointly with Oracle. For details of this and future workshops, please see:

http://kamerad.parsizoroastrianism.com/S_and_I_GridControlWorkshopSeries2009_Porushh.htm
(Note: this URL is a new one from the URL I had supplied a few days ago).

Thanks to Oracle demonstration services known as TDS, hitherto available only to Oracle Pre-Sales but now avaialable to Oracle Certified Partners as well, we were able to show Grid Control in action managing a 4-node 11g RAC cluster database. We showed the easy addition and management of Cluster database services and their mapping to preferred and available instances, and the failover of the service to the available instance when the preferred instance is shut down.

We also demonstrated easy performance diagnosis and tuning of the RAC database using Grid Control. Also, a quick glance was taken at the Deployment Procedures supplied by Oracle for Rolling upgrades of the RAC database and also of the Oracle Clusterware. We also showed the Configuration managment capabilities that captured a history of changes at the server, cluster as well as database level. Thus, we consider Grid Control to be essential for efficient RAC database and cluster management.

The Enterprise Manager packs that would be essential for the above operations would be the Diagnostic / Tuning / Configuration Managment / Provisioning Packs.

Survey on Enterprise Manager :Patch Automation - Recoverability

Oracle is conducting a survey on Enterprise Manager :Patch Automation - Recoverability

Survey URL : http://www.zoomerang.com/Survey/?p=WEB229F23FFEF3

Can you please take 10 minutes to give your inputs/feedback. The survey captures your recommendation on how recoverability should be for Patching.This survey creates a direct impact on the upcoming EM product's patch automation experience.

Note,Survey closes on 23rd July 2009. Really appreciate a quick turn around.

We really appreciate your help in making Better Products

Thanks and Regards,

Hari

Hari prasanna Srinivasan Sr.Product Manager

Oracle

New One-Off Patch on top of 10.2.0.5 for Patch Automation

Download the latest one-off patch introducing fixes on Patch Automation feature from Enterprise Manager 10.2.0.5(Patch Number: 8653501)

Download the patch and apply it on one/all of the OMS server. Make sure you complete the additional steps in the readme.

Patch Summary

This one-off patch is applicable on top of 10.2.0.5 Enterprise manager, the patch introduces:

1. Enhanced Pre-requisites checks to standard deployment procedures,which can be invoked in Analyze mode to verify the applicability of the patch on the database environment.

Run the Deployment Procedures in Analyze Mode to do complete patchability check before the actual patching cycle. The below two modes are seen in the Review page during the Deployment procedure execution.

On submitting in ‘Analyze’ mode, the pre-req checks are run and you could check for the results with recommendations under RESULTS tab

2. Two new reports, accessible under Reports tab:
a. EM Target Patchability Report: Provides a comprehensive report on the automated patchability of the database environment managed by Grid control.
b. Patching Deployment Procedure Execution Summary Report: Provides a summary view of the deployment procedure's execution status across the given time period.

Deployment Procedures that are modified by the patch,
1. 'Patch Oracle Database'
2. 'Patch Oracle RAC - Rolling'
3. 'Patch Oracle RAC - All Nodes'
These procedures are accessible under 'Patching through Deployment Procedures'
in 'Deployments' page and versioned 5.1.

Related Metalink Note: 427577.1 : OMS and Agent Patches required for setting up Provisioning,Patching and Cloning in 10.2.0.3 to 10.2.0.5 GC

del.icio.us Tags: Patching,One-off

Oracle OpenWorld 2009 Vote-a-Session

Please vote for the following Oracle OpenWorld 2009 session on Oracle Mix.

S308732 Dell IT's Implementation of Oracle Enterprise Manager Patch
Provisioning

In order to do that, you must create an account on Mix. Go to:
https://mix।oracle.com/oow/proposals/10416-dell-it-s-implementation-of-oracle-enterprise-manager-patch-provisioning
Voting ends on midnight, July 3।

Another Fine Blog on Grid Automation

I just noticed another fine blog on Grid Automation:

http://blogs.oracle.com/gridautomation/

This blog is focussed on Enterprise Manager - Configuration Management and Provisioning, and is masterminded by Hariprasanna Srinivasan, who is the product manager for EM - Provisioning and Patch Automation pack.

Please do visit the blog.

Oracle ACE Director Meeting in USA

On an invitation from Oracle Corporation, I attended the Oracle ACE Director meeting at the Oracle HQ in Redwood Shores, California.

Oracle had everyone at the meeting sign a non-disclosure agreement so we cant say much about what was discussed. The main discussions were on Fusion Middleware.

One interesting probability for the future is that Weblogic server will be big in everything - including Enterprise Manager. This will enable Grid Control to scale enormously.

The ACE Director meeting was followed by the ODTUG conference in Monterey, California where I met industry stalwarts such as Tom Kyte, Don Burleson, and Cary Millsap.

A few photos of my trip to the Oracle HQ are enclosed.

RMAN via GRID CONTROL Article published on OTN

An RMAN via GRID CONTROL Article of mine has been published on the Oracle Technology Network (OTN). The article, based on real world experience, shows the power of Grid Control in setting up and scheduling RMAN backups of your database.

Architect: Grid
Oracle RMAN Backups: Pushing the "Easy" Button
By Porus Homi Havewala
How to redesign the setup and scheduling of Oracle RMAN backups the "easy" way, using Oracle Enterprise Manager
Published April 2009
http://www.oracle.com/technology/pub/articles/havewala-rman-grid.html

Please read. Hope you enjoy the article.

Applied Patches View

I was testing patching in Grid Control 10.2.0.4 a few weeks ago.
I found that when the Schedule Deployment Job executes, if the
patching step of the database completes, the Grid Control repository
view mgmt$applied_patches is not updated with the applied patch
immediately, but waits till the entire job completes.

If a later step fails, the entire job is marked as failed, and the
Grid Control Repository Applied Patches view is not updated,
even though the database has already been patched.

Instead of this, I felt, the Grid Control Repository Applied Patches
should be updated as soon as the patching step completes. Actually
it works in a different way. Let me explain.

MGMT$APPLIED_PATCHES is a view which gathers the information
from the following 4 tables in the repository:

mgmt_inv_container con,
mgmt_ecm_snapshot snap,
mgmt_inv_patch patch,
mgmt_targets tgt

The definition of the view is:

CREATE OR REPLACE FORCE VIEW "SYSMAN"."MGMT$APPLIED_PATCHES"
("PATCH", "BUGS", "INSTALLATION_TIME", "HOST",
"HOME_LOCATION", "HOME_NAME", "CONTAINER_GUID", "TARGET_GUID") AS
SELECT
to_char(patch.id) as patch,
ecm_util.concat_col('distinct BUG_NUMBER',
'mgmt_inv_patch_fixed_bug',
'PATCH_GUID = ''' || patch.patch_guid || '''',',') as bugs,
patch.timestamp as installation_time,
tgt.target_name as host,
con.container_location as home_location,
con.container_name as home_name,
con.container_guid,
tgt.target_guid
FROM
mgmt_inv_container con,
mgmt_ecm_snapshot snap,
mgmt_inv_patch patch,
mgmt_targets tgt
WHERE
con.snapshot_guid = snap.snapshot_guid AND
snap.is_current = 'Y' AND
snap.snapshot_type = 'host_configuration' AND
con.container_guid = patch.container_guid AND
tgt.target_name = snap.target_name
WITH READ ONLY;

As you can see there is a WHERE clause saying:
snap.is_current = 'Y' AND
snap.snapshot_type = 'host_configuration'.

Host configuration is manually executed at user request from
Deployment tabs or every 24 hours. If a patch is applied with
Opatch (ie. outside Grid Control at the unix or dos prompt)
then you MUST run the RefreshHostConfiguration job from
Deployments tab if you want the view updated immediately,
if not after 24 hours the info will be available.

If a patch is applied with Deployment procedures then there is
a step at the end of the DP that runs the RefreshHostConfiguration
job. If the whole DP job fails before reaching the
RefreshHostConfiguration job then of course this step
will not execute and the view will not be updated.

Thanks to the Oracle EM support team for this information.

My profile in Peer to Peer of Oracle Magazine

For the first time, I have been profiled in the Peer to Peer Section
in Oracle Magazine of May June 2009. Here is the page. What did I talk about? Grid Control, of course!

For space reasons, the magazine had to leave out some lines from the profile,
that I am reproducing here fully:

What technology has most changed your life? Oracle grid technology, including Oracle Enterprise Manager 10g Grid Control and all the great management packs. I am an Oracle Grid Computing enthusiast: I really like the advanced capabilities of Oracle Grid Control for tasks such as patching databases in an automated schedule, setting and scheduling Oracle RMAN [Oracle Recovery Manager] backups on the fly, and issuing SQL scripts and OS commands against any database or server. Oracle Grid Control has the potential to automate a lot of tedious DBA tasks.

What would you like to see Oracle, as a company, do more of? Oracle has demonstrated technical and thought leadership since its inception in 1977, the year of Star Wars. The most recent example of innovative technology is the announcement that we can use Oracle Grid Control to back up our databases directly to the Amazon Elastic Compute Cloud, thus negating the requirement for tape drives and offline storage of backups. My request would just be to keep up the technical and thought leadership. That’s what all Oracle DBAs appreciate.

What green practices do you use in your DBA work? I don’t print anything—I keep everything as electonic copies. Recently my office moved locations, and I was the only one to have a clean, green desk. All I had to move was my laptop and phone. The others were astonished.

Do you have a favorite vacation spot? There is no country as beautiful as India. I love the countryside, the undulating hills, and the villages. This is life at its most spiritual and simplest. The ocean shores of India are also very beautiful, and I love to visit these sandy beaches. I also like to vacation in Udwada in Gujarat, which is a pilgrimage spot for our ancient Zoroastrian religion. There’s a holy fire there that’s been burning for more than 1200 years.

Scripts Guru shows Database control

I just attended my fellow ACE Director Tanel Poder's 2-day "Celebrity" seminar in Singapore and I can say it is definitely one of the best troubleshooting and performance tuning seminars I have ever attended. Tanel has a unique "out of the box" approach to Oracle tuning in his extensive use of Unix-level utilities. His scripts are also amazing. I would definitely recommend his seminar to anyone else.

By the way Tanel recently became the second ACE Director in Singapore based on my recommendation of his community work to Oracle. It couldnt have happened to a more deserving person.

Everyone calls him the "Scripts Guru" since he has a script for almost anything (and even remembers all the names of his scripts, well almost always!). But what was amazing was that even this scripts Guru showed, right at the end of the seminar, that he was impressed with Database Control 11g! In particular, he demonstrated 11.1.0.7's use of Real Time Sql monitoring using DBMS_SQLTUNE.REPORT_SQL_MONITOR. This actually shows, in Enterprise Manager Database Control, where a SQL statement is in its execution plan AT THE SAME TIME the sql statement is executing! This requires the license of the Diagnostic and Tuning Packs. For more detail, have a look at this blog entry.

Oracle's Dev2DBA Newsletter of April 2009

Oracle's Dev2DBA Newsletter of April 2009 features in this issue:
"One Console to Rule Them All" - Oracle Enterprise Manager 10g Grid Control Release 5". In this newsletter, Justin Kestelyn, Editor-in-Chief, OTN, mentions:

"From admins of the Oracle stack, Oracle Enterprise Manager 10g Grid Control Release 5 is receiving the warmest welcome. With a new slate of database- and application-lifecycle management capabilities - including new support for Oracle WebLogic Server, Oracle Coherence, and Oracle VM - Release 5 takes cost-saving IT stack management to a new level."

This newsletter also mentions my article on Grid Control:

"Tech Article: Oracle Enterprise Manager Grid Control Architecture for Very Large Sites
Need to guarantee high scalability for Enterprise Manager Grid Control? This architecture, described by Oracle ACE Director Porus Homi Havewala based on real-world experiences, may be for you."

New EM Blog

I recently found a new EM Blog which was started in December 2008 by Oracle.
This would be the second blog dedicated solely to EM and we welcome it.
It has a lot of good information about the new Release 5 of EM10g:

http://blogs.oracle.com/oem/

Please visit the blog and see for yourself.

New EM Newsletter

You can now sign up to receive the new Enterprise Manager newsletter from Oracle.
Sign in to your OTN account, and click on "Account" at the top of the screen.
Then select the new EM newsletter which shows up in the "Opt-in to Oracle Communications" list as follows:

Oracle Enterprise Manager - News, reviews, and information on Oracle's solutions for managing, monitoring, and optimizing the full range of Oracle products including quality assurance, end-user experience monitoring, SOA runtime governance, complete IT service management, and more.

Retain historical data in EM Repository

Rajesh Bhogadi said...

"Is there any feature/solution within Grid to retain historical data from the repository which is helpful for capacity planning ex. database growth over the last x years etc?"

My answer:

Of course. As per the Enterprise Manager Advanced Configuration guide,

"Enterprise Manager aggregates your management data by hour and by day to minimize the size of the Management Repository. Before the data is aggregated, each data point is stored in a raw data table. Raw data is rolled up, or aggregated, into a one-hour aggregated metric table. One-hour records are then rolled up into a one-day table.

"After Enterprise Manager aggregates the data, the data is then considered eligible for purging. A certain period of time has to pass for data to actually be purged. This period of time is called the retention time.

"The raw data, with the highest insert volume, has the shortest default retention time, which is set to 7 days. As a result, 7 days after it is aggregated into a one-hour record, a raw data point is eligible for purging.

"One-hour aggregate data records are purged 31 days after they are rolled up to the one-day data table. The highest level of aggregation, one day, is kept for 365 days."

Section 9.2.3 explains how to increase the retention times for the raw or aggregated data:

To modify the default retention time for each level of management data in the Management Repository, you must insert additional rows into the MGMT_PARAMETERS table.
For increasing the retention in the one-day data table, connect as sysman and issue the command:

INSERT INTO MGMT_PARAMETERS (PARAMETER_NAME, PARAMETER_VALUE) VALUES ('mgmt_day_keep_window','');

Note the default is 365.

Regards,

Porus.

Clone Support Files

One Important note from the 10.2.0.5 patch Release notes:

As part of the upgrade of a Grid Control installation from 10.2.x.x to 10.2.0.3 or higher, you must download and install Clone Support Files from My Oracle Support and install the files onto each OMS as part of a post-patch configuration task to enable clone support for Oracle Tech stack components.

This step can often be overlooked and will result in issues for the cloning and provisioning subsystems.

Grid Control 10g Release 5 is out!!!

Grid Control 10g Release 5 is out since Wednesday morning, a lot of people were eagerly awaiting this release. I attended the Global launch Webcast on Tues., March 3, 9am PT which was 1 AM Singapore time. Richard Sarwal, Oracle SVP of Product Development, conducted the talk and slideshow. Towards the end of the demo, it was mentioned that Release 5 was now available for download from OTN. Both Windows and Linux versions are available. I downloaded it overnight and will install it soon. Files more than 2 GB for full download.

Main plus points of Release 5, from what I could gather, are:

1. Most stable release of Grid Control 10g and with the best performance
2. Manages Oracle VM for the first time
3. Manages Oracle Weblogic Server, apps and Service Bus
4. Real User Experience Insight (RUEI) with E-Biz, Peoplesoft and Siebel Accelerators
5. Oracle Communications Billing and Revenue Management
6. Oracle Beehive Management
7. Oracle Coherence Management
8. Supports ADDM for RAC for the first time
9. Real Time SQL monitoring, Automatic Sql tuning
10. Now supports 11g's new advisors such as Partition Advisor, Data Recovery Advisor
11. Now supports Database Vault
12. Secure Cloning and Masking = Automate masking of sensitive production data while cloning. Masking has been enhanced with deterministic masks, compound masks etc.
13. Application Diagnostics for Java = always-on, real-time, deploy on any Java server, JVM, or JRockit
14. Configuration Management detects unauthorized changes in real time. Continuous change capture for os, database, middleware and applications
15. Configuration Management now integrates Enterprise Manager with My Oracle Support so you can upload configurations directly to My Oracle Support
16. Database Replay automation now can capture workloads from 9.2.0.8 onwards.

Enjoy!!!

An article on using Grid Control for ITIL compliance

Just found an article written by David Yahalom:

"Using Grid Control for ITIL compliance"
http://www.davidyahalom.com/index.php/proactive-monitoring-with-oracle-grid-control/

Great article, he ends it with:
"And remember…. Once you go Grid, you’ll never go back!"

Voted leader of the Oracle RAC SIG (Special Interest Group) in Singapore

Today I was voted leader of the Oracle RAC SIG (Special Interest Group) in Singapore.
My name was proposed by RaviShankar Buddha of Oracle, seconded by Ashish Agarwal of
DBCON and accepted by the others. Thanks guys.

This means I will co-ordinate and conduct a RAC Round Table every month
in Singapore where clients using RAC are invited to share their experiences and
issues with RAC, like a peer discussion. At times presentations on RAC and
related technology can be held by Oracle or by any of us.

To my surprise I found most of the RAC users who came to the round table today
were not using Grid Control, only one of them was. RAC is very complicated and to manage it efficiently, we need good management software like Grid Control. For eg, rolling patch upgrades can be applied using the deployment procedures in Grid Control.

Recently Published Article on OTN

Please read "Grid Control Architecture for Very Large Sites":

http://www.oracle.com/technology/pub/articles/havewala-gridcontrol.html

This is an article of mine that was published today on OTN.

POC: Customized Patch Reports for Grid Control

This is part of the Grid Control POC we recently arranged for a large client.


As per the earlier post on the issue of patch reports, the supplied out-of-box
reports show the patch, bugs fixed, installation time, Host, Home directory
and Platform. The reports do not show the databases, the reason being that
the patch is actually at the Oracle Home level and not at the database level.
If you wish to add in the database name, it is possible to modify the report by
creating a copy and then modifying the Sql statement. For example the
above report is from this statement that can be modified:


SELECT distinct
patch as PATCH,
bugs as BUGS,
installation_time as TIMESTAMP,
host as HOST,
home_location as HOME_DIRECTORY,
platform as PLATFORM
from mgmt$applied_patches patch,
mgmt$em_homes_platform home,
mgmt$target tgt
where home.HOME_ID = patch.CONTAINER_GUID
and patch.target_guid = tgt.target_guid
and patch.installation_time>MGMT_VIEW_UTIL.ADJUST_TZ(??EMIP_BIND_START_DATE??,??EMIP_BIND_TIMEZONE_REGION??,tgt.TIMEZONE_REGION)
and patch.installation_time<= MGMT_VIEW_UTIL.ADJUST_TZ(??EMIP_BIND_END_DATE??,??EMIP_BIND_TIMEZONE_REGION??,tgt.TIMEZONE_REGION)

You can join this to MGMT$target_components which would have the same home as the mgmt$applied_patches home,
and then select the targets that are databases in this home.

SELECT distinct
patch as PATCH,
installation_time as TIMESTAMP,
host as HOST,
tgtcomp.target_name,
tgtcomp.target_type,
patch.home_location as HOME_DIRECTORY,
patch.home_name as HOME_NAME,
platform as PLATFORM
from mgmt$applied_patches patch,
mgmt$em_homes_platform home,
mgmt$target tgt,
mgmt$target_components tgtcomp
where home.HOME_ID = patch.CONTAINER_GUID
and patch.target_guid = tgt.target_guid
and tgtcomp.home_name = patch.home_name
and tgtcomp.target_type = 'oracle_database'
and patch.installation_time>MGMT_VIEW_UTIL.ADJUST_TZ(??EMIP_BIND_START_DATE??,??EMIP_BIND_TIMEZONE_REGION??,tgt.TIMEZONE_REGION)
and patch.installation_time<= MGMT_VIEW_UTIL.ADJUST_TZ(??EMIP_BIND_END_DATE??,??EMIP_BIND_TIMEZONE_REGION??,tgt.TIMEZONE_REGION)


This is just an example of the customization that is possible.


How do we know what info is in the Grid Control Repository? We recommend using the new Sql Developer
Data Modelling utility from Oracle. This is downloadable from the Oracle Technical Network (OTN) and currently
installs as a stand-alone utility, but will be incorporated into a new release of Sql Developer in the not-too-distant
future. ( Sql Developer keeps getting better and better, just like Grid Control ! )


The Data Modelling utility allows reverese engineering of an existing Oracle Database, you can access
this via the menu. Select File>Import>Data Dictionary. In this way, you can import form Oracle 9,10
and 11, MS SQL Server 2000 and 2005, DB2/390 7 and 8, DB2/UDB 7 and 8.


Use this to reverse engineer the SYSMAN schema in the Grid Control repository, especially the MGMT$ views.
Then you can understand what info is available.


As a test of the customization:

After applying database patch 7210195 via Grid Control to our FINPRD1 and FINDW1 database home,
we created a Grid Control customized report "Applied Interim Patches (Customized - Database Level)".


The Sql used in this report is:


SELECT distinct
patch as "Applied Interim Patch",
installation_time as "Time Applied",
tgtcomp.target_name as "Database Target Name",
patch.home_location as "Oracle Home Directory",
patch.home_name as "Oracle Home Name",
host as Host
from mgmt$applied_patches patch,
mgmt$em_homes_platform home,
mgmt$target tgt,
mgmt$target_components tgtcomp
where home.HOME_ID = patch.CONTAINER_GUID
and patch.target_guid = tgt.target_guid
and tgtcomp.home_name = patch.home_name
and tgtcomp.target_type = 'oracle_database'
and patch.installation_time>MGMT_VIEW_UTIL.ADJUST_TZ(??EMIP_BIND_START_DATE??,??EMIP_BIND_TIMEZONE_REGION??,tgt.TIMEZONE_REGION)
and patch.installation_time<= MGMT_VIEW_UTIL.ADJUST_TZ(??EMIP_BIND_END_DATE??,??EMIP_BIND_TIMEZONE_REGION??,tgt.TIMEZONE_REGION)
order by installation_time desc


As you can see, we have changed the headings of the field columns in the report, and we also changed the styled text in the
Report definition from “INSTR_APPLIED_PATCHES_ALL_HOSTS” to the customized line:


“The report shows the interim patches applied on Oracle Databases across all the hosts
in the last 31 days. Use the time period selector to view the interim patches applied within a time-period. Thanks.”


We also created a Report: "Applied Interim Patches (Customized - Database Level and Select which Database)" that does the above and also allows you to select which database.


The Sql statement for this is:


SELECT distinct
patch as "Applied Interim Patch",
installation_time as "Time Applied",
tgtcomp.target_name as "Database Target Name",
patch.home_location as "Oracle Home Directory",
patch.home_name as "Oracle Home Name",
host as Host
from mgmt$applied_patches patch,
mgmt$em_homes_platform home,
mgmt$target tgt,
mgmt$target_components tgtcomp
where home.HOME_ID = patch.CONTAINER_GUID
and patch.target_guid = tgt.target_guid
and tgtcomp.home_name = patch.home_name
and tgtcomp.target_type = 'oracle_database'
and tgtcomp.target_guid = ??EMIP_BIND_TARGET_GUID??
and patch.installation_time>MGMT_VIEW_UTIL.ADJUST_TZ(??EMIP_BIND_START_DATE??,??EMIP_BIND_TIMEZONE_REGION??,tgt.TIMEZONE_REGION)
and patch.installation_time<= MGMT_VIEW_UTIL.ADJUST_TZ(??EMIP_BIND_END_DATE??,??EMIP_BIND_TIMEZONE_REGION??,tgt.TIMEZONE_REGION)
order by installation_time desc


The line which selects the database target selected is “and tgtcomp.target_guid = ??EMIP_BIND_TARGET_GUID??”


Regards,


Porus Homi Havewala
(Oracle ACE Director)
Principal Consultant,
S & I Systems Pte Ltd.

Mentioned in Oracle's Dev2DBA Newsletter (Feb 2009)

The Grid Control Blog has been mentioned in the Feb 2009 edition of 

Oracle's Dev2DBA Newsletter (previously called Oracle Techblast), 

in the ACE Watch section as follows:

Visit Oracle ACE Director Porus Hami Havewala's blog for some interesting findings from a Grid Control POC he recently conducted for a large client.

To subscribe to Oracle Newsletters such as this popular newsletter, 

visit the following link:

http://www.oracle.com/newsletters/index.html?msgid=7360639#tech

More attempts to release the oci.dll lock

As a follow-up to the oci.dll locking issue and the use of unlocker that I explained

in an earlier blog post, when I conducted a session on "Patching with Grid Control",

a DBA suggested there was a Windows fix avaiable for the oci.dll locking issue.

I found Metalink Note 232827.1 which suggested that the svchost process

has the oci.dll locked on the machine. The note asked to shutdown Windows

services such as the Com+ Event System, MSDTC, and IIS. After

those services are shutdown the machine has to be rebooted in order for the

svchost process to release the oci.dll. The note suggested that these services

lock the oci.dll even though the user may not be using them.

I followed these steps, but it didnt make a difference. The ocii.dll

stayed locked even after the listeners were stopped.

The DBA then suggested to set the following registry key in Windows:

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer]

AlwaysUnloadDLL=1

I already had this registry key set. so this was not the solution. I had to

reply on the previous unlocker strategy (see the previous blog post).

A further bit of reading suggested that Windows keeps dlls in memory on

purpose for a certain period of time even after the dll is not is use. This is

for performance reasons.

I do wonder why Unix has no such issues and the performance is still ok

in unix without such "hold things in memory for some indefinite time"

tactics. We never have such locking issues on unix.

A side note on this issue: when I removed and reloaded the registry key

"AlwaysUnloadDLL=1" into the registry, from then on whenever I tried to

start the Grid Control Management service, it failed to load. I remembered

reading somewhere on the internet that this registry key may be causing

a Java load issue, so I deleted the "AlwaysUnloadDLL=1" key straight

away, and the Grid Control management service started loading fine.

Whew!

Windows - you can love it, you can hate it.

A New poll on patching

For patching your multiple Oracle databases, would you Use Grid Control Patch Deployment Procedures to automate, or do things the old-fashioned way? Or a bit of both? I have put a new poll on, go ahead and vote.

POC findings: Offline Patching of Previous CPUs and Modifying Reports

We were just going through a POC for Enterprise Manager Grid Control recently in a large client, and had some interesting results I like to share with everyone.

This particular client will not connect the Grid Control management service (OMS) to the internet for security reasons. This means the Refresh from Metalink job has to be run manually.

This has to be done after downloading certain xml files from Metalink (which contain the Patch Analysis metadata ie. which patches are available and what bugs they fix) and uploading these files, as per the following "Grid Automation Deployment Procedures" document on the Oracle OTN website:

http://www.oracle.com/technology/products/oem/pdf/grid-automation-deployment-procedures.pdf

After this the patch advisory on the Grid Control Home page rightly suggested that for a 9.2.0.1 home, the 9.2.0.8 patch as well as the October CPU had to be applied. Since the xml files had been downloaded manually from Metalink in December, the October CPU was the latest and therefore recommended to be applied.

The client wanted the earlier July CPU to be recommended instead. They said they rarely apply the latest patch but instead apply the earlier CPU patch. Now, the only way to do this would be to download the xml files from Metalink each month and store them on separate directories somwhere, but only upload the xml files to Grid Control as per the CPU you would like to be recommended by the Patch Advisory. For eg, if you want the Jul CPU to be recommended, then only upload the Jul or August or Sep xmls.

They also wanted a report to show all the patches applied on all their databases, showing each database one by one. The report as suggested was not fully available off-the-shelf in the reporting facility in Grid Control. The closest we come to this, is in this report:

Reports..Oracle Home Patch Advisories..Applied Interim Patches

This shows the patch, bugs fixed, installation time, Host, Home directory and Platform. It doesn’t show the databases, the reason being that the patch is actually at the Oracle Home level and not at the database level. This report may be appropriate for their needs, however if they wish to add in the database name, it is possible to modify the report by creating a copy andthen modifying the Sql statement. Oracle has mgmt$ views from which the info may be extracted.

Happy New Year and 11g Grid Control

This is the Blog's first post for the new year, so let me wish a Very Happy New Year 2009 to all the blog readers. All the Very Best for the New Year.

In 2009 I am looking forward to the release of 11g Grid Control, although I have no idea when it will be released. Hopefully, soon, this year.