Friday, May 30, 2014

Whitepaper: Best Practices for Managing Siebel Configurations

Contributing Author: Eunjoo Lee, Principal Product Manager, Oracle
The Oracle Enterprise Manager Applications Management team is pleased to announce the release of their latest whitepaper (available here) and associated screenwatch (available here) on configuration management for Siebel.
The task of managing configuration parameters in a dynamic, multi-target environment can be extremely challenging. A large, production scale environment can have dozens of unique targets, and each target can have several hundred different configuration parameters.
Oracle Enterprise Manager 12c, with the Siebel Plug-in, provides advanced, out-of-the-box tools to manage the complexity of Siebel configuration management. Oracle Enterprise Manager has advanced configuration capture, change detection, and comparison reporting capabilities that can dramatically reduce the amount of time that System Administrators spend on configuration management.
The purpose of the whitepaper and screenwatch is to provide customers with details on Oracle’s Best Practices for managing Siebel Configurations. It addresses several key areas including;
  • Understanding comparison templates in Oracle Enterprise Manager
  • Creating configuration templates for Siebel servers, components groups, and components
  • Using configuration templates to compare Siebel servers
  • Analysis of configuration comparison results
  • Using configuration templates for Siebel patch analysis
The benefits of using configuration templates to manage Siebel applications are substantial, and include:
  • Easily compare configuration parameters for targets within a Siebel Enterprise, or between two different Siebel Enterprises
  • Easily compare configuration parameters for a Siebel target against a gold standard or baseline.
  • Define comparison reports once and re-run multiple times
We hope this white paper and screenwatch help to provide you with a good starting point to manage your Siebel Application configurations more efficiently and pro-actively.
Contributing Author: Eunjoo Lee, Principal Product Manager, Oracle

Quickly Diagnose the Root Cause of Stuck Threads using Oracle Enterprise Manager 12c JVM Diagnostics

One of the hidden gems in Oracle Enterprise Manager 12c is JVM Diagnostics. If you purchased the Weblogic Management Pack license then you already own it. JVMD allows administrators to diagnose performance problems in production Java applications. By eliminating the need to reproduce these “production only” problems in QA, it reduces the time required to resolve them. It does not require complex instrumentation or restarting of the application to get in-depth application details. Application administrators will be able to identify Java problems or database issues that are causing application downtime without any detailed knowledge of the application internals. It is also very well suited to diagnosing issues with “Stuck Threads” which will be the focus of this blog.
What is a [STUCK] Thread
In a Weblogic server, all incoming requests are handled by a thread pool which is controlled by a work manager. Worker threads that are taken out of the pool and not returned after a specified time period are marked as [STUCK] by the work manager. This time period is 10 minutes by default but it is configurable on a per work manager basis using the "StuckThreadMaxTime" parameter (default is set to 600 seconds).
Note that it is possible that some of your threads are doing legitimate work for over 10 min with no issues. If you have such threads then you should consider placing them in a another work manager with proper setting for the "StuckThreadMaxTime" parameter
Why JVMD is Well Suited to Diagnosing [STUCK] Threads
Traditionally, developers will use a stack trace generated by jstack or kill -3 and try to determine the cause of a stuck thread. However, in my experience a majority of the time this stack is not even the culprit. The problem often lies in another tier of the application or even in another thread of the same application. JVMD has the ability to provide additional context such as the name of the request and which tier it called out to Eg: RDBMS servers, LDAP servers, Web servers, RMI servers, etc. Using fine grained thread states (i.e. DB, Network, IO, CPU, RMI, Lock, etc) and the ability to see additional details about the thread, JVMD users can quickly pinpoint the root cause of the problem. Since JVMD is always on, it can also debug these issues that happened in the past and can proactively notify you about stuck threads Eg: Get an email at 1am when you had stuck threads. And lastly, sometimes developers have no access to the target host due to lack of credentials needed to run command line applications.
On several occasions, the thread may be stuck but is doing legitimate work. In such scenarios JVMD allows you to scan back and forth through a large number of samples to see what work is being processed by the thread. In addition, you can take a look at other threads that were serviced the same request to see if they behaved similarly or not. This will allow you to quickly determine whether there is really a problem or not.
Real-Time [STUCK] Thread Analysis
With JVMD there are two real use cases for stuck thread analysis. If you get notified about a stuck thread in real-time (via email, etc) then you can perform a real-time stuck thread analysis. Alternatively, if you are investigating a thread that was stuck in the past but is not present any more, then you can perform a historical stuck thread analysis. In either case the first thing to do is to navigate to the JVM (or JVM pool) where the thread is stuck. We do this by clicking on Targets -> Middleware as shown
From here we can filter the list of targets by target type or by target name. Your most recent filter request will be remembered the next time you visit the page. Select the Target Type of JVM to see all of the JVM targets.
Pick the JVM for the Weblogic server which is having the stuck thread issues and click on it. This will take you to the target home page. Click the button at the top that says “Live Thread Analysis”. Type the word "stuck" into the thread name search box and click on the arrow to filter the table. Now you should see all the stuck threads. In this case we can see a thread that is stuck in the “Network Wait” state. It is stuck on line 358 in function writeBuffer() of which is in package com.sun.ssl.internal.ssl which makes it clear that this stuck thread has made an SSL call and the remote server has not responded in a reasonable amount of time so the client thread is stuck.
Here is another example of stuck threads, this time in the “DB Wait” state. Notice how the tool tip over the SQL ID field shows the SQL being executed. Click on it to view longer SQL statements. Also try clicking on the DB Wait link which takes you directly to this specific database session in the Oracle Database Diagnostics section of EM for further analysis. The columns displayed are controlled by “View” drop-down menu. Here we added the “User” column to show the logged in user who executed the request.
Historical [STUCK] Thread Analysis
In order to start historical stuck thread analysis you need to navigate to the JVM target home page in the same way as discussed in the real-time section. From the target home page clicking on the “JVM Performance Diagnostics” button at the top of the page. On the performance diagnostics page you can filter the data to make it more relevant to your task. The first filter to apply is of time. If you know the exact time you can use the “Edit Date and Time” button to specify it as shown. Otherwise use the handy shortcut links for Day, 1 Hours, 1 Hour or 15 Minutes as needed.
The next thing to filter is the Thread Name. Expand the filter options region if necessary and add the Thread Name filter to be “[STUCK]*” so you only see threads whose name starts with [STUCK]
Below the filter region the “General” tab will show you the Thread States, Top Requests, Top Methods, Top SQLs, Top DB Wait Events and Top Databases – only for the filtered data i.e. for only threads that are stuck. Try clicking on method names to see the call stack for the method. The charts are all interactive and fetch additional data about the item clicked.
If you want to find a specific thread move from the “General” tab over to the “Threads” tab. This is fine grained data with each sample and state transition visible. You can click on any sample to view it in the sample analyzer which should look familiar to you if you saw the threads in real-time. Details about SQLs, Wait states, etc are all available here also along with the complete call stack which can also be exported to a CSV file.
In conclusion, we can see the JVMD provides a rich set of additional details which are only a mouse click away that help you to diagnose the root causes of your stuck threads.
NOTE: Many of the screen shots taken here are using testing & debug code, which deliberately tries to create stuck threads. This does not and should not reflect on the nature of any Oracle products being shipped to customers.

The case for Snap Clone over point tools

Today, I stumbled over a competitor blog, conspicuous by its factual incorrectness on Enterprise Manager Snap Clone. However, I must compliment the author of the blog, because inadvertently, he has raised a point that we have been highlighting all along. The author, with reference to Dataguard and storage technologies, argues against the cobbling of technologies together and adding another technology stack to the mix without any automated management.  
Precisely the point! In the wide realm of technologies, there are necessities and there are accessories aka nice-to-haves. The necessities are technologies that are needed anyway, such as a high fidelity, high performance storage from a reputed vendor or a good DR solution for a mission critical database environment. Similarly, for any Oracle DBA worth his/her salt, Enterprise Manager 12c is a necessity, a part of the daily life. The Enterprise Manager agent, keeping vigil on every host, is therefore not an overhead, but the representative (the "agent" in true sense) of the DBA. Deep diagnostics, performance management, large scale configuration management, patching and compliance management make Enterprise Manager the darling of any Oracle DBA. All surveys suggest that any DBA spends considerable amount of time in Enterprise Manager for performing things beyond just data cloning, so why invest in an accessory for the cloning of Oracle test databases and unnecessarily proliferate the number of point tools (and possibly several instances of them) that you need to manage and maintain, not to ignore the past history that cites that very few such point tools solved customers' CAPEX and OPEX problems over the long run. It is like using spreadsheet for expenses and ERP for all other financial tasks.This is not to suggest that these point tools do not have good, innovative features. Over my tenure in the industry, I have come across several such tools with nice features, but often the hidden costs outweigh the benefits. Our position in this aspect has been consistent, whether it is on a competitor’s tool or our own. Few years back, we integrated My Oracle Support into Enterprise Manager with the same consistent goal that Enterprise Manager will serve as the single pane of glass for the Oracle ecosystem. Same has been our position on any product that we acquire.
Snap Clone's support for Dataguard and native storage stems from popular customer demand to leverage technologies they already invested in, and not create standalone islands of automation. Moreover, several customers have voiced in favor of the performance and scalability advantages that they would get by leveraging the native storage APIs. How else would you support one of the world's largest banks, a Snap Clone customer, who performs 60,000 (sixty thousand) data refreshes per year! In any case, that should not imply that we bind ourselves to any of those technologies. We do support cloning on various storage systems based on ZFS filesystem. Similarly, the Test Master refresh can be achieved with one among RMAN, Dataguard, Golden Gate or storage replication and optionally orchestrated with EM Job System.
Enterprise Manager 12c has taken a great step in delivering features via plugins that can be revisioned independent of the framework. An unwanted side effect is that the awareness often lags what is actually supported in the latest version of the product. For example, the filesystem support was introduced last Fall. And of course Enterprise Manager 12c Snap Clone supports RAC. My esteemed colleague and DBA par excellence, in her blog has highlighted some of these to dispel some of the prevalent awareness issues. Snap Clone's usage among the E-Business Suite and Developer community does not need any special accreditation. It is heavily used by the world's largest E-Business Suite Developer community-the Oracle E-Business Suite Engineering organization itself! It is true that Snap Clone does not support  restoration to any arbitrary point in time, but then our customers and prospects have not voiced a need for it. In reality, most customers want to perform intermediate data transformation such as masking and subsetting as they clone from production to test, and Enterprise Manager 12c already boasts of sophisticated data masking technologies, again via the same interface. It also includes testing features like Real Application Testing (RAT) that can complement and follow the test database creation. Future releases of Enterprise Manager will support a tighter integration among these features.
Snap Clone is delivered as a part of the Database as a Service feature set that has been pioneering, industry-leading and getting adopted at a great pace. Little wonder that we have already received a copious amount of Openworld paper submissions on the topic. In this emerging trend of DBaaS adoption, we find no reason to fragment the tasks such as fresh database creation, pluggable database provisioning and cloning across silo'ed point tools (not to mention broader PaaS capabilities which may be needed for complete application testing). Each use case could be different but needs a single service delivery platform. EM12c is that platform for Oracle. Period. So, think twice before 'adding another technology to the mix'. You do not need to.

Friday, May 9, 2014

Leveraging Target Properties to Enhance Enterprise Manager Capabilities

By Courtney Llamas

Do you still maintain a spreadsheet with Database or Server contact or business unit ownership?  In Oracle Enterprise Manager 12c (EM) Target Properties allow you to store descriptive target information, such as Contact or Location, which can then be used in dynamic/administration group definition, reports, incident rules and notifications.   This blog will show you how you can better leverage the features of EM to store your configuration data and utilize it to the fullest extent. 
The out-of-the-box target properties include:
  • Contact
  • Cost Center
  • Customer Support Identifier
  • Department
  • Life Cycle Status (Development, Test, Staging, Mission Critical, Production)
  • Line of Business
  • Location
  • Operating System – populated by collected configuration data
  • Platform Target Type – populated by collected configuration data
  • Target Version – populated by collected configuration data
LifeCycle Status
The LifeCycle Status target property has a special role within EM 12c.   The values are restricted to one of the predefined options:  Development, Test, Staging, Production, Mission Critical.   The value of this property is used by EM when it is under heavy load in prioritizing incoming alerts and data.   Mission Critical and Production targets are given the highest priority so that alerts always get loaded.   Targets of Development or Test status may be asked to "back off" until the load returns to a manageable level.  You can not add to these values, but you can modify the display name with EM CLI.   For example, you could use the command below to rename the Test status to QA.   
$ emcli modify_lifecycle_stage_name  -name="Test"  -new_name="QA" 
Setting Target Properties
To set target properties on an individual target, you can select the target menu (i.e. Host, Oracle Database) then selectTarget Setup / Properties and click the Edit button. This will provide an input screen such as the following:
As you can see, the Operating System, Platform, Target Version and Customer Support Identifier are not editable.  These are inherited from settings elsewhere or through configuration data collected by EM.
The most efficient time to set target properties is when you’re discovering or promoting a new target.   You will see a button Set Global Target Properties which will set the target properties for all selected targets. 
This will open a dialog box where you can set the target properties for all selected targets.  
If you're unable to set the properties while discovering a large number of databases, your best bet is to bulk set target properties using EM’s Command Line Interface (EM CLI).  In the example below we set the group MyProdGroup to a LifeCycle Status of Production.  The flag –propagate_to_members can be used to cascade target properties to a targets members, such as a group, domain or cluster.  In this example, all members of MyProdGroup would have a LifeCycle Status of Production.
$ emcli set_target_property_value 
-property_records="MyProdGroup:composite:LifeCycle Status:Production" 
Using EM CLI can be very helpful when setting target properties on a large number of targets, or when you want to modify a large number of targets (e.g. changing Department of Support to Operations).  For a sample script that can be leveraged for setting target properties in bulk please see the previously published blog Using EM CLI for mass update of Lifecycle Status Property Value.
User Defined Target Properties
EM also allows you to create custom target properties, or User Defined Target Properties.  While you cannot use custom target properties in dynamic or administration group selection criteria, you can use them for reporting and notifications.  Creating custom target properties is accomplished by using EM CLI and providing the property name and the target type it applies to.  While it is possible to set a custom property for just one target type, it is recommended to standardize these custom properties across all target types as they are stored in a sequential fashion (udtp_1, udtp_2, etc).  If you set per target type, udtp_1 may be Owner on Host targets and Function on Oracle Database targets.  This can cause confusion in reporting and notifications as you will see later.   
To add a custom target property for Owner, we use the EM CLI command below:
$ emcli add_target_property –target_type=”*” –property=”Owner”
Next, we’re going to set the value of the Owner target property to Courtney for a particular target (emrep) of target type (oracle_database).
$ emcli set_target_property_value 
On the following screen you can see that we’ve got a new property “Owner” with a value set to Courtney.
Utilizing Target Properties
All target properties are stored in the MGMT_ALL_TARGET_PROPS table with corresponding internal names (udtp_1, udtp_2, etc.).   The following query will show you what the internal names are:

SQL>  select property_display_name, property_name      from mgmt$all_target_prop_defs;
PROPERTY_DISPLAY_NAME          PROPERTY_NAME------------------------------ -------------------------Comment                        orcl_gtp_commentContact                        orcl_gtp_contactCost Center                    orcl_gtp_cost_centerCustomer Service Identifier    orcl_gtp_csiDepartment                     orcl_gtp_departmentDeployment Type                orcl_gtp_deployment_typeLifeCycle Status               orcl_gtp_lifecycle_statusLine of Business               orcl_gtp_line_of_busLocation                       orcl_gtp_locationOperating System               orcl_gtp_osPlatform                       orcl_gtp_platformTarget Version                 orcl_gtp_target_versionOwner                          udtp_1
Now you can access the custom target properties through their internal names in custom notification scripts. For instance, OS Command notifications can call the environment variables such %UDTP_1%, %ORCL_GTP_LIFECYCLE_STATUS%, etc.    
Properties can also be accessed in reports against the MGMT$ view tables, which will allow you to build custom reports that show additional target information such as Owner or LifeCycle Status. 
SQL> select t.target_name, t.target_type,
       d.property_display_name, p.property_value
     from mgmt$target t,          mgmt$target_properties p,          mgmt$all_target_prop_defs d     where t.target_guid=p.target_guid       and p.property_name=d.property_name       and p.property_value is not null;
TARGET_NAME          TARGET_TYPE      PROPERTY_DISPLAY_NAME PROPERTY_VALUE-------------------- ---------------- --------------------- --------------emrep                oracle_database  Target Version                oracle_database  Owner                 Courtneyemrep                oracle_database  Operating System      Linuxemrep                oracle_database  Platform              x86_64
By leveraging the out-of-the-box Target Properties and the User Defined Target Properties, you can easily extend the flexibility of EM’s groups, incident rules, reports and notifications to suit your business needs. For additional information on Target Properties used in Administration Groups see the Oracle Enterprise Manager Cloud Control 12c Administrator's Guide.   

By Courtney Llamas


Opinions expressed in this blog are entirely the opinions of the writers of this blog, and do not reflect the position of Oracle corporation. No responsiblity will be taken for any resulting effects if any of the instructions or notes in the blog are followed. It is at the reader's own risk and liability.

Blog Archive