Saturday, July 27, 2013

Thomas Kurian on Oracle Enterprise Manager


Thomas Kurian, Oracle's EVP of Product Development explains in succinct detail the enhancements in Oracle Enterprise Manager.



This is a great video with lots of information.

Regards,

Porus.

Friday, July 26, 2013

Oracle Enterprise Manager 12c Release 3: What’s New in EMCLI


If you have been using the classic Oracle Enterprise Manager Command Line interface ( EMCLI ), you are in for a treat. Oracle Enterprise Manager 12c R3 comes with a new EMCLI kit called ‘EMCLI with Scripting Option’. Not my favorite name, as I would have preferred to call this EMSHELL since it truly provides a shell similar to bash or cshell. Unlike the classic EMCLI, this new kit provides a Jython-based scripting environment along with the large collection of verbs to use. This scripting environment enables users to use established programming language constructs like loops (for, or while), conditional statements (if-else), etc in both interactive and scripting mode.

Benefits of ‘EMCLI with Scripting Option’

Some of the key benefits of the new EMCLI are:
  • Jython based scripting environment
  • Interactive and scripting mode
  • Standardized output format using JSON
  • Can connect to any EM environment (no need to run EMCLI setup …)
  • Stateless communication with OMS (no user data is stored with the client)
  • Generic list function for EM resources
  • Ability to run user-defined SQL queries to access published repository views
Before we go any further, there are two topics that warrant some discussion – Jython and JSON.

Jython

Jython is the Java implementation of the Python programming language. I have been working with Python (or CPython) and Jython for the last 10 years, and to me it is the best scripting language ever. It is fun, easy to learn, the syntax is simple, is self formatted, and is dynamically typed. This comic from XKCD summarizes it the best:


There are numerous tutorials for Python/Jython on the web, so feel free to pick anyone you like but just remember that the Jython version supported by the new kit is v2.5.1.

JSON

JSON stands for JavaScript Object Notation. It is a data interchange format, much like XML, which is easier to read and write for both humans and machines, but unlike XML it contains very little metadata (elements and attribute names). JSON format is quite simple; it basically represents data as a collection of name/value pairs. These pairs can be contained within arrays, lists, or maps. Here is a sample:
{"menu": {
"id": "file",
"value": "File",
"popup": {
"menuitem": [
{"value": "New", "onclick": "CreateNewDoc()"},
{"value": "Open", "onclick": "OpenDoc()"},
{"value": "Close", "onclick": "CloseDoc()"}
]
}
}}
JSON is quite popular. You will often find it used with REST based web services APIs or even with some modern databases like MongoDB. Most programming languages provide libraries to work with JSON.
The EMCLI kit uses JSON as its output format as well. Many of the verbs return output in JSON format for ease of programmatic use. I say many, since there are still some verbs that don’t, but this is only matter of time.
Now let’s get back to EMCLI.

Steps to setup the kit for ‘EMCLI with Scripting Option’

1. To download the new EMCLI kit, go to Setup->Command Line Interface. Here you will notice the new section for ‘EMCLI with Scripting Option’. Click on the link to download the kit to your desktop or desired server.
Download

You can also download the kit directly from the following url:
http(s)://:/em/public_lib_download/emcli/kit/emcliadvancedkit.jar
 
2. Copy the kit (emcliadvancedkit.jar) to a directory where you wish to install EMCLI
kit
3. To install, run the following command. Note that we need the Java version to be 1.6.0_43 or greater.
java -jar emcliadvancedkit.jar client -install_dir=
Verify Java version 
4. The last step to complete the setup is to run ‘sync’. Before using EMCLI you have to connect to the OMS to install all verb-related command line help. In classic EMCLI, this happens automatically when you run the ‘setup’ command. But in the new EMCLI, since we do not run setup, we run the ‘sync’ command instead.
The ‘sync’ verb now accepts some additional arguments. Run the following command:
emcli sync -url=http(s)://:/em -username= -trustall
It will prompt for the user password and then take a few minutes to download and install all the help content.
emcli sync
5. Now confirm the setup with a simple test. We do this using the interactive mode. Just run ‘emcli’, and once you see the prompt run ‘help()’. This will print list all verbs along with their description.
emcli interactive mode
With the setup complete, let’s now have some fun.

Using the ‘EMCLI with Scripting Option’

Connect to the interactive mode by running ‘emcli’ from the command prompt. Now try the following commands:
1. Basic Jython: Since EMCLI is built using the Jython interpreter, you can run Jython commands at the EMCLI prompt. For example, you can try the following:
>>1+2
>>print “Hello Jython”
>>mylist = [1,2,3]
>>print mylist
Jython test
2. EMCLI Status: Next, print the status of the EMCLI session using the ‘status()’ command.
emcli status
You will notice that the EM url and user are not set. To do this we have to set the client_properties. Run ‘help('client_properties')’ for more details.
client properties
The help text instructs us to set the client properties to connect to a specific EM environment. The 4 properties of interest to us are the following:
Name
Details
EMCLI_OMS_URL
The EM url
EMCLI_USERNAME
The EM user to connect as. We will use the login() function to set this.
EMCLI_TRUSTALL
I like to set this to true, but the default is false.
EMCLI_OUTPUT_TYPE
I like to set this to JSON even for interactive mode
To set these properties run the following:
>>set_client_property('EMCLI_OMS_URL','http(s)://:/em')
>>set_client_property('EMCLI_TRUSTALL','TRUE')
>>set_client_property('EMCLI_OUTPUT_TYPE', 'JSON')
>>login(username="",password="")
You should see the message on successful login. Now we are connected to EM.
login
3. Understanding help and verb invocations: Most of the help text presented in EMCLI is tailored towards the classic interface. Since Jython is a programming language, verb invocations are done in the function form. There is a simple mechanism for converting the classic invocation format for use in both interactive and scripting mode. Let’s use the login() verb as an example.
The EMCLI help for login is as follows:
>>help('login')
emcli login
-username=
[-password=]
[-force]
This means, when using classic EMCLI, you would invoke it as follows:
emcli login –username=”foo” –password=”bar” -force
Instead, in the interactive or script mode, the invocation would look like:
login(username="",password="",force=True)
Essentially, all verbs are now functions, and all arguments to the verb are now parameters passed to the function. Since the –force argument does not take any value, it is treated as a Boolean in Jython and takes the values of True or False.
Note: The -force parameter in the login() function is not applicable to the interactive or script mode, but is being used in this example to explain the concept of passing Boolean values. Again, you should never use the -force parameter in the interactive or script mode.
Another such conversion that you may come across is for list of values. For example,
In classic EMCLI, some verbs will ask for the same attribute to be repeated with varying values to represent a list.
emcli grant_privs -name='jan.doe' 
         -privilege="USE_ANY_BEACON"
         -privilege="FULL_TARGET;TARGET_NAME=host1.acme.com:TARGET_TYPE=host"
In interactive or script mode, you can use native Jython listes instead and pass it as parameters. In Jython, lists are represented within square brackets ([]).
>>priv_list = ['USE_ANY_BEACON','FULL_TARGET;TARGET_NAME=host1.acme.com:TARGET_TYPE=host']
>>grant_privs(name='jan.doe',privilege=priv_list)
4. Sample Use Case: Let’s take a very simple use case to demonstrate the interaction with EMCLI in the interactive mode. So our sample use case is to ‘List all targets of type oracle_database and those whose name starts with the characters ‘db’”.
For this use case, we will make use of the new generic ‘list’ verb. Traditionally, each feature in EM provided its own verbs for list, get, show, and describe. Rather than working with multiple such variants, the new generic ‘list’ verb takes a page from the REST web service specification and provides a generic action that can work against different EM resources.
To learn more about this verb, we ru:
>>help('list')
help
The help text shows us the format of this verb. Essentially, there are 3 parameters that we care about:
  • resource = the EM resource which is to be queried
  • columns = specify the different resource attributes to display
  • search = filters to narrow down the result
First, we need to know the list of resources that are supported by this verb. For this we run
>>List(‘help’)
list help
From the output, it is obvious that for our sample use case we want to query the Targets resource.
Second, we need to know which columns are supported by the Targets resource. For this, we run
>>list('help',resource="Targets")

help resourcesl

From the output, we can determine that we need the column related to target name and type. With this we have all the information we need to construct the final function call for our sample use case.
For ease of explanation, I will break down the process of determining the final function call into small incremental steps. Once you gain proficiency, you will be able to define this function in a single pass.
       1. List all targets in the EM environment. For this we run,
>>list(resource="Targets")
This command will spew a lot of text on your screen as there are likely to be numerous targets in your EM environment. So instead of listing all of them on the screen, let’s just get a count. For this, we need to understand the output format of this verb.
Any function that you run in the interactive or script mode returns an object of class Response (). The Response class has 4 key methods:
Function
Description
out()
Provides the verb execution output. The output can be text, or the JSON.
 isJson() method on the Response object can be used to determine whether the output is JSON.
error()
Provides the error text (if any) of the verb execution if there are any errors or exceptions during verb execution.
exit_code()
Provides the exit code of the verb execution. The exit code is zero for a successful execution and non-zero otherwise.
isJson()
Provides details about the type of output. It returns True if response.out() can be parsed into a JSON object.
So let’s look at a code snippet. snippet
For the first function call to list all targets in EM, we store the results into a variable called ‘all_tgts’. This variable contains the response object. ‘all_tgts.out()’ will give us the actual output. The output returned is in JSON format which automatically gets converted into a Jython dictionary (collection of name-value pairs represented by curly brackets). The output dictionary has a key name called ‘data’ which contains all search results in the form of a Jython list as its value. Finally, len() is a native Jython function which returns the number of elements in a Jython list. As seen in the output, we found 878 targets in the EM environment which is clearly not what we desire.
 2. Now we add search parameters to filter our results. We add two search filters, first the target type should be equal to oracle_database, and second the target name be like db%. You can add multiple search filters to the function call, but all these filters should be encapsulated in a Jython list. The search filter supports various operators: =, !=, >, <, >=, <=, like, null, and not null. Similar to a SQL query, you can also control which columns are to be displayed in the output.
So let’s run our final function.
>>search_filters=["TARGET_TYPE ='oracle_database'","TARGET_NAME like 'db%'"]
>>list(resource="Targets", columns="TARGET_NAME,TARGET_TYPE", search=search_filters)
The formatted output looks like this. As mentioned before it is in the form of a Jython dictionary which can be easily accessed programmatically. The value of the ‘data’ key is a Jython list that contains all search results, while the other keys provide other metadata related to the result.
{
'exceedsMaxRows': False,
'columnHeaders': ['TARGET_NAME', 'TARGET_TYPE'],
'columnLength': [256, 64],
'columnNames': ['TARGET_NAME', 'TARGET_TYPE'],
'data':


[

{'TARGET_NAME': 'db9328.acme.com', 'TARGET_TYPE': 'oracle_database'},

{'TARGET_NAME': 'db3092.acme.com', 'TARGET_TYPE': 'oracle_database'},
],
'filler': '\n\n\n'}
You must have noticed that I hardly talk about the scripting mode. This is on purpose, as I believe that interactive mode is the best interface to learn the new EMCLI. Once you master the interactive mode, converting your code snippets into a script is fairly easy. In future blog posts, I will cover scripting mode and numerous other use cases that seem like a perfect fit for the new EMCLI.
In summary, ‘EMCLI with Scripting Option’ is a new kit that is built on top of a Jython interpreter. It is much superior to the classic EMCLI, as it provides a complete programming environment with the ability to use native Jython functions and primitives. The output is presented in the JSON format which is both human and machine readable, and avoids the need for parsing text output. The client is completely stateless, which means no user data is stored with the client. This means numerous sessions can be launched from a single client, each connecting to a different EM environment, and as a different user.
I encourage you to play around with this new EMCLI kit, and post the different use cases that you found interesting and would benefit the community. You can reach me on twitter @AdeeshF.

Additional Reading:

Harness the power of Configuration Search to know what's out there and drive automation


Oracle Enterprise Manager collects and monitors configuration information for every target it manages and layers support for sophisticated lifecycle operations on top of this foundation. Within the configuration management area itself Enterprise Manager has formalized support for Drift Management, Inventory Management, Topology visualization and Compliance Management. Although these features cover a good segment of configuration management use cases there will always be additional uses for this important information. Enterprise Manager makes it easy to leverage this valuable information both inside and outside the product using the Configuration Search feature.
Within the Enterprise Manager UI, users can build sophisticated queries against Enterprise Manager’s configuration management repository to generate reports without writing even one line of SQL. If the Oracle provided Search library does not already contain a matching search a user can build a search to their exact specification completely graphically.
Outside of the Enterprise Manager UI, users can find and run saved searches using the EMCLI in both interactive and the new script mode. When used in a script, configuration search results can be used to drive other lifecycle operations like patch automation and provisioning.
In this article we will take a closer look at the Configuration Search feature using some common real world examples.
Arguably one of the most important configuration items collected by Enterprise Manager is applied patches. Finding the location of applied patches can cause some confusion at first owing to the new target model introduced in Enterprise Manager 12c. Oracle Home is now a separate and proper target with its own configuration collection which includes patch information. This makes great sense as patches are in fact applied to the Oracle Home and not the software running out of it.
The question is how can you figure out which targets ( ie databases ) are using which Oracle Homes? The answer is using relationships. Enterprise Manager 12c now discovers and collects relationships between targets. These relationships include both physical (observed) and logical (inferred from configuration). As an example, all databases running out of a given Oracle Home will have an “Installed At” relationship to its specific Oracle Home target. These relationships can be graphically viewed using the topology viewer available under the configuration menu of all targets. They can also be used when building a Configuration Search when starting with a well known target like database instance.

Find all single instance databases with Advanced Compression option that do not have a patch applied. – Step by Step

Let’s build a configuration search to find all single instance databases with Advanced Compression option that do NOT have a patch applied to their Oracle Home. Since patches are typically specific to a version let’s narrow it down to version 11.2.0.3 databases and patch 14275605. ( Database Patch Set Update : 11.2.0.3.4 )
1. Start by navigating to the Configuration Search Library. Enterprise->Configuration –>Search…
2. Click Create to start building a new Configuration Search.


3. Select Database Instance from the Target Type list of value.

Next we need to narrow the list of databases to those of version 11.2.0.3 and single instance. To do this we will use the target model to choose properties which contain this data so we can filter it further.
4. Click Properties on the Database Instance row.
5. Open the Target Properties and Instance Information folders and Select Property Name, Property Value,Version, Name and Selected as shown.

6. Click OK.
To filter down the results, we enter criteria into the text boxes to the right of the properties.
7. Enter 11.2.0.3 next to Version.
8. Enter ‘Advanced Compression’ for Name and ‘TRUE’ for Selected under Database Options
9. Select ‘Metric Scope’ for property Name and Enter ‘DB’ for value. ( Metric scope can have a value of DB for single instance and RACINST for RAC instances. )
Your search should look something like this:

At any point while you are creating a Configuration Search, you can see how your search is coming along by clicking Search. Doing so at this point will show results similar to the results shown here. ( Note: If you are not interested in seeing the results of a column you can uncheck the property to remove it from the results. )

At this point we need to pull the Oracle Home target into the picture to get at the applied patches configuration information.
10. Click Relationships on the Database Instance row.
11. Choose “Oracle Home” as the Destination Target Type then Click Search. This should result in one relationship type “Installed At”. Select this row and click OK.

We now have something that looks like this:

To add collected patch information from the Oracle Home target we need to use the target model again.
12. Click Properties on the Oracle Home row.
13. Open the “Patches installed in Oracle Home” folder and select “Patch ID” property.
14. Click OK.

15. Enter 14275605 in the text box next to “Patch ID” to narrow the results to this patch.
16. Click Search. You should see something similar to the results below.

But wait, this shows the databases that HAVE patch 14275605 installed. We are after databases that DON’T have this patch installed.
Fortunately we can achieve this result by using the “Advanced Options” capabilities.
17. Click the “Advanced Options” button on the “Patches installed in Oracle Home” row. ( Be sure to select the correct one! )

18. Change the Condition in the resulting dialog box to “NOT EXISTS”. ( The explanatory text shown just happens to use patch search as an example. )
19. Click OK.

Notice the addition of “Condition : NOT EXISTS” on the “Patches installed in Oracle Home” row. This will show targets in which none of the targets matches the criteria. In our case, an Oracle Home may have hundreds of patches applied. Only if none of the patch IDs equal 14275605 will the target be in the results.

20. Click Search.
This time, the results finally display what we are after. That is “11.2.0.3 Single Instance database with Advanced Compression option that do NOT have patch 14275605 applied.”

21. Click ‘Save As’ to save the search with the name “11.2.0.3 SI AC DBs without patch 14275605”.
22. Click OK.

The library now shows our new search. You or any other user can run the search by selecting it and clicking Run. You can modify it by using Edit or make a copy with Create Like to continue to refine it without affecting the original.

Running Configuration Search using Interactive EMCLI

As mentioned at the opening, Enterprise Manager Release 3 now supports the execution of saved Configuration Search from the EMCLI. There are two verbs with which you can run configuration searches: get_targets and run_configuration_search
The get_targets verb has been available since Release 1 but now has an additional switch to specify a configuration search. This results in a standardized result containing the Target Name, Target Type and Status.
Here is an example using the configuration search we just built.

The run_config_search verb generates results exactly as you see them in the results of the configuration search. The results are a little harder to read but the output could be re-directed to an output file for import into something like a document editor or spreadsheet for easier viewing or analysis.

Scripting Lifecycle processes using EMCLI Script mode

Enterprise Manager Release 3 introduced the EMCLI Script mode which is especially effective when performing tasks in bulk or many tasks at once. This mode enables you to create Jython scripts, store them as files and pass them as an argument to EMCLI. For more information on EMCLI see the documentation here.
In this section, we will expand on our previous work to automate the creation of patch plans to automate the application of the missing patch. We will use a python script to retrieve the list of databases without a patch, and then create a patch plan for each database.
As a prerequisite you must create a sample patch plan for a single instance database which has the desired patch ( 14275605 ) added to the plan. We will use this plan to create the others.
The work flow is as follows:
  1. Retrieve specified patch plan metadata and extract required patch information.
  2. Get list of databases without the patch applied using a configuration search.
  3. Create a patch plan for each database.

Here is the script in its entirety. Obviously you will need to make modifications for you environment, specific patch plan and configuration search names.

 #emcli_config_search.py

from emcli import *
import xml.dom.minidom


# Set Connection properties and logon
set_client_property('EMCLI_OMS_URL','https://oem.example.com/em')
set_client_property('EMCLI_TRUSTALL','true')
login(username='DWWOLF1',password='password')

patch_id = []
release_id = []
platform_id = []
language_id = []
target_type = []

# Get Sample Patch Metadata
pp_xml = show_patch_plan(name='PSU4 Rollout').out()

# Parse plan metadata into XML
patchPlan = xml.dom.minidom.parseString(pp_xml)

# Retrieve metadata for each patch in the sample patch plan
for patchList in patchPlan.getElementsByTagName("patchList"):
        for patch in patchList.getElementsByTagName("patch"):
                patch_id.append(patch.getElementsByTagName("id")[0].toxml().replace("","").replace("",""))
                release_id.append(patch.getElementsByTagName("release_id")[0].toxml().replace("","").replace("",""))
                platform_id.append(patch.getElementsByTagName("platform_id")[0].toxml().replace("","").replace("",""))
                language_id.append(patch.getElementsByTagName("language_id")[0].toxml().replace("","").replace("",""))
                target_type.append(patch.getElementsByTagName("target_type")[0].toxml().replace("","").replace("",""))

# Run stored configuration search to get list of databases missing the patch
target_array = get_targets(config_search='11.2.0.3 SI AC DBs without patch 14275605').out()['data']

# For each target create a patch plan containing the patches in the sample patch plan
for targets in target_array:
    tn = targets['Target Name']
    nodeCount = 0
    f = open('patchplan.txt', mode='w')
        for node in patch_id:
                f.write( "patch." + str(nodeCount) + ".patch_id=" + patch_id[nodeCount] + "\n")
                f.write( "patch." + str(nodeCount) + ".release_id=" + release_id[nodeCount] + "\n")
                f.write( "patch." + str(nodeCount) + ".platform_id=" + platform_id[nodeCount] + "\n")
                f.write( "patch." + str(nodeCount) + ".language_id=" + language_id[nodeCount] + "\n")
                f.write( "patch." + str(nodeCount) + ".target_name=" + tn + "\n")
                f.write( "patch." + str(nodeCount) + ".target_type=" + target_type[nodeCount] + "\n")
                nodeCount += 1
        f.close()
        planName = 'PSU4 ' + tn
        create_patch_plan(name=planName,input_file='data:patchplan.txt',impact_other_targets='add_all')

exit()


A zip of the script is available for download here.
Run the script by passing it as an argument to EMCLI:
>emcli @emcli_config_search.py
Here we can see all of the patch plans created by the script plus the sample patch plan “PSU4 Rollout”

Conclusion

Enterprise Manager’s Configuration Search feature is a powerful tool that can be leveraged both inside and outside of the UI. It can quickly and easily provide answers to difficult configuration questions without writing any SQL. When used via through the EMCLI it can be used to dynamically generate a target list which can be used to drive complex and otherwise time consuming tasks in the UI quickly and efficiently.
Configuration Search and patch automation are both features of Enterprise Manager’s Database lifecycle management pack.
For more information on Enterprise Manager’s database lifecycle management capabilities, visit http://www.oracle.com/technetwork/oem/lifecycle-mgmt/index.html

By Dave Wolf

What's New in Database Lifecycle Management in Enterprise Manager 12c Release 3


Disclaimer

Opinions expressed in this blog are entirely the opinions of the writers of this blog, and do not reflect the position of Oracle corporation. No responsiblity will be taken for any resulting effects if any of the instructions or notes in the blog are followed. It is at the reader's own risk and liability.

Blog Archive