Deciding on using an Oracle AIA PIP

Oracle Application Integration Architecture (AIA) Process Integration Packs (PIP) is a concept that was created by Oracle to provide out of the box integrations between their various ERP systems.  Their initial intention was to create a marketplace where Oracle partners could create PIPs and sell them to Oracle customers.  This concept initially led to several PIPs to be created but as time has gone on many of the Oracle partners have limited or stopped supporting the PIPs that they have created. There are a couple of things to consider when evaluating the use of PIPs:

• Do you have vanilla installations of your ERP systems or do they have a lot of customizations? If your ERP systems have been customized, then the PIP will need to be modified to support the customizations.  This includes something as simple as adding additional fields to the records that need to be integrated.
• How often to you upgrade your ERP systems?  If your upgrade cycle is slow, you run the risk of using a PIP that is no longer supported.  If your upgrade cycle is fast, you run the risk of needing a PIP version that has not been created/updated yet.
• The Oracle partner that created the PIP is also a very important because each partner has different support levels, including discontinuing support.
• The Oracle PIP marketplace never really gained much traction so it will be interesting to see how long Oracle still supports and pushes PIPs.

Canonical Best Practices

Here are some quick notes on best practices for creating a canoncial data model that is easily understandable and useable across an organization.

Define clear schema creation guidelines

• Versioning, namespaces, and naming
• Use file comments and xsd:documentation

Centralize "common" types and elements in common schemas

• Define entities shared across schemas in a common schema
• Have a simpleType schema and a commonTypes schemas

Beware of enumerations (make sure it can be strictly defined and will not change). There are actually very few things that can be represented by a truely definitive list.  For example, most people think the US states are a good example of an enumeration since there are only 50 of them but what about Guam or the District of Columbia or US Virgin Island.

Protocol-independent

• Create types in domain schema and create separate message schema that contain messages. The message schemas would leverage the domain schemas.  (i.e. Guest.xsd and GuestMessages.xsd)

There are several XML schema design patterns at http://developers.sun.com/jsenterprise/archive/nb_enterprise_pack/reference/techart/design_patterns.html. Over the years we have found that several tools don't fully support the different patterns so care should be taken to make sure your current and future tools support the pattern that you choose.  We prefer a modified Garden of Eden pattern.  It is modified because we don't recommend exporting everything but think of it as an API and expose only the types that schema users want to reference.

Schema versioning is one of the greatest concerns when working with a canonical data model.  Once developers start using your CDM it becomes harder and harder to make changes without impacting the current services.  Determine a standard on how major and minor version numbers will be used to support backward compatibility.  We use minor version numbers to represent changes that are fully backward compatibility and major version numbers to represent changes that require users to modify their code to utilize the new schema.

Define magical words that are used consistantly across all schemas.

• If 'Id' is used to identify a single object use it consistantly across all schemas. Don't mix Code and Id (eg. CountryId, CountryCode)
• Use attributes only for metadata and/or optional data

Stay away from edge XSD elements since tool vendors support them differently

• xsd:any
• xsd:substitutionGroup

Executing on SOA strategy

Sitting in a packed house watch Eben Hewitt discuss Executing a SOA Strategy and sharing his experiences with rolling out SOA at Choice Hotels. Good services need to have several attributes:

- Strong interface
- interoperability
- reusable & composable
- standard contract
- independent lifecycle
- discoverable and governable - not necessary from the beginning but needed quickly after first services are rolled out.

Criteria for a pilot project:

- Meaningful business value but don't bet the farm
- non-mission critical
- flexible delivery data
- don't cross organizational boundries
- make sure you have a good executive sponsor

A nice shout out from Eben for the help with the SOA Maturity model.

Don't confuse and merge SOA governance and SOA Compentency Center.  They have different but complementary roles in an organization.

SOA compentancy Center
- Act as a focal point for training and knowledge transfer
- Early adopters of new technology so they can set direction of where and how to use various tool sets
- Act as a deep bench to development teams when additional SOA bandwidth is necessary

Using SOA tools does not mean you are doing SOA. All of your SOA concepts should be tool independent.  Make sure you don't buy tools and then recreate the current state diagram using the new tools.  Look at how the current state diagram can be improved by SOA independent of the tool.

Oracle Service Bus (OSB) and JCA Adapters

We were recently working on a project that had the need to create a custom JCA adapter to connect from the OSB to a legacy C application.  We figured this would be something fairly easy to do since the WebLogic Server, BPEL and OSB all support JCA adapter out of the box.  It turns out that the current OSB (11g) doesn't fully support the JCA adapter specification.  It only supports the following JCA compliant adapters provided by Oracle:

• Oracle Adapter for Oracle Applications
• Oracle JCA Adapter for AQ
• Oracle JCA Adapter for Database
• Oracle JCA Adapter for Files
• Oracle JCA Adapter for FTP
• Oracle BAM Adapter
• PeopleSoft Adapter
• SAP R/3 Adapter
• Siebel
• J.D. Edwards

The OSB already has transports that support File/FTP so these really are not needed.  We actually tried using these on the bus and didn't have much success so quickly reverted back to the built in OSB functionality.

Anyway, we were looking for a way to utilize the custom JCA adapter on the bus and came across the OSB Transport API. It appeared that this would do the trick but seemed extremely complicated so we kept digging. We eventually settled on deploying the JCA adapter to the WebLogic Server and writing a small wrapper class that made it easy to access the adapter with a JNDI lookup. 

We were then able to simply use a Java Callout from the OSB to access the adapter.  This is obviously not the most elegant solution but has proved to work extremely well.  We were a little concerned about performance so we did a little testing.  We were able to push a single WLS managed server to over 3,000 adapter calls/second which is more then sufficient for our application. 

A few details:

• The JCA wrapper class takes and returns an XML document as a string
• It then uses JAXB to marshall/unmarshall it and calls the adapter
• The marshaller has been optimized to not destory itself every time to prevent marshalling bottlenecks
• The JCA wrapper also caches the JCA connection information

Which SOA Suite tool to use when reading a JMS topic

We have been doing a lot of event driven architecture with our clients lately and one question that comes up a lot is which SOA Suite component to use as the consumer. There are a couple of options available: OSB business service and BPEL using the JMS adapter.

Ideally you would like to use the OSB for its performance, statelessness, and low overhead but there is little gotcha that is pretty important when reading from topics. The current version of the OSB (11.1.1.5) doesn't support setting the JMS consumer name. The OSB auto-generates the JMS consumer name that is pretty unreadable and impossible to determine the client it is associated with. When connecting to a topic this is very important because that is the only way to know which consumers are connected to the topic and which ones are missing.

The JMS JCA adapter used by BPEL allows the developer to set a meaningful name to a durable subscriber. This makes it really easy when looking at the list of durable subscribes on a topic to determine which consumers are connected and listening and which consumers are not there. BPEL has some additional overhead (dehydration) but in most cases this can mitigated making the BPEL with JMS JCA adapter the better choice.

Service Based Chaos

A lot of companies that I come across become a victim of their own SOA success. They successfully complete their first project and the word gets out about the potential of SOA. This leads to many groups wanting to take advantage of the benefits of SOA. This is a good thing right???? The problem is processes, governance, and standards are normally an after thought when trying to get the first project out the door.

It is not a big problem during the first project because the team is normally small and the scope is controlled. Once the gates start to open up and new developers outside of the initial project are added with different skill levels chaos normally ensues. This is because there is little to no time to focus on best practices, standards and processes to share knowledge and put controls in place on the new projects. This leads to service based chaos.

Here is a link to a podcast that I did with Oracle that covers this topic and how we help a client avoid issues.

http://streaming.oracle.com/ebn/podcasts/media/10048058_Choice_Hotels.mp3

How to create a canonical data model?

When used properly the canonical form can provide great benefits in an SOA world such as loose coupling of applications, ease of integration maintenance and a common understanding of information, but when used improperly a canonical form can create a maintenance nightmare. So, how do you create the canonical form for an object?

I have seen several different approaches to addressing this issue that fall into two basic categories: create a superset of information or create the minimal subset of information. The sections below describe those approaches and their drawbacks.

Use all of the information that is available in the source system since sooner or later some application will want it.
This approach is probably the simplest way to create a canonical format, essentially every piece of information that is known about an object is passed around regardless of its usefulness to other applications. This approach has several easily identifiable drawbacks:

• The size of your canonical form will be unnecessarily large for what is needed. This means that the consuming applications will have to sift through all of the unneeded data to get to what it needs. This adds to the potential for development bugs as developers try to figure out what fields they need to use.
• The application that produces the event must needlessly create and provide information that no other application will never use.
• The XML representation of the form will be large and consume a lot of resources creating, transporting, and consuming the message.
• The benefit of this approach is that since the canonical form already has all of the information that can be provided so you will never have to modify your canonical form (Until the source application is enhanced with new information :)).

Try to think of every possible piece of information that any application now or in the future could ever need.This approach is another way to create a superset of fields in the canonical form.

• This approach obviously will take a long time to determine all of the information that will be required by receiving applications.
• I have seen several integration projects try to take this approach and fail before they even define the canonical form and start to integrate things.
• Each application will have their own unique set of information that they require and all of the other applications will need to sift through it needlessly.
• The benefit of this approach is if you can actually create a true superset then you will never have to modify your canonical form.
• If you can predict what information future applications will need you are in the wrong business you should be perdicting the stock market.

Create a base form and then as new applications are added the new data fields are added to the canonical format.
This approach starts out as a minimal subset approach and then quickly turns into a superset.

• The canonical form will grow to be extremely large as more and more applications are added.
• Each time the canonical form is modified/updated all of the consuming applications need to account for the changes.
• Eventually you will end up with the unmanageable supersets described above.
• The benefit to this approach is that you can create the initial canonical form fairly quickly since it will contain only the information required by the currently know consuming applications.

Provide only the minimal amount of information required to identify an object as unique.

• This approach is obviously the simplest to create. All that is required are the fields that make an object unique.
• This approach normally leads to unnecessary work/overhead because now every application needs to make an extra service call back to the source to retrieve the information that is not in the message. As more and more applications are added the burden on the source application becomes greater until it can no longer handle the load.

So, what is the answer? How can you create the canonical format that provides all of the perceived benefits without the maintenance nightmare?

There are a couple of good approaches to take and rules of thumb that will provide the balance between to much information and too little information in your canonical form.

1. Start small - Begin with the unique fields that make an object unique.
2. Add common fields - Add the fields that are common among most of the consuming applications that you are currently working with. This one can be tricky so I normally go with the rule of thumb that if 80% of the applications need it I should provide it.
3. Add information that is expensive to retrieve later - If there is information that is not in the message that is required by a consuming application, the application will need to retrieve it from the source application. If retrieving it from the source application is expensive (either expensive to recreate the information or their would be a high volume of retrievals) then provide that information to the canonical form.

Following these simple rules should prevent you from changing the form a lot. If the canonical form is good you should only need to change it when:

• The source application is enhanced/upgraded to contain additional information that is needed by 80% of the consuming applications.
• The cost of consuming applications retrieving additional information from the source application becomes to high.

If you find that your canonical form is changing a lot after it is initial created instead of just adding new fields every time step back and try to determine why all of the additional fields are required and how they were missed in the initial creation process.

In the end the canonical form should contain 80-90% of the information that is required by all of the consuming applications (the ultimate superset). This will minimize the time spent creating the initial canonical form and reduce the number of times that the canonical form needs to change.

There are also several standards organizations (like the Object Modeling Group) that have already created object representations that can be used as a starting point. These forms have been thought out over the years by many industry experts that are very knowledgeable about their respective spaces. Use these forms as a starting point, as they normally contain a lot of additional information that you may not need and can trim out.

Now that you have a canonical form in place the dirty word "governance" comes into play as other developers in other groups need information that is not already in the canonical form. A governance model must but put in place to prevent fields from being added to the form to make integrating to a single consuming application easier. Governance must look at the use of the canonical form as a whole and not just change the form to please the needs of one applications as this will lead to a complete superset and too many changes to the form.