MDM, SOA and BPM – Partners for Success, by Steve Minor
There’s a very interesting relationship between master data management (MDM), service oriented architecture (SOA) and business process management (BPM).
The purpose of this article is to discuss how each helps strengthen the others, sharing some of the experiences I’ve had implementing MDM with these technologies.
Based on the size of the organization and how centrally managed enterprise services are managed, MDM, SOA and BPM each require an IT initiative (sponsored by the business) to land if the goal is to establish a foundational IT capability.
Let’s be clear, these three capabilities are cost centers—they themselves don’t generate revenue or profit. Anyone implementing these solutions knows that pain. Any positive ROI story hinges on what they enable and how effectively we can make that connection. They are a means to an end, sharing a common vision of helping the business.
I’m not going to delve into the various possible business benefits in this paper (e.g. MDM’s single authoritative version of the truth or 360 view of customer, SOA’s “real-time” enablement and BPM’s business process re-engineering opportunities). I am going to point out that each is to some degree reliant on the other for the successful enablement of business goals.
First, I’d like to clarify this article is written largely from an MDM standpoint. I’d like to give a brief background on the two successful MDM implementations I’ve helped steward in order to give context. These two MDM solutions are both around “customer master” data but are fairly radically different.
MDM Solution #1 – I had the privilege of being the architect and development lead for implementing a Siperian-based MDM implementation for customer “organizations” (B2B) consisting of roughly 200,000 customer records.
The key points for this MDM solution included:
- MDM core was a central transactional hub, storing the single version of the truth, and syndicating that truth to the rest of the enterprise.
- MDM exposed a real-time web service (WS) interface for all updates (outside of initial data loads) that were managed by a SOA Enterprise Service Bus (ESB).
- We centralized all customer master data updates for the enterprise into a common BPM process that orchestrated human workflow steps (using IBM’s FileNet eForms and a Web user interface) and SOA MDM services (MDM was the only system of origin for organization master data)
- Applications in the eco-system (i.e. SAP ERP) were updated via an asynchronous publish/subscribe model using canonical messaging
- The solution implemented a Party pattern and additional MDM entities were on the strategy map: Vendor, Employee, Product
MDM Solution #2 – As a technical Program Manager Lead I’ve been responsible for a variety of tasks including establishing a multi-year roadmap, and leading a team of program managers to design and manage dozens of MDM features to land a customer MDM platform for “individual” data (B2C), consisting of over 1 billion customer persona records. This DIY solution is arguably one of the largest scaling MDM solutions in the world.
The key points of this MDM solution included:
- MDM core was a central transactional hub, storing the single version of the truth, and syndicating that truth to the rest of enterprise
- MDM exposed real-time sync, async, batch and pub-sub integration patterns.
- MDM harmonization approach was used to harmonize the many “systems of origin” of this data across the enterprise
- No BPM aspect to the solution
- MDM solution had to handle ~1500 transactions per second in update traffic at peak, with sub 400ms response time, with strong emphasis on engineering excellence around high availability (HA), disaster recovery (DR) and supporting online upgrades (“always on”).
- Solution implemented an MDM platform (scale up and out) designed to store any number of MDM entities (into the billions of records total).
Both of the above were successful: each landed an MDM solution meeting the original aspirations of the initiative—albeit, with considerably different technical solutions. Both involved real-time SOA, but implemented it a bit differently. Both involved workflow, but only solution #1 involved BPM in the sense of enabling the business to define process flows externally to MDM, orchestrating MDM and human workflow.
The key question enters my mind: what have I learned from these experiences? What additional strategies and/or approaches could I promote or evangelize with the business stakeholders for helping solve the business problems being discussed? I feel that having a strong understanding of how MDM, SOA and BPM can work together will really equip you with the tools and approaches to be successful.
For the purpose of this article, I’m looking at Operational MDM, and customer data specifically. I assume you’re familiar with the key capabilities provided by an operational MDM solution, including establishing a single authoritative version of the truth for shared and common enterprise data and having that data then be used across the enterprise.
MDM and SOA
Real-time is the reality of business today; business agility and savings are the battle cry. Whether driven by just-in-time aspirations, quicker time to market, or faster customer service, IT is tasked with making data highly accessible.
MDM operational solutions (whether Commercial Off The Shelf or Do It Yourself) are center stage and a facilitator of this objective. As such, most MDM solutions involve a service layer in the form of Web Services (SOAP, REST, HTTP, etc.) whereby clients send service requests to a real-time interface that receives the request and sends back a response.
A simple example would be MDM providing a basic set of Create-Read-Update-Delete (CRUD) service operations that can be used to lookup or update master entities (with proper security).
Clearly, having a SOA strategy – perhaps involving an Enterprise Service Bus (ESB) and directory services – is of value to MDM. Operational MDM needs it. If a SOA strategy is not in place, some serious discussions are needed to make sure you don’t go down the wrong path with MDM services.
Does SOA benefit by having MDM? Yes.
Without MDM, you risk limiting your SOA solution to tactical integrations with one-off point-to-point integrations where individual systems hand-shake on specific needs, or you risk limiting SOA to intra-application scope.
You risk simply paving the cow paths of the past—yes, it is in real time, but you risk implementing a new flavor of spaghetti (we’ve all seen those integration diagrams!).
The opportunity here with merging MDM and SOA is recognizing that the enterprise data definitions in MDM change the game from tactical to strategic. MDM establishes a common identifier/key and a common definition of the attribution of the entity or entities in question.
It may even allow an application to consider enterprise uses beyond their narrow immediate requirement—concepts like creating composite services come to mind, whether combining disparate entities, or with data augmentation from key engagement/transactional data.
In my experience, both of the MDM solutions I’ve helped design and implement had a SOA strategy as the preferred integration approach. Clearly, offering a secure integration interface that allows for synchronous interoperability within an enterprise, or even to external partners, is a key for success in a real-time world.
When working on MDM solution #1 (mentioned above), I appreciated the benefits of an ESB, including:
- Abstracting MDM from the various enterprise applications integrating with MDM. MDM’s service operations were locked down to only the ESB, while access security was implemented on the ESB per a standard security policy
- Providing ability to create targeted listener end-points on the ESB that pointed to a common general service in MDM
- Providing a basic versioning strategy allowing MDM to deprecate older service contracts earlier, and using the ESB to manage deprecation of the old contracts (both old and new mapping to the new contracts).
MDM solution #2 did not include an ESB because it wasn’t present. We accomplished some of the same benefits by using virtual IPs. Not quite as elegant architecturally, but it worked nicely.
MDM needs SOA and without MDM, a SOA strategy is at risk of being ineffectual and likely the breeding ground for a future re-factoring in the future.
SOA and BPM
The word complementary comes to mind. SOA + BPM represents a perfect example of the whole being greater than the sum of the parts.
SOA’s real-time nature and interoperability lend themselves well towards enabling time critical business orchestrations. Add in human workflow and now the business has the ability to choreograph end-to-end processes. True business process management can occur, limited only by the maturity of the SOA strategy and the ability of the BPM tooling.
BPM without SOA is possible, but the opportunities are likely constrained, and the results involve cumbersome point-to-point integrations.
While there has been some tension in the past between SOA and BPM camps (I’ve not been in any of those religious wars), the benefits of the two working together are strong in my experience. In fact, I would argue that to maximize the value of SOA, understanding the larger meta-point around what scenarios the services could facilitate in business processes is a must.
BPM and MDM
Let’s face it, most, if not all, of the core business processes that generate revenue involve the core and common Master Data Entities (Organization and Individual customers, products/materials, etc).
Companies sell products or services, and they sell them to organizations or individuals. The ability to look-up and potentially update MDM data as part of these business processes is a given.
As a side note, in in attempting to keep this document simple, I won’t be introducing or discussing multi-mastering (multiple systems of origin) or the implications of whether MDM is the sole mechanism for updating data or if harmonization is required. BPM integration with MDM would / could be impacted accordingly.
While MDM services are likely part of many BPM processes, the MDM solution itself should remain agnostic to business processes as much possible. With that stated, MDM can provide capabilities that support a BPM strategy while largely remaining agnostic.
As an Operational MDM solution, it is assumed that the master data is the single authoritative version of the truth. Sometimes organizations call this the “golden” data. Along those lines, an MDM solution can be used to store pre-golden data using terms like “copper” or “silver” to reflect that the data is not yet fully certified.
By implementing a “copper/silver” data strategy to the existing “gold” data strategy in MDM, you can open up the possibility of having MDM store the “in-flight” update data that is being choreographed via BPM workflows. Using a BPM provided parameter value that asserts the level of the data is a fairly simple approach that keeps MDM agnostic to the many BPM workflows, but allows for delineating between “copper” and “gold” lookups or updates—a lineage capability is a natural fit for storing this in-flight data.
Some form or record locking is likely also needed to eliminate collisions. Note that part of the update business rules in MDM would/should enforce an appropriate level of validation prior to being asserted as gold. (A more in-depth discussion on these concepts would be appropriate as a separate article.)
While SOA and BPM should be loosely coupled, there are MDM design considerations that should be considered ahead of time that can allow for MDM to more strongly support workflow concepts. I strongly recommend that these considerations be evaluated as soon as possible in an MDM high level design.
Pulling It All Together
The following diagram is a high level view on how BPM, SOA and MDM can fit together. This example is very MDM centric, effectively representing updates to master data. Other examples could be made where MDM data is retrieved as part of order-to-pay processes, etc. This MDM example best aligns to business processes involving management of the master data by company representative (working on behalf of customers), as opposed to consumer self-service scenarios.
In the example above, many interactions occur, and implementation details may distinguish the BPM from the user interfaces involved. I do want to call out that I’ve simplified the “BPM” layer and combined the UI element. Several design options are available that would impact whether BPM and/or a UI element interacts directly with MDM.
Key points include:
Step 1 – A change is initiated by a triggering event—likely a person initiating a process. This initiates a new workflow item. An e-form or web-based interface then allows the user to search/lookup the record, make necessary changes, and submit.
Step 2 – The BPM manages the workflow items, choreographing the necessary human and automated steps in the business workflow (direct integration with SOA depends on architectural/design approach for how the UIs interact with BPM or if they call the ESB directly). BPM of course manages the work queues, assignments, creating metrics on age of requests (SLA), status dashboard, etc.
Step 3 – ESB listeners receive requests, verify security credentials, map requests to the appropriate listener, and map responses back to the originating caller.
Step 4 – MDM services process the incoming requests. The validate operation is tasked with ensuring a master record (or object) passes necessary minimum data standards and quality bars. Using a rules engine, a given validation request can be configured to apply rules based on a passed parameter defining the “phase” of the request. This can allow for defining a ramping level of quality gates. For example, the initiating validation may be basic rules, while the final approval from the steward could require that all applicable rules fire.
Step 5 – Leveraging the copper/gold concepts mentioned above (see “BPM and MDM”), similar to the Validate operation, the Update operation can use the “phase” parameter to determine lifecycle of the update received. Updates can then be evaluated and executed accordingly until the appropriate “gold” request occurs.
Step 6 – Both “copper” and “gold” updates can be stored to the MDM store. Once this data is persisted as “gold”, thus becoming the authoritative truth, it is available for lookup and for syndicating out to subscribing systems.
Step 7 – The majority of MDM solutions require syndicating out updated master data to subscribing systems, even if only to an EDW/BI solution. In an ideal perfect world, all operational systems in your enterprise will use MDM as their store for master data, but that may be a vision that takes a long time to achieve (if ever).
Step 8 – A Pub/Sub type application (which can be distinct from or part of the ESB platform) receives these syndications and brokers, transforms and routes these changes to subscribers. Concepts around canonical message forms and come into play here.
The benefit of experience is that you have firsthand knowledge of what works, what doesn’t work, and what opportunities you might have in the future the next time you find yourself in a similar situation.
Whether I day dream about “if I could do it over again” or if indeed I find myself defining another MDM strategy/solution in the future, I can safely say that I’ll be exposing the opportunities and options to the business on how MDM, SOA and BPM can be leveraged to meet business needs.