Loqate: Accuracy and Intelligence from Address Data
The Hub Designs MDM Think Tank recently received a briefing from Martin Turvey, CEO of Loqate.
Read more
MDM’s Blind Spot: Social Networks by Peter Perera
The convergence of Master Data Management (MDM) and social networking is inevitable. Read more 
Final Deadline for the MDM Track at COLLABORATE 2012
The final deadline for the COLLABORATE 2012 conference Call for Papers is TODAY - Monday, October 17.
The MDM Track at COLLABORATE 2012
The deadline for the Call for Papers for the 2012 COLLABORATE conference is coming up fast – Friday, October 14.
Call for Papers for MDM Track at COLLABORATE 2012
by Dan Power
I’m still a volunteer on the Education Committee of the Oracle Application Users Group, which is a robust users group that is completely independent of Oracle Corporation. I’ve been involved in the group as a whole for over 15 years now, and have been on the Education Committee for more than five years. Read more
Informatica Progress
Misti Lusher and Ravi Shankar from Informatica were kind enough to do an analyst briefing for Hub Designs recently, to bring us up to date on what’s been happening with Informatica in the past few months.
The combination of Siperian with Informatica has exceeded their expectations so far, with MDM revenue running significantly ahead of quota and Informatica landing customers in a number of new vertical industries such as retail, healthcare, aerospace/defense, agriculture, education, and hospitality. Informatica continues to penetrate EMEA and has had its first successes in Asia Pacific and Latin America as well.
There’s also a healthy sales pipeline being built for future quarters, with the top three verticals being healthcare and life sciences, financial services and insurance, and high tech and retail. Growth is being seen all over the world, with a large percentage of the bigger sales opportunities for Informatica involving MDM, regardless of the region.
Ravi highlighted how the Informatica Master Data Management (MDM) solution is solving multidomain business problems like physician spend compliance, product mastering, high volume reference data mastering, clinical trial management, customer and channel management, and Salesforce.com enablement. He also discussed how Informatica’s other products usually fit into an MDM solution.
As the Informatica MDM product has evolved, it has remained true to its roots, and continues to offer complex hierarchy management, to be business user focused, and to allow for fast time to value. What Informatica has done, building on what Siperian had created before its acquisition, is to provide for true multidomain master data management, which allows for a much wider range of problems to be solved.
Informatica continues to increase its market share beyond the pharmaceutical vertical, and shows a strong track record of expanding its footprint within existing customers as well.
Informatica MDM Data Director has been widely used as well, with every new customer since its release in March 2009 buying it along with the MDM hub.
Informatica just finished up an 18-city MDM road show in the U.S. and Canada, and featured its MDM product prominently at Informatica World in early November. It has both a horizontal and a vertical industry marketing strategy.
Ravi previewed for us the materials for their “Customer and Channel Management Solution”, which manages hierarchies and relationships between customers, channel partners, products, and resources, in order to maximize account penetration, optimize coverage, and enable business agility and speed.
Ravi also gave us a demo of the latest version of the Informatica MDM product, with built-in dashboards using Data Director measuring data quality for individual customers and organizational customers. He also demonstrated the integration of MDM with the rest of the Informatica Platform – Power Center Business Glossary and Metadata Manager, and Informatica Data Quality.
Another impressive feature is enabling business applications, such as Salesforce.com, to be MDM aware. New records can be entered in the Salesforce.com application and instantly be bounced up directly against the Informatica MDM hub, and customer hierarchies can be viewed in a Salesforce.com tab, rather than requiring the user to jump back and forth between a Salesforce window and an Informatica MDM window. And the Salesforce user can see a timeline of a record “as of” a particular date, including all the hierarchy data.
At the end of the briefing, I came away feeling (again) that Informatica had made a great move in purchasing Siperian, and that Informatica’s MDM business has clearly gained momentum since the acquisition. This is clearly one of those cases where one plus one equals three. Informatica has done a great job integrating Siperian into the company, in taking advantage of the synergies between the two companies, and in promoting the product. Opportunities exist to take it even further, but the Informatica team is to be congratulated, since almost 60% of all mergers and acquisitions fail to create shareholder value according to the Boston Consulting Group.
New Article by Dan Power in Information Management
I’ve written an article for the most recent edition of Information Management magazine titled “How to Be an MDM Process Owner”.
Here’s an except:
“I’ve been an MDM evangelist since 2004 and have worked with a lot of people either becoming MDM process owners or helping shape another person within their company into one. Here are some tips if you’ve had that role handed to you, are aspiring to it or want to interest someone else in your company in stepping up to it.
First, embrace the political aspects of MDM. I wrote an article in the March 2008 issue of Information Management called “The Politics of Master Data Management and Data Governance,” in which I recommended that people start by understanding the political landscape at their company when creating a plan. Who are your likely allies and opponents? How will you get your initial funding and accomplish implementation? And don’t forget to plan for data governance.”
Click here if you’d like to read the full article.
Call for Papers for MDM Track at OAUG COLLABORATE 2011
I’ve been involved in the Oracle Applications Users Group (OAUG) since 1995, and have been a member of the OAUG Education Committee for several years now. The Education Committee is starting to plan next April’s COLLABORATE 11 Conference, and I’m managing the “Master Data Management” track.
Together with the Special Interest Group (SIG) coordinators for the Customer Data Management SIG and the Oracle Enterprise Product Lifecycle Management SIG, we invite YOU to submit a paper for the 2011 conference’s MDM track.
Our vision for the MDM track at COLLABORATE 11 is to have:
- a good mix of beginner, intermediate and advanced topics,
- presented by a range of well known and first time presenters,
- covering master data management, data governance and data quality,
- including domains such as customer, product, supplier and site,
- discussing Oracle products such as Oracle Customer Hub, Oracle Product Hub, Oracle Supplier Hub, Oracle Site Hub, Oracle MDM Data Quality, Oracle Higher Education Constituent Hub, Hyperion Data Relationship Management, and Oracle Fusion MDM.
Here are the important facts from the OAUG Call for Papers:
You’ll have the opportunity to connect with more than 5,000 users, technology leaders, Oracle executives and solution innovators gathering for the user-driven education and networking event April 10 – 14, 2011 at the Orange County Convention Center West in Orlando, Florida. Proposals are now being accepted. The deadline is Friday, October 1, 2010 at 11:59 p.m. EDT. To submit a paper, go to http://collaborate.oaug.org/submit/. For more information, you can go to http://collaborate.oaug.org/presenterinfo/.
Note to Oracle Employees: All Oracle employees interested in speaking at COLLABORATE 11 are to submit your papers through the Call for Papers submission form. Please contact speakerprograms@oaug.com for assistance with technical difficulties. For all other inquiries, please contact Lisa Stuart at lisa.stuart@oracle.com.
What I Learned on My Summer Vacation
I recently got back from my summer vacation, a 16-day, 300-mile sailing trip with my wife and two boys. We co-organized the trip for 15 boats, all members of the Blue Water Sailing Club. We went from the Boston area, through the Cape Cod Canal, down Buzzards Bay to Rhode Island Sound, spending four days on Block Island, and stopping off at great places like Padanaram, Cuttyhunk and Newport along the way.
Continuing a tradition we started in 2008 with an article called “Lessons on MDM from My Summer Vacation“, I’ll try to sum up some things I learned along the way, and apply them to master data management and data governance where I can.
1. Be Prepared for Storms
On one passage from Red Brook Harbor to Cuttyhunk, we were hit with a nasty thunderstorm that wasn’t forecast to go through until much later in the day. Winds were clocked at 50 knots (58 miles per hour). We prepared by dousing our sails, getting our foul weather gear on, battening down the hatches, and getting the boys in safe positions down below. But when the storm hit, the rain on my face felt like needles, visibility dropped to zero, our dinghy flipped over on its towing bridle, and I had to concentrate on avoiding a buoy in the area.
The application to MDM is that, given how political these projects can be, there will be storms. So be prepared for them. Have a good crew (project team), work hard at instilling loyalty between the team members, and maintain a united front. In our case, the storm, though intense, passed quickly, and we were able to get our dinghy right side up and resume our course for Padanaram with no injuries or damage.
2. Don’t Try to Control Too Much
Co-organizing a sailing trip with 15 boats can be a bit like herding cats. Sailors are very independent by nature at best, and even though we had regular check-ins by radio, some people would skip them completely, and others would forget (including me). Traveling with two young children increased the chaos factor. We’ve learned to go with it a bit – it’s like riding a wave. You can’t plan every minute of every day – sometimes you’ve got to be spontaneous, put the plan aside and just see what happens.
In the MDM and data governance world, the business community as a whole, even though they may not be on your project team directly, is going to be directly affected. They’ll want to have a say in how things are done, and they’ll have good ideas for you. Don’t shut them down. Learn to listen, actually consider what they’ve got to say, and be inclusive. Have town hall meetings where the broader business community gets a chance to tell you about their concerns, where you communicate the project’s progress and milestones, and where you can reach out to them and pull them in to upcoming phases.
3. Accept the Kindness of Others
Previously, we had a 32 foot boat, but at the beginning of June, we took delivery of a 38 foot boat, which we were still getting the hang of. A couple of club members on the cruise took the time to help us get to know the systems on our new boat, and it was great to have experienced friends walking us through what was, to me, new territory. Whether it was the selector switch between water tanks, the fresh water pressure pump, the anchor wash down pump, or various other things, our friends took the time to mentor us on the ins and outs of our new boat. And on the last day of the cruise, our friend Fred remembered that our son Brendan wanted a ride in his skiff, so he came alongside as we were leaving the harbor, picked him up, and gave him the ride of a lifetime.
The application to MDM and data governance is that you should be open to mentoring within and outside the enterprise. People like sharing their experience and wisdom with others, once you’ve established a strong relationship. If you reach out and develop a network of contacts inside and outside the company, then when the stuff hits the fan, you’ll be able to call on them for help. And even when you don’t need help, you’ll find a ready group of mentors who’ll take you under their wing, to teach you the finer points of leadership skills, project management tips and tricks, communications and marketing excellence, business process redesign and organizational change management basics — all the things you’ll need to succeed in your MDM and data governance initiative.
4. Stay on Schedule
There were several times during our sailing trip when we were tempted to stay an extra day or leave a day early from one place or another. We talked it over as a group and decided to stay on schedule. Many of us had made mooring reservations at marinas with strict cancellation policies, and we would have ended up paying for those moorings even though we didn’t use them. Not a big deal in and of itself, but we asked ourselves, what’s the worst that could happen if we stuck with the original schedule? It turned out that it wasn’t that different from what would happen if we went with a changed schedule.
In the MDM and data governance world, as in any technology implementation, there are going to be unforeseen obstacles. Try to build some cushion into your project plan, so the smallest little delay doesn’t impact your critical path and delay the overall project. When you get to the point that to stay on schedule means sacrificing functionality or increasing costs (the famous “triple constraint“), the discussions start getting pretty heated. There will be many times when your project will feel like you’re herding cats too, but remember how important it is to stay on schedule. You can’t finish on time if you get behind shortly after you start.
5. Look for Those Special Moments You’ll Always Remember
There were quite a few special moments on this vacation. Shortly after we arrived in Cuttyhunk, both of my boys put on their bathing suits and dove off the boat into the harbor. They swam fearlessly from Blue Water boat to Blue Water boat, saying hello to their friends, until we had a bunch of kids in the water doing the same thing, including one little girl that had never done that before (and who made her dad very proud). That night, after going to the beach, we had a lobster bake that I organized for 33 people on the lawn overlooking the harbor. I will remember the conviviality and friendship of that dinner for a long time. And there were small moments too: body surfing with my youngest son in Westport, getting airborne in the dinghy, slogging through the passage to Block Island against 25 knot winds, foul currents and 4-6 foot seas (even the hard times can be good memories after you get through them).
For MDM and data governance practitioners, there are many rewards: the satisfaction of bringing in a challenging project on time and on budget, forging relationships with team members that will last a lifetime, learning new things and expanding professional horizons, being recognized by the company as a valuable player capable of big things, mastering MDM and data governance at a time when having those technologies on one’s resume certainly doesn’t hurt one’s career prospects, and so on. For a good look at what is involved in being a “data champion”, and the rewards involved, read “So You Want to be a Data Champion?” by my friend, Tom Carlock.
To sum it up, if you’re prepared for the inevitable storms that will come your way and don’t try to control things too much, and are open to the kindness of others while remembering the importance of staying on schedule, you’ll certainly be blessed, as I have been, with a wealth of those special moments you’ll always remember. Master data management and data governance can be challenging, but they can be very rewarding as well, both for the organizations which take on the initiatives and for the individuals who make up those teams.
Data Profiling For All The Right Reasons, Part 5
The Hub Designs Blog welcomes the final installment of this great series by Rob DuMoulin, an information architect with more than 26 years of IT experience, specializing in master data management, database administration and design, and business intelligence.
Part 5: The Profiling Payoff
This is the final part of a five-part series, describing how data profiling benefits both IT projects and business operations. In Part One, we discussed profiling perspectives. In Parts Two, Three and Four, we introduced the value of system, entity, and attribute-level metrics. This part discusses the archival and beneficial uses of profile results.
If you have defined your corporate data profiling strategy similar to the methods discussed in the preceding parts of this series, you’ll have amassed a robust collection of metadata spanning relevant systems across your business. Although systems may be of different types and locations, the structured approach and common metrics you collected create a centralized repository of information that can be examined holistically. Ideally, this information will exist in an open-source database repository with reports made available across the enterprise. System and Entity information help planners and developers organize information strategies. Attribute-level domains, constraints, and business rules help data architects understand existing systems. Relationships and value patterns are readily available to support validation of information-related hypotheses as needed.
If you plan to design your own repository, consider adding timestamps and indicators to help you manage and present the information. To keep your repository relevant to business needs, design collection rules to be configurable. This allows you to easily ignore superfluous information or enable tests only at certain critical times. Allow initial system profiling efforts to gather a large set of metrics and store them as your baseline. As you learn about the information, you will see which tests or which data objects add no value. Us geeky DBA-types who understand system-level catalogs have our own scripts to do much of what was described inParts Two,Three and Four. Those less-inclined may prefer to use a third-party tool for profiling. Either way works as long as the business needs are satisfied and the entire enterprise standardizes on one approach (and thus one integrated repository).
You will find that collecting and maintaining this level of detail has a definite cost. Even if the collection is automated, interrogations of large data sets places an overhead on production systems that may not be practical. Record and monitor profile execution metrics to identify bottlenecks or tuning opportunities. Realize that the extent of data profiling is contingent on the project phase, specific data elements, and most of all, business value. Review profiling goals on a regular basis and eliminate unnecessary and redundant checks.
How much profile history to maintain is another consideration. Even though disk is “relatively” cheap, maintaining all historical entries in a live repository may not be necessary. Consider business needs and value for historical profile information. Even consider archiving at a summarized (or less frequent) level and keep only a limited time window of statistics online.
This discussion on data profiling was intended to broaden perceptions of what it means to a business and the value it can bring if done in a sustainable way. The blog format is not conducive to in-depth discussions, but hopefully the topics covered here spur some thoughts into how you can add value to your business by implementing some of these concepts. Use your imagination, but remember that no matter how cool it might be to collect and store some profile output, if it does not add business value to somebody, it might not be worth the overhead to continue recording it.
Data Profiling For All The Right Reasons, Part 4
The Hub Designs Blog welcomes Part 4 of this series by Rob DuMoulin, an information architect with more than 26 years of IT experience, specializing in master data management, database administration and design, and business intelligence.
Part 4: Profiling Relationships and Patterns
This is part four of a five-part series describing how data profiling assists in all aspects of system development, from design through deployment.
Part One introduced different perspectives on data profiling. Part Two identified valuable system and entity metrics to track. Part Three discussed attributes. In this segment, we dive deeper into attribute relationships and pattern recognition. Also, we expand on primary key identification discussion and discuss hidden relationships.
Pattern grouping provides a mask of distinct format patterns within an attribute data set and a count of the number of occurrences. Patterns give insight into the type of values found in an attribute. For example, a numeric pattern analysis may show values such as 999.99999, 99, or -.9999.
Observing distinct patterns gives insight into the maximum digits and precision, and also domains such as integer or real. Pattern of a database date or date-time type provides unremarkably similar patterns for all dates. Because the database management system typically enforces the domain, date analysis provides no value and can be ignored. If dates are stored in character format, however, patterns quickly show variations in date formatting. Character patterns only have significance to a limited number of positions. It makes no sense to pattern a description field of 200 or 2000 characters. Smaller code attributes of less than 10 characters though do provide value. Ignore pattern profiling for character strings over 20 characters at first, then refine to shorter character strings if the results do not add value.
In pure database theory, referential integrity (RI) is your friend. In practice, designers and software vendors often forgo RI to improve system performance on data inserts. These designers place the data quality burden on the application and do not endorse external data manipulation outside the application interfaces. In the real world, though, data corruption occurs and without RI or routine data quality checks, corruptions may not be found for a long time or not at all. Personally, I have identified over $50,000 of recent orphaned sales in a retail client resulting from deliberately disabled RI. These unreported sales were not added to the ledger and were allowed to occur for performance reasons until I found them through simple profiling. Enforcement of RI is a topic for another discussion but is mentioned here because it does identify a valid reason for data profiling.
In even presumably good relational designs, some parent-child relationships are not enforced for different reasons. First, interrogate the RI listed in the system catalogs to identify all enforced relationships. Reverse-engineering a system with a good modeling tool is probably the best way to do this. A harder and more valuable analysis is to identify unenforced relationships and determining the probability of the relationship if not all values are an exact match. Do this by counting all the candidate child attribute values that exist within a known parent attribute table. If all match and there are a non-trivial number of matches, there is a good probability of a non-identified relationship. A small number of mismatches could identify data quality issues.
In Part 5, we tie all the techniques discussed in the first four parts together to show the value of a repeatable data profiling process.
Data Profiling For All The Right Reasons, Part 3
The Hub Designs Blog welcomes Part 3 of this series by Rob DuMoulin, an information architect with more than 26 years of IT experience, specializing in master data management, database administration and design, and business intelligence.
Part 3: Attribute-Level Analyses
This is part three of a five-part series on data profiling.
In Part One, we took a light-hearted view of where profiling benefits an organization and in Part Two, we discussed the fundamentals of a profiling strategy. The remaining three parts discuss attributes, relationships, patterns, and how to use the combined data profiling information you collect. In this section, we introduce attributes, the lowest-level components of a profiling effort.
An attribute is simply a individual data element. Alone, an attribute has no context. Given the simple descriptor of “Cost” for an attribute tells us very little about the attribute’s true purpose and immediately drives a need for additional information, such as units (hours, Dollars, Euros…), type (weighted, unit, gross…), and use (invoice, sum, average…). Attributes therefore must be analyzed within the context of their business purpose to have meaning.
Some characteristics require business knowledge to define and others can be determined through interrogation of existing values and underlying rules of the storage medium. It takes both analyses to get a complete picture of information within a system. While assembling this puzzle, though, keep in mind that until you validate the enforcement of business rules, only assumptions can result from physical profiling or business context.
Analyses of values, domains, and constraints allows insight into use (or abuse) of an attribute. The larger the sample size, the better confidence you gain in the results. Without explicit proof of business rule enforcement, though, you must assume that just because a value does not presently exist does not mean it cannot exist. Business rules are defined by business experts and enforced through database constraints, data type/precision, and application code. Knowing the methods of enforcement allow you to narrow a domain but not totally understand it. Profiling of actual values provides additional refinement in terms of percentage of NULL values, percentage of distinct values, minimum, maximum, and average values, top x and bottom x recurring values along with their counts, and minimum, maximum, and average data lengths.
Some attributes within a data set serve valuable purposes that are important to identify. Attributes that individually or in conjunction with others define uniqueness of the data set also may support relationships between entities. Uniqueness can be further classified as being either members of a system-enforced primary key or of a business key (outside of the defined primary key). System-enforced primary keys are relatively easy to define within a database system through interrogation of the system catalog. Business keys that exist in tables in addition to a primary key may be more difficult to identify, especially if more than one attribute is needed to define uniqueness.
Attribute-level information of interest includes: data type (size and precision), the number and percent of NULL values, column descriptions, number and percent of distinct values, and the minimum-maximum-average values and lengths. Uses of the system catalog provides some of this information, but others must be collected through sampling the data.
Other types of attributes that may help in identifying relevancy are those that provide system-level auditing or change control. Knowing which attributes fill these roles may either allow you to (a) ignore them for profiling purposes or (b) use them to help explain versions or data anomalies.
Part 4 expands on attribute profiling with the introduction of relationships and patterns.
Data Profiling For All The Right Reasons, Part 2
The Hub Designs Blog welcomes Part 2 of this series by Rob DuMoulin, an information architect with more than 26 years of IT experience, specializing in master data management, database administration and design, and business intelligence.
Part 2: Profiling the Basics
This discussion is the second of a five-part series on data profiling. In Part 1, we discussed the project roles that benefit from data profiling and how better understanding information results in more reliable information systems. Important goals of any profiling strategy include automation of metric collection and socializing results to support the differing objectives of a data-centric project.
Early in a system development life cycle, profiling helps define sources, data storage requirements, and data transformations. As a system goes into production (or if profiling is added to an existing system for quality control purposes), routine profiling is useful to audit system quality and business rule enforcement. The frequency of collection and amount of effort you expend to automate your profiling methods should be based on the ability of the organization to benefit from the profile results.
This section discusses the beginnings of a profiling effort. Information assembled here forms the foundation of other profiling activities. For this discussion, consider a Profile Group as a set of information sharing a common purpose and data management methods. Examples of profile groups include tables within a single database schema or a group of spreadsheets with the same format but each spreadsheet representing a different time slice of data.
The underlying System managing a set of information within the profile group may be a named relational database, a file system directory, or even a web site being accessed through web services. The reason we abstract information into Systems is to group the information into distinct governance methods common to the underlying information. Relevant metadata and governance methods we track at the system-level include: technical contacts, backup schedules, system descriptors, connection strings, business unit owners, and host operating systems. System-level metadata common to a profile group helps us understand and troubleshoot future analyses. This level of information also provides developers with an understanding of inherent restrictions (or freedoms) they may encounter when trying to use or integrate the information.
Entities within a profile group belong to the same system, may have a common unique identifier, and, for database entities, have the same schema owner. Typically, entities are database tables, but may also be similar files or spreadsheet tabs containing like attribute lists. For entities, we track characteristics common to all the attributes they contain. These include: row counts, entity-level descriptors, growth characteristics (size and frequency), last analyzed date, and various customized indicators such as active/inactive, existence of change data management attributes such as insert/update timestamps, and existence of audit traceability indicators such as insert/update username.
The combination of system and entity level profiling supply the foundation for the attribute-level profiling, which is where physical information in a system resides. It also provides valuable metadata to classify information and allows for future correlation of like information across systems. Assembly and publication of entity and system level information benefits the various consumers of the information by providing a centralized “master” source of contact and context information.
In Part 3, we will dive into the attribute level analyses around data profiling.
Data Profiling For All The Right Reasons, Part 1
The Hub Designs Blog welcomes a guest post by Rob DuMoulin, an information architect with more than 26 years of IT experience, specializing in master data management, database administration and design, and business intelligence.
Part 1: The Psychology of Data Profiling
Swiss psychologist Carl Gustav Jung founded the Analytical School of Psychology. His word association theories form the basis of the Myers-Briggs Type Indicator Assessment test to identify career aptitude in today’s high school students. Dr. Jung’s approach assigned personality profiles based on how an individual’s thoughts associated to various phrases. By analyzing responses, he could understand how an individual viewed the world around them and perceived themselves. Typically, subjects are asked to speak the first thought entering their minds after hearing a trigger phrase. For the following example, remember, there are no wrong answers. If I say the words “Data Profiling”, what is the first thing you think of?
If you thought of food, cats, country music, CSI NY, or residential plumbing, you are either not in IT or are an IT Manager.
If your first thought was “Quality Assurance”, you align yourself with data quality professionals having anti-social thoughts of failing test cases and sadistically reporting lazy developers for buggy code. You gleefully scour test cases looking for any evidence of truncation, missing values, non-matching codes, numeric precision errors, and inconsistent abbreviation, text, and date formatting.
If “Integration” comes first in your mind, past legacy integration projects have scarred you with a disdain for source system data quality levels. You view production apps with contempt and loathe the time it takes to track down data issues caused by system integrations. You investigate upstream sources to create detailed mappings and transformation rules. Typical debugging sessions consist of validating relationships to identify orphaned data, identifying attributes that contain overloaded columns (attributes containing more than one distinct data element), or fixing format errors from implied decimals.
Some of you responded with “Value Domains” or “Data Types”, indicating you are obsessive compulsive data architects compelled to organize the world into strict and orderly fashion with some degree of normalization, though you are not considered “normal” by your peers. Your concerns lie in understanding and regulating naming conventions, relationships, existence of NULL or default values, and understanding the meaning of each data element to accurately identify business rules and when two or more objects are related or redundant.
Lastly, if “Debugging” is the first item in your thought queue, you are a coder justifying why presumably good code is not working. Extreme paranoia has taught you to assume nothing about data quality, so you add tests to identify duplicates, validate relationships, enforce business rules, track change data capture, provide substitute values. Your phobia of early morning phone calls cause you to add auditing to your code to inform a DBA of data issues rather than waking you up in the middle of the night.
It is truly amazing how much we can conclude from the response to one simple phrase.
As stated before, there are no wrong answers. Aside from the innocent jab at Managers and non-IT resources, we all realize the benefits of information quality and absolutely need business involvement to understand context and domains of business information. The meaning and actions of Data Profiling change both by role and by project phase. Through profiling, we are able to identify best sources of information, learn proper ways to categorize and store it, reactively identify quality issues, and proactively define business rules to prevent future issues.
Identifying what is important to profile, when and how profiling is done, and how to share our findings across business and project resources is key. Done properly, profile results integrate to a master metadata repository and are periodically refreshed through an automated process.
This five-part series provides a tool-agnostic approach to comprehensive data profiling, focusing on information meaning and use. The next part of the series discusses system and table-level profiling. In particular, what information is important to collect at the system and table level and how can that information be leveraged by the Enterprise to help assure quality. The third part dives into attribute-level profiling and the fourth discusses attribute patterns and relationships. The final part discusses the benefits and utility of gathering profiled information into a single repository.
Orchestra Networks Webinar Replay
Yesterday, I attended a great webinar hosted by Orchestra Networks and Michelin. I found it full of good insights into how Michelin has deployed proactive data governance with real benefits to the business.
To view the webinar replay and download the presentations, just register at http://www.orchestranetworks.com/webinar-master-data-governance-michelin.html. And if you have any questions, please contact info@orchestranetworks.com.
About Orchestra Networks
Orchestra Networks provides advanced Data Governance and MDM software, with a strong focus on governance for all shared data across an organization. This model-driven approach with its dynamic user interface and data services provides the means for business and IT to finally collaborate on MDM. Founded in 2000, Orchestra Networks has presence across Europe and North America.
Oracle’s MDM Strategy and Roadmap
At the Oracle Applications Users Group (OAUG) COLLABORATE 2010 conference this week, I attended a session by Pascal Laik, Oracle’s VP of Master Data Management Strategy.
He started out by talking about several Oracle MDM customers, their success stories and their return on investment, across drivers like growth, efficiency, improved IT agility, and compliance.
Pascal moved on to talk about MDM implementation challenges. Oracle surveys its MDM customers every two years. Measuring actual ROI achieved is the most difficult challenge reported. Next is breaking down organizational silos, and then demonstrating incremental business value.
Five out of the top ten challenges were related to data governance and project/organization. These were big themes two years ago as well. So Oracle worked with an outside partner on the areas of strategy, policies & processes, organization, measurement & monitoring, technology, and communication. They got a group of 10-15 customers together 2-3 times per year, and that group put together a set of requirements for a product that Oracle has now created called Data Governance Manager. This product helps data governance professionals to operate and monitor the hub and to define and enforce policies.
Pascal showed a short video from an Oracle customer, Areva. Their program was called STOCK – Strategic and Operational Customer Knowledge, to ensure the high quality of customer data. They used a five step approach: Collect, Harmonize, Merge, Enrich, and Publish. The benefits included saving employees time, ensuring that internal people can rely on customer and prospect data, and providing the entire enterprise with a clear vision of the customer database.
The second set of challenges related to ROI and business case – measuring actual ROI achieved. Oracle now has a web-based ROI model available through its sales team. Oracle also has a group of people that do a 3-5 week management consulting exercise called “Insight” that delivers a full business case.
The third set of challenges is the first one involving technical issues: #10 and #11 (integration and data quality).
Two years ago, the #1 issue was procuring skilled resources. So Oracle has been working closely with systems integrators, so now this issue is down to #7. Integration with operational applications has gone from #2 to #11.
Lastly, Pascal discussed Oracle solutions, investments and its strategy going forward. Oracle now has Customer Hub, Supplier Hub, Product Hub, and Site Hub. Data Relationship Management, which is a financial hub to manage financial entities such as the chart of accounts and other hierarchies, is also an analytical hub.
Oracle Customer Hub (formerly known as Universal Customer Master) is now on release 8.2, which shipped in January 2010, and includes the new Data Governance Manager module. This is the largest customer release in four years.
Oracle’s MDM strategy has two legs – embedded “best in class”. Oracle has OEM’d the Informatica solution, using the Identity Systems solution (now owned by Informatica) and the Address Doctor solution (also from Informatica) for postal cleansing for 200+ countries. The other leg is “open” – Oracle is providing a “Universal DQ Connector” for selected vendors like Trillium, Acxiom, D&B and Datanomic. (Note: the embedded “best in class” approach is somewhat controversial, since Informatica is now competing directly with Oracle, since it has acquired the Siperian MDM hub).
The end-to-end data quality framework (the Data Quality “Machine”) has a Rules Manager for design, development and validation (IDQ). There is a process (Analyze/Profile, Standardize/Cleanse, Match & De-Duplicate, Enrich) with a Scorecard & Reporting, and an Exception Management Process. The output is to load the MDM system with zero rejects.
Oracle has also acquired Silver Creek Systems, which is focused on product data quality. It is a self-learning semantic engine to handle the complexities of product information.
Pascal talked about some of the newer MDM hubs, Supplier Hub and Site Hub. Site Hub in particular has experienced strong interest from retailers, fast food companies and large enterprises, which are using it to manage stores and locations.
Oracle’s MDM investments are critical for Oracle in terms of its differentiation strategy, and data governance is the number one item from its customer advisory board. Oracle has reached 1,000 MDM customers across all of its various MDM products.
Pascal wrapped up by talking about how competitive the MDM space is and the recent acquisitions in the market. Oracle’s history is in applications. Oracle brings a pre-built, flexible schema with enterprise-grade, verticalized hub applications. Oracle MDM hubs are pre-integrated with both Oracle and non-Oracle applications. And Oracle provides best-in-class data quality and data governance solutions.
Our Booth at the Gartner MDM Summit
Hub Designs was a Silver sponsor at the Gartner MDM Summit 2010. Here’s the new, 3-minute video we produced to describe what Hub Designs does as an consulting firm focused specifically on MDM:
Great New White Paper and Other Collateral Available at Our Booth
At the event, we announced with Equifax a new product that integrates Equifax commercial information with Oracle E-Business Suite and Oracle Customer Data Hub. This product simplifies the process of integrating Equifax credit and marketing information with prospect and customer data in Oracle. Both the joint press release and a one-page product overview were distributed at the booth.
Also available was a new whitepaper written in collaboration with Informatica titled, When Data Governance Turns Bureaucratic: How Data Governance Police Can Constrain the Value of Your Multidomain Master Data Management Initiative. This updated version of an earlier white paper written with Siperian in 2009 added both new content and industry insights. It was very well received at the Gartner conference this week.
Finally, we handed out one of the most popular recent articles from this blog, Hidden Costs of Duplicate Customer Data.
The conference drew attendees from many different market sectors, so discussions and meetings were both informative and valuable from an MDM perspective. Several Hub Designs clients were able to join us there, from the insurance, software and transportation industries, and we had four of our team members there as well. I’m going to write a separate article with my thoughts on the sessions and the mood of the conference, but I wanted to provide a look at our booth as well, for our readers who weren’t able to make it to Las Vegas this week.
Hub Designs and Equifax Introduce Oracle Integration Solution
Today, at the Gartner Master Data Management Summit in Las Vegas, Hub Designs and Equifax jointly announced a new product, Hub Designs Equifax Integration for Oracle, bringing the power of Equifax Commercial Information Solutions data to the Oracle E-Business Suite and Oracle Customer Data Hub platforms.
The solution smoothly integrates Equifax data into Release 12 of Oracle’s enterprise resource planning (ERP) and master data management (MDM) suites.
Hub Designs Equifax Integration for Oracle provides access to vital credit and marketing data in Oracle’s MDM and ERP modules including:
- Oracle Customers Online
- Oracle Sales Online
- Oracle Receivables
Equifax commercial information helps businesses to:
- Make credit decisions, expedite collections, reduce bad debt and pre-qualify prospects;
- Reveal linkage between related companies;
- Standardize name & address information and prevent duplicates;
- Enrich prospect and customer records with marketing and credit information from Equifax;
- Increase productivity by creating new parties and party relationships in Oracle automatically
The joint press release describes the solution in more detail, and a one-page overview is available as well. If you’re interested in learning more, please contact us via our web site or drop by Booth #7 during the exhibit hall hours at the Gartner MDM Summit.
Informatica Analyst Briefing
Arvind Parthasarathi, Ken Hoang and Ravi Shankar from Informatica were kind enough recently to give me a detailed briefing on Informatica’s master data management (MDM) strategy after its acquisition of Siperian.
First, there’s no doubt this was a game-changing move, for both Siperian and for Informatica. With over 4,000 Informatica installed base customers to leverage, and 200 Informatica sales reps going through training and certification, Siperian’s sales momentum should increase dramatically. And in fact, several new deals have closed just since the acquisition was announced in late January.
And being acquired by Informatica eliminates the “company viability” question that some Fortune 500 IT shops would have about any software company under a certain size (not just Siperian). Informatica itself might be acquired by one of the mega-vendors at some point, but with annual revenue of $500 million, it’s big enough not to be subject to the financial viability question.
Informatica also provides a large partner ecosystem and a significant marketing budget, so living on under the Informatica banner, Siperian can compete more readily for mind share both with partners and with potential customers.
But what impressed me the most was the strategic nature of the other purchases that Informatica has made over the past couple of years, such as Identity Systems for entity resolution (i.e. matching) and Address Doctor for address cleansing. With the addition of Siperian as a strong player in the multidomain MDM hub space, Informatica has declared itself a real competitor against the likes of Oracle, IBM, Initiate Systems (an IBM company) and SAP.
And in some ways, Informatica is better positioned than most of these, for two reasons. First, it has a complete suite of leading products for data integration, data quality and all of the associated things that make up the “MDM ecosystem”. And second, many of its competitors are dependent on it for those components (Ramon Chen wrote a great article on Informatica’s OEM agreements with various competitors).
Informatica’s product lineup supports all of these MDM requirements:
- Multiple MDM architectural styles including the ability to support Registry style (competes most directly with Initiate Systems)
- Multiple data domains, i.e. multidomain MDM (competes most directly with Oracle, IBM and SAP)
- Data Integration and Data Quality as a foundation for MDM (competes with a wide variety of products)
So in some ways, Informatica wins even if customers buy a competitor’s MDM hub product, because there’s a good chance they’ll still buy Informatica’s data integration and/or data quality solutions, to help them with data integration, data profiling and data quality, or to help build the inevitable data services, once the master data is gathered in a centralized hub and able to deliver timely, trusted and relevant to the rest of the enterprise.
Informatica sees its MDM products used in both Operational MDM (where the master data is actively managed by data stewards, governed and improved and then synchronized back to the operational systems), and in Analytical MDM (where for various reasons, the improved master data does not flow back to the operational systems, but flow instead to data warehousing, analytical and business intelligence applications).
Informatica has such a strong, integrated story, with its PowerCenter data integration, Informatica Data Quality, and Informatica MDM products, that it’s able to accommodate customers’ maturity needs starting with data integration and progressing to data quality and MDM.
And Informatica, by giving customers the ability to solve any MDM-related business problem with a unified architecture, spanning data integration, data profiling, data quality, identity resolution, address validation, and all major styles of master data management, has pulled together a great set of solutions for MDM.
I’m looking forward to seeing the Informatica folks at this week’s Gartner MDM Summit conference in Las Vegas. If you’re going to be there, stop by and see the Hub Designs team at Booth #7 during the exhibit hall hours. We’ll be announcing a new product with Equifax, and we’ll be releasing a data governance white paper with Informatica.
Hub Designs at Gartner MDM Summit
We’re in the final stages of getting ready for the Gartner MDM Summit at this point. It will be held on April 14-16, 2010 at the Mandalay Bay Hotel in Las Vegas, NV.
This will be our third time at this event, and our second as an exhibitor. Last October, we exhibited as a Kiosk sponsor, and this year we will be there as a Silver sponsor. We’ll be in Booth 7 during the exhibit hall hours, and if you’re going to be attending and would like to meet with us, just contact us via our web site.
We’ll be announcing an exciting new product, and publishing a new version of one of our most popular white papers.
Here’s what you’ll learn about if you go:
- Multi-domain MDM
- MDM vision and strategy
- Customer data
- Product data
- Data warehousing, data quality and MDM
- Enterprise information architecture
- Enterprise information management
- SOA and MDM
And here are the key benefits of attending:
- Insight into creating a successful MDM program
- Persuading the business to take a leadership role
- Delivering measurable ROI by linking your MDM to business metrics
- Reducing costs and increasing efficiency by removing duplication and creating consistency
- Improving customer acquisition and cross- selling/upselling activities
- Complying with regulations and leveraging your master data to manage risk
- Consolidating and leveraging data faster following mergers & acquisitions
- Accelerating your new product introductions
- Managing your supply chain more efficiently
It’s not too late to register at the special rate of $1,795 – a $300 savings on the standard rate of $2,095! Go to gartner.com/us/mdm or call 1-866-405-2511 and mention priority code: MDMHUB.
We’d love to see you in Las Vegas! These events are like “old home week” – getting to catch up with people we haven’t seen in a while and find out what everyone in the MDM space is up to. So come along for the ride, catch a few sessions, maybe hit the tables a bit, and head home with a little less cash in your pockets but a little more knowledge in your head. And if you need help convincing the “powers that be” to let you go to the conference, Gartner has very thoughtfully put together an Attendee Justification Kit to help you convince them.
Answering Questions from LinkedIn
I got a good question via LinkedIn the other day, so I thought I’d answer it here:
Dear Dan,
I am a database architect but I am new to MDM and data governance, and I’m very interested in this area.
Can you please suggest where to start? I’ve found some information on the web (sometimes a bit disconnected), but I seem to be lost with so much information. Also, the tools that are currently available in the market – do they address all the challenges in this space?
One question I have is: if data quality is given the importance it deserves from the beginning of any project (operational or data warehouse), are MDM initiatives necessary? Are MDM projects needed because of the proliferation of applications that are developed in silos and that don’t consider what information is already available to the enterprise? In essence, should MDM be part of any project?
Thanks for your time.
Not to sound too self-serving, but I’d start with this blog and the MDM Community.
As for your question about whether MDM initiatives are necessary if data quality is given sufficient importance, please realize that MDM is a relatively new discipline which includes embedded data quality; it does not replace data quality.
What MDM does is sit between the source systems (typically CRM and ERP) and the data warehouse and business intelligence. So instead of trying to flow master data and transactional data directly into the warehouse for analysis, we bring it into the MDM system first, where it can be “mastered” – which includes fixing data quality issues. We then flow those corrections back into the source systems and downstream into the analytical systems. Which of course you can’t do without data quality tools. But data quality tools by themselves are not sufficient, because they typically don’t persist or store the data.
Your next question, are MDM projects necessary because of the proliferation of apps developed as silos – yes, that’s a big part of it. Essentially, if you developed a new architecture from scratch, you’d put a multi-domain MDM hub, able to handle many types of master data at the core, and you’d build data quality into it, then you’d surround it with integration so you can flow data from there to where ever it’s needed. So clean, accurate, consistent and timely master data would be available to any other IT project that was going on, but it would only have to be built once. “Build once, use many”, as they say.
Please keep reading and I hope you stay interested in MDM and data governance!
I got an answer from this person today that I thought I’d share with you here:
Dan,
Thank you very much for your insights. Whatever documents I had read about MDM had more to do with the people, process or technology, but didn’t cover the essence of MDM. I’ve gone through some of your blogs and I’m beginning to understand MDM.
AMB Releases Community Edition
Hub Designs has been a partner of AMB, a provider of information governance, quality and discovery software, since November 2008.
Now AMB is launching an open source version of its Information Governance Suite, called the Community Edition. AMB delivers tools to facilitate real time governance, and is now extending its reach as one of the first major vendors with an open source version of its core product.
This might be an ideal way for companies looking to familiarize themselves with a data profiling and data quality product to learn the tool, get a data governance proof of concept up and running in a cost effective way, and then demonstrate value to the business.
The Community Edition allows you to:
- become familiar with the concept of data profiling as a way of identifying and fixing information anomalies
- enable enterprises embarking on a data stewardship program to use the Community Edition to spotlight, identify and determine the priority of their internal information issues
- enable organizations to define and automate a repeatable process, using software to administer the information governance program that aligns with the repeatable process, not the other way around
The Community Edition should provide a core set of data profiling and governance, and training and support is available, as are upgrades to the Professional and Enterprise Editions.
For more information, contact AMB at 1-847-899-5154 or community@ambpdm.com, or visit http://www.ambpdm.com.
Long Live MDM
Editor’s Note: Today’s post was written by Jeff Schaffzin. Jeff is an independent consultant with over 15 years of experience in high tech. He’s worked with a number of leading software vendors in roles such as product marketing, professional services and information technology. Specializing in data management, Jeff has spent the last three years focusing on Customer Data Integration and Master Data Management and has worked with a number of high profile companies in the United States and abroad.
DISCLAIMER: While the facts that I’ve included here are true, I’m speculating on the reasons why they’re taking place. I have no affiliation with any company mentioned here, nor should my opinions be construed as knowledge of their actions.
If you, like me, have followed MDM for the past year or two, you knew that what has been happening recently was going to happen sooner or later. Whether it was due to choice or necessity, MDM has been in the IT press a lot lately. Oracle acquired Silver Creek to enrich its product information management offering. Talend has announced and started to promote its open source MDM application. Data integration provider Informatica acquired Siperian recently in order to enter the MDM space and IBM recently acquired Initiate Systems as well.
Each of these events leads to one key question – how will this impact MDM in the short term and in the future? Given my understanding of the space, I think three scenarios are likely:
Scenario 1
It is hard to ignore the movements that IBM and Oracle have been making in the past year or so. In a market economy, the goal is to have as much market share as possible. In order to do this, you either build new products or acquire existing companies that have the technologies that you want.
While each company has done a combination of both building and buying solutions, their strategic plans are hardly a secret. IBM has proposed a vision of an end-to-end data management platform, which includes their MDM offering as well as reporting tools like Cognos and analytics/statistics from SPSS. Now that IBM has acquired Initiate Systems to complement their MDM stack, the question is why. Do they want to be known as a serious player in the health care industry? There could be other reasons too – they may consider MDM just a small piece of their data management toolkit and this could solidify that position, moving MDM from one of the hottest ‘technologies’ out there to just a “means to an end” to increase market share for their software business unit. Regardless of the reason, it means one less major MDM player in the market.
Then we have Oracle. For as long as I can remember, Oracle has been promoting its Fusion strategy. For those of you who are not familiar with it, Fusion is Oracle’s attempt to provide one code base that would pull together the applications it has built and purchased. This momentous undertaking was finally demonstrated at last year’s Oracle Open World (while Oracle continued to acquire other companies such as Silver Creek Systems).
However, like IBM, one can speculate on where MDM fits in this Fusion strategy. Oracle has always promoted its database first and sold its applications second. Even with the numerous special purpose hubs they’ve been developing lately, could this finally be the technology that enables Oracle to transcend from being a database vendor to a true platform player. Only time will tell with this one.
Scenario 2
There’s always the possibility that MDM has been considered the “secret sauce” – the so-called missing link – that rounds out the product lines for data integration/migration vendors.
Talend’s acquisition of French software company Amalto provided them a way to enter the MDM space. The open source vendor has been a darling of the analysts for a number of years and even won an award by Gartner, one of the first (if not the first) they offered such a company. However, since I don’t have contacts within Talend, it’s not clear what their next step will be, since they seem to be focusing their energies mostly in MDM after hiring two people to drive that effort within the past 6 months or so.
As the de facto leader in data integration, Informatica needed to extend its reach beyond that space. If you look at their job listings, they are looking for someone to market their CEP (Complex Event Processing) efforts. Relatively recently, they were looking to hire someone who had experience with ERP or MDM, but it is unclear which path they decided to take with that. Regardless, there were always looming rumors of them wanting to add MDM to their portfolio with the press suggesting that they would acquire Initiate Systems. However, instead of buying them, they purchased Siperian – a company half its size in terms of customer base and revenue.
In either of these cases, MDM may not be their flagship product, but at least it shows that it is a viable technology and shows that it is something that won’t be going away any time soon.
Scenario 3
People like me who have been in the data management space are always interested in improving something. I believe in the statement, “even if something isn’t broken, there’s always a reason to make it better.” This was clear when Customer Data Integration (CDI) first came about and many companies hopped on that bandwagon, knowing that they wanted a way to track their customers more efficiently.
At the same time, other companies explored Product Information Management (PIM), a way to have a single source of product information which was sourced from PLM, inventory and supply chain systems. Following that was the concept of MDM – a beautiful vision – having a single source of truth that can be used by an entire company.
Now we have a new concept that has been promoted – Multi-domain MDM. Siperian and other companies have began to promote this to show the world that they are truly the most advanced players out there. While this has been going on, there have been rumblings about Enterprise Information Management (EIM). What I’m still not clear on is – what’s the difference between multi-domain MDM and EIM? Are they the same? If not, what are the differences between the two concepts?
In any case, there’s a lot to think about. I don’t know where you stand, but one thing is certain – MDM is not going away, at least for the foreseeable future.
Data Profiling For All The Right Reasons, Part 5
The Hub Designs Blog welcomes the final installment of this great series by Rob DuMoulin, an information architect with more than 26 years of IT experience, specializing in master data management, database administration and design, and business intelligence.
Part 5: The Profiling Payoff
This is the final part of a five-part series, describing how data profiling benefits both IT projects and business operations. In Part One, we discussed profiling perspectives. In Parts Two, Three and Four, we introduced the value of system, entity, and attribute-level metrics. This part discusses the archival and beneficial uses of profile results.
If you have defined your corporate data profiling strategy similar to the methods discussed in the preceding parts of this series, you’ll have amassed a robust collection of metadata spanning relevant systems across your business. Although systems may be of different types and locations, the structured approach and common metrics you collected create a centralized repository of information that can be examined holistically. Ideally, this information will exist in an open-source database repository with reports made available across the enterprise. System and Entity information help planners and developers organize information strategies. Attribute-level domains, constraints, and business rules help data architects understand existing systems. Relationships and value patterns are readily available to support validation of information-related hypotheses as needed.
If you plan to design your own repository, consider adding timestamps and indicators to help you manage and present the information. To keep your repository relevant to business needs, design collection rules to be configurable. This allows you to easily ignore superfluous information or enable tests only at certain critical times. Allow initial system profiling efforts to gather a large set of metrics and store them as your baseline. As you learn about the information, you will see which tests or which data objects add no value. Us geeky DBA-types who understand system-level catalogs have our own scripts to do much of what was described inParts Two,Three and Four. Those less-inclined may prefer to use a third-party tool for profiling. Either way works as long as the business needs are satisfied and the entire enterprise standardizes on one approach (and thus one integrated repository).
You will find that collecting and maintaining this level of detail has a definite cost. Even if the collection is automated, interrogations of large data sets places an overhead on production systems that may not be practical. Record and monitor profile execution metrics to identify bottlenecks or tuning opportunities. Realize that the extent of data profiling is contingent on the project phase, specific data elements, and most of all, business value. Review profiling goals on a regular basis and eliminate unnecessary and redundant checks.
How much profile history to maintain is another consideration. Even though disk is “relatively” cheap, maintaining all historical entries in a live repository may not be necessary. Consider business needs and value for historical profile information. Even consider archiving at a summarized (or less frequent) level and keep only a limited time window of statistics online.
This discussion on data profiling was intended to broaden perceptions of what it means to a business and the value it can bring if done in a sustainable way. The blog format is not conducive to in-depth discussions, but hopefully the topics covered here spur some thoughts into how you can add value to your business by implementing some of these concepts. Use your imagination, but remember that no matter how cool it might be to collect and store some profile output, if it does not add business value to somebody, it might not be worth the overhead to continue recording it.
Go back to Part 4.
Initiate Systems Acquired By IBM
Today, IBM announced that it is acquiring Initiate Systems.
This was widely rumored last week, but the announcement of Informatica’s acquisition of Siperian took my mind off this temporarily.
On the face of it, it makes all the sense in the world. IBM knows a good product when it sees it, and Initiate has been doing well in the MDM world lately, particularly in the healthcare vertical, where it grew up, and in the public sector vertical. IBM’s press release explicitly mentions Initiate as a leader in “data integrity software for information sharing” among healthcare and government organizations. I thought it was interesting that the IBM release didn’t mention the terms “master data management” or “MDM” even once.
I was a little surprised that IBM’s release made no mention of the financial terms, since IBM is a public company, but I’m sure it will only be a matter of time before those details become available to those who know where to look or whom to ask.
It wasn’t a surprise to see the IBM release mention the stimulus funding being invested around the globe. When I first saw the rumors last week, I immediately thought – IBM is buying Initiate to be better prepared for the various e-Healthcare initiatives that are coming down the pike.
Where things may get a bit tricky is explaining the multiple MDM platforms from IBM to potential customers, and managing several different development roadmaps and product portfolios. There’s the IBM InfoSphere MDM Server (the former DWL product) and there’s also IBM InfoSphere MDM Server for Product Information Management (the former Trigo product). And now there’s the Initiate product too.
While the acquisition does make sense, there is an “embarrassment of riches” factor. IBM will, of course, develop a sales playbook explaining what situations at what type of customer are a good fit for each product.
I think the lingering feeling I have with Initiate Systems is that it may be headed for a “golden ghetto” at IBM – never to reach its full potential as a solution across many different industries, and eventually to handle many different domains of master data. IBM may (and rightly so, in its mind) pigeonhole it into the healthcare and government verticals.
But Initiate’s had some good success outside those two industries. In the Financial Services vertical, they’ve got customers like Capital One Financial, Countrywide Financial (now Bank of America), eSure Insurance, and Wells Fargo. In the Hospitality industry, they’ve got Choice Hotels. In manufacturing, they’ve got Mitsubishi Motors Australia. In the Logistics vertical, they’ve got Federal Express. In the retail sector, Barnes & Noble, CVS, Longs Drug Stores and SuperValu are all customers. And in the high tech space, they’ve got Dell, Ingenix, Intuit, LocatePLUS, Microsoft and National Instruments.
Unfortunately, they didn’t achieve enough critical mass in any of these other verticals to compete with the strong momentum they’d developed in healthcare and government.
As I said last week, these are interesting times in the MDM world. The recent M&A activity, the healthy demand from large and medium sized corporations, the large number of consultants from other areas claiming to now have experience in MDM – these are all signals to me of a large and fast-growing market. So the New Year, for those of us in the MDM space, is off to a good start.
Siperian Acquired By Informatica
Siperian, one of the last best-of-breed providers of master data management (MDM) technology, is being acquired by Informatica.
The two firms were already working together closely, having an alliance and OEM relationship through Informatica’s acquisitions in 2008 of Identity Systems (for entity resolution and matching) and in 2009 of Address Doctor (for customer address cleansing).
This will strengthen the Siperian product further by bringing Informatica’s technology even more tightly into the Siperian MDM Hub.
At the same time, it eliminates the “company viability” question mark that sometimes gets raised in large IT shops’ minds when evaluating enterprise software vendors. When a Fortune 500 company is evaluating a smaller company, they sometimes wonder how long a company like Siperian can last against companies like IBM, Oracle and SAP. I’ve never been a big fan of that argument, since some of the best software gets created at small and medium-sized companies, but there’s no doubt it’s a obstacle to be overcome with the larger enterprises. Now, it shouldn’t be an issue.
As a Siperian partner, Hub Designs is excited about this acquisition. Based on the information we’ve got at this point, it seems like a good thing for Siperian’s customers, products, shareholders, partners and people. In today’s economic climate, dreams of a big IPO (for any venture-backed technology company) are unlikely, so an acquisition by a well-run larger company is a good outcome.
I know a lot of the people at Siperian personally, and have worked closely with them over the last few years. I hope the people at Informatica realize what a strong team they are getting in this acquisition, and do everything they can to hang onto them all.
I do suggest they stop using the term “MDM Infrastructure” though (which appeared 5 times in Informatica’s press release announcing the acquisition). First, it’s not accurate – MDM projects typically need to be drive by the business to be successful, so they can’t and shouldn’t be thought of as “IT Infrastructure” projects. Secondly, from a marketing perspective, “infrastructure” is about as exciting as mud – it’s hard to get senior management excited about spending money on something with the word “infrastructure” in the name.
As for the acquisition’s impact on the rest of the MDM market, it’s still growing pretty quickly, but the number of players is shrinking. So I think we’ll see it become even more competitive, and with Informatica now becoming a strong player in the MDM hub market, that’s got to cool its relationship with Oracle, who selected Informatica as an OEM component of its Oracle Fusion MDM hub.
IBM is rumored to be acquiring Initiate Systems, which is an interesting play in its own right, especially given the expected growth in spending in the e-healthcare space over the next few years.
And SAP continues to improve its products slowly but steadily, while D&B/Purisma is doing some interesting things with web services access to the D&B central database of information on businesses.
As for the remaining independent MDM vendors, like Orchestra Networks and Kalido, or Product Information Management (PIM) solutions like Stibo and Riversand, they should see this as further validation of the strength of the MDM market. Kalido feels that it’s the only independent MDM provider who can manage every master data domain. That may be true. I plan on learning more about Kalido over the next few months.
So like the Chinese curse, “may you live in interesting times”, the beginning of 2010 promises to be interesting for all of us in the MDM business!
If you’d like to continue the discussion on the “Impact of Informatica’s Acquisition of Siperian”, click http://ning.it/aJ1Xj5.
Data Profiling For All The Right Reasons, Part 4
The Hub Designs Blog welcomes Part 4 of this series by Rob DuMoulin, an information architect with more than 26 years of IT experience, specializing in master data management, database administration and design, and business intelligence.
Part 4: Profiling Relationships and Patterns
This is part four of a five-part series describing how data profiling assists in all aspects of system development, from design through deployment.
Part One introduced different perspectives on data profiling. Part Two identified valuable system and entity metrics to track. Part Three discussed attributes. In this segment, we dive deeper into attribute relationships and pattern recognition. Also, we expand on primary key identification discussion and discuss hidden relationships.
Pattern grouping provides a mask of distinct format patterns within an attribute data set and a count of the number of occurrences. Patterns give insight into the type of values found in an attribute. For example, a numeric pattern analysis may show values such as 999.99999, 99, or -.9999.
Observing distinct patterns gives insight into the maximum digits and precision, and also domains such as integer or real. Pattern of a database date or date-time type provides unremarkably similar patterns for all dates. Because the database management system typically enforces the domain, date analysis provides no value and can be ignored. If dates are stored in character format, however, patterns quickly show variations in date formatting. Character patterns only have significance to a limited number of positions. It makes no sense to pattern a description field of 200 or 2000 characters. Smaller code attributes of less than 10 characters though do provide value. Ignore pattern profiling for character strings over 20 characters at first, then refine to shorter character strings if the results do not add value.
In pure database theory, referential integrity (RI) is your friend. In practice, designers and software vendors often forgo RI to improve system performance on data inserts. These designers place the data quality burden on the application and do not endorse external data manipulation outside the application interfaces. In the real world, though, data corruption occurs and without RI or routine data quality checks, corruptions may not be found for a long time or not at all. Personally, I have identified over $50,000 of recent orphaned sales in a retail client resulting from deliberately disabled RI. These unreported sales were not added to the ledger and were allowed to occur for performance reasons until I found them through simple profiling. Enforcement of RI is a topic for another discussion but is mentioned here because it does identify a valid reason for data profiling.
In even presumably good relational designs, some parent-child relationships are not enforced for different reasons. First, interrogate the RI listed in the system catalogs to identify all enforced relationships. Reverse-engineering a system with a good modeling tool is probably the best way to do this. A harder and more valuable analysis is to identify unenforced relationships and determining the probability of the relationship if not all values are an exact match. Do this by counting all the candidate child attribute values that exist within a known parent attribute table. If all match and there are a non-trivial number of matches, there is a good probability of a non-identified relationship. A small number of mismatches could identify data quality issues.
In Part 5, we tie all the techniques discussed in the first four parts together to show the value of a repeatable data profiling process.
Data Profiling For All The Right Reasons, Part 3
The Hub Designs Blog welcomes Part 3 of this series by Rob DuMoulin, an information architect with more than 26 years of IT experience, specializing in master data management, database administration and design, and business intelligence.
Part 3: Attribute-Level Analyses
This is part three of a five-part series on data profiling.
In Part One, we took a light-hearted view of where profiling benefits an organization and in Part Two, we discussed the fundamentals of a profiling strategy. The remaining three parts discuss attributes, relationships, patterns, and how to use the combined data profiling information you collect. In this section, we introduce attributes, the lowest-level components of a profiling effort.
An attribute is simply a individual data element. Alone, an attribute has no context. Given the simple descriptor of “Cost” for an attribute tells us very little about the attribute’s true purpose and immediately drives a need for additional information, such as units (hours, Dollars, Euros…), type (weighted, unit, gross…), and use (invoice, sum, average…). Attributes therefore must be analyzed within the context of their business purpose to have meaning.
Some characteristics require business knowledge to define and others can be determined through interrogation of existing values and underlying rules of the storage medium. It takes both analyses to get a complete picture of information within a system. While assembling this puzzle, though, keep in mind that until you validate the enforcement of business rules, only assumptions can result from physical profiling or business context.
Analyses of values, domains, and constraints allows insight into use (or abuse) of an attribute. The larger the sample size, the better confidence you gain in the results. Without explicit proof of business rule enforcement, though, you must assume that just because a value does not presently exist does not mean it cannot exist. Business rules are defined by business experts and enforced through database constraints, data type/precision, and application code. Knowing the methods of enforcement allow you to narrow a domain but not totally understand it. Profiling of actual values provides additional refinement in terms of percentage of NULL values, percentage of distinct values, minimum, maximum, and average values, top x and bottom x recurring values along with their counts, and minimum, maximum, and average data lengths.
Some attributes within a data set serve valuable purposes that are important to identify. Attributes that individually or in conjunction with others define uniqueness of the data set also may support relationships between entities. Uniqueness can be further classified as being either members of a system-enforced primary key or of a business key (outside of the defined primary key). System-enforced primary keys are relatively easy to define within a database system through interrogation of the system catalog. Business keys that exist in tables in addition to a primary key may be more difficult to identify, especially if more than one attribute is needed to define uniqueness.
Attribute-level information of interest includes: data type (size and precision), the number and percent of NULL values, column descriptions, number and percent of distinct values, and the minimum-maximum-average values and lengths. Uses of the system catalog provides some of this information, but others must be collected through sampling the data.
Other types of attributes that may help in identifying relevancy are those that provide system-level auditing or change control. Knowing which attributes fill these roles may either allow you to (a) ignore them for profiling purposes or (b) use them to help explain versions or data anomalies.
Part 4 expands on attribute profiling with the introduction of relationships and patterns.
Data Profiling For All The Right Reasons, Part 2
The Hub Designs Blog welcomes Part 2 of this series by Rob DuMoulin, an information architect with more than 26 years of IT experience, specializing in master data management, database administration and design, and business intelligence.
This discussion is the second of a five-part series on data profiling. In Part 1, we discussed the project roles that benefit from data profiling and how better understanding information results in more reliable information systems. Important goals of any profiling strategy include automation of metric collection and socializing results to support the differing objectives of a data-centric project.
Early in a system development life cycle, profiling helps define sources, data storage requirements, and data transformations. As a system goes into production (or if profiling is added to an existing system for quality control purposes), routine profiling is useful to audit system quality and business rule enforcement. The frequency of collection and amount of effort you expend to automate your profiling methods should be based on the ability of the organization to benefit from the profile results.
This section discusses the beginnings of a profiling effort. Information assembled here forms the foundation of other profiling activities. For this discussion, consider a Profile Group as a set of information sharing a common purpose and data management methods. Examples of profile groups include tables within a single database schema or a group of spreadsheets with the same format but each spreadsheet representing a different time slice of data.
The underlying System managing a set of information within the profile group may be a named relational database, a file system directory, or even a web site being accessed through web services. The reason we abstract information into Systems is to group the information into distinct governance methods common to the underlying information. Relevant metadata and governance methods we track at the system-level include: technical contacts, backup schedules, system descriptors, connection strings, business unit owners, and host operating systems. System-level metadata common to a profile group helps us understand and troubleshoot future analyses. This level of information also provides developers with an understanding of inherent restrictions (or freedoms) they may encounter when trying to use or integrate the information.
Entities within a profile group belong to the same system, may have a common unique identifier, and, for database entities, have the same schema owner. Typically, entities are database tables, but may also be similar files or spreadsheet tabs containing like attribute lists. For entities, we track characteristics common to all the attributes they contain. These include: row counts, entity-level descriptors, growth characteristics (size and frequency), last analyzed date, and various customized indicators such as active/inactive, existence of change data management attributes such as insert/update timestamps, and existence of audit traceability indicators such as insert/update username.
The combination of system and entity level profiling supply the foundation for the attribute-level profiling, which is where physical information in a system resides. It also provides valuable metadata to classify information and allows for future correlation of like information across systems. Assembly and publication of entity and system level information benefits the various consumers of the information by providing a centralized “master” source of contact and context information.
In Part 3, we will dive into the attribute level analyses around data profiling.
Data Profiling For All The Right Reasons, Part 1
The Hub Designs Blog welcomes another guest post by Rob DuMoulin, an information architect with more than 26 years of IT experience, specializing in master data management, database administration and design, and business intelligence.
Part 1: The Psychology of Data Profiling
Swiss psychologist Carl Gustav Jung founded the Analytical School of Psychology. His word association theories form the basis of the Myers-Briggs Type Indicator Assessment test to identify career aptitude in today’s high school students. Dr. Jung’s approach assigned personality profiles based on how an individual’s thoughts associated to various phrases. By analyzing responses, he could understand how an individual viewed the world around them and perceived themselves. Typically, subjects are asked to speak the first thought entering their minds after hearing a trigger phrase. For the following example, remember, there are no wrong answers. If I say the words “Data Profiling”, what is the first thing you think of?
If you thought of food, cats, country music, CSI NY, or residential plumbing, you are either not in IT or are an IT Manager.
If your first thought was “Quality Assurance”, you align yourself with data quality professionals having anti-social thoughts of failing test cases and sadistically reporting lazy developers for buggy code. You gleefully scour test cases looking for any evidence of truncation, missing values, non-matching codes, numeric precision errors, and inconsistent abbreviation, text, and date formatting.
If “Integration” comes first in your mind, past legacy integration projects have scarred you with a disdain for source system data quality levels. You view production apps with contempt and loathe the time it takes to track down data issues caused by system integrations. You investigate upstream sources to create detailed mappings and transformation rules. Typical debugging sessions consist of validating relationships to identify orphaned data, identifying attributes that contain overloaded columns (attributes containing more than one distinct data element), or fixing format errors from implied decimals.
Some of you responded with “Value Domains” or “Data Types”, indicating you are obsessive compulsive data architects compelled to organize the world into strict and orderly fashion with some degree of normalization, though you are not considered “normal” by your peers. Your concerns lie in understanding and regulating naming conventions, relationships, existence of NULL or default values, and understanding the meaning of each data element to accurately identify business rules and when two or more objects are related or redundant.
Lastly, if “Debugging” is the first item in your thought queue, you are a coder justifying why presumably good code is not working. Extreme paranoia has taught you to assume nothing about data quality, so you add tests to identify duplicates, validate relationships, enforce business rules, track change data capture, provide substitute values. Your phobia of early morning phone calls cause you to add auditing to your code to inform a DBA of data issues rather than waking you up in the middle of the night.
It is truly amazing how much we can conclude from the response to one simple phrase.
As stated before, there are no wrong answers. Aside from the innocent jab at Managers and non-IT resources, we all realize the benefits of information quality and absolutely need business involvement to understand context and domains of business information. The meaning and actions of Data Profiling change both by role and by project phase. Through profiling, we are able to identify best sources of information, learn proper ways to categorize and store it, reactively identify quality issues, and proactively define business rules to prevent future issues.
Identifying what is important to profile, when and how profiling is done, and how to share our findings across business and project resources is key. Done properly, profile results integrate to a master metadata repository and are periodically refreshed through an automated process.
This five-part series provides a tool-agnostic approach to comprehensive data profiling, focusing on information meaning and use. The next part of the series discusses system and table-level profiling. In particular, what information is important to collect at the system and table level and how can that information be leveraged by the Enterprise to help assure quality. The third part dives into attribute-level profiling and the fourth discusses attribute patterns and relationships. The final part discusses the benefits and utility of gathering profiled information into a single repository.
Continue with Part 2.
2009 Year in Review
As we’re about to enter 2010, it’s a good time to reflect on what happened in 2009 and what it all means.
“It was the best of times; it was the worst of times…” So Dickens begins “A Tale of Two Cities”, but it’s also a good description of the past year.
The first half of the year was one of the most challenging I’ve faced in my twenty-three year career in business and technology. The second half of 2009 was better – not without its speed bumps but every month was a little better than the one before it.
The macro-economic climate has been tumultuous at best. But the second half of the year showed enough improvement that Hub Designs’ revenue for the year was up 33%. Not bad for a two and a half year old company during the worst economic conditions in 80 years …
Marketing and Thought Leadership
We launched a new web site in January, and it’s been well received. Total visits to www.hubdesigns.com were up 14% over 2008.
A little later in the year, we updated the “look and feel” of the Hub Designs Blog, branding it as the “world’s fastest growing blog covering master data management and data governance”. We’ve gotten more than 43,000 hits since we started writing in July 2007, and our readership more than doubled in 2009, to about 27,000 hits per year.
We published six issues of our “Best Practices in Master Data Management” newsletter this year. We publish the newsletter about six times a year to roughly 3,300 subscribers.
I wrote six articles for Information Management magazine, including some popular ones on “Product Information Management Challenges”, how to build a business case for master data management, and how to select the right MDM vendor for your organization. I also wrote for Identity Resolution Daily, on “The Growing Role of Identity Resolution in MDM” and “Matching – MDM’s Secret Sauce”.
With our partner Siperian, we wrote a white paper in August called “When Data Governance Turns Bureaucratic: How Data Governance Police Can Constrain the Value of Your MDM Initiative” that has generated quite a bit of discussion. You can download a copy of it here.
A second white paper, called “Best Practices for Leveraging D&B in Oracle E-Business Suite”, was written in partnership with Dun & Bradstreet. It describes using D&B information to drive better supply chain performance for companies using Oracle E-Business Suite. You can download it here.
I volunteer for the Education Committee of the Oracle Applications Users Group (OAUG). A big part of that effort lies in programming the MDM track for the annual conference. This year, it was in Orlando in May, and I really enjoyed speaking there and seeing people from the Oracle community that I don’t see very often. Here’s a link to my OAUG presentation.
We participated in conference calls with Oracle Development during the year, and ultimately attended the Oracle Fusion “Hands-On Validation & Testing” session for Customer MDM at Oracle headquarters in August. It was a great chance to get some early insights into Oracle’s next major product release and to see the progress Oracle has made in building out its Fusion MDM vision, which is striking in its powerful hub technology and its elegant & productive user interface.
In 2008, we attended the Gartner MDM Summit to decide whether to exhibit there in 2009. We were impressed enough with the conference that we did exhibit in 2009, in October in Los Angeles. We had a positive experience, so we’ll be a Silver level sponsor in April 2010 in Las Vegas. Since we specialize in MDM and data governance, we find the association with Gartner’s MDM event a powerful one.
I didn’t attend Oracle OpenWorld for the past couple of years, but this year I was glad I did. It was like “old home week”, seeing people from Oracle itself and from the broader Oracle community that I’ve met over the past 15 years. David Butler, Senior Director of MDM Marketing at Oracle, posted my presentation on Oracle’s web site, and said “you were our cleanup hitter and you hit a home run with the bases loaded”.
We also did webinars with our partners Siperian and Initiate Systems. The Siperian webinar covered the differences between MDM platforms like Siperian and ERP platforms like SAP from a master data perspective. The Initiate webinar, with Initiate’s CTO Marty Moseley, discussed developing strong MDM business case, deploying core MDM technologies and lessons learned on the “build vs. buy” question.
Social Networking
After experimenting with social networking in 2008, this year we had a coordinated strategy to use the Hub Designs Blog, Facebook, LinkedIn and Twitter to communicate & collaborate with our clients, potential clients, team members, partners, suppliers, etc.
It’s a pretty simple strategy. Short updates (140 characters or less) go out on Twitter, and are re-published on both LinkedIn and Facebook. Longer updates (i.e. blog articles) are published on the Hub Designs Blog. We encourage responses and feedback using @replies on Twitter and comments on LinkedIn and Facebook, as well as longer-form comments on the blog. And we get them – almost every blog article gets at least one comment, sometimes as many as a dozen.
When a new blog article comes out, we notify everyone via a single update on Twitter. What’s amazing is that during 2009, social networking now drives about 15% of the Hub Designs Blog’s total traffic. And one of our clients gave us some good feedback that our social networking activities help her stay current on what we’re up to, and help her feel connected to us as a company.
Another social networking experiment that developed further in 2009 was the MDM Community. We started this using Ning (a “social network in a box”) in November 2008, out of frustration with LinkedIn’s “Group” functionality. It now has more than 210 members, from 23 different countries. It’s still a work in progress, but if you’re interested in master data management or data governance, you should check it out at http://mdmcommunity.ning.com. It’s becoming an international “who’s who” of the MDM world.
Summary of Client Projects
In case you think the Hub Designs team has been doing nothing but marketing, writing white papers and magazine articles, speaking at conferences, and volunteering for user groups, here’s a summary of our 2009 client projects:
- Technology provider for vehicle dealers: integration of Oracle E-Business Suite with D&B data
- Payroll services company: integration of Oracle E-Business Suite with external credit information
- Information services company: technical support for customers using Oracle E-Business Suite
- Legal information services company: readiness assessment and product MDM strategy & design
- Simulation and engineering software company: advisor to data governance board
- Manufacturer of oil and gas equipment: integration of Oracle E-Business Suite R12 with D&B
- Software company: built connector between Oracle AR and D&B’s DNBi risk management solution
- Technology company: customer MDM strategy workshop
Out With The Old, In With The New
This past year has been a lot of fun, but it has also been somewhat exhausting. So I’m looking forward to a bit more deliberate pace in 2010.
We’re very excited about the coming year at Hub Designs. We’ve got some great projects underway and in the pipeline, and we’ll be continuing to grow and expand to meet our clients’ needs and market demands.
In closing, I’d like to say how grateful I am to my family, for their patience with my traveling so much and for their unconditional love.
Hidden Costs of Duplicate Customer Data
A client asked me last week about what rate of duplicate data was “normal” in customer master data.
My initial answer was that, among companies that don’t have any formal master data management, data governance or data quality initiatives in place, duplication rates of 10%-30% (or more) are not uncommon.
When I was at D&B, we used to routinely see that level of duplication in client’s customer files.
In a study in the healthcare field, Children’s Medical Center Dallas engaged an outside firm to help clean up their duplicate data:
“Solving both the current and future problems around duplicate records helped Children’s improve the quality of patient care and increase physician acceptance of the new EHR. The duplicate record rate was initially reduced from 22.0% to 0.2% and five years later it remains an exceptionally low 0.14%. The 5 FTEs initially tasked with resolving duplicate records have been reduced to less than 1 FTE.”
“For the Children’s Medical Center, the results were heartening, not only from a care delivery standpoint but also because of the significant cost-savings that can be realized. A study conducted on Children’s data showed that on average, a duplicate medical record costs the organization more than $96.”
So it is possible to get the duplication rate down to really low levels through careful analysis and the application of the right tools, as part of an ongoing data governance program. Even the hospital above (and hospitals are usually not mentioned as practitioners of best practices) was able to maintain a duplication rate of only 0.14% after 5 years.
And there are very real costs to not de-duplicating your customer data. Depending on the functional area (marketing, sales, finance, customer service, etc.) and the business activities you undertake, high levels of duplicate customer data can:
- annoy customers or undermine their confidence in your company,
- increase mailing costs,
- cause hundreds of hours of manual reconciliation of data,
- increase resistance to implementation of new systems,
- result in multiple sales people, sales teams or collectors calling on the same customer,
- etc.
The best studies I’ve seen of the cost of duplicate data have been in the healthcare industry. One study I saw said:
“According to Just Associates, the direct cost of leaving duplicates in an Master Patient Index database is anywhere from $20 per duplicate to several hundred dollars. The lower cost reflects the organization’s labor and supply costs to identify and fix the record while the higher expense reflects the costs of repeated diagnostic tests done on a patient whose previous medical records could not be located.
The American Health Information Management Association (AHIMA) estimates that it costs between $10 and $20 per pair of duplicates to reconcile the records. If the records aren’t reconciled, however, the costs are even higher.”
Here are three more case studies backing up the range I quoted of 10%-30%:
- Once the analysis was complete, Sentara discovered they had a significant duplication rate, over 18%. They had attempted to address the duplication rate in the past through a remediation process, but due to either technology issues or because the cost of merging and cleaning up the duplicates across their many different systems was too high, they had not yet successfully reduced their duplication rate. Source: Initiate Systems success story
- Emerson Process Management faced a tremendous challenge four years ago in getting its CRM data in order: There were potentially 400 different master records for each customer, based on different locations or different functions associated with the client. “You have to begin to think about a customer as an organization you do business with that has a set of addresses tied to it,” says Nancy Rybeck, the data warehouse architect at Emerson who took charge of the cleanup. Working with Group 1, Rybeck analyzed the customer records for similarities and connections using everything from postal standards to D&B data, and managed to eliminate the 75 percent site-duplication rate the company suffered in its data. “That’s going to ripple through everything,” she says. Source: DestinationCRM.com
- Problem: Number of duplicate records: 20.9% of Utah Statewide Immunization Information System records. Impact of Problem: Difficult to find patients in system—key barrier to provider participation, risk of over-immunization—unable to find reliable patient record, cost of unnecessary immunizations, risk of adverse effects on patients. Source: health.utah.gov.
And here’s a good quote from a white paper titled “Data Quality and the Bottom Line” by The Data Warehousing Institute:
“Peter Harvey, CEO of Intellidyn, a marketing analytics firm, says that when his firm audits recently ‘cleaned’ customer files from clients, it finds that 5 percent of the file contains duplicate records. The duplication rate for untouched customer files can be 20 percent or more.”
Every organization will need its own metrics, but left unchecked, the duplication problem is a hidden cost that drags at your company, slowing down your processes and making your analyses less reliable.
If your sales analysis reports can’t be sure that there’s one and only one record for each of your largest customers, then the sales figures for those customers are probably not right. So the entire report becomes suspect at that point.
I’d like to end with a great quote on data quality by Ken Orr from the Cutter Consortium in “The Good, The Bad, and The Data Quality”:
“Ultimately, poor data quality is like dirt on the windshield. You may be able to drive for a long time with slowly degrading vision, but at some point, you either have to stop and clear the windshield or risk everything.”
Please let us know what you think by commenting here. We’re interested in hearing your thoughts on data quality and the issue of customer data duplication.
First Look at Oracle Fusion MDM Hub
“All NDAs are lifted” were the magic words uttered by Steve Miranda from Oracle at the Fusion Inner Circle Event at Oracle OpenWorld on October 15th.
Just to make sure, I asked Steve explicitly during the Q&A section of the program if it was okay under the non-disclosure agreement we had all signed to write about Fusion on my blog, and he said “Yes.”
Hub Designs was invited back in February to help Oracle’s Fusion MDM team with some design review, validation, and testing activities. In return for our assistance, we’ve gotten to see Fusion MDM inside and out, and we can proudly say that we are one of the very few trusted partners who helped Oracle to design and develop the application.
We participated in a lot of conference calls with Haidong Song, Oracle’s Product Strategy Director for Customer MDM, and other members of his team. And we attended a week-long “hands-on validation” event at Oracle headquarters in August, looking specifically at the customer data management aspects of the Fusion MDM hub.
My first impressions of Fusion MDM during that hands-on session were very favorable. I remember thinking to myself, “Oracle could almost start selling this into the MDM hub market right now!”
Of course, Fusion isn’t scheduled to ship until sometime in 2010, and there’s still plenty of work to be done between now and then. But the core functionality needed for master data management was there, and the Oracle Fusion MDM team had a room full of customers and partners banging on it for a week without any significant crashes or issues.
There was plenty to like in Fusion that didn’t relate specifically to master data management – the new and improved user interface, the embedded analytics, the modern, standards-based architecture, the usability research that Oracle has done, the improved business processes, the built-in collaboration capabilities …
But the fundamentals of MDM were strong as well. Haidong and his team demonstrated how to import and consolidate customer data from outside sources, and we did our first hands-on lab session bringing in a small customer data load from a desktop file, such as a list of trade show leads.
We also tested a larger volume of customer data being brought into Fusion MDM through the Bulk Import process.
We did another exercise simulating how a typical customer data steward would identify potential duplicate customers, and then resolve those duplicates by merging the duplicate parties.
We also got a good look at the Informatica components that Oracle is bundling into Fusion on an OEM basis: the former Identity Systems matching engine and the former Address Doctor address cleansing tool. Previous Oracle MDM products like Customer Data Hub have had loose integration with Trillium and Firstlogic for address cleansing, but it’s refreshing to see Oracle investing in deep integration with industry leading solutions.
I think there are going to be a lot of Oracle customers who will move to Fusion MDM as the first wave of their overall migration to Fusion, who will see Fusion MDM as a good way to get some early experience with the Fusion applications family, before committing their mission critical Enterprise Resource Planning (ERP) applications to the Fusion platform.
And in 2010 and beyond, I think will be a lot of potential customers who evaluate Fusion MDM positively on its own merits against competitive MDM hubs. Oracle brings a robust data model, open architecture, and a next-generation approach to master data management, with state-of-the-art matching, data quality, middleware, and business process management.
Please let me know by commenting here what your thoughts and expectations are for Oracle’s Fusion MDM hub.
