by Julie Hunt

It’s always great to get a briefing from a very knowledgeable and flexible founder of a young company: tapping into original vision that has been tempered by the experience of customer implementations can be a rich experience. Recently the MDM Think Tank chatted with Martin Turvey, CEO of Loqate, about location intelligence and data quality. Loqate may be a young company, but in many ways, it is not a “start-up” due to the extensive expertise of founders Turvey and Paul Flew, in the areas of address quality practices and location intelligence — and the technologies that support them. Flew’s strong NLP (natural language processing) background is also an important asset for Loqate.

For most businesses, accurate address data is essential; poor address quality costs companies tons of money and lost customers. Not only is missing address information a problem, but inconsistent address data across repositories impedes data synchronization and calls into question the accuracy of all of the data. Turvey provided an interesting point: over 80% of data has a location element (source: PBBI). This means that verifying and cleansing address data, and then adding geocoding, can greatly enhance the usability of this data.

Use cases for location intelligence include: fraud detection, insurance risk assessments, improving mailing and shipping operations, real time mapping of web content and news feeds. Loqate maintains a repository of reference data comprised of address, geocodes, languages and other data covering 242 countries and territories.

Loqate is one of the few independent software vendors for address quality / location intelligence services, since most of the other vendors have been acquired by the usual suspects. Traditional data management stacks from the large infrastructure vendors are not an option for many companies, especially in emerging markets and the larger international arenas for address quality needs.

So Loqate is positioning its independence as a competitive differentiator that allows it to target solution niches not usually covered by the big infrastructure vendors.  This positioning is strengthened by Loqate solution innovation and future direction, including an upcoming Cloud / SaaS offering. Turvey commented that most MDM solutions fail at what Loqate does well for address quality and location intelligence. As to sales model, the Loqate solution is only available through Loqate partner companies that OEM or sub-license the technology.

The components of Loqate are broken down into:  Verify, Geocode, Register, and Identify. Additional technology includes the Geo-Data Quality Engine and the Global Knowledge Repository.

In Verify, where validation and clean up are performed, powerful differentiation exists in the parsing methodology. Based on lexicon and context, parsing algorithms understand what the data should be, based on context parameters. Loqate parsing capabilities analyze and remediate data in situ, and do the hard work on single strings without breaking the string into components first. In fact, the string’s structural context provides cues for Loqate’s parsing / validation. Most other data quality tools require users to do quite a bit of hands-on mapping and other tedious work as preparation for remediation. This isn’t the case for Loqate Verify. Parsing is also trans-global for processing records and strings with mixed international character and language sets. Data can also be transliterated from one character set to another character set (e.g. Roman to Katakana in Japan)

Turvey cited a use case for Fraud Detection: cheats will use fake addresses made up of components that are valid on an individual basis (real street address, real city), but in combination the components do not add up to a valid address. Most other address quality and data quality solutions will vet the individual components but not the composite, which means they may validate a fraudulent address.

Loqate provides real value by trapping errors at data entry time when remediation is at the cheapest point. It’s also essential to trap inconsistencies at data entry to prevent fraud and block bad data from infecting processes such as shipping that can greatly impact costs.

Geocoding is available for every address data element “out of the box”. The geocode assignment results from high granularity generation of a latitude-longitude coordinate for any address worldwide. From the geocode a map visualization can be created, accurate to delivery point / rooftop levels.

Register provides smart data entry to reduce errors and speed up operational processes. This is particularly useful when working with customers, whether it’s a call center employee talking to a customer or a self-service interface for eCommerce.

Identify:  The Loqate Entity Extraction module enables location, additional geographic and contact information to be identEified and extracted from web content and other unstructured information sources. LocationV extractions enable content to be localized, driving location-based services and geographically targeted advertising.

Loqate has included the ability to quantify address data quality through its Accuracy Code analysis. A verification accuracy match score portrays the similarity between input data and the closest reference data match as a percentage between 0 and 100, with 100 as best match, as well as detailing what changes have been made to each key component during processing and determining validity of those components.

Right now, user tweaking is limited and is mainly accomplished through processing rules. There is no means as yet to tweak algorithms. Loqate is taking a slow approach here, being careful not to lead users astray, since algorithm changes can end in unexpected and harmful consequences.

Next up for Loqate is an impending Cloud / SaaS offering that should be available Spring 2012. Loqate is touting their Cloud services as “developer source for anything location”. It is also a great opportunity for further innovation since the Cloud can take software vendors in some very interesting directions.

About the author: Julie Hunt is a software industry strategist and analyst, providing market and competitive insights. Her 25+ years as a software professional range from the very technical side to customer-centric work in solutions consulting, sales and marketing.  Julie shares her takes on the software industry via her blog Highly Competitive and on Twitter at @juliebhunt   For more information, please visit Julie Hunt Consulting – Strategic Product & Market Intelligence Services. 

Not because we need to jump on the social networking bandwagon to keep MDM interesting and relevant. But because companies like Google (with Google+) and data.com (formerly Jigsaw and now SalesForce.com) have already cemented the relationship between the two.

The evolving conversation regarding MDM and social networking has two perspectives. One angle considers how to use members of a social network in the management of master data. The other addresses how to use MDM for managing data about members of a social network.

The first context for discussing MDM and social networking has been a subject of interest for some time. Crowd-sourcing is the basic social networking mechanism used to accomplish this goal. The “crowd” can be a well-defined group of employees or a coalition of like-minded members of an online service.

The second recently came to the forefront when Google chairman Eric Schmidt proclaimed that the company will only accept “real names” in Google+. According to GigaOM, Schmidt admitted in an interview “Google is taking a hard line on the real-name issue because it sees Google+ as an ‘identity service’.”

That sounds a lot like an MDM policy in the making.

Marrying MDM and Social Networking

The idea of bringing social networking and MDM together has its roots in “community data stewarding,” which some in the MDM space started touting a few years back. The idea is to leverage the collective efforts of individuals within an organization to help manage master data. More recently the notion of community data stewarding expanded to include using “the masses” for MDM.

Online data services like data.com, which is the reincarnation of Jigsaw after it was acquired by SalesForce.com in 2010, is a good example of mass data stewardship.  Jigsaw is probably the first real adaptation of social networking, specifically crowdsourcing, to managing data about people and organizations.

Even though the stated intent of Jigsaw was to share contact names, I can easily argue that Jigsaw is a social MDM application. By tapping the universe of salespeople for information about people and organizations, Jigsaw, now data.com is, in effect, the first commercially successful crowdsourced Master Data as a Service (DaaS).

Master DaaS is different than MDM SaaS, which is the delivery of MDM software capabilities in the cloud. Personally, I believe all the action is (or will be) at the intersection of Master DaaS and MDM SaaS, which is where data.com is likely heading. But that’s a conversation for another time.

One can also show that even LinkedIn is readily re-purposed as a Master DaaS, where individuals manage their own identity. Consider offerings such as “LinkedIn for Salesforce.com.” It’s not much of a leap to see how contact names in SalesForce.com are indirectly managed by the contacts themselves through their LinkedIn profiles.

While understanding how to adapt crowdsourcing dynamics to data stewardship is worthy of further consideration, this article focuses on the second context: how to use MDM to manage data about members of the crowd, i.e. a social network. This is where the conversation gets contentious, as Schmidt unwittingly demonstrates.

At first blush, all the fuss about banning pseudonyms on Google+ seemingly has nothing to do with MDM. But it does. Very much so.

Master Identity Keeper

According to GigaOM, during an interview with Andy Carven, National Public Radio’s digital editor, Google’s Schmidt stated:

“G+ was build (sic) primarily as an identity service, so fundamentally, it depends on people using their real names if they’re going to build future products that leverage that information. If you think about it, the Internet would be better if we had an accurate notion that you were a real person as opposed to a dog, or a fake person, or a spammer or what have you… So if we knew that it was a real person, then we could sort of hold them accountable, we could check them, we could give them things, we could you know bill them, you know we could have credit cards and so forth.”

Fred Wilson at Union Square Ventures further notes, Google “wants to be an identity gatekeeper,” where individual crowd members basically manage their own identities.

Schmidt and Wilson suggest what sounds like the ultimate Master DaaS for person-type parties. But it raises some interesting questions. Does a business need to definitively know the individuals that they interact and transact business with? While this may seem absurd to even ask, apparently we are at that crossroad. And from the dialogue it seems not everyone agrees on the answer.

Is MDM Evil?

To date, a business naturally assumes that a customer cooperates by exposing their identity. After all, how else can you conduct business?

That said, there are many instances of businesses that don’t know the identity of their customer. P&G, for example, cannot readily identify the consumers who purchase, say, Tide®. The “customer” represented in P&G’s business systems are retailers and wholesalers and some of their employees, but not likely the person pouring Tide® into their washing machines.

On the other hand, take a retailer like Target. Recently, they had some challenges with handling online purchases as described in this story on CNN.

“…shoppers who bought the Missoni Target line are posting on social media websites Facebook and Twitter that they won’t shop at Target again because their online orders are being delayed — or worse, canceled — by the retailer.”

Before the internet, Target did not likely know the identity of many “shoppers,” i.e. customers who made their purchases at a store, unless they had a Target credit card or exposed their identities at the cash register. And even if they did have shoppers’ identities, they would not likely have the capability to resolve the relationships among them, which is a major goal for social media sites.

In the past Target’s customers for MDM purposes were suppliers and wholesalers and not “shoppers.” Just like P&G. With online purchases, however, this is no longer the case.

Of course, Target needs to know the identity of the suppliers and wholesalers and those related contact names. The question is this. Will they ever know the identity of the supplier or wholesaler contact, who, when behaving also as a shopper, is bad mouthing Target on some social media site under a pseudonym…or is part of some anti-Target group? Meanwhile, a Target representative may just have had a seemingly pleasant lunch with the person. They’re being blindsided.

Now consider this. The blog GIZMODO states “Google’s horrible new policy on using real names in Google+ effectively means that the service is now a danger to real people.” (As opposed to fake people?) And “…the policy is evil.”

GIZMODO adds:

“You can’t use initials (even if that’s what you go by). You can’t use a pseudonym (even if that’s what you go by). And you can’t use numbers or symbols (even if they are part of your name).

Æ, e.e. cummings, Malcolm X, and T.S. Eliot would all be in violation of Google’s policy. So, too (by my reading) would be Mark Twain, George Eliot and doubly so, R.U. Sirius. I’m pretty sure nobody whose name you actually know in the band U2 can use Google+ or, by extension, Gmail.”

Play Nice

Apparently Google seeks birth certificate-quality names. Is this the same strength of policy making that MDM needs to succeed?

Traditionally, the answer is not necessarily, because any decent MDM technology will have the ability to relate multiple aliases to a person party. But as MDM moves to manage master data about your customers who are also members of business and social networks, not all aliases are willingly shared.

Everyone knows Mark Twain and Samuel Clements is the same person. And even if we did not, someone besides Twain himself would ultimately have to know his real identity. Otherwise how would he collect his royalties?

And here is the blind spot for MDM moving into the social realm. A 360-degree view will be without those structured or unstructured data relationships of a party that may only be revealed under a fantasy name…and not disclosed to you or anyone else.

Selling and marketing may get a whole lot harder as individuals increasingly armor themselves with pseudonyms and fantasy names (or personas). In response, we see the rise of strict policies enforcing identification…or don’t play on my social media site.

Unfortunately, that’s a double edged-sword, and the physics of it all has yet to get worked out. On the one hand, Facebook and Google have a threshold mass where people have to play by the rules or get booted. On the other hand, the resistive power of the masses can influence those policies. Just ask Netflix.

When a Customer Identity Is the Product

Consider Don Norman’s observation about Google+. Normanis the author of The Design of Everyday Things. He says, “Real names, they (Google) say, turn out to be the names on your driver’s license and your passport and your credit cards so that they can track you.”

He goes on to say Google’s “goal is to gather all the knowledge in the world in one place, but really their goal is to gather all of the people in the world and sell them.”

This appears very much like a goal of MDM: to gather all of the people in your business circle in one place so you can ultimately track them. All, of course, with the intent to manage a relationship with them…um sell them something.

I unequivocally say, yes, this is a goal of MDM. But there is a slight twist to the Google+ story.

Norman also submits that “Most people would say (of Google) ‘we’re the users, and the product is advertising. But in fact, the advertisers are the users and you (we) are the product.”

Typically, MDM assumes we are talking about the identities of persons and organizations like customers, suppliers, employees, partners and the like. Norman fundamentally asks, when a person’s identity is a product, do we have a right to enforce them to share that identity as a condition of use…I mean, participation in a service?

Frankly, I am not quite sure what the big deal is. Credit bureaus, like Experian and other companies like Acxiom, have sold our identities as a product for years.

Data Is Not An Asset. A Person’s Identity and Relationships Are.

According to GIZMODO, “Forget social networking, the big goldmine of the future is online identity verification.” Bingo, that’s Master DaaS.

Two questions related to MDM are at play here. (Since we are solely discussing individuals and groups of individuals …and for simplicity, I exclude master data on other entities such as products.)

1. How do you have transactions and interactions and perform analytics without, well, master data? After all, a party’s name is master data. In other words, how do you create an order or an account without header information about the party?

2. Is there an obligation for people who do business with you to correctly identify themselves? I am not sure how else business can occur. Even setting up a Swiss bank account requires identifying the account holder. Although, I cannot say for certain. I’ve never had enough money to hide from anyone!

Speaking of bank accounts, Dave Winer thinks Google “wants to effectively become a bank.” According to Weiner, knowing the relationships among persons is where the value of sites like Facebook and Google+ is. Knowing and managing the relationships among persons happens to be an important function of MDM too. We can think, then, of a MDM hub as a sort of bank of person and organization identities and the relationships among them.

If nothing else, one thing has been settled. Data is not an asset. The true identity…and corresponding relationships of a party are. It’s an asset to a person. It’s an asset to a business. Even if we mean “asset” rhetorically.

In my opinion, Google+ and other social media sites purporting to be identity services should offer an incentive like Jigsaw does. You gain points by sharing the identities of people you know. If anything, Jigsaw is the evil one. It’s a platform for everyone to “rat out” everyone else they know by disclosing their name, employer, work address, phone number and email address.

Identity and Relationship Resolution and Keeping

The concern voiced by Google resonates in MDM because a single representation of customer information and a “single version of truth” are frequently stated desired outcomes. Given the largest percentage of MDM initiatives involves resolving and managing identities and relationships, we need to reconsider what and who is a party to a transaction and interaction.

For starters, we need to rethink the notion of “customer.” It’s becoming an increasingly marginalized, if not obsolete and irrelevant concept. “Customer” seems inadequate for collectively describing the myriad roles played by all the participants in business transactions and interactions.

If abandoning “customer” altogether is too radical, we should limit it to a very narrow meaning. Take a health insurance company for example. They narrowly define a customer as the employer sponsoring a health plan. That’s it.

An employer a.k.a customer is only one of many participants in their business transactions and interactions. Other participants include subscribers, providers, members, administrators and brokers. They collectively refer to these participants as constituents, and any participant may play more than one role. A provider, for example, can also be a subscriber or member, or if the provider is an organization, such as a hospital, the provider is also an employer that sponsors a health plan for its own employees.

Additionally, we need to recognize that the collection of participants forming a crowd, group, organization, circle or whatever we call it is also an entity. Crowds, circles and groups are similar to organizations, except they usually don’t have a legal standing and tend to be virtual. Nevertheless, they are living, breathing entities that matter and require identity and relationship resolution. This is particularly the case if they are participants or potential participants in a transaction, interaction or analysis.

We can no longer easily segregate the persons and organizations we engage in a transaction or interaction into separate entities like customer, supplier, employee, partner, user, shopper, guest, donor or whatnot. It’s time to recognize these concepts and terms for what they are: role designations, not discrete entities.

The entity is the person or group. That’s why some organizations now collectively recognize all of them as a single entity and dub it a “party” or “constituent” or the like. Where the entity can behave in one or more roles as a customer, supplier, employee, partner, shopper or whatever.

Personally, I prefer “participant” to describe the individuals in a group. “Member” or “party” is probably easier on the brain, but “participant” is an action word, even if it doesn’t nicely roll off the tongue. “Constituent” works too, but sounds a little formal.

I also prefer Google+’s “Circles” to simply “group” or to, say, Facebook’s “Smart Lists” when depicting a crowd. After all, people travel in circles, not on lists! A “list” is too linear, even hierarchical sounding. “Circle” is a clean break from the chronic misrepresentation of all relationships as hierarchical. Plus, we can be participants in multiple related or unrelated circles.

“Smart group” might work too, where smart means identifiable. Then, of course, there is always “network.” A network is neither linear nor hierarchical but it’s too clinical sounding. We can always just say persons and organizations. But that’s unimaginative.

Instead of categorizing persons and organizations by their role in transactions and interactions, just designate them as participants in a circle. Some seek to distinguish social networking and business networking, but in the end, we can’t always distinguish interactions as social or business.

The following table depicts the evolving conceptual models for representing participants and circles in a system. While the goal is to move toward a Stage VI model, most organizations will have systems representing participants and circles in transactions and interactions at different stages.

Where Do We Go From Here

Deriving a panoramic view of fragmented customer data for transaction processing, business interactions and analytics involves multiple parties arranged in complex network configurations. But not all these parties traditionally and simply behave as a “customer.”

Business applications must rapidly evolve to a Stage VI model that can support different sets of relationships among persons, organizations and groups and different sets of data relationships for them. A blind spot may still exist where the identity of participants in a combined social and business circle are irreconcilable.

Until business applications can support a Stage VI model, MDM will have an interim function as keeper of participant and circle identities and relationships. However, even with MDM, separate data relationships based on participant role may not be fully supportable until business systems themselves can support complex network configurations.

Once business applications can support a Stage VI model, they still may not be suitable for managing all participant relationships for two reasons. One, if enterprise applications are not optimized for a large, complex arrangement of participant and data relationships, performance could be a significant roadblock. And two, a production transaction system may not be optimal for dedicated data stewards and curators to maintain participant identities and relationships.

For these reasons, the importance of MDM will increase as it becomes the main mechanism for managing current and historical participant and circle identities and relationships with pointers to corresponding data relationships. As cloud computing penetrates enterprises, resolving and keeping participant identities and relationships will increasingly occur as a combination of MDM SaaS and Master DaaS.

The Master Data Management Track of this conference will be the best we’ve ever had.  There’s been a separate MDM track for five years, since I joined the Education Committee of the Oracle Applications Users Group in 2006.

This year’s COLLABORATE conference will be held in Las Vegas, Nevada at the Mandalay Bay Convention Center on April 22-26, 2012.

We’re looking for papers from new people and returning speakers, on topics like Customer MDM and Product MDM, as well as other domains of MDM. Multidomain MDM – the ability to manage many different domains of data within one MDM hub – would be a great topic.  Data governance is another hot topic that people definitely want to hear about.  Here are the sub-categories that you can use within the MDM Track:

• Best Practices
• Data Quality
• Data Governance
• External Content
• MDM Hubs
• Organizational Change

We’d like a good mix of presentations from end users and consultants. This year, we want more case studies, more real world presentations, and more examples from different industries.

To respond to the MDM Track Call for Papers before the October 17th final deadline, just click here.

VERY IMPORTANT: be sure to indicate the “Master Data Management (MDM)” track (and only that track) in your entry so it doesn’t get lost in the shuffle. When we “build the room”, it’s hard to find your submission if it’s got 4 other tracks added on to it. So just stick with the MDM track, please.

If your paper is selected, you’ll get a free conference registration.

Don’t forget – click here to submit your paper before the October 17th final deadline! 

The Master Data Management Track of this conference will be the best we’ve ever had.  There’s been a separate MDM track for five years, since I joined the Education Committee of the Oracle Applications Users Group in 2006.

This year’s COLLABORATE conference will be held in Las Vegas, Nevada at the Mandalay Bay Convention Center on April 22-26, 2012.

We’re looking for papers from new people and returning speakers, on topics like Customer MDM and Product MDM, as well as other domains of MDM. Multidomain MDM – the ability to manage many different domains of data within one MDM hub – would be a great topic.  Data governance is another hot topic that people definitely want to hear about.  Here are the sub-categories that you can use within the MDM Track:

• Best Practices
• Data Quality
• Data Governance
• External Content
• MDM Hubs
• Organizational Change

We’d like a good mix of presentations from end users and consultants. This year, we want more case studies, more real world presentations, and more examples from different industries.

To respond to the MDM Track Call for Papers before the October 14th deadline, just click here.

VERY IMPORTANT: be sure to indicate the “Master Data Management (MDM)” track (and only that track) in your entry so it doesn’t get lost in the shuffle. When we “build the room”, it’s hard to find your submission if it’s got 4 other tracks added on to it. So just stick with the MDM track, please.

If your paper is selected, you’ll get a free conference registration.

Don’t forget – click here to submit your paper before the October 14th deadline! 

I’m still a volunteer on the Education Committee of the Oracle Application Users Group, which is a robust users group that is completely independent of Oracle Corporation. I’ve been involved in the group as a whole for over 15 years now, and have been on the Education Committee for more than five years.

The Call for Papers for the 2012 COLLABORATE conference is open, and I am starting to plan the MDM track of the conference.

We are going to make this conference – to be held in Las Vegas, Nevada, USA at the Mandalay Bay Convention Center on April 22-26, 2012 – our best ever. We are looking for papers on Customer Master Data Management (MDM) and Product MDM, as well as other domains of MDM as well. Multidomain MDM – the ability to manage several different domains of data within one MDM hub – is a popular topic, as is anything to do with data governance.

We’re looking for real world success stories, with presentations from end users at deploying companies as well as from consultants who have been part of MDM and data governance projects.

Last year, we had 32 different MDM track presentations, with titles such as:

• Managing Global Customer Data Using Oracle Data Quality Management and Party Merge – One Year Later, by Mike Foxcroft and Nick Soh
• MDM Case Study: Making Information Management the Foundation of the Future, by William McKnight
• The Strategic Nature of MDM, by David Butler of Oracle
• MDM Best Practices and Implementation Approaches, by Mani Kumar Manda
• Oracle MDM Strategy and Roadmap, by Manoj Tahiliani of Oracle

But this year, we want to go above and beyond last year. We want bigger and better case studies, more real world / end user presentations, different examples from different industries. Share what you’ve learned with your fellow practitioners in the MDM Community.

To respond to the MDM Track Call for Papers before the October 14th deadline, just click here. Be sure to indicate the MDM Track in your entry so it doesn’t get lost in the shuffle. If your paper is selected, you’ll get a free conference registration.

Don’t forget – click here to submit your paper before the October 14th deadline! 

The combination of Siperian with Informatica has exceeded their expectations so far, with MDM revenue running significantly ahead of quota and Informatica landing customers in a number of new vertical industries such as retail, healthcare, aerospace/defense, agriculture, education, and hospitality. Informatica continues to penetrate EMEA and has had its first successes in Asia Pacific and Latin America as well.

There’s also a healthy sales pipeline being built for future quarters, with the top three verticals being healthcare and life sciences, financial services and insurance, and high tech and retail. Growth is being seen all over the world, with a large percentage of the bigger sales opportunities for Informatica involving MDM, regardless of the region.

Ravi highlighted how the Informatica Master Data Management (MDM) solution is solving multidomain business problems like physician spend compliance, product mastering, high volume reference data mastering, clinical trial management, customer and channel management, and Salesforce.com enablement. He also discussed how Informatica’s other products usually fit into an MDM solution.

As the Informatica MDM product has evolved, it has remained true to its roots, and continues to offer complex hierarchy management, to be business user focused, and to allow for fast time to value. What Informatica has done, building on what Siperian had created before its acquisition, is to provide for true multidomain master data management, which allows for a much wider range of problems to be solved.

Informatica continues to increase its market share beyond the pharmaceutical vertical, and shows a strong track record of expanding its footprint within existing customers as well.

Informatica MDM Data Director has been widely used as well, with every new customer since its release in March 2009 buying it along with the MDM hub.

Informatica just finished up an 18-city MDM road show in the U.S. and Canada, and featured its MDM product prominently at Informatica World in early November. It has both a horizontal and a vertical industry marketing strategy.

Ravi previewed for us the materials for their “Customer and Channel Management Solution”, which manages hierarchies and relationships between customers, channel partners, products, and resources, in order to maximize account penetration, optimize coverage, and enable business agility and speed.

Ravi also gave us a demo of the latest version of the Informatica MDM product, with built-in dashboards using Data Director measuring data quality for individual customers and organizational customers. He also demonstrated the integration of MDM with the rest of the Informatica Platform – Power Center Business Glossary and Metadata Manager, and Informatica Data Quality.

Another impressive feature is enabling business applications, such as Salesforce.com, to be MDM aware. New records can be entered in the Salesforce.com application and instantly be bounced up directly against the Informatica MDM hub, and customer hierarchies can be viewed in a Salesforce.com tab, rather than requiring the user to jump back and forth between a Salesforce window and an Informatica MDM window. And the Salesforce user can see a timeline of a record “as of” a particular date, including all the hierarchy data.

At the end of the briefing, I came away feeling (again) that Informatica had made a great move in purchasing Siperian, and that Informatica’s MDM business has clearly gained momentum since the acquisition. This is clearly one of those cases where one plus one equals three. Informatica has done a great job integrating Siperian into the company, in taking advantage of the synergies between the two companies, and in promoting the product. Opportunities exist to take it even further, but the Informatica team is to be congratulated, since almost 60% of all mergers and acquisitions fail to create shareholder value according to the Boston Consulting Group.

Here’s an except:

“I’ve been an MDM evangelist since 2004 and have worked with a lot of people either becoming MDM process owners or helping shape another person within their company into one. Here are some tips if you’ve had that role handed to you, are aspiring to it or want to interest someone else in your company in stepping up to it.

First, embrace the political aspects of MDM. I wrote an article in the March 2008 issue of Information Management called “The Politics of Master Data Management and Data Governance,” in which I recommended that people start by understanding the political landscape at their company when creating a plan. Who are your likely allies and opponents? How will you get your initial funding and accomplish implementation? And don’t forget to plan for data governance.”

Click here if you’d like to read the full article.

I’ve been involved in the Oracle Applications Users Group (OAUG) since 1995, and have been a member of the OAUG Education Committee for several years now. The Education Committee is starting to plan next April’s COLLABORATE 11 Conference, and I’m managing the “Master Data Management” track.

Together with the Special Interest Group (SIG) coordinators for the Customer Data Management SIG and the Oracle Enterprise Product Lifecycle Management SIG, we invite YOU to submit a paper for the 2011 conference’s MDM track.

Our vision for the MDM track at COLLABORATE 11 is to have:

• a good mix of beginner, intermediate and advanced topics,
• presented by a range of well known and first time presenters,
• covering master data management, data governance and data quality,
• including domains such as customer, product, supplier and site,
• discussing Oracle products such as Oracle Customer Hub, Oracle Product Hub, Oracle Supplier Hub, Oracle Site Hub, Oracle MDM Data Quality, Oracle Higher Education Constituent Hub, Hyperion Data Relationship Management, and Oracle Fusion MDM.

Here are the important facts from the OAUG Call for Papers:

You’ll have the opportunity to connect with more than 5,000 users, technology leaders, Oracle executives and solution innovators gathering for the user-driven education and networking event April 10 – 14, 2011 at the Orange County Convention Center West in Orlando, Florida. Proposals are now being accepted. The deadline is Friday, October 1, 2010 at 11:59 p.m. EDT. To submit a paper, go to http://collaborate.oaug.org/submit/.  For more information, you can go to http://collaborate.oaug.org/presenterinfo/.

Note to Oracle Employees: All Oracle employees interested in speaking at COLLABORATE 11 are to submit your papers through the Call for Papers submission form. Please contact speakerprograms@oaug.com for assistance with technical difficulties. For all other inquiries, please contact Lisa Stuart at lisa.stuart@oracle.com.

Continuing a tradition we started in 2008 with an article called “Lessons on MDM from My Summer Vacation“, I’ll try to sum up some things I learned along the way, and apply them to master data management and data governance where I can.

1. Be Prepared for Storms

On one passage from Red Brook Harbor to Cuttyhunk, we were hit with a nasty thunderstorm that wasn’t forecast to go through until much later in the day.  Winds were clocked at 50 knots (58 miles per hour). We prepared by dousing our sails, getting our foul weather gear on, battening down the hatches, and getting the boys in safe positions down below.  But when the storm hit, the rain on my face felt like needles, visibility dropped to zero, our dinghy flipped over on its towing bridle, and I had to concentrate on avoiding a buoy in the area.

The application to MDM is that, given how political these projects can be, there will be storms.  So be prepared for them.  Have a good crew (project team), work hard at instilling loyalty between the team members, and maintain a united front.  In our case, the storm, though intense, passed quickly, and we were able to get our dinghy right side up and resume our course for Padanaram with no injuries or damage.

2. Don’t Try to Control Too Much

Co-organizing a sailing trip with 15 boats can be a bit like herding cats.  Sailors are very independent by nature at best, and even though we had regular check-ins by radio, some people would skip them completely, and others would forget (including me). Traveling with two young children increased the chaos factor.  We’ve learned to go with it a bit – it’s like riding a wave.  You can’t plan every minute of every day – sometimes you’ve got to be spontaneous, put the plan aside and just see what happens.

In the MDM and data governance world, the business community as a whole, even though they may not be on your project team directly, is going to be directly affected.  They’ll want to have a say in how things are done, and they’ll have good ideas for you.  Don’t shut them down.  Learn to listen, actually consider what they’ve got to say, and be inclusive.  Have town hall meetings where the broader business community gets a chance to tell you about their concerns, where you communicate the project’s progress and milestones, and where you can reach out to them and pull them in to upcoming phases.

3. Accept the Kindness of Others

Previously, we had a 32 foot boat, but at the beginning of June, we took delivery of a 38 foot boat, which we were still getting the hang of. A couple of club members on the cruise took the time to help us get to know the systems on our new boat, and it was great to have experienced friends walking us through what was, to me, new territory. Whether it was the selector switch between water tanks, the fresh water pressure pump, the anchor wash down pump, or various other things, our friends took the time to mentor us on the ins and outs of our new boat. And on the last day of the cruise, our friend Fred remembered that our son Brendan wanted a ride in his skiff, so he came alongside as we were leaving the harbor, picked him up, and gave him the ride of a lifetime.

The application to MDM and data governance is that you should be open to mentoring within and outside the enterprise.  People like sharing their experience and wisdom with others, once you’ve established a strong relationship. If you reach out and develop a network of contacts inside and outside the company, then when the stuff hits the fan, you’ll be able to call on them for help.  And even when you don’t need help, you’ll find a ready group of mentors who’ll take you under their wing, to teach you the finer points of leadership skills, project management tips and tricks, communications and marketing excellence, business process redesign and organizational change management basics — all the things you’ll need to succeed in your MDM and data governance initiative.

4. Stay on Schedule

There were several times during our sailing trip when we were tempted to stay an extra day or leave a day early from one place or another.  We talked it over as a group and decided to stay on schedule.  Many of us had made mooring reservations at marinas with strict cancellation policies, and we would have ended up paying for those moorings even though we didn’t use them.  Not a big deal in and of itself, but we asked ourselves, what’s the worst that could happen if we stuck with the original schedule?  It turned out that it wasn’t that different from what would happen if we went with a changed schedule.

In the MDM and data governance world, as in any technology implementation, there are going to be unforeseen obstacles.  Try to build some cushion into your project plan, so the smallest little delay doesn’t impact your critical path and delay the overall project.  When you get to the point that to stay on schedule means sacrificing functionality or increasing costs (the famous “triple constraint“), the discussions start getting pretty heated. There will be many times when your project will feel like you’re herding cats too, but remember how important it is to stay on schedule. You can’t finish on time if you get behind shortly after you start.

5. Look for Those Special Moments You’ll Always Remember

There were quite a few special moments on this vacation. Shortly after we arrived in Cuttyhunk, both of my boys put on their bathing suits and dove off the boat into the harbor.  They swam fearlessly from Blue Water boat to Blue Water boat, saying hello to their friends, until we had a bunch of kids in the water doing the same thing, including one little girl that had never done that before (and who made her dad very proud).  That night, after going to the beach, we had a lobster bake that I organized for 33 people on the lawn overlooking the harbor.  I will remember the conviviality and friendship of that dinner for a long time.  And there were small moments too: body surfing with my youngest son in Westport, getting airborne in the dinghy, slogging through the passage to Block Island against 25 knot winds, foul currents and 4-6 foot seas (even the hard times can be good memories after you get through them).

For MDM and data governance practitioners, there are many rewards: the satisfaction of bringing in a challenging project on time and on budget, forging relationships with team members that will last a lifetime, learning new things and expanding professional horizons, being recognized by the company as a valuable player capable of big things, mastering MDM and data governance at a time when having those technologies on one’s resume certainly doesn’t hurt one’s career prospects, and so on.  For a good look at what is involved in being a “data champion”, and the rewards involved, read “So You Want to be a Data Champion?” by my friend, Tom Carlock.

To sum it up, if you’re prepared for the inevitable storms that will come your way and don’t try to control things too much, and are open to the kindness of others while remembering the importance of staying on schedule, you’ll certainly be blessed, as I have been, with a wealth of those special moments you’ll always remember. Master data management and data governance can be challenging, but they can be very rewarding as well, both for the organizations which take on the initiatives and for the individuals who make up those teams.

Part 5: The Profiling Payoff

This is the final part of a five-part series, describing how data profiling benefits both IT projects and business operations.  In Part One, we discussed profiling perspectives.  In Parts Two, Three and Four, we introduced the value of system, entity, and attribute-level metrics.  This part discusses the archival and beneficial uses of profile results.

If you have defined your corporate data profiling strategy similar to the methods discussed in the preceding parts of this series, you’ll have amassed a robust collection of metadata spanning relevant systems across your business.  Although systems may be of different types and locations, the structured approach and common metrics you collected create a centralized repository of information that can be examined holistically. Ideally, this information will exist in an open-source database repository with reports made available across the enterprise. System and Entity information help planners and developers organize information strategies. Attribute-level domains, constraints, and business rules help data architects understand existing systems. Relationships and value patterns are readily available to support validation of information-related hypotheses as needed.

If you plan to design your own repository, consider adding timestamps and indicators to help you manage and present the information.  To keep your repository relevant to business needs, design collection rules to be configurable. This allows you to easily ignore superfluous information or enable tests only at certain critical times. Allow initial system profiling efforts to gather a large set of metrics and store them as your baseline.  As you learn about the information, you will see which tests or which data objects add no value.  Us geeky DBA-types who understand system-level catalogs have our own scripts to do much of what was described inParts Two,Three and Four. Those less-inclined may prefer to use a third-party tool for profiling. Either way works as long as the business needs are satisfied and the entire enterprise standardizes on one approach (and thus one integrated repository).

You will find that collecting and maintaining this level of detail has a definite cost.  Even if the collection is automated, interrogations of large data sets places an overhead on production systems that may not be practical. Record and monitor profile execution metrics to identify bottlenecks or tuning opportunities. Realize that the extent of data profiling is contingent on the project phase, specific data elements, and most of all, business value. Review profiling goals on a regular basis and eliminate unnecessary and redundant checks.

How much profile history to maintain is another consideration.  Even though disk is “relatively” cheap, maintaining all historical entries in a live repository may not be necessary. Consider business needs and value for historical profile information. Even consider archiving at a summarized (or less frequent) level and keep only a limited time window of statistics online.

This discussion on data profiling was intended to broaden perceptions of what it means to a business and the value it can bring if done in a sustainable way. The blog format is not conducive to in-depth discussions, but hopefully the topics covered here spur some thoughts into how you can add value to your business by implementing some of these concepts.  Use your imagination, but remember that no matter how cool it might be to collect and store some profile output, if it does not add business value to somebody, it might not be worth the overhead to continue recording it.

Part 4: Profiling Relationships and Patterns

This is part four of a five-part series describing how data profiling assists in all aspects of system development, from design through deployment.

Part One introduced different perspectives on data profiling. Part Two identified valuable system and entity metrics to track. Part Three discussed attributes. In this segment, we dive deeper into attribute relationships and pattern recognition. Also, we expand on primary key identification discussion and discuss hidden relationships.

Pattern grouping provides a mask of distinct format patterns within an attribute data set and a count of the number of occurrences. Patterns give insight into the type of values found in an attribute. For example, a numeric pattern analysis may show values such as 999.99999, 99, or -.9999.

Observing distinct patterns gives insight into the maximum digits and precision, and also domains such as integer or real. Pattern of a database date or date-time type provides unremarkably similar patterns for all dates. Because the database management system typically enforces the domain, date analysis provides no value and can be ignored. If dates are stored in character format, however, patterns quickly show variations in date formatting. Character patterns only have significance to a limited number of positions. It makes no sense to pattern a description field of 200 or 2000 characters. Smaller code attributes of less than 10 characters though do provide value. Ignore pattern profiling for character strings over 20 characters at first, then refine to shorter character strings if the results do not add value.

In pure database theory, referential integrity (RI) is your friend. In practice, designers and software vendors often forgo RI to improve system performance on data inserts. These designers place the data quality burden on the application and do not endorse external data manipulation outside the application interfaces. In the real world, though, data corruption occurs and without RI or routine data quality checks, corruptions may not be found for a long time or not at all. Personally, I have identified over $50,000 of recent orphaned sales in a retail client resulting from deliberately disabled RI. These unreported sales were not added to the ledger and were allowed to occur for performance reasons until I found them through simple profiling. Enforcement of RI is a topic for another discussion but is mentioned here because it does identify a valid reason for data profiling.

In even presumably good relational designs, some parent-child relationships are not enforced for different reasons. First, interrogate the RI listed in the system catalogs to identify all enforced relationships. Reverse-engineering a system with a good modeling tool is probably the best way to do this. A harder and more valuable analysis is to identify unenforced relationships and determining the probability of the relationship if not all values are an exact match. Do this by counting all the candidate child attribute values that exist within a known parent attribute table. If all match and there are a non-trivial number of matches, there is a good probability of a non-identified relationship. A small number of mismatches could identify data quality issues.

In Part 5, we tie all the techniques discussed in the first four parts together to show the value of a repeatable data profiling process.

Part 3: Attribute-Level Analyses

This is part three of a five-part series on data profiling.

In Part One, we took a light-hearted view of where profiling benefits an organization and in Part Two, we discussed the fundamentals of a profiling strategy.  The remaining three parts discuss attributes, relationships, patterns, and how to use the combined data profiling information you collect.  In this section, we introduce attributes, the lowest-level components of a profiling effort.

An attribute is simply a individual data element.  Alone, an attribute has no context.  Given the simple descriptor of “Cost” for an attribute tells us very little about the attribute’s true purpose and immediately drives a need for additional information, such as units (hours, Dollars, Euros…), type (weighted, unit, gross…), and use (invoice, sum, average…).  Attributes therefore must be analyzed within the context of their business purpose to have meaning.

Some characteristics require business knowledge to define and others can be determined through interrogation of existing values and underlying rules of the storage medium. It takes both analyses to get a complete picture of information within a system. While assembling this puzzle, though, keep in mind that until you validate the enforcement of business rules, only assumptions can result from physical profiling or business context.

Analyses of values, domains, and constraints allows insight into use (or abuse) of an attribute. The larger the sample size, the better confidence you gain in the results. Without explicit proof of business rule enforcement, though, you must assume that just because a value does not presently exist does not mean it cannot exist. Business rules are defined by business experts and enforced through database constraints, data type/precision, and application code. Knowing the methods of enforcement allow you to narrow a domain but not totally understand it. Profiling of actual values provides additional refinement in terms of percentage of NULL values, percentage of distinct values, minimum, maximum, and average values, top x and bottom x recurring values along with their counts, and minimum, maximum, and average data lengths.

Some attributes within a data set serve valuable purposes that are important to identify. Attributes that individually or in conjunction with others define uniqueness of the data set also may support relationships between entities.  Uniqueness can be further classified as being either members of a system-enforced primary key or of a business key (outside of the defined primary key).  System-enforced primary keys are relatively easy to define within a database system through interrogation of the system catalog.  Business keys that exist in tables in addition to a primary key may be more difficult to identify, especially if more than one attribute is needed to define uniqueness.

Attribute-level information of interest includes: data type (size and precision), the number and percent of NULL values, column descriptions, number and percent of distinct values, and the minimum-maximum-average values and lengths.  Uses of the system catalog provides some of this information, but others must be collected through sampling the data.

Other types of attributes that may help in identifying relevancy are those that provide system-level auditing or change control. Knowing which attributes fill these roles may either allow you to (a) ignore them for profiling purposes or (b) use them to help explain versions or data anomalies.

Part 4 expands on attribute profiling with the introduction of relationships and patterns.

Part 1: The Psychology of Data Profiling

Swiss psychologist Carl Gustav Jung founded the Analytical School of Psychology. His word association theories form the basis of the Myers-Briggs Type Indicator Assessment test to identify career aptitude in today’s high school students. Dr. Jung’s approach assigned personality profiles based on how an individual’s thoughts associated to various phrases. By analyzing responses, he could understand how an individual viewed the world around them and perceived themselves. Typically, subjects are asked to speak the first thought entering their minds after hearing a trigger phrase. For the following example, remember, there are no wrong answers. If I say the words “Data Profiling”, what is the first thing you think of?

If you thought of food, cats, country music, CSI NY, or residential plumbing, you are either not in IT or are an IT Manager.

If your first thought was “Quality Assurance”, you align yourself with data quality professionals having anti-social thoughts of failing test cases and sadistically reporting lazy developers for buggy code. You gleefully scour test cases looking for any evidence of truncation, missing values, non-matching codes, numeric precision errors, and inconsistent abbreviation, text, and date formatting.

If “Integration” comes first in your mind, past legacy integration projects have scarred you with a disdain for source system data quality levels. You view production apps with contempt and loathe the time it takes to track down data issues caused by system integrations. You investigate upstream sources to create detailed mappings and transformation rules. Typical debugging sessions consist of validating relationships to identify orphaned data, identifying attributes that contain overloaded columns (attributes containing more than one distinct data element), or fixing format errors from implied decimals.

Some of you responded with “Value Domains” or “Data Types”, indicating you are obsessive compulsive data architects compelled to organize the world into strict and orderly fashion with some degree of normalization, though you are not considered “normal” by your peers. Your concerns lie in understanding and regulating naming conventions, relationships, existence of NULL or default values, and understanding the meaning of each data element to accurately identify business rules and when two or more objects are related or redundant.

Lastly, if “Debugging” is the first item in your thought queue, you are a coder justifying why presumably good code is not working. Extreme paranoia has taught you to assume nothing about data quality, so you add tests to identify duplicates, validate relationships, enforce business rules, track change data capture, provide substitute values. Your phobia of early morning phone calls cause you to add auditing to your code to inform a DBA of data issues rather than waking you up in the middle of the night.

It is truly amazing how much we can conclude from the response to one simple phrase.

As stated before, there are no wrong answers. Aside from the innocent jab at Managers and non-IT resources, we all realize the benefits of information quality and absolutely need business involvement to understand context and domains of business information. The meaning and actions of Data Profiling change both by role and by project phase. Through profiling, we are able to identify best sources of information, learn proper ways to categorize and store it, reactively identify quality issues, and proactively define business rules to prevent future issues.

Identifying what is important to profile, when and how profiling is done, and how to share our findings across business and project resources is key. Done properly, profile results integrate to a master metadata repository and are periodically refreshed through an automated process.

This five-part series provides a tool-agnostic approach to comprehensive data profiling, focusing on information meaning and use. The next part of the series discusses system and table-level profiling. In particular, what information is important to collect at the system and table level and how can that information be leveraged by the Enterprise to help assure quality. The third part dives into attribute-level profiling and the fourth discusses attribute patterns and relationships. The final part discusses the benefits and utility of gathering profiled information into a single repository.

To view the webinar replay and download the presentations, just register at http://www.orchestranetworks.com/webinar-master-data-governance-michelin.html.  And if you have any questions, please contact info@orchestranetworks.com.

About Orchestra Networks

Orchestra Networks provides advanced Data Governance and MDM software, with a strong focus on governance for all shared data across an organization. This model-driven approach with its dynamic user interface and data services provides the means for business and IT to finally collaborate on MDM. Founded in 2000, Orchestra Networks has presence across Europe and North America.

He started out by talking about several Oracle MDM customers, their success stories and their return on investment, across drivers like growth, efficiency, improved IT agility, and compliance.

Pascal moved on to talk about MDM implementation challenges. Oracle surveys its MDM customers every two years. Measuring actual ROI achieved is the most difficult challenge reported. Next is breaking down organizational silos, and then demonstrating incremental business value.

Five out of the top ten challenges were related to data governance and project/organization. These were big themes two years ago as well.  So Oracle worked with an outside partner on the areas of strategy, policies & processes, organization, measurement & monitoring, technology, and communication. They got a group of 10-15 customers together 2-3 times per year, and that group put together a set of requirements for a product that Oracle has now created called Data Governance Manager. This product helps data governance professionals to operate and monitor the hub and to define and enforce policies.

Pascal showed a short video from an Oracle customer, Areva. Their program was called STOCK – Strategic and Operational Customer Knowledge, to ensure the high quality of customer data. They used a five step approach: Collect, Harmonize, Merge, Enrich, and Publish. The benefits included saving employees time, ensuring that internal people can rely on customer and prospect data, and providing the entire enterprise with a clear vision of the customer database.

The second set of challenges related to ROI and business case – measuring actual ROI achieved. Oracle now has a web-based ROI model available through its sales team. Oracle also has a group of people that do a 3-5 week management consulting exercise called “Insight” that delivers a full business case.

The third set of challenges is the first one involving technical issues: #10 and #11 (integration and data quality).

Two years ago, the #1 issue was procuring skilled resources. So Oracle has been working closely with systems integrators, so now this issue is down to #7. Integration with operational applications has gone from #2 to #11.

Lastly, Pascal discussed Oracle solutions, investments and its strategy going forward. Oracle now has Customer Hub, Supplier Hub, Product Hub, and Site Hub. Data Relationship Management, which is a financial hub to manage financial entities such as the chart of accounts and other hierarchies, is also an analytical hub.

Oracle Customer Hub (formerly known as Universal Customer Master) is now on release 8.2, which shipped in January 2010, and includes the new Data Governance Manager module. This is the largest customer release in four years.

Oracle’s MDM strategy has two legs – embedded “best in class”. Oracle has OEM’d the Informatica solution, using the Identity Systems solution (now owned by Informatica) and the Address Doctor solution (also from Informatica) for postal cleansing for 200+ countries. The other leg is “open” – Oracle is providing a “Universal DQ Connector” for selected vendors like Trillium, Acxiom, D&B and Datanomic. (Note: the embedded “best in class” approach is somewhat controversial, since Informatica is now competing directly with Oracle, since it has acquired the Siperian MDM hub).

The end-to-end data quality framework (the Data Quality “Machine”) has a Rules Manager for design, development and validation (IDQ). There is a process (Analyze/Profile, Standardize/Cleanse, Match & De-Duplicate, Enrich) with a Scorecard & Reporting, and an Exception Management Process. The output is to load the MDM system with zero rejects.

Oracle has also acquired Silver Creek Systems, which is focused on product data quality. It is a self-learning semantic engine to handle the complexities of product information.

Pascal talked about some of the newer MDM hubs, Supplier Hub and Site Hub. Site Hub in particular has experienced strong interest from retailers, fast food companies and large enterprises, which are using it to manage stores and locations.

Oracle’s MDM investments are critical for Oracle in terms of its differentiation strategy, and data governance is the number one item from its customer advisory board. Oracle has reached 1,000 MDM customers across all of its various MDM products.

Pascal wrapped up by talking about how competitive the MDM space is and the recent acquisitions in the market. Oracle’s history is in applications. Oracle brings a pre-built, flexible schema with enterprise-grade, verticalized hub applications. Oracle MDM hubs are pre-integrated with both Oracle and non-Oracle applications. And Oracle provides best-in-class data quality and data governance solutions.

Great New White Paper and Other Collateral Available at Our Booth

At the event, we announced with Equifax a new product that integrates Equifax commercial information with Oracle E-Business Suite and Oracle Customer Data Hub. This product simplifies the process of integrating Equifax credit and marketing information with prospect and customer data in Oracle.  Both the joint press release and a one-page product overview were distributed at the booth.

Also available was a new whitepaper written in collaboration with Informatica titled, When Data Governance Turns Bureaucratic: How Data Governance Police Can Constrain the Value of Your Multidomain Master Data Management Initiative. This updated version of an earlier white paper written with Siperian in 2009 added both new content and industry insights. It was very well received at the Gartner conference this week.

Finally, we handed out one of the most popular recent articles from this blog, Hidden Costs of Duplicate Customer Data.

The conference drew attendees from many different market sectors, so discussions and meetings were both informative and valuable from an MDM perspective. Several Hub Designs clients were able to join us there, from the insurance, software and transportation industries, and we had four of our team members there as well. I’m going to write a separate article with my thoughts on the sessions and the mood of the conference, but I wanted to provide a look at our booth as well, for our readers who weren’t able to make it to Las Vegas this week.

The solution smoothly integrates Equifax data into Release 12 of Oracle’s enterprise resource planning (ERP) and master data management (MDM) suites.

Hub Designs Equifax Integration for Oracle provides access to vital credit and marketing data in Oracle’s MDM and ERP modules including:

• Oracle Customers Online
• Oracle Sales Online
• Oracle Receivables

Equifax commercial information helps businesses to:

• Make credit decisions, expedite collections, reduce bad debt and pre-qualify prospects;
• Reveal linkage between related companies;
• Standardize name & address information and prevent duplicates;
• Enrich prospect and customer records with marketing and credit information from Equifax;
• Increase productivity by creating new parties and party relationships in Oracle automatically

The joint press release describes the solution in more detail, and a one-page overview is available as well. If you’re interested in learning more, please contact us via our web site or drop by Booth #7 during the exhibit hall hours at the Gartner MDM Summit.

First, there’s no doubt this was a game-changing move, for both Siperian and for Informatica. With over 4,000 Informatica installed base customers to leverage, and 200 Informatica sales reps going through training and certification, Siperian’s sales momentum should increase dramatically. And in fact, several new deals have closed just since the acquisition was announced in late January.

And being acquired by Informatica eliminates the “company viability” question that some Fortune 500 IT shops would have about any software company under a certain size (not just Siperian). Informatica itself might be acquired by one of the mega-vendors at some point, but with annual revenue of $500 million, it’s big enough not to be subject to the financial viability question.

Informatica also provides a large partner ecosystem and a significant marketing budget, so living on under the Informatica banner, Siperian can compete more readily for mind share both with partners and with potential customers.

But what impressed me the most was the strategic nature of the other purchases that Informatica has made over the past couple of years, such as Identity Systems for entity resolution (i.e. matching) and Address Doctor for address cleansing. With the addition of Siperian as a strong player in the multidomain MDM hub space, Informatica has declared itself a real competitor against the likes of Oracle, IBM, Initiate Systems (an IBM company) and SAP.

And in some ways, Informatica is better positioned than most of these, for two reasons. First, it has a complete suite of leading products for data integration, data quality and all of the associated things that make up the “MDM ecosystem”. And second, many of its competitors are dependent on it for those components (Ramon Chen wrote a great article on Informatica’s OEM agreements with various competitors).

Informatica’s product lineup supports all of these MDM requirements:

• Multiple MDM architectural styles including the ability to support Registry style (competes most directly with Initiate Systems)
• Multiple data domains, i.e. multidomain MDM (competes most directly with Oracle, IBM and SAP)
• Data Integration and Data Quality as a foundation for MDM (competes with a wide variety of products)

So in some ways, Informatica wins even if customers buy a competitor’s MDM hub product, because there’s a good chance they’ll still buy Informatica’s data integration and/or data quality solutions, to help them with data integration, data profiling and data quality, or to help build the inevitable data services, once the master data is gathered in a centralized hub and able to deliver timely, trusted and relevant to the rest of the enterprise.

Informatica sees its MDM products used in both Operational MDM (where the master data is actively managed by data stewards, governed and improved and then synchronized back to the operational systems), and in Analytical MDM (where for various reasons, the improved master data does not flow back to the operational systems, but flow instead to data warehousing, analytical and business intelligence applications).

Informatica has such a strong, integrated story, with its PowerCenter data integration, Informatica Data Quality, and Informatica MDM products, that it’s able to accommodate customers’ maturity needs starting with data integration and progressing to data quality and MDM.

And Informatica, by giving customers the ability to solve any MDM-related business problem with a unified architecture, spanning data integration, data profiling, data quality, identity resolution, address validation, and all major styles of master data management, has pulled together a great set of solutions for MDM.

I’m looking forward to seeing the Informatica folks at this week’s Gartner MDM Summit conference in Las Vegas.  If you’re going to be there, stop by and see the Hub Designs team at Booth #7 during the exhibit hall hours.  We’ll be announcing a new product with Equifax, and we’ll be releasing a data governance white paper with Informatica.

This will be our third time at this event, and our second as an exhibitor. Last October, we exhibited as a Kiosk sponsor, and this year we will be there as a Silver sponsor. We’ll be in Booth 7 during the exhibit hall hours, and if you’re going to be attending and would like to meet with us, just contact us via our web site.

We’ll be announcing an exciting new product, and publishing a new version of one of our most popular white papers.

Here’s what you’ll learn about if you go:

• Multi-domain MDM
• MDM vision and strategy
• Customer data
• Product data
• Data warehousing, data quality and MDM
• Enterprise information architecture
• Enterprise information management
• SOA and MDM

And here are the key benefits of attending:

• Insight into creating a successful MDM program
• Persuading the business to take a leadership role
• Delivering measurable ROI by linking your MDM to business metrics
• Reducing costs and increasing efficiency by removing duplication and creating consistency
• Improving customer acquisition and cross- selling/upselling activities
• Complying with regulations and leveraging your master data to manage risk
• Consolidating and leveraging data faster following mergers & acquisitions
• Accelerating your new product introductions
• Managing your supply chain more efficiently

It’s not too late to register at the special rate of $1,795 – a $300 savings on the standard rate of $2,095! Go to gartner.com/us/mdm or call 1-866-405-2511 and mention priority code: MDMHUB.

We’d love to see you in Las Vegas! These events are like “old home week” – getting to catch up with people we haven’t seen in a while and find out what everyone in the MDM space is up to. So come along for the ride, catch a few sessions, maybe hit the tables a bit, and head home with a little less cash in your pockets but a little more knowledge in your head.  And if you need help convincing the “powers that be” to let you go to the conference, Gartner has very thoughtfully put together an Attendee Justification Kit to help you convince them.

Dear Dan,

I am a database architect but I am new to MDM and data governance, and I’m very interested in this area.

Can you please suggest where to start? I’ve found some information on the web (sometimes a bit disconnected), but I seem to be lost with so much information. Also, the tools that are currently available in the market – do they address all the challenges in this space?

One question I have is: if data quality is given the importance it deserves from the beginning of any project (operational or data warehouse), are MDM initiatives necessary? Are MDM projects needed because of the proliferation of applications that are developed in silos and that don’t consider what information is already available to the enterprise? In essence, should MDM be part of any project?

Thanks for your time.

Not to sound too self-serving, but I’d start with this blog and the MDM Community.

As for your question about whether MDM initiatives are necessary if data quality is given sufficient importance, please realize that MDM is a relatively new discipline which includes embedded data quality; it does not replace data quality.

What MDM does is sit between the source systems (typically CRM and ERP) and the data warehouse and business intelligence. So instead of trying to flow master data and transactional data directly into the warehouse for analysis, we bring it into the MDM system first, where it can be “mastered” – which includes fixing data quality issues. We then flow those corrections back into the source systems and downstream into the analytical systems. Which of course you can’t do without data quality tools. But data quality tools by themselves are not sufficient, because they typically don’t persist or store the data.

Your next question, are MDM projects necessary because of the proliferation of apps developed as silos – yes, that’s a big part of it. Essentially, if you developed a new architecture from scratch, you’d put a multi-domain MDM hub, able to handle many types of master data at the core, and you’d build data quality into it, then you’d surround it with integration so you can flow data from there to where ever it’s needed. So clean, accurate, consistent and timely master data would be available to any other IT project that was going on, but it would only have to be built once. “Build once, use many”, as they say.

Please keep reading and I hope you stay interested in MDM and data governance!

I got an answer from this person today that I thought I’d share with you here:

Dan,

Thank you very much for your insights. Whatever documents I had read about MDM had more to do with the people, process or technology, but didn’t cover the essence of MDM. I’ve gone through some of your blogs and I’m beginning to understand MDM.

Now AMB is launching an open source version of its Information Governance Suite, called the Community Edition. AMB delivers tools to facilitate real time governance, and is now extending its reach as one of the first major vendors with an open source version of its core product.

This might be an ideal way for companies looking to familiarize themselves with a data profiling and data quality product to learn the tool, get a data governance proof of concept up and running in a cost effective way, and then demonstrate value to the business.

The Community Edition allows you to:

• become familiar with the concept of data profiling as a way of identifying and fixing information anomalies
• enable enterprises embarking on a data stewardship program to use the Community Edition to spotlight, identify and determine the priority of their internal information issues
• enable organizations to define and automate a repeatable process, using software to administer the information governance program that aligns with the repeatable process, not the other way around

The Community Edition should provide a core set of data profiling and governance, and training and support is available, as are upgrades to the Professional and Enterprise Editions.

For more information, contact AMB at 1-847-899-5154 or community@ambpdm.com, or visit http://www.ambpdm.com.

DISCLAIMER: While the facts that I’ve included here are true, I’m speculating on the reasons why they’re taking place. I have no affiliation with any company mentioned here, nor should my opinions be construed as knowledge of their actions.

If you, like me, have followed MDM for the past year or two, you knew that what has been happening recently was going to happen sooner or later. Whether it was due to choice or necessity, MDM has been in the IT press a lot lately. Oracle acquired Silver Creek to enrich its product information management offering. Talend has announced and started to promote its open source MDM application. Data integration provider Informatica acquired Siperian recently in order to enter the MDM space and IBM recently acquired Initiate Systems as well.

Each of these events leads to one key question – how will this impact MDM in the short term and in the future? Given my understanding of the space, I think three scenarios are likely:

Scenario 1

It is hard to ignore the movements that IBM and Oracle have been making in the past year or so. In a market economy, the goal is to have as much market share as possible. In order to do this, you either build new products or acquire existing companies that have the technologies that you want.

While each company has done a combination of both building and buying solutions, their strategic plans are hardly a secret. IBM has proposed a vision of an end-to-end data management platform, which includes their MDM offering as well as reporting tools like Cognos and analytics/statistics from SPSS. Now that IBM has acquired Initiate Systems to complement their MDM stack, the question is why. Do they want to be known as a serious player in the health care industry? There could be other reasons too – they may consider MDM just a small piece of their data management toolkit and this could solidify that position, moving MDM from one of the hottest ‘technologies’ out there to just a “means to an end” to increase market share for their software business unit. Regardless of the reason, it means one less major MDM player in the market.

Then we have Oracle. For as long as I can remember, Oracle has been promoting its Fusion strategy. For those of you who are not familiar with it, Fusion is Oracle’s attempt to provide one code base that would pull together the applications it has built and purchased. This momentous undertaking was finally demonstrated at last year’s Oracle Open World (while Oracle continued to acquire other companies such as Silver Creek Systems).

However, like IBM, one can speculate on where MDM fits in this Fusion strategy. Oracle has always promoted its database first and sold its applications second. Even with the numerous special purpose hubs they’ve been developing lately, could this finally be the technology that enables Oracle to transcend from being a database vendor to a true platform player. Only time will tell with this one.

Scenario 2

There’s always the possibility that MDM has been considered the “secret sauce” – the so-called missing link – that rounds out the product lines for data integration/migration vendors.

Talend’s acquisition of French software company Amalto provided them a way to enter the MDM space. The open source vendor has been a darling of the analysts for a number of years and even won an award by Gartner, one of the first (if not the first) they offered such a company. However, since I don’t have contacts within Talend, it’s not clear what their next step will be, since they seem to be focusing their energies mostly in MDM after hiring two people to drive that effort within the past 6 months or so.

As the de facto leader in data integration, Informatica needed to extend its reach beyond that space. If you look at their job listings, they are looking for someone to market their CEP (Complex Event Processing) efforts. Relatively recently, they were looking to hire someone who had experience with ERP or MDM, but it is unclear which path they decided to take with that. Regardless, there were always looming rumors of them wanting to add MDM to their portfolio with the press suggesting that they would acquire Initiate Systems. However, instead of buying them, they purchased Siperian – a company half its size in terms of customer base and revenue.

In either of these cases, MDM may not be their flagship product, but at least it shows that it is a viable technology and shows that it is something that won’t be going away any time soon.

Scenario 3

People like me who have been in the data management space are always interested in improving something. I believe in the statement, “even if something isn’t broken, there’s always a reason to make it better.” This was clear when Customer Data Integration (CDI) first came about and many companies hopped on that bandwagon, knowing that they wanted a way to track their customers more efficiently.

At the same time, other companies explored Product Information Management (PIM), a way to have a single source of product information which was sourced from PLM, inventory and supply chain systems. Following that was the concept of MDM – a beautiful vision – having a single source of truth that can be used by an entire company.

Now we have a new concept that has been promoted – Multi-domain MDM. Siperian and other companies have began to promote this to show the world that they are truly the most advanced players out there. While this has been going on, there have been rumblings about Enterprise Information Management (EIM). What I’m still not clear on is – what’s the difference between multi-domain MDM and EIM? Are they the same? If not, what are the differences between the two concepts?

In any case, there’s a lot to think about. I don’t know where you stand, but one thing is certain – MDM is not going away, at least for the foreseeable future.

Part 5: The Profiling Payoff

This is the final part of a five-part series, describing how data profiling benefits both IT projects and business operations.  In Part One, we discussed profiling perspectives.  In Parts Two, Three and Four, we introduced the value of system, entity, and attribute-level metrics.  This part discusses the archival and beneficial uses of profile results.

If you have defined your corporate data profiling strategy similar to the methods discussed in the preceding parts of this series, you’ll have amassed a robust collection of metadata spanning relevant systems across your business.  Although systems may be of different types and locations, the structured approach and common metrics you collected create a centralized repository of information that can be examined holistically. Ideally, this information will exist in an open-source database repository with reports made available across the enterprise. System and Entity information help planners and developers organize information strategies. Attribute-level domains, constraints, and business rules help data architects understand existing systems. Relationships and value patterns are readily available to support validation of information-related hypotheses as needed.

If you plan to design your own repository, consider adding timestamps and indicators to help you manage and present the information.  To keep your repository relevant to business needs, design collection rules to be configurable. This allows you to easily ignore superfluous information or enable tests only at certain critical times. Allow initial system profiling efforts to gather a large set of metrics and store them as your baseline.  As you learn about the information, you will see which tests or which data objects add no value.  Us geeky DBA-types who understand system-level catalogs have our own scripts to do much of what was described inParts Two,Three and Four. Those less-inclined may prefer to use a third-party tool for profiling. Either way works as long as the business needs are satisfied and the entire enterprise standardizes on one approach (and thus one integrated repository).

You will find that collecting and maintaining this level of detail has a definite cost.  Even if the collection is automated, interrogations of large data sets places an overhead on production systems that may not be practical. Record and monitor profile execution metrics to identify bottlenecks or tuning opportunities. Realize that the extent of data profiling is contingent on the project phase, specific data elements, and most of all, business value. Review profiling goals on a regular basis and eliminate unnecessary and redundant checks.

How much profile history to maintain is another consideration.  Even though disk is “relatively” cheap, maintaining all historical entries in a live repository may not be necessary. Consider business needs and value for historical profile information. Even consider archiving at a summarized (or less frequent) level and keep only a limited time window of statistics online.

This discussion on data profiling was intended to broaden perceptions of what it means to a business and the value it can bring if done in a sustainable way. The blog format is not conducive to in-depth discussions, but hopefully the topics covered here spur some thoughts into how you can add value to your business by implementing some of these concepts.  Use your imagination, but remember that no matter how cool it might be to collect and store some profile output, if it does not add business value to somebody, it might not be worth the overhead to continue recording it.

Go back to Part 4.

This was widely rumored last week, but the announcement of Informatica’s acquisition of Siperian took my mind off this temporarily.

On the face of it, it makes all the sense in the world. IBM knows a good product when it sees it, and Initiate has been doing well in the MDM world lately, particularly in the healthcare vertical, where it grew up, and in the public sector vertical. IBM’s press release explicitly mentions Initiate as a leader in “data integrity software for information sharing” among healthcare and government organizations. I thought it was interesting that the IBM release didn’t mention the terms “master data management” or “MDM” even once.

I was a little surprised that IBM’s release made no mention of the financial terms, since IBM is a public company, but I’m sure it will only be a matter of time before those details become available to those who know where to look or whom to ask.

It wasn’t a surprise to see the IBM release mention the stimulus funding being invested around the globe. When I first saw the rumors last week, I immediately thought – IBM is buying Initiate to be better prepared for the various e-Healthcare initiatives that are coming down the pike.

Where things may get a bit tricky is explaining the multiple MDM platforms from IBM to potential customers, and managing several different development roadmaps and product portfolios. There’s the IBM InfoSphere MDM Server (the former DWL product) and there’s also IBM InfoSphere MDM Server for Product Information Management (the former Trigo product). And now there’s the Initiate product too.

While the acquisition does make sense, there is an “embarrassment of riches” factor. IBM will, of course, develop a sales playbook explaining what situations at what type of customer are a good fit for each product.

I think the lingering feeling I have with Initiate Systems is that it may be headed for a “golden ghetto” at IBM – never to reach its full potential as a solution across many different industries, and eventually to handle many different domains of master data. IBM may (and rightly so, in its mind) pigeonhole it into the healthcare and government verticals.

But Initiate’s had some good success outside those two industries. In the Financial Services vertical, they’ve got customers like Capital One Financial, Countrywide Financial (now Bank of America), eSure Insurance, and Wells Fargo. In the Hospitality industry, they’ve got Choice Hotels. In manufacturing, they’ve got Mitsubishi Motors Australia. In the Logistics vertical, they’ve got Federal Express. In the retail sector, Barnes & Noble, CVS, Longs Drug Stores and SuperValu are all customers. And in the high tech space, they’ve got Dell, Ingenix, Intuit, LocatePLUS, Microsoft and National Instruments.

Unfortunately, they didn’t achieve enough critical mass in any of these other verticals to compete with the strong momentum they’d developed in healthcare and government.

As I said last week, these are interesting times in the MDM world. The recent M&A activity, the healthy demand from large and medium sized corporations, the large number of consultants from other areas claiming to now have experience in MDM – these are all signals to me of a large and fast-growing market. So the New Year, for those of us in the MDM space, is off to a good start.

The two firms were already working together closely, having an alliance and OEM relationship through Informatica’s acquisitions in 2008 of Identity Systems (for entity resolution and matching) and in 2009 of Address Doctor (for customer address cleansing).

This will strengthen the Siperian product further by bringing Informatica’s technology even more tightly into the Siperian MDM Hub.

At the same time, it eliminates the “company viability” question mark that sometimes gets raised in large IT shops’ minds when evaluating enterprise software vendors. When a Fortune 500 company is evaluating a smaller company, they sometimes wonder how long a company like Siperian can last against companies like IBM, Oracle and SAP. I’ve never been a big fan of that argument, since some of the best software gets created at small and medium-sized companies, but there’s no doubt it’s a obstacle to be overcome with the larger enterprises. Now, it shouldn’t be an issue.

As a Siperian partner, Hub Designs is excited about this acquisition. Based on the information we’ve got at this point, it seems like a good thing for Siperian’s customers, products, shareholders, partners and people. In today’s economic climate, dreams of a big IPO (for any venture-backed technology company) are unlikely, so an acquisition by a well-run larger company is a good outcome.

I know a lot of the people at Siperian personally, and have worked closely with them over the last few years. I hope the people at Informatica realize what a strong team they are getting in this acquisition, and do everything they can to hang onto them all.

I do suggest they stop using the term “MDM Infrastructure” though (which appeared 5 times in Informatica’s press release announcing the acquisition). First, it’s not accurate – MDM projects typically need to be drive by the business to be successful, so they can’t and shouldn’t be thought of as “IT Infrastructure” projects. Secondly, from a marketing perspective, “infrastructure” is about as exciting as mud – it’s hard to get senior management excited about spending money on something with the word “infrastructure” in the name.

As for the acquisition’s impact on the rest of the MDM market, it’s still growing pretty quickly, but the number of players is shrinking. So I think we’ll see it become even more competitive, and with Informatica now becoming a strong player in the MDM hub market, that’s got to cool its relationship with Oracle, who selected Informatica as an OEM component of its Oracle Fusion MDM hub.

IBM is rumored to be acquiring Initiate Systems, which is an interesting play in its own right, especially given the expected growth in spending in the e-healthcare space over the next few years.

And SAP continues to improve its products slowly but steadily, while D&B/Purisma is doing some interesting things with web services access to the D&B central database of information on businesses.

As for the remaining independent MDM vendors, like Orchestra Networks and Kalido, or Product Information Management (PIM) solutions like Stibo and Riversand, they should see this as further validation of the strength of the MDM market. Kalido feels that it’s the only independent MDM provider who can manage every master data domain. That may be true.  I plan on learning more about Kalido over the next few months.

So like the Chinese curse, “may you live in interesting times”, the beginning of 2010 promises to be interesting for all of us in the MDM business!

If you’d like to continue the discussion on the “Impact of Informatica’s Acquisition of Siperian”, click http://ning.it/aJ1Xj5.

Part 4: Profiling Relationships and Patterns

This is part four of a five-part series describing how data profiling assists in all aspects of system development, from design through deployment.

Part One introduced different perspectives on data profiling. Part Two identified valuable system and entity metrics to track. Part Three discussed attributes. In this segment, we dive deeper into attribute relationships and pattern recognition. Also, we expand on primary key identification discussion and discuss hidden relationships.

Pattern grouping provides a mask of distinct format patterns within an attribute data set and a count of the number of occurrences. Patterns give insight into the type of values found in an attribute. For example, a numeric pattern analysis may show values such as 999.99999, 99, or -.9999.

Observing distinct patterns gives insight into the maximum digits and precision, and also domains such as integer or real. Pattern of a database date or date-time type provides unremarkably similar patterns for all dates. Because the database management system typically enforces the domain, date analysis provides no value and can be ignored. If dates are stored in character format, however, patterns quickly show variations in date formatting. Character patterns only have significance to a limited number of positions. It makes no sense to pattern a description field of 200 or 2000 characters. Smaller code attributes of less than 10 characters though do provide value. Ignore pattern profiling for character strings over 20 characters at first, then refine to shorter character strings if the results do not add value.

In pure database theory, referential integrity (RI) is your friend. In practice, designers and software vendors often forgo RI to improve system performance on data inserts. These designers place the data quality burden on the application and do not endorse external data manipulation outside the application interfaces. In the real world, though, data corruption occurs and without RI or routine data quality checks, corruptions may not be found for a long time or not at all. Personally, I have identified over $50,000 of recent orphaned sales in a retail client resulting from deliberately disabled RI. These unreported sales were not added to the ledger and were allowed to occur for performance reasons until I found them through simple profiling. Enforcement of RI is a topic for another discussion but is mentioned here because it does identify a valid reason for data profiling.

In even presumably good relational designs, some parent-child relationships are not enforced for different reasons. First, interrogate the RI listed in the system catalogs to identify all enforced relationships. Reverse-engineering a system with a good modeling tool is probably the best way to do this. A harder and more valuable analysis is to identify unenforced relationships and determining the probability of the relationship if not all values are an exact match. Do this by counting all the candidate child attribute values that exist within a known parent attribute table. If all match and there are a non-trivial number of matches, there is a good probability of a non-identified relationship. A small number of mismatches could identify data quality issues.

In Part 5, we tie all the techniques discussed in the first four parts together to show the value of a repeatable data profiling process.

Continue with Part 5 or go back to Part 3.

Part 3: Attribute-Level Analyses

This is part three of a five-part series on data profiling.

In Part One, we took a light-hearted view of where profiling benefits an organization and in Part Two, we discussed the fundamentals of a profiling strategy.  The remaining three parts discuss attributes, relationships, patterns, and how to use the combined data profiling information you collect.  In this section, we introduce attributes, the lowest-level components of a profiling effort.

An attribute is simply a individual data element.  Alone, an attribute has no context.  Given the simple descriptor of “Cost” for an attribute tells us very little about the attribute’s true purpose and immediately drives a need for additional information, such as units (hours, Dollars, Euros…), type (weighted, unit, gross…), and use (invoice, sum, average…).  Attributes therefore must be analyzed within the context of their business purpose to have meaning.

Some characteristics require business knowledge to define and others can be determined through interrogation of existing values and underlying rules of the storage medium. It takes both analyses to get a complete picture of information within a system. While assembling this puzzle, though, keep in mind that until you validate the enforcement of business rules, only assumptions can result from physical profiling or business context.

Analyses of values, domains, and constraints allows insight into use (or abuse) of an attribute. The larger the sample size, the better confidence you gain in the results. Without explicit proof of business rule enforcement, though, you must assume that just because a value does not presently exist does not mean it cannot exist. Business rules are defined by business experts and enforced through database constraints, data type/precision, and application code. Knowing the methods of enforcement allow you to narrow a domain but not totally understand it. Profiling of actual values provides additional refinement in terms of percentage of NULL values, percentage of distinct values, minimum, maximum, and average values, top x and bottom x recurring values along with their counts, and minimum, maximum, and average data lengths.

Some attributes within a data set serve valuable purposes that are important to identify. Attributes that individually or in conjunction with others define uniqueness of the data set also may support relationships between entities.  Uniqueness can be further classified as being either members of a system-enforced primary key or of a business key (outside of the defined primary key).  System-enforced primary keys are relatively easy to define within a database system through interrogation of the system catalog.  Business keys that exist in tables in addition to a primary key may be more difficult to identify, especially if more than one attribute is needed to define uniqueness.

Attribute-level information of interest includes: data type (size and precision), the number and percent of NULL values, column descriptions, number and percent of distinct values, and the minimum-maximum-average values and lengths.  Uses of the system catalog provides some of this information, but others must be collected through sampling the data.

Other types of attributes that may help in identifying relevancy are those that provide system-level auditing or change control. Knowing which attributes fill these roles may either allow you to (a) ignore them for profiling purposes or (b) use them to help explain versions or data anomalies.

Part 4 expands on attribute profiling with the introduction of relationships and patterns.

Continue with Part 4 or go back to Part 2.

Part 1: The Psychology of Data Profiling

Swiss psychologist Carl Gustav Jung founded the Analytical School of Psychology. His word association theories form the basis of the Myers-Briggs Type Indicator Assessment test to identify career aptitude in today’s high school students. Dr. Jung’s approach assigned personality profiles based on how an individual’s thoughts associated to various phrases. By analyzing responses, he could understand how an individual viewed the world around them and perceived themselves. Typically, subjects are asked to speak the first thought entering their minds after hearing a trigger phrase. For the following example, remember, there are no wrong answers. If I say the words “Data Profiling”, what is the first thing you think of?

If you thought of food, cats, country music, CSI NY, or residential plumbing, you are either not in IT or are an IT Manager.

If your first thought was “Quality Assurance”, you align yourself with data quality professionals having anti-social thoughts of failing test cases and sadistically reporting lazy developers for buggy code. You gleefully scour test cases looking for any evidence of truncation, missing values, non-matching codes, numeric precision errors, and inconsistent abbreviation, text, and date formatting.

If “Integration” comes first in your mind, past legacy integration projects have scarred you with a disdain for source system data quality levels. You view production apps with contempt and loathe the time it takes to track down data issues caused by system integrations. You investigate upstream sources to create detailed mappings and transformation rules. Typical debugging sessions consist of validating relationships to identify orphaned data, identifying attributes that contain overloaded columns (attributes containing more than one distinct data element), or fixing format errors from implied decimals.

Some of you responded with “Value Domains” or “Data Types”, indicating you are obsessive compulsive data architects compelled to organize the world into strict and orderly fashion with some degree of normalization, though you are not considered “normal” by your peers. Your concerns lie in understanding and regulating naming conventions, relationships, existence of NULL or default values, and understanding the meaning of each data element to accurately identify business rules and when two or more objects are related or redundant.

Lastly, if “Debugging” is the first item in your thought queue, you are a coder justifying why presumably good code is not working. Extreme paranoia has taught you to assume nothing about data quality, so you add tests to identify duplicates, validate relationships, enforce business rules, track change data capture, provide substitute values. Your phobia of early morning phone calls cause you to add auditing to your code to inform a DBA of data issues rather than waking you up in the middle of the night.

It is truly amazing how much we can conclude from the response to one simple phrase.

As stated before, there are no wrong answers. Aside from the innocent jab at Managers and non-IT resources, we all realize the benefits of information quality and absolutely need business involvement to understand context and domains of business information. The meaning and actions of Data Profiling change both by role and by project phase. Through profiling, we are able to identify best sources of information, learn proper ways to categorize and store it, reactively identify quality issues, and proactively define business rules to prevent future issues.

Identifying what is important to profile, when and how profiling is done, and how to share our findings across business and project resources is key. Done properly, profile results integrate to a master metadata repository and are periodically refreshed through an automated process.

This five-part series provides a tool-agnostic approach to comprehensive data profiling, focusing on information meaning and use. The next part of the series discusses system and table-level profiling. In particular, what information is important to collect at the system and table level and how can that information be leveraged by the Enterprise to help assure quality. The third part dives into attribute-level profiling and the fourth discusses attribute patterns and relationships. The final part discusses the benefits and utility of gathering profiled information into a single repository.

Continue with Part 2.

“It was the best of times; it was the worst of times…” So Dickens begins “A Tale of Two Cities”, but it’s also a good description of the past year.

The first half of the year was one of the most challenging I’ve faced in my twenty-three year career in business and technology. The second half of 2009 was better – not without its speed bumps but every month was a little better than the one before it.

The macro-economic climate has been tumultuous at best. But the second half of the year showed enough improvement that Hub Designs’ revenue for the year was up 33%. Not bad for a two and a half year old company during the worst economic conditions in 80 years …

Marketing and Thought Leadership

We launched a new web site in January, and it’s been well received. Total visits to www.hubdesigns.com were up 14% over 2008.

A little later in the year, we updated the “look and feel” of the Hub Designs Blog, branding it as the “world’s fastest growing blog covering master data management and data governance”. We’ve gotten more than 43,000 hits since we started writing in July 2007, and our readership more than doubled in 2009, to about 27,000 hits per year.

We published six issues of our “Best Practices in Master Data Management” newsletter this year. We publish the newsletter about six times a year to roughly 3,300 subscribers.

I wrote six articles for Information Management magazine, including some popular ones on “Product Information Management Challenges”, how to build a business case for master data management, and how to select the right MDM vendor for your organization. I also wrote for Identity Resolution Daily, on “The Growing Role of Identity Resolution in MDM” and “Matching – MDM’s Secret Sauce”.

With our partner Siperian, we wrote a white paper in August called “When Data Governance Turns Bureaucratic: How Data Governance Police Can Constrain the Value of Your MDM Initiative” that has generated quite a bit of discussion. You can download a copy of it here.

A second white paper, called “Best Practices for Leveraging D&B in Oracle E-Business Suite”, was written in partnership with Dun & Bradstreet. It describes using D&B information to drive better supply chain performance for companies using Oracle E-Business Suite. You can download it here.

I volunteer for the Education Committee of the Oracle Applications Users Group (OAUG). A big part of that effort lies in programming the MDM track for the annual conference. This year, it was in Orlando in May, and I really enjoyed speaking there and seeing people from the Oracle community that I don’t see very often. Here’s a link to my OAUG presentation.

We participated in conference calls with Oracle Development during the year, and ultimately attended the Oracle Fusion “Hands-On Validation & Testing” session for Customer MDM at Oracle headquarters in August. It was a great chance to get some early insights into Oracle’s next major product release and to see the progress Oracle has made in building out its Fusion MDM vision, which is striking in its powerful hub technology and its elegant & productive user interface.

In 2008, we attended the Gartner MDM Summit to decide whether to exhibit there in 2009.  We were impressed enough with the conference that we did exhibit in 2009, in October in Los Angeles. We had a positive experience, so we’ll be a Silver level sponsor in April 2010 in Las Vegas. Since we specialize in MDM and data governance, we find the association with Gartner’s MDM event a powerful one.

I didn’t attend Oracle OpenWorld for the past couple of years, but this year I was glad I did. It was like “old home week”, seeing people from Oracle itself and from the broader Oracle community that I’ve met over the past 15 years. David Butler, Senior Director of MDM Marketing at Oracle, posted my presentation on Oracle’s web site, and said “you were our cleanup hitter and you hit a home run with the bases loaded”.

We also did webinars with our partners Siperian and Initiate Systems. The Siperian webinar covered the differences between MDM platforms like Siperian and ERP platforms like SAP from a master data perspective. The Initiate webinar, with Initiate’s CTO Marty Moseley, discussed developing strong MDM business case, deploying core MDM technologies and lessons learned on the “build vs. buy” question.

Social Networking

After experimenting with social networking in 2008, this year we had a coordinated strategy to use the Hub Designs Blog, Facebook, LinkedIn and Twitter to communicate & collaborate with our clients, potential clients, team members, partners, suppliers, etc.

It’s a pretty simple strategy. Short updates (140 characters or less) go out on Twitter, and are re-published on both LinkedIn and Facebook. Longer updates (i.e. blog articles) are published on the Hub Designs Blog.  We encourage responses and feedback using @replies on Twitter and comments on LinkedIn and Facebook, as well as longer-form comments on the blog. And we get them – almost every blog article gets at least one comment, sometimes as many as a dozen.

When a new blog article comes out, we notify everyone via a single update on Twitter. What’s amazing is that during 2009, social networking now drives about 15% of the Hub Designs Blog’s total traffic. And one of our clients gave us some good feedback that our social networking activities help her stay current on what we’re up to, and help her feel connected to us as a company.

Another social networking experiment that developed further in 2009 was the MDM Community.  We started this using Ning (a “social network in a box”) in November 2008, out of frustration with LinkedIn’s “Group” functionality.  It now has more than 210 members, from 23 different countries. It’s still a work in progress, but if you’re interested in master data management or data governance, you should check it out at http://mdmcommunity.ning.com. It’s becoming an international “who’s who” of the MDM world.

Summary of Client Projects

In case you think the Hub Designs team has been doing nothing but marketing, writing white papers and magazine articles, speaking at conferences, and volunteering for user groups, here’s a summary of our 2009 client projects:

1. Technology provider for vehicle dealers: integration of Oracle E-Business Suite with D&B data
2. Payroll services company: integration of Oracle E-Business Suite with external credit information
3. Information services company: technical support for customers using Oracle E-Business Suite
4. Legal information services company: readiness assessment and product MDM strategy & design
5. Simulation and engineering software company: advisor to data governance board
6. Manufacturer of oil and gas equipment: integration of Oracle E-Business Suite R12 with D&B
7. Software company: built connector between Oracle AR and D&B’s DNBi risk management solution
8. Technology company: customer MDM strategy workshop

Out With The Old, In With The New

This past year has been a lot of fun, but it has also been somewhat exhausting. So I’m looking forward to a bit more deliberate pace in 2010.

We’re very excited about the coming year at Hub Designs. We’ve got some great projects underway and in the pipeline, and we’ll be continuing to grow and expand to meet our clients’ needs and market demands.

In closing, I’d like to say how grateful I am to my family, for their patience with my traveling so much and for their unconditional love.

My initial answer was that, among companies that don’t have any formal master data management, data governance or data quality initiatives in place, duplication rates of 10%-30% (or more) are not uncommon.

When I was at D&B, we used to routinely see that level of duplication in client’s customer files.

In a study in the healthcare field, Children’s Medical Center Dallas engaged an outside firm to help clean up their duplicate data:

“Solving both the current and future problems around duplicate records helped Children’s improve the quality of patient care and increase physician acceptance of the new EHR. The duplicate record rate was initially reduced from 22.0% to 0.2% and five years later it remains an exceptionally low 0.14%. The 5 FTEs initially tasked with resolving duplicate records have been reduced to less than 1 FTE.”

“For the Children’s Medical Center, the results were heartening, not only from a care delivery standpoint but also because of the significant cost-savings that can be realized. A study conducted on Children’s data showed that on average, a duplicate medical record costs the organization more than $96.”

So it is possible to get the duplication rate down to really low levels through careful analysis and the application of the right tools, as part of an ongoing data governance program. Even the hospital above (and hospitals are usually not mentioned as practitioners of best practices) was able to maintain a duplication rate of only 0.14% after 5 years.

And there are very real costs to not de-duplicating your customer data.  Depending on the functional area (marketing, sales, finance, customer service, etc.) and the business activities you undertake, high levels of duplicate customer data can:

• annoy customers or undermine their confidence in your company,
• increase mailing costs,
• cause hundreds of hours of manual reconciliation of data,
• increase resistance to implementation of new systems,
• result in multiple sales people, sales teams or collectors calling on the same customer,
• etc.

The best studies I’ve seen of the cost of duplicate data have been in the healthcare industry. One study I saw said:

“According to Just Associates, the direct cost of leaving duplicates in an Master Patient Index database is anywhere from $20 per duplicate to several hundred dollars. The lower cost reflects the organization’s labor and supply costs to identify and fix the record while the higher expense reflects the costs of repeated diagnostic tests done on a patient whose previous medical records could not be located.

The American Health Information Management Association (AHIMA) estimates that it costs between $10 and $20 per pair of duplicates to reconcile the records. If the records aren’t reconciled, however, the costs are even higher.”

Here are three more case studies backing up the range I quoted of 10%-30%:

• Once the analysis was complete, Sentara discovered they had a significant duplication rate, over 18%. They had attempted to address the duplication rate in the past through a remediation process, but due to either technology issues or because the cost of merging and cleaning up the duplicates across their many different systems was too high, they had not yet successfully reduced their duplication rate. Source: Initiate Systems success story

• Emerson Process Management faced a tremendous challenge four years ago in getting its CRM data in order: There were potentially 400 different master records for each customer, based on different locations or different functions associated with the client. “You have to begin to think about a customer as an organization you do business with that has a set of addresses tied to it,” says Nancy Rybeck, the data warehouse architect at Emerson who took charge of the cleanup. Working with Group 1, Rybeck analyzed the customer records for similarities and connections using everything from postal standards to D&B data, and managed to eliminate the 75 percent site-duplication rate the company suffered in its data. “That’s going to ripple through everything,” she says. Source: DestinationCRM.com

• Problem: Number of duplicate records: 20.9% of Utah Statewide Immunization Information System records. Impact of Problem: Difficult to find patients in system—key barrier to provider participation, risk of over-immunization—unable to find reliable patient record, cost of unnecessary immunizations, risk of adverse effects on patients. Source: health.utah.gov.

And here’s a good quote from a white paper titled “Data Quality and the Bottom Line” by The Data Warehousing Institute:

“Peter Harvey, CEO of Intellidyn, a marketing analytics firm, says that when his firm audits recently ‘cleaned’ customer files from clients, it finds that 5 percent of the file contains duplicate records. The duplication rate for untouched customer files can be 20 percent or more.”

Every organization will need its own metrics, but left unchecked, the duplication problem is a hidden cost that drags at your company, slowing down your processes and making your analyses less reliable.

If your sales analysis reports can’t be sure that there’s one and only one record for each of your largest customers, then the sales figures for those customers are probably not right. So the entire report becomes suspect at that point.

I’d like to end with a great quote on data quality by Ken Orr from the Cutter Consortium in “The Good, The Bad, and The Data Quality”:

“Ultimately, poor data quality is like dirt on the windshield. You may be able to drive for a long time with slowly degrading vision, but at some point, you either have to stop and clear the windshield or risk everything.”

Please let us know what you think by commenting here.  We’re interested in hearing your thoughts on data quality and the issue of customer data duplication.

Just to make sure, I asked Steve explicitly during the Q&A section of the program if it was okay under the non-disclosure agreement we had all signed to write about Fusion on my blog, and he said “Yes.”

Hub Designs was invited back in February to help Oracle’s Fusion MDM team with some design review, validation, and testing activities. In return for our assistance, we’ve gotten to see Fusion MDM inside and out, and we can proudly say that we are one of the very few trusted partners who helped Oracle to design and develop the application.

We participated in a lot of conference calls with Haidong Song, Oracle’s Product Strategy Director for Customer MDM, and other members of his team. And we attended a week-long “hands-on validation” event at Oracle headquarters in August, looking specifically at the customer data management aspects of the Fusion MDM hub.

My first impressions of Fusion MDM during that hands-on session were very favorable. I remember thinking to myself, “Oracle could almost start selling this into the MDM hub market right now!”

Of course, Fusion isn’t scheduled to ship until sometime in 2010, and there’s still plenty of work to be done between now and then. But the core functionality needed for master data management was there, and the Oracle Fusion MDM team had a room full of customers and partners banging on it for a week without any significant crashes or issues.

There was plenty to like in Fusion that didn’t relate specifically to master data management – the new and improved user interface, the embedded analytics, the modern, standards-based architecture, the usability research that Oracle has done, the improved business processes, the built-in collaboration capabilities …

But the fundamentals of MDM were strong as well. Haidong and his team demonstrated how to import and consolidate customer data from outside sources, and we did our first hands-on lab session bringing in a small customer data load from a desktop file, such as a list of trade show leads.

We also tested a larger volume of customer data being brought into Fusion MDM through the Bulk Import process.

We did another exercise simulating how a typical customer data steward would identify potential duplicate customers, and then resolve those duplicates by merging the duplicate parties.

We also got a good look at the Informatica components that Oracle is bundling into Fusion on an OEM basis: the former Identity Systems matching engine and the former Address Doctor address cleansing tool. Previous Oracle MDM products like Customer Data Hub have had loose integration with Trillium and Firstlogic for address cleansing, but it’s refreshing to see Oracle investing in deep integration with industry leading solutions.

I think there are going to be a lot of Oracle customers who will move to Fusion MDM as the first wave of their overall migration to Fusion, who will see Fusion MDM as a good way to get some early experience with the Fusion applications family, before committing their mission critical Enterprise Resource Planning (ERP) applications to the Fusion platform.

And in 2010 and beyond, I think will be a lot of potential customers who evaluate Fusion MDM positively on its own merits against competitive MDM hubs. Oracle brings a robust data model, open architecture, and a next-generation approach to master data management, with state-of-the-art matching, data quality, middleware, and business process management.

Please let me know by commenting here what your thoughts and expectations are for Oracle’s Fusion MDM hub.