Unifying Data from Multiple 3rd Parties Into One Business Application

Eventually the burden of having your data across multiple 3rd party CRMs and apps as well as your own custom software starts to cause a problem with cohesive visibility across your business. 

The common question is how do you cross-match the data to gain greater visibility? The answer is by creating your own holistic data model.

1. Connect to all data sources
2. Synchronize data from all sources
3. Create links between data silos
4. Build an abstracted data model
5. Create insights based on your abstractions
6. Implement AI

Advantages of 3rd Party Systems

It is important to remember that there are very valid reasons for having multiple 3rd party systems powering your business. Instead of re-inventing the wheel, you can pick the industry leading provider in each area to give you the biggest competitive advantage. 

Paying a subscription fee will almost always be incomparably cheaper than bringing this in-house because software companies can spread the cost over vast numbers of users and collate feedback to improve their offering rapidly.

Building on Top of 3rd Party Systems via APIs

Almost every software as a service provider has an API that allows you to programmatically integrate with their systems to push and pull data. If your 3rd party does not provide an API that is enough reason to find a new provider as there is almost certainly no chance they are their industry leader and you will find better value elsewhere.

It is best to imagine each of your 3rd parties as a repository of some data for your company, a HR system for example could be the driving force on tracking headcounts, holiday, sick pay. Your CRM is responsible for lead generation and onboarding statistics and your marketing software is responsible for brand visibility and source effectiveness.

Isolated silos of information give you limited visibility of how your business operates. Having a unified data model allows you to know that a dip in performance was caused by staff holiday and as such it is nothing to worry about and may instead allow you to implement a staff-cover process to ensure business continuity.

Connecting To All Data Sources

The starting point for your unification strategy is to identify all data sources, the majority of these may be 3rd party systems although you may also have a custom piece of software in-house that will need to be included.

Once you know every data silo, you can decide which ones you require to build the picture you need for your business. It may be that you do not want to include every single data source, for example the performance of your websites may be irrelevant if they only serve as holding pages for your brand. 

At this stage you would require an API integration to be created for each data source, this is to lay down the tracks that data will eventually move from and to the data sources.

Synchronising Data From Sources

This will be the heart of your new development, you will require a new application whose sole purpose is to:

• Hold all of the API integrations
• Utilise each integration to:
• Pull existing data
• Pull new data when it is created
• Update existing records when it detects changes
• Manage deletions from the source
• Operate in an efficient manner

Each of your data sources may introduce some limits on their APIs, so it could be that you can only pull so-much data at one time, there is a delay to when data comes through, or it could even be that there are issues with your 3rd party API’s that we must work around.

In all cases these limitations may be passed onto your data model, so it could be that live data is not possible, if this is an issue and goes against your goals then you will want to re-evaluate your data provider before starting or seek to replace them in the future.

Creating Links Between Data Silos

Once data is suitably running into your system and being kept fresh this is when you can begin to stitch together 3rd parties through some common elements.

At this stage we will begin to look for equivalents between data silos, for example a record of a ‘sales person’ in your CRM, may match to an ‘employee’ record in your HR system. Your system will need to give you the ability to flag which data is not matched and let you match these.

Once you match the linking records you will be able to programmatically dive deeper to grab all associated data of those links. For our fictional ‘John Doe’ with just one link-up we have now gained access to all of their HR information and all of their sales performance data.

Build An Abstracted Data Model

Now we will be in the situation where you have data from multiple sources sitting together on your system with cross-links between them all. Now comes the exciting stage of defining your data model. This is where you get to decide what your holistic view of your entire business looks like.

This stage requires cooperation from the board and developers, the reason is that your board will know what information they want to see from all the atomic data fragments and the developers will know how to piece the data together to make this shape.

You may also have to canonicalise certain data elements, for example a ‘Sales person’ in one system and an ‘employee’ in another you may collectively decide you refer to this as a ‘Person’ in your view of data. This is part of your abstraction.

Abstraction is extremely important because it adds a layer of separation between 3rd party system’s data and what you consider the company standard. In the future it also means that you can swap out any 3rd party system with a new one and it won’t matter what they call their version of a ‘Person’, because the business abstraction model has already been established.

Creating Insights

Once you have your abstracted data model we can be sure that these concepts will not change, therefore we can build new ideas on top of them, generally developers refer to these as ‘views’ while managers refer to them as ‘insights’.

Your insights can be amalgamations of existing information to create new metrics. For example you can combine a ‘Sales Target’ from one system with the ‘Holiday Taken’ in another system to create a ‘Real Sales Target’ which can automatically reduce the target for days that your staff are off to create a realistic sales target and potential forecast. 

Insights will be created by developers but it is important to remember that the business must drive the imagination behind what insights they want to see. The power of unified data application is that you are totally empowered to make these insights and the more creative and analytical you can be about it the more value you will pull from this project.

Implementing AI

Insights are normally the end-goal for most businesses as this provides enough information to allow them to act and grow, however despite many businesses not reaching this holy grail of information, there is still a step further that is available to us and it is the implementation of AI.

We can automatically look at your insights and data to provide correlations and predictions that humans alone simply can’t do.

AI or more specifically in this case machine learning is capable of reading through all of your data and in-fact is a requirement, you have to feed it every single piece of data you have to give it more certainty in its predictions.

An example of how this works is to look at the growth in performance of each individual member of your sales team over their entire career at your company. You can begin to get a trajectory of their success. With AI you can take in various factors such as hours worked, % of leads called to start to identify which factors make sales people successful.

Now that your AI system has a profile of what makes a successful sales person it can look at new sales people and with limited information on their performance to gauge what their potential trajectory in the company will be. 

This is a game changing tool to be able to offer staff support from day one to develop their skill set to their full potential.

Summary

While having your data across many 3rd party systems may initially have you feeling locked out of analysing your own data, this is not the case. 3rd party systems with a unification strategy is absolutely an effective way to run your business because you can:

• Have industry leading tools
• Create a minimal development solution
• Access all of your data coherently
• Scale your business without scaling your IT