Data onboarding

What is data onboarding?

Data onboarding is the process of taking data from offline sources and converting it into a digital format that your online tools can use. 

For example, your sales teams, your survey tools, and your customer relationship management (CRM) platform gather data about customers all the time. However, to get that data into your data warehouse, you need to go through a data onboarding process.

Online data vs. offline data

Because you will merge offline data with your online data, it’s important to understand the differences between the two. Put simply, offline customer data is any data that you collect from an offline source, and online data is data that comes from online sources. But, let’s go deeper:

• An offline source can include a customer survey, a signup for a loyalty program, in-store purchase history, or information a sales rep enters into your CRM after a phone call with a customer. This involves any data that your tools are not automatically picking up. 
• Online sources include your website and behavioral analytics tools, social media channels, and other online platforms. Often this is the data you collect automatically through data pipelines and digital marketing tools. 

It’s important to remember that “offline” doesn’t necessarily mean that the data was gathered in a completely analog fashion. While it can be, such as a paper email sign-up sheet for a store mailing list, it can also include customer-related notes in an invoicing or CRM platform, or notes recorded in a spreadsheet file.

Data onboarding examples 

Here are three examples of offline data and how onboarding that data can give a company useful insights. 

• A cell phone company sends its customers a survey. The survey gathers psychographic data about how customers feel about their service (such as “rate this service on a scale from 1 to 5”), and the respondents’ demographic data (such as age). The responses give information about what customers like or dislike about their service. Data onboarding could help connect that psychographic data with the company’s broader demographic data, and provide marketing insights for future campaigns.
• An athletic shoe company’s customer service team notices an uptick in calls and emails about an issue with the laces on a popular shoe. If the reps take notes on these issues and the company onboards that data, they may learn something about those customers. For example, the shoes may not be suited to long-distance running, but perfect for another sport.
• A retail storefront makes hundreds of sales per day. If the point-of-sale (POS) systems are not onboarding that data, the company may be losing geographic and inventory-related data that could provide usable insights into the customers’ or region’s spending habits.

Benefits of data onboarding

The main benefit of data onboarding is the ability to combine all data from all customer touchpoints into a single source of truth. If all data is in one place, organizations get more insights and value from the data they already have, gain a more holistic view of their customers, and potentially assign a persona to some of their anonymous online data.

With the number of data-related compliance laws and frameworks around the world, consolidating offline and online data through onboarding can also streamline opt-out processes. Having all of your offline data in the same secure location as your online data can also help with data security, privacy, and governance. 

How data onboarding helps marketing campaigns

Having a 360-degree view of customer data can be a huge benefit to marketing campaigns because it gives a complete picture of how customers are responding to marketing efforts and allow teams to create more personalized experiences. This can be useful for multiple reasons: 

• Conversion tracking - If a clothing store has recently sent out an email campaign with an in-store coupon code, a data onboarding process can help the marketing team track whether people are using that coupon code. This gives insight into how many customers are going to the store after receiving those codes.
• Audience suppression and retargeting - The same clothing store advertises on Instagram and uses its onboarded data to determine who among its followers are and aren’t making purchases in their store. They can then use that information to target their ad spends, either putting more ads in front of people likely to make purchases or putting fewer ads in channels that the non-customers use. 
• Joining disparate data sources - Your data may exist in a variety of locations, meaning you need to go to separate tools for analytics and reporting. Introducing a data onboarding process allows you to see trends from the combined data sources. For example, you can combine survey data with statistics from call centers, giving you insights that you may miss if looking at the data separately.

What is the data onboarding process?

There are four main steps to the data onboarding process:

1. Collection - Start by finding out what data sources you have, and how you can extract data from those sources. This may require making changes to those data sources. For example, you may need to change a survey question from a long text box to a pick list of options to gather more useful data. 
2. Uploading - The organized data can now be added to the  main data warehouse. This should be done with a secure shell (SSH) or over a connection with secure sockets layer or transport layer security (SSL/TLS) encryption, to ensure data is transferred securely 
3. Preparation - Collected data won’t be organized in any meaningful way. The data needs to be cleaned — duplicate entries consolidated, extraneous or incorrect information removed, and any PII or sensitive details anonymized. This is also where your identity resolution for that data will occur. Make sure there is a system to handle errors, such as collisions (two pieces of data going into the same place).
4. Validation - This step ensures the uploading process worked. It’s important to validate and make any adjustments to the process before repeating it. This kind of testing can be automated. Validation checks should include: 
   1. Performing spot QA checks
   2. Ensuring data has unique IDs
   3. Verifying that the uploaded and existing data is linked properly
   4. Checking that the output meets expectations
   5. Ensuring that the onboarding process meets your compliance requirements

After this, the onboarded data (formerly the offline data) can be activated and used in other tools. 

If we imagine a data warehouse as a physical warehouse, the process would be similar to:

1. Gather a bunch of files and folders (Collection)
2. Organize and box them, toss out anything that’s not being stored (Preparation)
3. Put them away in their correct places within the warehouse (Uploading)
4. Verify that everything is in its right place (Validation)
   ‍

Which roles manage data onboarding?

Data onboarding is a group responsibility because the relevant data sources could come from any number of teams or departments within your organization. Some of the key roles in the data onboarding process are: 

• Data and engineering teams - Because an organization’s data is already under the data and engineering teams’  purview, the teams responsible for the data pipeline, data collection tools, and other data sources should be involved in any conversations about a data onboarding process. 
• Product or website management - These teams should be involved if an organization relies on collecting data through its website or other product-based analytics. If there are new types of data the organization wants to collect, these may also be a good resource for figuring out how to gather it.
• Customer service or success - For data collected through a CRM or customer satisfaction survey, an organization will need to empower customer-facing teams to collect and manage that data. Because they are the ones collecting it, they will likely also need some ways to fix mistakes or maintain that data. 
• Marketing teams - The marketing team may be sending out customer surveys or similar data-gathering tools as part of a campaign, or to prepare for one, so they will be in a position to review and onboard that data afterward.

Before implementing a data onboarding strategy, organizations should also consult their legal team to make sure that you understand what your data privacy and security responsibilities are. Some countries have laws that surround the use of personally identifiable information (PII), like the EU’s General Data Protection Regulation, or some regulations set forth by the US’s Federal Trade Commission Act.

When considering a data onboarding strategy, it may be easy to overlook one final, but important role: 

• The customer - Customers are providing their data, and for some sources — like surveys — they will manually enter the data for your process or tools to onboard. Keep this in mind so that offline data collectors are as smooth and painless as possible, or customers may not use them.

Data onboarding best practices

Onboarding data can be a complex process, especially on the first instance. But there are some lessons you can learn and take note of during the process to make things easier the next time:

Continuous improvement

It may be tempting to view data onboarding as a one-and-done process, but organizations should revisit it periodically to make sure it is still working. 

• Be prepared to refine data collection tools.
• Empower all of your staff to say something if they see that something isn’t importing properly. 
• Revisit the process after a set number of iterations, and make sure that all the data is going to the right places.
• Develop a data observability dashboard that provides insights into how the onboarding and preparation steps are performing.

Look for automation strategies

Because a lot of offline data can be collected manually, it may seem like the fastest way to get it into the data warehouse is to enter each item by hand periodically. Instead, businesses should consider creating or purchasing a data onboarding or importing tool. While it may take a few seconds per item today, that may not scale with the business in a few years’ time. 

Data governance

Offline data may be a mess when first collected. As you complete the preparation stage of the onboarding process, you may notice some ways it could have been collected to make preparing the data easier. Lean into that. If you can refine your collection sources or train your data-collecting staff to enter the data properly, this can streamline the process later. These are the first steps in establishing data governance rules, which dictate how the data is collected and structured.

What should you look for in a data onboarding provider?

Before seeking any sort of data onboarding tools, first find out what your data stack looks like. Consider questions like:

• Are there data onboarding tools designed to work with the data warehouse you are using?
• What is the connection type of your data sources? For example, are they CSV files, API, or a Database?
• What tools are already part of your stack?  Are they performing adequately, or do they need to be replaced?

Having this information on hand during a conversation with a potential vendor means you can ask about integrations with their tools. If their solution can’t integrate with your tools, they may not be the right partner for you.

One key feature for your data onboarding tool is data security and privacy. You need to make sure your data onboarding strategy meets any legal requirements for organizations in your country and industry. Before talking to your vendor, meet with your information security and legal teams to find out what questions you need to ask.

After that, it’s time to compare features, look at integrations, and figure out which solutions are the best fit for your organization. This means research and conversations with vendors and any teams that will be using the onboarding tools.

Data onboarding tools and solutions

There are several tools on the market that can help streamline or manage a data onboarding process, including:

• LiveRamp
• Osmos
• Integrate.io
• Flatfile