Tips, tricks and techniques to improve data quality   

It is estimated that up to 30 per cent of an organisation’s revenue could be misspent due to poor data quality. Improving the frameworks for good data quality can deliver greater operational efficiency for businesses and reduce unnecessary expenditure. 

Data quality ensures the business has access to consistently accurate data in a timely fashion that is unique across the organisation and available to all. It can be quantified under six dimensions: Timeliness, Accuracy, Validity, Completeness, Uniqueness, and Consistency. At Sempre, we have a simple mission, which is to enable our customers to achieve better business outcomes through the intelligent use of data. Our strengths lie in supporting our customers to achieve greater financial resilience and operational efficiency. 

Data quality and data management are key to all of this and underpin the ability for Sempre to deliver those business outcomes for our customers. 

Why is data quality important? 

Poor data quality can remain undetected for prolonged periods and show up in several ways over time. For example, data quality had a significant impact on a global research project aimed at mapping the human genome. The project made some exciting promises in the advancement of medicine and treatments for human diseases. One gene identified through this research was the Membrane Associated Ring – CH type 1, which was abbreviated to MARCH 1. This was considered a good name until Excel was used to support the processing of data in the research and as such Excel changed the MARCH 1 input to a date, March 1st.  

A study in 2016 identified that as many as 20% of all published scientific papers on MARCH 1 contained one or more data quality issues. The resolution to this challenge was to change the name of the genome globally. 

This is just one compelling example that underscores the importance of good data quality and demonstrates the importance of it for organisations, whether for day-to-day operations, expansion plans, regulatory compliance, or transformational programs.  

“Do not underestimate the importance of having a critical approach toward managing the health and fitness of data as poor data quality completely destroys business value.” Ankush Jain, Melody Chien, Gartner 01 November 2022

What does poor data quality mean for business? 

For a housing provider, poor data quality can result in significant costs. An example of this is the incorrect logging of postcodes in a database. These incorrect entries lead to maintenance contractors being dispatched to the incorrect address for maintenance tasks. As well as the short-term operational cost of a missed visit, there is the medium-term reputational cost of not turning up to a scheduled appointment, as well as the cost of rebooking the maintenance service. Over the long term, this could negatively influence the health of the company’s revenue and sustainability, decision-making influenced by incorrect data relating to profitability, carbon emissions, and other critical targets.  

The Data Management Body of Knowledge (DAMA) is a global data management community who have collaborated with the UK Government to define the six dimensions of data quality: accuracy, completeness, uniqueness, consistency, timelines and validity (source: gov.uk). Their findings suggest that these data quality dimensions help organisations assess if the data is good enough to use, or if improvements are required. They have also worked with other global agencies to gather information on the costs of poor data quality to businesses. DAMA’s estimates suggest that between 10 and 30 percent of the company’s revenue is spent unnecessarily, due to poor data quality. 

Benefits of a DQ framework 

A challenge that growing companies often face when trying to improve and control their data quality is the siloed data within Excel spreadsheets. These offline data stores proliferate throughout organisations, containing out-of-date information and incorrect definitions. Little consideration is ever given to the control or validity of the data and its quality within the files. In this scenario, it is much more common to find different data formats and definitions across the various departments. 

Promoting a data-driven culture within an organisation can help resolve these issues and yield many benefits, including but not limited to:  

1. Improved trust in relationships across the company and with customers.  
2. Trusted regulatory reporting. Examples of this are varied, for instance, food manufacturers must maintain food safety standards, while financial services must comply with consumer duty regulations.  
3. Securing operational efficiency. It is always more expensive to go back and fix things afterwards than do it right in the first place. Identifying the cause of poor data quality in business can lead to huge gains in operational efficiency. 
4. Improved NPS and reputation with customers. Customers report greater satisfaction working with suppliers that have accurate records. 
5. Enhanced downstream systems. An example might be moving a record from prospect to customer, through the billing system and ERP, along with a variety of other internal systems. At each stage there is the possibility of compromised data quality, the consequence is that numerous records will need to be updated and improved upstream – it becomes a system problem to clean. Improved data quality throughout the process can reduce this headache. 
6. Data Quality can improve data governance. It is the first step on the ladder to good data governance. 

Data Quality Framework 

The process of software development is rigorous and incorporates several stages that ensure quality output. The first stage of development is coding, and once the application is developed it moves on to testing and ironing out any bugs through a couple of iterations. Once that has been completed, the development moves to pre-production and is tested again before being released. The same level of focus is required to ensure the quality of data.  

A recognised framework is to standardise, monitor, collaborate and ensure the quality of data that is required.  

Principles – appoint people within an organisation to monitor and control data quality. By calculating the costs associated with operational inefficiencies resulting from poor data quality, it will be quick to build a business case using the cost savings achieved from delivering good-quality data. 

Process – analysing the processing of data is crucial to improving data quality. To address the challenge of poor data entry it is important to consider if training is required, or if there is an issue with the data entry design. It is important to consider whether free-text fields can be replaced with lookups, or if the drop-down menu is overly long and therefore causing difficulties in system use. To make accurate conclusions you need to follow the process back to the source, review the challenges and understand where the data was disrupted. 

Technology – The days of manually solving data quality issues by reviewing data extracts in spreadsheets are long gone. Data is in a constant state of evolution, with volumes increasing at an unprecedented rate, and the rules governing the data growing more complex. What businesses now need are technology solutions that can profile and analyse the data to identify patterns and irregularities, to allow for a timely resolution as part of standard data processing. 

Getting it right 

To achieve high levels of data quality across the organisation it is important to start with the outcome – for example, what are the business goals for implementing good data quality? Defining this is the first step to enabling project leaders to find sponsors within the business to back the project. 

Once that buy-in is achieved, the route to success is: 

• Analyse the challenge, assess data quality as it is now and score it. 
• Review the cause, which will help to plan for a resolution. 

• Build a framework to correct the processes that have caused quality issues in the system. 
• Implement controls across the business to prevent the issue from reoccurring. 
• Remember that this remains a continuous improvement process.  

Sempre Analytics can help at every stage in securing data quality for your business. If you would like to know more, contact Sempre and include Data Quality in the subject line.

About the Author

Tom Clark With more than 25 years’ experience in technology solutions, Tom leads the Data & Analytics practice within Sempre. Tom has been with Sempre from the beginning and has worked across the full programme lifecycle to advise, scope, deliver, and manage, high-value programmes for our customers. He has a wide breadth of experience in all aspects of the data management lifecycle, including data solutions, strategy, governance, master data management and quality, having delivered transformational data solutions across multiple industry sectors.