Data Quality Management in the context of Big Data

Nowadays, rapidly growing volumes of structured and unstructured data (i.e. Big Data) make data quality a highly relevant topic. For instance, large volumes of unstructured data from different, distributed sources in diverse formats are collected and analyzed (often in real time) to derive business-relevant insights and to support business decisions. To ensure well-founded and reliable results, the quality of the underlying data is indispensable, which has also been explicitly highlighted by current surveys and studies (Economist Intelligence Unit 2011, IBM Institute for Business Value 2012, SAS Institute Inc. 2013).

Even though ensuring the quality of data is extremely important in today’s information age, according to a recent survey (Packowski und Gall 2013), 60 % of the respondents see an enormous backlog with regard to the assessment, management and improvement of data quality in their companies. Recent surveys (e.g., IBM Institute for Business Value 2012) emphasize that data quality is of utmost priority, especially for unstructured and distributed data. The reason is that insufficient data quality results in incorrect findings that lead to wrong decisions, doing more damage than good. Thus, the well founded and adequate assessment, management and improvement of data quality generate a substantial benefit and as a result a strong competitive advantage for companies.

More precisely, in our research, we develop and evaluate quantitative methods and models to assess, manage and improve data quality. Our research objectives can be classified as follows:

1. Development of approaches for assessing the quality of structured and unstructured data: We develop efficient quantitative approaches for assessing the quality of data values considering data quality dimensions such as correctness, consistency, currency, completeness, and unambiguity. Due to the large data volumes companies face, these approaches need to be applicable in an automated way. Moreover, they need to be suitable for different data formats (e.g. structured and unstructured data) as well as for distributed data (e.g., internal and external data).
    
2. Development of data and text mining methods considering data quality: To analyze structured and unstructured data, we develop and enhance data and text mining methods in order to directly take into account the assessed data quality level (cf. 1.). Thereby, taking into account data quality does not only result in considerable changes of the derived results (e.g., document classification or assignment to clusters), which highlights the fact that not considering data quality will lead to wrong decisions. Rather, the quality of the results is derived and reported in dependence of the quality of the input data in order to enable a better decision support.
    
3. Economic management and evaluation of data quality measures: We develop decision support approaches to enable a cost-benefit evaluation of data quality measures in the context of Big Data. Due to the characteristics of Big Data, both the benefits and the costs have to be determined in an automated way. The main benefit from the application of data quality improvement measures is the better decision quality due to higher data quality. However, both the particular application context and the applied BI method play an important role here.
    
4. Ensuring data quality in the context of Big Data considering IT security issues: In the context of Big Data, data from a plethora of application fields and data sources (e.g., internet platforms, Social Media as well as mobile devices and services) are considered, integrated and analyzed. Thus, in addition to traditional data quality dimensions (e.g., completeness, correctness, consistency and currency), special emphasis is placed on analyzing how the availability, reputation, and integrity can be ensured in the context of Big Data and how IT security and data privacy issues can be adequately considered.

Selected Publications:

• Heinrich, Bernd und Klier, Mathias (2015) Metric-based Data Quality Assessment - Developing and Evaluating a Probability-based Currency Metric. Decision Support Systems. in: DECISION SUPPORT SYSTEMS (DSS), Vol. 72, April 2015, pp 82-96.
• Heinrich, Bernd und Hristova, Diana (2014) A Fuzzy Metric for Currency in the Context of Big Data. In: 22nd European Conference on Information Systems (ECIS), 2014, Tel Aviv, Israel.
• Hristova, Diana (2014) Considering Currency in Decision Trees in the Context of Big Data. In: 2014 International Conference on Information Systems, 14.-17.12.2014, Auckland, New Zealand.
• Heinrich, Bernd und Klier, Mathias (2011) Assessing data currency — a probabilistic approach. Journal of Information Science 37 (1), S. 86-100.
• Heinrich, Bernd und Klier, Mathias (2009) A Novel Data Quality Metric for Timeliness considering Supplemental Data. In: 17th European Conference on Information Systems (ECIS), 2009, University of Verona.
• Heinrich, Bernd und Kaiser, Marcus und Klier, Mathias (2009) A Procedure to Develop Metrics for Currency and its Application in CRM. ACM journal of data and information quality: JDIQ 1 (1).
• Heinrich, Bernd und Kaiser, Marcus und Klier, Mathias (2008) Does the EU Insurance Mediation Directive Help to Improve Data Quality? - A Metric-based Analysis. In: 16th European Conference on Information Systems (ECIS), 2008, National University of Galway, Ireland.
• Heinrich, Bernd und Kaiser, Marcus und Klier, Mathias (2007) DQ Metrics: A Novel Approach to Quantify Timeliness and its Application in CRM. In: 12th International Conference on Information Quality (ICIQ), 2007, University of Cambridge.
• Heinrich, Bernd (2007) Der effiziente Einsatz proaktiver und reaktiver Datenqualitätsmaßnahmen - Ein modellbasierter Ansatz und seine Anwendung bei einem Finanzdienstleister. Die Betriebswirtschaft 67 (5), S. 539-562.
• Heinrich, Bernd und Kaiser, Marcus und Klier, Mathias (2007) How to measure Data Quality? A Metric-based Approach. In: 28th International Conference of Information Systems (ICIS), 2007, Queen's University Montreal, Kanada.
• Heinrich, Bernd und Klier, Mathias (2006) Ein Optimierungsansatz für ein fortlaufendes Datenqualitätsmanagement und seine praktische Anwendung bei Kundenkampagnen. Zeitschrift für Betriebswirtschaft 76 (6), S. 559-587.
• Heinrich, Bernd und Helfert, Markus (2003) Nützt Datenqualität wirklich im CRM? - Wirkungszusammenhänge und Implikationen. In: 6. Internationale Tagung Wirtschaftsinformatik 2003: Medien - Märkte - Mobilität, 2003, TU Dresden.