Over the last decade, enterprises have come to realize how valuable and influential their data can be for overall business success. However, the challenge arises when firms effectively govern and manage large volumes of data while maintaining security, integrity, and quality. Incorrect, incomplete, inconsistent, or untimely data quickly trickles down the firm’s pipeline, affecting every aspect of the business. Low-quality data hinders an enterprise’s operational efficiency, regulatory compliance, and decision-making capabilities. Therefore, improving and maintaining data quality (such as data timeliness) through effective data governance has become a top priority for all data-driven businesses.
In a previous article, we discussed three dimensions of data quality. Today, we are going to focus on the importance of timeliness or having timely data.
One of the most important dimensions of data quality is timeliness. When a firm makes a business decision about new market opportunities, growth initiatives, regulatory compliance, operational efficiency, etc., timely data is essential.
Timeliness refers to data availability and accessibility—data should be received and accessed at the expected time by the right stakeholders for it to be used efficiently for decision-making.
When a piece of data is not timely, other data quality dimensions can also be affected. For example, correctness is closely influenced by timeliness, as data accuracy can easily decay over time. Using outdated or incorrect data can lead to inaccurate analytics or insights, thereby negatively impacting decision-making based on false information. Additionally, data consistency should be based on timeliness – you may have systems that produce data at different times that need to be brought together to make the right decisions. This same data needs to be measured and optimized, which can only happen if it is consistent and timely.
Data timeliness can be measured as the time between when a specific data element is expected (say, at time TE) and when it is readily available for use (say, at time TA). Data that is available on time or ahead of time (TA ≤ TE) is beneficial because business decisions will correspondingly be taken on time or ahead of time. No further analysis is necessary. However, when data is not available in a timely manner (TA ≥ TE ), it becomes necessary to investigate the root cause, and for that, we need to understand both the factors that contribute to data timeliness and how to measure data timeliness as a factor of process cycle time.
Because data is created, enriched, and consumed as part of one or more business processes, data timeliness must be measured at every step in a process, not just at the end of a process, i.e., at the output stage. For example, the timely availability of cash balances or outstanding payments does not depend solely on the timeliness of the specific balance report or payment report. Why is this? A report consists of multiple line items. Each line item can either be an individual reported item, say, a balance of a specific account, or an aggregate, say, a balance of a portfolio. Typically, the data used to generate the line item may reside in sources external to the application that generates the report. If this data is not made available on a timely basis, the line item(s) will not be calculated in time, hence delaying the entire report. Therefore, for measuring data timeliness of this example report, we need to measure the timeliness of each data element that the report directly includes, or depends on. In other words, it is essential to measure process cycle time, which is made up of each element’s time of availability.
Cycle time is the total elapsed time to take a unit of work from start to finish through all steps of a process. The specific unit of work depends on the business process. It could be a customer application, a payment instruction, a customer order, an insurance claim, a buy/sell order for a stock, etc. For example, fulfilling a payment instruction in a timely manner depends on the timely completion of each step in the payment instruction business process. For simplicity, assume these steps are:
Typically, internal and external service-level agreements (SLAs) govern mandated completion times for each step of a business process. Data timeliness, then, can be measured by ensuring that all relevant SLA requirements are met. This is because each of the SLA requirements is only met by evaluating underlying data being provided to that step. In theory, this seems pretty straightforward. In the next section, we examine the challenges involved in measuring data timeliness specifically in the context of major global enterprises.
At any firm, business processes are automated by a wide variety of software applications. Some applications, such as order entry systems, may involve human input, while others, such as order validation and fulfillment applications, may be fully automated. For a business process at a major global enterprise, there may be dozens—even hundreds—of applications involved in creating, enriching, and transforming data.
Consider the example in the prior section, that of a payment instruction. A customer may submit a payment instruction online using a web portal, or a customer agent may create a payment instruction on behalf of the customer. This results in a new transaction or record with a specific state, like “new”. This record may be stored in a database after performing basic validation. The payment instruction may then be dispatched to a second application that performs AML verification, after which its state may change to “AML-verified”. The AML verification application may maintain a copy of the record in a separate database. Next, a routing application may route the instruction to one of the payment channels. It would then be transformed into a format required by the specific payment network, and the state may change to “dispatched to network”. The network then processes the payment and returns an acknowledgement, at which point the state of the payment instruction is updated to “completed”.
Each of the applications that have created or transformed the specific instruction may be implemented in a different language (Java, C++, Python, etc.), deployed either on-prem or in a cloud, may use different databases (Oracle, NoSQL, etc.) and store data in different formats (CSV, Avro, Parquet, domain-specific formats like FpML, etc.). Inter-process communication may be via APIs, or through a publisher/subscriber interface, such as a message broker.
Why are these details important from a data timeliness perspective? Assume that the global firm offering payment processing to its customers wants to ensure timely completion of payments. Failure to do so may expose the firm to legal and regulatory penalties. So the firm compiles a report consisting of all payments by volume and client, with completion status and time is taken to process each transaction.
In case of delays, this means examining each step in the process that we described, along with the time taken for each step and each state change. It means querying multiple databases and application logs, which may include tens or even hundreds of applications and databases. Even if this happens, the firm then must construct a time-series view for every payment instruction. In many cases, a graph view may be needed, especially where orders are split or aggregated as part of a business process.
Now, imagine constructing this end-to-end (E2E) transactional workflow view for hundreds of thousands, even millions of transactions, and doing so in a near real-time manner. Firms try to address this complex problem by using application performance management (APM) tools. However, APM tools provide low-level information, such as request/response times, database query response times, etc. This is required by IT teams to monitor application, database, and network performance, but do not include the business contextual information needed to measure the timeliness of data and process cycle times accurately.
A platform is needed that provides an E2E, front-to-back view of all business transactional workflows in an easily understandable representation. The representation should include both a time-series view that accurately measures how data and its state have changed over time, as well as a graph view that represents each step of each business process that has acted upon the data. It should be able to handle complex operations such as “fan-outs” to represent orders being split into sub-orders and “fan-ins” that represent sub-orders being aggregated into consolidated or “netted” orders. The platform should be able to create this view in a near-real-time manner, across disparate data sources, and at massive scale, with no IT effort.
Siloed and Fragmented Enterprise Landscape
Currently, firms struggle with timely access to critical data due to fragmented and siloed enterprise landscapes. A silo is a singular collection of data or information that isn’t effectively shared across an enterprise. Data is acquired from and stored in disparate systems, applications, and workflows throughout the enterprise infrastructure. Each of these data sources consists of scattered, outdated, and duplicate records, leading to availability and accessibility issues.
Lack of E2E Data Quality
Using existing metadata tools, firms cannot monitor data quality throughout the data’s lifecycle and enterprise landscape. As a result of silos and fragmented systems, businesses struggle to measure timeliness across workflows, resulting in reduced operational efficiency, regulatory compliance, and decision-making. Most of these are modeled manually with samples after the fact, instead of being able to see all permutations.
Static Data Governance Tools
Currently, many enterprises use passive data governance tools to govern and manage their data. These tools perform data quality checks as a last step in the pipeline, instead of perpetually checking throughout the data’s lifecycle. With this passive approach, firms are manually fixing errors or discrepancies within each individual silo, instead of across the enterprise as a whole. Timeliness is measured at specific points for individual data sources instead of across E2E systems, applications, and workflows.
Lack of Ongoing Data Quality Checks and Bilateral Tools
Additionally, many data quality tools are batch-oriented and perform data quality checks sporadically. These solutions are often strictly bilateral, which means that for enterprises to perform E2E data quality checks on complete business flows, they must perform multiple bilateral data quality checks. This results in high false positives, which means that the same upstream error can lead to duplicate issues in downstream data quality checks. This increases the costs to firms as well as affects timeliness for other processes.
Without monitoring data quality and timeliness holistically throughout the enterprise and lifecycle of the data, even the smallest error can trickle down and cause significant distress to an enterprise’s operational efficiency, regulatory compliance, and decision-making.
An effective data governance model solves a plethora of challenges around enterprise-wide data quality and specifically, data timeliness.
By unifying enterprise silos, effective data governance creates a framework for E2E data quality improvement and management. A functional data governance tool can successfully assist to break down data silos by providing E2E visibility across the firm. With easier accessibility and availability to enterprise data (regardless of where it is stored, located, or managed), firms can ensure that its data is timely.
With effective data governance, firms implement processes and rules to ensure that low-quality data is identified and addressed on an ongoing basis throughout the organization. This model reconciles the processes, rules, and systems that created the data quality issue in the first place.
The PeerNova® Cuneiform® Platform is an active data governance tool that enables E2E trust and transparency of data and business flows. The solution ensures data quality dimensions are met by perpetually running Data Quality and Timeliness Rules on live data across the enterprise landscape. Through automation and ongoing data quality checks, the solution not only remedies existing data issues but also continuously manages and monitors data throughout its entire lifecycle across siloed sources.
The platform reduces root-cause analysis and resolution time for data quality, SLA, and timeliness metrics, ultimately leading to better operational efficiency and faster time to market. Additionally, PeerNova’s solution improves existing processes by using timeliness metrics to manage issues that could result in regulatory fines and capital charges by reducing exceptions over time. By providing E2E data quality and unifying fragmented landscapes, leaders are able to instill confidence in the analytics and insights that fuel their decision making.
Using this dynamic approach to data quality and data management, the platform creates E2E, integrated, and active lineages across disparate tools and systems, making the solution significantly more efficient than other data governance tools.
If you are interested in learning more about PeerNova’s Cuneiform Platform and how it can improve your enterprise’s data quality, please get in touch with us and request a demo today.
