Background
Organisations of all types – be it for-profit, not-for-profit, or Governmental – have all been shaken by the pandemic for over a year now. The need to be in constant touch with the end-users or customers, as well as the key service delivery agents or suppliers, has never been found wanting earlier than during the pandemic. Software of various types already shape up the core business processes that run such large organisations. However, in order to cut across the islands of digital automation and ensure that the needs of the various stakeholders are addressed in real-time or near real-time, it is imperative to put in place the Data factory for the organisation. This will consist of all the tools, frameworks, processes, and skilled personnel who would be held accountable for ensuring that the right kind of data supports such real-time needs.
A Banking case
For instance, a Bank may like to address the transactional needs of its retail customers. Perhaps, over 30% of the real-time cash transfers are having an issue in the past week. This is a real scenario that will result in an adverse customer experience and the fallout of this event that may end up damaging the reputation of the Bank. Managers and leaders responsible for customer engagement need access to real-time information, possibly through dashboards that present the state of all the operations. This may give the necessary firepower in terms of understanding why the ongoing issue is happening in the first place. Once key insights are available, the Bank may then like to fix the faults that lead to such transactional failures. Perhaps it had to do with the batch job that was keeping the golden customer records in sync. Perhaps it was the latest software upgrade for one of the key payment components that resulted in the issue. Identifying the root causes for such failures at a high level, and then arriving at fixes is often a complex task.
The Need for Data Engineering
The complexity stems from the scores of legacy systems – be it for capturing customer inputs, processing those inputs against rules and regulations, and eventually committing transactions to reflect the modifications to various databases – and the way these systems hide the root causes. The first step toward ensuring that the Bank is able to understand the state of affairs, leave alone leveraging its resources and assets to take necessary right action, is to put in place a Data Engineering practice. This would encompass a variety of disciplines including architecting a Data platform that will host and curate the data required for operational decision making and transactions. While handling large-scale data processing and ensuring that systems required for this purpose and built and maintained, the Data Engineering practice will also be normally responsible to deliver data for proper consumption via analytics, analysis, data science and other operational use cases. The infrastructure that is required for such a platform to be functional also becomes the responsibility for this practice.
Integrating with Business Case
Coming back to our Banking use case, this will entail building a Data lake that has all curated data and information regarding customers, transactions, Bank employees, external parties, and the like. A quick access Customer-360 application is also par for the course. Also, the necessary views and dashboards that are required for diagnosing the transaction failures would have been built. More importantly, users of various categories including Bank managers, marketing leaders, or Data specialists will have their own ways of accessing and leveraging such a platform. An issue such as transaction failure can be very quickly sensed, analysed, and responded to. What would have normally taken a few weeks to diagnose and fix can be addressed in a few hours, thanks to the Data engineering practice.
Conclusions
With the availability of trustworthy data and insights, understanding the process inefficiencies and more importantly identifying the right levers to respond to any process distress events is made possible thanks to a solid Data Engineering practice within the enterprise. This calls for a Vision to be put in place, followed by the discipline to implement with appropriate change management, training, and process interventions. When architected and staffed well, Data Engineering will indeed prove to be the panacea for enabling a truly Digital organisation.
