Data Engineering Requires Human+Machine Fusion

Automation is abundant. We sit at the point of an extended automation revolution where basic notions of robotic process automation (software “robots” designed to screen-scape our clicks and interactions and speed up simple and some not so simple forms-based application interactions) are now starting to look like legacy relics from a bygone era, before modern AI.

As we now formalize and codify the next era of automation, its mechanics will stem from data engineering i.e. the practices, methodologies and implementation techniques software developers use to beautifully orchestrate our IT services at the platform level, at the networking level and upward to the application level itself.

However, data engineering is not an open and shut case. Managing data mechanics in this arena to power smart cloud-native applications poses significant challenges, including tasks related to data duplication, verification and management. Those are data automation tasks that need a fair degree of human oversight; automation is here, but it needs a human and machine (AI) toolset working in collaborative harmony to make it happen effectively and safely.

Zettabytes Of Data

The European Commission estimates the value of the EU data economy will be over €800 billion by the end of 2025. Similar growth figures are anticipated for the US market where an estimated 180 zettabytes of data could be produced by 2025, the value of which obviously stretches to several billion dollars. All of which mans that data engineering, as part of a wider corporate data operations strategy (now widely being called DataOps), is core to business innovation and commercial growth.

“The key to working at this tier of data engineering and overcoming the challenges inherent to the practice lies in the power of human and machine cohesion,” said Gary Sidhu, SVP of product engineering atGTT. “Harnessing the synergies between human data engineers and AI is crucial to achieving best-in-class data engineering that happens under the auspices of disciplined data management practices. However good it is, the technology stack alone will not deliver high-performance data engineering functionality; humans are just as important, if not more, as they set the standards and ensure oversight.”

Sidhu bases his comments on two decades of experience working for a telecommunications company and his current tenure at GTT, an organization known for its Envision orchestration and control platform designed to connect people, data and machines. He says that a big obstacle for data engineering occurs when there are gaps in terms of how data can be pulled together across different sources within a business, especially when companies are scaling rapidly.

Prioritize, Synchronize, Operationalize

“For example, let’s say a business has multiple instances of customer relationship management, inventory and order management systems due to inorganic growth. A lack of operational visibility and discipline in connecting data systems leads to fragmented data and hinders the ability to achieve a unified view of the customer. These issues could lead to lower productivity and innovation levels… leading then to poor customer and employee experiences,” said Sidhu. “To generate a unified knowledge landscape, data and IT teams need complete visibility over data operations. Companies should prioritize data integration and ensure that all systems are connected and synchronized.”

The suggestion here is that creating a central information management hub can help transform chaotic data practices into a more streamlined and enhanced experience. A hub can integrate data from various systems and by consolidating data into a single repository, companies can eliminate redundancies, reduce errors and ensure all stakeholders have access to the most accurate and up-to-date information. A central hub also enhances the speed and efficiency of data retrieval.

Intra-organizational Discipline

The success of data engineering initiatives at this human+machine level heavily relies on organizational discipline. It is down to the human developers, compliance managers and data engineers who design, communicate and enforce data governance policies. That’s no easy feat and this is where automation can come in and support human endeavours. This is especially so when the engineering laundry list involves decommissioning outdated systems, consolidating data into a single repository and ensuring that all data is accurately classified and indexed.

Organizational discipline also means having a clear data strategy and ensuring that all employees understand and subscribe to data management best practices. This includes delivering regular training and education on data governance, as well as establishing clear roles and responsibilities for data management from the start.

“By fostering a culture of data discipline within cross-functional teams from security to compliance and HR, companies can ensure their data engineering initiatives are sustainable. Cross-team functionality is crucial in this regard, as the embedded contextual business knowledge each employee has, combined with the collaboration between different departments and roles, enhances the overall effectiveness of data engineering projects,” explained Sidhu. “By integrating efforts across various teams and using automation, organizations can also ensure their data engineering processes are robust, secure and capable of supporting their business objectives.”

Securing Data Engineering

As data is vital to any business operation, securing data transmission from the core to the “edge” is critical. Securing data connectivity also maintains a high degree of data quality and integrity behind any data engineering initiative. Modern application programming interface management is essential for maintaining data security because APIs play a crucial role in securing the engineering lifecycle. By implementing robust API gateways and enforcing strict access management policies and controls, companies can protect their data assets and facilitate seamless data integration.

“When thinking beyond APIs, data engineering programmes also depend on the security of networks and applications as data passes through from one environment to another. This adds another dimension of complexity for IT security teams who need to ensure critical vulnerabilities are plugged and data compliance needs are met,” underlined Sidhu.

The growth in network virtualization capabilities are helping to simplify the journey to best practices for security, such as zero-trust frameworks. Research from analyst firmOmdia suggests enterprises are leaning toward security when considering what network functions to prioritize next, with functions for secure access and security applications such as logging, authentication, threat detection and analysis topping the list of use cases for virtualization in the coming years

The Future For Humans + Machines

Looking ahead, the collaboration between human data engineers and AI is sure to fuse more tightly.

Sidhu concludes his commentary on this subject by saying that human data engineers can design and implement robust data governance frameworks, ensuring that data is accurately classified, indexed and maintained across the business… but AI can then automate the enforcement of these frameworks, monitoring data quality and compliance in real-time, and flag where human attention is needed.

“Getting it right could lead to better employee and customer productivity by offering predictive insights and enabling a deeper degree of self-service. By creating a secure, central hub for unified data management, companies can overcome the frequent challenges of data duplication, verification, and management,” said Sidhu.

It’s a recurring theme isn’t it? Machines can work for us and AI can make our lives better, but we humans are still in control and the more tightly we can graft and fuse the work of both people and technology services, the more precision-engineered benefits we can look forward to reaping in the long run. Bionics of course epxlains the interdisciplinary zone that connects natural systems and technolgy engineering and that’s a part of what’s happening here.

Steve Austin would have been proud of us.