THE CHALLENGE: BUILDING A VIABLE INFRASTRUCTURE FOR AI
Managing and delivering a data pipeline and infrastructure that can meet demand for AI require a more sophisticated data foundation than what was seen in previous generations. However, much of the data needed is “dispersed across various data warehouses and data lakes, which lack the centralization necessary to operationalize AI and truly unlock its potential,” said Adam Glaser, SVP of engineering at Appian. That’s because AI models “are highly demanding, requiring significant management resources, frequent maintenance, and bespoke tooling.”
“Layering AI capabilities on top of these already-demanding, disconnected foundations would cause even more issues, as most organizations lack the expertise to train or fine-tune trained models on their data,” Glaser added. The source of this pain “ultimately comes from the inaccessible, disconnected nature of data silos.”
The trend: An inability to deliver data to AI models, “if not dealt with correctly or addressed at all, will only continue to get worse over time,” Glaser warned. “The amount of data enterprises generate today is increasing exponentially, and the enterprises that don’t tap into this data and make it work for them will get left behind. And with the advent of generative AI sparking new energy into enterprise transformation efforts, the stakes are even higher to solidify data foundations.”
The solution: Data fabric offers the best approach to building a viable foundation for AI models, Glaser recommended.
“With a data fabric, enterprises can work with data in a virtual layer so they don’t have to migrate data or refactor code when databases change. This not only provides easy access to data, but also a unified view of where it lies and how it’s being accessed, enabling better predictions and accuracy from AI, without having to change where it’s stored.”
As with any enterprise-wide IT challenge, Glaser also cautioned that “technology is only part of the solution. How the technology is implemented within the organization and subsequently managed is equally as important. It requires buy-in and support from not just the leadership teams, but a variety of stakeholders across the enterprise—and it cannot be done overnight.”
THE CHALLENGE: SCALING AI INITIATIVES
Along with establishing a solid data foundation for AI, enabling AI applications and systems to scale from pilot projects to enterprise needs is crucial.
“These challenges stem from various sources that have been ignored by organizations, such as data integrity, data privacy, security, infrastructure, and the cultural aspects of AI adoption,” said Doug Kachelmuss, senior director of data and AI at Launch Consulting Group. “Scaling AI isn’t merely a technical endeavor, but also requires a commitment to responsible AI use, involving aspects such as addressing bias, ensuring transparency, and establishing accountability in AI systems.”
The trend: “Organizations that have proactively cultivated a data-driven culture are better positioned to tackle the challenge of scaling AI,” said Kachelmuss. “Many organizations are now prioritizing the development of strategic plans that encompass people, process, technology, and data components to foster a well-defined, data-driven culture.”
The solution: “It’s imperative to have a strategic plan in place,” said Kachelmuss. “Start by conducting a workshop facilitated by a third-party expert to assess your organization’s current data maturity level. Remember that the journey toward a data-driven culture takes time and effort, so perseverance and continuous commitment are key to success.”
THE CHALLENGE: DATA GOVERNANCE
Data governance—long an issue as data became more core to company growth—is now also the top challenge and priority in implementing or building AI-driven applications and systems. “You cannot reap the benefits of AI without data governance, which remains one of the most challenging and complex tactics for organizations to master,” said Shadi Rostami, SVP of engineering at Amplitude. However, the challenge with data governance, she added, is one of alignment. “Many different individuals across many different departments may have access to your business’ critical data, and all have preferred methods of interacting with these findings. There are a lot of cooks in the kitchen, but not everyone studied at the same culinary school.”
The trend: The case for more effective data governance processes is increasing “with numerous new use cases for data insights being developed every single day across different functions,” said Rostami. “And if internal pressures weren’t enough, more and more regulations are being put in place to protect customer data, adding another layer of complexity.”
The solution: “Data governance is a collaborative effort, requiring the active participation from both the data team and the consumers of that data, such as marketing and product teams,” said Rostami. “This collaborative approach not only ensures consistent data quality and accessibility, but also enhances self-serve access for non-data teams, fostering a more data-driven culture within the organization.” Additional approaches include integrating machine learning to “help detect suspicious activities and monitor data security in real time, as well as automating access controls and change detection, will inevitably save time and resources for your data team.”
THE CHALLENGE: DATA QUALITY
All the excitement is on AI, but it could all grind to a halt without quality data. “Although it sounds simple, ensuring data quality has been a challenge that has not been easy to solve,” said Viraj Parekh, field CTO at Astronomer. “Every time data is collected, shared, or analyzed, there is potential for its quality to be impacted. As data volumes and sources continue to increase in 2024, prioritizing data quality will be critical.”
The trend: “The emergence and widespread adoption of generative AI has made data quality even more important,” said Parekh. “With increased use of black-box models to make decisions, the door is open for even greater scrutiny and uncertainty around data.”
OpenAI users, for example, “are never going to be able to deeply understand how the model is trained,” he continued. “This significantly impacts an organization’s ability to trust the quality of the data that’s being used to feed the model, as well as the quality of the data they receive as an output.”
The solution: Tools on the market include features for profiling, cleansing, and applying sanity checks to data. “Regardless of which tool they choose, data quality has to be baked into the process of orchestration platforms,” said Parekh. “Since data is constantly being moved through data orchestration, those platforms act as the interface for data governance, observability, and quality and ultimately are the interface to ensure that only good data is fed to dashboards, AI applications, and other data products. To mitigate data quality challenges, enterprises should prioritize their data orchestration strategy in the year ahead and beyond.”