Though the AI era conjures a futuristic, tech-advanced image of the present, AI fundamentally depends on the same data standards that have been around forever. These data standards—such as being clean, accessible, and processed efficiently—are assured through data engineering, a crucial role that helps transform data into valuable insights. Staying up to date with the latest evolutions of data engineering is critical to continue to feed quality data into today’s—and tomorrow’s—big tech.
Experts joined DBTA’s webinar, Top Trends in Data Engineering for 2025, to explore the new trends and best practices shaping the future of engineering, from AI-driven tools to real-time data processing, new data architecture patterns, and more.
“Data teams are subjected to this overwhelming amount of toil,” according to Sean Knapp, founder and CEO, Ascend.io. “And data engineering wasn’t always the most enjoyable experience.”
This is echoed by data from a DataAware Pulse Survey that showed that despite 83% of respondents expressing their productivity has improved through AI technology, data teams are still drowning in current workloads:
- ¼ of people on data teams report they are significantly over capacity.
- 54% feel they are somewhat over or significantly over capacity.
- 95% of data practitioners report being at or above their work capacity for a fifth year in a row.
Furthermore, nearly half of all data engineering time is spent on maintenance, which ultimately inhibits their ability to take on new projects. Knapp emphasized that when we introduce new technology such as AI into the business picture, that burden is falling onto data engineers—and without much consideration for their workloads.
It’s no surprise then that productivity is a major focus for data engineers in 2025. Automation is the next frontier for helping data engineering teams meet the increased demand and work around the maintenance burden—but what does data automation look like?
At Ascend, “automation isn’t just about pulling together the verbs of data engineering…we think about it in…three layers,” explained Knapp, which include a unified metadata layer; a DataAware automation layer that uses metadata to power things such as intelligent orchestration, workload optimization, adaptive pipelines, and more; and an AI Agents layer that delivers seamless experiences that close the semantic gap.
“Our mission at Ascend is to make data engineering truly delightful,” said Knapp. “[It’s] the combination of these three that we think are superpowers for teams.”
Marc Lamberti, head of customer education, Astronomer, identified the following as the major data engineering trends for 2025:
- Data operations are moving from analytics to operational use cases.
- Data engineering teams are looking increasingly like traditional software engineering teams.
- Platform teams want standardization but need to get creative to bridge the skill gap for non-DEs.
- Modern data teams build data products, which are now considered a means to an end.
Astronomer, the driving force being Apache Airflow—the standard for data pipelines in a cloud-native world—has been evolving much like data engineering. Growing in popularity over the past four years, Lamberti teased its next powerful evolution: Apache Airflow 3.
Airflow 3.0 is a massive update to the platform, offering new features such as:
- Run anywhere, in any language with remote execution and task execution beyond Python
- Expanded data awareness and GenAI support
- More intuitive and easy to use with historical DAG versioning, modernized UI, extended MLOps support for backfill runs, and task isolation
As a fourth-gen data and workflow integration platform, Nexla is designed to address today’s most pressing data engineering trend: generative AI (GenAI), according to Saket Saurabh, CEO and co-founder, Nexla. This next stage of data engineering is defined by a single word: converged.
“Where Nexla sits today and where the world is headed is in a converged space, where different patterns of working with data and applications together is how businesses ultimately solve their problems,” said Saurabh.
Nexla simplifies these converged integrations with schema-centric abstraction, a method of abstracting different types of sources—such as events, files, SaaS, databases, and more—into a common entity, or a Nexset.
Existing not as another copy of data but as a new virtual entity, Nexset abstracts how data is piped into different target systems by decoupling the data producer and consumer. Nexsets are discoverable and reusable, simplifying the workloads of data engineers who are managing an excess of data and information.
This method of converged integration also serves GenAI use cases, unifying unstructured data, structured data, retrieval-augmented generation (RAG) pipelines, and agentic workflows within Nexsets.
This is only a snippet of DBTA’s latest webinar event. For the full, in-depth conversation of data trends for 2025, you can view an archived version of the webinar here.