Data engineers should be working faster than ever. AI-powered tools promise to automate pipeline optimization, accelerate data integration and handle the repetitive grunt work that has defined the profession for decades.Yet, according to a new survey of 400 senior technology executives by MIT Technology Review Insights in partnership with Snowflake, 77% say their data engineering teams’ workloads are getting heavier, not lighter.The culprit? The very AI tools meant to help are creating a new set of problems.While 83% of organizations have already deployed AI-based data engineering tools, 45% cite integration complexity as a top challenge. Another 38% are struggling with tool sprawl and fragmentation.”Many data engineers are using one tool to collect data, one tool to process data and another to run analytics on that data,” Chris Child, VP of product for data engineering at Snowflake, told VentureBeat. “Using several tools along this data lifecycle introduces complexity, risk and increased infrastructure management, which data engineers can’t afford to take on.”The result is a productivity paradox. AI tools are making individual tasks faster, but the proliferation of disconnected tools is making the overall system more complex to manage. For enterprises racing to deploy AI at scale, this fragmentation represents a critical bottleneck.From SQL queries to LLM pipelines: The daily workflow shiftThe survey found that data engineers spent an average of 19% of their time on AI projects two years ago. Today, that figure has jumped to 37%. Respondents expect it to hit 61% within two years.But what does that shift actually look like in practice?Child offered a concrete example. Previously, if the CFO of a company needed to make forecast predictions, they would tap the data engineer …
