Data engineer jobs: Hadoop and Scala are gone, dbt and governance took their place
Part of a 14-post series on IT jobs in the AI era. Start with the overview: IT job market 2022 to 2026: openings fell, jobs held.
In the six months around ChatGPT's launch, 17% of data engineer, analyst and scientist job descriptions asked for Scala and 14% named Hadoop or its ecosystem. In May to October 2026, Scala appears in 5% and Hadoop in 4% [job-page study, 2026]. Over the same period dbt went from 10% to 28% and data quality and governance from 20% to 37%. I counted 241 company job pages from October 2022 to March 2023 and 541 from 2026, from the Greenhouse, Lever and Ashby boards tech employers use. I left out every role with AI or ML in its title and removed AI vocabulary from the text, so these numbers describe the data job that existed before the AI boom.
What dropped out of the data job ad
Source: Mustard Seed Solutions job-page study, 241 vs 541 data job pages without AI/ML titles; 2026 shares re-weighted to 2022-23 text length.
Most of what left is the 2022 big-data toolkit. Besides Scala and Hadoop, Java fell from 17% to 6% of data job pages, NoSQL from 11% to 1.5%, Kafka from 16% to 7% and stream processing from 13% to 6% [job-page study, 2026]. Generic "ETL" went from 30% to 21% and Redshift from 17% to 9%. AWS fell from 36% to 20%, while Google Cloud and Azure held between 14% and 16%, so the ads name a specific cloud less often without moving to a rival.
These falls hold inside the same employers: among companies that posted data roles in both periods, 19 of the 20 biggest skill moves point the same way as in the full sample [job-page study, 2026]. The Hacker News check is weaker here, because the data posts there are few (70 and 40), and it agrees on 11 of the 20.
Unlike JavaScript in developer ads, Scala is also used less. On GitHub, Scala's share of developers' language activity fell from 0.21% to 0.12% in Europe and from 0.18% to 0.10% in the US between late 2022 and late 2025 [GitHub Innovation Graph, 2025]. In Poland it went from 0.23% to 0.12%, and the number of Polish developers pushing Scala code fell from 674 to 506. Stack Overflow's professional developers report the same direction, 3.0% using Scala in 2022 and 2.2% in 2026 [Stack Overflow survey, 2022-2026].
Data engineering was changing fast before ChatGPT as well: Lightcast's Skill Disruption Index found it was the occupation whose skills changed most between 2016 and 2021, out of 680 it tracked [Lightcast, 2022].
What replaced it: the warehouse, dbt and data quality
Source: Mustard Seed Solutions job-page study, same samples as above.
dbt rose from 10% to 28% of data job pages and "analytics engineering" from 2% to 12% [job-page study, 2026]. Data modelling went from 26% to 36% and A/B testing and experimentation from 10% to 20%. On storage, Databricks and the open table formats around it went from 12% to 25% and BigQuery from 11% to 18%, while Snowflake held at 22% to 23%. Airflow went from 20% to 25%. LinkedIn's 2026 list of fastest-growing skills in US IT puts dbt at number 8 and Databricks at number 9 [LinkedIn, 2026].
The biggest risers are about running data in production, the same shift I measured in software developer ads. "Scalability" went from 37% to 59% of data pages, data quality and governance from 20% to 37%, monitoring from 13% to 22%, "reliability" from 6% to 26% and observability from 2% to 11%.
The basics did not move. SQL sat in 78% of data pages in 2022-23 and 81% in 2026; Python stayed at 80% [job-page study, 2026].
Source: Mustard Seed Solutions job-page study; terms below 5% of pages drop out of their box. AI row: share of the same pages that mention AI anywhere.
Fewer juniors, more stakeholders
Data ads ask for more experience. Across all data job pages, the median minimum experience asked rose from three years to five [job-page study, 2026]. Among roles without AI titles, ads open to one or two years of experience fell from 22% to 11%, and ads asking for eight or more years rose from 2.5% to 8%. Degree asks moved little: a bachelor's degree is mentioned in 31.5% of 2022-23 pages and 28% of 2026 pages, a master's in 16% and 19%. Indeed reported a longer fall in the UK, where postings in mathematics, which includes data science, requiring at least a bachelor's went from 35.3% in the first half of 2018 to 28.3% in the first half of 2024 [Indeed Hiring Lab, 2024].
The soft requirements changed more than the formal ones. "Stakeholders" appears in 51% of 2026 data pages, up from 39%. "Ownership" went from 24.5% to 38%, "business acumen" from 7.5% to 16% and "judgment" from 2% to 10.5% [job-page study, 2026]. "Passion" fell from 40% to 27%.
The job titles moved, and AI moved into the ad
Across all data job pages, roles with AI or ML in the title went from 30% to 58% [job-page study, 2026]. Among the rest, data engineer titles went from 43% to 37% of pages, data analyst from 26% to 21%, data scientist from 26% to 32% and analytics engineer from 5% to 11%.
AI vocabulary is in almost every data ad now. Across the whole data family, 57% of job pages mentioned AI somewhere in 2022-23 and 91% in 2026, and AI in the requirements section went from 42% to 75% [job-page study, 2026]. Even among roles without AI in the title, the share of pages mentioning AI went from 43% to 82%.
Europe is thin in this sample, with 27 European data pages in 2022-23, so I cannot give European shares for these terms. GitHub is the best European signal I have: besides the Scala decline, Python's share of developer activity rose from 7.3% to 8.9% in Europe and from 6.9% to 8.7% in Poland between late 2022 and late 2025 [GitHub Innovation Graph, 2025].
What this means if you are a data engineer
- If your CV leads with Hadoop, Hive, Scala or Kafka, move them down and lead with the warehouse work. dbt (28%), Databricks (25%) and data modelling (36%) are what 2026 ads ask for, next to SQL and Python at around 80%.
- Show that you run data as a service. Name the data-quality checks, monitoring and incident handling you own, with a number if you have one. Data quality and governance (37%) and "reliability" (26%) are now written into the ad.
- Show who used your data. Half of 2026 data ads mention stakeholders, and the median minimum experience they ask for is now five years. One example of a business decision your model or pipeline changed shows both the experience and the stakeholder work.