What's in Store for Data Teams in 2023?
Here’s what three dozen industry practitioners had to say.
Erik Edelmann
December 14, 2022
|18 minutes
What is December for if not for reflection, goal-setting, and a little good-natured crystal balling?! “Next year prediction” posts are pretty standard this time of year, but we’re doing ours with a twist: rather than bore you with our own predictions, we reached out to three dozen data practitioners and leaders from around the community in hopes of drafting a list of "23 predictions for 2023."
Well...we asked, and you all delivered! Below is a list of over 35 (!) predictions going into the new year as sourced from the community 🙌. Some high-level themes definitely emerged, so we have organized the list accordingly.
The three broader themes we identified were The Economy, Teams and Culture, and Tools and Trends, each with some common threads. A heartfelt “thank you” to all of the folks featured in the post who provided their pointed views on how next year might shape up. We hope this curated list gets you as excited for what’s to come in data in 2023 as it does us!
Theme #1: Economic Headwinds Come for the Data Industry
A fair number of the predictions we gathered speculated about the impact of the changing economic climate on the world of data. Some were more optimistic, others were ominous, but all predicted changes in how data is seen or done in response to the shifting economic climate. Below are our collected predictions relating to the economy and how it will impact data in 2023.
Natural Foreboding
First up, many of us were thinking it, and these folks said it out loud. The tightening we’re seeing in the market will translate directly to pressure in data, too. Scroll through the quotes below to get folks’ take.
“...Technology leaders are going to have to manage a major inflection point with this recession…Budgets/hiring are going to be scrutinized. If your projects don't have clearly defined business outcomes that impact the top or bottom line, they aren't going to get off the ground. Now is the time to engage with the business for both funding and guidance to make sure both tech leaders and business leaders are driving towards the same end goals.”
John Iwanski
Regional Account Director
, phData
“The good times for data teams and the modern data stack are, unfortunately, over. The focus for businesses in 2023 will be firstly, staying alive and saving money, and then a far second will be growth in the business, data teams, and budgets. Expect the focus for data analytics to be on business efficiencies, cost reduction and profitability, and increasing team productivity through AI and data-driven smart workflows.”
Mark Rittman
CEO
, Rittman Analytics
“...Technology leaders are going to have to manage a major inflection point with this recession…Budgets/hiring are going to be scrutinized. If your projects don't have clearly defined business outcomes that impact the top or bottom line, they aren't going to get off the ground. Now is the time to engage with the business for both funding and guidance to make sure both tech leaders and business leaders are driving towards the same end goals.”
John Iwanski
Regional Account Director
, phData
“The good times for data teams and the modern data stack are, unfortunately, over. The focus for businesses in 2023 will be firstly, staying alive and saving money, and then a far second will be growth in the business, data teams, and budgets. Expect the focus for data analytics to be on business efficiencies, cost reduction and profitability, and increasing team productivity through AI and data-driven smart workflows.”
Mark Rittman
CEO
, Rittman Analytics
Consistent Optimism
Others, however, have a more upbeat take on how market conditions will impact data.
“In 2023, companies will continue pushing to be more data-driven. A slowing economy will mean a shift to data projects focusing on financial resiliency and cost mitigation [not a de-emphasis on data in general]. Companies that have already built a strong data culture will be able to do so more effectively.”
Georgi Tasev
Sr. Data Scientist
, Schneider National Logistics
“Data teams will become points of increasingly high leverage for businesses that invest in them correctly. As economic conditions change around us, businesses will look to ground themselves in numbers that help them become more efficient. Data teams already hold incredibly rich context about the operation of their business, and are well positioned to take advantage of the opportunity to lead business transformation narratives.”
Anna Filippova
Director of Community
, dbt Labs
“Budget constraints won’t devalue data. Companies will need to lean heavily into their data, their most valuable asset. Data will help them uncover (in)efficiencies as well as provide valuable insights on where to prioritize resources for the greatest impact.”
Dayna Shoemaker
Sr. Manager, Enterprise Product Marketing
, Fivetran
“In 2023, companies will continue pushing to be more data-driven. A slowing economy will mean a shift to data projects focusing on financial resiliency and cost mitigation [not a de-emphasis on data in general]. Companies that have already built a strong data culture will be able to do so more effectively.”
Georgi Tasev
Sr. Data Scientist
, Schneider National Logistics
“Data teams will become points of increasingly high leverage for businesses that invest in them correctly. As economic conditions change around us, businesses will look to ground themselves in numbers that help them become more efficient. Data teams already hold incredibly rich context about the operation of their business, and are well positioned to take advantage of the opportunity to lead business transformation narratives.”
Anna Filippova
Director of Community
, dbt Labs
Focus on Business Value
Others have more neutral views on this theme, but still predict an increased focus on the ROI of data projects and functions, or an increased willingness to use lighter-weight, cost-effective solutions in response to this new focus on ROI.
“With the tightening of budgets being seen across the board due to market conditions, I predict that within smaller organizations we will see companies rely less on self-service tools and more on people doing manual data pulls from their data warehouse. This could mean an increase of either open source tools (if orgs have technical teams to do so) or relying more on Excel or Google Sheets to do more of their reporting.”
Lorena Vazquez
Associate Director, Business Analytics
, Wonder
“Data teams are going start focusing on OPEX and profitability more than in 2021/2022, whether this means controlling their spend on cloud and/or focusing more on their business rather than spending as much time on infrastructure and engineering. The business is going to lean on the data team to help them squeeze out efficiencies, to grow in a sustainable way.”
David Jayatillake
Head of Data
, Metaplane
“Teams and companies will become increasingly interested in the ROI of data projects. As the recession takes effect and budgets tighten, projects without clear purpose will struggle for funding. Data teams can help themselves by learning to measure the costs of projects and the projected value. Data teams that have this awareness will also naturally seek to get more value out of existing data products through evangelization, education, user support, and investments in reliability.”
Gordon Wong
Principle Consultant & Founder
, Wong Decision Intelligence
“With the tightening of budgets being seen across the board due to market conditions, I predict that within smaller organizations we will see companies rely less on self-service tools and more on people doing manual data pulls from their data warehouse. This could mean an increase of either open source tools (if orgs have technical teams to do so) or relying more on Excel or Google Sheets to do more of their reporting.”
Lorena Vazquez
Associate Director, Business Analytics
, Wonder
“Data teams are going start focusing on OPEX and profitability more than in 2021/2022, whether this means controlling their spend on cloud and/or focusing more on their business rather than spending as much time on infrastructure and engineering. The business is going to lean on the data team to help them squeeze out efficiencies, to grow in a sustainable way.”
David Jayatillake
Head of Data
, Metaplane
Theme #2: Data Teams And Culture Will Begin to Look Different
Even more predictions were around how data teams fit into and add value within their organizations. This isn’t totally surprising given the proliferation of conversations about things like “Data Contracts” around the data community. Some recurring themes included a new focus on data “process”, changes to data career paths and roles, and shifts in how data practitioners do their work.
New Process to Complement New Tech
Several predictions centered around the idea that even the “perfect” technology stack can’t fulfill its potential without proper business processes.
“Companies will become less fixated on implementing the “perfect” stack and develop a healthier data culture of evaluating and defining their business challenges and objectives before they design and build technical solutions that are meant to address them.”
Rhys Berkwitt
Data Strategy Manager
, Data Culture
“When I think of data maturity, I think the three main factors are people, process, and tools. Investments in people and tools have come along way in just the last five years…but I still hear a lot of questions around process. Data teams are unique in that they serve the entire business but often get housed in a specific department…We are still learning what works and what doesn’t, and so I think we will see an increased focus around developing and defining data team processes in 2023.”
Kelly Burdine
Director of Data Science & Analytics
, Wellthy
“We’ll start to witness more discussion around the people and process side of things (as opposed to technology and techniques). At this point, the high-performing data teams have nailed the fundamentals: they have reliable data models, their data consumers are able to self-service, and their data is operationalized. As these teams continue to climb the analytical maturity ladder via experimentation and predictive modeling, they’ll learn to navigate the friction associated with poor process. They’ll learn the value of being intentional and deliberate upfront when it comes to planning cross-functional initiatives. Data professionals at all levels will find themselves stepping up as leaders and influencing their organizations to standardize processes.”
Michelle Ballen-Griffin
Head of Data Analytics
, Future
“Companies will become less fixated on implementing the “perfect” stack and develop a healthier data culture of evaluating and defining their business challenges and objectives before they design and build technical solutions that are meant to address them.”
Rhys Berkwitt
Data Strategy Manager
, Data Culture
“When I think of data maturity, I think the three main factors are people, process, and tools. Investments in people and tools have come along way in just the last five years…but I still hear a lot of questions around process. Data teams are unique in that they serve the entire business but often get housed in a specific department…We are still learning what works and what doesn’t, and so I think we will see an increased focus around developing and defining data team processes in 2023.”
Kelly Burdine
Director of Data Science & Analytics
, Wellthy
Data-to-Business SLAs
Some others touched on how data teams work with their stakeholders, foretelling that 2023 will see a new emphasis on the SLA between data teams and their business stakeholders.
“This will be the year companies see data as more than just an internal resource; they will start thinking of it as an actual product, adding context and actionability for their partners, customers, and vendors.”
Dan Goldstein
Sales Manager, Data Analytics
“Companies will start realizing when metrics are “good enough” when directionally accurate but not precise, and that will allow data team members to be more effective.”
Emilie Schario
CEO
, Turbine
“Data teams will increasingly understand, and make strides towards helping their organizations and stakeholders understand, that data governance is not about numbers being 'right' or 'wrong', but about numbers being understood.”
Teresa Kovich
Principal Consultant
, DAS42
“This will be the year companies see data as more than just an internal resource; they will start thinking of it as an actual product, adding context and actionability for their partners, customers, and vendors.”
Dan Goldstein
Sales Manager, Data Analytics
“Companies will start realizing when metrics are “good enough” when directionally accurate but not precise, and that will allow data team members to be more effective.”
Emilie Schario
CEO
, Turbine
Workflows And Expectations
Others thought there would be some shifts in the technical aspects of the data practitioner’s workflow, or changes to expectations around skillsets and deliverables.
“... As analytics engineering continues to make technical and professional strides, I think data engineers will become more focused on infrastructure and less on data. I think it'll be important in the field for data engineers to learn more about AWS [and other cloud] services, containers, and software engineering paradigms.”
Alisa Aylward
Principal Data Engineer, Technical Design Lead
, Toast
“We’ll see evolution in the data development lifecycle. As companies build business-critical and customer-facing functionality on top of their data warehouses, data developers (data engineers, analytics engineers, etc.) will be held to the same development lifecycle standards as software developers. [e.g. version control, CI, etc.]”
Rachel Bradley-Haas
Co-Founder
, Big Time Data
“Data teams will own and execute on more projects and workflows that were traditionally done by software engineers. Given the advanced capabilities of many data platforms on the market, it is now easier than ever for data teams to own projects that action on their firm’s data. The innovation of cloud platforms puts data teams in a position to own not just their traditional responsibilities, but also some of the business workflows data enables.”
Laura McKinley
Principal Consultant
, DAS42
“... As analytics engineering continues to make technical and professional strides, I think data engineers will become more focused on infrastructure and less on data. I think it'll be important in the field for data engineers to learn more about AWS [and other cloud] services, containers, and software engineering paradigms.”
Alisa Aylward
Principal Data Engineer, Technical Design Lead
, Toast
“We’ll see evolution in the data development lifecycle. As companies build business-critical and customer-facing functionality on top of their data warehouses, data developers (data engineers, analytics engineers, etc.) will be held to the same development lifecycle standards as software developers. [e.g. version control, CI, etc.]”
Rachel Bradley-Haas
Co-Founder
, Big Time Data
Roles and Responsibilities
To round out the Team and Culture theme, we received contributions that speak to the nature of the roles on data teams and the way business responsibilities around them might evolve.
“The creation and adoption of enterprise data “movement” tools have replaced the need for traditional data engineers, or at least the need for large teams of data engineers. Many of these traditional data engineers then pivoted to other specific enterprise data needs that were still more manual…what’s coming next is automating modeling, going from source data straight into a business metric…so analytics engineers will go the way of traditional data engineers, either with a decrease of analytics engineering positions (especially embedded AEs vs. a centralized enterprise data team) or a pivot into a new role entirely.”
Brian Pei
Analytics Engineer
, Spotify
“...I think there will be a wider audience for certain data tools (dbt, Hightouch, etc.) within a company. More and more non-data team members will be more closely interacting with data tools to prevent bottlenecks and increase efficiency. With a strong set of best practices, solid governance, and robust testing in place, teams can enable other technical (but not data-team) teammates to work more quickly and confidently.”
Schylar Brock
Sr. Analytics Engineering Lead
, Vendr
“The creation and adoption of enterprise data “movement” tools have replaced the need for traditional data engineers, or at least the need for large teams of data engineers. Many of these traditional data engineers then pivoted to other specific enterprise data needs that were still more manual…what’s coming next is automating modeling, going from source data straight into a business metric…so analytics engineers will go the way of traditional data engineers, either with a decrease of analytics engineering positions (especially embedded AEs vs. a centralized enterprise data team) or a pivot into a new role entirely.”
Brian Pei
Analytics Engineer
, Spotify
“...I think there will be a wider audience for certain data tools (dbt, Hightouch, etc.) within a company. More and more non-data team members will be more closely interacting with data tools to prevent bottlenecks and increase efficiency. With a strong set of best practices, solid governance, and robust testing in place, teams can enable other technical (but not data-team) teammates to work more quickly and confidently.”
Schylar Brock
Sr. Analytics Engineering Lead
, Vendr
Theme #3: Speculation on Tools and Trends
Another broad theme we’ve pulled out from our collections is generalized into “tools and trends” in the modern data stack. No predictions list would be complete without it 🤓. These vary a good bit, but we’ve also identified some sub-themes here, from CDPs to the metrics layer to data observability.
The Modern CDP
One prominent theme is the increasing importance of centralizing customer data in the data warehouse, and the rise of the “Modern CDP” or “composable CDP": a warehouse-driven version of what was previously done through traditional customer data platforms. MarTech, in general, seems to be on our contributors’ minds, including education about the technologies emerging in the space.
“The value proposition of a data clean room for customer insights and program measurement is now well understood. As clean room adoption continues to accelerate, marketers are looking for the ability to bring together planning, measurement, and targeting capabilities. They will demand that clean rooms have better connectivity to multiple activation platforms to create more meaningful experiences at scale.”
Craig Howard
Chief Solutions Officer
, Actable
“The composable solutions that will win are likely the ones that align with other partners to package their composable stories together. Of those marketers that stay with a CDP, they will demand that the platform has interoperability/integrations with cloud solutions. The CDPs that lack an integration with cloud solutions will lose market share.”
David Wells
Business Development
, Snowflake
“As organizations progress along the data maturity curve, they will increasingly look to combine internal data with external sources of data to get a holistic view of their market and customers. Enriching the centralized data will be top of mind, and companies will find easier ways to obtain and integrate quality partner and third-party data (e.g., from the Snowflake data marketplace) to be better poised for strategic, data-backed decisions.”
Onkita Ganguly
Data and Analytics Manager
, Brooklyn Data Co.
“In 2023, interoperability will take center stage as the composability of the MarTech stack evolves from CDPs to data clean rooms and beyond.”
Dan Morris
Sr. Director, Industry Solutions
, Databricks
“With the continued difficulty of finding MarTech talent in the APAC region, we see 2023 being about building internal capability. How can brands continue to build their marketers’ technical skills and bridge the gaps between marketing, digital, and tech? Finding, retaining, and growing these people will be key to obtaining or maintaining competitive advantage. We’re seeing less of “can you implement this technology for us”, but instead, “can you help my team run this technology ongoing?” As such, if you have marketers who understand both the business and technology – hold onto them.”
Phil Wild
Sr. Advisory Consultant
, The Lumery
“The value proposition of a data clean room for customer insights and program measurement is now well understood. As clean room adoption continues to accelerate, marketers are looking for the ability to bring together planning, measurement, and targeting capabilities. They will demand that clean rooms have better connectivity to multiple activation platforms to create more meaningful experiences at scale.”
Craig Howard
Chief Solutions Officer
, Actable
“The composable solutions that will win are likely the ones that align with other partners to package their composable stories together. Of those marketers that stay with a CDP, they will demand that the platform has interoperability/integrations with cloud solutions. The CDPs that lack an integration with cloud solutions will lose market share.”
David Wells
Business Development
, Snowflake
The Metrics Layer
What good would a 2023 look ahead piece about data be without references to “the metrics layer”? Some of our collected predictions touched on this increasingly central topic in communal data discourse.
“As analytics teams get leaner, and their stakeholders need to dig deeper to drive additional value in a tougher economic climate, I’d expect to see even greater pushes toward efficiencies and collaboration and the rise of the metrics store. Implementing a metrics store, where teams can publish standardized logic for key business indicators, can simplify knowledge sharing across distributed analytics teams, reduce swirl in reporting at the exec level, and enable more self-service even among business users with some data know-how.”
Mary Alfheim
Head of Product Analytics, Prime Video
, Amazon
“I’m betting on the idea of the semantic layer, even though I’m not sure which framework or tool will win in the end. So what does a BI tool built for the semantic layer look like? Will the idea of “headless BI” (i.e. one tool for the semantic layer and a different one for BI) win out? I’m not convinced. I’m betting that next-gen BI tools will have their own charting capabilities, but will surface great APIs and database connectors so other visualization tools can query the semantic layer easily. From my vantage point, if governance and modeling is “solved” by the semantic layer, the best BI tools will excel at answering questions quickly.”
Josh Temple
Co-Founder
, Spectacles
“As analytics teams get leaner, and their stakeholders need to dig deeper to drive additional value in a tougher economic climate, I’d expect to see even greater pushes toward efficiencies and collaboration and the rise of the metrics store. Implementing a metrics store, where teams can publish standardized logic for key business indicators, can simplify knowledge sharing across distributed analytics teams, reduce swirl in reporting at the exec level, and enable more self-service even among business users with some data know-how.”
Mary Alfheim
Head of Product Analytics, Prime Video
, Amazon
“I’m betting on the idea of the semantic layer, even though I’m not sure which framework or tool will win in the end. So what does a BI tool built for the semantic layer look like? Will the idea of “headless BI” (i.e. one tool for the semantic layer and a different one for BI) win out? I’m not convinced. I’m betting that next-gen BI tools will have their own charting capabilities, but will surface great APIs and database connectors so other visualization tools can query the semantic layer easily. From my vantage point, if governance and modeling is “solved” by the semantic layer, the best BI tools will excel at answering questions quickly.”
Josh Temple
Co-Founder
, Spectacles
Data Observability
Similarly timely were the predictions about “data observability”, a newer category of tooling growing lately in response to increasingly business-critical workloads being driven by cloud data warehouses.
“Data observability is becoming more and more relevant for companies. Particularly if it can be automated in anyway. Companies are realizing that they can't foretell all the testing they need to write into test coverage, and they don't have the headcount/budget/time to create the monitoring and visibility needed for their stack.”
Callie White
Sr. Analytics Consultant
, Montreal Analytics
“2023 will be the year of data quality. Organizations have adopted modern data stacks and self-service analytics but are still plagued with data downtime. We’ll see increased adoption of data observability platforms as success stories spread through the industry.”
Joey Bryan
Product Manager
, Monte Carlo
“Data observability is becoming more and more relevant for companies. Particularly if it can be automated in anyway. Companies are realizing that they can't foretell all the testing they need to write into test coverage, and they don't have the headcount/budget/time to create the monitoring and visibility needed for their stack.”
Callie White
Sr. Analytics Consultant
, Montreal Analytics
“2023 will be the year of data quality. Organizations have adopted modern data stacks and self-service analytics but are still plagued with data downtime. We’ll see increased adoption of data observability platforms as success stories spread through the industry.”
Joey Bryan
Product Manager
, Monte Carlo
Security, Governance, and Privacy
Just as some of our contributors see the emergence of data observability as a big trend that will support more important workloads being driven by data, others predict a renewed emphasis on security, governance, and privacy in data.
“...In 2023 and beyond, organizations will be asking: How can we allow marketers access to the data they need while maintaining the right security standards? We need to enable marketers to do their jobs, but we also want to give them guardrails to work within. When those guardrails exist, it means marketers are spending less time wrangling data, and more time delivering value to the customer.”
Simon O'Day
Director of Vendor Partnerships
, The Lumery
“...Data privacy and governance are going to become even more important as the US trends more towards GDPR-type regulation. We've already seen this with California (CCPA), but more and more states are following with their own data privacy laws in 2023.”
Emily Hawkins
Data Engineering Manager, Data Platform
, Drizly
“I think data management and curation will become increasingly important as large models proliferate. Enterprises will increasingly look to software to uncover, curate, and label new sources of data to fine-tune models to their specific domains, driving the last mile of getting these large models into useful production circumstances.”
Rak Garg
Principal
, Bain Capital Ventures
“In 2023, we believe businesses will no longer be able to capture data without telling their users why and delivering near-immediate value in return. For example: using browsing history to deliver more relevant communications across channels (email, push, SMS, in-app), using purchase behavior to inform the next offer, using engagement data to decide which channels to communicate on…all this at an individual user level.”
Thierry Sequiera
CEO
, Massive Rocket
“...In 2023 and beyond, organizations will be asking: How can we allow marketers access to the data they need while maintaining the right security standards? We need to enable marketers to do their jobs, but we also want to give them guardrails to work within. When those guardrails exist, it means marketers are spending less time wrangling data, and more time delivering value to the customer.”
Simon O'Day
Director of Vendor Partnerships
, The Lumery
“...Data privacy and governance are going to become even more important as the US trends more towards GDPR-type regulation. We've already seen this with California (CCPA), but more and more states are following with their own data privacy laws in 2023.”
Emily Hawkins
Data Engineering Manager, Data Platform
, Drizly
Bonus Predictions!
Some of the predictions we gathered were tough to categorize! Here are some of the more unique predictions, ranging from niche technical trends to the cloud wars.
“I expect the average data engineer's blood pressure will drop by 0.2mmHg as better-tailored storage frameworks, formats, and compression gain traction. I don't profess to know whether all of these will soar in the coming year, but surely some will: Delta Lake, Iceberg, Deep Lake, Quantile Compression. These are already on the rise, but I believe they are still considerably undervalued.”
Martin Loncaric
Research Infrastructure Engineer
, Jane Street
“We may see the return of the data cube! The OLAP cube was a critical piece of the last generation of enterprise data tools. Data teams will increasingly see version-controlled, curated datasets as their output, and allow business analysts to manage their own metrics and reporting.”
Leah Weiss
CEO, Co-Founder
, Preql
“A new generation of distributed data warehouse platforms are coming online and will, this year, start gaining traction. Some of these are true CDWs offered as SaaS platforms, some intended to run in a data center or a self-managed solution. Their attributes are all very similar in that they bring extreme efficiency, ease of use, and extensive features. Because of this, the use of specialized high-performance data solutions will be reduced dramatically. Data pipelines will get shorter and simpler, and the utilization of summary generation tools will be reduced as they're simply not as necessary. As a side effect, SQL will bolster its dominance in the data space.”
Robert Harmon
Solutions Architect
, Firebolt
“As companies spin up streaming (Materialize, Tinybird) and event-driven (Zapier, LogicLoop) infrastructure in parallel with existing batch infrastructure, it becomes harder and more important to differentiate where a source of truth comes from.”
Ian Macomber
Head of Analytics Engineering & Data Science
, Ramp
“I believe GCP is finally positioned with its data suite to compete in a very real way with Amazon and Microsoft, as well as BigQuery against Snowflake. The consolidation of the analytics brand under Looker and the parity BigQuery now has with Snowflake is more obvious to consumers (if anything, BigQuery has an edge with BQML, integrations with Google Sheets, and a more favorable pricing model). It’s taken time for Google to make these investments and get the leadership in place - but I think 2023 is likey the year it comes together.”
Brien Bernstein
Sales Leader
, Retool
“I expect the average data engineer's blood pressure will drop by 0.2mmHg as better-tailored storage frameworks, formats, and compression gain traction. I don't profess to know whether all of these will soar in the coming year, but surely some will: Delta Lake, Iceberg, Deep Lake, Quantile Compression. These are already on the rise, but I believe they are still considerably undervalued.”
Martin Loncaric
Research Infrastructure Engineer
, Jane Street
“We may see the return of the data cube! The OLAP cube was a critical piece of the last generation of enterprise data tools. Data teams will increasingly see version-controlled, curated datasets as their output, and allow business analysts to manage their own metrics and reporting.”
Leah Weiss
CEO, Co-Founder
, Preql
Conclusion
Thank you to all of the practitioners and community members who helped craft this crystal ball. We couldn’t be more excited for the year ahead, and for all of the work and innovation the community will drive that may turn some of these predictions into reality. Wishing you and yours a joyful holiday season and looking forward to what the new year has in store!
