What’s next for the analytics engineer? - reflections from Coalesce 2022

Hightouch at dbt Coalesce

I watched Coalesce online and–even at home, more than a thousand miles from New Orleans–I felt the energy. Almost seven years ago, Fishtown Analytics created a command line tool they called dbt (data build tool). Today, dbt is more than just a command line tool. It’s become a movement. Sometimes I joke that I talk about dbt more than anyone who is not on the dbt Labs payroll, but a lot of us probably feel that way. That’s what makes it a movement.

Four years ago, I started working with dbt and since then I have trained, presented, and introduced the tool to hundreds of people. So I felt like I was part of something important when I watched the auditorium fill up for last week’s keynote at the first-ever in-person dbt Coalesce conference. I can only imagine what it must have felt like to be in that crowd waiting for the lights to come on. Just seeing it online was visually overwhelming. It took me a few minutes to digest the enormity of the moment.

It was an overdue moment. Coalesce was originally scheduled to happen in 2020, long before the dbt community amassed the following it has today. If the pandemic hadn’t forced its cancellation, the first Coalesce would have had a smaller audience, likely the people who were actually using dbt and maybe some of their managers.

Last week when dbt finally got to have its in-person celebration, the “Analytics Engineering Conference” of 2022 featured more than just analytics engineers. There were over two hundred presenters, including directors, founders, and CEOs. It was an overwhelming representation of thought leadership among a burgeoning community of modern data practitioners.

Unfortunately, as a virtual attendee, I felt that the intensity with which Coalesce started kind of … dissipated after the keynote. I didn’t get to experience the magic that in-person attendees posted about during the week: the feeling of being back in college, and the excitement of finally meeting three-dimensional versions of the Slack avatars we know so well.

I’ve attended many conferences in my career, so I know that the value of the content doesn’t always materialize fully within the sessions–but later, in the conversations about the sessions. It’s about finding people you identify with, who have seen and done things outside your realm of experience, and then processing it all together. That’s why we go to conferences.

I watched a lot of the presentations at their scheduled time, mainly because the dbt Labs team did such a phenomenal job of delivering content to remote attendees. We could see the slides and presenters clearly, and there was a Slack channel for every talk where we could ask questions and react to the material in real-time. If I had been at the conference physically, I would not have attended as many talks as I did. I might have watched the videos, but again, not as many as I did. Silver linings.

The Talks at Coalesce

The Talks at Coalesce

There were over a hundred talks at Coalesce and some difficult decisions to make because multiple talks were scheduled in a single time slot. The dbt community really showed up to deliver high-quality content!

Here are a few that stood out to me by category:

Most Relatable

• From worst to first, revamping your dbt project to be world-class (Kelly Burdine & Michelle Ballen)
• Buried Alive: Refactoring an inherited project (Christine Berger & Lauren Benezra)

Phew! I’m not the only one who has spent a significant chunk of my analytics career stuck in the world of tech debt. Talks like these are important because they give everyone a glimpse into the side of analytics engineering that is not super glamorous, but a foundational skill that you absolutely have to build.

I Wish I Was at Coalesce for This

• Babies and bathwater: Is Kimball still relevant? (Sydney Burns & Josh Devlin)

Data modeling technique has become an “agree to disagree” situation among the analytics engineering crowd. Kimball’s method dates back to 1996 so it doesn’t feel like a natural fit in the modern data stack. Or does it? This talk explained why dimensional modeling is still relevant today–well, at least until something better comes along to replace it.

Most Likely to Recommend to Non-Data Practioners

• How not to be a terrible writer (Justin Gage)

The topic of writing is near and dear to my heart. As data practitioners, I believe we can learn a lot about how to deliver compelling analytics when we consider the art and process behind storytelling. This talk had some really substantial writing advice that can apply far beyond the scope of analytics and the audience of Coalesce.

My Own Journey, Post-Analytics Engineer

My Own Journey, Post-Analytics Engineer

I became an analytics engineer in 2019. Since then, the role has exploded, both in numbers and demand, which we saw in the massive turnout at Coalesce last week. Reflecting on my job experiences though, I can’t quite reconcile the enthusiasm I witnessed at the conference with my own day-to-day responsibilities working as an analytics engineer.

For me, the role felt too tightly scoped to using dbt to prepare data for analytics, without having any insight into whether my work created real value. To expand on the tasks I was assigned, I had to reach into an area that already belonged to a different role on the team.

In an established organization, most teams agree that the data warehouse administration and pipelines belong to data engineers, and that data analysts own the BI tools and collaborate with the business stakeholders to define key metrics.

The analytics engineer lies somewhere in the middle, focusing on the “T” of the ELT (extract, load, transform) process. This includes shaping the data and resolving inconsistencies. In my experience, transformation work can become repetitive and can feel like a somewhat low-profile task without reach outside of the data team. While the work is important, there’s not a lot of room for skill development once you achieve competence with dbt.

I never saw the role as a permanent fit but rather an entry point into working in the modern data stack. So while the data community gathered in New Orleans, I began a new journey at Hightouch, as a data advocate. I’m excited about the advocate role because I’ll be able to draw from my previous engineering background (from the pre-modern data stack days) and explore modern data team challenges with a broader and more altruistic perspective, rather than being limited to one organization’s specific use case.

Today's Analytics Engineer

Today's Analytics Engineer

Analytics engineering has become a staple of most data teams, somehow inextricably linked to our use of dbt as a product. There’s a lot of hype around the job, but much of that hype comes from outside the role itself. Within the data community, a small portion of the thought leadership that is talking about analytics engineering actually holds the title of analytics engineer.

Analytics engineering is still a new and relatively accessible discipline. Most companies expect a baseline of SQL experience to become an entry-level analytics engineer, with the expectation that a person will learn dbt on the job. Some job postings also mention data modeling, but there’s no widespread consensus about the right technique for modeling data within cloud data warehouses. Recently I’ve observed that the use of dbt and the technique of data modeling have become somewhat interchangeable, as though being the product of a dbt run inherently makes something a model. This is an area I’m eager to explore from an advocate’s perspective.

The day-to-day tasks you can expect when you become an analytics engineer are hard to classify because so much depends on the organization and team. Some analytics engineers are tasked with implementing a new project from the ground up while others are tasked with unraveling tech debt from an inventory of spreadsheets and messy queries. Both use dbt, but it’s not really the same job at all.

Looking to the Future

Looking to the Future

There are a lot of resources that describe how to become an analytics engineer and how to advance your dbt skills, but not as many about how to navigate your career once you become an analytics engineer. It’s an area I expect to expand post-Coalesce, as more analytics engineers increase their tenure in the role and as more and more data practitioners migrate into the modern data stack.

There are many possibilities for the evolution of the role. My guess is that we will see different variations of analytics engineers as the people doing the job shape the future. Because of my own path, I can see analytics engineers moving into either a more technical role, like a data engineer, or into a leadership role with more influence. I can also imagine the scope of the analytics engineer growing as we further refine data modeling practices for cloud warehouses and expand the ways we use our data models to drive business action.

In my new role as a data advocate at Hightouch, I’m excited to delve into these topics with the community and plan to share my learnings, resources, and tips along the way.