Dopamine Balance Tool
Emotional and psychological reasons to be an Analytics Engineer
Benn spoke about why people want to be Analytics Engineers (AE) last week:
I enjoyed and mostly agreed with the post, but it looks at this topic from the perspective of a Data Analyst (DA of varying flavours Product/Growth/Finance/BI…) whereas I will look at it from the perspective of an AE. I have more affinity towards AE than any other data role.
Here is a key quote from my perspective:
Though analytics engineers aren’t fully removed from business problems and organizational politics, they’re often protected from its messiest edges. Building crisp models and designing efficient DAGs are tasks with well-defined starting and ending points, and lots of space for creativity in between. For an analyst, a job well done is a more convinced executive, an adjusted decision, and lingering doubt about what stones were left unturned. For an analytics engineer, success is a humming system, a clean codebase, and the satisfying tick of dbt jobs completing in your terminal. Despite telling ourselves that exploration and analysis are the reasons we’re here, I think a lot of us, like more traditional engineers, find a lot of satisfaction in the latter.
I’d like to think about this further from a few different angles:
Risk and reward
Politics and impact
Role suitability, progression and experience
Risk and Reward
As primates (and not machines), we’re set up to seek reward. The reward can take many forms but, ultimately, achieving the reward generates a positive hormonal response in our bodies. This response to the action is part of a feedback loop that forms a habit.
There are moral and ethical aspects which I’ve covered here, too. Where some of these aspects are not addressed, and there isn’t alignment of reward to group activity, you can see some pretty raw primate behaviour in the office. This is why office interactions have been described as similar to those of four year old children. Where you take away moral and ethical aspects to behaviour, you are left with reward cycles and who they belong to. This is why you can see territorial, aggressive, underhanded behaviour in workplaces - people feel their reward is at risk from others and they will act to defend it. Even under better circumstances, reward plays a big part in whether a role is satisfying. You can work for a fantastic company but still be very unsatisfied if the role doesn’t provide you with the specific reward you need.
This reward can take many forms including financial, affinity and achievement. The additional financial reward from being an AE has been covered before.
Affinity
AEs are often better integrated than DAs into Engineering organisations inside companies. They are much more likely to be brought into discussions about potential upstream changes than DAs are, they use more similar tools to SWEs (eg git, terraform) and approaches (eg modularity, DRYness). Being included and better understood is rewarding at work. AEs also get to understand how DA/DS/SWE are going to use what they are building - AEs get to sit in the middle of many flows. This reward is consistent, you plan what you’re going to do this sprint, speak to upstream and downstream teams, then execute in collaboration with them.
DAs, being often worse integrated into Engineering, are more likely to struggle to find what they need or even know if it exists. They may feel isolated from the teams who are responsible for the data they need, especially if they have been embedded into another discipline with no central connection. They are better connected with the end consumers of data; their work can be the last mile of data work - the research with recommendation, the dashboard for monitoring the OKR, the statistical analysis to test whether a null hypothesis can be rejected. However, their affinity reward is hugely linked to whether their stakeholders value them, much more so than for AEs. Every reorg holds substantial risk for a DA, whereas the impact on an AE is smaller.
DA roles can be undermined by the presence of AEs in an org. If stakeholders don’t trust their DAs, they can often go directly to the AEs with an expectation that they know the data better, given they built it. This is without realising that they may lose all of the nuance and context a good DA will provide that an AE may not. This can make the AE role more appealing in an organisation - there is reward in being trusted.
Achievement
In comparison to DA work, the end result required, whether the work is possible and what inputs are needed, are all clearer with AE work. With the right skills, tools and inputs, you can achieve highly in AE. Whether the data models you build are used or whether they are impactful doesn’t define whether an AE was successful. Ultimately, as an Engineer, they are asked to make something according to a specification. The way AEs achieve is much more similar to how SWEs achieve, than to DAs.
The achievement reward for impactful work in this space is arguably much higher with DA than with AE work. The whole org can recognise good DA work by following the recommendations made. However, these rewards can be few and far between. The ability of a DA to gain this bigger reward from their work is fragile. It takes leadership that wants to listen to recommendations from a DA. It requires the data to exist in order to drive the conclusions to be available in the first place. It needs significance and clarity in the results to uncover truth. There is much less clarity around what DAs should do to achieve highly - they need to understand what their business is trying to achieve and orient themselves accordingly. Often they make the requirements that AEs work to, in order to get the inputs they need - they help AEs to achieve highly.
Politics and Impact
AEs are shielded from politics, reorgs and layoffs in a way that DAs aren’t. Cutting back on AEs feels like cutting back on your core ability to deliver on Data… DAs seem more expendable to management, as they understand what analysis is compared to the black magic that is Engineering or Science. This gives AE roles a sense of security over DA roles. For candidates who are risk-averse and choosing between DA and AE roles, this may well make AE more appealing. AEs are much more likely to be centrally placed within an org than DAs - this means there is less instability for AEs during any reorg. For candidates who don’t like a lot of change in where they sit in a company or line management, AE roles may seem more appealing.
DAs often have to surf the waves of politics and organisational change - better yet they should be driving them with their analysis and input. DA roles which truly do this are probably the most impactful roles in Data, but they are incredibly rare. There are many technical candidates who could choose between DA and AE roles, who previously only had DA roles to choose from. Some of these candidates would much prefer to not deal with the uncertainty and complexity of organisational dynamics, and stick with the technical side. AE roles offer them this choice. The “E” in AE means that non-technical stakeholders have little to no expectation for AEs to be skilled in this area - AEs can progress in their careers without it.
Role suitability, progression and experience
The role of an Analytics Engineer is very new and is interlinked with dbt as a technology - enabling it to exist as a role distinct to previous ones. I would say that Analytics Engineering as an activity did take place before, in Data Engineering and BI development, but it was much more complicated and unpleasant. Just as I have described an evolution in job role leading to Analytics Engineer here, this is a short evolution in tooling:
SQL Transactions run manually one after another
Stored Procedures running SQL Transactions serially
Airflow managing SQL Transactions
dbt
dbt (in conjunction with BigQuery and Snowflake) has made the workflow of data transformation in SQL enjoyable. The tools available before made systems so brittle that they couldn’t be complex and most of your time was spent fixing, updating or rerunning existing models. The size and complexity of the DAG you can cope with when using dbt is at least ten times as large as before. AEs have a development workflow that only SWEs have enjoyed until recently.
For people who enjoy building data systems and models, the AE role is very appealing. The joy of the role is similar to playing with Lego or a train set as a child. The waiting to see if your new branch line will cause the train to derail or stop - the sense of satisfaction when it smoothly passes through. The satisfaction of writing rigorous tests for your gold tables and seeing them all go green on the next run. The puzzle to solve when one goes amber or red. Fixing broken runs out of hours, and gaining the moral high ground over the SWEs that caused the problems (plus requesting an incident report) 😉. This joy of building is new in Data and mostly exclusive to the AE role. The Developer Experience in the other roles doesn’t allow for this joy, though some may disagree.
There is also a sense of satisfaction in how there is an element of cleanliness and order in the role. An example of when this might be felt is from converting a giant Looker PDT into a set of clean refactored incremental models that also run an order of magnitude faster=cheaper. Surely that’s worth a celebratory Slack message in the AE channel, with some 🎉 and 💥 responses from your team!
You can be rewarded with progression and remuneration for doing these things well as an AE! It’s easier to progress as an individual contributor as an AE than as a DA, where progression and pay rise usually comes with management responsibility and taking you away from the role you enjoy. There is less impact or political weighting to your progression as an AE - it can be about mastery alone. As a DA this isn’t the case, I always find myself advising DAs to ensure they’ve done visible work that has been published somewhere in the org ahead of promotion cycles… it shouldn’t necessarily matter but it’s much easier to persuade those who make the decisions when they’ve seen and read the work.
