Creating the Data Experience

Drivers behind the pay discrepancy between Engineer and Analyst in data

Photo by Bruno Figueiredo on Unsplash

The modern data experience, as experienced by data end-users, can take a few different forms, including (with common tools):

• Self-serve BI (Looker/Lightdash/Glean)

• Analyst-curated dashboards (eg Tableau/Preset)

• Analytical Research (Hex, Count, Deepnote)

• APIs (Transform, Cube, dbt)

This is without even considering the ML workflow or streaming.

There are two main roles that act to create this experience: Analytics Engineer (AE) and Data Analyst (DA). There has been some talk in the data community around value discrepancy between the two roles. My aim in this post is to explore the lineage and reasoning behind why this discrepancy might exist.

Historical Role Lineage

Pre "Big Data" Era

Analytics used to be performed on OLTP databases repurposed for OLAP use cases like SQL Server, Oracle, MySQL etc. DBAs and BI developers were a scarce resource, who had to be very knowledgeable about Platform in order to hand hold these databases. Typically, these two roles were better paid than an Analyst role. Analysts were also thought to be easy to find and train.

There were plentiful candidates who knew SQL or who could get by on Excel (data was often aggregated to be the right size for Excel). Analysts very rarely did meaningful work until they were very senior, they mostly did data pulls and Excel charts. In my own experience as a junior to mid level Analyst, the more interesting work available was straying into BI development (a forerunner to Analytics Engineering, or Analytics Engineering sans dbt).

Some DBAs and BI Engineers who were paid more than Analysts at the same companies have become Analytics Engineers today. As you might expect, this gap in pay between them and the Analysts has been maintained as they changed role.

The "Big Data" Era

During this era, there was a huge shortage of Data Engineers who had the skills to run these stacks. So much so, huge companies (Hortonworks, Cloudera, MapR) rose quickly to deploy these solutions, at great cost, for enterprises. This shortage of appropriate Data Engineers, coupled with the complexity of training staff on the stack, resulted in a wage boom for Data Engineers. Therefore, there was a big movement of Software Engineers into Data Engineering due to the salary potential difference.

Now that we're largely past that era, some of these Data Engineers have become Analytics Engineers. However, it's difficult to hire people on less than they were on before, and so they are still highly paid as Analytics Engineers.

This was also the era where Data Scientist was branded "the sexiest role of the 21st Century". The perceived higher level work was split left and right into DE and DS with Analysts sandwiched in the middle, again perceived to be of lower value, inheriting the lower relative pay into the next era. I have often wondered whether it was the E in AE and DE and S in DS that resulted in higher pay; commercial stakeholders don't feel they understand what an Engineer or a Scientist does and therefore it must be more difficult than what an Analyst does, which is easier to understand.

The MDS Era

Today we're in the MDS era and perhaps the latter half of it. Analysts moving directly from Pre Big Data and Big Data eras have inherited lower pay than the BI developers and Data Engineers who became Analytics Engineers.

The move to self-serve for everything further devalued Analysts and made AE worth more. We lost sight of the fact that there is much more to DA work than reporting, because of what analysts were constrained to do in the past. Once AE matured we realised there was deeper work to be done to explain complex topics and drivers behind core metric changes - this is largely what I'm describing as analytical research. It's the work that Analysts dream of doing - impactful and not focused on data quality. Some Analysts would happily get the data they need from an API without writing code and then analyse it afterwards. You can see this thinking in integrations between Metrics Layer APIs and spreadsheets.

Output

Photo by Austrian National Library on Unsplash

One of the lessons from the "Big Data" era was that you could spend a fortune on very clever PhD-wielding Data Scientists, but if they didn't have any data to work with or had poor data, you got little value out of them. One of the companies I worked for had a central Data Science team that was largely paid to do nothing, while a "Big Data" infrastructure project was completed!

Why don't we think the same way about Analysts? Analysts have always hacked it together if they weren't supported. If they didn't have a good pipeline they'd download a .csv, manually open in Excel and build a manual process to ensure month end reporting got done. I've been here before, in my first Analyst role, the first 2 days of every week were taken up running manual processes to create these reports (until I got bored and automated them with VBA). The Analyst picks up the slack from all Engineering shortcomings in Data.

While this is obviously valuable it has been seen as hard to reward the ability to work around poor systems well. The working around takes up so much time that the Analyst has very little time to do meaningful work, especially as their hacked together processes fail regularly and take days to rerun. The data downtime from these workarounds failing also led to Analysts having a poor reputation - highly unfair given it's not really their work or specialty to build pipelines with no tooling. Engineering has typically been able to say no to work that was too much or out of scope, but Analysts have never had that luxury.

All of these factors have compounded to reduce the quality, quantity, depth and impact of Analyst work in the past. This is also true in MDS adopted companies that haven't established an Analytics Engineering discipline yet. When I joined Ruby Labs, the data team worked much like I had in past roles, instead of MSSQL stored procedures, they had BigQuery scheduled queries. They spent days doing data prep to enable basic data requests, and analysis of A/B tests took much longer than implementing the test features. I could see tremendous value in the discipline of Analytics Engineering producing safe data for Analysts and non-technical data users in that environment.

This led me to wonder if there is a phased effect, with the value of Analysts and Analytics Engineers based on the organisation's maturity in terms of Analytics Engineering.

The third rail

It could be the case that, to start with, AEs are more valuable, as otherwise all Analysts do is churn out unreproducible analysis based on poor data and very slowly. As AE matures and Analysts are allowed to shine with safe data to use, they could be paid more than AE, based on the impact they have.

There is also a balance and multiplier effect here - it could well be true that a team of 2 AEs + 4 DAs is more productive than a team of 8 DAs. Thus, the AEs are more valuable when they are scarce. However, this return diminishes as it's unlikely that a team of 6 AEs + 2 DAs is more productive than a team of 4 AEs + 4 DAs.

Accessibility

From my experience, in hiring DE/AE/DS vs DA, it's clear that many people who seem inappropriate for a DA role based on CV but still apply, don't apply for DE/AE/DS roles. I've found AE recruitment funnels a great place to find Analysts too. Sometimes, SMEs for business areas like Finance, Marketing and Customer Service apply for Analyst roles, as their domain knowledge makes them strong candidates. However, all of these areas are paid less than Engineers and so the newly-converted Analysts inherit a pay disparity.

Gender representation is also better in DA than AE typically, which could also explain part of the gap. Women are generally paid less than men to do the same role at the same proficiency, so, if a profession is more even in gender split, then its average pay will be lower. I worked at a company where tech was 80% male and the other functions were 80% female, tech was paid double on average than the other functions.

AEs are in short supply (see below pics on attempts to remedy this!) due to the tooling and discipline being so new and not all Analysts would make good AEs or would be interested in the role (the converse is also true). AEs allow DAs to be less technical, and therefore probably more gender diverse¹.

Should AEs be paid more or less than DAs? It depends.

In return for tardiness in releasing this post here are a couple of pics from the London dbt meetup last night, hosted by Datatonic:

Hamzah on the mike!

Jerco IRL (another attendee said he should be played by Jesse Eisenberg in the dbt biopic 😂)

1. https://www.thenation.com/article/society/gender-silicon-valley/ from Benn’s earlier post