This blog post summarizes my thoughts and dialogue with various companies over the past couple of months. I often see data organizations have data ingestion, modeling, exposure, and infrastructure split into four separate roles and responsibilities.
I believe that having a team of people handle the data pipeline from end-to-end, alongside a separate data platform team with no access to data, creates the most efficient, precise, and motivated roles. These roles take responsibility for their solutions.
Great post! I've been thinking a lot about this as well and fully agree with you.
I've only been in the data space for 2 years, but it really does seem that data work is best stratified by platform, engineering, and analytics. Within my current data org (which is fairly mature), there is a data platform org, an analytics engineering org, and a data analytics org. Everything you outlined as well is the philosophical basis for the current structure of our teams.
Especially with so many declarative frameworks for infra management/data engineering work and the technical enablement of developers by AI, the line between DE/AE is getting even more blurry. I also believe it will only be time for the DE/AE roles to evolve into a unified discipline.
I think most people in data leadership sense this change, but they are unsure how to make changes to legacy systems or what team patterns and technologies would give them a leg up over the next few years rather than just being considered the cost centre.
This blog post summarizes my thoughts and dialogue with various companies over the past couple of months. I often see data organizations have data ingestion, modeling, exposure, and infrastructure split into four separate roles and responsibilities.
I believe that having a team of people handle the data pipeline from end-to-end, alongside a separate data platform team with no access to data, creates the most efficient, precise, and motivated roles. These roles take responsibility for their solutions.
The responsibility angle is very true. In the UK we say “when something is everyone’s responsibility, then it’s no-one’s”.
You could write a book on this post, its fundamental
Great post! I've been thinking a lot about this as well and fully agree with you.
I've only been in the data space for 2 years, but it really does seem that data work is best stratified by platform, engineering, and analytics. Within my current data org (which is fairly mature), there is a data platform org, an analytics engineering org, and a data analytics org. Everything you outlined as well is the philosophical basis for the current structure of our teams.
Especially with so many declarative frameworks for infra management/data engineering work and the technical enablement of developers by AI, the line between DE/AE is getting even more blurry. I also believe it will only be time for the DE/AE roles to evolve into a unified discipline.
I think most people in data leadership sense this change, but they are unsure how to make changes to legacy systems or what team patterns and technologies would give them a leg up over the next few years rather than just being considered the cost centre.