I dreamt we spoke again
Looking forward to Coalesce 2023
I attended Coalesce 2022 in New Orleans, and it was easily the best conference, as a data person, that I had attended in person. There were so many great talks (I’m still catching up on some recordings!) that were just about “doing data” and not specific to any vendor.
It had a real festival vibe, especially in contrast to Snowflake Summit, which I had attended just before. I couldn’t walk down a corridor without bumping into someone I knew.
I also got to give a talk:
I have many great memories from last year, but, taking a look at the video of my talk, so much has changed since this time last year:
What I’m working on
What I’m hoping for from the conference
Where so many of the people I met last year are working
The rise of DuckDB
Increased criticism of the Modern Data Stack
Last year, the conference had a big focus on the semantic layer and it will again this year, with the launch of the new semantic layer - a result of the Transform acquisition.
I wonder if there will be any surprise announcements? If so, I’m expecting that they will be more focused on integration and ecosystem - where and how you can use the new semantic layer.
Will there be a new pricing model, more aligned to the idea that the semantic layer is a core driver of value for dbt Cloud customers in the future? While this is possible and perhaps logical, I’m not expecting it. A move to consumption-based pricing for use of the semantic layer might make sense for startups and SMB, but enterprise customers are willing to pay per seat and pay a lot… they just need to be able to accurately budget for their costs.
Many smaller data teams prefer using their own IDE to the dbt Cloud IDE, many prefer to run dbt-core with Dagster or Airflow, rather than using dbt Cloud jobs… this all changes at enterprise level.
Many of the people I met last year who were leading small, scrappy data teams have now moved to larger, established enterprises. I’ve spoken to people in this situation who have said they aren’t allowed to use open-source technology (despite it being used to make nearly all software products out there). Yes, part of this is suspicious info-sec people not trusting open-source, but, more more importantly, it’s enterprise legal teams needing someone to pass liability on to… the buck will not stop with them!
The enterprise is hungry for labour.
The enterprise is hungry for data.
The enterprise wants to use AI (but is also afraid of it).
The enterprise is sick of waiting for multi-year IT projects that originally had data in scope, but drops it along the way.
The enterprise is hiring people like us who have worked at faster paced orgs and used newer tooling.
We want to use dbt in our new role in large enterprise.
It’s the piece of the stack that we’ll quit for, if we can’t have it.
Therefore, the enterprise now wants dbt.
The enterprise doesn’t bother with small vendor contracts and is, in fact, suspicious of them (if we sue this vendor, will they be able to pay out?).
The enterprise isn’t going to let you run your choice of open-source software.
The enterprise expects to pay a substantial amount for a product.
The enterprise often wants on-prem, or to deploy in their own cloud VPC.
The enterprise doesn’t care about the difference between this price and that price, as long as it is a fixed price.
The enterprise doesn’t care about paying per seat - in fact, the enterprise likes it, as headcount is locked in for the next three years.
The enterprise likes multi-year contracts: once something is deployed in the enterprise, it’s going to be there for a long time.
The enterprise won’t let you install software on your machine.
The enterprise won’t let you use an open-source orchestrator to run anything on its infrastructure.
The enterprise expects that your purchased software is feature-complete and able to operate on its own without other MDS tooling.
The enterprise doesn’t care if the features built aren’t best in breed. The enterprise only cares if the features tick the boxes in the tender process.
The enterprise isn’t on data Twitter, nor does it care about what is said there.
Perhaps we’ll find less for MDS folks this year, but more for enterprise data folks… but the distinction between the two groups of people is now more blurry than ever.
Michael and I are heading to Coalesce 2023, with a new company logo, to boot 😎! If you see it on someone’s shirt, it will be one of us - so come say hi!
I think this is all true of some Enterprises... but for some it's the exact opposite. Some REQUIRE open source and/or self hosted solutions because they know exactly what code is being run, and where. For example Datacoves (https://datacoves.com/) primarily exists to provide dbt to Enterprise companies whose InfoSec teams won't let them use dbt Cloud.
The enterprise, this "illusive, quiet group", seemingly has a much different life than people at smaller companies. At the end of the day, data is still data. It's funny how it all works out like this, though.
"The enterprise won’t let you use an open-source orchestrator to run anything on its infrastructure." That gave me a good laugh, because it's so true. I've worked at massive enterprises and if I had brought this up, I'd have been laughed out of the room.