SQLMashed

What does this mean for data folks

As many of you will have heard, the company that looked after SQLGlot and SQLMesh, Tobiko Data, was recently acquired by Fivetran - one of the biggest companies to have come out of the modern data stack era.

Now that the dust has settled - it’s been a couple of weeks since the announcement - I thought it would be good to evaluate what it means for data folks and how I feel about it in general.

There is one sense in which I have no conflict in my thinking. I have got to know the Tobiko team since starting my series on SQLMesh, having since overseen real deployments. I’m very happy for them in the sense that they built something great, and that they have had what I assume is a successful¹ exit🫡.

After all, the reality of founding companies, especially in VC, is that the company will one day be sold and investors and founders make money. I know people don’t like to be so explicit about this: they like to talk about the mission and that they’re building a company not to be acquired, but to become a public company… one with its own tower in San Francisco, with its logo lit up for the whole city to see. There are very few such companies… you probably know most of their names. Most VC-backed companies that have a successful exit are acquired. These exits aren’t so glamorous, they don’t get to ring the bell at the stock exchange, the founders end up taking middle-management roles at the acquirer with a length of lock-in… they might get a cool LinkedIn announcement and article in TechCrunch.

So that’s fine, from an economic point of view everything makes sense for Tobiko as a company and that’s why it happened.

What about for us data folks?

So this is where, even with a bit of time passed, I still don’t think it’s good for us. As much as Fivetran are currently saying that the open source SQLMesh and SQLGlot projects will remain and be maintained… I don’t expect this to be the case for that long - all it takes is one difficult quarter and anything unnecessary gets axed. Fundamentally, it is difficult to justify spending expensive engineering time and resource on a free product that could cannibalise the paid one. This is the friction we’ve seen between dbt core and Cloud and we know where that has ended up: a solid enterprise offering with good feature release cadence but with support for open source neglected and now diverged since the dbt Fusion engine.

dbt Fusion Has People Divided- Here's Why

This negates the idea that the goodwill from community is valuable enough to support the open source project, because if anyone could have done this, dbt could have. Fivetran started as a profit-making company with a proprietary product, and haven’t been afraid to change prices (to their customer’s annoyance)… there is not much in the way of community love or trust. It’s just an easy to use product that is the right choice in many circumstances.

Let’s assume then that SQLMesh is long-term available inside Fivetran only. A lot of us buy Fivetran² as part of our data stack, so in that sense it’s great because we get something from a vendor we already work with and don’t have to pass another infosec review etc. Fivetran have also recently bought Census, so they can extract and load data. They were previously using dbt transforms, allowing them to also send transformed data to target systems. They were previously also relying on dbt open source for their transforms, but as I mentioned above, this is now a neglected project. It would be much better for Fivetran to own their own transformation technology, and now they do. As far as I’m concerned, it is a market leading technology, too. So now Fivetran really is the Informatica of this generation. I also think that this makes Fivetran one of dbt Labs’ biggest rivals now.

However, even for those of us who do buy Fivetran, this is not all good news. Fivetran will eventually charge for SQLMesh transformation… otherwise there was no point in buying it. Fivetran need to grow their revenue through cross-selling a new product to the customers they already sell connector services to. So, if you were previously using Fivetran for connectors, and using your orchestrator to run SQLMesh jobs afterwards, you will eventually need to pay Fivetran to run these SQLMesh jobs for you. Now there will certainly be savings to be had through reduced infrastructure work for data and analytics engineers, but it will also be constrained to work inside Fivetran, however this ends up looking… at best it will be like using a CLI (much like the dbt Cloud CLI) and VSCode extension that will run tasks on Fivetran remotely.

For those of us who don’t buy Fivetran, or want to come off it, or wanted to use SQLMesh independently… things become difficult. At the start of last year, we had the choice of SDF, dbt, SQLMesh and new upstarts including Quary. Long-term, we really just have dbt Cloud and Fivetran(SQLMesh)… you could hand-roll your own SQL in your orchestrator or you can pay up or you can fork one of dbt core and SQLMesh and maintain it yourself.

There are also data warehouse/lakehouse transformation methods offered by Databricks, Snowflake and some newer entrants like Bauplan, which we never used to consider because of lock-in. However, now that lock-in of some kind is inevitable, we might as well consider them, too. It feels like there is room for a new challenger, but at the same time I’m getting tired of having to learn new frameworks to achieve the same thing.

Perhaps it’s actually reasonable enough to vibe code your own transformation framework that works for your specific project, and suits your style of writing transformation code. It’s free like a puppy, but at least it’s your dog.

1. I don’t know specific numbers, but I don’t think Tobiko were running out of money, so they didn’t need to sell. Therefore, they were made an offer that wouldn’t make sense to refuse. In my mind, this must be at least their last valuation or near to avoid pref stack collapse on equity for the team… if you aren’t running out of money you probably wouldn’t choose that.

2. The last four data stacks I have worked on used Fivetran for all or part of batch extraction.