A Second Look
Old
I remember Drew Banin coming to Lyst’s office in late 2019 and giving my team a first look at the all-new dbt Cloud IDE. We were sitting in one of the break-out spaces in the office, peering round at Drew’s screen. I was interested, but at the time I wasn’t entirely sure I needed this. However, I felt it might be good for new starters who weren’t as technically minded as me.
VSCode has been my go-to tool for most of the last three years (after a very brief spell with Atom, which I’m now glad I dropped!). I initially used it for dbt, but now I use it for all other dev projects and languages, too. I’ve ended up working on 5 dbt repos in the past 9 months, including setting up 4 in that time 😅. In order to keep things simple, I’ve often used the dbt Cloud IDE on the dbt projects that aren’t my main focus at the time. For example, where I’m helping an organisation that I’m advising or consulting for. Even at Metaplane, I started off using the dbt Cloud IDE, as I only spend a minority of my time building in dbt.
There were times where I felt the dbt Cloud IDE was far too slow to start up and to respond in general. Git actions, saving, renaming and creating files were all terribly slow, especially when compared to it being nearly instant in VSCode. The UI had been recently refreshed too, but it didn’t feel like anything other than aesthetics had been changed. There were also the regular and lengthy outages that even needed to be addressed on the dbt blog.
There were a number of times I felt like slacking/zooming dbt Labs folks and saying: “When are you going to call time on this… it’s been bad for so long. You could easily do this via GitHub codespaces with good dbt specific extensions and most users would be happy. Or just buy something like Backlight or Paradime and repurpose it.” I didn’t do this however… I kept hoping a new release would make it better and I fed back any issues I found. I believe dbt Labs often plays the role of the tortoise - slow and steady but will eventually win the race. This is natural for a large community - it’s hard to go fast and take everyone with you. Then I started to hear rumours of a new version…
New
Getting into the new IDE is quick, but it also gives you an idea of progress to the instance being spun up, which funnily enough isn’t really needed any more, as it’s so quick. The old spinner that seemed to last forever is gone! That’s on cold start too - when the IDE has already been open recently, it’s almost instant. I had to be quick on the draw to get the screenshot above (cmd+shift+3 and then clip the picture, too fast to take the area of the screen I wanted).
Creating files/folders and saving/renaming files are all really quick now. Still slower than VSCode locally, but not enough for me to really notice. It would be nice to be able to right click instead of having to use the three dots… but not that big a deal.
The lineage seems slightly faster and automatically resizes better than before. There is a new ‘build’ button which lets you build, run or test the model that’s open and its upstream and downstream models. This is a nice new feature for people who aren’t that familiar with dbt CLI and model selection syntax. It also allows them to learn through watching which commands are run as a result of their button clicks - a mark of a good no code/low code feature.
There is an enhanced code preview ribbon next to the scroll bar on the right of the text editor, which is very similar to VSCode and other IDEs. It feels like a lot of the acute problems with the IDE have been resolved.
We’re then left with the benefits of easier git flow and abstraction, visualised lineage, interactive compiled SQL, no local install requirements (many aren’t allowed to change software on their work machines, especially once you cross the chasm) and environment management from dbt Cloud. There are still trade-offs where running locally might be better for advanced users, but I don’t know what they might be now for those who are less technically minded. It’s beginning to feel how development mode in LookML used to, inside the Looker platform. Time will tell whether new users will default to developing in the dbt Cloud IDE as opposed to their local machines, as was the case with most Looker Developers and the Looker platform. I think it may well be the case for enterprise dbt users.
There still seems to be the odd outage, as there always will be with any service. As long as they are few and far between, and increasingly so, I think the new IDE is in a decent place.
London Data Quality Meetup #2
We had some great talks and follow up questions at our second Meetup!
Lynsey Marshall of Marks & Spencer - "Building a Culture of Data Quality"
Unfortunately, Lynsey had to give her talk over video due to transport issues, but it was a really interesting non-technical talk on people, process and the mindset needed to drive organisational change towards Data Quality. An interesting point that I took away was the idea that you may need to talk to certain stakeholders about data without mentioning that you’re talking about data.
Kevin Hu - CEO @ Metaplane - "Five root causes of Data Quality issues"
Another great talk from Kevin, that connected on a practical and theoretical level. I particularly enjoyed his slide linking a data stack to business value and the layers in between.
Andrew Jones - Senior Data Engineer @ GoCardless - "Improving Data Quality with Data Contracts"
As far as I know, Andrew Jones coined the term Data Contracts, but not only did he come up with the concept, he did what any good engineer would do and implemented it. It was a real pleasure to hear from him and how the Data Contracts roll-out has been going at GoCardless.
Dale McDiarmid - Product Marketing Engineer @ Clickhouse - "Clickhouse-local as an ETL/data cleansing tool"
Dale had a very interactive session, where he showed off what Clickhouse can do on a practical level and shared his deep understanding of how Clickhouse works under the hood.
I look forward to hosting London Data Quality Meetup #3 nearer Christmas - keep an eye out for the recording that I will share from yesterday!