Start at the end...
...and why data leaders need to have impact
Happy New Year and welcome to the first post in my Substack!
I decided to start my own Substack as a way to blog my journey into a new role where I’m going to be building a new (very modern) stack and a new data team in the context of a new way of working. It’s as much a way for me to debrief how things are going and organize my own thoughts as a way to share my experience and give back to the data community.
While I was mulling over the idea of doing this, Benn, whose Substack many of you will also read, threw down the gauntlet for other people to follow in his footsteps and make their own blogs.
And here we are in my Substack!
I’m aiming to be very open and honest about my experiences and thoughts without offending. We have to be able to talk about both the positive and negative parts of our experiences; without this, we can’t share our learnings or propose doing things differently.
Leaving Lyst
Before thinking about my new role I think it’s worth debriefing my current and very soon-to-be ending role as Senior Director of Data @ Lyst. I joined Lyst as Lead Business Intelligence Analyst in September 2019 after leaving my role as Head of BI & Analytics @ Elevate Credit Ltd, and was looking for a new cultural experience more than anything else; I wanted to have the freedom to make the choices I needed to deliver, without being blocked as I had been in past roles.
It’s been a great ride!
True to the promise of autonomy in my interview process, I was allowed to come in and make the changes I needed to succeed; two weeks after joining and dealing with Redshift’s concurrency issues I proposed bringing in Snowflake and dbt. We managed to achieve migrating from Redshift in eight weeks (with two weeks spare for testing) ahead of a key Black Friday shopping period, where stakeholders were previously concerned they wouldn’t have the data needed to operate.
This successful migration to dbt and Snowflake served as a platform for success at Lyst, with the enablement of new analytical use cases and research. We grew the team from 2 to 12 with specialization between analytics engineers and analysts - embedding analysts and analytics engineers into domains with a hub and spoke model. With this success in my role came two promotions and also being given the data science chapter on the second.
I’m so grateful for my time at Lyst: the impact I’ve been able to have here, amazing people to work with and learn from, the teams I’ve built, and future Lyst data leaders I have been able to hire and invest in. I honestly wouldn’t have been prepared and ready for my new role without my time here.
Why leave?
Lyst operates based on the Spotify model (squads, tribes, and chapters), possibly more rigorously than Spotify itself. This has allowed Lyst to experiment quickly with different ideas and to allow different squads to succeed in what they need to do quickly. Any operating model has pros and cons, and one difficulty I have encountered with this one is that it is difficult to influence strategy in a way I have in previous roles; I have found out about key company issues through informal chats rather than being asked for advice - the model seems to block information flow through an org as the org itself is so decentralized.
In my early days at Lyst, I was so focused on engineering that this lack of influence in my role wasn’t felt acutely. However, as I grew my team and brought in/up leaders in data to look after analytics, analytics engineering, data science… the BAU and engineering didn’t need my direct involvement any more. However, I didn’t have a new higher-order strategic level of work to do and nor was I able to help steer the ship as the most senior person in data. This also coincided with the departure of our CTO at Lyst who was a lens into the strategic workings of Lyst for me and acted as CDO too.
I recently read Benn’s article on this topic, and it felt like a lightbulb went on… while I sit on the fence between individual contributor and people manager, and enjoy both, I did need a place to progress in order to influence my organization in the way I should and could.
Feeling lost at Lyst, I delved into software engineering as a way to improve myself where this wasn’t possible in my role. It’s quite clear in hindsight, from my contributions history graph on my Github profile, when I started feeling this way. The positive to this is that I feel much more knowledgeable about software engineering, recently contributing to the open-source BI tool Lightdash. I’m someone who is an all-rounder, I like to be technical and contribute as an individual even if just to prototype, but I also like to be an architect of a tech stack, an investor in people, and a designer of an organization. I’ve found that doing some of all of these activities provides synergies that specializing in just one could preclude, and also makes me able to see the bigger picture and able to contribute to strategy.
You may ask if I spoke to anyone at Lyst about this, and I did; I believe given another year or so and possible IPO that the kind of role I needed may exist. It’s taken me a few weeks to truly understand what growth I needed next and exploring the job market has helped too.
In the UK, the engineering and data job market has been furious since the start of 2021. Companies have decided that they need the talent to compete and after being cautious in 2020, they are aggressively hiring in a way not seen for a long time. Engineers and Data people being offered 20 to 40% more than in 2020 is commonplace. As anyone else in Data has most likely been, I have been approached about roles almost non-stop in 2021; in the months leading up to Christmas, quite a few roles were opened that really could not be ignored on all accounts (impact, interest, culture, package). This led me to also apply for roles that interested me, including the role at Ruby Labs, as I didn’t want to only select an opportunity from roles I had been approached for.
Leaving Lyst is one of the hardest things I’ve ever done in my career, I’m really going to miss the teams I’ve built and the people I love working with. However, on balance, I think it’s the right time for my own growth to move onwards into a bigger role with the impact I need. Lyst has a new generation of data leaders to take over and I’m proud to have been part of that.
My story of reaching the end of the road at Lyst will be repeated for many other data professionals in many other orgs - without being given space for an impactful role “the treadmill at the mountain top” will keep turning.
The Data Leadership Problem
Data teams are often similar in size to Finance and Marketing teams in many orgs and are usually larger than Product teams. It’s unthinkable that an org wouldn’t have a CFO, CMO or CPO but it still seems that many orgs don’t see the need for CDO or CAO roles. Often, I feel it’s because one of the existing C-suite wants to control Data because they see it as part of their domain OR that the rest of the C-suite don’t understand it as a discipline.
CTO’s, some of whom don’t truly understand data as a separate, but closely associated discipline to Engineering (there is a lot of engineering needed to support Data), often want to have control over Data but don’t really understand it as a discipline. It could be argued that Finance runs on similar tech to Data but on a much smaller scale with specialized scope - but you wouldn’t put Finance under a CTO. Analytics, if done well, often has a commercial impact and is then seen as a good fit in Finance or Marketing… These are all good reasons for Data to support or be supported by these groups but not be a subset of them. I strongly believe Data needs its own distinct part of an organization and a voice at the top table.
Looking forward to Ruby Labs
Having the impact I need
One of the things I really wanted from my new role was the strategic input into the organization that I don’t currently have at Lyst. I confirmed this with Ruby Labs with a great amount of clarity during my interview process: as a VP I would be part of the Senior Leadership Team (SLT) which meet weekly, much as the C-suite do at Lyst or most other companies; they confirmed they want data to be at the heart of their decisions and want someone at SLT level who can guide them from this viewpoint, a present and not missing analytics executive. I think it’s better to be bold in the interview process and be very clear about what you need and want from the role - don’t assume much especially around nuances like influence, which can greatly vary from company to company.
Just being part of a strategic meeting doesn’t guarantee impact, though. I think it will require a much more mindful, purposeful, and proactive approach. I’m a naturally curious person who likes to understand why things happen and to test and confirm these whys; I plan to use this as a strength in my new role. I’m planning to use things like SLT meeting agendas as ideas for research in advance and topics discussed as further areas to look into.
As a commercially-minded person, I’ve found it natural to then translate this data-driven knowledge of an organization into novel analytics and ideas for optimization and exploration; this is then how data has a proactive voice in an org. There is also space for data technology and methods to be embedded into products, which can happen in parallel with analytics.
Thanks for reading if you’re still here and let me know if you have any questions: I’m on twitter @DSJayatillake and Locally Optimistic and dbt slacks.
Interesting read on your journey so far David, looking forward to hear how the next chapter shapes up. All the best!
This is great, appreciate you sharing your experiences. Do you think that something like the executive-level analyst described in Benn's post would have scratched your itch for "higher-order strategic level of work" at Lyst and kept you there? Or was there an equal amount of looking for a new problem space, new environment?