Hi David - appreciate you diving in on what we built at SDF.
Two quick clarifications:
1. Our Write Audit Publish implementation did not need to understand breaking downstream changes, since these were all caught at compile-time. Local compiles were run before sdf run and sdf build, thereby catching all downstream breaking changes in a matter of seconds as opposed to waiting for queries or audits to run against the remote database.
2. We actually did store state. We leveraged the local file system for this (particularly a `sdftarget` directory), and would incrementally rebuild the DAG based on changes to both models (code) and source data.
Thanks for your clarifications. I used this https://docs.sdf.com/tutorials/debugging to understand how sdf compile would find breaking changes and it seemed as though it would highlight that there were upstream breaking changes but that you would have to traverse the CLL manually to find them.
That’s why I was looking for something automated in build too.
Happy to chat and get to know you and some sdf folks!
Hi David - appreciate you diving in on what we built at SDF.
Two quick clarifications:
1. Our Write Audit Publish implementation did not need to understand breaking downstream changes, since these were all caught at compile-time. Local compiles were run before sdf run and sdf build, thereby catching all downstream breaking changes in a matter of seconds as opposed to waiting for queries or audits to run against the remote database.
2. We actually did store state. We leveraged the local file system for this (particularly a `sdftarget` directory), and would incrementally rebuild the DAG based on changes to both models (code) and source data.
Thanks for your clarifications. I used this https://docs.sdf.com/tutorials/debugging to understand how sdf compile would find breaking changes and it seemed as though it would highlight that there were upstream breaking changes but that you would have to traverse the CLL manually to find them.
That’s why I was looking for something automated in build too.
Happy to chat and get to know you and some sdf folks!
I see, the second part (bonus section) of that tutorial showcases how running compile will catch breaking changes.
And yeah - would be happy to connect! Feel free to connect with me on LI