Spot on: 'Data Contracts is that they are a way for us to go down the stack and change our foundations to be on rock' & 'companies that have “won with data” have been using the concept of Data Contracts'. I totally aree.
I would propose these additions to the contents of a data contract:
1) Access: The tech details of where to consume the data and pointers to the process of getting access. These technical aspects like the storgage system do matter as they often imply contraints and capabilities of the underlying system and hence of the data.
2) Monitoring: Many (but not all) parts of the data contract can be monitored. That is the big advantage of going down the stack. Executable data contracts imply that monitoring for it can be automated. The data contract should include which aspects are already monitored by the producer. Monitoring may also include a producer vs consumer view. As a producer, you want to monitor what you produce. As you mention correct, the responsibility of the quality of the beer is with the producer, not the consumer. But the consumer may have specific usage requirements that go beyond the default producer checks. It should be possible to also add usage data contracts that expresses the specific usage requirements. This also enables better impact analysis.
I feel like I want to put as little in the data contract as possible. I think access can be dealt with in the same way as for other APIs. I feel our RBAC models for access in data and engineering are currently ok. Could be improved but are workable.
I also worry about use cases from the consumer being put into the data contract that can't easily be defined in terms of types and structure etc.
We really want application engineers to look at the contract and think it's very clear what I'm agreeing to. I don't think the beer producer should have to care that you mix their beer with cider and blackcurrant 😂.
Great work, David! Love it.
Spot on: 'Data Contracts is that they are a way for us to go down the stack and change our foundations to be on rock' & 'companies that have “won with data” have been using the concept of Data Contracts'. I totally aree.
I would propose these additions to the contents of a data contract:
1) Access: The tech details of where to consume the data and pointers to the process of getting access. These technical aspects like the storgage system do matter as they often imply contraints and capabilities of the underlying system and hence of the data.
2) Monitoring: Many (but not all) parts of the data contract can be monitored. That is the big advantage of going down the stack. Executable data contracts imply that monitoring for it can be automated. The data contract should include which aspects are already monitored by the producer. Monitoring may also include a producer vs consumer view. As a producer, you want to monitor what you produce. As you mention correct, the responsibility of the quality of the beer is with the producer, not the consumer. But the consumer may have specific usage requirements that go beyond the default producer checks. It should be possible to also add usage data contracts that expresses the specific usage requirements. This also enables better impact analysis.
You mentioned not having seen anything open-sourced. Check out https://docs.soda.io/soda-cl/soda-cl-overview.html It's a Apache licensed and big step towards data contracts imo.
Thanks again for sharing your thorough research!
Hi Tom,
Thanks for the feedback!
I feel like I want to put as little in the data contract as possible. I think access can be dealt with in the same way as for other APIs. I feel our RBAC models for access in data and engineering are currently ok. Could be improved but are workable.
I also worry about use cases from the consumer being put into the data contract that can't easily be defined in terms of types and structure etc.
We really want application engineers to look at the contract and think it's very clear what I'm agreeing to. I don't think the beer producer should have to care that you mix their beer with cider and blackcurrant 😂.
Best,
David