https://300watts.me/tags/kafka/Recent content in Kafka on 300.WattsHugoenmorristai01@gmail.com (Morris)morristai01@gmail.com (Morris)This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.Wed, 26 Jan 2022 10:04:55 +0800https://300watts.me/posts/make-kafka-schema-easier/Wed, 26 Jan 2022 10:04:55 +0800morristai01@gmail.com (Morris)https://300watts.me/posts/make-kafka-schema-easier/<p>TL;DR: You can find the script on my GitHub repository: <a href="https://github.com/morristai/kschema-table" target="_blank" rel="noopener noreferrer">kschema-table</a></p> <h2 id="coordinating-schema-with-data-scientist" class="headerLink"> <a href="#coordinating-schema-with-data-scientist" class="header-mark"></a>Coordinating schema with Data Scientist</h2><p>As a data engineer, you&rsquo;ll need to collaborate closely with data scientists/domain experts to design data schemas. The optimal schema will depend heavily on the business domain and how the product is used. For instance, in a cybersecurity context, threat experts and data scientists would likely be the ones designing the data schema. They may work with infrastructure and data teams to define these schemas in a common format like, for example: Avro IDL (Avro Interface Description Language).</p>