Schema - Tag - 300.Wattshttps://300watts.me/tags/schema/Schema - Tag - 300.WattsHugo -- gohugo.ioenmorristai01@gmail.com (Morris)morristai01@gmail.com (Morris)This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.Wed, 26 Jan 2022 10:04:55 +0800Make Kafka Schema Easierhttps://300watts.me/posts/make-kafka-schema-easier/Wed, 26 Jan 2022 10:04:55 +0800Morrishttps://300watts.me/posts/make-kafka-schema-easier/TL;DR: You can find the script on my GitHub repository: kschema-table

As a data engineer, you’ll need to collaborate closely with data scientists/domain experts to design data schemas. The optimal schema will depend heavily on the business domain and how the product is used. For instance, in a cybersecurity context, threat experts and data scientists would likely be the ones designing the data schema. They may work with infrastructure and data teams to define these schemas in a common format like, for example: Avro IDL (Avro Interface Description Language).

]]>