Etsy's Kafka Skills for Machine Learning
How AI agents are simplifying the development of data streaming applications
1Etsy's data streaming needs
Etsy hosts over 100 million unique listings from 5.6 million active sellers. Without a standard product catalog, machine learning models rely on data streams to understand user searches and product features. Kafka powers this real-time data pipeline. For example, when a user favorites a handmade ceramic mug, this information influences the recommendation system within minutes.
2Developer challenges
In the traditional process, engineers manually coded Kafka applications. They needed to filter events from terabyte-scale topics, write transformation logic, and route outputs to feature stores. The real challenge wasn't writing Java or YAML, but selecting correct schemas, identifying base classes, and anticipating edge cases. Static templates could generate files but couldn't accelerate the iterative process.
3Birth of Kafka Skills
Etsy's team developed reusable playbooks called 'Skills' to simplify Kafka feature development. These AI-assisted guides teach language models how to execute specific tasks. For each workflow, the agent automates steps like code generation, config validation, and local testing. The system significantly reduces manual coding requirements for streaming applications.
4Collaborative development process
Kafka Skills enable conversational workflows between developers and LLMs. They can be invoked via slash commands like '/new-streaming-feature' or natural language. The agent creates Java classes implementing shared interfaces for each streaming feature. YAML configuration files follow a fill-in-the-blanks approach, making the setup process more accessible to developers.
5Local application execution
The system automates all steps required to run applications locally. Developers no longer need to create test topics, build JARs, or enter manual commands. The agent handles these processes, reports findings, and generates relevant output links automatically. This streamlined approach saves significant development time and reduces potential errors.
6Observability and usage analysis
The team implemented lightweight telemetry to track Kafka Skills usage. When a Skill is invoked, a Python helper emits a start event. When the agent opens a pull request, a wrapper emits a completion event. Both events flow through Kafka and are stored in BigQuery tables, providing valuable insights into developer adoption patterns and workflow efficiencies.
7Adoption challenges
Initially, developers weren't using Kafka Skills simply because they weren't aware of them. While using coding agents, they weren't being offered these capabilities. The common workflow involved copying existing feature code and modifying it. To address this, the team added multiple entry points for different vocabularies and implemented repository-level routing to trigger Kafka Skills contextually.
8Future and security considerations
Kafka Skills were designed to prevent agents from inventing Kafka applications from scratch. The agent handles ambiguous aspects like understanding developer requests and selecting validation steps. Stanford's 2023 security research supports this structured tool-use approach. The system balances developer productivity with necessary guardrails, ensuring both efficiency and security in the development process.
Original article: Etsy Engineering | Code as Craft — Etsy'nin Makine Öğrenimi için Kafka Becerileri