Text Classification Approaches in the LLM Era
How Medium used LLMs to revamp its NSFW model
1The Role of LLMs in Text Classification
Medium uses machine learning models to automatically label stories, which determines the content recommended to readers. LLMs (Large Language Models) offer a new approach in this process. Unlike traditional models, LLMs can perform text classification without requiring training. This is particularly useful for critical tasks like detecting NSFW (Not Safe for Work) content.
2The Need to Update the NSFW Model
Medium's NSFW model has been in use since 2021 and now requires an update. The model plays a critical role in filtering inappropriate content in open office environments. However, it runs on outdated Python versions and is not regularly retrained. This reduces the model's performance and reliability. Therefore, the model needs to be refreshed and integrated with modern technologies.
3Advantages of the LLM-Based Approach
LLMs offer a more flexible and faster solution for text classification compared to traditional models. They can classify text accurately without requiring a specific training process. Medium aims to improve performance and reduce costs by using LLMs to revamp its NSFW model. This approach can be particularly effective with large datasets.
4Creating the Evaluation Dataset
To test the LLM-based approach, Medium created an evaluation dataset consisting of randomly sampled stories. This dataset includes a balanced mix of NSFW and SFW (Safe for Work) content. The curation team labeled each story and clarified the definition of NSFW, providing a solid foundation for the LLMs. This step is crucial for measuring the model's performance.
5Prompt Design for LLMs
For LLMs to classify text accurately, Medium designed a prompt that includes the NSFW rules defined by the curators. The prompt contains the story's content and the expected output from the LLM (a score between 0-100 and a short explanation). For example, 'graphic description of sexual intercourse' helps the LLM make a decision.
6Comparing LLMs and the Traditional Model
Medium compared the predictions of LLMs and the traditional model on the evaluation dataset to measure the performance of the LLM-based approach. The results showed that some LLMs matched or outperformed the traditional model. Mistral and Claude models performed particularly well in terms of false positives and false negatives.
7The Importance of Performance
Both false positives and false negatives can create significant issues in the NSFW model. False positives lead to blocking appropriate content, while false negatives allow inappropriate content to be shown. The LLM-based approach is seen as an important step in minimizing these risks. Performance is a critical factor for the model's reliability.
8Cost Analysis
The costs of the LLM-based approach are calculated based on the cost per input token for each model. Medium is evaluating whether this approach is feasible in terms of both performance and cost. The flexibility and speed of LLMs have the potential to reduce costs. This could be a significant advantage, especially in large-scale applications.
9The Future of LLMs
LLMs offer significant advantages over traditional models in areas like text classification. Medium's NSFW model revamp project provides an example of how this technology can be used in practice. The performance and cost-effectiveness of LLMs could lead more companies to adopt this technology in the future.
10Protecting User Data
Medium ensures the security of user data when using LLMs. The LLMs are hosted on Snowflake's infrastructure, and the data is not used for training by other companies. This is an important step in protecting user privacy. Medium continues to adhere to ethical standards when using AI technologies.
11Results and Next Steps
The LLM-based approach offers a promising solution for revamping Medium's NSFW model. Performance and cost analyses show that this approach is feasible. Medium plans to use this technology in more models. LLMs could be increasingly used in areas like text classification and content recommendation in the future.
12The Potential of LLMs
LLMs have great potential in areas like text classification and content recommendation. Medium's NSFW model revamp project provides an example of how this technology can be used in practice. The flexibility and performance of LLMs could lead more companies to adopt this technology. This is an important step in the evolution of AI technologies.
Original article: Medium Engineering — LLM Çağında Metin Sınıflandırma Yaklaşımları