Fine-Tuning SigLip: When Is It the Right Choice?
Balancing cost and performance in AI classification tasks.
1Introduction: AI Image Classification
Alma Media, a Finnish digital services company, develops AI/ML solutions for real estate listing services. Handling hundreds of thousands of listings annually, the company uses AI to automatically tag images. This significantly enhances search, recommendations, and internal use cases. For example, determining whether a photo shows a kitchen, floor plan, or garden improves user experience.
2The Challenge of Multi-Labeling
Alma Media's AI solution automatically tags photos with room-type and content classes, such as LIVING ROOM, KITCHEN, and BEDROOM. However, challenges arise. For instance, if a photo shows 10% living room and 90% dining area, how should it be tagged? These decisions depend on application requirements.
3Classification Approaches
Image classifiers can be built using third-party APIs or custom classifiers trained on open-source models. Alma Media fine-tuned Google SigLip to develop its own classifier. This approach offers significant cost and performance advantages, especially for large-scale projects, compared to API usage.
4Cost Comparison
Classifying 1 million photos using Google’s Gemini-3.5-flash model costs approximately $1,500. Alma Media uses its own classifier on an AWS EC2 g4dn.xlarge instance, reducing the cost to just $0.37. This represents a 1/4000 cost advantage over API solutions. However, for those processing a few hundred photos daily, the cost difference is negligible.
5Reliability and Accuracy
Classification reliability is crucial, especially for user-facing results. For example, when presenting kitchen photos, high-confidence matches are preferred. Reliability estimates from APIs are often poorly calibrated. Training your own classifier provides usable per-class scores that can be calibrated as needed.
6Data Acquisition and Cleaning
Training a custom classifier requires thousands of labeled examples. When launching a new product, this data may not be available. Starting with unannotated photos, predictions from a VLM can gradually build a labeled dataset. However, this dataset may inherit the VLM's mistakes, necessitating a cleanup process.
7Approach Based on Use Cases
The type of classification task determines the approach. For example, classifying cars or kitchen appliances is easily promptable, while predicting YouTube thumbnail click-through rates is more complex. Alma Media's real estate use case falls between these extremes, with some classes easily promptable and others more ambiguous.
8The Role of Open-Source Models
If you decide to train your own classifier, starting with a pretrained open-source vision model is the most logical choice. However, for business use, checking the model's license is crucial. Alma Media fine-tuned Google SigLip to develop its classifier, achieving significant cost and performance benefits.
9API vs. Custom Classifier Comparison
Using APIs offers a quick solution for small-scale projects. However, costs escalate rapidly for large-scale projects. Custom classifiers provide lower costs and higher reliability. Alma Media's experience shows that custom classifiers are more suitable for large-scale projects.
10The Importance of Reliability Estimates
Classification reliability is critical, especially for user-facing results. Reliability estimates from APIs are often poorly calibrated. Training your own classifier provides usable per-class scores that can be calibrated as needed, offering more accurate and reliable results.
11Conclusion: Choosing the Right Approach
In AI classification tasks, selecting the right approach requires considering the use case, cost, and reliability. Alma Media's experience demonstrates that custom classifiers are more suitable for large-scale projects. However, API usage offers a quick and practical solution for small-scale projects.
12Future Perspective
AI classification technologies hold significant potential, especially in sectors like real estate with heavy visual content. Alma Media's success in fine-tuning SigLip highlights the importance of custom solutions. In the future, more companies are expected to adopt similar approaches.
Original article: Towards Data Science — SigLip Modelinin İnce Ayarı: Ne Zaman Doğru Seçim?