We introduce SuperCM, a differentiable clustering module that uses a model's few labeled examples to guide how it groups all of its data.

By Durgesh Kumar Singh, Postdoctoral Research Fellow at SFI Visual Intelligence

Image classification means teaching a computer to look at a picture and sort it into the right category — say, recognizing whether a photo shows a cat, a dog, or a car. Most machine learning models for image classification are only as good as the labeled images they're trained on — but labeling images by hand is slow and expensive, so in practice we usually end up with a small, labeled set and a much larger unlabeled one. Consider a real-world example: a hospital might have thousands of unlabeled chest X-rays but only a few hundred confirmed diagnoses, or a model trained on images from one scanner might need to generalize to images from a completely different scanner at another hospital.  

Two research areas try to address this kind of gap:

• semi-supervised learning (SSL), which combines a small, labeled set with a much larger unlabeled set from the same distribution, and
• unsupervised domain adaptation (UDA), which reuses labels from one domain (the source domain) to help a model work on a new, unlabeled domain (the target domain).

Both SSL and UDA lean on a clustering assumption: images that look similar to the model probably belong to the same category. In SSL, if an unlabeled image looks a lot like a group of labeled images from one category, it's probably that same category. In UDA, the same logic applies even when the unlabeled images come from a different domain altogether. Earlier approaches have used the clustering assumption in an indirect way, incorporating it into their “training process” and “loss function” design.

In our work, we take a more direct route. We attach a small, trainable "clustering module" onto a neural network that explicitly sorts images into clusters while the network trains. The key twist: we use the available labeled images to anchor where each cluster's centroid should sit within the model's internal feature representation, continuously updating a running average position for each class. This gives the model a stable target to organize around, rather than letting clusters drift apart or collapse into each other — a common failure mode in earlier clustering-based approaches.

We call this method SuperCM, and evaluated it on standard image classification benchmarks for SSL (such as CIFAR-10, SVHN, and STL-10) and UDA (such as Office-31 and Office-Home) — both as a standalone approach and as a plug-in regularizer for five widely used SSL techniques and several UDA methods. The results show consistent improvements, strongest exactly where it matters most: when very few labels are available.

Visualizing the model's internal feature representations confirm the intuition: with SuperCM, images belonging to the same class cluster tightly together, rather than forming one indistinguishable blob. The same pattern holds when adapting a model from one domain to another.

Semi-Supervised Learning (SSL) and Unsupervised Domain Adaptation (UDA) enhance the model performance by exploiting information from labeled and unlabeled data. The clustering assumption has proven advantageous for learning with limited supervision and states that data points belonging to the same cluster in a high-dimensional space should be assigned to the same category. Recent works have utilized different training mechanisms to implicitly enforce this assumption for the SSL and UDA. In this work, we take a different approach by explicitly involving a differentiable clustering module which is extended to leverage the supervised data to compute its centroids. We demonstrate the effectiveness of our straightforward end-to-end training strategy for SSL and UDA over extensive experiments and highlight its benefits, especially in low supervision regimes, both as a standalone model and as a regularizer for existing approaches.