The Data Collection Problem
Labeling data for every new manufacturing defect on an assembly line is completely unsustainable. You might only see a specific type of scratch once a month, making it impossible to train a traditional CNN supervised classifier.
Deploying CLIP
We deployed a zero-shot pipeline using OpenAI's CLIP architecture (loaded via the Hugging Face transformers library). CLIP projects both images and text into the same embedding space.
We pass the image of the manufactured part and compare its embedding against two text prompts:
- "A close-up photo of a perfect, flawless metal product."
- "A close-up photo of a metal product with a scratch, dent, or severe defect."
from transformers import CLIPProcessor, CLIPModel
import torch
from PIL import Image
model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32")
processor = CLIPProcessor.from_pretrained("openai/clip-vit-base-patch32")
image = Image.open("factory_cam_01.jpg")
prompts = ["flawless metal product", "defective metal product with scratch"]
inputs = processor(text=prompts, images=image, return_tensors="pt", padding=True)
outputs = model(**inputs)
probs = outputs.logits_per_image.softmax(dim=1) # The image-text similarity scoreIt works surprisingly well for high-level semantic anomalies, though it struggles with millimeter-level precision. We use it as a highly effective first-pass filter before sending borderline cases to a specialized, heavier model.