What is CLIP?
CLIP (Contrastive Language-Image Pre-training) is a foundational vision-language model from OpenAI that learns to connect images and text in a shared embedding space. Trained on a huge set of image-caption pairs from the web, it learns which images and which text descriptions go together. The breakthrough is that this single training objective yields a model capable of zero-shot image classification — recognising categories it was never explicitly trained on — and of capable image-text search, making CLIP one of the most influential building blocks in modern multimodal AI.
How it works
CLIP has two encoders: an image encoder (a Vision Transformer or ResNet) and a text encoder. During training it uses a contrastive objective: given a batch of image-text pairs, it pulls each image's embedding close to its matching caption and pushes it away from non-matching ones. The result is a shared space where similarity between an image and a piece of text is just a dot product. To classify zero-shot, you embed candidate labels as text (e.g. 'a photo of a cat') and pick the one closest to the image.
What it is good at
CLIP excels at zero-shot classification, image-text retrieval and semantic image search — find images by describing them, or tag images against an open-ended label set without training. Its embeddings are widely reused: they guide and evaluate image-generation models, power content moderation and de-duplication, enable multimodal search, and serve as the vision backbone for many later vision-language systems. Its flexibility and reusability are its defining strengths.
Licensing & access
CLIP is open source (OpenAI released it under the MIT licence), with weights and code available, plus the popular community project OpenCLIP offering many additional models trained on open datasets like LAION. It is supported in Hugging Face Transformers and runs on a single GPU (small variants even on CPU). A range of sizes (ViT-B, ViT-L and larger) trade speed for accuracy.
Practical considerations
CLIP is an understanding/embedding model, not a generator — it scores and matches but does not produce images or captions on its own (though it guides models that do). Zero-shot accuracy depends on good prompt phrasing for the text labels, and CLIP can reflect biases and gaps in its web training data, so evaluate on your task. For best results pick an appropriate model size and consider OpenCLIP variants trained on larger open datasets.
How it compares
ERNIE-ViL adds structured scene-graph knowledge to vision-language pretraining; CogVLM and LLaVA-NeXT are generative multimodal assistants. CLIP is different and more fundamental: a contrastive embedding model whose strength is flexible matching, retrieval and zero-shot recognition, and which serves as a backbone inside many of those newer systems. For embeddings, search and zero-shot classification, CLIP (or OpenCLIP) is the standard tool.
Getting started
Load CLIP from Hugging Face or OpenCLIP, embed your images and candidate text labels, and compare them by similarity to classify or search in a few lines. For zero-shot classification, phrase labels as descriptive prompts; for search, embed a query and rank images by closeness. Pick a model size for your accuracy/speed needs, consider OpenCLIP's larger-dataset variants, and evaluate prompts and bias on your own data.
