The translator-pair analogy
Imagine two specialists in a room. One reads images and only speaks vector. The other reads vectors and writes English. To answer a question about a photo, the first looks at the picture and writes out a long vector summary; the second reads that summary plus your question and writes the answer.
A vision-language model (VLM) is exactly that pair. The translator is a small projection layer that maps "image vectors" into the same space the language model already speaks. Once you have that bridge, the LLM treats the image as if it were a paragraph of text.
The standard architecture
- Vision encoder — usually a Vision Transformer (ViT). Splits an image into patches (e.g. 14×14 pixels each), runs each through Transformer blocks, outputs a sequence of patch embeddings.
- Projection / adapter — a small MLP or set of learnable queries that maps vision embeddings into the language model's token space. This is where most of the "vision-language" specific training happens.
- Language model — a regular LLM. It receives
[image-tokens] [text-tokens]as input and generates text autoregressively.
The whole image becomes ~256–4096 "tokens" that take up context window like text does.
Why it works at all
The vision encoder is trained on massive image-text pairs (CLIP-style or similar). It learns to produce embeddings that sit near the embeddings of captions describing the image. Once those vision embeddings live near text-embeddings in the same space, an LLM can be coaxed to read them with a small adapter and not much else.
Two-stage training is common:
- Stage 1 — alignment — freeze the LLM and the vision encoder, train only the adapter.
- Stage 2 — instruction tuning — unfreeze (often with LoRA), fine-tune on multimodal instruction data so the model answers visual questions, follows visual instructions, etc.
What VLMs are good at
- OCR-light tasks — reading menus, screenshots, charts.
- Visual question answering — "what is the woman holding?", "is this code Python?"
- Document understanding — parse a screenshot of a PDF table.
- UI agents — see a screen, plan a click.
What they're still bad at
- Precise spatial reasoning — "is the cat to the left or right of the chair?"
- Counting — many objects in the same image trip them up.
- Tiny text / fine detail — depends heavily on input resolution; many VLMs downscale aggressively.
- Reading between the lines — implicit relationships, sarcasm in memes.
Resolution is destiny
A 224×224 input gives the encoder very little to work with — illegible text, blurred faces. Modern VLMs use either higher native resolution (672×672, 896×896) or a tiling scheme where the image is split into multiple crops, each encoded separately, then concatenated. Both approaches expand the effective token count significantly.
If your task is reading screenshots or documents, resolution is the first thing to check, before model size.
Practical engineering tips
- Compress images sensibly. Send only what the model needs to see; oversized inputs blow context budget.
- Match training resolution when you can. A 1024×1024 PNG sent to a model trained at 336×336 is downsampled lossy.
- Pair with OCR for documents. Hybrid pipelines (OCR → text → LLM) often beat pure VLMs on dense pages.
- Test on your domain. General VLM benchmarks rarely predict performance on UI screenshots, schematics, or medical scans.