Generally, multimodal LLM-s are really good in understanding what's on the screen and what it means. But they don't seem to be really good in terms of grasping exact image layout, small displacements etc. So we'll explore in this post where the problem lies and what alternatives there are.
Contemporary vision models are really good at processing visual information so naturally using one in any scenario connected with images seems logical, especially given the diversity of questions we could ask about an image, like "Is this the checkout page?", "Did it break?", "How many errors are on the screen?", or "Are these two screenshots the same?".
But not all of these are good questions to ask. Some of them work well with the model but "are these two the same" is probably one of the worst you can come up with, and its failure even resembles success.
In this post we will show how this failure looks like with a comparison of 2 screenshots, explain why it happens, and provide an alternative that might not be perfect but works in a more reliable way.
Can a vision model find differences between two screenshots?
For the most direct experiment you can just show the LLM-s 2 screenshots and ask "are they identical?". Short version: it catches a big, obvious change but misses a small one — and it sounds just as sure when it's wrong. To check it we prepared simple analytics app with a few KPI cards and the campaigns table, rendered headless on a fixed size with our open-source cravy. Then we generated three versions:
the same as the first one,
one with changed number (conversions 196 → 186),
one where the value is moved by 4px to the right.
Screenshot 1 — the baseline.
We asked a current frontier vision model (September 2026) if these are the same. The question was "are they the same and where?", no tooling involved, just asking by looking at them. Four runs per pair:
196 → 186: found it in all runs, pointing to the exact cell; good as this is a high-contrast discrete text
identical: correctly told that in all runs, saying they are identical every time and not even trying to come up with sth to say to satisfy us; one of them actually said it would rather tell us what it sees than make sth up
4px shift: missed in all runs, saying it's the same in all of them; two of them additionally confirmed position and alignment so very certain about its false negative
As you can see, the model is not very good at finding differences between screenshots. It can point a bigger, more distinctive change but struggles with smaller ones (even if it says it checked and is certain about the verdict).
This makes using it as a tool for finding changes unreliable as it's not possible to distinguish such cases from right ones.
Screenshot 2 — one changed number: Newsletter Drive conversions, 196 → 186.
Does the model catch a real layout shift, not just a changed number?
The 196 → 186 change was just a number. But the layout shift is the interesting one — we moved one value by 4px to the right and this is what we actually wanted to achieve with visual regression testing — finding layout bugs in a production app.
Screenshot 3 — the Churn value (1.9%) moved 4px to the right. Everything else is identical to the baseline.
But as you can see, the model missed it even though:
it was a real change (798 changed pixels), and
it was a meaningful change (the element was moved so there's a layout bug).
What's more the model was certain about its false negative, so if it were in CI it would have green-flagged a broken build.
Notably though, we did a similar comparison earlier but had access to terminal so we tried using ImageMagick for pixel diffs. The model did it in this way in 7 out of 8 runs:
creating the diff,
calculating checksum,
comparing it with the previous one.
Only then it checked the diff visually. So when it has the option, the model itself treats image equality as something to compute rather than something to see — which is the right call.
Why is the model blind to it?
It fails because images are processed in a different way by models — they're being read to understand their semantic layer. This way the model turns an image into thousands of tokens, thinking about it, but at the same time loses the information about its geometry (exact coordinates, small offsets, faint colour differences, precise counts).
In other words, it's good where natural language is good (identity, content, purpose) and bad where a ruler or calculator are needed.
That's not only our opinion though — these are actual measures:
Vision language models are blind (Rahmanzadehgervi et al., ACCV 2024): they checked the tasks a child can do (circle overlap, line-crossing counts, which letter is circled) with four top models and it turned out that they score 58% on average (with humans at around 100%), even Claude 3.5 Sonnet, the best model at the time, scored only around 78%. These are the figures from 2024 for the frontier models.
BLINK (Fu et al., ECCV 2024): they tested 14 classic vision tasks including spatial relations, correspondence, depth, jigsaw and so on with humans and top vision models — humans scored 95.7%, GPT-4V 51.3% and Gemini Pro 45.7% (barely above random, under it for some of the tasks). Keep in mind though that 2025 models perform much better on these tasks so you can treat this number from 2024 as a starting point rather than the current state but the shape of the weakness is still there.
VLMCountBench (2025): they checked how counting works when the number of types increases and it turned out that it doesn't work well — GPT-4o got 44% to 23%, Gemini 2.5 Flash 58% to 30%. So "how many X-s are on the screen?" is not a great question.
ScreenSpot-Pro (2025): they checked how well general models work in terms of clicking on high-resolution professional UIs and it turned out that even for GPT-4o it works under 2% of times.
CharXiv (Wang et al., NeurIPS 2024): they measured how good vision models are in terms of chart reasoning and found out that GPT-4o scores 47% compared to 80.5% for humans; the best model now is at around 60% so still 20 percentage points behind.
Even OpenAI are aware of this — their own GPT-4V system card flags problems with spatial locations and colour, and warns it shouldn't be used for high-risk or precision-critical tasks. The model is better now but the shape of this limitation hasn't changed.
These are all real-life screens — a lot of near-duplicates elements, small deltas, position and alignment, dense text, faint colour differences. The demo where there's one big change on an uncluttered page is actually the least typical case.
When to give an image to a model?
If you want to ask a model about an image you should only do it when you're interested in its semantics, never if you want to measure sth — if a ruler, calculator, diff or grep could answer it just don't use the model. Here's the best tool for a given question:
| You want to know… | Reach for… |
|---|---|
| Did anything change between two screens? | A deterministic image diff — pixelmatch, Playwright snapshot assertions, BackstopJS |
| What does this screen say / is this the right page / does the error read correctly? | A vision model — its strength |
| What's the value or label in a specific cell? | OCR (Tesseract, PaddleOCR), then string comparison |
| Where is an element, is it aligned, did it move? | The DOM instead of a screenshot — layout assertions, or let the LLM read it — or a purpose-built grounding model |
| How many X are on the screen? | Object detection (YOLO, Grounding DINO), then count |
| Is this already-located change acceptable? | A vision model — given the one region and minimal context |
Using pixel diffs is also not perfect as they can sometimes produce false positives but these are more visible and easier to address by excluding noisier areas. Here, the model's failure is much more silent, it's certain and points to the very change that was actually crucial.
How to properly integrate a vision model into a visual check?
If you really need a model to decide if a change is acceptable rather than if it happened, the architecture is even more important than the model. If you need an LLM to find differences between two screenshots, here's what you can do:
Use a deterministic image diff (pixel or structural) to understand if and where there's a change; if there's no change — don't call the model. If something is detected, get only the changed part of the image.
Make it not possible for the model to say "there's no difference"; the failure isn't only missing a change, it's being certain that there's none. If the deterministic layer detected a change, make the model confront its outcome with the ground truth or explain why it can't.
Don't give it the context it doesn't need — previous conversation or additional frames. In most cases isolating the comparison step increases the quality of the comparison more than prompt tweaks.
Describe the images and decide in separate steps instead of asking about them all together.
Don't count on switching models to save you — moving the same naive "are these the same?" prompt from one frontier model to another barely changes anything. The gains are in the architecture; the perceptual ceiling is roughly shared.
The bottom line is that when you need an LLM to diff two screenshots it's all about the architecture. Split things into separate tasks and direct them to specialised tools — detector for locating and counting, OCR for text, diff for changes, model for understanding what a screen says; two experts are more effective than one generalist.
That's just engineering practice for a tool that's really good at one thing and not very good in another — give it the sub-task it's good at and guard the rest.
In other words, models are decent screens interpreters but not good diffs, rulers or counters. If a workflow needs these they become a liability because of their certainty.
Demos always work
Even though the model has recognised what's on the screen, it wasn't able to spot that one element is shifted by 4px. Models will always work well on demos so remember to validate against pixel diff before shipping your solution. The experiment was run with the September 2026 model; all benchmarks are for the models used in the respective papers and reflect their state at that point given they change.
