TPIPS: Cracking the Code of Human Visual Similarity

Our brains effortlessly judge image similarity across multiple dimensions, but AI has long struggled to keep up. A new study introduces TPIPS (Text-Prompted Image Perceptual Similarity), exposing just how far behind our machines really are, and offering a path forward.

The problem? Current AI treats visual similarity as a one-dimensional score. But human perception is far more nuanced. We might see two images as similar in shape but wildly different in color, or sharing a theme while diverging in style. Traditional metrics flatten these distinctions, losing critical information.

To tackle this, researchers built a dataset that captures the multifaceted nature of human similarity judgments. They didn't just ask 'Are these similar?' Instead, they collected free-form descriptions of how images relate across various semantic aspects.

The results were sobering. When benchmarked against this dataset, even advanced vision-language models (VLMs) fell woefully short of human-level understanding.

Enter TPIPS. By fine-tuning a VLM on this rich dataset, the researchers created a metric that adapts its similarity assessment based on text prompts. Want to find images with matching color palettes but different subjects? TPIPS can do that. Need thematic similarities regardless of visual style? It's got you covered.

This flexibility enables new possibilities:

1. Text-guided image retrieval: Find visually similar images based on specific criteria.
2. Compositional search: Combine multiple similarity aspects for precision.
3. Fine-grained generative model evaluation: Assess AI-created images across various perceptual dimensions.

TPIPS represents a significant step towards AI systems that can reason about images in ways that align with human perception. However, questions remain. Can the dataset truly capture cultural and individual variations in similarity judgments? How will TPIPS perform on edge cases or adversarial examples?

The researchers have open-sourced their code, data, and models, inviting further exploration. As we push towards AI that genuinely understands the visual world, TPIPS offers both a valuable tool and a humbling reminder of how far we still have to go in replicating the nuanced perceptual abilities we take for granted.