Datalad tool

Embedding Similarity Explorer

Retrieval comes down to turning text into numbers and measuring the angle between them. This does that in front of you: real vectors, real cosine similarity, and the per word arithmetic behind every score. It also shows the exact point where matching on words alone falls over, which is the reason dense embeddings exist at all.

1

Your documents and your query

The vectors are TF-IDF, built with the same formula scikit-learn uses, so the numbers here are the numbers you would get in a notebook. The geometry is identical to the geometry a dense embedding model uses. What differs is where the numbers come from, and that difference is the whole point of the example loaded below.

2

How the documents rank

Three ways of measuring the same vectors. They do not agree, and knowing why is most of what there is to know about retrieval scoring.

3

Where the top score came from

4

What this tells you

View Comments (5)

Leave a Reply

Subscribe to My Newsletter

Subscribe to my email newsletter to get the latest posts delivered right to your email. Pure inspiration, zero spam.

Discover more from Discuss Data Science, Machine Learning and Analytics

Subscribe now to keep reading and get access to the full archive.

Continue reading