Vector Semantics and Embeddings

How can a computer represent the meaning of a word? Vector semantics answers this by representing a word as a point in a high-dimensional space, derived from the contexts in which it appears. The idea is load-bearing for modern natural language processing: virtually every language model begins by mapping tokens to vectors — an embedding.

This chapter develops the classical pipeline of vector semantics:

1. Words and vectors — representing words and documents through co-occurrence matrices (term-document and term-term matrices).

2. TF-IDF — re-weighting raw counts so that words which are frequent in a document but rare across documents dominate, with an application that builds a small movie recommender.

3. Cosine similarity — measuring the closeness of two vectors by the angle between them, with an implementation and an application to word similarity.

The treatment follows closely Jurafsky and Martin [Jurafsky and Martin, 2022].

Table of Contents

• Words and Vectors
  • Concept: Words and Vectors
• Term Frequency-Inverse Document Frequency (TF-IDF)
  • TF-IDF Application: IMDB Recommender System
• Cosine Similarity and Notion of Closeness
  • Concept: Cosine Similarity
  • Cosine Similarity Implementation
  • Cosine Similarity Application: Word Similarity

References and Further Readings

• Jurafsky, Dan, and James H. Martin. “Chapter 6. Vector Semantics and Embeddings.” In Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition. Pearson, 2022.