Word2vec

word2vec Stay organized with collections Save and categorize content based on your preferences. View on TensorFlow.org Run in Google Colab View on GitHub Download notebook Continuous bag-of-words model : predicts the middle word based on surrounding context words. The context consists of a few words before and after the current (middle) word. This architecture is called a bag-of-words model as the order of words in the context is not important. Continuous skip-gram model : predicts words within a certain range before and after the current word in the same sentence. A worked example of this is given below. You'll use the skip-gram approach in this tutorial. First, you'll explore skip-grams and other concepts using a single sentence for illustration. Next, you'll train your own word2vec model on a small dataset. This tutorial also contains code to export the trained embeddings and visualize them in the TensorFlow Embedding Projector . where c is the size of the training context. The basic skip-gram formulation defines this probability using the softmax function. Computing the denominator of this formulation involves performing a full softmax over the entire vocabulary words, which are often large (105-107) terms. The noise contrastive estimation (NCE) loss function is an efficient approximation for a full softmax. With an objective to learn word embeddings instead of modeling the word distribution, the NCE loss can be simplified to use negative sampling. The simplified negative sampling objective for a target word is to distinguish the context word from num ns negative samples drawn from noise distribution Pn(w) of words. More precisely, an efficient approximation of full softmax over the vocabulary is, for a skip-gram pair, to pose the loss for a target word as a classification problem between the context word and num ns negative samples. A negative sample is defined as a (target word, context word) pair such that the context word does not appear in the window size neighborhood of the target word . For the example sentence, these are a few potential negative samples (when window size is 2 ). In the next section, you'll generate skip-grams and negative samples for a single sentence. You'll also learn about subsampling techniques and train a classification model for positive and negative training examples later in the tutorial. 2024-07-19 11:19:50.266209: E external/local xla/xla/stream executor/cuda/cuda fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered 2024-07-19 11:19:50.287751: E external/local xla/xla/stream executor/cuda/cuda dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered 2024-07-19 11:19:50.294095: E external/local xla/xla/stream executor/cuda/cuda blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered sentence = "The wide road shimmered in the hot sun" tokens = list(sentence.lower().split()) print(len(tokens)) inverse vocab = {index: token for token, index in vocab.items()} print(inverse vocab) The tf.keras.preprocessing.sequence module provides useful functions that simplify data preparation for word2vec. You can use the tf.keras.preprocessing.sequence.skipgrams to generate skip-gram pairs from the example sequence with a given window size from tokens in the range [0, vocab size) . window size = 2 positive skip grams, = tf.keras.preprocessing.sequence.skipgrams( example sequence, vocabulary size=vocab size, window size=window size, negative samples=0) print(len(positive skip grams)) The skipgrams function returns all positive skip-gram pairs by sliding over a given window span. To produce additional skip-gram pairs that would serve as negative samples for training, you need to sample random words from the vocabulary. Use the tf.random.log uniform candidate sampler