Natural Language Processing has become a crucial area in our knowledge society. In this field, word embeddings have meant a significant advance. This technique has improved web searches, sentimental analysis, and content suggestions. However, what are word embeddings? How can we use them for our projects?
Let’s learn more about this technique in this post.
What are word embeddings?
Word embeddings are vector representations of words. In this technique, the position of the words is important. Let’s imagine a cloud of words. The placement of the words in this cloud determines their meaning. Banana and apple are in the same section of the cloud. Meanwhile, giraffe and elephant are in another section. This structure was developed by computer scientists, and, in this way, they were able to do arithmetic with words.
Arithmetic with words? It sounds weird at the beginning, but the results are interesting. For example, subtracting man to king gives a similar result to subtracting woman to queen. Meanwhile, adding green to transport can give us public transport, bicycle or urban transport as result.
A brief history of word embeddings
The term word embeddings was coined by Bengio et al in 2003 in the paper A Neural Probabilistic Language Model, published in the Journal of Machine Learning Research. However, they didn’t invent the technique. In distributional semantics, vector space models have been used since the 1990s.
In 2008, Collobert and Weston demonstrate the power of the technique in their paper A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning. They showed that word embeddings were a powerful tool in downstream tasks and developed a neural network architecture used for many today.
In 2013, Mikolov et al. created the toolkit word2vec, and a year later, Pennington et al. created GloVe, a set of pre-trained embeddings. Since then, this technique have become popular for applications of unsupervised learning tasks.
How can this technique be used?
Some practical uses of word embeddings are:
- Analyzing Survey Responses
- Analyzing Verbatim Comments
- Music/Video Recommendation System
Developers can use libraries for training models such as:
- Spacy: an NLP library for Python with word embedding models built in.
- Gensim: Topic modeling library for Python with access to Word2Vec and other word embedding algorithms for training. It allows pre-trained word embeddings.
Resources for learning
For mastering word embeddings, developers can check the next resources:
- Tensor Flow: A tutorial that contains an introduction to word embedding. In this tutorial, developers can train their word models using a simple Keras model for a sentiment classification task.
- Lil’log: For those who want to discover the theory behind word embedding, this resource is a great starting point. They are the learning notes of Lilian Weng, an applied AI reasearcher.
- Coursera: As part of the Deep Learning Specialization, there’s a lesson focus on word embedding in Coursera.
- Pathmind: A Beginner’s Guide to Word2Vec and Neural Word Embeddings.
For more visual learners, some tutorials explain word embedding. Check this one:
Was this post useful? Share it!