Comparison of texts
Let's say we have three texts: two of them about dogs and one about cats. How do you compare them with each other?
We can count how many each word occurs in the text, in our case we will count cats and dogs, and if there are more dogs than cats in the text, then we can conclude that they (texts) are about the same thing.
In fact, this is not always the case. Imagine a situation that there is a very long text about dogs and there are more words in it. Fortunately, you can get out of this situation by comparing the cosine distances.
bag-of-words , , , «» «» . , , . , ? . bag-of-words , «» «» .
. , , . , «» «». – . python : pymorphy pymystem. , . .
. , , .
, . , «» — «» — «», , «» «», «».
Word2Vec
, . , Word2Vec. ? , . , , . Word2Vec :
, .. . , , . 10 . .
, , , IT, .. . ? () «». «» «» , , , , .
, pyLDAvis , .
, . , . . udpipe.
, , : « ». , , «».
, udpipe , , .
Despite the fact that all tools were considered using simple examples, NLP has a wide range of tasks to be solved: classifying employee requests, evaluating customer reviews, analyzing messages from a chatbot. Thus, several more instruments appeared in our hands.