0 votes
1 view
in Machine Learning by (32.3k points)
I am having a hard time understanding the concept of Tokenization in NLP, can anyone please help me with that?

Also, can anyone provide me an example on Sentiment Analysis using NLTK?

1 Answer

0 votes
by (16.8k points)

Tokenization is a very common task in NLP, it is basically a task of chopping a character into pieces, called as token, and throwing away the certain characters at the same time, like punctuation. 

If I speak in a common term, it is just to split apart the text into the individual units, and each individual unit, should have a value associated with it.

For an example, if you Input like this, 

Input: World, Americans, Countrymen, Borrow.



Here is an informative end-to-end tutorial on Natural language processing, covering the concept of Tokenization, and with an implementation of Sentiment Analysis using NLTK, and that is not it, since it is an end-to-end video, you will also learn the Components of NLP, Natural Language Understanding, Natural Language Generation, Packages of NLP, Uni-gram, Bi-gram, Tr-gram, stemming, lemmatization, parts of speech tagging, named entity recognition and many more, make sure to follow this video, it will be very helpful.

Welcome to Intellipaat Community. Get your technical queries answered by top developers !