Tokenization in NLP!

Question

I am having a hard time understanding the concept of Tokenization in NLP, can anyone please help me with that?

Also, can anyone provide me an example on Sentiment Analysis using NLTK?

Answer 1

Tokenization is a very common task in NLP, it is basically a task of chopping a character into pieces, called as token, and throwing away the certain characters at the same time, like punctuation. 

If I speak in a common term, it is just to split apart the text into the individual units, and each individual unit, should have a value associated with it.

For an example, if you Input like this, 

Input: World, Americans, Countrymen, Borrow.

Output:

Here is an informative end-to-end tutorial on Natural language processing, covering the concept of Tokenization, and with an implementation of Sentiment Analysis using NLTK, and that is not it, since it is an end-to-end video, you will also learn the Components of NLP, Natural Language Understanding, Natural Language Generation, Packages of NLP, Uni-gram, Bi-gram, Tr-gram, stemming, lemmatization, parts of speech tagging, named entity recognition and many more, make sure to follow this video, it will be very helpful.