Explore Courses Blog Tutorials Interview Questions
0 votes
in Machine Learning by (45.3k points)
I am having a hard time understanding the concept of Tokenization in NLP, can anyone please help me with that?

Also, can anyone provide me an example on Sentiment Analysis using NLTK?

1 Answer

0 votes
by (16.8k points)

Tokenization is a very common task in NLP, it is basically a task of chopping a character into pieces, called as token, and throwing away the certain characters at the same time, like punctuation. 

If I speak in a common term, it is just to split apart the text into the individual units, and each individual unit, should have a value associated with it.

For an example, if you Input like this, 

Input: World, Americans, Countrymen, Borrow.



Here is an informative end-to-end tutorial on Natural language processing, covering the concept of Tokenization, and with an implementation of Sentiment Analysis using NLTK, and that is not it, since it is an end-to-end video, you will also learn the Components of NLP, Natural Language Understanding, Natural Language Generation, Packages of NLP, Uni-gram, Bi-gram, Tr-gram, stemming, lemmatization, parts of speech tagging, named entity recognition and many more, make sure to follow this video, it will be very helpful.

Related questions

0 votes
1 answer
0 votes
1 answer
0 votes
1 answer
asked Mar 5, 2020 in AI and Deep Learning by Sudhir_1997 (55.6k points)
0 votes
1 answer
Welcome to Intellipaat Community. Get your technical queries answered by top developers!

30.4k questions

32.5k answers


108k users

Browse Categories