How to calculate TF*IDF for a single new document to be classified?

Question

I am using document-term vectors to represent a collection of document. I use TF*IDF to calculate the term weight for each document vector. Then I could use this matrix to train a model for document classification.

I am looking forward to classifying a new document in the future. But in order to classify it, I need to turn the document into a document-term vector first, and the vector should be composed of TF*IDF values, too.

My question is, how could I calculate the TF*IDF with just a single document?

As far as I understand, TF can be calculated based on a single document itself, but the IDF can only be calculated with a collection of document. In my current experiment, I actually calculate the TF*IDF value for the whole collection of documents. And then I use some documents as the training set and the others as the test set.

I just suddenly realized that this seems not so applicable to real life.

ADD 1

So there are actually 2 subtly different scenarios for classification:

1. to classify some documents whose content are known but label is not known.
2. to classify some totally unseen document.

For 1, we can combine all the documents, both with and without labels. And get the TF*IDF over all of them. This way, even we only use the documents with labels for training, the training result will still contain the influence of the documents without labels.

But my scenario is 2.

Suppose I have the following information for term T from the summary of the training set corpus:

• document count for T in the training set is n
• total number of training documents is N

Should I calculate the IDF of t for an unseen document D as below?

IDF(t, D)= log((N+1)/(n+1))

ADD 2

And what if I encounter a term in the new document which didn't show up in the training corpus before? How should I calculate the weight for it in the doc-term vector?

Answer 1

TF-IDF does not add up for one document, independent of a corpus. It's fundamentally about emphasizing relatively rare and informative words.

You need to stay corpus summary data so as to calculate TF-IDF weights. In specific, you need the document count for each term and the total number of documents.

Whether you would like to use summary data from the full training set and test set for TF-IDF, or for just the training set is a matter of your problem formulation. If it is the case that you just solely care to use your classification system to documents whose contents you've got, but whose labels you do not have (this is actually pretty common), then using TF-IDF for the entire corpus is okay. If you would like to use your classification system to thoroughly unseen documents once you train, then you only want to use the TF-IDF summary information from the training set.

For unseen words, TF calculation is not a problem as TF is a document specific metric. While computing IDF, you can use the smoothed inverse document frequency technique.

IDF = 1 + log(total documents / document frequency of a term)

Here the lower bound for IDF is 1. So if a word isn't seen within the training corpus, its IDF is 1. Since there is no universally agreed single formula for computing tf-idf or even idf, your formula for tf-idf calculation is also reasonable.

Note that, in many cases, unseen terms are ignored if they don't have much impact on the classification task. Sometimes, individuals replace unseen tokens with a special symbol like UNKNOWN_TOKEN and do their computation.

Alternative of TF-IDF: Another way of computing the weight of each term of a document is using the Maximum Likelihood Estimation. While computing MLE, you can smooth using an additive smoothing technique which is also known as Laplace smoothing. MLE is used in case you are using Generative models like the Naive Bayes algorithm for document classification.

For a more holistic approach study the Machine Learning Courses as well.