How to approach machine learning problems with high dimensional input space?

Question

How should I approach a situation when I try to apply some ML algorithms (classification, to be more specific, SVM in particular) over some high dimensional input, and the results I get are not quite satisfactory?

1, 2 or 3-dimensional data can be visualized, along with the algorithm's results, so you can get the hang of what's going on, and have some idea how to approach the problem. Once the data is over 3 dimensions, other than intuitively playing around with the parameters I am not really sure how to attack it?

Answer 1

In your case, you made no changes in data. The SVMs are designed to handle high-dimensional data. There is no impact of dimensionality reduction prior to classification. Preprocessing the data features using PCA/LDA can not significantly increase classification accuracy of the SVM.

You should simply change your SVM parameters. For example, with libsvm link, the parameters C and gamma are crucially important to classification success:

1. Scale your features before classification.
2. Try to obtain balanced classes. If impossible, then penalize one class more than the other. 
3. Check the SVM parameters. Try other combinations to arrive at the best one.
4. Use the RBF kernel first. It almost always works best.
5. Perform Cross validation.

Hope this answer helps you!

These problems are normally asked in Machine Learning Interview Questions. Thus for this, students and aspirants are always advised to go through Machine Learning Course.