weight_change = learning_rate * input * output

Question

This is a beginner question on regularization with regression. Most information about Elastic Net and Lasso Regression online replicates the information from Wikipedia or the original 2005 paper by Zou and Hastie (Regularization and variable selection via the elastic net).

Resource for simple theory? Is there a simple and easy explanation somewhere about what it does, when and why regularization is necessary, and how to use it - for those who are not statistically inclined? I understand that the original paper is the ideal source if you can understand it, but is there somewhere that more simply the problem and solution?

How to use in sklearn? Is there a step by step example showing why the elastic net is chosen (over the ridge, lasso, or just simple OLS) and how the parameters are calculated? Many of the examples on sklearn just include alpha and rho parameters directly into the prediction model, for example:

from sklearn.linear_model import ElasticNet

alpha = 0.1

enet = ElasticNet(alpha=alpha, rho=0.7)

y_pred_enet = enet.fit(X_train, y_train).predict(X_test)

However, they don't explain how these were calculated. How do you calculate the parameters for the lasso or net?

Answer 1

In an elastic net implementation, to automatically tune the value of alpha, it is indeed possible to use ElasticNetCV, which will spare excess computation as opposed to using GridSearchCV. 

You can use a regular GridSearchCV for finding the optimal value of rho. 

ElasticNet will tend to select more variables hence lead to larger models, but they are more accurate generally. 

Lasso is quite sensitive to the correlation between features, that’s why it might select randomly. one out of 2 very correlated informative features while ElasticNet is more likely to select both which should lead to a more stable model.

Hope this answer helps.