The cost function in Machine Learning is a measure of the performance of a model when it works on the input data. It is used as a quantifier that puts forth the amount of error between the predicted value and the expected value. The cost function is minimized when the requirement is to find the value of a parameter. The cost function can also be maximized, and in this case, it is considered to be called a reward. The direct estimation of how far the variance is in between the input and the output variables is what makes the usage of cost functions vital in the working of any Machine Learning model.
If you are looking for an online course to learn Machine Learning, I recommend this Machine Learning Certification program by Intellipaat.