You cannot exactly say which algorithm should be used for fraud detection in terms of accuracy and consistency in data mining. It depends upon your business or task you want to accomplish. Suppose, you want to restrict it to some fixed condition and mark the transaction as “fraud” or “genuine” then you can use logistic regression.
So, you must learn Machine Learning concepts and for that, you should start by this wonderful Artificial Intelligence tutorial:
You can check out Fraud Detection Algorithms for more information.
If you want the algorithm to respond to larger conditions of fraud with greater accuracy then you might go for neural networks. For knowing more about machine learning algorithms which can be used in fraud detection you can watch this Machine Learning tutorial:
Thank You!!
Hope this answer helps!!