How can I apply reinforcement learning to continuous action spaces?

Question

I'm trying to get an agent to learn the mouse movements necessary to best perform some task in a reinforcement learning setting (i.e. the reward signal is the only feedback for learning).

I'm hoping to use the Q-learning technique, but while I've found a way to extend this method to continuous state spaces, I can't seem to figure out how to accommodate a problem with a continuous action space.

I could just force all mouse movement to be of a certain magnitude and in only a certain number of different directions, but any reasonable way of making the actions discrete would yield a huge action space. Since standard Q-learning requires the agent to evaluate all possible actions, such an approximation doesn't solve the problem in any practical sense.

Answer 1

The common way of dealing with this problem is with actor-critic methods. These naturally extend to continuous action spaces.

Reinforcement Learning often considers real-world domains which deal with continuous state and action spaces to obtain optimal rewards. Many solutions are applied on RL Algorithms to solve continuous state problems, the same techniques can be hardly extended to continuous action spaces, where, besides the computation of a good approximation of the value function, a fast method for the identification of the highest-valued action is needed. 

However, it can be novel by using the actor-critic approach in which the policy of the actor is estimated through sequential Monte Carlo methods as proposed in “Reinforcement Learning in Continuous Action Spaces through Sequential Monte Carlo Methods”

In this paper, learn the problem statement with a controller to drive a boat from the left bank to the right bank of a river, with a strong non-linear current. The Proposed SMC Algorithm and Continuous Q-learning are combined to measure the results obtained in a control problem consisting of steering a boat across a river.

Thus, for more details, Reinforcement Learning will be quite important as far as the software domain is concerned. Also, Machine Learning Course also helps in making a desirable career.