In a 32-bit Windows R, you can only use up to 3Gb of available RAM.
In the 64-bit version, you can use to the maximum size of your RAM.
To check the current size of memory available to R, run the following:
memory.limit()
#OR
round(memory.limit(),2)
To increase the memory in a 64-bit Windows R version, run the following:
memory.limit(size=25000)
You can use any size greater than the current size, if there is more RAM available otherwise, you can always buy more RAM.