R memory management / cannot allocate vector of size n Mb

Question

I am running into issues trying to use large objects in R. For example:

> memory.limit(4000)

> a = matrix(NA, 1500000, 60)

> a = matrix(NA, 2500000, 60)

> a = matrix(NA, 3500000, 60)

Error: cannot allocate vector of size 801.1 Mb

> a = matrix(NA, 2500000, 60)

Error: cannot allocate vector of size 572.2 Mb # Can't go smaller anymore

> rm(list=ls(all=TRUE))

> a = matrix(NA, 3500000, 60) # Now it works

> b = matrix(NA, 3500000, 60)

Error: cannot allocate vector of size 801.1 Mb # But that is all there is room for

I understand that this is related to the difficulty of obtaining contiguous blocks of memory

Error messages beginning cannot allocate vector of size indicate a failure to obtain memory, either because the size exceeded the address-space limit for a process or, more likely, because the system was unable to provide the memory. Note that on a 32-bit build there may well be enough free memory available, but not a large enough contiguous block of address space into which to map it.

How can I get around this? My main difficulty is that I get to a certain point in my script and R can't allocate 200-300 Mb for an object... I can't really pre-allocate the block because I need the memory for other processing. This happens even when I dilligently remove unneeded objects.

Answer 1

To allocate memory to a very large object, you can try the following steps:

• Try creating the matrix using the Matrix package particularly for sparse matrices.

For example:

m1 <- matrix(0, nrow = 10000, ncol = 10000)

> m2 <- Matrix(0, nrow = 10000, ncol = 10000, sparse = TRUE)

> object.size(m1)

800000216 bytes

> object.size(m2)

41728 bytes

• Use the gc() function to clear unused memory or create the object of your interest in a separate R session.

• You can install more RAM and allocate to R using memory.limit() function  in a 64-bit Windows version.

• You can use any remote computing online service to complete your task.