Explore Courses Blog Tutorials Interview Questions
0 votes
in R Programming by (5.3k points)

I am running into issues trying to use large objects in R. For example:

> memory.limit(4000)

> a = matrix(NA, 1500000, 60)

> a = matrix(NA, 2500000, 60)

> a = matrix(NA, 3500000, 60)

Error: cannot allocate vector of size 801.1 Mb

> a = matrix(NA, 2500000, 60)

Error: cannot allocate vector of size 572.2 Mb # Can't go smaller anymore

> rm(list=ls(all=TRUE))

> a = matrix(NA, 3500000, 60) # Now it works

> b = matrix(NA, 3500000, 60)

Error: cannot allocate vector of size 801.1 Mb # But that is all there is room for

I understand that this is related to the difficulty of obtaining contiguous blocks of memory

Error messages beginning cannot allocate vector of size indicate a failure to obtain memory, either because the size exceeded the address-space limit for a process or, more likely, because the system was unable to provide the memory. Note that on a 32-bit build there may well be enough free memory available, but not a large enough contiguous block of address space into which to map it.

How can I get around this? My main difficulty is that I get to a certain point in my script and R can't allocate 200-300 Mb for an object... I can't really pre-allocate the block because I need the memory for other processing. This happens even when I dilligently remove unneeded objects.

1 Answer

0 votes

To allocate memory to a very large object, you can try the following steps:

  • Try creating the matrix using the Matrix package particularly for sparse matrices.

For example:

m1 <- matrix(0, nrow = 10000, ncol = 10000)

> m2 <- Matrix(0, nrow = 10000, ncol = 10000, sparse = TRUE)

> object.size(m1)

800000216 bytes

> object.size(m2)

41728 bytes

  • Use the gc() function to clear unused memory or create the object of your interest in a separate R session.
  • You can install more RAM and allocate to R using memory.limit() function  in a 64-bit Windows version.
  • You can use any remote computing online service to complete your task.
Welcome to Intellipaat Community. Get your technical queries answered by top developers!

30.5k questions

32.6k answers


108k users

Browse Categories