There are multiple memory allocators in TensorFlow, so that memory can be used in different ways. Their behavior has some adaptive aspects.
For GPU users, there is a PoolAllocator for CPU memory that is pre-registered with the GPU for fast DMA. A tensor that is expected to be transferred from CPU to GPU, e.g., will be allocated from this pool.
The main working concept of PoolAllocators is to decompose the cost of calling a more expensive underlying allocator by keeping around a pool of allocated then freed chunks that are eligible for immediate reuse. Their default approach is to grow slowly until the removal rate drops below some constant. The removal rate is the proportion of free calls where we return an unused part from the pool to the underlying pool in order not to exceed the size limit.) In the log messages above, you see "Raising pool_size_limit_" lines that show the pool size growing. If your program actually has a steady state behavior with a maximum size collection of chunks it needs, the pool will grow to accommodate it, and then grow no more. It behaves this way rather than simply retaining all chunks ever allocated so that sizes needed only rarely, or only during program startup, are less likely to be retained in the pool.
These messages should only be a reason for concern if you run out of memory. In such a case the log messages may help diagnose the problem. The peak execution speed may only be attained after the memory pools have grown to the proper size.
Back