0 votes
1 view
in AWS by (19.2k points)

Environment Details:

Server: Amazon ec2 Linux

Web Server: Apache

Web Framework: Django with mod_wsgi

Following I have found in the mysql_err.log file.

The InnoDB memory heap is disabled

120823  3:21:40 InnoDB: Mutexes and rw_locks use GCC atomic builtins

120823  3:21:40 InnoDB: Compressed tables use zlib 1.2.3

120823  3:21:40 InnoDB: Using Linux native AIO

120823  3:21:41 InnoDB: Initializing buffer pool, size = 128.0M

InnoDB: mmap(137363456 bytes) failed; errno 12

120823  3:21:41 InnoDB: Completed initialization of buffer pool

120823  3:21:41 InnoDB: Fatal error: cannot allocate memory for the buffer pool

120823  3:21:41 [ERROR] Plugin 'InnoDB' init function returned error.

120823  3:21:41 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.

120823  3:21:41 [ERROR] Unknown/unsupported storage engine: InnoDB

120823  3:21:41 [ERROR] Aborting

Looks like the system memory is not sufficient to allocate memory to the buffer pool. Same error happens when I was using Amazon ec2 micro instance, So I moved to the small instance. It works fine for some days but now it is breaking once in a day again. Is there a permanent fix for that? I can move to medium instance but the issue is will that be fixed or not? Should I decrease the innodb_buffer_pool_size, what is the preferred size?

The result of cat /proc/meminfo is following:

MemTotal:        1697824 kB

MemFree:          125744 kB

Buffers:          109704 kB

Cached:           481408 kB

SwapCached:            0 kB

Active:          1212396 kB

Inactive:         266840 kB

Active(anon):     888192 kB

Inactive(anon):       76 kB

Active(file):     324204 kB

Inactive(file):   266764 kB

Unevictable:           0 kB

Mlocked:               0 kB

SwapTotal:             0 kB

SwapFree:              0 kB

Dirty:                 4 kB

Writeback:             0 kB

AnonPages:        888144 kB

Mapped:            15604 kB

Shmem:               144 kB

Slab:              63752 kB

SReclaimable:      53680 kB

SUnreclaim:        10072 kB

KernelStack:         800 kB

PageTables:        16436 kB

NFS_Unstable:          0 kB

Bounce:                0 kB

WritebackTmp:          0 kB

CommitLimit:      848912 kB

Committed_AS:    1417140 kB

VmallocTotal:   34359738367 kB

VmallocUsed:       10988 kB

VmallocChunk:   34359725168 kB

DirectMap4k:     1748992 kB

DirectMap2M:           0 kB

OS version (uname -a): Linux ip-10-246-134-149 3.2.21-1.32.6.amzn1.x86_64 #1 SMP Sat Jun 23 02:32:15 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

I checked the ps aux command whet the server has like only 15MB of memory left and these are the httpd process running at that time:

The result of free -m

total       used free     shared buffers   cached

Mem:          1657 1628         29 0 3         19

-/+ buffers/cache:       1605 51

Swap:          895 875         20

The result of ps aux

apache   21123 0.1  1.2 394652 20464 ?        S 19:35 0:06 /usr/sbin/httpd

apache   21146 0.1  1.2 394280 20796 ?        S 19:38 0:06 /usr/sbin/httpd

apache   21152 0.1  1.2 394284 21560 ?        S 19:38 0:05 /usr/sbin/httpd

apache   21155 0.2  1.4 396244 24528 ?        S 19:38 0:06 /usr/sbin/httpd

apache   21156 0.1  1.1 392552 20344 ?        S 19:38 0:06 /usr/sbin/httpd

apache   21157 0.1  1.1 394284 18884 ?        S 19:38 0:05 /usr/sbin/httpd

apache   21159 0.1  1.4 396200 25040 ?        S 19:38 0:06 /usr/sbin/httpd

apache   21161 0.1  1.2 394856 21724 ?        S 19:38 0:06 /usr/sbin/httpd

apache   21162 0.1  1.3 394864 22400 ?        S 19:38 0:06 /usr/sbin/httpd

apache   21163 0.1  1.3 394860 22204 ?        S 19:38 0:06 /usr/sbin/httpd

apache   21164 0.1  1.1 392560 19204 ?        S 19:38 0:06 /usr/sbin/httpd

apache   21165 0.1  1.3 394832 22280 ?        S 19:38 0:06 /usr/sbin/httpd

apache   21166 0.1  1.3 395276 22932 ?        S 19:38 0:06 /usr/sbin/httpd

apache   21172 0.2  1.4 396320 24820 ?        S 19:38 0:06 /usr/sbin/httpd

apache   21174 0.2  1.7 400672 29452 ?        S 19:39 0:06 /usr/sbin/httpd

apache   21178 0.1  1.4 400540 25304 ?        S 19:39 0:06 /usr/sbin/httpd

apache   21179 0.2  1.6 400580 27856 ?        S 19:39 0:06 /usr/sbin/httpd

apache   21184 0.1  1.7 400628 29320 ?        S 19:39 0:06 /usr/sbin/httpd

apache   21185 0.1  1.6 397944 27292 ?        S 19:39 0:05 /usr/sbin/httpd

apache   21186 0.1  1.5 397960 25648 ?        S 19:39 0:05 /usr/sbin/httpd

apache   21187 0.1  1.7 400576 29120 ?        S 19:39 0:06 /usr/sbin/httpd

apache   21191 0.1  1.4 400576 24400 ?        S 19:39 0:06 /usr/sbin/httpd

apache   21193 0.1  1.4 400536 24940 ?        S 19:39 0:05 /usr/sbin/httpd

apache   21194 0.1  1.5 400572 26096 ?        S 19:39 0:06 /usr/sbin/httpd

apache   21203 0.1  1.6 400580 28808 ?        S 19:39 0:05 /usr/sbin/httpd

apache   21206 0.1  1.7 400584 29732 ?        S 19:39 0:06 /usr/sbin/httpd

apache   21207 0.1  1.6 400576 27940 ?        S 19:39 0:06 /usr/sbin/httpd

apache   21224 0.1  1.2 400624 20768 ?        S 19:39 0:06 /usr/sbin/httpd

apache   21225 0.1  1.6 400576 28468 ?        S 19:39 0:05 /usr/sbin/httpd

apache   21226 0.1  1.6 400576 28048 ?        S 19:39 0:06 /usr/sbin/httpd

apache   21228 0.1  1.4 400572 23880 ?        S 19:39 0:06 /usr/sbin/httpd

apache   21237 0.1  1.5 400628 26124 ?        S 19:39 0:06 /usr/sbin/httpd

apache   21265 0.1  1.6 400536 28592 ?        S 19:39 0:06 /usr/sbin/httpd

apache   21276 0.1  1.2 400544 21456 ?        S 19:39 0:05 /usr/sbin/httpd

apache   21277 0.1  1.3 400624 22676 ?        S 19:39 0:05 /usr/sbin/httpd

apache   21278 0.1  1.6 400536 27360 ?        S 19:39 0:06 /usr/sbin/httpd

apache   21282 0.1  1.4 400612 24996 ?        S 19:39 0:06 /usr/sbin/httpd

apache   21292 0.1  1.4 400532 24780 ?        S 19:39 0:05 /usr/sbin/httpd

apache   21302 0.2  1.2 400540 21332 ?        S 19:39 0:06 /usr/sbin/httpd

apache   21303 0.1  1.3 400628 22228 ?        S 19:39 0:06 /usr/sbin/httpd

apache   21305 0.2  1.2 400536 21116 ?        S 19:39 0:06 /usr/sbin/httpd

apache   21306 0.1  1.3 400572 22380 ?        S 19:39 0:06 /usr/sbin/httpd

apache   21307 0.1  1.1 397956 20056 ?        S 19:39 0:05 /usr/sbin/httpd

apache   21308 0.1  1.2 400624 21520 ?        S 19:39 0:06 /usr/sbin/httpd

apache   21319 0.1  1.1 400540 19468 ?        S 19:39 0:05 /usr/sbin/httpd

apache   21320 0.1  1.3 400628 22712 ?        S 19:39 0:05 /usr/sbin/httpd

apache   21335 0.1  1.0 400540 17236 ?        S 19:39 0:05 /usr/sbin/httpd

apache   21336 0.1  1.3 400628 22188 ?        S 19:39 0:06 /usr/sbin/httpd

apache   21352 0.1  1.1 394276 18972 ?        S 19:40 0:04 /usr/sbin/httpd

apache   21356 0.1  1.1 394280 19028 ?        S 19:40 0:05 /usr/sbin/httpd

apache   21358 0.1  1.1 394280 19004 ?        S 19:40 0:05 /usr/sbin/httpd

apache   21361 0.2  0.7 400452 12632 ?        S 19:40 0:06 /usr/sbin/httpd

apache   21610 0.2  1.6 400536 27660 ?        S 19:46 0:06 /usr/sbin/httpd

apache   21643 0.2  1.3 400156 23272 ?        S 19:55 0:04 /usr/sbin/httpd

apache   21647 0.2  1.0 400544 17556 ?        S 19:57 0:05 /usr/sbin/httpd

apache   21654 0.2  1.5 400188 26884 ?        S 19:58 0:05 /usr/sbin/httpd

apache   21719 0.3  1.9 400192 32264 ?        S 20:14 0:03 /usr/sbin/httpd

apache   21725 0.2  2.0 400044 35340 ?        S 20:15 0:03 /usr/sbin/httpd

apache   21738 0.0  0.8 257648 13792 ?        S 20:26 0:00 /usr/sbin/httpd

Can anyone has an idea about it why there is so much httpd process??

1 Answer

0 votes
by (44.6k points)

Use 50% of available RAM to test:

You can decrease the innodb_buffer_pool_size very low to examine if it helps:

#/etc/my.cnf

innodb_buffer_pool_size = 1M

A rule of thumb is to line innodb_buffer_pool_size to 500th of available RAM for your low memory testing. This means you begin the server and everything except MySQL InnoDB. See how much RAM you have. Then use 50% of that for InnoDB.

Apache:

Are you using Apache and/or another web server? If so, try to decrease its RAM usage. For example in Apache conf, think about low RAM settings like these:

StartServers 1

MinSpareServers 1

MaxSpareServers 5

MaxClients 5

And cap the requests like this:

MaxRequestsPerChild 300

Then restart Apache.

mod_wsgi:

If you're using Apache with mod_python, switch to Apache with mod_wsgi.

Pympler:

If it's still happening, possibly your Django is steadily growing. Try Django memory profiling with Pympler:

http://www.rkblog.rk.edu.pl/w/p/profiling-django-object-size-and-memory-usage-pympler/

SAR:

Your report of once-per-day failures, then once-per-week failures, could point to some kind of cron job running daily or weekly. For example, maybe there is a batch process that takes up tons of RAM or a database dump, etc.

To track RAM use and look for RAM spikes in the hour before MySQL dies, take a look at SAR, which is a great tool: http://www.thegeekstuff.com/2011/03/sar-examples/

Related questions

0 votes
1 answer
0 votes
1 answer
0 votes
1 answer
0 votes
1 answer
0 votes
1 answer
Welcome to Intellipaat Community. Get your technical queries answered by top developers !


Categories

...