У клиента возникла такая ситуация. На centos'е крутится PostgreSQL с Java'ским application сервером, и периодически при действиях когда через сервер приложений надо пропихнуть метров 800 (отдельный вопрос почему так получается), срабатывает oom killer который грохает один из postgre'х процессов, после чего валятся все остальные (так как по мнению postgresql могут быть corrupted shared buffers). При этом на сервере 8 гигов памяти и что наиболее не понятно 10 гиговый своп. Учитывая их нагрузку забить 18 гигов даже теоретически тяжеловато. И если посмотреть /var/log/messages:

Oct 28 11:15:03 xcl kernel: jsvc invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
Oct 28 11:15:03 xcl kernel: jsvc cpuset=/ mems_allowed=0
Oct 28 11:15:03 xcl kernel: Pid: 9815, comm: jsvc Not tainted 2.6.32-358.23.2.el6.x86_64 #1
Oct 28 11:15:03 xcl kernel: Call Trace:
Oct 28 11:15:03 xcl kernel: [<ffffffff810cb641>] ? cpuset_print_task_mems_allowed+0x91/0xb0
Oct 28 11:15:03 xcl kernel: [<ffffffff8111ce40>] ? dump_header+0x90/0x1b0
Oct 28 11:15:03 xcl kernel: [<ffffffff8121d4ec>] ? security_real_capable_noaudit+0x3c/0x70
Oct 28 11:15:03 xcl kernel: [<ffffffff8111d2c2>] ? oom_kill_process+0x82/0x2a0
Oct 28 11:15:03 xcl kernel: [<ffffffff8111d201>] ? select_bad_process+0xe1/0x120
Oct 28 11:15:03 xcl kernel: [<ffffffff8111d700>] ? out_of_memory+0x220/0x3c0
Oct 28 11:15:03 xcl kernel: [<ffffffff8112c3dc>] ? __alloc_pages_nodemask+0x8ac/0x8d0
Oct 28 11:15:03 xcl kernel: [<ffffffff81160d6a>] ? alloc_pages_vma+0x9a/0x150
Oct 28 11:15:03 xcl kernel: [<ffffffff81143f0b>] ? handle_pte_fault+0x76b/0xb50
Oct 28 11:15:03 xcl kernel: [<ffffffff8104bac7>] ? pte_alloc_one+0x37/0x50
Oct 28 11:15:03 xcl kernel: [<ffffffff8117b869>] ? do_huge_pmd_anonymous_page+0xb9/0x380
Oct 28 11:15:03 xcl kernel: [<ffffffff8114452a>] ? handle_mm_fault+0x23a/0x310
Oct 28 11:15:03 xcl kernel: [<ffffffff8109c22a>] ? down_read_trylock+0x1a/0x30
Oct 28 11:15:03 xcl kernel: [<ffffffff810474e9>] ? __do_page_fault+0x139/0x480
Oct 28 11:15:03 xcl kernel: [<ffffffff810097cc>] ? __switch_to+0x1ac/0x320
Oct 28 11:15:03 xcl kernel: [<ffffffff8150e1c0>] ? thread_return+0x4e/0x76e
Oct 28 11:15:03 xcl kernel: [<ffffffff81513bfe>] ? do_page_fault+0x3e/0xa0
Oct 28 11:15:03 xcl kernel: [<ffffffff81510fb5>] ? page_fault+0x25/0x30
Oct 28 11:15:03 xcl kernel: Mem-Info:
Oct 28 11:15:03 xcl kernel: Node 0 DMA per-cpu:
Oct 28 11:15:03 xcl kernel: CPU 0: hi: 0, btch: 1 usd: 0
Oct 28 11:15:03 xcl kernel: CPU 1: hi: 0, btch: 1 usd: 0
Oct 28 11:15:03 xcl kernel: CPU 2: hi: 0, btch: 1 usd: 0
Oct 28 11:15:03 xcl kernel: CPU 3: hi: 0, btch: 1 usd: 0
Oct 28 11:15:03 xcl kernel: Node 0 DMA32 per-cpu:
Oct 28 11:15:03 xcl kernel: CPU 0: hi: 186, btch: 31 usd: 2
Oct 28 11:15:03 xcl kernel: CPU 1: hi: 186, btch: 31 usd: 39
Oct 28 11:15:03 xcl kernel: CPU 2: hi: 186, btch: 31 usd: 76
Oct 28 11:15:03 xcl kernel: CPU 3: hi: 186, btch: 31 usd: 58
Oct 28 11:15:03 xcl kernel: Node 0 Normal per-cpu:
Oct 28 11:15:03 xcl kernel: CPU 0: hi: 186, btch: 31 usd: 36
Oct 28 11:15:03 xcl kernel: CPU 1: hi: 186, btch: 31 usd: 21
Oct 28 11:15:03 xcl kernel: CPU 2: hi: 186, btch: 31 usd: 53
Oct 28 11:15:03 xcl kernel: CPU 3: hi: 186, btch: 31 usd: 60
Oct 28 11:15:03 xcl kernel: active_anon:1634022 inactive_anon:309847 isolated_anon:0
Oct 28 11:15:03 xcl kernel: active_file:39 inactive_file:100 isolated_file:0
Oct 28 11:15:03 xcl kernel: unevictable:0 dirty:8 writeback:1 unstable:0
Oct 28 11:15:03 xcl kernel: free:25314 slab_reclaimable:4046 slab_unreclaimable:6457
Oct 28 11:15:03 xcl kernel: mapped:174215 shmem:271795 pagetables:16651 bounce:0
Oct 28 11:15:03 xcl kernel: Node 0 DMA free:15692kB min:124kB low:152kB high:184kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15272kB mlocked:0
kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Oct 28 11:15:03 xcl kernel: lowmem_reserve[]: 0 3512 8057 8057
Oct 28 11:15:03 xcl kernel: Node 0 DMA32 free:47556kB min:29404kB low:36752kB high:44104kB active_anon:2664088kB inactive_anon:594444kB active_file:0kB inactive_file:120kB unevictable:0kB isolated(anon):0kB isolated(file):0kB pres
ent:3596500kB mlocked:0kB dirty:0kB writeback:0kB mapped:441660kB shmem:476456kB slab_reclaimable:3580kB slab_unreclaimable:636kB kernel_stack:64kB pagetables:13700kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:220 all_unrecl
aimable? yes
Oct 28 11:15:03 xcl kernel: lowmem_reserve[]: 0 0 4545 4545
Oct 28 11:15:03 xcl kernel: Node 0 Normal free:38008kB min:38052kB low:47564kB high:57076kB active_anon:3872000kB inactive_anon:644944kB active_file:168kB inactive_file:280kB unevictable:0kB isolated(anon):0kB isolated(file):0kB p
resent:4654080kB mlocked:0kB dirty:32kB writeback:4kB mapped:255200kB shmem:610724kB slab_reclaimable:12604kB slab_unreclaimable:25192kB kernel_stack:1960kB pagetables:52904kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:717 a
ll_unreclaimable? yes
Oct 28 11:15:03 xcl kernel: lowmem_reserve[]: 0 0 0 0
Oct 28 11:15:03 xcl kernel: Node 0 DMA: 3*4kB 2*8kB 1*16kB 1*32kB 2*64kB 1*128kB 0*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15692kB
Oct 28 11:15:03 xcl kernel: Node 0 DMA32: 598*4kB 493*8kB 334*16kB 185*32kB 88*64kB 51*128kB 23*256kB 5*512kB 5*1024kB 2*2048kB 0*4096kB = 47424kB
Oct 28 11:15:03 xcl kernel: Node 0 Normal: 639*4kB 386*8kB 277*16kB 165*32kB 120*64kB 58*128kB 23*256kB 3*512kB 0*1024kB 0*2048kB 0*4096kB = 37884kB
Oct 28 11:15:03 xcl kernel: 307836 total pagecache pages
Oct 28 11:15:03 xcl kernel: 35820 pages in swap cache
Oct 28 11:15:03 xcl kernel: Swap cache stats: add 375539898, delete 375504078, find 557420548/581925676
Oct 28 11:15:03 xcl kernel: Free swap = 0kB
Oct 28 11:15:03 xcl kernel: Total swap = 10485752kB
Oct 28 11:15:03 xcl kernel: 2097151 pages RAM
Oct 28 11:15:03 xcl kernel: 82251 pages reserved
Oct 28 11:15:03 xcl kernel: 797360 pages shared
Oct 28 11:15:03 xcl kernel: 1811719 pages non-shared
Oct 28 11:15:03 xcl kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Oct 28 11:15:03 xcl kernel: [ 524] 0 524 2793 0 2 -17 -1000 udevd
Oct 28 11:15:03 xcl kernel: [ 1234] 0 1234 2279 1 0 0 0 dhclient
Oct 28 11:15:03 xcl kernel: [ 1272] 0 1272 62367 237 2 0 0 rsyslogd
Oct 28 11:15:03 xcl kernel: [ 1301] 0 1301 2704 37 0 0 0 irqbalance
Oct 28 11:15:03 xcl kernel: [ 4371] 81 4371 5350 41 2 0 0 dbus-daemon
Oct 28 11:15:03 xcl kernel: [ 4400] 0 4400 1019 0 2 0 0 acpid
Oct 28 11:15:03 xcl kernel: [ 4409] 68 4409 6299 101 0 0 0 hald
Oct 28 11:15:03 xcl kernel: [ 4410] 0 4410 4526 1 0 0 0 hald-runner
Oct 28 11:15:03 xcl kernel: [ 4438] 0 4438 5055 1 2 0 0 hald-addon-inpu
Oct 28 11:15:03 xcl kernel: [ 4453] 68 4453 4451 1 0 0 0 hald-addon-acpi
Oct 28 11:15:03 xcl kernel: [ 4476] 0 4476 114768 38 2 0 0 automount
Oct 28 11:15:03 xcl kernel: [ 4492] 0 4492 1691 0 0 0 0 mcelog
Oct 28 11:15:03 xcl kernel: [ 4504] 0 4504 16563 0 0 -17 -1000 sshd
Oct 28 11:15:03 xcl kernel: [ 4515] 0 4515 13036 0 3 0 0 vsftpd
Oct 28 11:15:03 xcl kernel: [ 4576] 497 4576 19156 4 0 0 0 zabbix_agentd
Oct 28 11:15:03 xcl kernel: [ 4578] 497 4578 19156 76 0 0 0 zabbix_agentd
Oct 28 11:15:03 xcl kernel: [ 4579] 497 4579 19156 41 0 0 0 zabbix_agentd
Oct 28 11:15:03 xcl kernel: [ 4580] 497 4580 19156 44 2 0 0 zabbix_agentd
Oct 28 11:15:03 xcl kernel: [ 4581] 497 4581 19156 43 2 0 0 zabbix_agentd
Oct 28 11:15:03 xcl kernel: [ 4588] 0 4588 29312 22 0 0 0 crond
Oct 28 11:15:03 xcl kernel: [ 4599] 0 4599 5373 0 2 0 0 atd
Oct 28 11:15:03 xcl kernel: [ 4647] 0 4647 1015 1 1 0 0 mingetty
Oct 28 11:15:03 xcl kernel: [ 4649] 0 4649 1015 1 2 0 0 mingetty
Oct 28 11:15:03 xcl kernel: [ 4651] 0 4651 1015 1 1 0 0 mingetty
Oct 28 11:15:03 xcl kernel: [ 4653] 0 4653 1015 1 2 0 0 mingetty
Oct 28 11:15:03 xcl kernel: [ 4655] 0 4655 1015 1 1 0 0 mingetty
Oct 28 11:15:03 xcl kernel: [ 4657] 0 4657 1015 1 1 0 0 mingetty
Oct 28 11:15:03 xcl kernel: [ 4659] 0 4659 2792 0 0 -17 -1000 udevd
Oct 28 11:15:03 xcl kernel: [ 4660] 0 4660 2792 0 2 -17 -1000 udevd
Oct 28 11:15:03 xcl kernel: [ 4695] 0 4695 23293 28 0 -17 -1000 auditd
Oct 28 11:15:03 xcl kernel: [ 4884] 0 4884 20216 24 2 0 0 master
Oct 28 11:15:03 xcl kernel: [ 4887] 89 4887 20279 16 1 0 0 qmgr
Oct 28 11:15:03 xcl kernel: [16500] 26 16500 323285 72 0 0 0 postmaster
Oct 28 11:15:03 xcl kernel: [16502] 26 16502 45006 49 0 0 0 postmaster
Oct 28 11:15:03 xcl kernel: [16504] 26 16504 324117 85020 0 0 0 postmaster
Oct 28 11:15:03 xcl kernel: [16505] 26 16505 323451 49075 2 0 0 postmaster
Oct 28 11:15:03 xcl kernel: [16506] 26 16506 323414 1064 0 0 0 postmaster
Oct 28 11:15:03 xcl kernel: [16507] 26 16507 323658 292 0 0 0 postmaster
Oct 28 11:15:03 xcl kernel: [16508] 26 16508 45481 354 0 0 0 postmaster
Oct 28 11:15:03 xcl kernel: [16761] 0 16761 2609 1 0 0 0 jsvc
Oct 28 11:15:03 xcl kernel: [16762] 500 16762 733778 36997 1 0 0 jsvc
Oct 28 11:15:03 xcl kernel: [ 9812] 0 9812 2610 1 2 0 0 jsvc
Oct 28 11:15:03 xcl kernel: [ 9813] 501 9813 1689304 810222 0 0 0 jsvc
Oct 28 11:15:03 xcl kernel: [ 9830] 26 9830 324456 1422 0 0 0 postmaster
Oct 28 11:15:03 xcl kernel: [ 9863] 26 9863 338059 134611 0 0 0 postmaster
Oct 28 11:15:03 xcl kernel: [ 9864] 26 9864 1754324 16936 2 0 0 postmaster
Oct 28 11:15:03 xcl kernel: [ 9934] 26 9934 453490 44269 0 0 0 postmaster
Oct 28 11:15:03 xcl kernel: [29354] 26 29354 566182 332724 0 0 0 postmaster
Oct 28 11:15:03 xcl kernel: [ 921] 26 921 341831 138505 3 0 0 postmaster
Oct 28 11:15:03 xcl kernel: [10310] 26 10310 1731158 618645 0 0 0 postmaster
Oct 28 11:15:03 xcl kernel: [10407] 26 10407 338239 55648 0 0 0 postmaster
Oct 28 11:15:03 xcl kernel: [ 9461] 26 9461 336734 33343 1 0 0 postmaster
Oct 28 11:15:03 xcl kernel: [ 9779] 26 9779 338499 39082 0 0 0 postmaster
Oct 28 11:15:03 xcl kernel: [10182] 89 10182 20321 216 2 0 0 pickup
Oct 28 11:15:03 xcl kernel: [10246] 26 10246 332818 28755 2 0 0 postmaster
Oct 28 11:15:03 xcl kernel: [10975] 26 10975 325280 5698 2 0 0 postmaster
Oct 28 11:15:03 xcl kernel: Out of memory: Kill process 10310 (postmaster) score 329 or sacrifice child
Oct 28 11:15:03 xcl kernel: Killed process 10310, UID 26, (postmaster) total-vm:6924632kB, anon-rss:1836248kB, file-rss:638332kB

То сумма всех rss даже близко не подходит к 18 гигам (а Free Swap = 0kb). В чем может быть подвох и куда еще смотреть?

Версия ОС:

Linux xcl.atservers.net 2.6.32-358.23.2.el6.x86_64 #1 SMP Wed Oct 16 18:37:12 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

LSB Version: :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch
Distributor ID: CentOS
Description: CentOS release 6.4 (Final)
Release: 6.4
Codename: Final

Журавлев Дениспокажите
cat /proc/meminfo

MemTotal: 8059600 kB
MemFree: 431800 kB
Buffers: 74840 kB
Cached: 2141408 kB
SwapCached: 6096 kB
Active: 5607692 kB
Inactive: 1721228 kB
Active(anon): 5369336 kB
Inactive(anon): 848664 kB
Active(file): 238356 kB
Inactive(file): 872564 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 10485752 kB
SwapFree: 10434624 kB
Dirty: 396 kB
Writeback: 0 kB
AnonPages: 5107312 kB
Mapped: 1111904 kB
Shmem: 1105328 kB
Slab: 189508 kB
SReclaimable: 163660 kB
SUnreclaim: 25848 kB
KernelStack: 2048 kB
PageTables: 34420 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 14515552 kB
Committed_AS: 6522528 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 28520 kB
VmallocChunk: 34359703644 kB
HardwareCorrupted: 0 kB
AnonHugePages: 4765696 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 8180 kB
DirectMap2M: 8380416 kB

Это сейчас ессно а на момент падения. На момент падения, а точнее после него тяжело посмотреть, обычно повторяется раз в месяц макс.

Nitro_Junkie,

* а не на момент падения