INFORMIX ODS 7.31 FD6 SunFire 880 Solaris 8

Выглядит следующим образом
Первичка замерзает в состоянии сброса чекпойнта на длительное время,
в это время на вторичке в лог валится каждую секунду

15:35:09 Assert Failed: Buffer manager internal error
15:35:09 Informix Dynamic Server Version 7.31.FD6
15:35:09 Who: Session(754, informix@ss880, 0, 1521202384)
Thread(824, xchg_1.3, 15aa93108, 6)
File: rsbuff.c Line: 135
15:35:09 Results:
15:35:09 Action: Internally corrected
15:35:09 See Also: /tmp/af.72063dd

в первом случае в течении ~20 минут HADR таки упал и первичка поехала дальше.
После чего HADR поднять не удалось и его восстанавливали вечером через архив нулевого уровня.

Во втором случае ждать было некогда и INFORMIX был перзапущен без HADR. Ночью опять же восстановили и ждем опять.

содержимое файла /tmp/af.72063dd

15:35:09 bp 11a919158 pagenum 38:134966 pg_flags 90
15:35:09 Consistency Check: page not physically logged.
buffer header
000000011a919158: 00000000 00000000 00000000 00000000 ........ ........
000000011a919168: 00000001 19561598 00000001 19ede058 .....V.. .......X
000000011a919178: 00000001 1a384358 00000001 1c3dd5d8 .....8CX .....=..
000000011a919188: 00000802 00000001 0000001c 00000000 ........ ........
000000011a919198: 00000001 5aa93108 00000000 00000000 ....Z.1. ........
000000011a9191a8: 00000000 00000080 7e317fe1 00a00003 ........ ~1......
000000011a9191b8: 000a73a3 00000001 00000000 00000000 ..s..... ........
000000011a9191c8: 02620f36 00000001 00000001 2b60b000 .b.6.... ....+`..
page header
000000012b60b000: 02620f36 7e317fe1 00010090 036d048b .b.6~1.. .....m..
000000012b60b010: 0003bedc 000a706f ......po
15:35:09
15:35:09 Informix Dynamic Server Version 7.31.FD6 Software Serial Number AAC#J213346

15:35:09 Assert Failed: Buffer manager internal error
15:35:09 Who: Session(754, informix@ss880, 0, 1521202384)
Thread(824, xchg_1.3, 15aa93108, 6)
File: rsbuff.c Line: 135
15:35:09 Results:
15:35:09 Action: Internally corrected
---------------------------------
Begin System Alarm Program Output
---------------------------------

Assertion Failure Type: Warning
Host Name: ss880
Database Server Name: ss880on
Time of failure: Mon Oct 31 15:35:09 GMT 2005
AF file: /tmp/af.72063dd
Shared memory file: None
System Blocking: OFF


-------------------------------
End System Alarm Program Output
-------------------------------

15:35:09 sh /opt/informix/etc/evidence.sh 1 0 /tmp/af.72063dd 754 0x5aa93108 824 0x5b757410 1 0 0 0 0
15:35:09 See Also: /tmp/af.72063dd
15:35:09
------------------ End of assertion failure 0 -----------------

Первичный сервер сказал вот что

15:33:38 Logical Log 391455 Complete.
15:33:39 Process exited with return code 142: /bin/sh /bin/sh -c /opt/informix
15:34:09 Logical Log 391456 Complete.
15:34:10 Process exited with return code 142: /bin/sh /bin/sh -c /opt/informix
15:36:27 Checkpoint Completed: duration was 75 seconds.
15:36:27 Checkpoint loguniq 391457, logpos 0x234018

15:38:49 Checkpoint Completed: duration was 22 seconds.
15:38:49 Checkpoint loguniq 391457, logpos 0x298790

15:49:57 DR: ping timeout
15:49:57 DR: Receive error
15:49:59 DR: Turned off on primary server
15:55:06 WARNING: Checkpoint appears stalled and may not complete
before the database server shuts down.

#**************************************************************************
#
# INFORMIX SOFTWARE, INC.
#
# Title: onconfig.std
# Description: Informix Dynamic Server Configuration Parameters
#
#**************************************************************************

# Root Dbspace Configuration

ROOTNAME rootdbs # Root dbspace name
ROOTPATH /dev/md/rdsk/d66 # Path for device containing root dbspace
ROOTOFFSET 0 # Offset of root dbspace into device (Kbytes)
ROOTSIZE 1048575 # Size of root dbspace (Kbytes)

# Disk Mirroring Configuration Parameters

MIRROR 0 # Mirroring flag (Yes = 1, No = 0)
MIRRORPATH # Path for device containing mirrored root
MIRROROFFSET 0 # Offset into mirrored device (Kbytes)

# Physical Log Configuration

PHYSDBS logdbs # Location (dbspace) of physical log
PHYSFILE 800000 # Physical log file size (Kbytes)

# Logical Log Configuration

LOGFILES 260 # Number of logical log files
LOGSIZE 2000 # Logical log size (Kbytes)

# Diagnostics

MSGPATH /opt/informix/online.log # System message log file path
CONSOLE /dev/console # System console message path
ALARMPROGRAM /opt/informix/etc/log_full.sh # Alarm program path
SYSALARMPROGRAM /opt/informix/etc/evidence.sh # System Alarm program path
TBLSPACE_STATS 0

# System Archive Tape Device

TAPEDEV /dev/rmt/0 # Tape device path
#TAPEDEV /dev/null
TAPEBLK 4096 # Tape block size (Kbytes)
TAPESIZE 70000000 # Maximum amount of data to put on tape (Kbytes)

# Log Archive Tape Device

LTAPEDEV /dev/null # Log tape device path
LTAPEBLK 4096 # Log tape block size (Kbytes)
LTAPESIZE 10240 # Max amount of data to put on log tape (Kbytes)

# Optical

STAGEBLOB # Informix Dynamic Server/Optical staging area

# System Configuration
SERVERNUM 0 # Unique id corresponding to a Dynamic Server instance
DBSERVERNAME sf880on # Name of default database server
DBSERVERALIASES fast0spx,sf880tli # List of alternate dbservernames
NETTYPE ipcshm,2,200,CPU # Configure poll thread(s) for nettype
NETTYPE tlitcp,1,10,NET # Configure poll thread(s) for nettype
#NETTYPE tlispx,5,10,NET # Configure poll thread(s) for nettype
DEADLOCK_TIMEOUT 60 # Max time to wait of lock in distributed env.
RESIDENT 1 # Forced residency flag (Yes = 1, No = 0)

MULTIPROCESSOR 1 # 0 for single-processor, 1 for multi-processor
NUMCPUVPS 8 # Number of user (cpu) vps
SINGLE_CPU_VP 0 # If non-zero, limit number of cpu vps to one

NOAGE 1 # Process aging
AFF_SPROC 0 # Affinity start processor
AFF_NPROCS 8 # Affinity number of processors

# Shared Memory Parameters
CCFLAGS 0x400000 # fix bug 101062 (sfclub)
LOCKS 3000000 # Maximum number of locks
BUFFERS 500000 # Maximum number of shared buffers
NUMAIOVPS 22 # Number of IO vps
PHYSBUFF 128 # Physical log buffer size (Kbytes)
LOGBUFF 32 # Logical log buffer size (Kbytes)
LOGSMAX 400 # Maximum number of logical log files
CLEANERS 34 # Number of buffer cleaner processes
SHMBASE 0x10a000000 # Shared memory base address
SHMVIRTSIZE 1000000 # initial virtual shared memory segment size
SHMADD 512000 # Size of new shared memory segments (Kbytes)
SHMTOTAL 0 # Total shared memory (Kbytes). 0=>unlimited
CKPTINTVL 120 #300 # Check point interval (in sec)
LRUS 34 # Number of LRU queues
LRU_MAX_DIRTY 2 #60 # LRU percent dirty begin cleaning limit
LRU_MIN_DIRTY 1 #50 # LRU percent dirty end cleaning limit
LTXHWM 50 # Long transaction high water mark percentage
LTXEHWM 60 # Long transaction high water mark (exclusive)
TXTIMEOUT 0x12c # Transaction timeout (in sec)
STACKSIZE 256 # Stack size (Kbytes)

# System Page Size
# BUFFSIZE - Dynamic Server no longer supports this configuration parameter.
# To determine the page size used by Dynamic Server on your platform
# see the last line of output from the command, 'onstat -b'.


# Recovery Variables
# OFF_RECVRY_THREADS:
# Number of parallel worker threads during fast recovery or an offline restore.
# ON_RECVRY_THREADS:
# Number of parallel worker threads during an online restore.

OFF_RECVRY_THREADS 10 # Default number of offline worker threads
ON_RECVRY_THREADS 1 # Default number of online worker threads

# Data Replication Variables
# DRAUTO: 0 manual, 1 retain type, 2 reverse type
DRAUTO 0 # DR automatic switchover
DRINTERVAL -1 # DR max time between DR buffer flushes (in sec)
DRTIMEOUT 150 # DR network timeout (in sec)
DRLOSTFOUND /opt/informix/etc/dr.lostfound # DR lost+found file path

# CDR Variables
CDR_LOGBUFFERS 2048 # size of log reading buffer pool (Kbytes)
CDR_EVALTHREADS 1,2 # evaluator threads (per-cpu-vp,additional)
CDR_DSLOCKWAIT 5 # DS lockwait timeout (seconds)
CDR_QUEUEMEM 4096 # Maximum amount of memory for any CDR queue (Kbytes)
CDR_LOGDELTA 30 # % of log space allowed in queue memory
CDR_NUMCONNECT 16 # Expected connections per server
CDR_NIFRETRY 300 # Connection retry (seconds)
CDR_NIFCOMPRESS 0 # Link level compression (-1 never, 0 none, 9 max)

# Backup/Restore variables
BAR_ACT_LOG /tmp/bar_act.log
BAR_DEBUG_LOG /usr/informix/bar_dbug.log
# ON-Bar Debug Log - not in /tmp pleas
BAR_MAX_BACKUP 0
BAR_RETRY 1
BAR_NB_XPORT_COUNT 10
BAR_XFER_BUF_SIZE 31

# Informix Storage Manager variables
ISM_DATA_POOL ISMData # If the data pool name is changed, be sure to
# update $INFORMIXDIR/bin/onbar. Change to
# ism_catalog -create_bootstrap -pool <new name>
ISM_LOG_POOL ISMLogs

# Read Ahead Variables
RA_PAGES # Number of pages to attempt to read ahead
RA_THRESHOLD # Number of pages left before next group

# DBSPACETEMP:
# Dynamic Server equivalent of DBTEMP for SE. This is the list of dbspaces
# that the Dynamic Server SQL Engine will use to create temp tables etc.
# If specified it must be a colon separated list of dbspaces that exist
# when the Dynamic Server system is brought online. If not specified, or if
# all dbspaces specified are invalid, various ad hoc queries will create
# temporary files in /tmp instead.

DBSPACETEMP tempdbs # Default temp dbspaces

# DUMP*:
# The following parameters control the type of diagnostics information which
# is preserved when an unanticipated error condition (assertion failure) occurs
# during Dynamic Server operations.
# For DUMPSHMEM, DUMPGCORE and DUMPCORE 1 means Yes, 0 means No.

DUMPDIR /tmp # Preserve diagnostics in this directory
DUMPSHMEM 0 # Dump a copy of shared memory
DUMPGCORE 0 # Dump a core image using 'gcore'
DUMPCORE 0 # Dump a core image (Warning:this aborts Dynamic Server)
DUMPCNT 1 # Number of shared memory or gcore dumps for
# a single user's session

FILLFACTOR 90 # Fill factor for building indexes

# method for Dynamic Server to use when determining current time
USEOSTIME 0 # 0: use internal time(fast), 1: get time from OS(slow)
# when the Dynamic Server system is brought online. If not specified, or if
# all dbspaces specified are invalid, various ad hoc queries will create
# temporary files in /tmp instead.

DBSPACETEMP tempdbs # Default temp dbspaces

# DUMP*:
# The following parameters control the type of diagnostics information which
# is preserved when an unanticipated error condition (assertion failure) occurs
# during Dynamic Server operations.
# For DUMPSHMEM, DUMPGCORE and DUMPCORE 1 means Yes, 0 means No.

DUMPDIR /tmp # Preserve diagnostics in this directory
DUMPSHMEM 0 # Dump a copy of shared memory
DUMPGCORE 0 # Dump a core image using 'gcore'
DUMPCORE 0 # Dump a core image (Warning:this aborts Dynamic Server)
DUMPCNT 1 # Number of shared memory or gcore dumps for
# a single user's session

FILLFACTOR 90 # Fill factor for building indexes

# method for Dynamic Server to use when determining current time
USEOSTIME 0 # 0: use internal time(fast), 1: get time from OS(slow)

# Parallel Database Queries (pdq)
MAX_PDQPRIORITY 100 # Maximum allowed pdqpriority
DS_MAX_QUERIES # Maximum number of decision support queries
DS_TOTAL_MEMORY # Decision support memory (Kbytes)
DS_MAX_SCANS 1048576 # Maximum number of decision support scans
DATASKIP off # List of dbspaces to skip

# OPTCOMPIND
# 0 => Nested loop joins will be preferred (where
# possible) over sortmerge joins and hash joins.
# 1 => If the transaction isolation mode is not
# "repeatable read", optimizer behaves as in (2)
# below. Otherwise it behaves as in (0) above.
# 2 => Use costs regardless of the transaction isolation
# mode. Nested loop joins are not necessarily
# preferred. Optimizer bases its decision purely
# on costs.
OPTCOMPIND 1 # To hint the optimizer

ONDBSPACEDOWN 2 # Dbspace down option: 0 = CONTINUE, 1 = ABORT, 2 = WAIT
LBU_PRESERVE 0 # Preserve last log for log backup
OPCACHEMAX 0 # Maximum optical cache size (Kbytes)

# HETERO_COMMIT (Gateway participation in distributed transactions)
# 1 => Heterogeneous Commit is enabled
# 0 (or any other value) => Heterogeneous Commit is disabled
HETERO_COMMIT 0

# Optimization goal: -1 = ALL_ROWS(Default), 0 = FIRST_ROWS
OPT_GOAL -1

# Optimizer DIRECTIVES ON (1/Default) or OFF (0)
DIRECTIVES 1

# Status of restartable restore
RESTARTABLE_RESTORE off

Началось собствено совсем недавно

29-10-2005 первый раз глюкнуло, ночью переподняли HADR, отработали 30го
31го глюкнуло еще раз, ночью опять восстановили. летим дальше


S&T support ответил, что это баг номер 124003 , который пофиксили в более ранних версиях по декларации IBM. И посоветовали подождать , если будет стабильно повторяться -- обращаться в IBM.

Выбегаллоявный баг на втором сервере. можете stack trace запостить, можете сразу открывать тикет с IBM (если саппорт есть, но с такой техникой смешно на саппорте экономить ). А что, кстати, на первичном происходило в это время ? Давно это у вас началось ?

В таком вот аксепте

лог с первички в третьем посте

скажите, а что есть stack trace ?

Журавлев Денис cpr
скажите, а что есть stack trace ?

Одна из секций af файла. Там показывается стек вызовов функций и видно функцию в которой выпал эксепшин.

Возможно в твоем /tmp/af.72063dd этой секции и не было?

Понял, это в onstat команда stk [tid]
нет в в моем af файле этого не было.
Обычно когда сервак падает, тогда он все вываливает в af файл, а здесь поскольку ошибка корректировалась, видимо не счел нужным.

после второго падения вторички сначала передаунили первичку, а потом восстановили HADR. В первый раз первичку не презапускали. После второго раза до сих пор (тук-тук-тук) не падал.