добрый день.
подскажите причину такого резкого падения DB2

DB2 v10.5.0.4 - AIX
LEVEL: Severe
Section relink failure: Relink ID 00000000 @ 0A0001000270F2C0 to ID D4B9AACB @ 0A00020036654FE0

ADM14005E The following error occurred: "AppErr". First Occurrence
Data Capture (FODC) has been invoked in the following mode:
"Automatic". Diagnostic information has been recorded in the

Спасибо.

Mark Barinstein,

SOFTWARE PROGRAM

Recommended Actions
RERUN THE APPLICATION PROGRAM
IF PROBLEM PERSISTS THEN DO THE FOLLOWING
CONTACT APPROPRIATE SERVICE REPRESENTATIVE

Detail Data
SIGNAL NUMBER
11
USER'S PROCESS ID:
3736014
FILE SYSTEM SERIAL NUMBER
14
INODE NUMBER
69760
CORE FILE NAME
/db2dump/FODC_Trap_2017-03-17-11.30.00.665683_0000/3736014.000.core/core
PROGRAM NAME
db2sysc
STACK EXECUTION DISABLED
0
COME FROM ADDRESS REGISTER
??
PROCESSOR ID
hw_fru_id: 4
hw_cpu_id: 36

ADDITIONAL INFORMATION
sqlra_cac D4C
sqlra_cac D48
sqlrr_sig C68
glue_poin 58
sqloExecu 5C
??
sqloDumpD 488
??

Symptom Data
REPORTABLE
1
INTERNAL ERROR
0
SYMPTOM CODE
PCSS/SPI2 FLDS/db2sysc SIG/11 FLDS/sqlra_cac VALU/d4c FLDS/sqlrr_sig

Mark Barinstein,


2017-03-17-11.30.00.679013+180 E19060924A789 LEVEL: Error
PID : 3736014 TID : 82548 PROC : db2sysc 0
EDUID : 82548 EDUNAME: db2agent (DB) 0
FUNCTION: DB2 UDB, relation data serv, sqlrr_dump_ffdc, probe:30
MESSAGE : ADM14005E The following error occurred: "AppErr". First Occurrence
Data Capture (FODC) has been invoked in the following mode:
"Automatic". Diagnostic information has been recorded in the
directory named
"/db2dump/FODC_AppErr_2017-03-17-11.30.00.660471_3736014_8254
8_000/".

2017-03-17-11.30.00.680268+180 E19061714A798 LEVEL: Error
PID : 3736014 TID : 84301 PROC : db2sysc 0
EDUID : 84301 EDUNAME: db2agent (DB) 0
FUNCTION: DB2 UDB, oper system services, sqloEDUCodeTrapHandler, probe:30
MESSAGE : ADM14005E The following error occurred: "AppErr". First Occurrence
Data Capture (FODC) has been invoked in the following mode:
"Automatic". Diagnostic information has been recorded in the
directory named
"/db2dump/FODC_AppErr_2017-03-17-11.30.00.671886_3736014_8430
1_000/".
MESSAGE : ADM0001C A severe error has occurred. Examine the administration
notification log and contact IBM Support if necessary.

2017-03-17-11.30.00.681728+180 I19063098A900 LEVEL: Severe
PID : 3736014 TID : 82548 PROC : db2sysc 0
EDUID : 82548 EDUNAME: db2agent (DB) 0
FUNCTION: DB2 UDB, relation data serv, sqlrr_dump_ffdc, probe:250
DATA #1 : SQLCA, PD_DB2_TYPE_SQLCA, 136 bytes
sqlcaid : SQLCA sqlcabc: 136 sqlcode: -901 sqlerrml: 40
sqlerrmc: Section relink: Section IDs do not match
sqlerrp : SQLRI3D2
sqlerrd : (1) 0x00000000 (2) 0x00000000 (3) 0x00000000
(4) 0x00000000 (5) 0xFFFFFDC4 (6) 0x00000000
sqlwarn : (1) (2) (3) (4) (5) (6)
(7) (8) (9) (10) (11)
sqlstate:

2017-03-17-11.30.00.682123+180 I19063999A658 LEVEL: Severe
PID : 3736014 TID : 82548 PROC : db2sysc 0
EDUID : 82548 EDUNAME: db2agent (DB) 0
FUNCTION: DB2 UDB, relation data serv, sqlrr_dump_ffdc, probe:275
MESSAGE : MASTER TIME INFO
DATA #1 : Hexdump, 32 bytes
0x0A00020115960D30 : 0000 0000 58B1 EA5D 02A3 0000 000A 4FCC ....X..]......O.
0x0A00020115960D40 : 00B4 296A 02D8 24CB 0000 0000 001F 4000 ..)j..$.......@.

2017-03-17-11.30.00.682528+180 I19064658A230 LEVEL: Severe
PID:3736014 TID:82548 NODE:000 Title: MASTER TIME INFO
Dump File:/db2dump/FODC_AppErr_2017-03-17-11.30.00.660471_3736014_82548_000/3736014.82548.000.dump.bin

2017-03-17-11.30.00.687072+180 E19065790A973 LEVEL: Critical
PID : 3736014 TID : 17107 PROC : db2sysc 0
EDUID : 17107 EDUNAME: db2agent (DB) 0
FUNCTION: DB2 UDB, SQO Memory Management, sqloDiagnoseFreeBlockFailure, probe:10
MESSAGE : ADM14001C An unexpected and critical error has occurred: "Panic".
The instance may have been shutdown as a result. "Automatic" FODC
(First Occurrence Data Capture) has been invoked and diagnostic
information has been recorded in directory
"/db2dump/FODC_Trap_2017-03-17-11.30.00.665683_0000/". Please
look in this directory for detailed evidence about what happened and
contact IBM support if necessary to diagnose the problem.

2017-03-17-11.30.00.694186+180 I19066764A219 LEVEL: Severe
PID:3736014 TID:82548 NODE:000 Title: SQLCA
Dump File:/db2dump/FODC_AppErr_2017-03-17-11.30.00.660471_3736014_82548_000/3736014.82548.000.dump.bin

2017-03-17-11.30.00.695249+180 I19066984A559 LEVEL: Severe
PID : 3736014 TID : 82548 PROC : db2sysc 0
EDUID : 82548 EDUNAME: db2agent (DB) 0
FUNCTION: DB2 UDB, relation data serv, sqlrr_dump_ffdc, probe:400

Mark Barinstein,
!

EDUID : 82548 EDUNAME: db2agent (DB) 0
FUNCTION: DB2 UDB, runtime interpreter, sqlriSectRelink, probe:572
DATA #1 : <preformatted>
Section relink failure: Relink ID 00000000 @ 0A0001000270F2C0 to ID D4B9AACB @ 0A00020036654FE0

2017-03-17-11.30.00.679013+180 E19060924A789 LEVEL: Error
PID : 3736014 TID : 82548 PROC : db2sysc 0

все в таком духе((


2017-03-17-11.30.00.695630+180 I19067544A563 LEVEL: Severe
PID : 3736014 TID : 82548 PROC : db2sysc 0
EDUID : 82548 EDUNAME: db2agent (DB) 0
FUNCTION: DB2 UDB, relation data serv, sqlrr_dump_ffdc, probe:450
MESSAGE : PACKAGE
DATA #1 : Hexdump, 11 bytes
0x0A0001002C545FE0 : 5031 3332 3633 3633 3838 37 P1326363887

2017-03-17-11.30.00.696000+180 I19068108A221 LEVEL: Severe
PID:3736014 TID:82548 NODE:000 Title: CREATOR
Dump File:/db2dump/FODC_AppErr_2017-03-17-11.30.00.660471_3736014_82548_000/3736014.82548.000.dump.bin

2017-03-17-11.30.00.696226+180 I19068330A221 LEVEL: Severe
PID:3736014 TID:82548 NODE:000 Title: PACKAGE
Dump File:/db2dump/FODC_AppErr_2017-03-17-11.30.00.660471_3736014_82548_000/3736014.82548.000.dump.bin

2017-03-17-11.30.00.696462+180 I19068552A226 LEVEL: Severe
PID:3736014 TID:82548 NODE:000 Title: REQUEST TYPE
Dump File:/db2dump/FODC_AppErr_2017-03-17-11.30.00.660471_3736014_82548_000/3736014.82548.000.dump.bin

2017-03-17-11.30.00.731689+180 E19068779A2506 LEVEL: Severe
PID : 3736014 TID : 17107 PROC : db2sysc 0
EDUID : 17107 EDUNAME: db2agent (DB) 0
FUNCTION: DB2 UDB, SQO Memory Management, sqloDiagnoseFreeBlockFailure, probe:999
MESSAGE : Memory validation failure, diagnostic file dumped.
DATA #1 : String, 28 bytes
Corrupt pool free tree node.
DATA #2 : File name, 33 bytes
3736014.17107.mem_diagnostics.txt
CALLSTCK: (Static functions may not be resolved correctly, as they are resolved to the nearest symbol)
[0] 0x09000000088770D8 diagnoseMemoryCorruptionAndCrash__13SQLO_MEM_POOLFUlCPCcCb + 0x448
[1] 0x090000000AFB3578 MemTreePut__13SQLO_MEM_POOLFP8SMemNodeUlP17SqloChunkSubgroup + 0x88
[2] 0x090000000AF997C8 sqlofmblkEx + 0x274
[3] 0x090000000AF9A6C0 __dt__16sqlrlCatalogScanFv + 0xC0
Standard input
[4] 0x09000000098920EC sqlrlcAuthidsFindRolelist__FP8sqlrr_cbP11sqlr_aainfoUlP13SQLO_MEM_POOLPP19sqlrlcRolesRequiredPUlT6T3
+ 0x1F8
[5] 0x0900000009892D64 sqlrlcAuthidsFindRolelist__FP8sqlrr_cbP11sqlr_aainfoUlP13SQLO_MEM_POOLPP19sqlrlcRolesRequiredPUlT6T3
+ 0x5D8
[6] 0x090000000988F3A0 sqlrlcRequestRolelistsForAuthid__FP8sqlrr_cbP11sqlr_aainfobN23 + 0x278
[7] 0x090000000A681A1C sqlrr_sql_request_pre__FP14db2UCinterfaceUiiP16db2UCprepareInfoP15db2UCCursorInfo + 0x754
[8] 0x090000000A37A87C sqlrr_open__FP14db2UCinterfaceP15db2UCCursorInfo + 0x390
[9] 0x090000000A37A87C sqlrr_open__FP14db2UCinterfaceP15db2UCCursorInfo + 0x390
[10] 0x090000000A4CFD14 sqljsParseRdbAccessed__FP13sqljsDrdaAsCbP13sqljDDMObjectP14db2UCinterface + 0x2DA0
[11] 0x090000000A4DE3EC @72@sqljsSqlam__FP14db2UCinterfaceP8sqeAgentb + 0x7E4
[12] 0x090000000A4DE3EC @72@sqljsSqlam__FP14db2UCinterfaceP8sqeAgentb + 0x7E4
[13] 0x090000000A4DDFE0 @72@sqljsSqlam__FP14db2UCinterfaceP8sqeAgentb + 0x3D8
[14] 0x09000000097A53A4 @72@sqljsDriveRequests__FP8sqeAgentP14db2UCconHandle + 0xA8
[15] 0x09000000097A5EB0 @72@sqljsDrdaAsInnerDriver__FP18SQLCC_INITSTRUCT_Tb + 0x5F4
[16] 0x0900000009E357D0 RunEDU__8sqeAgentFv + 0x48254
[17] 0x0900000009DD44F8 RunEDU__8sqeAgentFv + 0x124
[18] 0x090000000AE9EFD4 EDUDriver__9sqzEDUObjFv + 0x12C
[19] 0x0900000009FBD018 sqloEDUEntry + 0x394
[20] 0x0900000000542E10 _pthread_body + 0xF0
[21] 0xFFFFFFFFFFFFFFFC ?unknown + 0xFFFFFFFF

2017-03-17-11.30.00.768481+180 I19071286A229 LEVEL: Severe
PID:3736014 TID:82548 NODE:000 Title: APM COMMON INFO
Dump File:/db2dump/FODC_AppErr_2017-03-17-11.30.00.660471_3736014_82548_000/3736014.82548.000.dump.bin

2017-03-17-11.30.00.768871+180 I19071516A225 LEVEL: Severe
PID:3736014 TID:82548 NODE:000 Title: CONTEXT MCB
Dump File:/db2dump/FODC_AppErr_2017-03-17-11.30.00.660471_3736014_82548_000/3736014.82548.000.dump.bin

2017-03-17-11.30.00.769088+180 I19071742A227 LEVEL: Severe
PID:3736014 TID:82548 NODE:000 Title: PACKAGE ENTRY
Dump File:/db2dump/FODC_AppErr_2017-03-17-11.30.00.660471_3736014_82548_000/3736014.82548.000.dump.bin

2017-03-17-11.30.00.769291+180 I19071970A234 LEVEL: Severe
PID:3736014 TID:82548 NODE:000 Title: CACHED PACKAGE ENTRY
Dump File:/db2dump/FODC_AppErr_2017-03-17-11.30.00.660471_3736014_82548_000/3736014.82548.000.dump.bin

2017-03-17-11.30.00.769502+180 I19072205A233 LEVEL: Severe
PID:3736014 TID:82548 NODE:000 Title: Recursive dump_sect
Dump File:/db2dump/FODC_AppErr_2017-03-17-11.30.00.660471_3736014_82548_000/3736014.82548.000.dump.bin

2017-03-17-11.30.00.769722+180 I19072439A228 LEVEL: Severe
PID:3736014 TID:82548 NODE:000 Title: SECTION NUMBER
Dump File:/db2dump/FODC_AppErr_2017-03-17-11.30.00.660471_3736014_82548_000/3736014.82548.000.dump.bin

2017-03-17-11.30.00.769927+180 I19072668A227 LEVEL: Severe
PID:3736014 TID:82548 NODE:000 Title: SECTION ENTRY
Dump File:/db2dump/FODC_AppErr_2017-03-17-11.30.00.660471_3736014_82548_000/3736014.82548.000.dump.bin

2017-03-17-11.30.00.770122+180 I19072896A227 LEVEL: Severe
PID:3736014 TID:82548 NODE:000 Title: SIBLING ENTRY
Dump File:/db2dump/FODC_AppErr_2017-03-17-11.30.00.660471_3736014_82548_000/3736014.82548.000.dump.bin

2017-03-17-11.30.00.770334+180 I19073124A233 LEVEL: Severe
PID:3736014 TID:82548 NODE:000 Title: common section info
Dump File:/db2dump/FODC_AppErr_2017-03-17-11.30.00.660471_3736014_82548_000/3736014.82548.000.dump.bin

2017-03-17-11.30.00.770543+180 I19073358A231 LEVEL: Severe
PID:3736014 TID:82548 NODE:000 Title: workspace sibling
Dump File:/db2dump/FODC_AppErr_2017-03-17-11.30.00.660471_3736014_82548_000/3736014.82548.000.dump.bin

2017-03-17-11.30.00.770741+180 I19073590A231 LEVEL: Severe
PID:3736014 TID:82548 NODE:000 Title: workspace mhashcb
Dump File:/db2dump/FODC_AppErr_2017-03-17-11.30.00.660471_3736014_82548_000/3736014.82548.000.dump.bin

2017-03-17-11.30.00.771055+180 I19073822A660 LEVEL: Severe
PID : 3736014 TID : 82548 PROC : db2sysc 0
EDUID : 82548 EDUNAME: db2agent (DB) 0
FUNCTION: DB2 UDB, relation data serv, sqlrr_dump_section, probe:25
MESSAGE : MASTER TIME INFO
DATA #1 : Hexdump, 32 bytes
0x0A00020115960D30 : 0000 0000 58B1 EA5D 02A3 0000 000A 4FCC ....X..]......O.
0x0A00020115960D40 : 00B4 296A 02D8 24CB 0000 0000 001F 4000 ..)j..$.......@.

2017-03-17-11.30.00.771435+180 I19074483A230 LEVEL: Severe
PID:3736014 TID:82548 NODE:000 Title: MASTER TIME INFO
Dump File:/db2dump/FODC_AppErr_2017-03-17-11.30.00.660471_3736014_82548_000/3736014.82548.000.dump.bin

2017-03-17-11.30.00.771631+180 I19074714A233 LEVEL: Severe

Причём до этого все хорошо и онлайн бекап прошёл и а 11:30
Все просто упало!

Amiri
2017-03-17-11.30.00.695630+180 I19067544A563 LEVEL: Severe
PID : 3736014 TID : 82548 PROC : db2sysc 0
EDUID : 82548 EDUNAME: db2agent (DB) 0
FUNCTION: DB2 UDB, relation data serv, sqlrr_dump_ffdc, probe:450
MESSAGE : PACKAGE
DATA #1 : Hexdump, 11 bytes
0x0A0001002C545FE0 : 5031 3332 3633 3633 3838 37 P1326363887

2017-03-17-11.30.00.696000+180 I19068108A221 LEVEL: Severe
PID:3736014 TID:82548 NODE:000 Title: CREATOR
Dump File:/db2dump/FODC_AppErr_2017-03-17-11.30.00.660471_3736014_82548_000/3736014.82548.000.dump.bin

2017-03-17-11.30.00.696226+180 I19068330A221 LEVEL: Severe
PID:3736014 TID:82548 NODE:000 Title: PACKAGE
Dump File:/db2dump/FODC_AppErr_2017-03-17-11.30.00.660471_3736014_82548_000/3736014.82548.000.dump.bin

2017-03-17-11.30.00.696462+180 I19068552A226 LEVEL: Severe
PID:3736014 TID:82548 NODE:000 Title: REQUEST TYPE
Dump File:/db2dump/FODC_AppErr_2017-03-17-11.30.00.660471_3736014_82548_000/3736014.82548.000.dump.bin

Mark Barinstein,

В понедельник смогу выполнить.

Mark Barinstein,

Добрый день.

увы, ничего нет в логе такого рода.

$ db2diag -t 2017-03-17-11.29.00:2017-03-17-11.32.00 -g "funcname=sqlrr_dump_ffdc,data:=MESSAGE : CREATOR"
$
$ db2diag -t 2017-03-17-11.29.00:2017-03-17-11.32.00 -g "funcname=sqlrr_dump_ffdc,data:=MESSAGE : PACKAGE"
$

Mark Barinstein,

2017-03-17-11.30.00.695249+180 I19066984A559 LEVEL: Severe
PID : 3736014 TID : 82548 PROC : db2sysc 0
EDUID : 82548 EDUNAME: db2agent (DB) 0
FUNCTION: DB2 UDB, relation data serv, sqlrr_dump_ffdc, probe:400
MESSAGE : CREATOR
DATA #1 : Hexdump, 8 bytes
0x0A0001002C545FEB : 5341 5054 4F4F 4C53 SAPTOOLS

2017-03-17-11.30.00.695630+180 I19067544A563 LEVEL: Severe
PID : 3736014 TID : 82548 PROC : db2sysc 0
EDUID : 82548 EDUNAME: db2agent (DB) 0
FUNCTION: DB2 UDB, relation data serv, sqlrr_dump_ffdc, probe:450
MESSAGE : PACKAGE
DATA #1 : Hexdump, 11 bytes
0x0A0001002C545FE0 : 5031 3332 3633 3633 3838 37 P1326363887

Так что-же случилось?
Все файлы трейсов бинарные и нечитабельны.
Как найти причину такого поведения БД db2