sqloDiagnoseFreeBlockFailure 数据库宕机？
数据库实例当掉了，数据库版本信息、日志报错信息如下；
自己网路上查询有人说可能是BUG：IC78144: NON-THREADED STORED PROCEDURE COMPILED IN 32 BIT MODE MAY CAUSE INSTANCE CRASH
我还是不明白什么是non-threaded stored procedure?? 我最近在做db2move db export的操作会不会是导出操作引起，请专家们帮忙分析一下，万分感谢！！！！
OS：AIX5.3
Informational tokens are "DB2 v9.5.0.7", "s101129", "IP23136", and Fix Pack
"7"
2018-05-07-19.44.55.335479+480 E3552755258A955 LEVEL: Critical
PID : 5361864 TID : 97807 PROC : db2sysc 0
INSTANCE: db2inst1 NODE : 000 DB : DB
APPHDL : 0-8379 APPID: *LOCAL.db2inst3.180507030630
AUTHID : DB2INST3
EDUID : 97807 EDUNAME: db2agent (CRVDB) 0
FUNCTION: DB2 UDB, SQO Memory Management, sqloDiagnoseFreeBlockFailure, probe:10
MESSAGE : ADM14001C An unexpected and critical error has occurred: "Panic".
The instance may have been shutdown as a result. "Automatic" FODC
(First Occurrence Data Capture) has been invoked and diagnostic
information has been recorded in directory
"/home/db2inst1/sqllib/db2dump/FODC_Panic_2018-05-07-19.44.55.303663/
". Please look in this directory for detailed evidence about what
happened and contact IBM support if necessary to diagnose the
problem.
2同行回答
- 报错前后的日志： 2018-05-07-19.37.37.430998+480 E3552752407A465 LEVEL: Event PID : 5361864 TID : 17146 PROC : db2sysc 0 INSTANCE: db2inst3 NODE : 000 DB : REPDB APPHDL : 0-42202 APPID: *LOCAL.db2inst3.180507131323 AUTHID : PATROL EDUID : 17146 EDUNAME: db2agent (REPDB) 0 FUNCTION: DB2 UDB, base sys utilities, sqeLocalDatabase::FirstConnect, probe:1000 START : DATABASE: REPDB : ACTIVATED: NO 2018-05-07-19.37.48.259435+480 E3552752873A467 LEVEL: Event PID : 5361864 TID : 17146 PROC : db2sysc 0 INSTANCE: db2inst3 NODE : 000 DB : REPDB APPHDL : 0-42202 APPID: *LOCAL.db2inst3.180507131323 AUTHID : PATROL EDUID : 17146 EDUNAME: db2agent (idle) 0 FUNCTION: DB2 UDB, base sys utilities, sqeLocalDatabase::TermDbConnect, probe:2000 STOP : DATABASE: REPDB : DEACTIVATED: NO 2018-05-07-19.37.58.594383+480 E3552753341A465 LEVEL: Event PID : 5361864 TID : 17146 PROC : db2sysc 0 INSTANCE: db2inst3 NODE : 000 DB : REPHB APPHDL : 0-42233 APPID: *LOCAL.db2inst3.180507131347 AUTHID : PATROL EDUID : 17146 EDUNAME: db2agent (REPHB) 0 FUNCTION: DB2 UDB, base sys utilities, sqeLocalDatabase::FirstConnect, probe:1000 START : DATABASE: REPHB : ACTIVATED: NO 2018-05-07-19.38.08.319358+480 E3552753807A467 LEVEL: Event PID : 5361864 TID : 17146 PROC : db2sysc 0 INSTANCE: db2inst3 NODE : 000 DB : REPHB APPHDL : 0-42233 APPID: *LOCAL.db2inst3.180507131347 AUTHID : PATROL EDUID : 17146 EDUNAME: db2agent (idle) 0 FUNCTION: DB2 UDB, base sys utilities, sqeLocalDatabase::TermDbConnect, probe:2000 STOP : DATABASE: REPHB : DEACTIVATED: NO 2018-05-07-19.40.43.846323+480 I3552754275A348 LEVEL: Event PID : 5283896 TID : 772 PROC : db2acd 0 INSTANCE: db2inst3 NODE : 000 EDUID : 772 EDUNAME: db2acd 0 FUNCTION: DB2 UDB, Administrative Task Scheduler, db2HmonAtsPollDatabase, probe:250 REPORT : Database REPDB is unregistered 2018-05-07-19.44.55.232833+480 I3552754624A633 LEVEL: Error PID : 5337316 TID : 1 PROC : db2move INSTANCE: db2inst3 NODE : 000 APPID : *LOCAL.db2inst3.180507030630 EDUID : 1 FUNCTION: <0>, <0>, <0>, probe:2110 MESSAGE : SQL30020N Execution of the command or SQL statement failed because of a syntax error in the communication data stream that will affect the successful execution of subsequent commands and SQL statements: Reason Code "". DATA #1 : String, 12 bytes sqluexpm.SQC DATA #2 : String, 44 bytes Error from Import/Export or Load Processing. 2018-05-07-19.44.55.335479+480 E3552755258A955 LEVEL: Critical PID : 5361864 TID : 97807 PROC : db2sysc 0 INSTANCE: db2inst3 NODE : 000 DB : CRVDB APPHDL : 0-8379 APPID: *LOCAL.db2inst3.180507030630 AUTHID : DB2INST3 EDUID : 97807 EDUNAME: db2agent (CRVDB) 0 FUNCTION: DB2 UDB, SQO Memory Management, sqloDiagnoseFreeBlockFailure, probe:10 MESSAGE : ADM14001C An unexpected and critical error has occurred: "Panic". The instance may have been shutdown as a result. "Automatic" FODC (First Occurrence Data Capture) has been invoked and diagnostic information has been recorded in directory "/home/db2inst3/sqllib/db2dump/FODC_Panic_2018-05-07-19.44.55.303663/ ". Please look in this directory for detailed evidence about what happened and contact IBM support if necessary to diagnose the problem. 2018-05-07-19.44.55.362479+480 E3552756214A1400 LEVEL: Severe PID : 5361864 TID : 97807 PROC : db2sysc 0 INSTANCE: db2inst3 NODE : 000 DB : CRVDB APPHDL : 0-8379 APPID: *LOCAL.db2inst3.180507030630 AUTHID : DB2INST3 EDUID : 97807 EDUNAME: db2agent (CRVDB) 0 FUNCTION: DB2 UDB, SQO Memory Management, sqloDiagnoseFreeBlockFailure, probe:999 MESSAGE : Memory validation failure, diagnostic file dumped. DATA #1 : String, 28 bytes Corrupt pool free tree node. DATA #2 : File name, 33 bytes 5361864.97807.mem_diagnostics.txt CALLSTCK: [0] 0x0900000008210F6C pdLog + 0x88 [1] 0x09000000077206C8 diagnoseMemoryCorruptionAndCrash__13SQLO_MEM_POOLFUlCPCc + 0x258 [2] 0x0900000007720394 diagnoseMemoryCorruptionAndCrash__13SQLO_MEM_POOLFUlCPCc@glue6D2 + 0x74 [3] 0x090000000B353ACC .allocateMemoryBlock__13SQLO_MEM_POOLFCUlUlT1UiT1PPvPP17SqloChunkSubgroupPP12SMemLostNodeCP12SMemLogEvent_fdprpro_clone_151 + 0xA4 [4] 0x090000000B056514 ?unknown + 0xFFFFB6CF [5] 0x09000000081D98A0 sqlogmblkEx + 0xFFFFFFFFFFFFFD1C [6] 0x09000000081D83C0 sqlmoAllocBlock + 0x90 [7] 0x09000000081D859C sqlogmblkEx + 0xFFFFFFFFFFFFFE8C [8] 0x0900000008248C38 sqlra_sqlW_mem_get_block_ex__FP8sqlrr_cbUlN22UiPPvPCcCUl + 0x68 [9] 0x0900000008248ADC sqlra_sqlW_copy_section__FP8sqlrr_cbP23sqlra_workspace_siblingP9sqlri_shd + 0x94 2018-05-07-19.44.55.429015+480 I3552757615A561 LEVEL: Error PID : 5361864 TID : 97807 PROC : db2sysc 0 INSTANCE: db2inst1 NODE : 000 DB : CRVDB APPHDL : 0-8379 APPID: *LOCAL.db2inst1.180507030630 AUTHID : db2inst1 EDUID : 97807 EDUNAME: db2agent (CRVDB) 0 FUNCTION: DB2 UDB, base sys utilities, sqleagnt_sigsegvh, probe:1 MESSAGE : Error in agent servicing application with coor_node: DATA #1 : Hexdump, 2 bytes 0x0700000048009330 : 0000 .. 2018-05-07-19.44.55.429157+480 I3552758177A568 LEVEL: Error PID : 5361864 TID : 97807 PROC : db2sysc 0 INSTANCE: db2inst1 NODE : 000 DB : CRVDB APPHDL : 0-8379 APPID: *LOCAL.db2inst1.180507030630 AUTHID : db2inst1 EDUID : 97807 EDUNAME: db2agent (CRVDB) 0 FUNCTION: DB2 UDB, base sys utilities, sqleagnt_sigsegvh, probe:2 MESSAGE : Error in agent servicing application with coor_agent_index: DATA #1 : Hexdump, 2 bytes 0x0700000048009332 : 20BB . 2018-05-07-19.44.55.429258+480 I3552758746A567 LEVEL: Error PID : 5361864 TID : 97807 PROC : db2sysc 0 INSTANCE: db2inst1 NODE : 000 DB : CRVDB APPHDL : 0-8379 APPID: *LOCAL.db2inst1.180507030630 AUTHID : db2inst1 EDUID : 97807 EDUNAME: db2agent (CRVDB) 0 FUNCTION: DB2 UDB, base sys utilities, sqleagnt_sigsegvh, probe:3 MESSAGE : Error in agent servicing application with CLIENT PID: DATA #1 : Hexdump, 7 bytes 0x0700000048009350 : 3533 3337 3331 36 5337316 2018-05-07-19.44.55.429357+480 I3552759314A666 LEVEL: Error PID : 5361864 TID : 97807 PROC : db2sysc 0 INSTANCE: db2inst1 NODE : 000 DB : CRVDB APPHDL : 0-8379 APPID: *LOCAL.db2inst1.180507030630 AUTHID : db2inst1 EDUID : 97807 EDUNAME: db2agent (CRVDB) 0 FUNCTION: DB2 UDB, base sys utilities, sqleagnt_sigsegvh, probe:4 MESSAGE : Error in agent servicing application with INBOUND APPLICATION ID: DATA #1 : Hexdump, 28 bytes 0x078000000319290C : 2A4C 4F43 414C 2E64 6232 696E 7374 332E *LOCAL.db2inst1. 0x078000000319291C : 3138 3035 3037 3033 3036 3330 180507030630 2018-05-07-19.44.55.429466+480 I3552759981A575 LEVEL: Error PID : 5361864 TID : 97807 PROC : db2sysc 0 INSTANCE: db2inst1 NODE : 000 DB : CRVDB APPHDL : 0-8379 APPID: *LOCAL.db2inst1.180507030630 AUTHID : db2inst1 EDUID : 97807 EDUNAME: db2agent (CRVDB) 0 FUNCTION: DB2 UDB, base sys utilities, sqleagnt_sigsegvh, probe:5 MESSAGE : Error in agent servicing application with INBOUND SEQUENCE NUMBER: DATA #1 : Hexdump, 2 bytes 0x0780000003192450 : 0593 .. 2018-05-07-19.44.55.429564+480 I3552760557A481 LEVEL: Error PID : 5361864 TID : 97807 PROC : db2sysc 0 INSTANCE: db2inst1 NODE : 000 DB : CRVDB APPHDL : 0-8379 APPID: *LOCAL.db2inst1.180507030630 AUTHID : db2inst1 EDUID : 97807 EDUNAME: db2agent (CRVDB) 0 FUNCTION: DB2 UDB, base sys utilities, sqleagnt_sigsegvh, probe:6 MESSAGE : Error in agent servicing application with OUTBOUND APPLICATION ID: 2018-05-07-19.44.55.429660+480 I3552761039A482 LEVEL: Error PID : 5361864 TID : 97807 PROC : db2sysc 0 INSTANCE: db2inst1 NODE : 000 DB : CRVDB APPHDL : 0-8379 APPID: *LOCAL.db2inst1.180507030630 AUTHID : db2inst1 EDUID : 97807 EDUNAME: db2agent (CRVDB) 0 FUNCTION: DB2 UDB, base sys utilities, sqleagnt_sigsegvh, probe:7 MESSAGE : Error in agent servicing application with OUTBOUND SEQUENCE NUMBER:
- 这就很明显了吧，在报错内容上一条，就说明是db2move引起的，在导出过程中是不是有其他的什么操作？
- 没有起他操作哦，只是执行dbmove db export操作。
- 总共宕机了三次，有两次之前都是执行了db2move，看来真的是他引起的。
2018-05-07-19.37.37.430998+480 E3552752407A465 LEVEL: Event
PID : 5361864 TID : 17146 PROC : db2sysc 0
INSTANCE: db2inst1 NODE : 000 DB : REPDB
APPHDL : 0-42202 APPID: *LOCAL.db2inst1.180507131323
AUTHID : PATROL
EDUID : 17146 EDUNAME: db2agent (REPDB) 0
FUNCTION: DB2 UDB, base sys utilities, sqeLocalDatabase::FirstConnect, probe:1000
START : DATABASE: REPDB : ACTIVATED: NO
2018-05-07-19.37.48.259435+480 E3552752873A467 LEVEL: Event
PID : 5361864 TID : 17146 PROC : db2sysc 0
INSTANCE: db2inst1 NODE : 000 DB : REPDB
APPHDL : 0-42202 APPID: *LOCAL.db2inst1.180507131323
AUTHID : PATROL
EDUID : 17146 EDUNAME: db2agent (idle) 0
FUNCTION: DB2 UDB, base sys utilities, sqeLocalDatabase::TermDbConnect, probe:2000
STOP : DATABASE: REPDB : DEACTIVATED: NO
2018-05-07-19.37.58.594383+480 E3552753341A465 LEVEL: Event
PID : 5361864 TID : 17146 PROC : db2sysc 0
INSTANCE: db2inst1 NODE : 000 DB : REPHB
APPHDL : 0-42233 APPID: *LOCAL.db2inst1.180507131347
AUTHID : PATROL
EDUID : 17146 EDUNAME: db2agent (REPHB) 0
FUNCTION: DB2 UDB, base sys utilities, sqeLocalDatabase::FirstConnect, probe:1000
START : DATABASE: REPHB : ACTIVATED: NO
2018-05-07-19.38.08.319358+480 E3552753807A467 LEVEL: Event
PID : 5361864 TID : 17146 PROC : db2sysc 0
INSTANCE: db2inst1 NODE : 000 DB : REPHB
APPHDL : 0-42233 APPID: *LOCAL.db2inst1.180507131347
AUTHID : PATROL
EDUID : 17146 EDUNAME: db2agent (idle) 0
FUNCTION: DB2 UDB, base sys utilities, sqeLocalDatabase::TermDbConnect, probe:2000
STOP : DATABASE: REPHB : DEACTIVATED: NO
2018-05-07-19.40.43.846323+480 I3552754275A348 LEVEL: Event
PID : 5283896 TID : 772 PROC : db2acd 0
INSTANCE: db2inst1 NODE : 000
EDUID : 772 EDUNAME: db2acd 0
FUNCTION: DB2 UDB, Administrative Task Scheduler, db2HmonAtsPollDatabase, probe:250
REPORT : Database REPDB is unregistered
2018-05-07-19.44.55.232833+480 I3552754624A633 LEVEL: Error
PID : 5337316 TID : 1 PROC : db2move
INSTANCE: db2inst1 NODE : 000
APPID : *LOCAL.db2inst1.180507030630
EDUID : 1
FUNCTION: <0>, <0>, <0>, probe:2110
MESSAGE : SQL30020N Execution of the command or SQL statement failed because
of a syntax error in the communication data stream that will affect
the successful execution of subsequent commands and SQL statements:
Reason Code "".DATA #1 : String, 12 bytes
sqluexpm.SQC
DATA #2 : String, 44 bytes
Error from Import/Export or Load Processing.
2018-05-07-19.44.55.335479+480 E3552755258A955 LEVEL: Critical
PID : 5361864 TID : 97807 PROC : db2sysc 0
INSTANCE: db2inst1 NODE : 000 DB : CRVDB
APPHDL : 0-8379 APPID: *LOCAL.db2inst1.180507030630
AUTHID : db2inst1
EDUID : 97807 EDUNAME: db2agent (CRVDB) 0
FUNCTION: DB2 UDB, SQO Memory Management, sqloDiagnoseFreeBlockFailure, probe:10
MESSAGE : ADM14001C An unexpected and critical error has occurred: "Panic".
The instance may have been shutdown as a result. "Automatic" FODC
(First Occurrence Data Capture) has been invoked and diagnostic
information has been recorded in directory
"/home/db2inst1/sqllib/db2dump/FODC_Panic_2018-05-07-19.44.55.303663/
". Please look in this directory for detailed evidence about what
happened and contact IBM support if necessary to diagnose the
problem.
2018-05-07-19.44.55.362479+480 E3552756214A1400 LEVEL: Severe
PID : 5361864 TID : 97807 PROC : db2sysc 0
INSTANCE: db2inst1 NODE : 000 DB : CRVDB
APPHDL : 0-8379 APPID: *LOCAL.db2inst1.180507030630
AUTHID : db2inst1
EDUID : 97807 EDUNAME: db2agent (CRVDB) 0
FUNCTION: DB2 UDB, SQO Memory Management, sqloDiagnoseFreeBlockFailure, probe:999
MESSAGE : Memory validation failure, diagnostic file dumped.
DATA #1 : String, 28 bytes
Corrupt pool free tree node.
DATA #2 : File name, 33 bytes
5361864.97807.mem_diagnostics.txt
CALLSTCK:
[0] 0x0900000008210F6C pdLog + 0x88
[1] 0x09000000077206C8 diagnoseMemoryCorruptionAndCrash__13SQLO_MEM_POOLFUlCPCc + 0x258
[2] 0x0900000007720394 diagnoseMemoryCorruptionAndCrash__13SQLO_MEM_POOLFUlCPCc@glue6D2 + 0x74
[3] 0x090000000B353ACC .allocateMemoryBlock__13SQLO_MEM_POOLFCUlUlT1UiT1PPvPP17SqloChunkSubgroupPP12SMemLostNodeCP12SMemLogEvent_fdprpro_clone_151 + 0xA4
[4] 0x090000000B056514 ?unknown + 0xFFFFB6CF
[5] 0x09000000081D98A0 sqlogmblkEx + 0xFFFFFFFFFFFFFD1C
[6] 0x09000000081D83C0 sqlmoAllocBlock + 0x90
[7] 0x09000000081D859C sqlogmblkEx + 0xFFFFFFFFFFFFFE8C
[8] 0x0900000008248C38 sqlra_sqlW_mem_get_block_ex__FP8sqlrr_cbUlN22UiPPvPCcCUl + 0x68
[9] 0x0900000008248ADC sqlra_sqlW_copy_section__FP8sqlrr_cbP23sqlra_workspace_siblingP9sqlri_shd + 0x94
2018-05-07-19.44.55.429015+480 I3552757615A561 LEVEL: Error
PID : 5361864 TID : 97807 PROC : db2sysc 0
INSTANCE: db2inst1 NODE : 000 DB : CRVDB
APPHDL : 0-8379 APPID: *LOCAL.db2inst1.180507030630
AUTHID : db2inst1
EDUID : 97807 EDUNAME: db2agent (CRVDB) 0
FUNCTION: DB2 UDB, base sys utilities, sqleagnt_sigsegvh, probe:1
MESSAGE : Error in agent servicing application with coor_node:
DATA #1 : Hexdump, 2 bytes
0x0700000048009330 : 0000 ..
2018-05-07-19.44.55.429157+480 I3552758177A568 LEVEL: Error
PID : 5361864 TID : 97807 PROC : db2sysc 0
INSTANCE: db2inst1 NODE : 000 DB : CRVDB
APPHDL : 0-8379 APPID: *LOCAL.db2inst1.180507030630
AUTHID : db2inst1
EDUID : 97807 EDUNAME: db2agent (CRVDB) 0
FUNCTION: DB2 UDB, base sys utilities, sqleagnt_sigsegvh, probe:2
MESSAGE : Error in agent servicing application with coor_agent_index:
DATA #1 : Hexdump, 2 bytes
0x0700000048009332 : 20BB .
2018-05-07-19.44.55.429258+480 I3552758746A567 LEVEL: Error
PID : 5361864 TID : 97807 PROC : db2sysc 0
INSTANCE: db2inst1 NODE : 000 DB : CRVDB
APPHDL : 0-8379 APPID: *LOCAL.db2inst1.180507030630
AUTHID : db2inst1
EDUID : 97807 EDUNAME: db2agent (CRVDB) 0
FUNCTION: DB2 UDB, base sys utilities, sqleagnt_sigsegvh, probe:3
MESSAGE : Error in agent servicing application with CLIENT PID:
DATA #1 : Hexdump, 7 bytes
0x0700000048009350 : 3533 3337 3331 36 5337316
2018-05-07-19.44.55.429357+480 I3552759314A666 LEVEL: Error
PID : 5361864 TID : 97807 PROC : db2sysc 0
INSTANCE: db2inst1 NODE : 000 DB : CRVDB
APPHDL : 0-8379 APPID: *LOCAL.db2inst1.180507030630
AUTHID : db2inst1
EDUID : 97807 EDUNAME: db2agent (CRVDB) 0
FUNCTION: DB2 UDB, base sys utilities, sqleagnt_sigsegvh, probe:4
MESSAGE : Error in agent servicing application with INBOUND APPLICATION ID:
DATA #1 : Hexdump, 28 bytes
0x078000000319290C : 2A4C 4F43 414C 2E64 6232 696E 7374 332E *LOCAL.db2inst1.
0x078000000319291C : 3138 3035 3037 3033 3036 3330 180507030630
2018-05-07-19.44.55.429466+480 I3552759981A575 LEVEL: Error
PID : 5361864 TID : 97807 PROC : db2sysc 0
INSTANCE: db2inst1 NODE : 000 DB : CRVDB
APPHDL : 0-8379 APPID: *LOCAL.db2inst1.180507030630
AUTHID : db2inst1
EDUID : 97807 EDUNAME: db2agent (CRVDB) 0
FUNCTION: DB2 UDB, base sys utilities, sqleagnt_sigsegvh, probe:5
MESSAGE : Error in agent servicing application with INBOUND SEQUENCE NUMBER:
DATA #1 : Hexdump, 2 bytes
0x0780000003192450 : 0593 ..
2018-05-07-19.44.55.429564+480 I3552760557A481 LEVEL: Error
PID : 5361864 TID : 97807 PROC : db2sysc 0
INSTANCE: db2inst1 NODE : 000 DB : CRVDB
APPHDL : 0-8379 APPID: *LOCAL.db2inst1.180507030630
AUTHID : db2inst1
EDUID : 97807 EDUNAME: db2agent (CRVDB) 0
FUNCTION: DB2 UDB, base sys utilities, sqleagnt_sigsegvh, probe:6
MESSAGE : Error in agent servicing application with OUTBOUND APPLICATION ID:
2018-05-07-19.44.55.429660+480 I3552761039A482 LEVEL: Error
PID : 5361864 TID : 97807 PROC : db2sysc 0
INSTANCE: db2inst1 NODE : 000 DB : CRVDB
APPHDL : 0-8379 APPID: *LOCAL.db2inst1.180507030630
AUTHID : db2inst1
EDUID : 97807 EDUNAME: db2agent (CRVDB) 0
FUNCTION: DB2 UDB, base sys utilities, sqleagnt_sigsegvh, probe:7
MESSAGE : Error in agent servicing application with OUTBOUND SEQUENCE NUMBER: