Всем привет!
Wildfly 8.2.1, 2 серверных группы main и stage, в каждой 2 ноды. Периодически ноды тихо умирают. Последний раз вчера. В логах ноды куча записей типа
1.
2.
3.
4.
...................
2016-02-24 13:08:37,306 WARN [org.jgroups.protocols.TP$ProtocolAdapter] (INT-1,shared=udp) JGRP000031: master:main-2/web: dropping unicast message to wrong destination master:main-2/web
2016-02-24 13:08:37,627 WARN [org.jgroups.protocols.TP$ProtocolAdapter] (OOB-21,shared=udp) JGRP000031: master:main-2/ejb: dropping unicast message to wrong destination master:main-2/ejb
2016-02-24 13:08:37,657 WARN [org.jgroups.protocols.TP$ProtocolAdapter] (INT-1,shared=udp) JGRP000031: master:main-2/ejb: dropping unicast message to wrong destination master:main-2/ejb
.
Сам процесс умер. Кто-нибудь сталкивался с таким? И как диагностировать причину? По этим записям непонятно.
Возможно имеется связь, но в console.log периодически наблюдаются стектрейсы
1.
2.
3.
4.
5.
6.
[Server:stage-2] 00:15:24,490 ERROR [io.undertow.request] (default task-3) UT005023: Exception handling request to /broker/rs/reports/order/1406e623-0112-4a0b-afae-8dfb3d044aef/status: java.lang.IllegalStateException: Transaction DummyTransaction{xid=DummyXid{id=76671}, status=1} is not in a valid state to be invoking cache operations on.t(TxInterceptor.java:275) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
[Server:stage-2] at org.infinispan.interceptors.TxInterceptor.enlistIfNeeded(TxInterceptor.java:231) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
[Server:stage-2] at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:225) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
[Server:stage-2] at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:221) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
[Server:stage-2] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
......
и
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
[Server:stage-2] 00:42:57,056 ERROR [org.infinispan.interceptors.InvocationContextInterceptor] (default task-17) ISPN000136: Execution error: org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [15 seconds] on key [f-iqzKWLlU0OsJ59lzX9-OIa] for requestor [GlobalTransaction:<master:stage-2/web>:149334:local]! Lock held by [GlobalTransaction:<master:stage-2/web>:149322:local]
[Server:stage-2] at org.infinispan.util.concurrent.locks.LockManagerImpl.lock(LockManagerImpl.java:198) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
[Server:stage-2] at org.infinispan.util.concurrent.locks.LockManagerImpl.acquireLock(LockManagerImpl.java:171) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
[Server:stage-2] at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockKeyAndCheckOwnership(AbstractTxLockingInterceptor.java:169) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
[Server:stage-2] at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitGetKeyValueCommand(PessimisticLockingInterceptor.java:70) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
[Server:stage-2] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
[Server:stage-2] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
[Server:stage-2] at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:112) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
[Server:stage-2] at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:74) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
[Server:stage-2] at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:40) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
[Server:stage-2] at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:98) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
[Server:stage-2] at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:226) [infinispan-core-6.0.2.Final.jar:6.0.2.Final]
...........
