Repository navigation
fix(net): release sync requests when block deserialization fails #7093
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: release_v4.8.3
Are you sure you want to change the base?
Changes from all commits
6bd88e1
f12c112
cfdbfcf
dee5f44
7b1150b
31d366f
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -321,9 +321,14 @@ private synchronized void handleSyncBlock() { | |
| isProcessed[0] = true; | ||
| BlockCapsule block; | ||
| try { | ||
| block = new BlockCapsule(unparsedBlock.getData()); | ||
| block = new BlockMessage(unparsedBlock.getData()).getBlockCapsule(); | ||
| } catch (Exception e) { | ||
| logger.warn("Deserialize block {} failed", blockId.getString(), e); | ||
| logger.warn("Deserialize block {} from {} failed", | ||
| blockId.getString(), peerConnection.getInetSocketAddress(), e); | ||
| // Deferred validation may fail after ingress, e.g. on ALLOW_PROTO_FILTER_NUM changes | ||
| // or interruption. Release sync state so this block remains eligible for retry. | ||
| peerConnection.getSyncBlockInProcess().remove(blockId); | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [SHOULD] Please use the same deserialization path as the initial network parse: block = new BlockMessage(unparsedBlock.getData()).getBlockCapsule();The reproduced stall comes from using different Protobuf overloads: Please adjust the regression to verify that, with filtering disabled, the block is processed successfully and the provider stays connected. With filtering enabled, the payload should still be rejected at the initial parse. Keep the existing catch; cleanup for other deferred-parsing failures can be considered separately.
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I disagree with making both parsing paths accept this payload. A block message with an extra, unmatched trailing In f12c112, BlockMessage now calls The tests verify ingress rejection, disconnect cleanup, and retry through another peer. Deferred-failure cleanup remains as a defensive safeguard, tested separately with damaged queued data.
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I understand that an unmatched top-level I still recommend using the same parsing path in this PR to fix the regression with a minimal change. If we instead reject the payload at ingress, it will never reach deferred parsing. In that case, adding the ingress check is sufficient to fix this reproduction, and I suggest removing the newly added state cleanup and
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Updated in 7b1150b0b5. Deferred decoding now uses Removed the unconditional The regression now exercises a filter change after enqueueing: it verifies that deferred rejection releases sync state without disconnecting the provider, then a canonical response from another peer is processed successfully. Validation: 49 focused tests,
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [NIT] Add one line above this catch noting it is a defensive safeguard, now that ingress rejects trailing END_GROUP With Not blocking; a single line here would settle both the reachability question and the "is this dead code" concern. Suggestion: Add one comment line above the catch, e.g.: "Ingress validation rejects malformed payloads; this cleanup is a defensive safeguard for queued data damaged after receipt or for future divergence between the two parse entries — do not remove without re-checking the #6717 stall recurrence."
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Added in 7b1150b0b5. The comment now explains that deferred validation may fail after successful ingress—for example, if Deferred decoding now also reuses BlockMessage. The unconditional BAD_BLOCK disconnect has been removed; the regression covers a filter change after enqueueing and verifies cleanup and retry without disconnecting the provider. |
||
| invalid(blockId, peerConnection); | ||
| return; | ||
| } | ||
| processSyncBlock(block, peerConnection); | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,55 @@ | ||
| package org.tron.core.net.message.adv; | ||
|
|
||
| import static org.junit.Assert.assertEquals; | ||
| import static org.junit.Assert.assertThrows; | ||
| import static org.mockito.Mockito.CALLS_REAL_METHODS; | ||
| import static org.mockito.Mockito.mockStatic; | ||
|
|
||
| import com.google.protobuf.ByteString; | ||
| import com.google.protobuf.InvalidProtocolBufferException; | ||
| import java.util.Arrays; | ||
| import org.junit.Test; | ||
| import org.mockito.MockedStatic; | ||
| import org.tron.common.overlay.message.Message; | ||
| import org.tron.protos.Protocol.Block; | ||
| import org.tron.protos.Protocol.BlockHeader; | ||
|
|
||
| public class BlockMessageTest { | ||
|
|
||
| private static Block sampleBlock() { | ||
| return Block.newBuilder().setBlockHeader(BlockHeader.newBuilder() | ||
| .setRawData(BlockHeader.raw.newBuilder().setNumber(100).setTimestamp(123456789L)) | ||
| .setWitnessSignature(ByteString.copyFrom(new byte[] {0x0c}))) | ||
| .build(); | ||
| } | ||
|
|
||
| @Test | ||
| public void trailingEndGroupIsRejectedWithEitherFilterSetting() { | ||
| byte[] canonical = sampleBlock().toByteArray(); | ||
| for (boolean filter : new boolean[] {false, true}) { | ||
| try (MockedStatic<Message> messages = mockStatic(Message.class, CALLS_REAL_METHODS)) { | ||
| messages.when(Message::isFilter).thenReturn(filter); | ||
| // Also reject an END_GROUP followed by more bytes, not just a final 0x0c byte. | ||
| for (byte[] suffix : new byte[][] {{0x0c}, {0x0c, 0x08, 0x01}}) { | ||
| byte[] malformed = Arrays.copyOf(canonical, canonical.length + suffix.length); | ||
| System.arraycopy(suffix, 0, malformed, canonical.length, suffix.length); | ||
| assertThrows(InvalidProtocolBufferException.class, () -> new BlockMessage(malformed)); | ||
| } | ||
| } | ||
| } | ||
| } | ||
|
|
||
| @Test | ||
| public void endGroupByteInsideABytesFieldIsAccepted() throws Exception { | ||
| Block block = sampleBlock(); | ||
| byte[] data = block.toByteArray(); | ||
| assertEquals(0x0c, data[data.length - 1]); | ||
| for (boolean filter : new boolean[] {false, true}) { | ||
| try (MockedStatic<Message> messages = mockStatic(Message.class, CALLS_REAL_METHODS)) { | ||
| messages.when(Message::isFilter).thenReturn(filter); | ||
| assertEquals(block, new BlockMessage(data).getBlockCapsule().getInstance()); | ||
| } | ||
| } | ||
| } | ||
|
|
||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -3,8 +3,10 @@ | |
| import static org.mockito.Mockito.mock; | ||
|
|
||
| import com.google.common.cache.Cache; | ||
| import com.google.protobuf.ByteString; | ||
| import java.lang.reflect.Field; | ||
| import java.lang.reflect.Method; | ||
| import java.math.BigInteger; | ||
| import java.net.InetSocketAddress; | ||
| import java.util.ArrayList; | ||
| import java.util.Arrays; | ||
|
|
@@ -13,21 +15,26 @@ | |
| import java.util.Map; | ||
| import org.junit.Assert; | ||
| import org.junit.Test; | ||
| import org.mockito.ArgumentCaptor; | ||
| import org.mockito.Mockito; | ||
| import org.springframework.context.ApplicationContext; | ||
| import org.tron.common.BaseMethodTest; | ||
| import org.tron.common.crypto.ECKey; | ||
| import org.tron.common.overlay.message.Message; | ||
| import org.tron.common.utils.ReflectUtils; | ||
| import org.tron.common.utils.Sha256Hash; | ||
| import org.tron.core.capsule.BlockCapsule; | ||
| import org.tron.core.net.P2pEventHandlerImpl; | ||
| import org.tron.core.net.TronNetDelegate; | ||
| import org.tron.core.net.message.MessageTypes; | ||
| import org.tron.core.net.message.adv.BlockMessage; | ||
| import org.tron.core.net.message.adv.FetchInvDataMessage; | ||
| import org.tron.core.net.messagehandler.BlockMsgHandler; | ||
| import org.tron.core.net.messagehandler.PbftDataSyncHandler; | ||
| import org.tron.core.net.peer.PeerConnection; | ||
| import org.tron.core.net.peer.PeerManager; | ||
| import org.tron.core.net.peer.TronState; | ||
| import org.tron.core.net.service.adv.AdvService; | ||
| import org.tron.core.net.service.sync.SyncService; | ||
| import org.tron.core.net.service.sync.UnparsedBlock; | ||
| import org.tron.p2p.connection.Channel; | ||
|
|
@@ -270,6 +277,177 @@ public void testProcessingBlockResumesThrottledFetch() throws Exception { | |
| sync.close(); | ||
| } | ||
|
|
||
| @Test | ||
| public void testStoredGenesisBlockRoundTripsThroughNetworkDecoder() throws Exception { | ||
| BlockCapsule stored = chainBaseManager.getBlockById(chainBaseManager.getGenesisBlockId()); | ||
| for (int filter : new int[] {0, 1}) { | ||
| chainBaseManager.getDynamicPropertiesStore().saveAllowProtoFilterNum(filter); | ||
| BlockMessage message = new BlockMessage(new BlockMessage(stored).getData()); | ||
| Assert.assertEquals(stored.getInstance(), message.getBlockCapsule().getInstance()); | ||
| Assert.assertEquals(stored.getBlockId(), message.getBlockId()); | ||
| Assert.assertArrayEquals(stored.getData(), message.getData()); | ||
| } | ||
| } | ||
|
|
||
| @Test | ||
| public void testIngressParseFailureAllowsRetryAfterDisconnect() throws Exception { | ||
| chainBaseManager.getDynamicPropertiesStore().saveAllowProtoFilterNum(0); | ||
| SyncService sync = new SyncService(); | ||
| try { | ||
| TronNetDelegate delegate = mock(TronNetDelegate.class); | ||
| ReflectUtils.setFieldValue(sync, "tronNetDelegate", delegate); | ||
| PeerConnection badPeer = Mockito.spy(context.getBean(PeerConnection.class)); | ||
| PeerConnection goodPeer = Mockito.spy(context.getBean(PeerConnection.class)); | ||
| Channel channel = mock(Channel.class); | ||
| Channel goodChannel = mock(Channel.class); | ||
| Mockito.when(channel.getInetSocketAddress()).thenReturn(inetSocketAddress); | ||
| Mockito.when(channel.getInetAddress()).thenReturn(inetSocketAddress.getAddress()); | ||
| Mockito.when(goodChannel.getInetSocketAddress()).thenReturn(inetSocketAddress); | ||
| Mockito.when(goodChannel.getInetAddress()).thenReturn(inetSocketAddress.getAddress()); | ||
| badPeer.setChannel(channel); | ||
| goodPeer.setChannel(goodChannel); | ||
| badPeer.setFetchAble(true); | ||
| goodPeer.setFetchAble(true); | ||
| ReflectUtils.setFieldValue(badPeer, "syncService", sync); | ||
| ReflectUtils.setFieldValue(badPeer, "advService", mock(AdvService.class)); | ||
| Mockito.when(delegate.getActivePeer()).thenReturn(Arrays.asList(badPeer, goodPeer)); | ||
|
|
||
| BlockCapsule block = new BlockCapsule(1, Sha256Hash.ZERO_HASH, | ||
| System.currentTimeMillis(), | ||
| ByteString.copyFrom(ECKey.fromPrivate(BigInteger.ONE).getAddress())); | ||
| BlockCapsule.BlockId blockId = block.getBlockId(); | ||
| badPeer.getSyncBlockToFetch().add(blockId); | ||
| goodPeer.getSyncBlockToFetch().add(blockId); | ||
| Method fetch = SyncService.class.getDeclaredMethod("startFetchSyncBlock"); | ||
| fetch.setAccessible(true); | ||
| fetch.invoke(sync); | ||
| Cache<BlockCapsule.BlockId, PeerConnection> requests = | ||
| (Cache<BlockCapsule.BlockId, PeerConnection>) | ||
| ReflectUtils.getFieldObject(sync, "requestBlockIds"); | ||
| Assert.assertSame(badPeer, requests.getIfPresent(blockId)); | ||
|
|
||
| byte[] data = block.getData(); | ||
| byte[] wire = new byte[data.length + 2]; | ||
| wire[0] = MessageTypes.BLOCK.asByte(); | ||
| System.arraycopy(data, 0, wire, 1, data.length); | ||
| wire[wire.length - 1] = 0x0c; | ||
| P2pEventHandlerImpl handler = new P2pEventHandlerImpl(); | ||
| BlockMsgHandler blockHandler = mock(BlockMsgHandler.class); | ||
| ReflectUtils.setFieldValue(handler, "blockMsgHandler", blockHandler); | ||
| Method process = P2pEventHandlerImpl.class | ||
| .getDeclaredMethod("processMessage", PeerConnection.class, byte[].class); | ||
| process.setAccessible(true); | ||
| process.invoke(handler, badPeer, wire); | ||
|
|
||
| Mockito.verifyNoInteractions(blockHandler); | ||
| Mockito.verify(badPeer).disconnect(Mockito.any(Protocol.ReasonCode.class)); | ||
| Assert.assertTrue(badPeer.getSyncBlockRequested().containsKey(blockId)); | ||
| Assert.assertTrue(badPeer.getSyncBlockInProcess().isEmpty()); | ||
| badPeer.onDisconnect(); | ||
| Assert.assertNull(requests.getIfPresent(blockId)); | ||
| Assert.assertTrue(badPeer.getSyncBlockRequested().isEmpty()); | ||
| Mockito.when(delegate.getActivePeer()).thenReturn(Collections.singletonList(goodPeer)); | ||
| fetch.invoke(sync); | ||
| Assert.assertSame(goodPeer, requests.getIfPresent(blockId)); | ||
| Assert.assertTrue(goodPeer.getSyncBlockRequested().containsKey(blockId)); | ||
| Mockito.verify(goodPeer).sendMessage(Mockito.any(FetchInvDataMessage.class)); | ||
| } finally { | ||
| sync.close(); | ||
| } | ||
| } | ||
|
|
||
| @Test | ||
| public void testDeserializeFailureAllowsRetryFromAnotherPeer() throws Exception { | ||
|
Collaborator
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [NIT] Declare the non-singleton isolation premise for these tests (class-level javadoc) This test (and the other Suggestion: add one line to the class-level javadoc of
Collaborator
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
That lifecycle is already documented in |
||
| chainBaseManager.getDynamicPropertiesStore().saveAllowProtoFilterNum(0); | ||
| SyncService sync = new SyncService(); | ||
| try { | ||
| TronNetDelegate delegate = mock(TronNetDelegate.class); | ||
| ReflectUtils.setFieldValue(sync, "tronNetDelegate", delegate); | ||
| ReflectUtils.setFieldValue(sync, "pbftDataSyncHandler", mock(PbftDataSyncHandler.class)); | ||
| Mockito.when(delegate.getSolidBlockId()) | ||
| .thenReturn(new BlockCapsule.BlockId(Sha256Hash.ZERO_HASH, 0)); | ||
| Mockito.when(delegate.getBlockLock()).thenReturn(new Object()); | ||
|
|
||
| PeerConnection badPeer = Mockito.spy(context.getBean(PeerConnection.class)); | ||
| PeerConnection goodPeer = Mockito.spy(context.getBean(PeerConnection.class)); | ||
| Channel badChannel = mock(Channel.class); | ||
| Channel goodChannel = mock(Channel.class); | ||
| Mockito.when(badChannel.getInetSocketAddress()).thenReturn(inetSocketAddress); | ||
| Mockito.when(badChannel.getInetAddress()).thenReturn(inetSocketAddress.getAddress()); | ||
| Mockito.when(goodChannel.getInetSocketAddress()).thenReturn(inetSocketAddress); | ||
| Mockito.when(goodChannel.getInetAddress()).thenReturn(inetSocketAddress.getAddress()); | ||
| badPeer.setChannel(badChannel); | ||
| goodPeer.setChannel(goodChannel); | ||
| badPeer.setFetchAble(true); | ||
| goodPeer.setFetchAble(true); | ||
| Mockito.when(delegate.getActivePeer()).thenReturn(Arrays.asList(badPeer, goodPeer)); | ||
|
|
||
| BlockCapsule block = new BlockCapsule(1, Sha256Hash.ZERO_HASH, | ||
| System.currentTimeMillis(), | ||
| ByteString.copyFrom(ECKey.fromPrivate(BigInteger.ONE).getAddress())); | ||
| BlockCapsule.BlockId blockId = block.getBlockId(); | ||
| byte[] data = block.getData(); | ||
| // Repeating the singular header field preserves the block but changes its wire length. | ||
| byte[] nonCanonical = Arrays.copyOf(data, data.length * 2); | ||
| System.arraycopy(data, 0, nonCanonical, data.length, data.length); | ||
| BlockMessage message = new BlockMessage(nonCanonical); | ||
| Assert.assertEquals(block.getInstance(), message.getBlockCapsule().getInstance()); | ||
| badPeer.getSyncBlockToFetch().add(blockId); | ||
| goodPeer.getSyncBlockToFetch().add(blockId); | ||
|
|
||
| Method fetch = SyncService.class.getDeclaredMethod("startFetchSyncBlock"); | ||
| fetch.setAccessible(true); | ||
| fetch.invoke(sync); | ||
| Assert.assertTrue(badPeer.getSyncBlockRequested().containsKey(blockId)); | ||
| Assert.assertTrue(goodPeer.getSyncBlockRequested().isEmpty()); | ||
| Cache<BlockCapsule.BlockId, PeerConnection> requests = | ||
| (Cache<BlockCapsule.BlockId, PeerConnection>) | ||
| ReflectUtils.getFieldObject(sync, "requestBlockIds"); | ||
| Assert.assertSame(badPeer, requests.getIfPresent(blockId)); | ||
|
|
||
| BlockMsgHandler handler = new BlockMsgHandler(); | ||
| ReflectUtils.setFieldValue(handler, "syncService", sync); | ||
| handler.processMessage(badPeer, message); | ||
| Assert.assertTrue(badPeer.getSyncBlockRequested().isEmpty()); | ||
| Assert.assertTrue(badPeer.getSyncBlockInProcess().contains(blockId)); | ||
| // A filter change while the block is queued can make deferred validation fail. | ||
| chainBaseManager.getDynamicPropertiesStore().saveAllowProtoFilterNum(1); | ||
| Method handle = SyncService.class.getDeclaredMethod("handleSyncBlock"); | ||
| handle.setAccessible(true); | ||
| ReflectUtils.setFieldValue(sync, "fetchFlag", false); | ||
| handle.invoke(sync); | ||
|
|
||
| Assert.assertNull("A failed parse must release the global request", | ||
| requests.getIfPresent(blockId)); | ||
| Assert.assertFalse(badPeer.getSyncBlockInProcess().contains(blockId)); | ||
| Mockito.verify(badPeer, Mockito.never()).disconnect(Mockito.any(Protocol.ReasonCode.class)); | ||
| Mockito.verify(delegate, Mockito.never()) | ||
| .processBlock(Mockito.any(BlockCapsule.class), Mockito.anyBoolean()); | ||
| Assert.assertTrue((boolean) ReflectUtils.getFieldObject(sync, "fetchFlag")); | ||
|
|
||
| Mockito.when(delegate.getActivePeer()).thenReturn(Collections.singletonList(goodPeer)); | ||
| fetch.invoke(sync); | ||
| ArgumentCaptor<FetchInvDataMessage> request = | ||
| ArgumentCaptor.forClass(FetchInvDataMessage.class); | ||
| Mockito.verify(goodPeer).sendMessage(request.capture()); | ||
| Assert.assertEquals(Collections.singletonList(blockId), request.getValue().getHashList()); | ||
| Assert.assertSame(goodPeer, requests.getIfPresent(blockId)); | ||
|
|
||
| // Leave a next block queued so successful processing does not request a chain summary. | ||
| BlockCapsule.BlockId next = new BlockCapsule.BlockId(Sha256Hash.ZERO_HASH, 2); | ||
| goodPeer.getSyncBlockToFetch().add(next); | ||
| handler.processMessage(goodPeer, new BlockMessage(block)); | ||
| handle.invoke(sync); | ||
| Mockito.verify(delegate).processBlock(Mockito.any(BlockCapsule.class), Mockito.eq(true)); | ||
| Assert.assertEquals(blockId, goodPeer.getBlockBothHave()); | ||
| Assert.assertEquals(next, goodPeer.getSyncBlockToFetch().peek()); | ||
| Assert.assertFalse(goodPeer.getSyncBlockInProcess().contains(blockId)); | ||
| Mockito.verify(goodPeer, Mockito.never()).disconnect(Mockito.any(Protocol.ReasonCode.class)); | ||
| } finally { | ||
| sync.close(); | ||
| } | ||
| } | ||
|
|
||
| @Test | ||
| public void sameAddressPeersKeepSyncRequestsAndResponsesIndependent() throws Exception { | ||
| InetSocketAddress address = new InetSocketAddress("127.0.0.2", 10001); | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[NIT] Document the two-parse-entry asymmetry and proto-filter precondition at this catch site
This failure path is only reachable because the network-side parse (
BlockMessage(byte[])->Block.parseFrom(CodedInputStream)) accepts a trailing0x0c(END_GROUP) byte while the deferredBlockCapsule(byte[])parse (Block.parseFrom(byte[])) rejects it — verified against protobuf-java 3.25.8. With proto filtering enabled (allowProtoFilterNum == 1), the entry path'sMessage.compareByteslength check rejects the payload first, so this catch is nearly unreachable there; on filter-disabled networks it is reachable. If a future protobuf upgrade unifies the two parse entries, or a maintainer 'unifies' them by hand, the reachability of this path changes silently and the regression test degrades into testing dead code.Suggestion: add a short comment above this catch block noting that the second parse is stricter than the entry parse and that a failure here means the supplying peer is not trustworthy; re-verify the regression test semantics whenever protobuf-java is upgraded.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated in f12c112.
BlockMessagenow callscheckLastTagWas(0)after the initial parse, rejecting unmatched top-levelEND_GROUPtags regardless of the proto-filter setting. A comment at that check explains the EOF requirement.The deferred-failure regression now explicitly injects damaged queued data to verify defensive cleanup independently of ingress validation. Separate tests cover ingress rejection and legitimate 0x0c bytes inside fields.