From edfa3226a0cc78d3cb1be2c8e6237874fecb9ce2 Mon Sep 17 00:00:00 2001 From: thephez Date: Mon, 23 Apr 2018 16:48:36 -0400 Subject: [PATCH] content - p2p - Update compact block messages - Add some info from bitcoin-dot-org/bitcoin.org#2092 - Add detail to cmpctblock, sendcmpct, getblocktxn, blocktxn messages - Add reference terms --- _includes/devdoc/ref_p2p_networking.md | 125 ++++++++++++++++++++----- _includes/references.md | 6 +- 2 files changed, 104 insertions(+), 27 deletions(-) diff --git a/_includes/devdoc/ref_p2p_networking.md b/_includes/devdoc/ref_p2p_networking.md index cc3c4267..7113285e 100644 --- a/_includes/devdoc/ref_p2p_networking.md +++ b/_includes/devdoc/ref_p2p_networking.md @@ -172,7 +172,7 @@ The currently-available type identifiers are: | 17 | [`MSG_GOVERNANCE_OBJECT`][msg_governance_object]{:#term-msg_governance_object}{:.term} | The hash is a Governance Object. | 18 | [`MSG_GOVERNANCE_OBJECT_VOTE`][msg_governance_object_vote]{:#term-msg_governance_object_vote}{:.term} | The hash is a Governance Object Vote. | 19 | [`MSG_MASTERNODE_VERIFY`][msg_masternode_verify]{:#term-msg_masternode_verify}{:.term} | The hash is a Masternode Verify. -| 20 | [`MSG_CMPCT_BLOCK`][msg_cmpct_block]{:#term-msg_cmpct_block}{:.term} | The hash is of a compact block. +| 20 | [`MSG_CMPCT_BLOCK`][msg_cmpct_block]{:#term-msg_cmpct_block}{:.term} | The hash is of a block header; identical to `MSG_BLOCK`. When used in a `getdata` message, this indicates the response should be a `cmpctblock` message. **Only for use in `getdata` messages.** Type identifier zero and type identifiers greater than twenty are reserved for future implementations. Dash Core ignores all inventories with @@ -211,7 +211,20 @@ different reasons: *Added in protocol version 70209 of Dash Core as described by BIP152* The `blocktxn` message sends requested block transactions to a node which -previously requested them with a `getblocktxn` message. +previously requested them with a `getblocktxn` message. It is defined as a message +containing a serialized `BlockTransactions` message. + +Upon receipt of a properly-formatted requested `blocktxn` message, nodes should: + +1. Attempt to reconstruct the full block by taking the prefilledtxn transactions from the original `cmpctblock` message and placing them in the marked positions +2. For each short transaction ID from the original `cmpctblock` message, in order, find the corresponding transaction (from either the `blocktxn` message or from other sources) +3. Place each short transaction ID in the first available position in the block +4. Once the block has been reconstructed, it shall be processed as normal. + +**Short transaction IDs are expected to occasionally collide. Nodes must +not be penalized for such collisions.** + +The structure of `BlockTransactions` is defined below. | Bytes | Name | Data Type | Encoding | Description| |----------|----------------------|----------------------|----------|------------| @@ -235,39 +248,63 @@ message header has been omitted.) *Added in protocol version 70209 of Dash Core as described by BIP152* -The `cmpctblock` message transmits a single serialized compact block. A -`HeaderAndShortIDs` structure is used to relay a block header, the short -transactions IDs used for matching already-available transactions, and a select -few transactions which we expect a peer may be missing (PrefilledTransactions). -It can be sent for two different reasons: +The `cmpctblock` message is a reply to a `getdata` message which +requested a block using the inventory type `MSG_CMPCT_BLOCK`. If the +requested block was recently announced and is close to the tip of the +best chain of the receiver and after having sent the requesting peer +a `sendcmpct` message, nodes respond with a `cmpctblock` message containing +data for the block. -1. **GetData Response:** Nodes will always send it in response to a - `getdata` message that requests the block with an inventory - type of `MSG_CMPCT_BLOCK` (provided the node has that block available for - relay). +**If the requested block is too old, the node responds with a *full non-compact block*** -2. **Unsolicited:** If a node has previously sent a `sendcmpct` message - requesting blocks be announced via a `cmpctblock` message. +Upon receipt of a `cmpctblock` message, after sending a `sendcmpct` message, +nodes should calculate the short transaction ID for each unconfirmed +transaction they have available (i.e. in their mempool) and compare each +to each short transaction ID in the `cmpctblock` message. After finding +already-available transactions, nodes which do not have all transactions +available to reconstruct the full block should request the missing transactions +using a `getblocktxn` message. -**HeaderAndShortIDs** +A node must not send a `cmpctblock` message unless they are able to respond to +a `getblocktxn` message which requests every transaction in the block. A node +must not send a `cmpctblock` message without having validated that the header properly +commits to each transaction in the block, and properly builds on top of the existing, +fully-validated chain with a valid proof-of-work either as a part of the current most-work +valid chain, or building directly on top of it. A node may send a `cmpctblock` message before +validating that each transaction in the block validly spends existing UTXO set entries. + +The `cmpctblock` message contains a vector of `PrefilledTransaction` whose +structure is defined below. A `PrefilledTransaction` is used in `HeaderAndShortIDs` +to provide a list of a few transactions explicitly. + +| Bytes | Name | Data Type | Encoding | Description| +|----------|----------------------|----------------------|----------|------------| +| 1 or 3 | index | CompactSize | Compact Size, differentially encoded since the last PrefilledTransaction in a list | The index into the block at which this transaction is +| *Varies* | tx | Transaction | As encoded in `tx` messages sent in response to `getdata MSG_TX` | Transaction which is in the block at index `index` + +The `cmpctblock` message is compromised of a serialized `HeaderAndShortIDs` +structure which is defined below. A `HeaderAndShortIDs` structure is used to +relay a block header, the short transactions IDs used for matching +already-available transactions, and a select few transactions which +we expect a peer may be missing. | Bytes | Name | Data Type | Encoding | Description| |----------|----------------------|----------------------|----------|------------| | 80 | header | Block header | First 80 bytes of the block as defined by the encoding used by `block` messages | The header of the block being provided | 8 | nonce | uint64_t | Little Endian | A nonce for use in short transaction ID calculations | 1 or 3 | shortids_
length | CompactSize | As used to encode array lengths elsewhere | The number of short transaction IDs in `shortids` (i.e. block tx count - `prefilledtxn`
`_length`) -| 6 * `shortids`
`_length` | shortids | List of 6-byte integers | Little Endian | The short transaction IDs calculated from the transactions which were not provided explicitly in `prefilledtxn` +| *Varies* | shortids | List of 6-byte integers | Little Endian | The short transaction IDs calculated from the transactions which were not provided explicitly in `prefilledtxn` | 1 or 3 | prefilledtxn
_length | CompactSize | As used to encode array lengths elsewhere | The number of prefilled transactions in `prefilledtxn` (i.e. block tx count - `shortids`
`_length`) -| *Varies* * `prefilledtxn`
`_length` | prefilledtxn | List of Prefilled
Transactions | As defined by `Prefilled`
`Transaction` definition below | Used to provide the coinbase transaction and a select few which we expect a peer may be missing +| *Varies* | prefilledtxn | List of Prefilled
Transactions | As defined by `Prefilled`
`Transaction` definition below | Used to provide the coinbase transaction and a select few which we expect a peer may be missing -**PrefilledTransaction** +**Short Transaction ID calculation** -A `PrefilledTransaction` structure is used in `HeaderAndShortIDs` to provide a list of a few transactions explicitly. +Short transaction IDs are used to represent a transaction without sending a full 256-bit hash. They are calculated as follows, -| Bytes | Name | Data Type | Encoding | Description| -|----------|----------------------|----------------------|----------|------------| -| 1 or 3 | index | CompactSize | Compact Size, differentially encoded since the last PrefilledTransaction in a list | The index into the block at which this transaction is -| *Varies* | tx | Transaction | As encoded in `tx` messages sent in response to `getdata MSG_TX` | Transaction which is in the block at index `index` +* A single-SHA256 hashing the block header with the nonce appended (in little-endian) +* Running SipHash-2-4 with the input being the transaction ID and the keys (k0/k1) set to the first two little-endian 64-bit integers from the above hash, respectively. +* Dropping the 2 most significant bytes from the SipHash output to make it 6 bytes. + The following annotated hexdump shows a `cmpctblock` message. (The message header has been omitted.) @@ -384,13 +421,24 @@ d39f608a7775b537729884d4e6633bb2 *Added in protocol version 70209 of Dash Core as described by BIP152* The `getblocktxn` message requests a `blocktxn` message for any transactions -that it has not seen after a compact block is received. +that it has not seen after a compact block is received. It is defined as a +message containing a serialized `BlockTransactionsRequest` message. Upon receipt +of a properly-formatted `getblocktxn` message, nodes which recently provided the +sender of such a message with a `cmpctblock` message for the block hash +identified in this message must respond with either an appropriate +`blocktxn` message, or a full block message. + +A `blocktxn` message response must contain exactly and only each transaction +which is present in the appropriate block at the index specified in the +`getblocktxn` message indexes list, in the order requested. + +The structure of `BlockTransactionsRequest` is defined below. | Bytes | Name | Data Type | Encoding | Description| |----------|-----------------|----------------------|----------|------| | 32 | blockhash | Binary blob | The output from a double-SHA256 of the block header, as used elsewhere | The blockhash of the block which the transactions being requested are in -| 1 or 3 | indexes_length | CompactSize | As used to encode array lengths elsewhere | The number of transactions requested -| (1 or 3) * `indexes_length` | indexes | List of CompactSizes | Differentially encoded | The indexes of the transactions being requested in the block +| *Varies* | indexes_length | CompactSize uint | As used to encode array lengths elsewhere | The number of transactions requested +| *Varies* | indexes | CompactSize uint[] | Differentially encoded | Vector of compactSize containing the indexes of the transactions being requested in the block. The following annotated hexdump shows a `getblocktxn` message. (The message header has been omitted.) @@ -1445,7 +1493,32 @@ header has been omitted.) The `sendcmpct` message tells the receiving peer whether or not to announce new blocks using a `cmpctblock` message. It also sends the compact block protocol -version it supports. +version it supports. The `sendcmpct` message is defined as a message containing +a 1-byte integer followed by a 8-byte integer. The first integer is interpreted +as a boolean and should have a value of either 1 or 0. The second integer +is be interpreted as a little-endian version number. + +Upon receipt of a `sendcmpct` message with the first and second integers +set to 1, the node should announce new blocks by sending a `cmpctblock` message. + +Upon receipt of a `sendcmpct` message with the first integer set to 0, the node +shouldn't announce new blocks by sending a `cmpctblock` message, but instead announce +new blocks by sending invs or headers, as defined by [BIP130][]. + +Upon receipt of a `sendcmpct` message with the second integer set to something +other than 1, nodes should treat the peer as if they had not received the message +(as it indicates the peer will provide an unexpected encoding in `cmpctblock` messages, +and/or other, messages). This allows future versions to send duplicate +`sendcmpct` messages with different versions as a part of a version handshake. + +Nodes should check for a protocol version of >= 70209 before sending `sendcmpct` +messages. Nodes shouldn't send a request for a `MSG_CMPCT_BLOCK` object to a peer +before having received a `sendcmpct` message from that peer. Nodes shouldn't +request a `MSG_CMPCT_BLOCK` object before having sent all `sendcmpct` messages +to that peer which they intend to send, as the peer cannot know what protocol +version to use in the response. + +The structure of a `sendcmpct` message is defined below. | Bytes | Name | Data Type | Description |----------|---------------|------------------|-------------- diff --git a/_includes/references.md b/_includes/references.md index 8682718c..e602cc5f 100644 --- a/_includes/references.md +++ b/_includes/references.md @@ -4,8 +4,11 @@ http://opensource.org/licenses/MIT. {% endcomment %} {% comment %}{% endcomment %} +[blocktransactions]: /en/developer-reference#blocktxn "A P2P Networking data structure used to provide some of the transactions in a block as requested" +[blocktransactionsrequest]: /en/developer-reference#getblocktxn "A P2P Networking data structure used to list transaction indexes in a block being requested by a peer" [coinbase block height]: /en/developer-reference#term-coinbase-block-height "The current block's height encoded into the first bytes of the coinbase field" [data-pushing opcode]: https://en.bitcoin.it/wiki/Script#Constants "Any opcode from 0x01 to 0x4e which pushes data on to the script evaluation stack" +[headerandshortids]: /en/developer-reference#cmpctblock "A P2P Networking data structure used to relay a block header, the short transactions IDs used for matching already-available transactions, and a select few transactions which a peer may be missing" [key index]: /en/developer-guide#term-key-index "An index number used in the HD wallet formula to generate child keys from a parent key" [key pair]: /en/developer-guide#term-key-pair "A private key and its derived public key" @@ -29,7 +32,7 @@ http://opensource.org/licenses/MIT. [msg_governance_object]: /en/developer-reference#term-msg_governance_object "Governance object data type identifier of an inventory on the P2P network" [msg_governance_object_vote]: /en/developer-reference#term-msg_governance_object_vote "Governance object vote data type identifier of an inventory on the P2P network" [msg_masternode_verify]: /en/developer-reference#term-msg_masternode_verify "Masternode Verify data type identifier of an inventory on the P2P network" -[msg_cmpct_block]: /en/developer-reference#term-msg_cmpct_block "Compact Block data type identifier of an inventory on the P2P network" +[msg_cmpct_block]: /en/developer-reference#term-msg_cmpct_block "An alternative to the block header hash data type identifier of an inventory on the P2P network used to request a compact block" [network]: /en/developer-guide#term-network "The Dash P2P network which broadcasts transactions and blocks" [op_checkmultisig]: /en/developer-reference#term-op-checkmultisig "Opcode which returns true if one or more provided signatures (m) sign the correct parts of a transaction and match one or more provided public keys (n)" @@ -42,6 +45,7 @@ http://opensource.org/licenses/MIT. [op_verify]: /en/developer-reference#term-op-verify "Operation which terminates the script if the entry below it on the stack is non-true (zero)" [output index]: /en/developer-guide#term-output-index "The sequentially-numbered index of outputs in a single transaction starting from 0" [point function]: /en/developer-guide#term-point-function "The ECDSA function used to create a public key from a private key" +[prefilledtransaction]: /en/developer-reference#cmpctblock "A P2P Networking data structure used to represent a vector of a few transactions" [previous block header hash]: /en/developer-reference#term-previous-block-header-hash "A field in the block header which contains the SHA256(SHA256()) hash of the previous block's header" [proper money handling]: /en/developer-reference#term-proper-money-handling "Bitcoin amounts need to be correctly processed without introducing rounding errors that could cause monetary loss" [standard block relay]: /en/developer-guide#term-standard-block-relay "The regular block relay method: announcing a block with an inv message and waiting for a response"