27 KiB
P2P Network
{% autocrossref %}
This section describes the Bitcoin P2P network protocol (but it is not a specification). It does not describe the discontinued direct [IP-to-IP payment protocol][], the [BIP70 payment protocol][payment protocol], the [GetBlockTemplate mining protocol][section getblocktemplate], or any network protocol never implemented in an official version of Bitcoin Core.
All peer-to-peer communication occurs entirely over TCP.
Note: unless their description says otherwise, all multi-byte integers mentioned in this section are transmitted in little-endian order.
{% endautocrossref %}
Constants And Defaults
{% autocrossref %}
The following constants and defaults are taken from Bitcoin Core's [chainparams.cpp][core chainparams.cpp] source code file.
Network | Default Port | [Start String][]{:#term-start-string}{:.term} | Max nBits |
---|---|---|---|
Mainnet | 8333 | 0xf9beb4d9 | 0x1d00ffff |
Testnet | 18333 | 0x0b110907 | 0x1d00ffff |
Regtest | 18444 | 0xfabfb5da | 0x207fffff |
Note: the testnet start string and nBits above are for testnet3; the original testnet used a different string and higher (less difficult) nBits.
Command line parameters can change what port a node listens on (see
-help
). Start strings are hardcoded constants that appear at the start
of all messages sent on the Bitcoin network; they may also appear in
data files such as Bitcoin Core's block database. The nBits displayed
above are in big-endian order; they're sent over the network in
little-endian order.
Bitcoin Core's [chainparams.cpp][core chainparams.cpp] also includes other constants useful to programs, such as the hash of the genesis blocks for the different networks as well as the alert keys for mainnet and testnet.
{% endautocrossref %}
Protocol Versions
{% autocrossref %}
The table below lists some notable versions of the P2P network protocol, with the most recent versions listed first. (If you would like to help document older protocol versions, please [open an issue][docs issue].)
Version | Implementation | Major Changes |
---|---|---|
70002 | Bitcoin Core 0.9.0 (Mar. 2014) |
- Added reject message (see [BIP61][]). - Send multiple inv messages in response to mempool message if necessary. |
70001 | Bitcoin Core 0.8.0 (Feb. 2013) |
- Added filterload message. - Added filteradd message. - Added filterclear message. - Added merkleblock message; also added relay field to version message and MSG_FILTERED_BLOCK inventory type to getdata message (see [BIP37][]). |
60002 | Bitcoin Core 0.7.0 (Sep. 2012) |
- Added mempool message. - Extended getdata message to allow download of memory pool transactions (see [BIP35][]). |
60001 | Bitcoin Core 0.6.1 (May. 2012) |
- Added pong message (see [BIP31][]). |
60000 | Bitcoin Core 0.6.0 (Mar. 2012) |
- Separated protocol version from Bitcoin Core version (see [BIP14][]). |
{% endautocrossref %}
Message Headers
{% autocrossref %}
All messages in the network protocol use the same container format, which provides a required multi-field header and an optional payload. The header format is:
Bytes | Name | Data Type | Description |
---|---|---|---|
4 | start string | char[4] | Magic bytes indicating the originating network; used to seek to next message when stream state is unknown. |
12 | command name | char[12] | ASCII string which identifies what message type is contained in the payload. Followed by nulls (0x00) to pad out byte count; for example: version\0\0\0\0\0 . |
4 | payload size | uint32_t | Number of bytes in payload. The current maximum number of bytes ([MAX_SIZE ][max_size]) allowed in the payload by Bitcoin Core is 32 MiB---messages with a payload size larger than this will be dropped or rejected. |
4 | checksum | char[4] | First 4 bytes of SHA256(SHA256(payload)) in internal byte order. If payload is empty, as in verack and getaddr messages, the checksum is always 0x5df6e0e2 (SHA256(SHA256(<empty string>))).The checksum field was introduced in protocol version 209; Bitcoin Core 0.2.9, May 2010. |
The following example is an annotated hex dump of a mainnet message
header from a verack
message which has no payload.
{% highlight text %} f9beb4d9 ................... Start string: Mainnet 76657261636b000000000000 ... Command name: verack + null padding 00000000 ................... Byte count: 0 5df6e0e2 ................... Checksum: SHA256(SHA256()) {% endhighlight %}
{% endautocrossref %}
Data Messages
{% autocrossref %}
The following network messages all request or provide data related to transactions and blocks.
Many of the data messages use [inventories][inventory]{:#term-inventory}{:.term} as unique identifiers for for transactions and blocks. Inventories have a simple 36-byte structure:
Bytes | Name | Data Type | Description |
---|---|---|---|
4 | type identifier | uint32_t | The type of object which was hashed. See list of type identifiers below. |
32 | hash | char[32] | SHA256(SHA256()) hash of the object in internal byte order. |
The currently-available type identifiers are:
Type Identifier | Name | Description |
---|---|---|
1 | [MSG_TX ][msg_tx]{:#term-msg_tx}{:.term} |
The hash is a TXID. |
2 | [MSG_BLOCK ][msg_block]{:#term-msg_block}{:.term} |
The hash is of a block header. |
3 | [MSG_FILTERED_BLOCK ][msg_filtered_block]{:#term-msg_filtered_block}{:.term} |
The hash is of a block header; identical to MSG_BLOCK . When used in a getdata message, this indicates the response should be a merkleblock message rather than a block message (but this only works if a bloom filter was previously configured). Only for use in getdata messages. |
Type identifier zero and type identifiers greater than three are reserved for future implementations. Bitcoin Core ignores all inventories with one of these unknown types.
{% endautocrossref %}
Block
{% autocrossref %}
The block
message transmits a single serialized block in the format
described in the [serialized blocks section][section serialized blocks].
See that section for an example hexdump.
It is sent in reply to a getdata
message which had an inventory type of
MSG_BLOCK
and the header hash of the particular block being requested.
{% endautocrossref %}
GetBlocks
{% autocrossref %}
The getblocks
message requests an inv
message that provides block
header hashes starting from a particular point in the block chain. It
allows a peer which has been disconnected or started for the first time
to get the data it needs to request the blocks it hasn't seen.
Peers which have been disconnected may have stale blocks in their
locally-stored block chain, so the getblocks
message allows the
requesting peer to provide the receiving peer with multiple header
hashes at various heights on their local chain. This allows the
receiving peer to find, within that list, the last header hash they had
in common and reply with all subsequent header hashes.
Note: the receiving peer itself may respond with an inv
message
containing header hashes of stale blocks. It is up to the requesting
peer to poll all of its peers to find the best block chain.
If the receiving peer does not find a common header hash within the
list, it will assume the last common block was the genesis block (block
zero), so it will reply with in inv
message containing header hashes
starting with block one (the first block after the genesis block).
Bytes | Name | Data Type | Description |
---|---|---|---|
4 | version | uint32_t | The protocol version number; the same as sent in the version message. |
Varies | hash count | compactSize uint | The number of header hashes provided not including the stop hash. There is no limit except that the byte size of the entire message must be below the [MAX_SIZE ][max_size] limit; typically from 1 to 200 hashes are sent. |
Varies | block header hashes | char[32] | One or more block header hashes (32 bytes each) in internal byte order. Hashes should be provided in reverse order of block height, so highest-height hashes are listed first and lowest-height hashes are listed last. |
32 | stop hash | char[32] | The header hash of the last header hash being requested; set to all zeroes to request an inv message with 500 header hashes (the maximum which will ever be sent as a reply to this message). |
The following annotated hexdump shows a getblocks
message. (The
message header has been omitted.)
{% highlight text %} 71110100 ........................... Protocol version: 70001 02 ................................. Hash count: 2
d39f608a7775b537729884d4e6633bb2 105e55a16a14d31b0000000000000000 ... Hash #1
5c3e6403d40837110a2e8afb602b1c01 714bda7ce23bea0a0000000000000000 ... Hash #2
00000000000000000000000000000000 00000000000000000000000000000000 ... Stop hash {% endhighlight %}
{% endautocrossref %}
GetData
{% autocrossref %}
The getdata
message requests one or more data objects from another
node. The objects are requested by an inventory, which the requesting
node typically previously received by way of an inv
message.
The response to a getdata
message can be a tx
message, block
message, merkleblock
message, or notfound
message.
This message cannot be used to request arbitrary data, such as historic
transactions no longer in the memory pool or relay set. Full nodes may
not even be able to provide older blocks if they've pruned old
transactions from their block database. For this reason, the getdata
message should usually only be used to request data from a node which
previously advertised it had that data by sending an inv
message.
The format and maximum size limitations of the getdata
message are
identical to the inv
message; only the message header differs.
{% endautocrossref %}
GetHeaders
{% autocrossref %}
The getheaders
message requests a headers
message that provides block headers
starting from a particular point in the block chain. It allows a
peer which has been disconnected or started for the first time to get
the headers it hasn’t seen yet.
The getheaders
message is nearly identical to the getblocks
message,
with one minor difference: the inv
reply to the getblocks
message
will include no more than 500 block header hashes; the headers
reply
to the getheaders
message will include as many as 2,000 block headers.
Headers
The headers
message sends one or more block headers to a node which
previously requested certain headers with a getheaders
message.
Bytes | Name | Data Type | Description |
---|---|---|---|
Varies | count | compactSize uint | Number of block headers up to a maximum of 2,000. |
Varies | headers | block_header | Block headers: each 80-byte block header is in the format described in the [block headers section][block header] with an additional 0x00 suffixed. This 0x00 is called the transaction count, but because the headers message doesn't include any transactions, the transaction count is always zero. |
The following annotated hexdump shows a headers
message. (The message
header has been omitted.)
{% highlight text %} 01 ................................. Header count: 1
02000000 ........................... Block version: 2 b6ff0b1b1680a2862a30ca44d346d9e8 910d334beb48ca0c0000000000000000 ... Hash of previous block's header 9d10aa52ee949386ca9385695f04ede2 70dda20810decd12bc9b048aaab31471 ... Merkle root 24d95a54 ........................... Unix time: 1415239972 30c31b18 ........................... Target (bits) fe9f0864 ........................... Nonce
00 ......... Transaction Count (0x00) {% endhighlight %}
{% endautocrossref %}
Inv
{% autocrossref %}
The inv
message (inventory message) transmits one or more inventories of
objects known to the transmitting peer. It can be sent unsolicited to
announce new transactions or blocks, or it can be sent in reply to a
getblocks
message or mempool
message.
The receiving peer can compare the inventories from an inv
message
against the inventories it has already seen, and then use a follow-up
message to request unseen objects.
Bytes | Name | Data Type | Description |
---|---|---|---|
Varies | count | compactSize uint | The number of inventory entries. |
Varies | inventory | inventory | One or more inventory entries up to a maximum of 50,000 entries. |
The following annotated hexdump shows an inv
message with two
inventory entries. (The message header has been omitted.)
{% highlight text %} 02 ................................. Count: 2
01000000 ........................... Type: MSG_TX de55ffd709ac1f5dc509a0925d0b1fc4 42ca034f224732e429081da1b621f55a ... Hash (TXID)
01000000 ........................... Type: MSG_TX 91d36d997037e08018262978766f24b8 a055aaf1d872e94ae85e9817b2c68dc7 ... Hash (TXID) {% endhighlight %}
{% endautocrossref %}
MemPool
{% autocrossref %}
The mempool
message requests the TXIDs of transactions that the
receiving node has verified are valid but which have not yet appeared in
a block. That is, transactions which are in the receiving node's memory
pool. The response to the mempool
message is one or more inv
messages containing the TXIDs in the usual inventory format. The
mempool
message was introduced in protocol version 60002 as
implemented in Bitcoin Core 0.7.0 (September 2012).
Sending the mempool
message is mostly useful when a program first
connects to the network. Full nodes can use it to quickly gather most or
all of the unconfirmed transactions available on the network; this is
especially useful for miners trying to gather transactions for their
transaction fees. SPV clients can set a filter before sending a
mempool
to only receive transactions that match that filter; this
allows a recently-started client to get most or all unconfirmed
transactions related to its wallet.
The inv
response to the mempool
message is, at best, one node's
view of the network---not a complete list of unconfirmed transactions
on the network. Here are some additional reasons the list might not
be complete:
-
Before Bitcoin Core 0.9.0, the response to the
mempool
message was only oneinv
message. Aninv
message is limited to 50,000 inventories, so a node with a memory pool larger than 50,000 entries would not send everything. Later versions of Bitcoin Core send as manyinv
messages as needed to reference its complete memory pool. -
The
mempool
message is not currently fully compatible with thefilterload
message'sBLOOM_UPDATE_ALL
andBLOOM_UPDATE_P2PUBKEY_ONLY
flags. Mempool transactions are not sorted like in-block transactions, so a transaction (tx2) spending an output can appear before the transaction (tx1) containing that output, which means the automatic filter update mechanism won't operate until the second-appearing transaction (tx1) is seen---missing the first-appearing transaction (tx2). It has been proposed in [Bitcoin Core issue #2381][] that the transactions should be sorted before being processed by the filter.
There is no payload in a mempool
message. See the [message header
section][message header] for an example of a message without a payload.
{% endautocrossref %}
MerkleBlock
{% autocrossref %}
The merkleblock
message is a reply to a getdata
message which
requested a block using the inventory type MSG_MERKLEBLOCK
. It is
only part of the reply: if any matching transactions are found, they will
be sent separately as tx
messages.
This message is part of the bloom filters described in BIP37, added in protocol version 70001 and implemented in Bitcoin Core 0.8.0 (February 2013).
If a filter has been previous set with the filterload
message, the
merkleblock
message will contain the TXIDs of any transactions in the
requested block that matched the filter, as well as any parts of the
block's merkle tree necessary to connect those transactions to the
block header's merkle root. The message also contains a complete copy
of the block header to allow the client to hash it and confirm its
proof of work.
Bytes | Name | Data Type | Description |
---|---|---|---|
80 | block header | block_header | The block header in the format described in the [block header section][block header]. |
4 | transaction count | uint32_t | The number of transactions in the block (including ones that don't match the filter). |
Varies | hash count | compactSize uint | The number of hashes in the following field. |
Varies | hashes | char[32] | One or more hashes of both transactions and merkle nodes in internal byte order. Each hash is 32 bits. |
Varies | flag byte count | compactSize uint | The number of flag bytes in the following field. |
Varies | flags | byte[] | A sequence of bits packed eight in a byte with the least significant bit first. May be padded to the nearest byte boundary but must not contain any more bits than that. Used to assign the hashes to particular nodes in the merkle tree as described below. |
The annotated hexdump below shows a merkleblock
message which
corresponds to the examples below. (The message header has been
omitted.)
{% highlight text %} 01000000 ........................... Block version: 1 82bb869cf3a793432a66e826e05a6fc3 7469f8efb7421dc88067010000000000 ... Hash of previous block's header 7f16c5962e8bd963659c793ce370d95f 093bc7e367117b3c30c1f8fdd0d97287 ... Merkle root 76381b4d ........................... Time: 1293629558 4c86041b ........................... nBits: 0x04864c * 256**(0x1b-3) 554b8529 ........................... Nonce
07000000 ........................... Transaction count: 7 04 ................................. Hash count: 4
3612262624047ee87660be1a707519a4 43b1c1ce3d248cbfc6c15870f6c5daa2 ... Hash #1 019f5b01d4195ecbc9398fbf3c3b1fa9 bb3183301d7a1fb3bd174fcfa40a2b65 ... Hash #2 41ed70551dd7e841883ab8f0b16bf041 76b7d1480e4f0af9f3d4c3595768d068 ... Hash #3 20d2a7bc994987302e5b1ac80fc425fe 25f8b63169ea78e68fbaaefa59379bbf ... Hash #4
01 ................................. Flag bytes: 1 1d ................................. Flags: 1 0 1 1 1 0 0 0 {% endhighlight %}
Note: when fully decoded, the above merkleblock
message provided the
TXID for a single transaction that matched the filter. In the network
traffic dump this output was taken from, the full transaction belonging
to that TXID was sent immediately after the the merkleblock
message as
a tx
message.
Parsing A MerkleBlock
As seen in the annotated hexdump above, the merkleblock
message
provides three special data types: a transaction count, a list of
hashes, and a list of one-bit flags.
You can use the transaction count to construct an empty merkle tree. We'll call each entry in the tree a node; on the bottom are TXID nodes---the hashes for these nodes are TXIDs; the remaining nodes (including the merkle root) are non-TXID nodes---they may actually have the same hash as a TXID, but we treat them differently.
Keep the hashes and flags in the order they appear in the merkleblock
message. When we say "next flag" or "next hash", we mean the next flag
or hash on the list, even if it's the first one we've used so far.
Start with the merkle root node and the first flag. The table below describes how to evaluate a flag based on whether the node being processed is a TXID node or a non-TXID node. Once you apply a flag to a node, never apply another flag to that same node or reuse that same flag again.
Flag | TXID Node | Non-TXID Node |
---|---|---|
0 | Use the next hash as this node's TXID, but this transaction didn't match the filter. | Use the next hash as this node's hash. Don't process any descendant nodes. |
1 | Use the next hash as this node's TXID, and mark this transaction as matching the filter. | The hash needs to be computed. Process the left child node to get its hash; process the right child node to get its hash; then concatenate the two hashes as 64 raw bytes and hash them to get this node's hash. |
Any time you descend into a node for the first time, evaluate the next flag. Never use a flag at any other time.
When processing a child node, you may need to process its children (the grandchildren of the original node) or further-descended nodes before returning to the parent node. This is expected---keep processing depth first until you reach a TXID node or a non-TXID node with a flag of 0.
After you process a TXID node or a non-TXID node with a flag of 0, stop processing flags and begin to ascend the tree. As you ascend, compute the hash of any nodes for which you now have both child hashes or for which you now have the sole child hash. See the [merkle tree section][section merkle trees] for hashing instructions. If you reach a node where only the left hash is known, descend into its right child (if present) and further descendants as necessary.
However, if you find a node whose left and right children both have the same hash, fail. This is related to CVE-2012-2459.
Continue descending and ascending until you have enough information to obtain the hash of the merkle root node. If you run out of flags or hashes before that condition is reached, fail. Then perform the following checks (order doesn't matter):
-
Fail if there are unused hashes in the hashes list.
-
Fail if there are unused flag bits---except for the minimum number of bits necessary to pad up to the next full byte.
-
Fail unless the hash of the merkle root node is identical to the merkle root in the block header.
-
Fail if the block header is invalid. Remember to ensure that the hash of the header is less than or equal to the target threshold encoded by the nBits header field. Your program should also, of course, attempt to ensure the header belongs to the best block chain and that the user knows how many confirmations this block has.
For a detailed example of parsing a merkleblock
message, please see
the corresponding [merkleblock examples section][section merkleblock
example].
Creating A MerkleBlock
It's easier to understand how to create a merkleblock
message after
you understand how to parse an already-created message, so we recommend
you read the parsing section above first.
Create a complete merkle tree with TXIDs on the bottom row and all the other hashes calculated up to the merkle root on the top row. For each transaction that matches the filter, track its TXID node and all of its ancestor nodes.
Start processing the tree with the merkle root node. The table below describes how to process both TXID nodes and non-TXID nodes based on whether the node is a match, a match ancestor, or neither a match nor a match ancestor.
TXID Node | Non-TXID Node | |
---|---|---|
Neither Match Nor Match Ancestor | Append a 0 to the flag list; append this node's TXID to the hash list. | Append a 0 to the flag list; append this node's hash to the hash list. Do not descend into its child nodes. |
Match Or Match Ancestor | Append a 1 to the flag list; append this node's TXID to the hash list. | Append a 1 to the flag list; process the left child node. Then, if the node has a right child, process the right child. Do not append a hash to the hash list for this node. |
Any time you descend into a node for the first time, a flag should be appended to the flag list. Never put a flag on the list at any other time, except when processing is complete to pad out the flag list to a byte boundary.
When processing a child node, you may need to process its children (the grandchildren of the original node) or further-descended nodes before returning to the parent node. This is expected---keep processing depth first until you reach a TXID node or a node which is neither a TXID nor a match ancestor.
After you process a TXID node or a node which is neither a TXID nor a match ancestor, stop processing and begin to ascend the tree until you find a node with a right child you haven't processed yet. Descend into that right child and begin processing again.
After you fully process the merkle root node according to the
instructions in the table above, processing is complete. Pad your flag
list to a byte boundary and construct the merkleblock
message using the
template near the beginning of this subsection.
{% endautocrossref %}
NotFound
{% autocrossref %}
The notfound
message is a reply to a getdata
message which
requested an object the receiving node does not have available for
relay. (Nodes are not expected to relay historic transactions which
are no longer in the memory pool or relay set. Nodes may also have
pruned spent transactions from older blocks, making them unable to
send those blocks.)
The format and maximum size limitations of the notfound
message are
identical to the inv
message; only the message header differs.
{% endautocrossref %}
Tx
{% autocrossref %}
The tx
message transmits a single transaction in the raw transaction
format. It can be sent in a variety of situations;
-
Transaction Response: Bitcoin Core and BitcoinJ will send it in response to a
getdata
message that requests the transaction with an inventory type ofMSG_TX
. -
MerkleBlock Response: Bitcoin Core will send it in response to a
getdata
message that requests a merkleblock with an inventory type ofMSG_MERKLEBLOCK
. (This is in addition to sending amerkleblock
message.) Eachtx
message in this case provides a matched transaction from that block. -
Unsolicited: BitcoinJ will send a
tx
message unsolicited for transactions it originates.
For an example hexdump of the raw transaction format, see the [raw transaction section][raw format].
{% endautocrossref %}