diff --git a/_autocrossref.yaml b/_autocrossref.yaml index a1fe539c..d413b9cb 100644 --- a/_autocrossref.yaml +++ b/_autocrossref.yaml @@ -76,6 +76,7 @@ inputs: input input: intermediate certificate: intermediate certificates: intermediate certificate +internal byte order: key index: key pair: '`label`': label @@ -165,6 +166,7 @@ root certificate: root seed: RPCs: rpc RPC: +RPC byte order: satoshi: satoshis: satoshi '`script`': pp script diff --git a/_includes/ref_core_rpc_intro.md b/_includes/ref_core_rpc_intro.md new file mode 100644 index 00000000..e24c01ba --- /dev/null +++ b/_includes/ref_core_rpc_intro.md @@ -0,0 +1,92 @@ +### Hash Byte Order + +{% autocrossref %} + +Bitcoin Core RPCs accept and return hashes in the reverse of their +normal byte order. For example, the Unix `sha256sum` command would display the +SHA256(SHA256()) hash of Mainnet block 300,000's header as the +following string: + + 5472ac8b1187bfcf91d6d218bbda1eb2405d7c55f1f8cc820000000000000000 + +The string above is also how the hash appears in the +previous-header-hash part of block 300,001's header: + +
020000005472ac8b1187bfcf91d6d218bbda1eb2405d7c55f1f8cc82000\
+0000000000000ab0aaa377ca3f49b1545e2ae6b0667a08f42e72d8c24ae\
+237140e28f14f3bb7c6bcc6d536c890019edd83ccf
+ +However Bitcoin RPCs use the reverse byte order for hashes, so if you +want to get information about block 300,000 using the `getblock` RPC, +you need to reverse the byte order: + + > bitcoin-cli getblock \ + 000000000000000082ccf8f1557c5d40b21edabb18d2d691cfbf87118bac7254 + +(Note: hex representation uses two characters to display each byte of +data, which is why the reversed string looks somewhat mangled.) + +The rational for the reversal is unknown, but it likely stems from +Bitcoin's use of hash digests (which are byte arrays in C++) as integers +for the purpose of determining whether the hash is below the network +target. Whatever the reason for reversing header hashes, the reversal +also extends to other hashes used in RPCs, such as TXIDs and merkle +roots. + +Off-site documentation such as the Bitcoin Wiki tends to use the terms +big endian and little endian as shown in the table below, but they +aren't always consistent. Worse, these two different ways of +representing a hash digest can confuse anyone who looks at the Bitcoin +Core source code and finds a so-called "big endian" value being stored +in a little-endian data type. + +As header hashes and TXIDs are widely used as global identifiers in +other Bitcoin software, this reversal of hashes has become the standard +way to refer to certain objects. The table below should make clear where +each byte order is used. + +
+ +|---------------+---------------------|-----------------| +| Data | Internal Byte Order ("Big Endian") | RPC Byte Order ("Little Endian") | +|---------------|---------------------|-----------------| +| Example: SHA256(SHA256(0x00)) | Hash: 1406...539a | Hash: 9a53...0614 | +|---------------|---------------------|-----------------| +| Header Hashes: SHA256(SHA256(block header)) | Used when constructing block headers | Used by RPCs such as `getblock`; widely used in block explorers | +|---------------|---------------------|-----------------| +| merkle Roots: SHA256(SHA256(TXIDs and merkle rows)) | Used when constructing block headers | Returned by RPCs such as `getblock` | +|---------------|---------------------|-----------------| +| TXIDs: SHA256(SHA256(transaction)) | Used in transaction inputs | Used by RPCs such as `gettransaction` and transaction data parts of `getblock`; widely used in wallet programs | +|---------------|---------------------|-----------------| +| P2PKH Hashes: RIPEMD160(SHA256(pubkey)) | Used in both addresses and pubkey scripts | **N/A:** RPCs use addresses which use internal byte order | +|---------------|---------------------|-----------------| +| P2SH Hashes: RIPEMD160(SHA256(redeem script)) | Used in both addresses and pubkey scripts | **N/A:** RPCs use addresses which use internal byte order | +|---------------|---------------------|-----------------| + +
+ +Note: RPCs which return raw results, such as `getrawtransaction` or the +raw mode of `getblock`, always display hashes as they appear in blocks +(internal byte order). + +The code below may help you check byte order by generating hashes +from raw hex. +{% endautocrossref %} + +{% highlight python %} +#!/usr/bin/env python + +from sys import byteorder +from hashlib import sha256 + +## You can put in $data an 80-byte block header to get its header hash, +## or a raw transaction to get its txid +data = "00".decode("hex") +hash = sha256(sha256(data).digest()).digest() + +print "Warning: this code only tested on a little-endian x86_64 arch" +print +print "System byte order:", byteorder +print "Internal-Byte-Order Hash: ", hash.encode('hex_codec') +print "RPC-Byte-Order Hash: ", hash[::-1].encode('hex_codec') +{% endhighlight %} diff --git a/_includes/ref_core_rpcs-abcdefg.md b/_includes/ref_core_rpcs-abcdefg.md index 0ca49970..c8b028f1 100644 --- a/_includes/ref_core_rpcs-abcdefg.md +++ b/_includes/ref_core_rpcs-abcdefg.md @@ -1542,8 +1542,7 @@ An array of *transactions* in [transaction object format][]{:#term-transaction-o {% autocrossref %} Each object in the array contains the -rawtransaction *data* in hex and the *hash* of the data in little-endian -hex. +rawtransaction *data* in hex and the *hash* of the data in RPC byte order. {% endautocrossref %} diff --git a/_includes/ref_transactions.md b/_includes/ref_transactions.md index da9cf728..e532c043 100644 --- a/_includes/ref_transactions.md +++ b/_includes/ref_transactions.md @@ -86,7 +86,7 @@ bytes commonly used by Bitcoin are: 2. Create a copy of the version and hash; then hash that twice with SHA256: `SHA256(SHA256(version . hash))` -3. Extract the four most significant bytes from the double-hashed copy. +3. Extract the first four bytes from the double-hashed copy. These are used as a checksum to ensure the base hash gets transmitted correctly. @@ -135,6 +135,10 @@ Bitcoin transactions are broadcast between peers and stored in the block chain in a serialized byte format, called [raw format][]{:#term-raw-format}{:.term}. Bitcoin Core and many other tools print and accept raw transactions encoded as hex. +The binary form of a raw transaction is SHA256(SHA256()) hashed to create +its TXID. Bitcoin Core RPCs use a reversed byte order for hashes; see the [subsection about hash byte +order][section hash byte order] for details. + A sample raw transaction is the first non-coinbase transaction, made in [block 170][block170]. To get the transaction, use the `getrawtransaction` RPC with that transaction's txid (provided below): @@ -142,7 +146,7 @@ that transaction's txid (provided below): {% endautocrossref %} ~~~ -> getrawtransaction \ +> bitcoin-cli getrawtransaction \ f4184fc596403b9d638783cf57adfe4c75c605f6356fbc91338530e9831e9e16 0100000001c997a5e56e104102fa209c6a852dd90660a20b2d9c352423e\ diff --git a/_includes/references.md b/_includes/references.md index 1155fdb4..ae5396b8 100644 --- a/_includes/references.md +++ b/_includes/references.md @@ -46,6 +46,7 @@ [input]: /en/developer-guide#term-input "The input to a transaction linking to the output of a previous transaction which permits spending of satoshis" [inputs]: /en/developer-guide#term-input "The input to a transaction linking to the output of a previous transaction which permits spending of satoshis" [intermediate certificate]: /en/developer-examples#term-intermediate-certificate "A intermediate certificate authority certificate which helps connect a leaf (receiver) certificate to a root certificate authority" +[internal byte order]: /en/developer-reference#hash-byte-order "The standard order in which hash digests are displayed as strings---the same format used in blocks" [key index]: /en/developer-guide#term-key-index "An index number used in the HD wallet formula to generate child keys from a parent key" [key pair]: /en/developer-guide#term-key-pair "A private key and its derived public key" [label]: /en/developer-guide#term-label "The label parameter of a bitcoin: URI which provides the spender with the receiver's name (unauthenticated)" @@ -118,6 +119,7 @@ [regression test mode]: /en/developer-examples#regtest-mode "A local testing environment in which developers can control blocks" [root certificate]: /en/developer-examples#term-root-certificate "A certificate belonging to a certificate authority (CA)" [root seed]: /en/developer-guide#term-root-seed "A potentially-short value used as a seed to generate a master private key and master chain code for an HD wallet" +[RPC byte order]: /en/developer-reference#hash-byte-order "A hash digest displayed with the byte order reversed; used in Bitcoin Core RPCs and other software." [satoshi]: /en/developer-guide#term-satoshi "The smallest unit of Bitcoin value; 0.00000001 bitcoins. Also used generically for any value of bitcoins" [satoshis]: /en/developer-guide#term-satoshi "The smallest unit of Bitcoin value; 0.00000001 bitcoins. Also used generically for any value of bitcoins" [sequence number]: /en/developer-guide#term-sequence-number "A number intended to allow time locked transactions to be updated before being finalized; not currently used except to disable locktime in a transaction" @@ -283,6 +285,7 @@ [RPCs]: /en/developer-reference#remote-procedure-calls-rpcs [secp256k1]: http://perso.univ-rennes1.fr/sylvain.duquesne/master/standards/sec2_final.pdf +[section hash byte order]: /en/developer-reference#hash-byte-order [section verifying payment]: /en/developer-guide#verifying-payment [bitcoin URI subsection]: /en/developer-guide#bitcoin-uri [SHA256]: https://en.wikipedia.org/wiki/SHA-2 diff --git a/en/developer-reference.md b/en/developer-reference.md index 9ef2905e..01801548 100644 --- a/en/developer-reference.md +++ b/en/developer-reference.md @@ -37,6 +37,8 @@ title: "Developer Reference - Bitcoin" -- * https://en.bitcoin.it/wiki/API_reference_(JSON-RPC) --> +{% include ref_core_rpc_intro.md %} + ### Remote Procedure Calls (RPCs) **Warning:** the block chain and memory pool can include arbitrary data