Merge branch 'peer-discovery' (pull #617)

_autocrossref.yaml

@@ -2,8 +2,11 @@
 ## List of words to match with references in _includes/references.md
 ## in developer documentation, used by autocrossref.rb plugin.
 ## "pattern to match in file" => "reference to give it"
+#
+## To prevent a ref for term, use: "term: DO NOT AUTOCROSSREF"
 
 51 percent attack:
+'`addr`': addr message
 address: 
 addresses: address
 '`amount`': pp amount
@@ -55,6 +58,8 @@ denominations: denomination
 DER format: der
 DER-formatted: der
 difficulty:
+dns seed:
+dns seeds: dns seed
 double spend:
 double-spend: double spend
 double spending: double spend
@@ -82,6 +87,8 @@ input:
 intermediate certificate:
 intermediate certificates: intermediate certificate
 internal byte order:
+IP address: DO NOT AUTOCROSSREF
+IP addresses: DO NOT AUTOCROSSREF
 key index:
 key pair:
 '`label`': label
@@ -89,6 +96,8 @@ leaf certificate:
 locktime:
 long-term fork:
 mainnet:
+man in the middle: man-in-the-middle
+man-in-the-middle:
 master chain code:
 master private key:
 '`memo`': pp memo

_includes/guide_p2p_network.md

@@ -2,7 +2,24 @@
 
 {% autocrossref %}
 
-The Bitcoin [network][network]{:#term-network}{:.term} uses simple methods to perform peer discovery and communicate between nodes. The following section applies to both full nodes and SPV clients, with the exception that SPV's Bloom filters take the role of block discovery.
+The Bitcoin network protocol allows full nodes
+([peers][peer]{:#term-peer}{:.term}) to collaboratively maintain a
+[peer-to-peer network][network]{:#term-network}{:.term} for block and
+transaction exchange. Many SPV clients also use this protocol to connect
+to full nodes.
+
+Consensus rules do not cover networking, so Bitcoin programs may use
+alternative networks and protocols, such as the [high-speed block relay
+network][] used by some miners and the [dedicated transaction
+information servers][electrum server] used by some wallets that provide
+SPV-level security.
+
+To provide practical examples of the Bitcoin peer-to-peer network, this
+section uses Bitcoin Core as a representative full node and [BitcoinJ][]
+as a representative SPV client. Both programs are flexible, so only
+default behavior is described. Also, for privacy, actual IP addresses
+in the example output below have been replaced with [RFC5737][] reserved
+IP addresses.
 
 {% endautocrossref %}
 
@@ -10,13 +27,91 @@ The Bitcoin [network][network]{:#term-network}{:.term} uses simple methods to pe
 
 {% autocrossref %}
 
-Bitcoin Core maintains a list of [peers][peer]{:#term-peer}{:.term} to connect to on startup. When a full node is started for the first time, it must be bootstrapped to the network. This is done automatically today in Bitcoin Core by a short list of trusted DNS seeds. The option `-dnsseed` can be set to define this behavior, though the default is `1`. DNS requests return a list of IP addresses that can be connected to. From there, the client can start connecting the Bitcoin network.
+When started for the first time, programs don't know the IP
+addresses of any active full nodes. In order to discover some IP
+addresses, they query one or more DNS names (called [DNS seeds][dns
+seed]{:#term-dns-seed}{:.term}) hardcoded into Bitcoin Core and
+BitcoinJ. The response to the lookup should include one or more [DNS
+A records][] with the IP addresses of full nodes that may accept new
+incoming connections. For example, using the [Unix `dig`
+command][dig command]:
 
-Alternatively, bootstrapping can be done by using the option `-seednode=<ip>`, allowing the user to predefine what seed server to connect to, then disconnect after building a peer list. Another method is starting Bitcoin Core with `-connect=<ip>` which disallows the node from connecting to any peers except those specified. Lastly, the argument `-addnode=<ip>` simply allows the user to add a single node to his peer list.
+    ;; QUESTION SECTION:
+    ;seed.bitcoin.sipa.be.	    IN  A
 
-After bootstrapping, nodes send out a `addr` message containing their own IP to peers. Each peer of that node then forwards this message to a couple of their own peers to expand the pool of possible connections.  
+    ;; ANSWER SECTION:
+    seed.bitcoin.sipa.be.	60  IN  A  192.0.2.113
+    seed.bitcoin.sipa.be.	60  IN  A  198.51.100.231
+    seed.bitcoin.sipa.be.	60  IN  A  203.0.113.183
+    [...]
 
-To see which peers one is connected with (and associated data), use the `getpeerinfo` RPC.
+The DNS seeds are maintained by Bitcoin community members: some of them
+provide dynamic DNS seed servers which automatically get IP addresses
+of active nodes by scanning the network; others provide static DNS
+seeds that are updated manually and are more likely to provide IP
+addresses for inactive nodes. In either case, nodes are added to the
+DNS seed if they run on the default Bitcoin ports of 8333 for Mainnet
+or 18333 for Testnet.
+
+<!-- paragraph below based on Greg Maxwell's email in
+     http://comments.gmane.org/gmane.comp.bitcoin.devel/5378 -->
+
+DNS seed results are not authenticated and a malicious seed operator or
+network man-in-the-middle attacker can return only IP addresses of
+nodes controlled by the attacker, isolating a program on the attacker's
+own network and allowing the attacker to feed it bogus transactions and
+blocks.  For this reason, programs should not rely on DNS seeds
+exclusively.
+
+Once a program has connected to the network, its peers can begin to send
+it [`addr`][addr message]{:#term-addr-message}{:.term}
+(address<!--noref-->) messages with the IP addresses and port numbers of
+other peers on the network, providing a fully decentralized method of
+peer discovery. Bitcoin Core keeps a record of known peers in a
+persistent on-disk database which usually allows it to connect directly
+to those peers on subsequent startups without having to use DNS seeds.
+
+However, peers often leave the network or change IP addresses, so
+programs may need to make several different connection attempts at
+startup before a successful connection is made. This can add a
+significant delay to the amount of time it takes to connect to the
+network, forcing a user to wait before sending a transaction or checking
+the status of payment.
+
+<!-- reference for "Bitcoin Core...11 seconds" below:
+     https://github.com/bitcoin/bitcoin/pull/4559 -->
+
+To avoid this possible delay, BitcoinJ always uses dynamic DNS seeds to
+get IP addresses for nodes believed to be currently active.
+Bitcoin Core also tries to strike a balance between minimizing delays
+and avoiding unnecessary DNS seed use: if Bitcoin Core has entries in
+its peer database, it spends up to 11 seconds attempting to connect to
+at least one of them before falling back to seeds; if a connection is
+made within that time, it does not query any seeds.
+
+<!-- reference for Bitcoin Core behavior described below: search for
+"FixedSeeds" in src/net.cpp; BitcoinJ has IPv4 seeds in its chainparams
+and a function to use them, but I don't see that function being used in
+any of the examples/wallet templates (but I'm not Java fluent, so
+maybe PEBKAC). -@harding -->
+
+Both Bitcoin Core and BitcoinJ also include a hardcoded list of IP
+addresses and port numbers to several dozen nodes which were active
+around the time that particular version of the software was first
+released. Bitcoin Core will start attempting to connect to these nodes
+if none of the DNS seed servers have responded to a query within 60
+seconds, providing an automatic fallback option.
+
+As a manual fallback option, Bitcoin Core also provides several
+command-line connection options, including the ability to get a list of
+peers from a specific node by IP address, or to make a persistent
+connection to a specific node by IP address.  See the `-help` text for
+details.  BitcoinJ can be programmed to do the same thing.
+
+**Resources:** [Bitcoin Seeder][], the program run by several of the
+seeds used by Bitcoin Core and BitcoinJ. The Bitcoin Core [DNS Seed
+Policy][].  The hardcoded list of IP addresses used by Bitcoin Core and
+BitcoinJ is generated using the [makeseeds script][].
 
 {% endautocrossref %}
 

_includes/references.md

@@ -1,5 +1,6 @@
 <!-- Terms; must have tooltip description in "quotes"; alphabetical order -->
 [51 percent attack]: /en/developer-guide#term-51-attack "The ability of someone controlling a majority of hashing power to revise transactions history and prevent new transactions from confirming"
+[addr message]: /en/developer-guide#term-addr-message "A Bitcoin network protocol message which relays IP addresses and port numbers of active nodes to other nodes and clients, allowing decentralized peer discovery."
 [addresses]: /en/developer-guide#term-address "A 20-byte hash formatted as a P2PKH or P2SH Bitcoin Address"
 [address]: /en/developer-guide#term-address "A 20-byte hash formatted as a P2PKH or P2SH Bitcoin Address"
 [base58Check]: /en/developer-reference#term-base58check "The method used in Bitcoin for converting 160-bit hashes into Bitcoin addresses"
@@ -34,6 +35,7 @@
 [consensus rules]: /en/developer-guide#term-consensus-rules "The block validation rules that full nodes follow to stay in consensus with other nodes."
 [denomination]: /en/developer-guide#term-denomination "bitcoins (BTC), bitcents (cBTC), millibits (mBTC), microbits (uBTC), or satoshis"
 [difficulty]: /en/developer-guide#term-difficulty "A number corresponding to the target threshold which indicates how difficult it will be to find the next block"
+[dns seed]: /en/developer-guide#term-dns-seed "A DNS server which returns IP addresses of full nodes on the Bitcoin network to assist in peer discovery."
 [double spend]: /en/developer-guide#term-double-spend "Attempting to spend the same satoshis which were spent in a previous transaction"
 [extended key]: /en/developer-guide#term-extended-key "A public or private key extended with the chain code to allow them to derive child keys"
 [extended private key]: /en/developer-guide#term-extended-private-key "A private key extended with the chain code so that it can derive child private keys"
@@ -278,13 +280,16 @@
 [BIP71]: https://github.com/bitcoin/bips/blob/master/bip-0071.mediawiki
 [BIP72]: https://github.com/bitcoin/bips/blob/master/bip-0072.mediawiki
 [CVE-2012-2459]: https://en.bitcoin.it/wiki/CVEs#CVE-2012-2459
+[RFC5737]: http://tools.ietf.org/html/rfc5737
 <!-- [secp256k1]: http://www.secg.org/index.php?action=secg,docs_secg -->
 [secp256k1]: http://perso.univ-rennes1.fr/sylvain.duquesne/master/standards/sec2_final.pdf
 
 <!-- Other external site links; alphabetical order -->
 [BFGMiner]: https://github.com/luke-jr/bfgminer
 [bitcoin core fee drop commit]: https://github.com/bitcoin/bitcoin/commit/6a4c196dd64da2fd33dc7ae77a8cdd3e4cf0eff1
+[Bitcoin Seeder]: https://github.com/sipa/bitcoin-seeder
 [bitcoin-documentation mailing list]: https://groups.google.com/forum/#!forum/bitcoin-documentation
+[BitcoinJ]: http://bitcoinj.github.io
 [block170]: https://www.biteasy.com/block/00000000d1145790a8694403d4063f323d499e655c83426834d4ce2f8dd4a2ee
 [casascius address utility]: https://github.com/casascius/Bitcoin-Address-Utility
 [core base58.h]: https://github.com/bitcoin/bitcoin/blob/master/src/base58.h
@@ -292,14 +297,21 @@
 [core paymentrequest.proto]: https://github.com/bitcoin/bitcoin/blob/master/src/qt/paymentrequest.proto
 [core script.h]: https://github.com/bitcoin/bitcoin/blob/master/src/script.h
 [DER]: https://en.wikipedia.org/wiki/Abstract_Syntax_Notation_One
+[dig command]: https://en.wikipedia.org/wiki/Dig_%28Unix_command%29
+[DNS A records]: http://tools.ietf.org/html/rfc1035#section-3.2.2
+[DNS Seed Policy]: https://github.com/bitcoin/bitcoin/blob/master/doc/dnsseed-policy.md
 [docs issue]: https://github.com/bitcoin/bitcoin.org/issues
 [ECDSA]: https://en.wikipedia.org/wiki/Elliptic_Curve_DSA
+[Electrum server]: https://github.com/spesmilo/electrum-server
 [Eloipool]: https://gitorious.org/bitcoin/eloipool
 [forum tech support]: https://bitcointalk.org/index.php?board=4.0
+[high-speed block relay network]: https://www.mail-archive.com/bitcoin-development@lists.sourceforge.net/msg03189.html
 [HMAC-SHA512]: https://en.wikipedia.org/wiki/HMAC
 [HTTP longpoll]: https://en.wikipedia.org/wiki/Push_technology#Long_polling
 [irc channels]: https://en.bitcoin.it/wiki/IRC_channels
 [libblkmaker]: https://gitorious.org/bitcoin/libblkmaker
+[makeseeds script]: https://github.com/bitcoin/bitcoin/tree/master/contrib/seeds
+[man-in-the-middle]: https://en.wikipedia.org/wiki/Man-in-the-middle_attack
 [MIME]: https://en.wikipedia.org/wiki/Internet_media_type
 [mozrootstore]: https://www.mozilla.org/en-US/about/governance/policies/security-group/certs/
 [Payment Request Generator]: http://bitcoincore.org/~gavin/createpaymentrequest.php

_plugins/autocrossref.rb

@@ -71,7 +71,13 @@ require 'yaml'
 		     ## becomes an issue, we can devise a more complex
 		     ## regex
             (?!\w)  ## Don't match inside words
-          /xmi, "[\\&][#{term[1]}]{:.auto-link}")
+          /xmi) {|s|
+              if term[1] == "DO NOT AUTOCROSSREF"
+                  s.gsub(/( |$)/, "<!--noref-->\\&")
+              else
+                  "[#{s}][#{term[1]}]{:.auto-link}"
+              end
+              }
       }
       output.gsub!(/<!--.*?-->/m,'')  ## Remove all HTML comments