BitTorrent’s Future? DHT, PEX and Magnet Links Explained

Written by Ernesto on November 20, 2009 

The Pirate Bay’s recent confirmation that they had closed down their tracker since DHT and Peer Exchange have matured enough to take over, was coupled with the news that they had added Magnet links to the site. This news has achieved its aim of stimulating discussion, but has also revealed that there is much confusion over how these technologies work.

The key thing to understand is that nobody is being forced to use Magnet links or trackerless torrents. While these long-standing technologies may prove to be the future, they will co-exist with tracker-enabled torrenting for quite some time. For now, nobody will be forced to immediately change their existing downloading habits, although it may be wise to switch to a BitTorrent client that is compatible with these technologies.

In an attempt to clear some of the mystique surrounding DHT, PEX and Magnet links we will walk through all three briefly, hoping to assure those who’ve become confused earlier this week.

Worlds most resiliant tracking

You might have noticed all the new magnet icons everywhere?

These are "magnet links", a link that lets you download a torrent directly in your BitTorrent client, instead of your browser. Most clients supports this (uTorrent, Vuze, rtorrent, whatever) and will get the relevant torrent data over the DHT network.

And DHT? It's a de-centralized peer to peer network that all modern clients join by default, even if they are currently not downloading any torrents. DHT can help you find peers and metadata when you choose to start a torrent download.

(If you want to learn more about DHT this Torrentfreak article might be a good place to start)

Busting Common Trackerless Torrent Myths

Written by Ben Jones on October 24, 2009 

You might also have noticed that the tracker has been down lately? And that the upload page don't recommend trackers anymore! The development of DHT has reached a stage where a tracker is no longer needed to use a torrent. DHT (combined with PEX) is highly effective in finding peers without the need for a centralized service. If you run uTorrent you might have noticed in the tracker tab of your torrents that the [Peer Exchange] (PEX) row is often reporting a lot more peers than the trackers you might have for that torrent. These peers all came to you without the use of a central tracker service! This is what we consider to be the future. Faster and more stability for the users because there is no central point to rely upon.

DHT has been included with many clients since it first debuted in the summer of 2005. however, over the 4 years of life, many myths and misunderstandings have been spread around. These can put people off using it and can give these users difficulties when a tracker goes down. Currently the Pirate Bay is popping on and offline, and Demonoid has been down for a week or two.

The main problem is that most people just don’t understand what DHT is, what it does, and how it works. Not really a surprise since the documentation and even the Wikipedia page are filled with technical jargon, and no simple explanation.  Without that basic understanding confusion is inevitable. We did explain DHT in our jargon piece back in 2006 but after 3 years, we decide to cover it again.

The easiest way to think about DHT is to imagine it as a form of ’super tracker’, in some ways a lot like WinMX and Kazaa of old. A large ad-hoc network of peers pass on information requests about torrents without a central server, meaning no control or single point of failure. No information about the contents or even the names of torrents are passed around, making this legal and hard to shut down.

Myth: You must turn off DHT when you use private trackers.

Wrong — There is an element to a torrent that is called the ‘private flag’. It’s a small flag that marks to a client that the torrent is ‘private’ and disables any method of sharing peers (including DHT), except via the tracker. This flag also changes the hash, so peers on a non-flagged torrent could not connect to a flagged torrent in any case. Most private torrent sites check for the flag, and add it if missing when the torrent is initially uploaded to their site.

Myth: Certain clients leak DHT data and should be avoided.

Wrong (with one exception) — There are always going to be people that want control. When it comes to torrent sites (especially the private ones) they like to express their control through lists of clients you can and can’t use (a form of DRM) and sometimes give reasons to support this. An example would be this statement from a staff member at a private tracker:

Not all torrent clients respect the private flag. But if you are using a client like Vuze, uTorrent or similar if the private flag is on (set by the tracker) the DHT, peer exchange settings etc are ignored. However, if you are using something like BitComet, BitLord or their ilk they ignore the private flag so if you have DHT etc enabled it is going to be enabled no matter what.

This statement is completely false. All torrent clients that support DHT respect the flag. The flag is set by the torrent file, not the tracker (although the tracker can add the flag to the file, it’s still set by the torrent), and BitComet does NOT ignore the flag. The one exception is a single build of BitComet (0.60) that was available for 2 weeks at the end of 2005, and even then, was a fallback only if the tracker was unable to be contacted for a 30minute period. Bitlord is unable to leak to DHT, as it doesn’t use DHT at all.

If you see staff making claims like this, it’s a good indication that the staff is clueless, which might be an idea to leave that tracker. If they can’t get the basics right who knows when else is wrong. Of course, we ask those claiming other clients leak to let us know so we can test it.

Myth: You can be tracked by DHT / AntiP2P groups use DHT to find you

Unlikely — It’s much easier and simpler to use the tracker. Blocklists, used on your client and on the trackers, are generally ineffective and easily circumvented through the use of residential connections. Last year’s University of Washington study showed that they will send letters just based on tracker info.

Myth: DHT slows your system down

Generally not true — It can slow down your connection depending mainly on network hardware. The actual data used in running DHT is low, generally less than 1kilobyte a second. Some routers and modems, however, can have problems with DHT causing lockups and restarts if they run out of ram. This mostly happens with lower spec ‘home’ equipment (such as older Belkins, Netgears and D-links), or telco-provided hardware.

Myth: You need to connect to a tracker, before you can use DHT

Wrong — When DHT is enabled (certainly in uTorrent) it connects to a bootstrap node (such as router.utorrent.com or router.bittorrent.com for mainline, or dht.aelitis.com for Vuze) and uses that to enter the DHT ’swarm’. It’s handed a set of DHT nodes and uses that to build up a small group of connected nodes. Those nodes are then used to get peers. No tracker is required at any time.

Myth: When enabled, it sends usage data back to [insert company]

Wrong — This is another case of people not knowing what they’re talking about. Generally they’re misinterpreting the bootstrap node connection for their client.

When the demonoid tracker was finally resurrected last year, many of it’s torrents were still active thanks mainly to DHT. DHT with Peer Exchange (PEX) is a very powerful addition to the torrenting world, and allows torrents to stay active, irrespective of the trackers stability or even existence. Also, Azureus/Vuze users, despite having their own DHT system, can join in using a mainline DHT plugin.

Busting Common Trackerless Torrent Myths

Now that the decentralized system for finding peers is so well developed, TPB has decided that there is no need to run a tracker anymore, so it will remain down! It's the end of an era, but the era is no longer up2date. We have put a server in a museum already, and now the tracking can be put there as well.

By moving to a more decentralized system of handling tracking (DHT+PEX) and distributions of torrent files (Magnet Links), BitTorrent will become less vulnerable to downtime and outages:

• With decentralized peer acquisition, there is no central tracker that can be down.
• With decentralized fetching of metadata (torrents) we don't need to rely on a single server that stores and distributes torrent files.

(Before you tech geeks out there start complaining about the info_hash in the magnet links being in HEX ("isn't it supposed to be in base32?") - No! According to the BitTorrent specification it should be in HEX but the client may choose to also support the old base32 encoding. If your client doesn't support the HEX encoding, please upgrade to the latest version of you client! If it still doesn't work, send an email to the developers of your client and ask them to add support for it.)

This is the future. And the present.