Villains and Vigilantes—Mapping the Dark Web

Dark. Mysterious. A den of thieves. A front for freedom fighters. It is many things for many different kinds of people, all of whom by nature or necessity find themselves driven to the fringes of digital society. It is the dark web. All this and more will be discussed in part two. 

There’s still plenty of misinformation floating around out there about this obscure corner of the internet. The average cyber citizen is unaware of its very existence. Even for those intimately familiar with the dark web, accurate predictions as to its behavior and future effect on broader internet culture have remained elusive; predictions foretelling its mainstreaming, for instance, seem less and less likely with each passing year. The problem is, this is one case where ignorance isn’t always bliss. Dark web relevance to the general population is becoming more painfully apparent with every breaking news story about yet another data breach.

The amount of personal information accessible via a web connection these days is staggering. Names, addresses, and phone numbers are only the tip of the iceberg. Credit card information, marital status, browsing histories, purchase histories, medical histories (a favorite target of hackers these days) and so much more—every bit and byte of this data is at risk of theft, ransom, exposure and exploitation. A person’s entire life can be up for sale on the dark web without them being any the wiser. That is until their credit card comes up overdrawn, or worse, a mysterious and threatening email graces their inbox threatening to expose some very private information.

But despite the fact that it is the individual being exposed, the ones who truly have to worry are those entities entrusted with storing the individual data of their millions of users. The dark web is a potential nightmare for banks, corporations, government bureaus, health care providers—pretty much any entity with large databases storing sensitive (i.e., valuable) information. Many of these entities are waking up to the dangers, some rudely so, and are too late to avoid paying out a hefty ransom or fine depending on how they handle the situation. Whatever the case, the true cost is often to the reputation of the entity itself, and it is sometimes unrecoverable.

It should be obvious at this point that the dark web cannot be ignored. The first step to taking it seriously is to understand what it is and where it came from.

The landscape

Perhaps the most common misconception regarding the dark web begins with the internet itself. Contrary to popular sentiment, Google does not know all. In fact, it is not even close. Sundar Pichai and his legions of Googlers only index pages they can access, which by current estimates hover in and around the $60 billion mark. Sounds like a lot, but in reality this is only the surface web, a paltry 0.2% to 0.25% of digital space.

Home for the bulk of our data, the other 99.75% is known as the deep web. Research on deep web size is somewhat dated but the conditions the findings are based on appear to point to a growing size disparity, if any changes have occurred at all.

Unlike the surface web, which is made up of all networked information discoverable via public internet browsing, the deep web is all networked information blocked and hidden from public browsing.

Take Amazon as an example. It has its product pages, curated specifically to customer browsing habits and seemingly eerily aware of conversations people have had around their Alexa—this is the Surface Web. But powering this streamlined customer experience are databases storing details for hundreds of millions of customers; including personal identifiable information (PII), credit card and billing information, purchase history, and the like. Then there are databases for the millions of vendors, warehouse databases, logistical databases, corporate intranet, and so on. All in all you are looking at a foundational data well some 400 to 500 times larger than the visible surface.

The dark web is technically a part of this deep web rubric, meeting the criteria of being hidden from indexing by common web browsers. And although microscopically small in comparison it can have an outsized effect on the overall superstructure, sort of like a virus or a cure, depending on how it is used. In the Amazon example, where the dark web fits in is that a portion of its members would like nothing better than to access its deep web data for any number of nefarious purposes, including sale, ransom, or just to sow a bit of plain old anarchic chaos.

Such activities do not interest all dark web users, of course, with many seeing anonymity as an opportunity to fight off corruption rather than be a part of it. The dark web is a complex place, and to fully appreciate this shadow war of villains and vigilantes, how it can affect millions of people every now and then when it spills over into the light, first you have to understand its origins.

Breaking down the numbers

Anonymity is not without its challenges when it comes to mapping out hard figures. The key is to focus on commerce, a clear and reliable demarcating line. For the most part, those only seeking anonymity can stick to hidden chat rooms and the like. However, if a user is looking to engage in illegal activity, in most instances they’re going to have to pay for it. Several past studies and more recent work provide workable insight when extrapolating along this logic path.

First, a 2013 study analyzing 2,618 services being offered found over 44% to involve illicit activity. That number jumped to 57% in a follow up study conducted in 2016. These studies alone project an accelerating upward trend. Short of a more recent comprehensive study, the tried and true investigative maxim of “follow the money” should suffice in convincing the rational mind that this number is only going to grow dramatically. Especially when comparing the $250 million in bitcoin spent in 2012 on the dark web with the projected $1 billion mark for 2019.

Origins and operation

It was the invention of none other than the U.S. military—the Navy, of all branches, if you’d believe it. Seeking an easy way for spy networks to communicate without having to lug heavy encryption equipment to remote and hostile corners of the globe, the U.S. Naval Research Laboratory (NRL) came up with an ingenious solution. Ditching the equipment, it created an overlay network of unique address protocols and a convoluted routing system, effectively masking both the source and destination of all its traffic. By forgoing the traditional DNS system and relying instead on software specific browsers like Tor and Freenet and communication programs like I2P among others, dark web traffic was rendered invisible to traditional crawlers. Furthermore, with these browsers routing traffic through multiple user stations around the world, accurate tracking became extremely difficult. This solution afforded both flexibility and mobility for quick and easy insertion and extraction of human assets while securing sensitive communication to and from the field.

There was only one element missing. As co-creator Roger Dingledine explained, if only U.S. Department of Defense (DoD) personnel used the network it wouldn’t matter that source and destination were masked between multiple user stations. All users would be identifiable as part of the spy network. It would be like trying to hide a needle in a stack of needles. What the dark web needed was a haystack of non DoD users. And so in 2002 the software was made open source and anyone seeking the option to communicate and transact globally was invited to download it. Thousands of freedom-conscious people heeded the call and thus the dark web was born.

But freedom is morally ambiguous, granting expression to the best and worst urges of humanity. This is why security officers and senior executives in banks and businesses, insurance providers and intelligence agencies, all need to know who is using the dark web, what it is being used for, and how imminent is the threat it poses to their operations.

All this and more will be discussed in part two.

Ariel Yosefi is the head of the technology and regulation department, at Israel-based law firm Herzog Fox & Neeman. Avraham Chaim Schneider is coordinator of the firm’s cyber and innovation media project.