Organization Calls for the Development of Tools to Monitor the Dark Web


The Dark Web (basically online content that is purposefully inaccessible from mainstream web browsers and search engines) poses many new challenges for investigators of online crime that requires a sophisticated approach to understanding its topography, according to a new working paper issued by the Global Commission on Internet Governance (GCIG).

The working paper, The Impact of the Dark Web on Internet Governance and Cyber Security, was co-authored by Michael Chertoff and Tobby Simon who represent the Chertoff Group and The Synergia Foundation, respectively. It is the sixth report in a GCIG series which looks at a range of Internet governance challenges, and is part of a leadup to a final report with set of recommendations it will publish in Spring 2016.

Anonymity Makes the Dark Web Appealing, But Also Prone to Crime

The “Deep Web” is distinguished from the public Internet because it includes content not indexed by normal search engines for various reasons. This might include content that isn’t illegal but that its owners don’t want made public for legitimate reasons such as internal business information or even family photos.

The Dark Web, however, includes content that is purposefully inaccessible from mainstream web browsers.

The Tor network, for instance, is a relatively popular anonymous network that can only be accessed with the Tor web browser in order to find Dark Web content, and to browse online material anonymously.

The authors note that individuals who would be in danger if their identities were known might benefit from the anonymity of the Dark Web, however, this anonymity is also used for nefarious activities such as the sale of drugs and weapons, financing terrorist activities, identity theft, and a variety of other online crime.

They write, “The Dark Web, in general, and the Tor network, in particular, offer a secure platform for cybercriminals to support a vast amount of illegal activities — from anonymous marketplaces to secure means of communication, to an untraceable and difficult to shut down infrastructure for deploying malware and botnets.”

Anonymous and Unregulated Online Activity Worries Governments

The report also references a November op-ed by Robert Hannigan, the director of Britain’s Government Communications Headquarters (GCHQ), a UK government intelligence and security organization. In the piece, he mentions that social media networks have helped spread terrorism, and privacy tools are making it difficult to collect intelligence on suspected terrorist activity.

Hannigan wrote, “Techniques for encrypting messages or making them anonymous which were once the preserve of the most sophisticated criminals or nation states now come as standard. These are supplemented by freely available programs and apps adding extra layers of security, many of them proudly advertising that they are ‘Snowden approved’. There is no doubt that young foreign fighters have learnt and benefited from the leaks of the past two years.”

Chertoff and Simon note that there’s a fear that tech companies like Google, Apple, Snapchat and Facebook have been developing security features and practices that make it difficult to monitor user activity. They also mention a low-profile Facebook announcement that it was using Tor to allow users to surf the web without being tracked and publish content that would not show up in normal search engines, and, in so doing, became the first US tech giant to officially provide Tor support.

This corresponds to the UK government’s recent aims to crack down on anti-surveillance techniques such as encryption, which could circumvent the ability of government agencies to eavesdrop on terrorist communication.

Because of the anonymity afforded by the Dark Web, it is largely considered an unregulated space.

“[T]he hidden ecosystem [of the Dark Web] is conducive for propaganda, recruitment, financing and planning, which relates to our original understanding of the dark Web as an unregulated space,” write Chertoff and Simon. “Providing evidence showing that the Dark Web has turned into a major platform for global terrorism and criminal activities is crucial in order for the necessary tools to be developed for monitoring all parts of the Internet.”

A Plan for Bringing Oversight to the Dark Web

The authors recommend several efforts to monitor and map the Dark Web including mapping the hidden services directory by deploying nodes into the Distributed Hash Table (which acts as a form of distributed DNS for resolving Dark Web hostnames) to possibly monitor requests coming from a given domain.

Dark Web users could be anonymously monitored in order to find connections to non-standard domains to scope out Dark Web locations of interest to investigators. And public Internet sites like Pastebin can reveal contact information and addresses for new hidden services on the Dark Web.

Chertoff and Simon also suggest investigators take a snapshot of every new site for ongoing or later analysis since most hidden services tend to go offline very often, and reappear later under new domains. Based on collected data, semantic analysis could be used to track illegal activities and malicious actors, and associate them with particular activities.

And, finally, it could be useful to focus on profiling transactions made on dark Web marketplaces to gather information about sellers, buyers, and the kinds of goods exchanged.

Up until this point, cybercriminals have had a relatively easy time circumventing law enforcement, however, authorities have made certain strides to bring order to the Dark Web, including the take down of the illegal goods marketplace Silk Road and conviction of its founder. However, other Dark Web marketplaces easily take its place.

Chertoff and Simon suggest that criminal elements will become more sophisticated as law enforcement gets more effective in navigating the Dark Web’s criminal underbelly, leading these nefarious actors to hide into more fragmented alternative “Dark Nets” (small niches within the “Deep Web) and private networks which make them more difficult to find.

They conclude, “Security researchers have to remain vigilant and find new ways to spot upcoming malicious services to deal with new phenomena as quickly as possible.”

Add Your Comments

  • (will not be published)


  1. I have a Dark Web Search engine with over 400,00 URL and web-content I crawled the last 2 weeks from this my next crawl should have over 2-3 million Dark Web sites and content. The dark web is Tor and i2p if you go to my site I even give for free i2p sites I found from my crawl. - I built this tool 2 years ago when nobody had a clue - So yes there are other tools for the Dark Web but everyone wants it free and it cost money to crawl the Dark Web. I have contacted the FBI and other agencies but since I am just a disable Veteran with 35+ years in IT I guess I don't qualify.

    • raw

      Mr Richards pls how do i get your tool