Robust hashing could be new tech tool to fight terror online

From rabid terror propaganda to dangerous rumour-mongering, the internet is an archive of inflammatory material that governments as well as social media networks across the world have struggled to contain and cope with. Since mid-2015 to the beginning of this year, Twitter suspended over 1,25,000 accounts linked to Islamic State terror group. Yet organizations promoting terror continue to survive, if not thrive, online.

Now, a cutting-edge technology could emerge as the biggest tool to combat the menace. Hany Farid, senior advisor to the US-based Counter Extremism Project, has helped develop a technology called robust hashing which uses a software to identify images, videos, and audio clips by comparing them against a bank of stored content.

With robust hashing, content can be taken down as soon as it is identified. “It can be fully automated. The only manual step would be if a service provider wants to check if posted material is by media outlet,“ Farid told TOI over email, adding, “We can create a white-list of approved sites and ban all other users.“

This is how it works: Machines see an image as a series of 1s and 0s. Images and its components are identified by a unique code, known as a hash. The hash remains the same even if you resize or crop an image. While filtering images, the software compares these hashes against those of a repository of images it already has. A hash match leads to flagging of content.

Farid had earlier developed a similar technology for identifying and taking down child pornography. The software, called PhotoDNA, was used by social networking giants like Facebook and Twitter. That was in 2013.

For the latest video and audio-enabled version of this technology, the CEP has developed a database of terror propaganda material. Potentially extremely useful in times when alarming extremist content can be rapidly shared by multiple user accounts, lack of human intervention can have its own pitfalls. Last week, soon after robust hashing was introduced, YouTube took down a counter-jihad video from its website for “hate speech”.


The video, titled “Killing for a Cause: Sharia Law & Civilization Jihad,” used propaganda material to counter the jihad narrative. Google, which runs YouTube, declined to comment on whether the takedown in this case was automated or manual.

Closer home in India, we see more examples of the problems that can come from blanket terms for classification of content. In February
this year, the website jihadology.net was blocked in India by several ISPs following a government order that sought blocking of ISIS propaganda online. Jihadology, run by research scholar Aaron Zelin, is a clearing house of information on Islamic terror around the world.

Errors like these have got some American academics talking about “algorithmic accountability” – or transparency in decisions made using algorithms.
Nicholas Diakopoulos is an assistant professor at the University of Maryland and a fellow at the Tow Center for Digital Journalism at Columbia University. His Feb 2016 paper titled “Accountability in Algorithmic Decision Making” points to the fallibility of algorithms. “The opportunities for bias, uncertainty, or outright mistakes are plentiful in automated classification. The training data that is the basis for supervised machine-learning algorithms is an important consideration, given the human biases that may be lurking there.”

“I think the biggest problem with this algorithmic policing is always going to be the fact that algorithms are ruthless,” says Nishant Shah of Center for Internet and Society. “Once they have been fed parameters of policing, they would punish everybody equally. So whether you are a first respondent reporting from a scene of terror or an organisation spreading hate speech, if the algorithm detects similar semantic patterns, it would censor both equally,” he says.

Shah recalls the online message boards of the early 90s, where bots weeded out pornography by policing terms like “pornography” or “porn”. Users found ways to work around the algorithm by replacing the term with “pron”, “p0rn” and eventually, even “prawn”.

In times when cyberspace has become a platform for brainwashing innocent youths, robust hashing can be an invaluable tool despite its limitations. It needs to constantly updated and improved though. As Farid points out, such algorithmic policing can never be a one-time solution. As law breakers evolve, law enforcers will have to follow suit. “As with spam/anti-spam, virus/anti-virus, this will be a forever evolving game of cat and mouse. We develop a technology to squash material, our adversary adapts with a new technology or approach, (and then) we adapt,” he says.