Google Incubator Jigsaw puzzle game Google has launched a counter-terrorism tool designed to help small businesses address extremist content shared across their online platforms.
While most major tech companies, including Meta, X, and Google itself, have considerable experience moderating content to remove illegal or toxic content, the resources required to do so are beyond the capabilities of many smaller digital players. This is particularly problematic at a time when legislation such as the European Digital Services Act and anti-terrorism content rules are putting increasing pressure on companies to remove illegal content quickly.
Formerly known as Google Ideas before rebranding in 2016, Jigsaw focuses on developing and testing new products that address societal issues such as misinformation, censorship, and the spread of toxic and extremist content online. Last November, Jigsaw launched Advertise A new online content management tool called to risehelping moderators verify whether content on their platforms matches what’s flagged in third-party databases, rather than having to check each source individually.
The elevation was developed in partnership with Technology Against Terrorisman initiative launched by the United Nations Counter-Terrorism Committee Executive Directorate in 2017. Tech Against Terrorism’s mission is to support the technology industry in developing more effective tools to address terrorists’ use of the Internet to spread harmful content. The tool was developed in partnership with Industrial led group The Global Internet Forum to Counter Terrorism (GIFCT), which includes most major technology companies as members.
Nine months later, Jigsaw is now officially handing Altitude over to Tech Against Terrorism, who will continue to develop and maintain it. The code is now also available on GitHub. Under the Apache 2.0 license, which means companies are able to self-host the tool, and third parties are able to inspect the code.
How does height work?
Altitude relies on “trusted” data sources provided by the likes of GIFCT, whose members collect their own information for the purpose of powering automated content moderation tools (like Altitude), as well as Tech Against Terrorism’s own Terrorist Content Analysis PlatformA database of verified terrorist content in real time. Trusted sources identify and flag this type of content using their own methods.
By driving these databases, Altitude is consolidating what it calls “terrorist and violent extremist content” into a single offering, where moderators can quickly view, review and remove content and adhere to their obligations under whatever jurisdiction they operate in. Notably, no content is automatically removed—it’s all about flagging potential extremist content and providing alerts so humans can evaluate it.
Neither Jigsaw nor Tech Against Terrorism disclosed which platforms Altitude used during the initial testing period, but the types of apps it was likely to use include messaging apps, URL shortening services, video hosting sites, online forums, social networks, and Pastebins.
Jigsaw books in Monday’s blog:
Over the course of a year, Jigsaw interviewed founders, trust and safety leaders, and TVEC content reviewers across eleven platforms, ranging in size. The platforms included photo-sharing platforms, social media sites, URL shortening services, and file and text storage sites. We found that platforms existed on a spectrum of resources and readiness to manage TVEC, and that this level of readiness was not clearly correlated with the size of their organization or user base. Even relatively mature services still sometimes lacked the signals needed to proactively identify harms on their platform. We also saw that while they may ultimately prefer to consolidate all tools into an internal platform, a separate dedicated harms interface could provide a way to get started in an under-resourced environment.
As an open source project, any company can integrate Altitude into their own platforms, or they can work with Tech Against Terrorism to access “custom routing support.”
Jigsaw and Tech Against Terrorism say they may add “specialized databases” to the data sources in the future, and plan to expand the tool beyond English.