On-Device Context: A defense against misinformation in the encrypted WhatsApp
Balancing privacy, security, and mitigation of misinformation
We expand upon our recently published proposal for how end-to-end encrypted messaging systems can address misinformation using “on-device context” while preserving privacy. Misinformation spread through such systems have had — and, if unchecked, will continue to have — significant negative impacts to communities, polities, and even health. Here we go deeper into explaining the rationale, approach, and challenges of the proposal.
How it works
We propose maintaining a list of rumors (including image, audio, and video hashes) along with corresponding fact-checks, similar to what Facebook uses when identifying misinformation in its news feed. This “context list” can be regularly supplied to WhatsApp clients, so that if someone receives a debunked rumor, WhatsApp can provide the relevant context or fact-check (similar to how WhatsApp currently flags suspicious links). The same list can also be used to warn people sending rumors in the first place.
This on-device context approach is simply a combination of ideas from two existing models. Facebook’s collaboration with the International Fact Checking Network provides a trusted source of debunks and context around viral rumors. Modern browsers regularly download a list of malware sites which they warn users about in order to protect them while still maintaining user privacy (many browser extensions including most ad blockers also use client-side lists).
To ensure fact checkers are aware of and can triage what to check, WhatsApp can rely on traditional methods, such as advertising to people to report rumors and monitoring public social media. In addition, WhatsApp could support “rumor gathering” through additional built-in reporting features to encourage people to share potential misinformation, and through differential privacy protected statistics on what people are seeing in different regions.
It’s also important to note that on-device context might only be applied to larger group chats e.g. 50+ or even 100+ member groups through which many rumors travel.
Types of threats
But why does this matter? What are we concerned about? A strawman argument might say that our goal is stop lynchings triggered by misinformation, given recent news coverage of them, but this is only a small slice of the problem.
The following is not an exhaustive list or a full threat model, but it provides context on the breadth of the threats resulting from misinformation.
- Health misinformation — enabling public health crises.
- Coordinated political misinformation — giving bad actors an advantage in elections.
- Rumors that can trigger mob or ethnic violence — causing gruesome lynchings in India, Mexico and other countries.
These are not academic issues, nor are they hypothetical. For example, from Brazil to Mexico to India, powerful political parties (or their surrogates) have coordinated end-to-end encrypted campaigns across many of the world’s largest democracies, involving millions of people that can be reached via hierarchies of groups (allegedly within minutes). Many of these campaigns involve misinformation, stirring up ethnic tensions, a particularly dangerous combination.
It is important to note that due to the content of the rumor messages, spam fighting techniques are not always applicable. The messages can be entertaining or amusing (say, in the form of memes) or shared by a trusted family member in a group WhatsApp chat. Expecting an ordinary person to question or doubt what someone they trust says is unrealistic.
We may provide a comprehensive threat model, spelling out Principals, Goals, Adversities, and Invariants in a longer piece in the future; please get in touch if you are interested in contributing to that effort.
Goals and Constraints
The goals of any intervention would be to make it harder for both coordinated and emergent misinformation to spread through end-to-end encrypted networks in ways that cause significant and lasting harm.
India and other countries purportedly want to “address misinformation” by forcing tech companies to break encryption to trace messages and prosecute people. We believe that will cause significant harm, as it leads to less privacy and security for users and potentially weaponization by governments.
Instead, we are propose the on-device context approach because it satisfies two core principles:
Core Principles
- No censorship: Every message sent should be received. This system can provide additional context, but it is never used to “disappear” messages.
- No spying: No new information should be available to WhatsApp about the user, or about messages they have sent or received without explicit user permission.
Potential Challenges
Unfortunately, this doesn’t come completely for free. There are a number of potential issues that can limit the effectiveness or applicability of the approach. As explained below, these challenges are significant, but we believe they are surmountable.
User resource constraints: Bandwidth, Storage, CPU, Battery
Regularly updating “context lists” might require additional bandwidth, in environments that have low connectivity, high costs, or both. Storing the list and a more complex app would increase the device’s storage requirements for an app, and potentially consume additional CPU and battery life. This could lead to a worse user experience in low-resource environments where cheap old phones are the norm.
While this is true, it also applies to many app changes, whether they defend against misinformation, or increase privacy and security. Practically, the question is whether or not the additional resource consumption would be egregious, and we don’t believe that to be the case. We also propose that metadata already captured by WhatsApp (e.g. region) can be used to selectively update lists for users based off the likelihood that they will encounter a particular rumor, reducing most of the costs mentioned above.
Fact checking credibility and costs
A major challenge of any effort that attempts to fact check or provide context, is understanding why the fact checkers or educators should themselves be trusted. We view IFCN’s work with Facebook as a model for this. It’s not perfect, but it enables similar “context lists”, which are used server-side by Facebook’s newsfeed to call out potential misinformation.
That said, fact checking and contextualization isn’t free. Facebook (and other platforms) and funders would ideally be supporting this work (via third parties) at the necessary scale of tens or even hundreds of millions of dollars each year across all regions they operate in.
UX and psychosocial challenges
Getting the user experience right for this is difficult. How do we tastefully provide context and minimize the extent to which people and communities react poorly to fact checks? How do we ensure people know that they can report potential rumors to be checked without clogging up the interface? How do we let users know that if they click a link to the details of the fact check, external sites can then track them? How do you prevent people from feeling like they are being watched when a fact checks pops up for a message — regardless of their actual privacy? How can we guarantee that flagging a subset of false information doesn’t make people automatically trust everything not flagged?
These are important challenges, and we will need significant research and testing to answer these questions. But, they are similar to the challenges that platform companies already take on in order to develop their products. We simply need sufficient resourcing, which is not an insurmountable deterrent.
Increased government pressure
Adding functionality such as on-device contextualisation could lead to a slippery slope scenario where governments compel companies to push their propaganda or censor fact checks that contradict their messaging.
There is a trade-off that needs to be weighed here: is the potential slippery slope more likely than the real and present danger to end-to-end encryption? Australia and Vietnam have already attempted to neutralize encryption and India is next in line, using the harms of misinformation as a primary rationale. We believe that checks against misinformation could reduce the likelihood that end-to-end encryption is made illegal.
Conclusions
Technology alone won’t “solve” or “stop” misinformation, and this piece addresses just one part of a single vector in a much larger ecosystem of challenges. To significantly alleviate the harms of misinformation, without creating a cure worse than the disease, will involve the careful actions of many stakeholders. We believe that this approach of on-device context, harnessing both fact checkers and technology changes, will likely to scale better than either standing alone, without the privacy loss that many other proposals require.
Thanks to Priyanjana Bengani and Christina Kelly for helping improve this proposal, and to Alec Muffett for critical feedback.
Aviv Ovadya is the founder of the Thoughtful Technology Project, and was previously Chief Technologist at the Center for Social Media Responsibility at UMSI. He is also a non-resident fellow at the German Marshall Fund’s Alliance for Securing Democracy.
If you believe there are important details not addressed here, please get in touch at av@aviv.me or on twitter at @metaviv.
