Metrics are key to how product teams at tech companies function

Image for post
Image for post
Photo: Chesnot/Getty Images

After the 2020 election a Twitter dashboard that I first prototyped four years before started going wild. It estimates misinformation prevalence by monitoring “the percent of retweets and likes pointing toward domains that had made a habit of sharing misinformation.” This metric had been going up throughout the election cycle from a low around 10% up to almost 20% on November 3rd. And then it jumped wildly to 30% over the next week and stayed there for almost a month. Something was likely very wrong.

Tracking this sort of change is a valuable step toward understanding the platform’s impact. It…

Image for post
Image for post
Improvement in image synthesis, from, via Ian Goodfellow

How, when, and why synthetic media can be used for harm

This is post is an excerpted section from a working paper with Jess Whittlestone (shared in 2019, but minimal updates were needed). While the full paper was focused on synthetic media research, this section is far more broadly applicable and often referenced in other contexts. This piece jumps into the meat, so for more background on this topic, see the paper overview here.

We aim to connect the dots between the theoretical potential for the malicious use (mal-use) of synthetic media technology, and what actually makes significant harm likely.

Factors Impacting the Likelihood of Harm

Below we explore the factors influencing whether a new capability overcomes…

Image for post
Image for post

Rumors on WhatsApp have led to mob violence. Here’s how to prevent them — without sacrificing privacy.

I originally published this in Bloomberg Opinion in 2019. Reprinted with permission. The opinions expressed are those of the author. For more detail addressing potential obstacles, see this follow-up post.

Last July, an engineer at Accenture was beaten to death by a crowd that thought he was a child kidnapper. They were angry, violent and completely wrong. The rumors about the man were “fake news” spread on WhatsApp, an incredibly popular messaging service owned by Facebook.

In response, the Indian government now wants to force companies to take several steps — including breaking encryption — that could compromise privacy and…

How can we fortify our “Knowledge Pipeline” in the face of synthetic media?

Close to two years ago, I started applying the framework described below in order explore ways to reduce the negative impacts of synthetic media (and other forms of misinformation). It’s not perfect, but I’ve found it useful enough to share in venues around the world, and I am continuing to expand on it with others. Using frameworks like this as a form of shared language can help us make sense of complex problems and share approaches for addressing them.

If we can’t distinguish fact from fiction, or reality from fakery, we can’t make effective decisions as society. Synthetic media technology…

Considerations and potential release practices for machine learning

Jess Whittlestone and I recently distributed a working paper exploring the challenges and options around ensuring that machine learning research is not used for harm, focusing on the challenges of synthetic media. This post is just a brief overview so read or skim the full paper here—and it was written specifically to be skimmable and referenceable! (Here is the citable arXiv link, though it might be missing some minor fixes given update delays).

Over the last few years, research advances — primarily in machine learning (ML) — have enabled the creation of increasingly convincing and realistic synthetic media: artificially generated…

Image for post
Image for post
Totally flirting for your password…

And how should that change the way we approach security and disclosure?

The technology now exists to create tools for gathering public information on people and spear-phishing them — automatically, at scale. Or creating a system that uses calls with forged realistic voices to impersonate someone. These new attack capabilities are being made possible by modern AI and may have significant implications on how we should approach security disclosure.

So what exactly is new?

Two examples: advances in AI enable conversation and impersonation

  • We can make text bots that far more realistically imitate a conversation with a human.
  • We can make human quality speech from text. We can even imitate voices of a particular person extremely well.

Image for post
Image for post

Balancing privacy, security, and mitigation of misinformation

We expand upon our recently published proposal for how end-to-end encrypted messaging systems can address misinformation using “on-device context” while preserving privacy. Misinformation spread through such systems have had — and, if unchecked, will continue to have — significant negative impacts to communities, polities, and even health. Here we go deeper into explaining the rationale, approach, and challenges of the proposal.

How it works

We propose maintaining a list of rumors (including image, audio, and video hashes) along with corresponding fact-checks, similar to what Facebook uses when identifying misinformation in its news feed. This “context list” can be regularly supplied to WhatsApp clients…

Imagine this: You work at Facebook and you have been instructed by Zuck himself to “fix news on the platform” — to do “the right thing” for users, even if it cuts into Facebook’s profits. So what do you do? And how do you measure if you’ve succeeded?

This scenario gets at much of the purpose of journalism and news consumption. To help answer it and elucidate some the implicit complexity, we propose a conceptual framework for news and news ecosystem “quality,” which can then form a basis for news ecosystem health measurement. …

Image for post
Image for post

Decentralization won’t solve our problems. Maybe multi-centralization can help?

I’ve had many conversations recently with very well meaning people who believe that if we just decentralize everything, it will fix the internet—and perhaps all of society! Decentralize social networks, decentralize money, decentralize the world…if only it was so simple.

This is the “magical decentralization fallacy” — the mistaken belief that decentralization on its own can address governance problems. What do I mean by governance problems? Things like misinformation and harassment.

I deeply wish that decentralization was a silver bullet that could “fix” this—that could ensure that misinformation wasn’t rewarded with attention and harassment wasn’t rewarded with silencing—but instead it…

I just confirmed this by analyzing the top 150,000 articles for the month

Charlie Warzel just published an interesting piece on how Facebook employees are reacting to election crisis. What I found especially interesting was this passage:

In the summer of 2015, a Facebook engineer was combing through the company’s internal data when he noticed something unusual. He was searching to determine which websites received the most referral traffic from its billion-plus users. The top 25 included the usual suspects — YouTube and the Huffington Post, along with a few obscure hyperpartisan sites he didn’t recognize. …

Aviv Ovadya

Founder of the Thoughtful Technology Project & GMF non-res fellow. Prev Tow fellow & Chief Technologist @ Center for Social Media Responsibility.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store