Over the past two halves, Integrity teams across a platform could come into alignment on one central idea: there should be more user-facing integrity treatments.
It’s not surprising that they come to this conclusion. It’s reflected in a broader literature on safety engineering (Wogalter, 2006), in particular in the, “The Hazard Control Hierarchy.” This well-established framework is laid out below.
The real-world expectation is not that you can “design out”/remove every harm, but that you continue to evaluate how best to effectively ameliorate the potential risk. When you can’t design a harm out and when you can’t guard against it, then you need to warn against it. We owe it to the user to do so. Because, despite our efforts, “people are dissatisfied with their experiences encountering and sharing information on our platform. — They feel overwhelmed by seemingly untrustworthy information and an atmosphere of negativity.” Participants believe it’s our responsibility to help them manage it (Sterling, 2019).
When removal isn’t possible, we, as software engineers, find reasons to introduce warnings, rather than simply altering demotion strengths (but to be clear — we are not arguing that it should only be one or the other). We know from multiple research studies that users want to feel more empowered; more in control of their experience (Powell, 2020). While they want the opportunity to assess information for themselves, they also want to have the relevant information to do so (Backstrom, 2019; Sterling 2019). Further, trust in a platform increases when 1) we tell users they have encountered integrity harms, 2) when users feel informed about our integrity efforts, and 3) when we remind users of our efforts (Motyl, 2020).
Given the difficulty communicating demotions, we also face pressure to develop a more transparent and externally defensible approach to moderating problematic content. Warnings, in the form of informative interventions, address both our desire to empower users and provide transparency on our efforts. By helping users identify the bad and learn more about the relevant pieces of information to pay attention to through warnings, we can create a product that users find helpful and valuable, while simultaneously reducing integrity harms, and increasing trust.
“Warnings are safety communications used to inform people about hazards so that undesirable consequences are avoided or minimized.” — The Handbook on Warnings.
Here are four main purposes of a warning (Wogalter, 2006):
- Warnings communicate important safety information.
- Warnings influence or modify behavior in ways that improve safety.
- Warnings are intended to reduce harms.
- Warnings can be a reminder to those who already know about a particular harm.
Warnings come in all shapes and sizes: They can be as simple as a warning about a wet floor, where no additional information is needed to understand the risk (falling). They can also be as complex as warnings about potential side effects from a prescription drug. They also can be active, like our “bullying comment” interstitial outlined below or more passive, like the search bar warnings on a suspicious website.
To expand, “an effective physical warning clearly communicates risk, consequences of not complying, and instructions to comply (although some of this information can be omitted if the risk is obvious or the consequences can be deduced from the warning)”.
2. Using Artificial Intelligence to detect COVID-19
3. Real vs Fake Tweet Detection using a BERT Transformer Model in few lines of code
4. Machine Learning System Design
An added complexity: digital risks are often hard to understand. A sign warning of a broken sidewalk is easy to understand. The risk is “open and obvious” . A sign about a potential financial scam, less so. Users are often not familiar with our terminology or complex technical “jargon” such as phrases like “domain age,” they do not have the complete context, and cannot “see” the potential risk in the same way you could see a broken sidewalk. This is, of course, sometimes by design — scams work because they look like websites and people you trust. On a platform, the risk is even more complex as the scam, or other harm, may be posted by a friend or Page you trust, reducing the level of skepticism you approach the situation with to start.
This complexity puts the onus on us for how to best communicate the potential risk. As Choice Architects (those who design the way choices are presented to users), our job is to help people overcome cognitive biases, make decisions more carefully, and “nudge” them to the decision they would make if they had complete information. We know that even a small nudge to be more discerning can change outcomes .
Further, when writing about usability and trust in download warnings, Krol, et al. (2012) conclude that to keep user trust in warnings high, the only case where a warning can be justified is “ where genuine danger has been detected within, but there is still a chance that the file in question is not malicious.” For many of the opportunities for warnings identified by development teams at a platform, this is the case. There is a disproportionate amount of bad (misinformation, hate, etc) detected within, but not all content in that bucket is bad.
Taken together, it becomes evident that warnings are a useful tool in our fight to reduce integrity harms on-platform, while providing users with information to help empower them to make more informed decisions in the future.
So what might an effective warning design look like on a platform?
How we can do this effectively, while not being overly prescriptive, judgemental, or annoying is tantamount to our success. To that end, there are several factors we should consider in this space, such as cognitive biases that influence decision-making.
First, we know adding visual information and complexity to features is sometimes fundamental to the success and safety of our platform, as it increases comprehension. Clean and simple aesthetics may look visually appealing, but for some users these hidden features or missing context can lead to errors in judgement and understanding.
Speaking on the topic of share interstitials, Zyskowski and Levin (2019) conclude, ”design decisions like this, which add visual complexity and friction, can be important means for increasing the comprehension of posts, promoting healthy skepticism of news information, and giving stronger signals when information is lower quality.” We also know from the literature that active warnings, or warnings that disrupt your intended flow, are more effective than passive warnings in altering behaviors and/or providing relevant information.
Transparency is not useful if, in its delivery, it requires too much context and education. Worse, presenting raw data without context and education can lead to incorrect and highly problematic assumptions.By providing additional cues or additional context, explaining why we believe you should be cautious, we can correct these problematic assumptions, and should be included in our design frameworks in this space.
Warnings are not without their own problems. There is strong user and company value in introducing warnings regardless: The security engineering literature argues that “even in situations where warning effectiveness might not be high, people have the right to be informed about safety issues”. This perspective aligns well with our intentions here. As we should not be seeking full compliance, yet to empower by providing relevant and helpful information to users, it is reasonable to proceed despite concerns about habituation.
We are concerned about warning fatigue and habituation and want to ensure we are appearing when necessary and in a way that is accessible and understandable to users.
It is necessary to innovate and update engineers’ approach to integrity harms if platforms hope to continue to reduce exposure to harmful content. Remove and reduce only help address part of the problem and are not always fully understood by the public. Additionally, in their current form they are not empowering for users. Users want to be informed about what we know, but they also want the space to make their own decisions.
Warnings provide an opportunity for both. We have often argued that we cannot fight integrity issues alone — this is true. Yet, we do not only need regulators and other external bodies; users are vital to our efforts toward reducing low-quality content on-platform, they just need to understand when and where these risks are.
Ideally, these warnings will eventually trigger less over time as users come to understand the risks associated with certain pieces of information and become more diligent themselves. Regardless of if that happens, users deserve to be presented with relevant information about what they intend to post and share and make judgments accordingly.