Designing Products for Abuse

Some forms of abuse:

  1. Spam: unsolicited commercial advertisement to your users, which could be annoying to your users, or may disrupt the way you monetize your product.
  2. Abusing product terms (EULA): primarily fraud, e.g., to receive unlimited free trials, or to use features in your free licenses to produce effectively the features only available in your paid licenses.
  3. Harassing other users: direct harassment through messages (or other forms of interaction), interfering with their use of the product, or breaching user privacy.

Some essential product features:

  1. The ability to block or mute users.
  2. A mechanism for reporting harassment or other forms of abuse.
  3. Policies and community standards.
  4. Metrics for tracking abuse.
  5. User privacy and security.
  6. Misuse cases.

Blocking or muting users:

What does it mean for Alex to block Kelly? Can Alex still access Kelly’s activity, messages, or user information? What can Kelly access about Alex? What can other users access about Alex and Kelly’s interactions?

Blocking is a single tool. Blocking protects a single person from a single user. If a user is harassed by many people, does your system offer tools for them to protect themselves? If a single user harasses many others, does your system detect and respond?

Twitter allows a user to protect their tweets, so that only the users they follow may see their tweets or @ them in tweets. This method of protection limits a user’s speech to protect them from others.

Reporting harassment or abuse:

Yes, your employees will need to devote time to handling harassment or abuse, whether they’re personally involved, evaluating a user’s report, or developing features to reduce harassment or abuse. This is an investment in your product, and valuable to users.

Policies and community standards

Does your product have a policy covering acceptable terms of use, community standards, or rules for behavior? Does your policy state consequences for violation? Is your policy enforced? Do you have C-level endorsement of your policies, including of potential consequences for violation?

Metrics for tracking abuse:

Track who is interacting with your product, when, and where they go.

User privacy and security:

What information is gathered from a user? What information can a user access about other users? How is that information protected? Do you have a plan to revisit protection of user information with an information security team on a regular basis?

Misuse cases

Use cases are essential — but have you written out a list of misuse cases to answer some of these questions?

What else?

What categories of harassment, abuse, and misuse am I missing? What strategies have I not included? What other questions should we ask?

About the author

Sarai Rosenberg is a mathematician, insecurity engineer, and queer femme woman dismantling systemic barriers in tech, one fencepost problem at a time.



Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store