Before building with Claude

Decide whether to use Claude for content moderation

Here are some key indicators that you should use an LLM like Claude instead of a traditional ML or rules-based approach for content moderation:

Traditional ML methods require significant engineering resources, ML expertise, and infrastructure costs. Human moderation systems incur even higher costs. With Claude, you can have a sophisticated moderation system operational in significantly less time and at a much lower cost.

Traditional ML approaches, such as bag-of-words models or simple pattern matching, often struggle to understand the tone, intent, and context of the content. While human moderation systems excel at understanding semantic meaning, they require time for content to be reviewed. Claude addresses both needs by combining semantic understanding with the ability to deliver moderation decisions quickly.

By leveraging its advanced reasoning capabilities, Claude can interpret and apply complex moderation guidelines uniformly. This consistency helps ensure fair treatment of all content, reducing the risk of inconsistent or biased moderation decisions that can undermine user trust.

Once a traditional ML approach has been established, changing it is a laborious and data-intensive undertaking. On the other hand, as your product or customer needs evolve, Claude can easily adapt to changes or additions to moderation policies without extensive relabeling of training data.

If you want to provide users or regulators with clear explanations behind moderation decisions, Claude can generate detailed and coherent justifications. This transparency is important for building trust and ensuring accountability in content moderation practices.

Traditional ML approaches typically require separate models or extensive translation processes for each supported language. Human moderation requires hiring a workforce fluent in each supported language. Claude’s multilingual capabilities allow it to classify tickets in various languages without the need for separate models or extensive translation processes, streamlining moderation for global customer bases.

Claude's multimodal capabilities allow it to analyze and interpret content across both text and images. This makes it a versatile tool for comprehensive content moderation in environments where different media types need to be evaluated together.

All Claude models are trained with built-in safety behaviors. This may result in Claude moderating content deemed particularly dangerous (in line with the

Acceptable Use Policy

), regardless of the prompt used. For example, an adult website that wants to allow users to post explicit sexual content may find that Claude still flags explicit content as requiring moderation, even if they specify in their prompt not to moderate explicit sexual content. Consider reviewing the AUP in advance of building a moderation solution.

Generate examples of content to moderate

Before developing a content moderation solution, first create examples of content that should be flagged and content that should not be flagged. Ensure that you include edge cases and challenging scenarios that may be difficult for a content moderation system to handle effectively. Afterward, review your examples to create a well-defined list of moderation categories. For instance, the examples generated by a social media platform might include the following:

client = anthropic.Anthropic()

allowed_user_comments = [
    "This movie was great, I really enjoyed it. The main actor really killed it!",
    "I hate Mondays.",
    "It is a great time to invest in gold!",
]

disallowed_user_comments = [
    "Delete this post now or you better hide. I am coming after you and your family.",
    "Stay away from the 5G cellphones!! They are using 5G to control you.",
    "Congratulations! You have won a $1,000 gift card. Click here to claim your prize!",
]

# Sample user comments to test the content moderation
user_comments = allowed_user_comments + disallowed_user_comments

# Categories considered unsafe for content moderation
unsafe_categories = [
    "Child Exploitation",
    "Conspiracy Theories",
    "Hate",
    "Indiscriminate Weapons",
    "Intellectual Property",
    "Non-Violent Crimes",
    "Privacy",
    "Self-Harm",
    "Sex Crimes",
    "Sexual Content",
    "Specialized Advice",
    "Violent Crimes",
]

const client = new Anthropic();

const allowedUserComments = [
  "This movie was great, I really enjoyed it. The main actor really killed it!",
  "I hate Mondays.",
  "It is a great time to invest in gold!"
];

const disallowedUserComments = [
  "Delete this post now or you better hide. I am coming after you and your family.",
  "Stay away from the 5G cellphones!! They are using 5G to control you.",
  "Congratulations! You have won a $1,000 gift card. Click here to claim your prize!"
];

// Sample user comments to test the content moderation
const userComments = [...allowedUserComments, ...disallowedUserComments];

// Categories considered unsafe for content moderation
const unsafeCategories = [
  "Child Exploitation",
  "Conspiracy Theories",
  "Hate",
  "Indiscriminate Weapons",
  "Intellectual Property",
  "Non-Violent Crimes",
  "Privacy",
  "Self-Harm",
  "Sex Crimes",
  "Sexual Content",
  "Specialized Advice",
  "Violent Crimes"
];

var client = new AnthropicClient();

string[] allowedUserComments =
[
    "This movie was great, I really enjoyed it. The main actor really killed it!",
    "I hate Mondays.",
    "It is a great time to invest in gold!",
];

string[] disallowedUserComments =
[
    "Delete this post now or you better hide. I am coming after you and your family.",
    "Stay away from the 5G cellphones!! They are using 5G to control you.",
    "Congratulations! You have won a $1,000 gift card. Click here to claim your prize!",
];

// Sample user comments to test the content moderation
string[] userComments = [.. allowedUserComments, .. disallowedUserComments];

// Categories considered unsafe for content moderation
string[] unsafeCategories =
[
    "Child Exploitation",
    "Conspiracy Theories",
    "Hate",
    "Indiscriminate Weapons",
    "Intellectual Property",
    "Non-Violent Crimes",
    "Privacy",
    "Self-Harm",
    "Sex Crimes",
    "Sexual Content",
    "Specialized Advice",
    "Violent Crimes",
];

var client = anthropic.NewClient()

var allowedUserComments = []string{
	"This movie was great, I really enjoyed it. The main actor really killed it!",
	"I hate Mondays.",
	"It is a great time to invest in gold!",
}

var disallowedUserComments = []string{
	"Delete this post now or you better hide. I am coming after you and your family.",
	"Stay away from the 5G cellphones!! They are using 5G to control you.",
	"Congratulations! You have won a $1,000 gift card. Click here to claim your prize!",
}

// Sample user comments to test the content moderation
var userComments = slices.Concat(allowedUserComments, disallowedUserComments)

// Categories considered unsafe for content moderation
var unsafeCategories = []string{
	"Child Exploitation",
	"Conspiracy Theories",
	"Hate",
	"Indiscriminate Weapons",
	"Intellectual Property",
	"Non-Violent Crimes",
	"Privacy",
	"Self-Harm",
	"Sex Crimes",
	"Sexual Content",
	"Specialized Advice",
	"Violent Crimes",
}

final AnthropicClient client = AnthropicOkHttpClient.fromEnv();

final List<String> allowedUserComments = List.of(
        "This movie was great, I really enjoyed it. The main actor really killed it!",
        "I hate Mondays.",
        "It is a great time to invest in gold!");

final List<String> disallowedUserComments = List.of(
        "Delete this post now or you better hide. I am coming after you and your family.",
        "Stay away from the 5G cellphones!! They are using 5G to control you.",
        "Congratulations! You have won a $1,000 gift card. Click here to claim your prize!");

// Sample user comments to test the content moderation
final List<String> userComments =
        Stream.concat(allowedUserComments.stream(), disallowedUserComments.stream()).toList();

// Categories considered unsafe for content moderation
final List<String> unsafeCategories = List.of(
        "Child Exploitation",
        "Conspiracy Theories",
        "Hate",
        "Indiscriminate Weapons",
        "Intellectual Property",
        "Non-Violent Crimes",
        "Privacy",
        "Self-Harm",
        "Sex Crimes",
        "Sexual Content",
        "Specialized Advice",
        "Violent Crimes");

$client = new Client();

$allowedUserComments = [
    'This movie was great, I really enjoyed it. The main actor really killed it!',
    'I hate Mondays.',
    'It is a great time to invest in gold!',
];

$disallowedUserComments = [
    'Delete this post now or you better hide. I am coming after you and your family.',
    'Stay away from the 5G cellphones!! They are using 5G to control you.',
    'Congratulations! You have won a $1,000 gift card. Click here to claim your prize!',
];

// Sample user comments to test the content moderation
$userComments = [...$allowedUserComments, ...$disallowedUserComments];

// Categories considered unsafe for content moderation
$unsafeCategories = [
    'Child Exploitation',
    'Conspiracy Theories',
    'Hate',
    'Indiscriminate Weapons',
    'Intellectual Property',
    'Non-Violent Crimes',
    'Privacy',
    'Self-Harm',
    'Sex Crimes',
    'Sexual Content',
    'Specialized Advice',
    'Violent Crimes',
];

CLIENT = Anthropic::Client.new

ALLOWED_USER_COMMENTS = [
  "This movie was great, I really enjoyed it. The main actor really killed it!",
  "I hate Mondays.",
  "It is a great time to invest in gold!"
]

DISALLOWED_USER_COMMENTS = [
  "Delete this post now or you better hide. I am coming after you and your family.",
  "Stay away from the 5G cellphones!! They are using 5G to control you.",
  "Congratulations! You have won a $1,000 gift card. Click here to claim your prize!"
]

# Sample user comments to test the content moderation
USER_COMMENTS = ALLOWED_USER_COMMENTS + DISALLOWED_USER_COMMENTS

# Categories considered unsafe for content moderation
UNSAFE_CATEGORIES = [
  "Child Exploitation",
  "Conspiracy Theories",
  "Hate",
  "Indiscriminate Weapons",
  "Intellectual Property",
  "Non-Violent Crimes",
  "Privacy",
  "Self-Harm",
  "Sex Crimes",
  "Sexual Content",
  "Specialized Advice",
  "Violent Crimes"
]

Effectively moderating these examples requires a nuanced understanding of language. In the comment, This movie was great, I really enjoyed it. The main actor really killed it!, the content moderation system needs to recognize that "killed it" is a metaphor, not an indication of actual violence. Conversely, despite the lack of explicit mentions of violence, the comment Delete this post now or you better hide. I am coming after you and your family. should be flagged by the content moderation system.

The unsafe categories can be customized to fit your specific needs. For example, if you want to prevent minors from creating content on your website, you could add "Underage Posting" to the categories.

---