AI trust and safety vs AI security

AI security protects the system and its data from attackers: prompt injection that leaks a user's files, stolen model weights, an agent tricked into sending email. AI trust and safety, which frontier labs often call safeguards, protects the world from people using the system as intended but for harm: generating malware, running influence operations, fraud, weapons uplift, abuse of other users.

The difference is who the adversary is and what they want. A security attacker wants the system to do something its owner never allowed. A misuse actor usually wants the model's ordinary capability, aimed at a target the policy forbids. Microsoft's AI Red Team, in its January 2025 paper on red teaming over 100 generative AI products, draws the same line: security harms such as data exfiltration and credential dumping on one side, responsible AI harms such as harmful content on the other, and notes the second kind is harder to measure because it is more subjective.

The two disciplines share methods. Both red-team the model, both build classifiers and detection pipelines, both investigate incidents from logs, and both measure success rates rather than trusting single examples. Jailbreaks sit in the middle: a jailbreak is a safety failure when it unlocks harmful content and a security failure when it unlocks another user's data or a tool the attacker should not reach.

Employers reflect the overlap. Anthropic's Red Team Engineer role sits on its Safeguards team and asks for penetration testing, application security and agentic prompt injection testing. A now-closed Google posting for trust and safety on Gemini was titled Security Engineering Manager and asked for years in both security engineering and abuse detection. The organisations stay separate, though: at Anthropic, threat intelligence on misuse of its models sits in Safeguards, while threat intelligence on attacks against the company sits in Security.

Engineering: safeguards, safety systems and integrity engineers build abuse detection, classifiers, enforcement tooling and human review systems; fraud and account-abuse engineers work on payment and signup abuse; red team engineers attack the products to find gaps in all of it.

Investigation and analysis: enforcement analysts review flagged activity in one harm area, such as cyber, influence operations, weapons or child safety, and turn it into enforcement decisions and better detection; threat investigators track actors misusing the models and write intelligence reports; data scientists define safety metrics and run experiments. Anthropic's postings for these roles, read on 4 November 2026, ask for SQL and Python in every case.

Policy and programme: policy designers write and revise the usage policy, and incident managers run response when a serious misuse case breaks.

Pay at frontier labs is in the same band as security engineering. Anthropic's Safeguards postings read on 4 November 2026 listed, as annual base in US dollars: $320,000 to $485,000 for Staff+ Software Engineer, Safeguards; $320,000 to $405,000 for Red Team Engineer, Safeguards; $285,000 to $380,000 for Data Scientist, Safeguards; $285,000 to $330,000 for Safeguards Enforcement Analyst, Integrity and Authenticity; and $230,000 to $290,000 for Technical Cyber Threat Investigator. Ranges span levels, equity is separate, and other employers pay differently.

No AI company publishes a trust and safety interview guide, so treat any precise round list with suspicion. What the postings make clear is the skill set. Engineers are tested on system design for detection: where signals come from, how a classifier's threshold trades precision against recall when human reviewers have a fixed capacity, how labels arrive late, and how adversaries adapt. Analysts and investigators are tested on data work in SQL and Python, on case studies that start from a suspicious pattern and end with an enforcement decision, and on judgment where policy is ambiguous. Red team roles test attack methodology and measurement and ask for a public body of work.

Anthropic states that live interviews are done without AI assistance unless the interviewer says otherwise, which applies to these loops as much as to engineering ones.

Read the primary material these teams publish: Anthropic's threat intelligence reports on detecting and countering misuse, OpenAI's reports on disrupting malicious uses of AI, and the Google Threat Intelligence Group's posts on adversarial misuse of generative AI. They show the actual cases teams handle and how they describe them.

Then build the technical base that both tracks share: how jailbreaks and prompt injection differ, how guardrails are layered on input and output, and how to judge a detector by its precision, recall and base rate rather than by a single headline number. The curriculum here covers those in the prompt injection, LLM application security and red teaming topics. Practise SQL on event data until a funnel or cohort query is routine.