An AI security audit is not a renamed penetration test. It covers a distinct set of risks — model manipulation, training data poisoning, inference attacks, third-party model dependencies — that standard IT security assessments were never designed to surface. If your organization deploys AI systems in regulated environments, understanding what auditors actually examine, and what evidence they expect, is the difference between a clean audit cycle and a remediation sprint you did not budget for. The sections below cover the scope of an AI security audit, the specific controls auditors evaluate, how adversarial testing works as a distinct workstream, what supply chain risks look like in practice, and how AI-specific risks map to SOC 2 Trust Service Criteria.
What an AI Security Audit Actually Covers
A traditional IT security audit focuses on infrastructure, access controls, patch management, and network perimeter. An AI security audit starts there and then goes further — into the model itself, the data it was trained on, the pipeline that feeds it, and the decisions it produces. The scope of an AI model security assessment typically spans four layers:
- Model layer — How the model was trained, what data it ingested, whether it can be manipulated through adversarial inputs, and whether its outputs can be reverse-engineered to expose training data.
- Infrastructure layer — Where the model runs, how inference endpoints are secured, and whether model weights are protected at rest and in transit.
- Data layer — Provenance and integrity of training data, access controls on datasets, and whether sensitive data was used in training without appropriate safeguards.
- Operational layer — Monitoring for model drift, anomaly detection on inference traffic, and incident response procedures specific to AI failures.
The AI risk assessment framework that auditors apply varies by regulatory context. NIST’s AI Risk Management Framework (AI RMF) organizes risk across four functions — Map, Measure, Manage, Govern — and is increasingly referenced by auditors as a baseline structure even when it is not the primary compliance requirement. ISO 42001 takes a similar lifecycle approach. Both treat security as one dimension of a broader governance posture, which is why an AI security audit rarely happens in isolation from governance work. The key distinction from a traditional audit: the attack surface is probabilistic, not deterministic. A misconfigured firewall either allows traffic or it does not. A model under adversarial pressure may behave correctly 99% of the time and fail in ways that are difficult to predict or detect. That probabilistic failure mode is what makes AI security audits structurally different — and more demanding to evidence. For a broader view of what auditors expect across the full governance stack, see AI Model Documentation and Transparency: What Auditors Want to See.
Core Security Controls for AI Systems
When auditors evaluate AI system security controls, they are looking for evidence that controls exist, that they are documented, and that they are actually operating. Assertions without artifacts do not pass. The control categories that appear most consistently across AI security assessments:
- Access and authentication controls
- Role-based access to model training environments, feature stores, and inference APIs
- Separation of duties between teams that develop models and teams that deploy them
- Audit logs covering who accessed model weights, training data, and configuration files
- Data integrity controls
- Checksums or cryptographic signatures on training datasets to detect tampering
- Version control for datasets, with documented lineage from source to model
- Controls preventing unauthorized modification of labeled data
- Model integrity controls
- Signed model artifacts to verify that deployed models match approved versions
- Change management procedures covering model updates and rollbacks
- Baseline performance benchmarks that trigger review if outputs drift outside defined thresholds
- Inference security controls
- Rate limiting and anomaly detection on inference endpoints to flag potential extraction attacks
- Output filtering for models that could produce sensitive or regulated content
- Logging of inference requests at a level sufficient to reconstruct what inputs produced what outputs
- Monitoring and incident response
- Continuous monitoring for model performance degradation, which can indicate data drift or adversarial interference
- Defined escalation paths for AI-specific incidents (model failure, data breach via inference, supply chain compromise)
- Documented recovery procedures, including model rollback and retraining protocols
An AI model robustness audit will probe whether these controls hold under realistic stress conditions — not just whether the policy documents say they should exist. The gap between documented controls and operating controls is where most findings land. For a structured checklist approach to preparing these controls before an audit engagement, the AI Audit Readiness: The Complete Checklist for Regulated Organizations covers the full evidence inventory.
Adversarial Testing and Model Robustness Assessment
Adversarial AI testing is a distinct workstream within an AI security audit. It is not a standard vulnerability scan run against the model’s hosting infrastructure — it is a structured attempt to make the model fail in security-relevant ways. The primary attack categories that adversarial testing covers:
- Evasion attacks — Inputs crafted to cause the model to misclassify or produce incorrect outputs. In a fraud detection model, this might mean manipulating transaction features to evade detection. In a content moderation model, it might mean encoding prohibited content in ways the model does not recognize.
- Model inversion and extraction attacks — Queries designed to reconstruct training data or replicate model behavior. Inversion attacks can expose PII or proprietary data that was present in training sets. Extraction attacks can allow a competitor or adversary to build a functional copy of a model by querying its API at scale.
- Poisoning attacks — Attacks on the training pipeline rather than the deployed model. If an adversary can influence what data the model trains on, they can embed backdoors that activate under specific conditions. This is particularly relevant for models that incorporate user-generated data or continuously retrain on production traffic.
- Prompt injection — Specific to large language models and other generative systems. Crafted inputs that override system instructions, cause the model to ignore safety constraints, or exfiltrate context window contents.
An AI model robustness audit documents which attack categories were tested, what test cases were used, what failure modes were observed, and what mitigations were applied or recommended. The output is not a pass/fail score — it is a risk-ranked finding set with remediation guidance. Auditors increasingly expect to see evidence that adversarial testing has been conducted as part of the model development lifecycle, not just as a one-time pre-deployment exercise. Models that retrain on live data need ongoing robustness assessment because the attack surface changes as the model changes. The AI risk assessment framework used to scope adversarial testing should be documented and defensible. NIST AI RMF’s "Measure" function provides a reasonable structure for this, specifying that AI risks should be analyzed, prioritized, and tracked with quantitative or qualitative metrics where feasible.
AI Supply Chain Security: Third-Party Models and Data Pipelines
Most production AI systems are not built from scratch. They incorporate pre-trained foundation models, third-party APIs, open-source libraries, and training datasets assembled from external sources. Each of those dependencies is a potential attack vector — and AI supply chain security is now a formal audit workstream. The specific risks auditors probe in this area:
Pre-trained model provenance
When an organization fine-tunes a foundation model from a public repository or commercial provider, auditors want to know: who trained the base model, on what data, under what security controls? A model downloaded from a public repository without provenance verification could contain embedded backdoors or have been trained on data that creates legal or regulatory exposure.
Third-party API dependencies
Organizations that call external model APIs (for inference, embeddings, or classification) inherit the security posture of that provider. Auditors will ask whether vendor security assessments have been conducted, whether data sent to external APIs is subject to appropriate data handling agreements, and what the fallback posture is if the API becomes unavailable or is compromised.
Training data provenance
Data pipelines that aggregate training data from multiple sources — web scrapes, licensed datasets, internal records — need documented lineage. Auditors look for evidence that data sources were vetted, that consent or licensing requirements were met, and that data integrity was maintained through the pipeline. Poisoned or mislabeled training data is a supply chain risk, not just a data quality issue.
Open-source library risks
The ML toolchain — frameworks, preprocessing libraries, experiment tracking tools — carries the same software supply chain risks as any other software dependency. Dependency scanning, software bill of materials (SBOM) documentation, and patch management procedures should cover ML-specific libraries, not just application-layer dependencies. AI system security controls applied to the supply chain should include vendor risk assessments with AI-specific questionnaires, contractual requirements around model provenance and data handling, and periodic re-evaluation as vendor relationships evolve. For the data governance dimension of supply chain risk — particularly where training data intersects with privacy regulations — AI Data Governance, Privacy, and GDPR Compliance for AI Systems covers the compliance requirements in detail.
SOC 2 Considerations for AI Systems
SOC 2 audits evaluate controls against the AICPA’s Trust Service Criteria (TSC). The five criteria — Security, Availability, Processing Integrity, Confidentiality, and Privacy — were written for general technology systems, but each maps to specific AI risks that auditors are increasingly expected to address.
Security (CC series)
The common criteria covering logical access, change management, and risk assessment apply directly to AI systems. Auditors will look for evidence that access to model training environments and inference APIs is controlled (CC6), that changes to models go through a documented change management process (CC8), and that the organization has identified AI-specific risks as part of its risk assessment program (CC3).
Availability (A series)
For AI systems that support critical business processes, availability criteria require evidence that the system meets defined uptime commitments and that there are documented procedures for handling model failures or degraded performance. This includes rollback procedures and the ability to fall back to non-AI processes if the model is unavailable.
Processing Integrity (PI series)
This is where SOC 2 AI systems face the most novel scrutiny. Processing integrity criteria require that system processing is complete, valid, accurate, timely, and authorized. For AI systems, this means demonstrating that model outputs are monitored for accuracy, that there are controls to detect when the model is producing anomalous results, and that there is a defined process for handling incorrect outputs.
Confidentiality (C series)
If the model processes confidential information — customer data, proprietary business data, regulated data — auditors will evaluate whether that data is protected during training, inference, and logging. This includes controls against model inversion attacks that could expose training data through inference queries.
Privacy (P series)
Where personal data is used in training or processed during inference, the privacy criteria apply. This overlaps significantly with GDPR and CCPA requirements and requires documented data handling practices, consent management where applicable, and data subject rights procedures. The practical implication for compliance teams: SOC 2 does not have AI-specific controls enumerated in the criteria. Auditors apply the existing TSC to AI systems, which means the evidence burden falls on the organization to demonstrate how its AI-specific controls satisfy criteria that were written without AI in mind. Organizations that have mapped their AI system security controls to specific TSC criteria before the audit engagement are in a materially better position than those that try to make the mapping during fieldwork. For a broader comparison of how different frameworks — including NIST AI RMF and ISO 42001 — approach AI governance requirements, AI Governance Frameworks Compared: ISO 42001, NIST AI RMF, and EU AI Act provides a side-by-side analysis.
An AI security audit surfaces findings that most organizations were not tracking — not because the risks were hidden, but because the internal processes for identifying and documenting AI-specific controls had not been built yet. The organizations that move through audit cycles efficiently are the ones that treat control documentation as a continuous practice rather than a pre-audit scramble. Start by inventorying where your current AI security controls are documented, tested, and evidenced — and where they exist only as informal practice. Gaps identified internally cost far less to close than gaps identified during fieldwork. For the full governance context that AI security audits sit within, the AI Governance Audit Readiness pillar covers the complete picture — from framework selection to board reporting to audit evidence management. If you want to assess your current AI security posture against what auditors actually examine, schedule a readiness review before your next SOC 2 cycle begins — not after the auditor’s opening meeting.