diff --git a/Document/content/2.2_Appendix_E.md b/Document/content/2.2_Appendix_E.md index f8e0e02..9f76d4d 100644 --- a/Document/content/2.2_Appendix_E.md +++ b/Document/content/2.2_Appendix_E.md @@ -101,68 +101,52 @@ Note: Component identifiers correspond to the SAIF numbering scheme illustrated ## (2) User Input -**Summary:** User Input is the front door of the system — every downstream component depends on it. Without strong input validation, filtering, and limits, it becomes the main vector for prompt injection, data leakage, DoS, and toxicity propagation. +**Summary:** User Input is the front door of the system, every downstream component depends on it. Without strong input validation, filtering, and limits, it becomes the main vector for prompt injection, data leakage, DoS, and toxicity propagation. -**Threats:** T01-DPIJ, T01-IPI J, T01-SID, T01-DoSM, T01-IOH, T01-MTU +#### Direct Prompt Injection (T01-DPIJ) & Indirect Prompt Injection (T01-IPIJ) -**Targeted CWEs:** -CWE-20, CWE-74, CWE-94, CWE-707, CWE-200, CWE-359, CWE-522, CWE-400, CWE-770, CWE-787, CWE-116, CWE-79 +**Mapped CWEs:** [CWE-20](https://cwe.mitre.org/data/definitions/20.html), [CWE-74](https://cwe.mitre.org/data/definitions/74.html), [CWE-94](https://cwe.mitre.org/data/definitions/94.html), [CWE-707](https://cwe.mitre.org/data/definitions/707.html) -### Direct Prompt Injection (T01-DPIJ) & Indirect Prompt Injection (T01-IPIJ) - -**Mapped CWEs:** CWE-20, CWE-74, CWE-94, CWE-707 - -**Rationale:** Maliciously crafted inputs (user prompts or embedded instructions) can override instructions or trigger unintended actions. +**Rationale:** Maliciously crafted inputs (user prompts or embedded instructions) can override instructions, alter reasoning chains, or trigger unintended actions in connected tools. **Recommendations:** - Apply strict input validation and canonicalization before passing content to the model. -- Use prompt isolation/sandboxing (separate user and system instructions). -- Enforce allowlist-based instruction patterns. -- Test with adversarial prompt fuzzing. +- Use prompt isolation or sandboxing (separate user and system instructions). +- Enforce allowlist-based instruction and function patterns. +- Perform adversarial prompt fuzzing and red-team testing. -### Sensitive Information Disclosure (T01-SID) +#### Denial of Service – Model (T01-DoSM) -**Mapped CWEs:** CWE-200, CWE-359, CWE-522 +**Mapped CWEs:** [CWE-400](https://cwe.mitre.org/data/definitions/400.html), [CWE-770](https://cwe.mitre.org/data/definitions/770.html), [CWE-787](https://cwe.mitre.org/data/definitions/787.html) -**Rationale:** Inputs may include secrets/PII that can be reflected in outputs or logs. +**Rationale:** Oversized, malformed, or adversarial inputs can exhaust tokenization, GPU, or compute capacity, leading to degraded performance or service unavailability. **Recommendations:** +- Enforce maximum input size and token limits. +- Apply rate-limits and per-user quotas at API gateways. +- Use circuit breakers and autoscaling to mitigate load spikes. -- Integrate DLP filters into input channels. -- Mask/tokenize secrets and PII before forwarding to the model. -- Restrict logging of raw inputs. +#### Insecure Output Handling Triggered by Inputs (T01-IOH) -### Denial of Service – Model (T01-DoSM) +**Mapped CWEs:** [CWE-116](https://cwe.mitre.org/data/definitions/116.html), [CWE-79](https://cwe.mitre.org/data/definitions/79.html) -**Mapped CWEs:** CWE-400, CWE-770, CWE-787 - -**Rationale:** Oversized or adversarial inputs can exhaust tokens/compute. +**Rationale:** Malicious inputs may propagate into rendered outputs (e.g., HTML, Markdown, or JSON), enabling injection or cross-site scripting attacks. **Recommendations:** +- Sanitize and contextually encode all rendered outputs. +- Separate data from control characters; use safe templating and rendering frameworks. +- Enforce strict content-type validation before presentation. -- Set input size and tokenization limits. -- Apply rate-limits and per-user quotas. -- Use circuit breakers/autoscaling. +#### Model Toxicity / Unreliable Outputs (T01-MTU) -### Insecure Output Handling Triggered by Inputs (T01-IOH) +**Mapped CWEs:** [CWE-707](https://cwe.mitre.org/data/definitions/707.html), [CWE-345](https://cwe.mitre.org/data/definitions/345.html), [CWE-1204](https://cwe.mitre.org/data/definitions/1204.html) -**Mapped CWEs:** CWE-116, CWE-79 - -**Rationale:** Malicious inputs may propagate to rendered outputs (e.g., XSS). +**Rationale:** Crafted or provocative user inputs can bias model behavior, steering it toward toxic, discriminatory, or ungrounded responses. **Recommendations:** -- Sanitize and encode outputs by context (HTML/MD/JSON). -- Separate data from control characters; use safe rendering frameworks. - -### Model Toxicity / Unreliable Outputs (T01-MTU) - -**Mapped CWEs:** CWE-707, CWE-345, CWE-1204 - -**Rationale:** Inputs can steer models toward toxic or unreliable content. - -**Recommendations:** -- Add toxicity/bias classifiers and context filters. -- Escalate high-risk cases to human review. +- Integrate toxicity and bias classifiers to pre-screen user prompts. +- Use contextual and sentiment filters on incoming requests. +- Escalate high-risk or policy-violating cases to human review workflows. --- @@ -170,102 +154,127 @@ CWE-20, CWE-74, CWE-94, CWE-707, CWE-200, CWE-359, CWE-522, CWE-400, CWE-770, CW **Summary:** The last mile to users/connected systems; without control, it’s a vector for excessive agency, prompt leakage, misinformation, and unsafe rendering. -**Threats:** T01-EA, T01-SPL, T01-MIS, T01-IOH +#### Excessive Agency (T01-EA) -**Targeted CWEs:** -CWE-284, CWE-285, CWE-200, CWE-209, CWE-359, CWE-532, CWE-116, CWE-79, CWE-75, CWE-345, CWE-1204 +**Mapped CWEs:** [CWE-284](https://cwe.mitre.org/data/definitions/284.html), [CWE-285](https://cwe.mitre.org/data/definitions/285.html) -### Excessive Agency (T01-EA) - -**Mapped CWEs:** CWE-284, CWE-285 - -**Rationale:** Action-bearing outputs can trigger privileged operations without proper scoping. +**Rationale:** Action-bearing model outputs (e.g., generated commands, API calls, workflow triggers) can execute privileged or irreversible operations without authorization or user oversight. **Recommendations:** -- Enforce least-privilege scopes for action outputs. -- Require policy checks before rendering actionable UI. -- Use allowlists and out-of-band approvals for high-risk actions. +- Enforce least-privilege scopes for all actionable outputs. +- Apply policy and authorization checks before rendering or executing UI-driven actions. +- Maintain allowlists and require explicit human approvals for high-impact or sensitive actions. -### Sensitive Prompt Leakage (T01-SPL) +#### Sensitive Prompt Leakage (T01-SPL) -**Mapped CWEs:** CWE-200, CWE-209, CWE-359, CWE-532 +**Mapped CWEs:** [CWE-200](https://cwe.mitre.org/data/definitions/200.html), [CWE-209](https://cwe.mitre.org/data/definitions/209.html), [CWE-359](https://cwe.mitre.org/data/definitions/359.html), [CWE-532](https://cwe.mitre.org/data/definitions/532.html) -**Rationale:** Hidden prompts/keys/PII can surface in responses, errors, or logs. -**Recommendations:** -- Redact secrets/PII/system instructions before render/logging. -- Wrap errors safely; never show raw tool/model errors. -- Separate user-visible and operator logs with DLP. - -### Misinformation (T01-MIS) - -**Mapped CWEs:** CWE-345, CWE-1204 - -**Rationale:** Ungrounded claims appear credible in UI. +**Rationale:** Model outputs, error messages, or logs may inadvertently reveal hidden prompts, credentials, API keys, or personal information embedded in the conversation context. **Recommendations:** -- Require grounding/citations for high-risk claims. -- Add verification metrics and “needs review” flags. +- Redact secrets, PII, and system instructions prior to rendering or logging. +- Use structured error wrappers; never expose raw stack traces or backend errors. +- Segregate user-visible and operator logs; apply DLP scanning to prevent prompt or secret leakage. -### Insecure Output Handling (T01-IOH) +#### Misinformation (T01-MIS) -**Mapped CWEs:** CWE-116, CWE-79, CWE-75 +**Mapped CWEs:** [CWE-345](https://cwe.mitre.org/data/definitions/345.html), [CWE-1204](https://cwe.mitre.org/data/definitions/1204.html) -**Rationale:** Unsanitized text can execute in rich renderers. +**Rationale:** Ungrounded, fabricated, or biased statements can appear credible when presented in the UI, eroding user trust or propagating false information. **Recommendations:** -- Render from structured formats; encode per context. -- Sanitize Markdown/HTML via allowlists; disable unsafe embeds. +- Require grounding and citation checks for high-risk or factual claims. +- Integrate verification confidence scores and “needs review” flags for uncertain responses. +- Route flagged outputs to human review or moderation pipelines. + +#### Insecure Output Handling (T01-IOH) + +**Mapped CWEs:** [CWE-116](https://cwe.mitre.org/data/definitions/116.html), [CWE-79](https://cwe.mitre.org/data/definitions/79.html), [CWE-75](https://cwe.mitre.org/data/definitions/75.html) + +**Rationale:** Unsanitized model outputs rendered in rich text, HTML, or Markdown can lead to script execution, injection, or UI manipulation in downstream clients. + +**Recommendations:** +- Render outputs from structured formats (e.g., JSON, plain text) with context-aware encoding. +- Sanitize HTML/Markdown through allowlisted elements and attributes. +- Disable unsafe embeds, links, and inline scripts in all rendering environments. --- ## (4) Application -**Summary:** Orchestration brain (sessions, APIs, business logic). Weak validation or access controls can cascade into systemic compromise. +**Summary:** The orchestration brain that manages sessions, APIs, and business logic. Weak validation, error handling, or access controls at this layer can cascade into systemic compromise across the entire application stack. -**Threats:** T01-DPIJ, T01-IPI J, T01-SID, T01-DoSM, T01-MTU, T01-IOH, T01-EA, T01-SPL, T01-MIS +#### Prompt Injection (T01-DPIJ, T01-IPIJ) -**Targeted CWEs:** -CWE-20, CWE-74, CWE-94, CWE-200, CWE-209, CWE-359, CWE-522, CWE-400, CWE-770, CWE-787, CWE-116, CWE-79, CWE-75, CWE-284, CWE-285, CWE-345, CWE-1204 +**Mapped CWEs:** [CWE-20](https://cwe.mitre.org/data/definitions/20.html), [CWE-74](https://cwe.mitre.org/data/definitions/74.html), [CWE-94](https://cwe.mitre.org/data/definitions/94.html) -### Prompt Injection (T01-DPIJ, T01-IPIJ) +**Rationale:** Unvalidated or unescaped input injected into model orchestration logic or prompt templates can override instructions, bypass business rules, or trigger unintended system actions. -**Mapped CWEs:** CWE-20, CWE-74, CWE-94 +**Recommendations:** +- Perform strict schema validation and canonicalization on all inputs. +- Separate roles for user-authored, developer, and system instructions. +- Introduce a safe interpreter or mediation layer between user input and model orchestration. +- Conduct adversarial prompt-injection testing as part of QA. -**Rationale:** Unvalidated inputs into core instruction sets allow overrides. +#### Sensitive Information Disclosure (T01-SID, T01-SPL) -**Recommendations:** Schema validation, role separation, safe interpreter layer. +**Mapped CWEs:** [CWE-200](https://cwe.mitre.org/data/definitions/200.html), [CWE-209](https://cwe.mitre.org/data/definitions/209.html), [CWE-359](https://cwe.mitre.org/data/definitions/359.html), [CWE-522](https://cwe.mitre.org/data/definitions/522.html) -### Sensitive Information Disclosure (T01-SID, T01-SPL) +**Rationale:** Secrets, credentials, or internal configuration details may leak through logs, prompt contexts, or plugin responses, exposing sensitive data or business logic. -**Mapped CWEs:** CWE-200, CWE-209, CWE-359, CWE-522 +**Recommendations:** +- Redact secrets and PII from logs, prompts, and API responses. +- Enforce RBAC and scoped access to sensitive configuration data. +- Implement safe, user-friendly error handling that hides stack traces and internal state. +- Apply DLP scanning on logs and telemetry. -**Rationale:** Secrets leak via logs/prompts/plugins. +#### Denial of Service – Model (T01-DoSM) -**Recommendations:** Redact secrets, RBAC on sensitive data, safe error handling. +**Mapped CWEs:** [CWE-400](https://cwe.mitre.org/data/definitions/400.html), [CWE-770](https://cwe.mitre.org/data/definitions/770.html), [CWE-787](https://cwe.mitre.org/data/definitions/787.html) -### Denial of Service – Model (T01-DoSM) +**Rationale:** Excessive or malformed requests to the orchestration or inference service can saturate compute, memory, or token resources, leading to service unavailability. -**Mapped CWEs:** CWE-400, CWE-770, CWE-787 +**Recommendations:** +- Apply rate-limiting and circuit breakers at API gateways and orchestration tiers. +- Enforce input size, token, and format validation. +- Implement workload isolation and quotas per tenant, API, or model instance. +- Monitor runtime metrics to detect anomalous consumption patterns. -**Recommendations:** Rate-limit orchestration, circuit breakers, size checks. +#### Model Toxicity / Misinformation (T01-MTU, T01-MIS) -### Model Toxicity / Misinformation (T01-MTU, T01-MIS) +**Mapped CWEs:** [CWE-345](https://cwe.mitre.org/data/definitions/345.html), [CWE-1204](https://cwe.mitre.org/data/definitions/1204.html) -**Mapped CWEs:** CWE-345, CWE-1204 +**Rationale:** Models embedded in the application can generate harmful, biased, or false content when the orchestration lacks grounding, confidence thresholds, or moderation layers. -**Recommendations:** Grounding checks, toxicity/bias filters, confidence flags. +**Recommendations:** +- Implement grounding and factual consistency checks using trusted data sources. +- Integrate toxicity and bias filters in the inference pipeline. +- Flag low-confidence or high-risk outputs for review before dissemination. +- Apply continuous evaluation of model reliability and fairness metrics. -### Insecure Output Handling (T01-IOH) +#### Insecure Output Handling (T01-IOH) -**Mapped CWEs:** CWE-79, CWE-116, CWE-75 +**Mapped CWEs:** [CWE-79](https://cwe.mitre.org/data/definitions/79.html), [CWE-116](https://cwe.mitre.org/data/definitions/116.html), [CWE-75](https://cwe.mitre.org/data/definitions/75.html) -**Recommendations:** Contextual encoding/sanitization; strip unsafe HTML/MD. +**Rationale:** Improperly sanitized or encoded model outputs (HTML, Markdown, or JSON) rendered in dashboards or downstream clients can lead to injection, cross-site scripting, or data corruption. -### Excessive Agency (T01-EA) +**Recommendations:** +- Apply contextual encoding and sanitization before rendering. +- Strip or escape unsafe HTML/Markdown tags and attributes. +- Use safe templating libraries or rendering frameworks. +- Enforce output validation and content-type boundaries between services. -**Mapped CWEs:** CWE-284, CWE-285 +#### Excessive Agency (T01-EA) -**Recommendations:** Least privilege, allowlists, secondary approvals. +**Mapped CWEs:** [CWE-284](https://cwe.mitre.org/data/definitions/284.html), [CWE-285](https://cwe.mitre.org/data/definitions/285.html) + +**Rationale:** Autonomous agents or model-driven APIs may perform privileged actions—such as initiating transactions or modifying files—without appropriate oversight or authorization. + +**Recommendations:** +- Enforce least-privilege access for model plugins, agents, and integrations. +- Maintain allowlists for sensitive operations and external service calls. +- Require secondary approvals or human-in-the-loop validation for high-impact actions. +- Log and audit all agent-initiated operations for accountability. --- @@ -273,35 +282,45 @@ CWE-20, CWE-74, CWE-94, CWE-200, CWE-209, CWE-359, CWE-522, CWE-400, CWE-770, CW **Summary:** Extended arms of the system; vulnerable to IPIJ, secrets handling, tampering, excessive actions, and unsafe workflows. -**Threats:** T01-IPI J, T01-SID, T01-MTD, T01-EA, T01-VEW +#### Indirect Prompt Injection (T01-IPIJ) -**Targeted CWEs:** -CWE-20, CWE-74, CWE-94, CWE-200, CWE-359, CWE-522, CWE-284, CWE-285, CWE-276, CWE-494, CWE-829, CWE-918, CWE-502 +**Mapped CWEs:** [CWE-20](https://cwe.mitre.org/data/definitions/20.html), [CWE-74](https://cwe.mitre.org/data/definitions/74.html), [CWE-94](https://cwe.mitre.org/data/definitions/94.html) -### Indirect Prompt Injection (T01-IPIJ) -**Mapped CWEs:** CWE-20, CWE-74, CWE-94 +**Rationale:** Plugins or connected tools may receive crafted or hidden instructions embedded within user or system prompts that manipulate downstream components, alter intended behavior, or trigger unsafe code execution. -**Rationale:** Plugins may receive crafted instructions through user or system prompts that alter tool behavior or execute unsafe code. +**Recommendations:** Enforce strict input/output schemas; escape or sanitize all parameters; prohibit dynamic code evaluation or direct command execution from model-generated content. -**Recommendations:** Strict I/O schemas, escape parameters, forbid dynamic eval. +#### Sensitive Information Disclosure (T01-SID) -### Sensitive Information Disclosure (T01-SID) -**Mapped CWEs:** CWE-200, CWE-359, CWE-522 -**Recommendations:** Scoped credentials, redact tool responses, data minimization. +**Mapped CWEs:** [CWE-200](https://cwe.mitre.org/data/definitions/200.html), [CWE-359](https://cwe.mitre.org/data/definitions/359.html), [CWE-522](https://cwe.mitre.org/data/definitions/522.html) -### Model Tampering / Disclosure (T01-MTD) -**Mapped CWEs:** CWE-276, CWE-285, CWE-494 -**Recommendations:** Hardened permissions, signed manifests, artifact signing. +**Rationale:** Model, plugin, or connected service exposes confidential data such as credentials, tokens, or personal information through logs, prompts, or API responses due to insufficient data protection or contextual awareness. -### Excessive Agency (T01-EA) -**Mapped CWEs:** CWE-284, CWE-285 -**Recommendations:** Per-action least privilege, policy gates, human-in-the-loop. +**Recommendations:** Use scoped, short-lived credentials; redact sensitive fields in tool and model outputs; apply data minimization and need-to-know access controls. -### Vulnerable External Workflow (T01-VEW) -**Mapped CWEs:** CWE-829, CWE-918, CWE-502 -**Recommendations:** Tool allowlists, egress proxy, safe content types. +#### Model Tampering / Disclosure (T01-MTD) -**Operational Hardening (cross-cutting):** Per-tool rate limits/timeouts; container isolation; telemetry; signed releases/SBOMs; tenant isolation for state. +**Mapped CWEs:** [CWE-276](https://cwe.mitre.org/data/definitions/276.html), [CWE-285](https://cwe.mitre.org/data/definitions/285.html), [CWE-494](https://cwe.mitre.org/data/definitions/494.html) + +**Rationale:** Model artifacts, weights, or configurations can be modified, replaced, or exfiltrated due to weak file permissions, missing integrity checks, or insecure deployment pipelines—allowing attackers to alter model behavior or leak intellectual property. + +**Recommendations:** Enforce hardened file and storage permissions; validate model integrity via signed manifests; require digital signing and verification of all model artifacts before deployment. + +#### Excessive Agency (T01-EA) + +**Mapped CWEs:** [CWE-284](https://cwe.mitre.org/data/definitions/284.html), [CWE-285](https://cwe.mitre.org/data/definitions/285.html) + +**Rationale:** Model or autonomous agent executes actions beyond its intended authority—such as invoking privileged APIs, modifying external systems, or performing unapproved transactions—due to insufficient access controls or unrestricted delegation. + +**Recommendations:** Enforce per-action least privilege; implement policy gates for sensitive operations; require human-in-the-loop approval for high-risk or irreversible actions. + +#### Vulnerable External Workflow (T01-VEW) + +**Mapped CWEs:** [CWE-829](https://cwe.mitre.org/data/definitions/829.html), [CWE-918](https://cwe.mitre.org/data/definitions/918.html), [CWE-502](https://cwe.mitre.org/data/definitions/502.html) + +**Rationale:** Model-integrated tools or external workflow components can be exploited through untrusted dependencies, SSRF vectors, or unsafe deserialization—allowing attackers to pivot into internal networks, exfiltrate data, or execute arbitrary code. + +**Recommendations:** Maintain strict tool allowlists and egress proxy controls; enforce validation of content types and schema for external responses. --- @@ -309,28 +328,53 @@ CWE-20, CWE-74, CWE-94, CWE-200, CWE-359, CWE-522, CWE-284, CWE-285, CWE-276, CW **Summary:** Bridges to the outside world; unverified data can inject poison, trigger unsafe actions, or spread misinformation. -**Threats:** T01-IPI J, T01-MTD, T01-SID, T01-EA, T01-VEW, T01-DMP +#### Indirect Prompt Injection (T01-IPIJ) -**Targeted CWEs:** -CWE-20, CWE-74, CWE-94, CWE-200, CWE-359, CWE-522, CWE-276, CWE-284, CWE-285, CWE-494, CWE-829, CWE-918, CWE-502, CWE-353, CWE-345 +**Mapped CWEs:** [CWE-20](https://cwe.mitre.org/data/definitions/20.html), [CWE-74](https://cwe.mitre.org/data/definitions/74.html), [CWE-94](https://cwe.mitre.org/data/definitions/94.html) -### Indirect Prompt Injection (T01-IPIJ) -**Recommendations:** Sanitize/normalize external content; restrict content types; segregate retrieved content. +**Rationale:** Plugins or retrieval components may process crafted or malicious content from external sources (web pages, documents, APIs) that inject hidden instructions or alter model behavior through prompt manipulation. -### Model Tampering/Disclosure (T01-MTD) -**Recommendations:** Integrity/signature checks; least-privilege access; explicit approvals; hardened storage permissions. +**Recommendations:** Sanitize and normalize all retrieved external content; restrict accepted content types and formats; segregate and label retrieved data to prevent cross-context prompt injection. -### Sensitive Information Disclosure (T01-SID) -**Recommendations:** Mask sensitive fields; scoped OAuth; DLP policies. +#### Model Tampering/Disclosure (T01-MTD) -### Excessive Agency (T01-EA) -**Recommendations:** RBAC and allowlists for sources; policy checks before executing; sandboxed connectors. +**Mapped CWEs:** [CWE-276](https://cwe.mitre.org/data/definitions/276.html), [CWE-285](https://cwe.mitre.org/data/definitions/285.html), [CWE-494](https://cwe.mitre.org/data/definitions/494.html) -### Vulnerable External Workflow (T01-VEW) -**Recommendations:** Egress proxy + allowlists; safe content types; SBOM verification. +**Rationale:** Model files, weights, or configurations can be modified or leaked through weak storage permissions, unverified updates, or insecure pipelines—allowing attackers to alter outputs, inject backdoors, or exfiltrate proprietary data. -### Data / Model Poisoning (T01-DMP) -**Recommendations:** Provenance/reputation scoring; adversarial sample testing; cryptographic integrity checks. +**Recommendations:** Implement integrity and signature verification for all model artifacts; enforce least-privilege access and explicit change approvals; apply hardened storage permissions across training and deployment environments. + +#### Sensitive Information Disclosure (T01-SID) + +**Mapped CWEs:** [CWE-200](https://cwe.mitre.org/data/definitions/200.html), [CWE-359](https://cwe.mitre.org/data/definitions/359.html), [CWE-522](https://cwe.mitre.org/data/definitions/522.html) + +**Rationale:** AI models or connected tools may expose confidential data (e.g., tokens, credentials, personal identifiers) through logs, responses, or stored context due to insufficient redaction or access controls. + +**Recommendations:** Mask sensitive fields in logs and outputs; use scoped OAuth credentials with minimal privileges; enforce data-loss-prevention (DLP) policies for prompt and response data flows. + +#### Excessive Agency (T01-EA) + +**Mapped CWEs:** [CWE-284](https://cwe.mitre.org/data/definitions/284.html), [CWE-285](https://cwe.mitre.org/data/definitions/285.html) + +**Rationale:** Model or agent autonomously performs privileged or unintended actions—such as calling sensitive APIs, modifying resources, or invoking external tools—without appropriate authorization or contextual policy validation. + +**Recommendations:** Enforce RBAC and allowlists for data sources and actions; perform policy and safety checks before executing model-initiated operations; use sandboxed or isolated connectors to restrict external access. + +#### Vulnerable External Workflow (T01-VEW) + +**Mapped CWEs:** [CWE-829](https://cwe.mitre.org/data/definitions/829.html), [CWE-918](https://cwe.mitre.org/data/definitions/918.html), [CWE-502](https://cwe.mitre.org/data/definitions/502.html) + +**Rationale:** Integrations or tools that interact with external workflows can be compromised via untrusted dependencies, SSRF, or unsafe deserialization, leading to unauthorized network access or remote code execution. + +**Recommendations:** Enforce egress proxy and strict allowlists for outbound connections; validate and enforce safe content types; verify software supply chain integrity through signed releases and SBOM verification. + +#### Data / Model Poisoning (T01-DMP) + +**Mapped CWEs:** [CWE-20](https://cwe.mitre.org/data/definitions/20.html), [CWE-494](https://cwe.mitre.org/data/definitions/494.html), [CWE-353](https://cwe.mitre.org/data/definitions/353.html) + +**Rationale:** Attackers inject malicious data or manipulate model artifacts during training, fine-tuning, or update pipelines, causing biased outputs, backdoors, or performance degradation. + +**Recommendations:** Establish data provenance and reputation scoring mechanisms; perform adversarial sample and anomaly testing; apply cryptographic integrity checks on datasets and model artifacts throughout the pipeline. --- @@ -338,25 +382,45 @@ CWE-20, CWE-74, CWE-94, CWE-200, CWE-359, CWE-522, CWE-276, CWE-284, CWE-285, CW **Summary:** The filter layer; weak parsing/schema enforcement lets adversarial inputs/injections slip through. -**Threats:** T01-DPIJ, T01-AIE, T01-SID, T01-LSID, T01-DoSM, T01-SPL, T01-VEW +#### Prompt Injection (T01-DPIJ) -**Targeted CWEs:** -CWE-20, CWE-74, CWE-94, CWE-200, CWE-359, CWE-522, CWE-532, CWE-209, CWE-400, CWE-770, CWE-787, CWE-79, CWE-116, CWE-75, CWE-918 +**Mapped CWEs:** [CWE-20](https://cwe.mitre.org/data/definitions/20.html), [CWE-74](https://cwe.mitre.org/data/definitions/74.html), [CWE-94](https://cwe.mitre.org/data/definitions/94.html) -### Prompt Injection (T01-DPIJ) -**Recommendations:** Strict schemas and typing; strip unsafe control sequences; sandbox inputs. +**Rationale:** Malicious user or system input manipulates model prompts to override instructions, inject new goals, or trigger unintended actions in downstream tools or connected systems. -### Adversarial Input Evasion (T01-AIE) -**Recommendations:** Unicode normalization; adversarial testing; layered validation. +**Recommendations:** Enforce strict input schemas and strong typing; strip unsafe control sequences and escape characters; sandbox and isolate user inputs before prompt assembly. -### Sensitive Information Disclosure (T01-SID, T01-LSID, T01-SPL) -**Recommendations:** Ingestion-time redaction; masked logging; sanitize logs and errors. +#### Adversarial Input Evasion (T01-AIE) -### Denial of Service – Model (T01-DoSM) -**Recommendations:** Input size/rate quotas; buffer validation. +**Mapped CWEs:** [CWE-20](https://cwe.mitre.org/data/definitions/20.html), [CWE-1384](https://cwe.mitre.org/data/definitions/1384.html) -### Vulnerable External Workflow (T01-VEW) -**Recommendations:** Domain allowlists + proxy; content-type validation. +**Rationale:** Attackers craft adversarial inputs (e.g., perturbed tokens, unicode tricks, or obfuscated payloads) to evade model detection or classification boundaries, resulting in mispredictions or bypassing safety filters. + +**Recommendations:** Normalize and sanitize Unicode and encoding variations; conduct adversarial robustness testing; apply layered input validation and confidence thresholding. + +#### Sensitive Information Disclosure (T01-SID, T01-LSID, T01-SPL) + +**Mapped CWEs:** [CWE-200](https://cwe.mitre.org/data/definitions/200.html), [CWE-359](https://cwe.mitre.org/data/definitions/359.html), [CWE-522](https://cwe.mitre.org/data/definitions/522.html) + +**Rationale:** Sensitive data (e.g., API keys, secrets, PII, or training data) is exposed during ingestion, inference, or logging due to unredacted inputs, verbose errors, or unsafe context retention. + +**Recommendations:** Apply ingestion-time redaction for sensitive terms; mask or tokenize secrets in logs; sanitize logs, error traces, and tool responses to prevent data leakage. + +#### Denial of Service – Model (T01-DoSM) + +**Mapped CWEs:** [CWE-400](https://cwe.mitre.org/data/definitions/400.html), [CWE-770](https://cwe.mitre.org/data/definitions/770.html), [CWE-787](https://cwe.mitre.org/data/definitions/787.html) + +**Rationale:** Oversized or malformed inputs and unbounded request rates can exhaust GPU, memory, or CPU resources in model inference services, leading to degraded performance or service outages. + +**Recommendations:** Enforce input size and rate quotas; validate buffer dimensions and tensor structures before inference execution. + +#### Vulnerable External Workflow (T01-VEW) + +**Mapped CWEs:** [CWE-829](https://cwe.mitre.org/data/definitions/829.html), [CWE-918](https://cwe.mitre.org/data/definitions/918.html), [CWE-502](https://cwe.mitre.org/data/definitions/502.html) + +**Rationale:** External toolchains, webhooks, or retrieval flows can be exploited through untrusted dependencies, SSRF, or unsafe deserialization to access internal networks or execute arbitrary code. + +**Recommendations:** Use domain-based allowlists with outbound proxy enforcement; validate and enforce safe content types for all retrieved or external resources. --- @@ -364,31 +428,61 @@ CWE-20, CWE-74, CWE-94, CWE-200, CWE-359, CWE-522, CWE-532, CWE-209, CWE-400, CW **Summary:** Safety gate before delivery; failure here leaks sensitive data, misinformation, and unsafe content. -**Threats:** T01-LSID, T01-SID, T01-DoSM, T01-SPL, T01-IOH, T01-TDL, T01-MTU, T01-EA, T01-MIS +#### Log/Storage Information Disclosure (T01-LSID) -**Targeted CWEs:** -CWE-79, CWE-116, CWE-75, CWE-200, CWE-209, CWE-359, CWE-532, CWE-522, CWE-400, CWE-770, CWE-787, CWE-284, CWE-285, CWE-345, CWE-1204 +**Mapped CWEs:** [CWE-200](https://cwe.mitre.org/data/definitions/200.html), [CWE-532](https://cwe.mitre.org/data/definitions/532.html), [CWE-522](https://cwe.mitre.org/data/definitions/522.html) -### Log/Storage Information Disclosure (T01-LSID) -**Recommendations:** Strip sensitive context; RBAC for logs; safe error messages. +**Rationale:** Logs or persistent storage may capture raw model outputs, user prompts, or tokens that contain sensitive information. Without redaction, encryption, or access controls, these records can expose secrets, PII, or proprietary context. -### Sensitive Information Disclosure (T01-SID, T01-SPL, T01-TDL) -**Recommendations:** Post-output DLP; encrypt/mask sensitive fields; prevent recall of sensitive training rows. +**Recommendations:** Strip sensitive context from stored logs and outputs; enforce RBAC and least privilege for log access; use sanitized and generic error messages. -### Denial of Service – Model (T01-DoSM) -**Recommendations:** Cap output size/tokens; quarantine oversized outputs; validate downstream buffers. +#### Sensitive Information Disclosure (T01-SID, T01-SPL, T01-TDL) -### Insecure Output Handling (T01-IOH) -**Recommendations:** Contextual encoding; allowlist sanitizers; disable rich rendering for untrusted text. +**Mapped CWEs:** [CWE-200](https://cwe.mitre.org/data/definitions/200.html), [CWE-359](https://cwe.mitre.org/data/definitions/359.html), [CWE-522](https://cwe.mitre.org/data/definitions/522.html) -### Training Data Leakage (T01-TDL) -**Recommendations:** Differential privacy; verbatim/entropy filters; redact prompts; restrict logging. +**Rationale:** Output layers may inadvertently reveal secrets, PII, or confidential training data through generated responses, summaries, or recalled examples. -### Model Toxicity / Misinformation (T01-MTU, T01-MIS) -**Recommendations:** Toxicity/bias filters; grounding/citations; fallbacks. +**Recommendations:** Apply post-output DLP scanning; encrypt or mask sensitive fields before returning to clients; prevent recall or verbatim exposure of sensitive training data rows. -### Excessive Agency (T01-EA) -**Recommendations:** Allowlisted commands; authorization checks; explicit confirmation. +#### Denial of Service – Model (T01-DoSM) + +**Mapped CWEs:** [CWE-400](https://cwe.mitre.org/data/definitions/400.html), [CWE-770](https://cwe.mitre.org/data/definitions/770.html), [CWE-787](https://cwe.mitre.org/data/definitions/787.html) + +**Rationale:** Excessively large or malformed outputs (e.g., runaway text generation, long JSON sequences) can overflow downstream buffers or consume significant rendering resources, impacting availability. + +**Recommendations:** Cap output size and token limits; quarantine or truncate oversized responses; validate downstream buffer and rendering capacities. + +#### Insecure Output Handling (T01-IOH) + +**Mapped CWEs:** [CWE-79](https://cwe.mitre.org/data/definitions/79.html), [CWE-116](https://cwe.mitre.org/data/definitions/116.html), [CWE-75](https://cwe.mitre.org/data/definitions/75.html) + +**Rationale:** Untrusted model outputs rendered as HTML, Markdown, or code without proper encoding can lead to injection attacks or content manipulation in client or downstream systems. + +**Recommendations:** Use contextual output encoding and allowlisted sanitization routines; disable rich rendering for untrusted text or code blocks; enforce strict content-type boundaries. + +#### Training Data Leakage (T01-TDL) + +**Mapped CWEs:** [CWE-201](https://cwe.mitre.org/data/definitions/201.html), [CWE-359](https://cwe.mitre.org/data/definitions/359.html) + +**Rationale:** Models may emit verbatim snippets or memorized content from their training data, including personally identifiable or proprietary information. + +**Recommendations:** Employ differential privacy during training; use verbatim and entropy-based leakage filters; redact prompt and output logs; restrict access to model telemetry or trace data. + +#### Model Toxicity / Misinformation (T01-MTU, T01-MIS) + +**Mapped CWEs:** [CWE-345](https://cwe.mitre.org/data/definitions/345.html), [CWE-1204](https://cwe.mitre.org/data/definitions/1204.html) + +**Rationale:** Generated outputs may include harmful, biased, or false information due to unfiltered model behavior or insufficient grounding in verified sources. + +**Recommendations:** Integrate toxicity and bias filters; require grounding and citations to trusted datasets; implement fallback responses when confidence is low or bias is detected. + +#### Excessive Agency (T01-EA) + +**Mapped CWEs:** [CWE-284](https://cwe.mitre.org/data/definitions/284.html), [CWE-285](https://cwe.mitre.org/data/definitions/285.html) + +**Rationale:** The model or its connected tools execute actions automatically (e.g., API calls, file writes, system changes) without explicit authorization or confirmation. + +**Recommendations:** Restrict actions to allowlisted commands; apply authorization and policy checks before execution; require explicit human confirmation for high-impact operations. --- @@ -396,38 +490,77 @@ CWE-79, CWE-116, CWE-75, CWE-200, CWE-209, CWE-359, CWE-532, CWE-522, CWE-400, C **Summary:** The core intelligence; targeted by injection, poisoning, theft, inversion, DoS, and unsafe outputs. -**Threats:** -T01-DPIJ, T01-IPI J, T01-SCMP, T01-AIE, T01-DPFT, T01-RMP, T01-DMP, T01-SID, T01-MIMI, T01-TDL, T01-DoSM, T01-LSID, T01-SPL, T01-VEW, T01-MTU, T01-IOH, T01-MTR, T01-EA, T01-MIS +#### Prompt Injection (T01-DPIJ, T01-IPIJ) -**Targeted CWEs:** -CWE-20, CWE-74, CWE-94, CWE-200, CWE-209, CWE-359, CWE-522, CWE-532, CWE-276, CWE-284, CWE-285, CWE-400, CWE-770, CWE-787, CWE-918, CWE-502, CWE-494, CWE-345, CWE-353, CWE-1204, CWE-116, CWE-119, CWE-830, CWE-829, CWE-640, CWE-693, CWE-75, CWE-79 +**Mapped CWEs:** [CWE-20](https://cwe.mitre.org/data/definitions/20.html), [CWE-74](https://cwe.mitre.org/data/definitions/74.html), [CWE-94](https://cwe.mitre.org/data/definitions/94.html) -### Prompt Injection (T01-DPIJ, T01-IPIJ) -**Recommendations:** Separate system/developer prompts; tokenizer-stage filtering; adversarial training. +**Rationale:** Crafted user or system inputs can override, manipulate, or insert instructions within model prompts—altering the model’s intended reasoning path or causing execution of untrusted actions. -### Supply Chain / Data & Fine-tuning Poisoning (T01-SCMP, T01-DPFT, T01-RMP, T01-DMP) -**Recommendations:** Signed weights/datasets; provenance scoring; adversarial sanitation; SBOMs. +**Recommendations:** Separate system, developer, and user prompts into isolated contexts; apply tokenizer-stage filtering and normalization; conduct adversarial training to harden against prompt manipulation. -### Adversarial Input Evasion (T01-AIE) -**Recommendations:** Normalize before tokenization; robustness testing; monitor embeddings. +#### Supply Chain / Data & Fine-tuning Poisoning (T01-SCMP, T01-DPFT, T01-RMP, T01-DMP) -### Sensitive Information Disclosure / Training Data Leakage (T01-SID, T01-TDL, T01-LSID, T01-SPL) -**Recommendations:** DP in training; block verbatim sequences; redact system prompts; restrict logging. +**Mapped CWEs:** [CWE-494](https://cwe.mitre.org/data/definitions/494.html), [CWE-353](https://cwe.mitre.org/data/definitions/353.html), [CWE-829](https://cwe.mitre.org/data/definitions/829.html) -### Model Inversion / Membership Inference (T01-MIMI) -**Recommendations:** DP-SGD; rate limits/randomization; run MI red-teaming. +**Rationale:** Model training or fine-tuning data, dependencies, or weights can be poisoned or replaced through compromised datasets, malicious model checkpoints, or tampered packages in the supply chain. -### Denial of Service – Model (T01-DoSM) -**Recommendations:** Cap context; detect anomalies; harden serving buffers. +**Recommendations:** Use digitally signed model weights and datasets; apply provenance and reputation scoring; sanitize fine-tuning data for adversarial patterns; maintain SBOMs for all model components. -### Insecure Output Handling / Unsafe Integrations (T01-IOH, T01-VEW) -**Recommendations:** Sanitize outputs; whitelist tools; enforce policy layers. +#### Adversarial Input Evasion (T01-AIE) -### Model Theft / Exfiltration (T01-MTR, T01-MTD) -**Recommendations:** Access controls; encryption at rest; monitor for exfil. +**Mapped CWEs:** [CWE-20](https://cwe.mitre.org/data/definitions/20.html), [CWE-1384](https://cwe.mitre.org/data/definitions/1384.html) -### Model Toxicity / Misinformation / Excessive Agency (T01-MTU, T01-MIS, T01-EA) -**Recommendations:** Toxicity/bias post-filters; grounding; restrict actionable outputs; approvals. +**Rationale:** Adversarially perturbed inputs exploit model weaknesses to evade detection or cause misclassification, often through subtle token-level or embedding-space manipulation. + +**Recommendations:** Normalize inputs prior to tokenization; perform robustness and adversarial testing across datasets; monitor embedding distributions for drift or anomalies. + +#### Sensitive Information Disclosure / Training Data Leakage (T01-SID, T01-TDL, T01-LSID, T01-SPL) + +**Mapped CWEs:** [CWE-200](https://cwe.mitre.org/data/definitions/200.html), [CWE-359](https://cwe.mitre.org/data/definitions/359.html), [CWE-201](https://cwe.mitre.org/data/definitions/201.html) + +**Rationale:** Model parameters or outputs may expose memorized training data, sensitive context, or private attributes through unfiltered responses or model inversion attempts. + +**Recommendations:** Apply differential privacy during training (e.g., DP-SGD); block verbatim sequence recall; redact sensitive tokens in system prompts; restrict or sanitize inference-time logging. + +#### Model Inversion / Membership Inference (T01-MIMI) + +**Mapped CWEs:** [CWE-203](https://cwe.mitre.org/data/definitions/203.html), [CWE-359](https://cwe.mitre.org/data/definitions/359.html) + +**Rationale:** Attackers query the model to infer whether specific data records were used during training, or reconstruct sensitive training examples via inversion techniques. + +**Recommendations:** Use DP-SGD or other noise-based privacy mechanisms; enforce rate limits and output randomization; conduct dedicated membership-inference red teaming to validate resilience. + +#### Denial of Service – Model (T01-DoSM) + +**Mapped CWEs:** [CWE-400](https://cwe.mitre.org/data/definitions/400.html), [CWE-770](https://cwe.mitre.org/data/definitions/770.html), [CWE-787](https://cwe.mitre.org/data/definitions/787.html) + +**Rationale:** Excessive model context lengths, complex prompt chains, or malformed inference payloads can overload GPU/CPU resources, leading to degraded performance or outages. + +**Recommendations:** Cap model context and token limits; detect abnormal inference patterns or anomalies; harden serving buffers and apply per-request resource quotas. + +#### Insecure Output Handling / Unsafe Integrations (T01-IOH, T01-VEW) + +**Mapped CWEs:** [CWE-79](https://cwe.mitre.org/data/definitions/79.html), [CWE-116](https://cwe.mitre.org/data/definitions/116.html), [CWE-829](https://cwe.mitre.org/data/definitions/829.html) + +**Rationale:** Model outputs may contain untrusted data or unsafe formatting passed to external systems, or integrations may process outputs without sanitization—leading to injection or workflow compromise. + +**Recommendations:** Sanitize and encode all model outputs; restrict integrations to whitelisted tools and trusted domains; enforce policy and validation layers between model and tool execution. + +#### Model Theft / Exfiltration (T01-MTR, T01-MTD) + +**Mapped CWEs:** [CWE-276](https://cwe.mitre.org/data/definitions/276.html), [CWE-285](https://cwe.mitre.org/data/definitions/285.html), [CWE-494](https://cwe.mitre.org/data/definitions/494.html) + +**Rationale:** Unauthorized access or exfiltration of model artifacts, weights, or parameters can lead to IP theft, cloning, or malicious redistribution of compromised versions. + +**Recommendations:** Apply strict access controls to model repositories and serving endpoints; encrypt weights and checkpoints at rest; monitor for unauthorized exfiltration or replication. + +#### Model Toxicity / Misinformation / Excessive Agency (T01-MTU, T01-MIS, T01-EA) + +**Mapped CWEs:** [CWE-345](https://cwe.mitre.org/data/definitions/345.html), [CWE-1204](https://cwe.mitre.org/data/definitions/1204.html), [CWE-284](https://cwe.mitre.org/data/definitions/284.html) + +**Rationale:** Models may generate biased, harmful, or false information—or take autonomous actions based on toxic or deceptive outputs—causing reputational, ethical, or operational harm. + +**Recommendations:** Integrate toxicity and bias post-filters; ground model outputs in verified sources; restrict actionable outputs via policy enforcement; require approvals for high-risk autonomous actions. --- @@ -435,22 +568,37 @@ CWE-20, CWE-74, CWE-94, CWE-200, CWE-209, CWE-359, CWE-522, CWE-532, CWE-276, CW **Summary:** Crown jewels at rest — must be encrypted, signed, and access-controlled. -**Threats:** T01-DPFT, T01-SCMP, T01-MTR, T01-MTD +#### Data/Prompt Fine-Tuning Poisoning (T01-DPFT) -**Targeted CWEs:** -CWE-276, CWE-284, CWE-285, CWE-200, CWE-359, CWE-522, CWE-494, CWE-353, CWE-922 +**Mapped CWEs:** [CWE-494](https://cwe.mitre.org/data/definitions/494.html), [CWE-353](https://cwe.mitre.org/data/definitions/353.html) -### Data/Prompt Fine-Tuning Poisoning (T01-DPFT) -**Recommendations:** Cryptographic signing + checksums; read-only versioned storage; attestation. +**Rationale:** Attackers may modify or replace stored training or fine-tuning datasets, prompt templates, or embeddings in model storage repositories—resulting in malicious model behavior or backdoored outputs. -### Supply Chain Model Poisoning (T01-SCMP) -**Recommendations:** Trusted registries; verify lineage; pin dependencies. +**Recommendations:** Apply cryptographic signing and checksums to all stored artifacts; maintain read-only and versioned storage for model and dataset files; require cryptographic attestation for model load operations. -### Model Theft / Exfiltration (T01-MTR) -**Recommendations:** Encrypt with KMS; least-privilege; monitor bulk downloads; harden defaults. +#### Supply Chain Model Poisoning (T01-SCMP) -### Model Tampering / Disclosure (T01-MTD) -**Recommendations:** WORM storage; integrity verification on load; restrict access to service accounts. +**Mapped CWEs:** [CWE-829](https://cwe.mitre.org/data/definitions/829.html), [CWE-494](https://cwe.mitre.org/data/definitions/494.html) + +**Rationale:** Model dependencies, pre-trained weights, or third-party registries can be compromised, introducing malicious code or poisoned weights into the build and deployment pipelines. + +**Recommendations:** Source models and dependencies only from trusted registries; verify lineage and digital signatures; pin dependency versions and verify integrity before loading or deployment. + +#### Model Theft / Exfiltration (T01-MTR) + +**Mapped CWEs:** [CWE-276](https://cwe.mitre.org/data/definitions/276.html), [CWE-284](https://cwe.mitre.org/data/definitions/284.html) + +**Rationale:** Unauthorized access or large-scale export of model artifacts, checkpoints, or container images can lead to theft of proprietary IP or replication of protected models. + +**Recommendations:** Encrypt stored models and weights using KMS-managed keys; enforce least-privilege access for repositories and buckets; monitor for bulk download or anomalous access; harden default permissions and configurations. + +#### Model Tampering / Disclosure (T01-MTD) + +**Mapped CWEs:** [CWE-276](https://cwe.mitre.org/data/definitions/276.html), [CWE-285](https://cwe.mitre.org/data/definitions/285.html), [CWE-494](https://cwe.mitre.org/data/definitions/494.html) + +**Rationale:** Stored models or weight files can be altered, replaced, or disclosed if access controls, integrity checks, or permissions are weak—allowing attackers to inject malicious behavior or leak proprietary data. + +**Recommendations:** Use WORM (Write Once, Read Many) or immutable storage for production models; perform integrity verification on model load; restrict access to service accounts with strict RBAC and scoped tokens. --- @@ -458,74 +606,161 @@ CWE-276, CWE-284, CWE-285, CWE-200, CWE-359, CWE-522, CWE-494, CWE-353, CWE-922 **Summary:** Execution gateway; must resist poisoning, theft, DoS, and unsafe outputs. -**Threats:** T01-SCMP, T01-MTU, T01-MTR, T01-DoSM +#### Supply Chain Model Poisoning (T01-SCMP) -**Targeted CWEs:** -CWE-276, CWE-284, CWE-285, CWE-400, CWE-770, CWE-787, CWE-494, CWE-353, CWE-345, CWE-1204, CWE-75 +**Mapped CWEs:** [CWE-494](https://cwe.mitre.org/data/definitions/494.html), [CWE-353](https://cwe.mitre.org/data/definitions/353.html), [CWE-829](https://cwe.mitre.org/data/definitions/829.html) -### Supply Chain Model Poisoning (T01-SCMP) -**Recommendations:** Signed container images; checksums; SBOM-enforced provenance; block untrusted registries. +**Rationale:** Model serving containers, preloaded weights, or dependencies may be replaced or tampered with during build or deployment, introducing malicious payloads or backdoored models into production pipelines. -### Model Toxicity / Unreliable Outputs (T01-MTU) -**Recommendations:** Moderation/toxicity filters; grounding checks; safe fallbacks. +**Recommendations:** Use signed and verified container images; validate checksums and digests for all model files; enforce SBOM-based provenance and signature verification; block deployment from untrusted or public registries. -### Model Theft / Exfiltration (T01-MTR) -**Recommendations:** Rate limits/anomaly detection; mTLS + RBAC; encrypt weights; harden FS perms. +#### Model Toxicity / Unreliable Outputs (T01-MTU) -### Denial of Service – Model (T01-DoSM) -**Recommendations:** Cap request size/tokens; quotas at gateway; circuit breakers/autoscaling; robust parsers. +**Mapped CWEs:** [CWE-345](https://cwe.mitre.org/data/definitions/345.html), [CWE-1204](https://cwe.mitre.org/data/definitions/1204.html), [CWE-75](https://cwe.mitre.org/data/definitions/75.html) + +**Rationale:** Deployed models may generate harmful, biased, or misleading content due to unmoderated outputs, missing grounding, or unreliable post-processing mechanisms. + +**Recommendations:** Integrate moderation and toxicity filters into inference pipelines; perform grounding checks against trusted data sources; implement fallback or neutral responses when confidence is low or results are potentially unsafe. + +#### Model Theft / Exfiltration (T01-MTR) + +**Mapped CWEs:** [CWE-276](https://cwe.mitre.org/data/definitions/276.html), [CWE-284](https://cwe.mitre.org/data/definitions/284.html), [CWE-285](https://cwe.mitre.org/data/definitions/285.html) + +**Rationale:** Insecure endpoints, weak authentication, or misconfigured storage permissions may allow adversaries to exfiltrate model weights, clone serving containers, or reconstruct models through inference scraping. + +**Recommendations:** Enforce API rate limits and anomaly detection on inference endpoints; require mutual TLS (mTLS) and RBAC-based authorization; encrypt model weights at rest; harden file system permissions and disable anonymous or default service accounts. + +#### Denial of Service – Model (T01-DoSM) + +**Mapped CWEs:** [CWE-400](https://cwe.mitre.org/data/definitions/400.html), [CWE-770](https://cwe.mitre.org/data/definitions/770.html), [CWE-787](https://cwe.mitre.org/data/definitions/787.html) + +**Rationale:** Oversized, malformed, or high-rate inference requests can exhaust serving resources such as memory, CPU, or GPU queues—causing degraded availability or total service outages. + +**Recommendations:** Cap input request sizes and token lengths; configure quotas and throttling at the API gateway; use circuit breakers and autoscaling for load protection; validate input buffers and parsers to prevent overflow or runaway generation. --- ## (12) Evaluation -**Summary:** The safety lens; poison/bypass here yields false assurance. +**Summary:** Where model quality and trustworthiness are validated; weak evaluation enables unsafe, biased, or manipulated outputs to pass undetected. -**Threats:** T01-AIE, T01-DMP, T01-LSID, T01-SID, T01-TDL, T01-DoSM, T01-MTU, T01-IOH, T01-MIS +#### Adversarial Input Evasion (T01-AIE) -**Targeted CWEs:** -CWE-20, CWE-116, CWE-200, CWE-209, CWE-359, CWE-532, CWE-400, CWE-770, CWE-787, CWE-345, CWE-1204 +**Mapped CWEs:** [CWE-20](https://cwe.mitre.org/data/definitions/20.html), [CWE-116](https://cwe.mitre.org/data/definitions/116.html), [CWE-1389](https://cwe.mitre.org/data/definitions/1389.html) -### Adversarial Input Evasion (T01-AIE) -**Recommendations:** Schema validation; normalization; adversarial red-teaming. +**Rationale:** Evaluation datasets and inputs can be crafted to evade detection or distort performance metrics, leading to false confidence in model robustness. -### Data/Model Poisoning (T01-DMP) -**Recommendations:** Verify dataset provenance; cross-check baselines; ensemble evaluation. +**Recommendations:** Normalize and validate evaluation inputs; perform adversarial testing under varied perturbations; apply outlier and embedding-space drift detection. -### Information Disclosure (T01-LSID, T01-SID, T01-TDL) -**Recommendations:** Sanitize logs; encrypt/ACL datasets; monitor for memorization leakage. +#### Data/Model Poisoning (T01-DMP) -### Denial of Service – Model (T01-DoSM) -**Recommendations:** Limit dataset size/runs; rate-limit jobs; fault isolation. +**Mapped CWEs:** [CWE-345](https://cwe.mitre.org/data/definitions/345.html), [CWE-353](https://cwe.mitre.org/data/definitions/353.html), [CWE-494](https://cwe.mitre.org/data/definitions/494.html) -### Model Toxicity / Unsafe Output / Misinformation (T01-MTU, T01-IOH, T01-MIS) -**Recommendations:** Include toxicity/factuality benchmarks; require grounding; scan for unsafe HTML/MD. +**Rationale:** Compromised datasets or poisoned models used during evaluation can skew metrics and conceal malicious alterations. + +**Recommendations:** Validate datasets with cryptographic checksums and signatures; maintain golden reference baselines; verify model lineage before evaluation. + +#### Log/Storage Information Disclosure (T01-LSID) + +**Mapped CWEs:** [CWE-117](https://cwe.mitre.org/data/definitions/117.html), [CWE-200](https://cwe.mitre.org/data/definitions/200.html), [CWE-532](https://cwe.mitre.org/data/definitions/532.html) + +**Rationale:** Logging of sensitive evaluation outputs, prompts, or internal metrics can expose confidential data or model behavior to unauthorized users. + +**Recommendations:** Sanitize and minimize logged output; redact sensitive context or metadata; restrict access to evaluation logs and reports. + +#### Sensitive Information Disclosure (T01-SID) + +**Mapped CWEs:** [CWE-200](https://cwe.mitre.org/data/definitions/200.html), [CWE-359](https://cwe.mitre.org/data/definitions/359.html), [CWE-522](https://cwe.mitre.org/data/definitions/522.html) + +**Rationale:** Evaluation pipelines may process datasets containing private or regulated information that could leak via reports, dashboards, or telemetry. + +**Recommendations:** Apply data masking and DLP filters in evaluation output; enforce least-privilege access; encrypt all evaluation artifacts and summaries at rest. + +#### Training Data Leakage (T01-TDL) + +**Mapped CWEs:** [CWE-201](https://cwe.mitre.org/data/definitions/201.html), [CWE-359](https://cwe.mitre.org/data/definitions/359.html) + +**Rationale:** Evaluation datasets overlapping with training data can cause inflated scores and unintentional exposure of memorized content. + +**Recommendations:** De-duplicate evaluation data against training sets; implement entropy and verbatim leakage filters; isolate training and evaluation environments. + +#### Denial of Service – Model (T01-DoSM) + +**Mapped CWEs:** [CWE-400](https://cwe.mitre.org/data/definitions/400.html), [CWE-770](https://cwe.mitre.org/data/definitions/770.html), [CWE-787](https://cwe.mitre.org/data/definitions/787.html) + +**Rationale:** Large or malformed evaluation inputs can overload inference services, exhausting compute resources or crashing evaluation pipelines. + +**Recommendations:** Limit input and output sizes; apply quotas and circuit breakers on evaluation workloads; validate and sanitize input buffers. + +#### Model Toxicity / Misinformation (T01-MTU, T01-MIS) + +**Mapped CWEs:** [CWE-345](https://cwe.mitre.org/data/definitions/345.html), [CWE-1204](https://cwe.mitre.org/data/definitions/1204.html) + +**Rationale:** Without toxicity, bias, or factual consistency tests, evaluation may miss unsafe, unreliable, or ungrounded model behaviors. + +**Recommendations:** Include toxicity and bias detection in evaluation metrics; perform grounding verification against trusted sources; use human validation for high-impact outputs. + +#### Insecure Output Handling (T01-IOH) + +**Mapped CWEs:** [CWE-79](https://cwe.mitre.org/data/definitions/79.html), [CWE-74](https://cwe.mitre.org/data/definitions/74.html), [CWE-75](https://cwe.mitre.org/data/definitions/75.html), [CWE-693](https://cwe.mitre.org/data/definitions/693.html) + +**Rationale:** Unsafe rendering or display of model outputs in dashboards or visualization tools can lead to injection, cross-site scripting, or data corruption. + +**Recommendations:** Apply contextual encoding for rendered outputs; sanitize HTML/Markdown before display; restrict rich content in evaluation interfaces. + +#### Unsafe Evaluation Practices (TO1-UEP) + +**Mapped CWEs:** [CWE-352](https://cwe.mitre.org/data/definitions/352.html), [CWE-825](https://cwe.mitre.org/data/definitions/825.html) + +**Rationale:** Lack of test isolation or dependency validation in evaluation frameworks can lead to contaminated results or untrusted code execution. + +**Recommendations:** Isolate evaluation from training environments; enforce CSRF protection in evaluation tools; validate external dependencies and ensure reproducible runs. --- ## (13) Training & Tuning -**Summary:** Where knowledge is forged; poor data embeds lasting bias/backdoors. +**Summary:** Where knowledge is forged; poor data embeds lasting bias and backdoors. -**Threats:** T01-AIE, T01-MIS, T01-DPFT, T01-SCMP, T01-MTD +#### Adversarial Input Evasion (T01-AIE) -**Targeted CWEs:** -CWE-20, CWE-116, CWE-345, CWE-353, CWE-494, CWE-276, CWE-284, CWE-285, CWE-200, CWE-359 +**Mapped CWEs:** [CWE-20](https://cwe.mitre.org/data/definitions/20.html), [CWE-116](https://cwe.mitre.org/data/definitions/116.html) -### Adversarial Input Evasion (T01-AIE) -**Recommendations:** Enforce schemas + canonical normalization; adversarial resilience tests; anomaly detection in preprocessing. +**Rationale:** Adversarial or malformed training inputs (e.g., mislabeled, perturbed, or poisoned samples) can distort model learning and weaken resilience against evasion or misclassification attacks. -### Misinformation (T01-MIS) -**Recommendations:** Validate vs trusted sources; human oversight; training-time grounding. +**Recommendations:** Enforce strict data schemas and canonical normalization during ingestion; perform adversarial resilience testing on training data; deploy anomaly detection to flag abnormal patterns in preprocessing pipelines. -### Data/Prompt Fine-Tuning Poisoning (T01-DPFT) -**Recommendations:** Signed datasets; immutable baselines; adversarial testing pre-deploy. +#### Misinformation (T01-MIS) -### Supply Chain Model Poisoning (T01-SCMP) -**Recommendations:** Trusted registries; signatures; hardened defaults and scoped access. +**Mapped CWEs:** [CWE-345](https://cwe.mitre.org/data/definitions/345.html), [CWE-200](https://cwe.mitre.org/data/definitions/200.html) -### Model Tampering / Disclosure (T01-MTD) -**Recommendations:** Encrypt checkpoints/logs; RBAC; regular permission audits. +**Rationale:** Training datasets or feedback loops can contain inaccurate, biased, or manipulated content that skews model reasoning and propagates false or unsafe knowledge into production models. + +**Recommendations:** Validate datasets against trusted reference sources; integrate human oversight for labeling and feedback verification; implement training-time grounding and periodic data quality audits. + +#### Data/Prompt Fine-Tuning Poisoning (T01-DPFT) + +**Mapped CWEs:** [CWE-353](https://cwe.mitre.org/data/definitions/353.html), [CWE-494](https://cwe.mitre.org/data/definitions/494.html) + +**Rationale:** Attackers can inject poisoned examples or tampered prompt templates during fine-tuning or reinforcement learning phases, embedding persistent backdoors or bias. + +**Recommendations:** Require cryptographically signed and versioned datasets; preserve immutable baselines for training runs; conduct adversarial and data integrity testing before deploying tuned models. + +#### Supply Chain Model Poisoning (T01-SCMP) + +**Mapped CWEs:** [CWE-494](https://cwe.mitre.org/data/definitions/494.html), [CWE-284](https://cwe.mitre.org/data/definitions/284.html), [CWE-285](https://cwe.mitre.org/data/definitions/285.html) + +**Rationale:** Compromised third-party packages, pre-trained weights, or data pipelines may introduce malicious code or tainted components into the model training environment. + +**Recommendations:** Use trusted registries for dependencies and pre-trained models; enforce signature verification and provenance checks; apply hardened configuration defaults and scoped access for all training assets. + +#### Model Tampering / Disclosure (T01-MTD) + +**Mapped CWEs:** [CWE-276](https://cwe.mitre.org/data/definitions/276.html), [CWE-285](https://cwe.mitre.org/data/definitions/285.html), [CWE-359](https://cwe.mitre.org/data/definitions/359.html) + +**Rationale:** Insecure permissions or lack of encryption on model checkpoints and logs can allow unauthorized modification or exposure of sensitive model parameters and training data. + +**Recommendations:** Encrypt model checkpoints, logs, and gradient data with strong key management (KMS); apply RBAC and access scoping to all storage locations; conduct regular permission audits and integrity checks across training infrastructure. --- @@ -533,19 +768,29 @@ CWE-20, CWE-116, CWE-345, CWE-353, CWE-494, CWE-276, CWE-284, CWE-285, CWE-200, **Summary:** ML runtime backbone; supply chain or unsafe integrations taint the system. -**Threats:** T01-SCMP, T01-MTD, T01-VEW +#### Supply Chain Model Poisoning (T01-SCMP) -**Targeted CWEs:** -CWE-94, CWE-95, CWE-829, CWE-494, CWE-353, CWE-276, CWE-284, CWE-285, CWE-918, CWE-502 +**Mapped CWEs:** [CWE-494](https://cwe.mitre.org/data/definitions/494.html), [CWE-353](https://cwe.mitre.org/data/definitions/353.html), [CWE-829](https://cwe.mitre.org/data/definitions/829.html) -### Supply Chain Model Poisoning (T01-SCMP) -**Recommendations:** Pin versions; require signed packages; scan dependencies; maintain SBOMs. +**Rationale:** Compromised ML frameworks, pre-compiled binaries, or third-party libraries can introduce backdoors, poisoned dependencies, or malicious behavior into runtime environments and training pipelines. -### Model Tampering / Disclosure (T01-MTD) -**Recommendations:** Harden runtimes; least-privilege service accounts; audit framework binaries. +**Recommendations:** Pin dependency versions and require signed packages; scan for known vulnerabilities and integrity mismatches; maintain comprehensive SBOMs for all model and runtime components. -### Vulnerable External Workflow / Unsafe Integration (T01-VEW) -**Recommendations:** Disable/sandbox dynamic eval; restrict plugin loading; isolate untrusted code; harden deserialization. +#### Model Tampering / Disclosure (T01-MTD) + +**Mapped CWEs:** [CWE-276](https://cwe.mitre.org/data/definitions/276.html), [CWE-284](https://cwe.mitre.org/data/definitions/284.html), [CWE-285](https://cwe.mitre.org/data/definitions/285.html) + +**Rationale:** Weak runtime permissions, insecure service accounts, or lack of binary integrity validation can allow unauthorized modification or inspection of core ML frameworks, leading to altered inference logic or model theft. + +**Recommendations:** Harden runtimes with restricted privileges; run services under least-privilege accounts; perform regular integrity and permission audits on framework binaries and configuration files. + +#### Vulnerable External Workflow / Unsafe Integration (T01-VEW) + +**Mapped CWEs:** [CWE-94](https://cwe.mitre.org/data/definitions/94.html), [CWE-95](https://cwe.mitre.org/data/definitions/95.html), [CWE-918](https://cwe.mitre.org/data/definitions/918.html), [CWE-502](https://cwe.mitre.org/data/definitions/502.html) + +**Rationale:** Unsafe plugin loading, dynamic evaluation, or insecure integrations within ML frameworks can enable remote code execution, SSRF, or deserialization exploits that compromise the serving environment. + +**Recommendations:** Disable or sandbox dynamic `eval` and code-generation features; restrict plugin or module loading to trusted sources; isolate untrusted or experimental code in containers; harden deserialization routines and enforce strict content-type validation. --- @@ -553,22 +798,37 @@ CWE-94, CWE-95, CWE-829, CWE-494, CWE-353, CWE-276, CWE-284, CWE-285, CWE-918, C **Summary:** Knowledge vault; poisoning/tampering/leaks here undermine integrity & confidentiality. -**Threats:** T01-RMP, T01-DMP, T01-DPFT, T01-SCMP, T01-SID, T01-MTD, T01-LSID +#### Runtime/Model/Data Poisoning (T01-RMP, T01-DMP, T01-DPFT, T01-SCMP) -**Targeted CWEs:** -CWE-276, CWE-284, CWE-285, CWE-200, CWE-359, CWE-522, CWE-532, CWE-400, CWE-770, CWE-787, CWE-494, CWE-353, CWE-345, CWE-922 +**Mapped CWEs:** [CWE-353](https://cwe.mitre.org/data/definitions/353.html), [CWE-494](https://cwe.mitre.org/data/definitions/494.html), [CWE-345](https://cwe.mitre.org/data/definitions/345.html) -### Runtime/Model/Data Poisoning (T01-RMP, T01-DMP, T01-DPFT, T01-SCMP) -**Recommendations:** Integrity checks; provenance scoring; append-only/versioned stores; anomaly monitoring. +**Rationale:** Malicious or manipulated data, runtime parameters, or stored model artifacts can inject backdoors or bias into downstream inference and retraining workflows, compromising model integrity. -### Sensitive Information Disclosure (T01-SID, T01-LSID) -**Recommendations:** Encrypt at rest + KMS; RBAC; sanitized logging; access monitoring. +**Recommendations:** Perform integrity checks and cryptographic verification on stored artifacts; apply provenance and reputation scoring; maintain append-only or versioned storage; monitor for anomalies and poisoning indicators. -### Model/Data Tampering or Exfiltration (T01-MTD) -**Recommendations:** Disable public/broad ACLs; per-tenant keys; least-privilege; immutable storage for critical data. +#### Sensitive Information Disclosure (T01-SID, T01-LSID) -### Denial of Service – Storage -**Recommendations:** Quotas and rate limits; hardened parsers/buffers; ingestion throttling. +**Mapped CWEs:** [CWE-200](https://cwe.mitre.org/data/definitions/200.html), [CWE-359](https://cwe.mitre.org/data/definitions/359.html), [CWE-522](https://cwe.mitre.org/data/definitions/522.html), [CWE-532](https://cwe.mitre.org/data/definitions/532.html) + +**Rationale:** Misconfigured databases, verbose logs, or shared storage buckets may expose credentials, tokens, or PII contained in datasets, checkpoints, or system logs. + +**Recommendations:** Encrypt all sensitive data at rest using KMS-managed keys; enforce RBAC and access segmentation; sanitize and minimize logging of secrets or identifiers; monitor data-access patterns for anomalies. + +#### Model/Data Tampering or Exfiltration (T01-MTD) + +**Mapped CWEs:** [CWE-276](https://cwe.mitre.org/data/definitions/276.html), [CWE-284](https://cwe.mitre.org/data/definitions/284.html), [CWE-285](https://cwe.mitre.org/data/definitions/285.html), [CWE-922](https://cwe.mitre.org/data/definitions/922.html) + +**Rationale:** Weak storage permissions, shared access tokens, or lack of immutability controls can enable attackers to alter or exfiltrate stored model or dataset assets. + +**Recommendations:** Disable public or overly broad ACLs; use per-tenant encryption keys; enforce least-privilege storage access; apply immutable or WORM storage for critical datasets and production models. + +#### Denial of Service – Storage (T01-DoSS) + +**Mapped CWEs:** [CWE-400](https://cwe.mitre.org/data/definitions/400.html), [CWE-770](https://cwe.mitre.org/data/definitions/770.html), [CWE-787](https://cwe.mitre.org/data/definitions/787.html) + +**Rationale:** Excessive data ingestion, unbounded file uploads, or malformed objects can exhaust storage capacity or crash parsers and metadata services, disrupting model training and access. + +**Recommendations:** Enforce quotas and rate limits for data ingestion; validate and harden file parsers and buffer handling; apply throttling and back-pressure controls for high-volume writes or uploads. --- @@ -576,22 +836,37 @@ CWE-276, CWE-284, CWE-285, CWE-200, CWE-359, CWE-522, CWE-532, CWE-400, CWE-770, **Summary:** Root of trust; compromise propagates to all downstream behavior. -**Threats:** T01-MIMI, T01-TDL, T01-SID +#### Model Inversion / Membership Inference (T01-MIMI) -**Targeted CWEs:** -CWE-200, CWE-359, CWE-522, CWE-345, CWE-353, CWE-494, CWE-276, CWE-284, CWE-285 +**Mapped CWEs:** [CWE-200](https://cwe.mitre.org/data/definitions/200.html), [CWE-359](https://cwe.mitre.org/data/definitions/359.html), [CWE-522](https://cwe.mitre.org/data/definitions/522.html) -### Model Inversion / Membership Inference (T01-MIMI) -**Recommendations:** Differential privacy; strict RBAC on raw data; detect inversion patterns. +**Rationale:** Attackers can query models or inspect intermediate representations to infer whether specific data records were included in training, exposing sensitive personal or proprietary information. -### Training Data Leakage (T01-TDL) -**Recommendations:** Encrypt datasets; keep creds out of pipelines; tokenize sensitive fields pre-ingestion. +**Recommendations:** Apply differential privacy (e.g., DP-SGD) to limit per-sample influence; enforce strict RBAC and isolation for raw training data; monitor inference activity to detect inversion or membership-inference patterns. -### Sensitive Information Disclosure (T01-SID) -**Recommendations:** Least-privilege; row/column-level policies; audit all access. +#### Training Data Leakage (T01-TDL) -### Data Authenticity -**Recommendations:** Signed/versioned datasets; provenance scoring; golden-set cross-validation. +**Mapped CWEs:** [CWE-200](https://cwe.mitre.org/data/definitions/200.html), [CWE-359](https://cwe.mitre.org/data/definitions/359.html), [CWE-353](https://cwe.mitre.org/data/definitions/353.html) + +**Rationale:** Sensitive or secret data can be inadvertently exposed during dataset preparation, preprocessing, or ingestion, allowing leakage through logs, pipelines, or model memory. + +**Recommendations:** Encrypt datasets at rest and in transit; scrub credentials or tokens from preprocessing pipelines; tokenize or mask sensitive fields prior to ingestion and model training. + +#### Sensitive Information Disclosure (T01-SID) + +**Mapped CWEs:** [CWE-276](https://cwe.mitre.org/data/definitions/276.html), [CWE-284](https://cwe.mitre.org/data/definitions/284.html), [CWE-285](https://cwe.mitre.org/data/definitions/285.html) + +**Rationale:** Inadequate access control on raw or processed training datasets enables unauthorized viewing or extraction of confidential or regulated data. + +**Recommendations:** Enforce least-privilege access; implement row- and column-level data-access policies; continuously audit and alert on all access to sensitive data stores. + +#### Data Authenticity (T01-DAU) + +**Mapped CWEs:** [CWE-345](https://cwe.mitre.org/data/definitions/345.html), [CWE-494](https://cwe.mitre.org/data/definitions/494.html) + +**Rationale:** Lack of dataset provenance or version control allows tampered, mislabeled, or malicious data to contaminate training, degrading model reliability and security. + +**Recommendations:** Maintain signed and version-controlled datasets; apply provenance and reputation scoring for all data sources; perform golden-set cross-validation to detect data drift or contamination. --- @@ -599,25 +874,45 @@ CWE-200, CWE-359, CWE-522, CWE-345, CWE-353, CWE-494, CWE-276, CWE-284, CWE-285 **Summary:** Gatekeeper stage; weak validation lets poisoned/sensitive data pass. -**Threats:** T01-RMP, T01-DMP, T01-DPFT, T01-SID, T01-MIMI, T01-TDL, T01-VEW, T01-MIS +#### Runtime / Data Poisoning (T01-RMP, T01-DMP, T01-DPFT) -**Targeted CWEs:** -CWE-20, CWE-116, CWE-200, CWE-359, CWE-345, CWE-353, CWE-494, CWE-276, CWE-284, CWE-285, CWE-400, CWE-770, CWE-787, CWE-829, CWE-918, CWE-502 +**Mapped CWEs:** [CWE-353](https://cwe.mitre.org/data/definitions/353.html), [CWE-494](https://cwe.mitre.org/data/definitions/494.html), [CWE-345](https://cwe.mitre.org/data/definitions/345.html) -### Runtime / Data Poisoning (T01-RMP, T01-DMP, T01-DPFT) -**Recommendations:** Signed datasets; hash verification; drift detection. +**Rationale:** Compromised or tampered datasets entering preprocessing pipelines can introduce malicious bias, backdoors, or instability in downstream models if integrity validation is weak. -### Sensitive Information Disclosure (T01-SID, T01-TDL, T01-MIMI) -**Recommendations:** DLP in preprocessing; masking/tokenization; RBAC for feature stores. +**Recommendations:** Require signed and versioned datasets; verify file hashes and checksums during ingestion; apply statistical drift and anomaly detection to identify poisoned or manipulated data. -### Vulnerable External Workflow (T01-VEW) -**Recommendations:** Sandbox transforms; egress filtering; forbid unsafe deserialization. +#### Sensitive Information Disclosure (T01-SID, T01-TDL, T01-MIMI) -### Misinformation (T01-MIS) -**Recommendations:** Reputation/ground-truth validation; cross-dataset checks; human review for high-risk domains. +**Mapped CWEs:** [CWE-200](https://cwe.mitre.org/data/definitions/200.html), [CWE-359](https://cwe.mitre.org/data/definitions/359.html), [CWE-522](https://cwe.mitre.org/data/definitions/522.html) -### Denial of Service on Pipelines -**Recommendations:** Size quotas; ingestion rate limits; anomaly monitoring. +**Rationale:** Preprocessing or feature extraction may expose raw sensitive data—such as PII, credentials, or proprietary information—through intermediate files, logs, or feature stores. + +**Recommendations:** Implement DLP scanning during preprocessing; mask or tokenize sensitive attributes before feature extraction; apply RBAC and access segmentation for all feature store operations. + +#### Vulnerable External Workflow (T01-VEW) + +**Mapped CWEs:** [CWE-829](https://cwe.mitre.org/data/definitions/829.html), [CWE-918](https://cwe.mitre.org/data/definitions/918.html), [CWE-502](https://cwe.mitre.org/data/definitions/502.html) + +**Rationale:** Data processing scripts, plugins, or third-party connectors may invoke untrusted resources or deserialize unsafe content, enabling SSRF, RCE, or data exfiltration through external workflows. + +**Recommendations:** Execute transformation jobs in sandboxed environments; apply outbound egress filtering and domain allowlists; prohibit unsafe deserialization and enforce strict content-type validation. + +#### Misinformation (T01-MIS) + +**Mapped CWEs:** [CWE-345](https://cwe.mitre.org/data/definitions/345.html), [CWE-353](https://cwe.mitre.org/data/definitions/353.html) + +**Rationale:** Preprocessing stages that fail to validate data sources or cross-check content may propagate incorrect or manipulated data into model training, resulting in biased or false learning outcomes. + +**Recommendations:** Validate dataset sources through reputation and ground-truth scoring; perform cross-dataset consistency checks; require human review for data from high-risk or low-trust domains. + +#### Denial of Service on Pipelines (T01-DoSP) + +**Mapped CWEs:** [CWE-400](https://cwe.mitre.org/data/definitions/400.html), [CWE-770](https://cwe.mitre.org/data/definitions/770.html), [CWE-787](https://cwe.mitre.org/data/definitions/787.html) + +**Rationale:** Excessive data volume, malformed records, or unbounded streaming inputs can overwhelm preprocessing pipelines, causing latency, storage exhaustion, or crashes. + +**Recommendations:** Enforce data size quotas and schema validation; apply ingestion rate limits and back-pressure controls; monitor for pipeline anomalies and memory spikes in ETL workloads. --- @@ -625,22 +920,37 @@ CWE-20, CWE-116, CWE-200, CWE-359, CWE-345, CWE-353, CWE-494, CWE-276, CWE-284, **Summary:** Entry point of truth; without provenance checks, they introduce poisoned/unsafe content. -**Threats:** T01-SID, T01-DMP, T01-VEW, T01-MIS +#### Sensitive Information Disclosure (T01-SID) -**Targeted CWEs:** -CWE-200, CWE-359, CWE-522, CWE-345, CWE-353, CWE-494, CWE-829, CWE-918, CWE-502 +**Mapped CWEs:** [CWE-200](https://cwe.mitre.org/data/definitions/200.html), [CWE-359](https://cwe.mitre.org/data/definitions/359.html), [CWE-522](https://cwe.mitre.org/data/definitions/522.html) -### Sensitive Information Disclosure (T01-SID) -**Recommendations:** DLP at ingestion; least-privilege credentials; encrypt sensitive datasets. +**Rationale:** Ingestion processes may inadvertently capture and store sensitive data such as PII, API keys, or confidential content without encryption or access control, leading to downstream exposure. -### Data/Model Poisoning (T01-DMP) -**Recommendations:** Signature/hash checks; reputation scoring; golden-set cross-validation. +**Recommendations:** Implement DLP scanning at ingestion; enforce least-privilege credentials for ingestion pipelines; encrypt sensitive datasets in transit and at rest. -### Vulnerable External Workflow (T01-VEW) -**Recommendations:** Proxy + allowlists; forbid unsafe formats; isolate connectors. +#### Data/Model Poisoning (T01-DMP) -### Misinformation (T01-MIS) -**Recommendations:** Reliability scoring; ground-truth cross-referencing; drift monitoring. +**Mapped CWEs:** [CWE-345](https://cwe.mitre.org/data/definitions/345.html), [CWE-353](https://cwe.mitre.org/data/definitions/353.html), [CWE-494](https://cwe.mitre.org/data/definitions/494.html) + +**Rationale:** Attackers can insert poisoned or manipulated data into ingestion sources, corrupting the model’s training corpus or runtime cache, leading to bias or hidden backdoors. + +**Recommendations:** Enforce digital signatures and hash verification for ingested datasets; apply source reputation and provenance scoring; perform golden-set cross-validation to detect inconsistencies or anomalies. + +#### Vulnerable External Workflow (T01-VEW) + +**Mapped CWEs:** [CWE-829](https://cwe.mitre.org/data/definitions/829.html), [CWE-918](https://cwe.mitre.org/data/definitions/918.html), [CWE-502](https://cwe.mitre.org/data/definitions/502.html) + +**Rationale:** Ingestion connectors or pipelines that pull data from external systems may process untrusted or malformed content, enabling SSRF, deserialization attacks, or malicious payload execution. + +**Recommendations:** Use egress proxies and strict domain allowlists; reject unsafe data formats or content types; isolate ingestion connectors and third-party integrations in sandboxed environments. + +#### Misinformation (T01-MIS) + +**Mapped CWEs:** [CWE-345](https://cwe.mitre.org/data/definitions/345.html), [CWE-353](https://cwe.mitre.org/data/definitions/353.html) + +**Rationale:** Unverified or low-quality data sources may introduce false, biased, or adversarial information into training or analysis pipelines, degrading model accuracy and trustworthiness. + +**Recommendations:** Apply reliability and reputation scoring for all data sources; cross-reference new data against ground-truth sets; perform continuous drift and consistency monitoring across ingestion pipelines. --- @@ -648,20 +958,36 @@ CWE-200, CWE-359, CWE-522, CWE-345, CWE-353, CWE-494, CWE-829, CWE-918, CWE-502 **Summary:** Outside the trust boundary; major vectors for poisoning, leakage, and misinformation. -**Threats:** T01-MIMI, T01-SID, T01-DMP, T01-MIS -**Targeted CWEs:** -CWE-200, CWE-359, CWE-522, CWE-345, CWE-353, CWE-494, CWE-918, CWE-829 +#### Model Inversion / Membership Inference (T01-MIMI) -### Model Inversion / Membership Inference (T01-MIMI) -**Recommendations:** Privacy-preserving APIs; throttle/detect anomalies; k-anonymity/data minimization. +**Mapped CWEs:** [CWE-200](https://cwe.mitre.org/data/definitions/200.html), [CWE-359](https://cwe.mitre.org/data/definitions/359.html), [CWE-522](https://cwe.mitre.org/data/definitions/522.html) -### Sensitive Information Disclosure (T01-SID) -**Recommendations:** Secret managers; token rotation; TLS + mutual auth. +**Rationale:** Adversaries may exploit external model endpoints or shared datasets to infer private training data or reconstruct sensitive inputs through repeated probing or correlation analysis. -### Data/Model Poisoning (T01-DMP) -**Recommendations:** Data signing/ checksums; cross-validate with references; vendor trust contracts. +**Recommendations:** Deploy privacy-preserving APIs with data minimization; implement throttling and anomaly detection on external access; apply k-anonymity and differential privacy where feasible. -### Misinformation (T01-MIS) -**Recommendations:** Source reliability scores; ground-truth validation; human review for high-impact feeds. +#### Sensitive Information Disclosure (T01-SID) + +**Mapped CWEs:** [CWE-200](https://cwe.mitre.org/data/definitions/200.html), [CWE-359](https://cwe.mitre.org/data/definitions/359.html), [CWE-522](https://cwe.mitre.org/data/definitions/522.html) + +**Rationale:** External data integrations or shared access credentials can leak secrets or confidential information through exposed endpoints or weak encryption. + +**Recommendations:** Manage all credentials through secret managers with rotation policies; enforce TLS with mutual authentication for external data exchanges; restrict and log all token usage. + +#### Data/Model Poisoning (T01-DMP) + +**Mapped CWEs:** [CWE-345](https://cwe.mitre.org/data/definitions/345.html), [CWE-353](https://cwe.mitre.org/data/definitions/353.html), [CWE-494](https://cwe.mitre.org/data/definitions/494.html) + +**Rationale:** External sources or third-party datasets may inject malicious data or corrupted models that contaminate pipelines, resulting in degraded accuracy or compromised behavior. + +**Recommendations:** Require data signing and checksum verification from external providers; cross-validate new data with reference or golden sets; establish vendor trust and supply-chain integrity contracts. + +#### Misinformation (T01-MIS) + +**Mapped CWEs:** [CWE-345](https://cwe.mitre.org/data/definitions/345.html), [CWE-353](https://cwe.mitre.org/data/definitions/353.html) + +**Rationale:** External feeds and open data sources may provide low-reliability or adversarial content that misguides training or inference outputs, spreading false narratives or bias. + +**Recommendations:** Assign reliability scores and reputation metrics to external sources; validate information against ground-truth datasets; require human review for high-impact or public-facing data feeds. ---