davisAIgov · AI Harness
AI Harness Detection Coverage Map
See which agent actions ten Elastic rules can observe, where the AI Harness needs its own telemetry, and what to verify before enabling automated containment.
Start with the coverage summary, select a node in the map, or review the disclaimer and validation scope.
01 · BOTTOM LINETen rules provide direct signals for three Harness components
Extensions and MCP (6.09), tool dispatch (6.10) and downstream systems (6.11) can produce matching process, file or network events.
Authorization (6.07), credential brokering (6.08), audit (6.12), oversight (6.15) and attestation (6.16) have related endpoint indicators. The control’s own decisions require separate telemetry.
Eleven components have no direct or proxy signal in this ten-rule subset.
These rules look for observable host activity or upstream security alerts. Elastic Defend supplies many of the signals. Some rules also accept other endpoint or Windows event sources. Consult the pinned rule definitions for each rule’s inputs and query.
An agent can use an approved tool on untrusted intent while producing ordinary-looking endpoint activity. Check application records for the content source, policy decision, delegated identity and downstream outcome.
The proposed design joins endpoint telemetry with independently collected Harness events such as policy decisions, retrieval scope, delegation and budget reservations. A versioned release manifest can anchor deterministic checks where the approved configuration and the observed runtime are comparable.
The map shows 76 distinct techniques and sub-techniques across 16 ATLAS tactics. The editorial rule crosswalk assigns 9 distinct ATLAS IDs; 7 also appear in the technique map. The pinned Elastic files publish MITRE ATT&CK tags.
How to close the telemetry gaps
Proposed implementation routes for the 11 components with no signal in this rule subset.
Orchestration, model calls, guardrails, delegation and evaluation. Verify what native telemetry the deployed tools emit.
Retrieval, memory and budget decisions. Add an observation or enforcement point where existing telemetry is insufficient.
Inputs, context assembly and recourse. Define and instrument the evidence needed to reconstruct those decisions.
Across all 19 components, nine are classified as ingest, six as insert and four as build. These are planning estimates based on the reference architecture. A deployed gateway, broker or audit service can change a component’s route.
Three joins drive the design: deployed runtime to approved release, delegated scope to parent authority, and untrusted content to downstream action. Validate the join keys and the loss rate before treating a missing record as a finding.
02 · THE MAPRules, Harness components and ATLAS tactics
Select a rule, component or tactic to isolate its connections. Click again or press Escape to clear. The selector provides the same detail without navigating the diagram.
Nothing selected. All 19 components, ten rules and sixteen tactics are shown.
03 · THE TEN RULESWhat each rule sees, and where it stops
Rule name, severity, default risk score and query type come from the pinned Elastic rule files. The published tags are MITRE ATT&CK. ATLAS links and Harness coverage grades are davisAIgov’s interpretation.
| # | Rule and what it sees | Severity | Query type · data scope | ATLAS crosswalk · davisAIgov | Components | Where it stops |
|---|---|---|---|---|---|---|
| R1 | Potential Destructive AWS CLI Command Executed by GenAI Agent Matches selected destructive AWS CLI commands launched from eligible process lineages. | high risk 73 | KQL query 1 listed data patterns | T0053 T0101 | 6.10 6.11 6.08 | Matches command patterns. Check downstream audit records to determine whether the API call succeeded and what changed. |
| R2 | GenAI Process Connection to Suspicious Top Level Domain Flags eligible process connections to a listed top-level domain. | medium risk 47 | EQL 3 listed data patterns | T0072 | 6.11 6.09 | Evaluates the configured domain suffixes. A match indicates network activity for triage and does not establish command and control. |
| R3 | Unusual Process Modifying GenAI Configuration File Flags unusual writers of monitored GenAI configuration files. | medium risk 47 | New terms 1 listed data patterns | T0081 | 6.09 6.16 6.07 | Flags matching file changes. Compare the changed entry with approved configuration. |
| R4 | Elastic Defend Alert from GenAI Utility or Descendant Surfaces qualifying Elastic Defend alerts linked to selected GenAI utilities or their process lineage. | critical risk 99 | ES|QL Defend alerts | T0053 T0011 | 6.10 6.11 6.12 | Requires a qualifying underlying Elastic Defend alert. It inherits the input alert and process-lineage limits. |
| R5 | GenAI Process Accessing Sensitive Files Flags matching access to sensitive files or creation and write events in monitored paths, subject to platform support. | high risk 73 | EQL 1 listed data patterns | T0085 T0085.001 T0055 | 6.08 6.10 | Observes matching endpoint file activity. Elastic documents Linux support for creation events, with access events unavailable at this revision. |
| R6 | GenAI Process Compiling or Generating Executables Flags matching compiler or packaging activity started by selected GenAI processes. | medium risk 47 | EQL 5 listed data patterns | T0053 | 6.10 6.11 | Flags matching compilation or packaging activity. Assess the workflow and executable provenance. |
| R7 | GenAI CLI Started with Unsafe Permission Bypass Flags matching command-line permission bypass or automatic approval flags. | medium risk 47 | EQL 1 listed data patterns | T0081 T0053 | 6.07 6.15 | Observes matching launch arguments. Verify runtime approval policy separately. |
| R8 | GenAI or MCP Server Child Process Execution Records matching child process execution from selected GenAI or MCP activity for correlation. | low risk 21 · building block | EQL 4 listed data patterns | T0053 | 6.10 6.09 6.12 | A building-block rule for matching child process events. Correlate it with task and configuration evidence. |
| R9 | GenAI Process Performing Encoding/Chunking Prior to Network Activity Correlates eligible encoding or archive process activity with an outbound connection. | medium risk 47 | EQL 6 listed data patterns | T0086 | 6.10 6.11 | Correlates matching local process activity and network activity. The sequence alone cannot establish what data was sent. |
| R10 | GenAI Process Connection to Unusual Domain Flags eligible macOS network events where a domain is new under the configured query and history. | medium risk 47 | New terms 1 listed data patterns | T0072 | 6.11 6.09 6.16 | Flags new terms for the configured macOS events. Review the lookback, process identity, destination and business context. |
04 · COVERAGE BY COMPONENTThree tiers of detection for each of the nineteen components
Tier A describes the ten-rule sample. Tier B lists additional Elastic candidates only where a matching rule can be linked at the pinned revision. Tier C contains 39 proposed specifications. Each specification needs verified telemetry, tests and an assigned response owner before use.
| Component | A · the ten rules | B · Other Elastic candidates | C · HoundBytes AI-DDR, proposed | Proposed response · approval required |
|---|---|---|---|---|
| Boundary | ||||
| 6.01 Inputs & external content | Dark Content provenance is outside the ten selected rule queries. |
| Quarantine the source record, tag it untrusted and hold dependent actions. | |
| Control plane | ||||
| 6.02 Orchestrator / agent loop | Dark Planning and agent recruitment need runtime or orchestration records. |
| Suspend the orchestration and withdraw the agent's discoverability. | |
| 6.05 Context assembly | Dark Context segments and truncation require application instrumentation. |
| Roll the template back to its attested digest. | |
| 6.06 Input / output guardrails | Dark Guardrail verdicts require application or provider telemetry. |
| Restore the guardrail configuration and strip external references from output. | |
| 6.00 Generative model | Dark Model identity, response and usage need inference or gateway logs. |
| Pin to the approved deployment and hold consequential actions. | |
| Data plane | ||||
| 6.03 RAG subsystem | Dark Verify retrieval scope and index changes with storage or application records. |
| Quarantine the entries and revert to the last attested index snapshot. | |
| 6.04 Memory & state | Dark Memory writes and checkpoints need application or store telemetry. |
| Freeze memory writes and purge entries by lineage. | |
| 6.11 Sandbox / downstream systems | Direct R1 R2 R4 R6 R9 R10 Matching egress, compilation and process events can help identify downstream activity. Confirm the outcome at the target. |
| Isolate the host or block egress, then restore from a verified backup. | |
| Identity & action plane | ||||
| 6.07 Policy / authorization gate | Partial R3 R7 Configuration changes and bypass flags are proxy signals. Collect gate decisions independently. |
| Set the gate to default deny for mutative tools. | |
| 6.08 Credential broker | Partial R1 R5 Sensitive file activity is a proxy. Collect broker records for issued credentials and their scopes. |
| Revoke or narrow the token and rotate any exposed secret. | |
| 6.09 Extensions / MCP / skills | Direct R2 R3 R8 R10 Matching child process, network and file events can support endpoint visibility. Registry checks need separate evidence. |
| Disable the extension and pin to the attested version. | |
| 6.10 Tool dispatcher | Direct R1 R4 R5 R6 R8 R9 Matching host process activity can support dispatch visibility. Hosted tool calls need telemetry from the execution path. |
| Kill the process tree and block the tool class. | |
| Evidence plane | ||||
| 6.12 Observability & audit | Partial R4 R8 The rules supply correlatable alerts. Assess record integrity and producer health at the collection layer. |
| Fail closed: no evidence, no consequential action. Switch to the secondary collector. | |
| Lifecycle plane | ||||
| 6.13 Identity & delegation | Dark Delegation needs identity, scope and lineage evidence. |
| Revoke the delegation chain, including running children. | |
| 6.14 Evaluation & conformance | Dark Conformance needs release and evaluation evidence. |
| Withdraw the release approval and revert to the last conformant configuration. | |
| 6.15 Human oversight console | Partial R7 Launch arguments provide a proxy for approval settings. Collect actual reviewer decisions and interrupt outcomes. | Verify platform and available rule inventory during onboarding. |
| Escalate to a second independent approver and interrupt out of band. |
| 6.16 AI BOM & attestation | Partial R3 R10 Configuration and destination anomalies can trigger review. Compare actual runtime state with the registered baseline. |
| Deny activation and re-attest. | |
| 6.17 Authority & consumption | Dark Budget reservations, enforcement and reconciliation require task or ledger evidence. |
| Freeze the budget and cap the rate. | |
| 6.18 Transparency & recourse | Dark Use business-process records for notices, decisions and recourse. | Verify platform and available rule inventory during onboarding. |
| Withhold publication and issue a corrected notice. |
05 · MARKET TOOLING BY COMPONENTReference tooling and evidence to collectAdded in v1.1
Each row pairs a Harness function with one documented example, a signal to validate and a proposed route to coverage. Actual telemetry depends on the deployed version, plan and configuration.
| Component and function | Documented example | Evidence to validate | Proposed route | Proposed implementation |
|---|---|---|---|---|
| Boundary | ||||
| 6.01 Dark Inputs & external content Brings prompts, files, web pages, email, API and tool responses, and other agents' messages into the system. | Microsoft Azure Prompt Shields | Record the request identifier and scanner verdict where the integration exposes them. | Build Define the event or evidence record. |
|
| Control plane | ||||
| 6.02 Dark Orchestrator / agent loop Plans, selects tools, and discovers and recruits other agents. | OpenTelemetry agent span conventions | Collect agent spans from instrumented runtimes. Confirm emitted fields and version per SDK. | Ingest Start with an available signal. |
|
| 6.05 Dark Context assembly Joins instructions, user intent, retrieved passages, memory and tool results into one token stream. | Langfuse Prompt Management | Record prompt versions. Carry segment-level provenance and truncation as application fields where needed. | Build Define the event or evidence record. |
|
| 6.06 Dark Input / output guardrails Filters content going in and coming out. Advisory to the orchestrator. | Amazon Bedrock Guardrails | Capture available verdicts at the integration and correlate them to the action trace. | Ingest Start with an available signal. |
|
| 6.00 Dark Generative model Inference. Proposes actions and holds no authority. | OpenTelemetry model inference conventions | Capture available request and response model identifiers and token counts. | Ingest Start with an available signal. |
|
| Data plane | ||||
| 6.03 Dark RAG subsystem Retrieves permitted knowledge from indexed sources. | Elastic document-level security | Verify retrieval permissions and record returned object IDs with the initiating principal where policy permits. | Insert Add or validate an observation point. |
|
| 6.04 Dark Memory & state Holds thread, persistent and shared state. | LangGraph persistence | Confirm checkpoint scope and retention, then instrument reads and writes that need independent review. | Insert Add or validate an observation point. |
|
| 6.11 Direct Sandbox / downstream systems Where code runs and where changes land. | AWS CloudTrail | Check which downstream events are logged for the enabled AWS resources and verify the action outcome. | Ingest Start with an available signal. |
|
| Identity & action plane | ||||
| 6.07 Partial Policy / authorization gate Decides independently whether a proposed action may proceed. | Open Policy Agent decision logs | Ingest policy decisions from participating enforcement points and redact sensitive decision inputs. | Insert Add or validate an observation point. |
|
| 6.08 Partial Credential broker Issues scoped, expiring, per-call credentials. | HashiCorp Vault audit devices | Collect configured broker audit records. Confirm which lease or token identifiers can be joined to the task. | Ingest Start with an available signal. |
|
| 6.09 Direct Extensions / MCP / skills Brings tools into the harness with their own code and their own trust. | Docker MCP Gateway | Compare observed tools and server configuration with the approved catalogue for the release. | Insert Add or validate an observation point. |
|
| 6.10 Direct Tool dispatcher Executes the tool call. The privileged edge. | OpenTelemetry GenAI tool execution | Pair tool execution spans with authorization and downstream results where equivalent identifiers can be propagated. | Ingest Start with an available signal. |
|
| Evidence plane | ||||
| 6.12 Partial Observability & audit Keeps the independent record of what happened. | Elastic Distribution of OpenTelemetry Collector | Validate trace export, sampling, retention and producer-loss reporting in the configured collector. | Ingest Start with an available signal. |
|
| Lifecycle plane | ||||
| 6.13 Dark Identity & delegation Tracks who each agent acts for and how authority is handed down. | IETF RFC 8693 OAuth 2.0 Token Exchange | Validate actor and subject claims when available, and add task scope and parent IDs where the identity system does not. | Ingest Start with an available signal. |
|
| 6.14 Dark Evaluation & conformance Tests the exact configuration before and after release. | Promptfoo evaluation docs | Retain evaluation results with the test suite, release identifier and configured pass thresholds. | Ingest Start with an available signal. |
|
| 6.15 Partial Human oversight console Lets a named person approve, reject or interrupt. | LangChain human-in-the-loop | Record approvals and interrupts independently with a decision ID, expiration and scope. | Insert Add or validate an observation point. |
|
| 6.16 Partial AI BOM & attestation Registers and verifies the release: model, tools, prompts and policies. | CycloneDX ML-BOM | Use the inventory as input to attestation, then check runtime state against what was approved. | Build Define the event or evidence record. |
|
| 6.17 Dark Authority & consumption Bounds calls, tokens, time and spend as shared authority. | Cloudflare AI Gateway spend limits | When the chosen gateway exposes usage and limits, confirm enforcement scope and correlate to the task budget. | Insert Add or validate an observation point. |
|
| 6.18 Dark Transparency & recourse Connects a decision to a notice, an explanation and a route to contest. | NIST Generative AI Profile | Keep a decision record, a minimized notice and a recourse reference where appropriate for the use case. | Build Define the event or evidence record. |
|
06 · BUILD PLANSeven workstreams for expanding telemetry and testingAdded in v1.1
Start with the signal each deployment can provide. Define the events, join keys and policy enforcement points that are missing, then test their coverage against agreed scenarios.
| Workstream | What exists to build on | What HoundBytes would have to do | Components lit |
|---|---|---|---|
| 1Harness span ingest Stage 2 | OpenTelemetry GenAI conventions and a collector configured for the deployment. | Publish an AI-DDR span profile pinned to one conventions version. Ship Elastic ingest pipelines and data views for it. Keep content capture off by default and store hashes. Collect outside the agent's write authority. | 6.00 6.02 6.10 6.12 |
| 2Platform, identity and broker log packs Stage 1 | Provider audit exports, identity activity and credential broker records where enabled. | Package onboarding per platform. Normalize one agent identity across sources. Rebuild the delegation chain per trace and add the scope-subset check. | 6.06 6.08 6.11 6.13 |
| 3Certified control points Stage 2 | An approved MCP or AI gateway, policy decision service and other enforcement points as needed. | Approve gateway and retrieval control points against defined requirements. Validate parsers and instrument retrieval or memory operations where records are missing. | 6.03 6.04 6.07 6.09 6.15 6.17 |
| 4Attested baseline as a service Stage 2 | An asset inventory, release manifest and component-level verification record. CycloneDX ML-BOM is one inventory option. | Deliver harness step 0 as a service: inventory agents, generate and sign an agent-level BOM, load registry, envelope and budget into Elastic lookup indices, and re-attest on change. | 6.14 6.16 |
| 5Detection content, tested Stage 2 | The 39 proposed Tier C specifications and a controlled evaluation environment. | Prioritize specifications by telemetry availability. Use controlled exercises to validate expected alerts, false positives and known blind spots. | All components, through Tier C |
| 6Response connectors and verified stop Stage 3 | Response APIs, identity and policy enforcement points, and a documented approval matrix. | Build a connector for each. End every playbook with a stop-state check across children, credentials, jobs and memory. Agree a pre-authorization matrix by consequence class with each client. | All components, through Tier C |
| 7The missing schemas Stage 2 | Schemas for provenance, context lineage, approval, task identity and decision records tailored to the deployment. | Define these fields as part of the implementation profile. Document which fields are local extensions and how they are versioned. | 6.01 6.05 6.18 |
Implementation principles
- Enforce authority at the action boundary with independently controlled policy.
- Prefer scoped credentials with short lifetimes. Verify the revocation behavior in the actual deployment.
- Pin the telemetry schema and collect only the fields the detection and evidence plan needs.
- Measure coverage against the approved tool and agent inventory.
Where the design fits
- Endpoint, gateway and SaaS detections can supply different evidence about the same event.
- The proposed AI-DDR design correlates those events against an approved operating baseline and verifies the containment outcome.
- For partner evaluation, document who controls the data, which signals each product contributes and which response actions each team can authorize.
07 · DETECTION AND RESPONSE MODELProposed detection and response operating model
AI deviation detection and response (AI-DDR) is the operating model proposed here for a HoundBytes engagement using the Harness and an Elastic Security deployment. Define the approved scope, collect independent signals and decide ahead of time which interventions responders may take. Confirm platform integrations and delivery commitments in the statement of work.
Collect endpoint and Harness telemetry
Collect eligible host, cloud and platform signals. Instrument the gate, dispatcher, broker, retriever, memory store and delegation service where needed. Plan to correlate agent and tool events with a propagated trace or other durable join key. Verify data ownership and collection outside the agent’s write permissions.
Set an approved operating baseline
Use the approved inventory and release manifest (6.16), delegation envelope (6.13), usage limits (6.17) and conformance evidence (6.14). Compare observed runtime state with the approved scope. Use observed baselines for statistical deviations once the data is adequate.
Sort every signal into three classes
- Boundary violation. Check for a material action without the required decision record or a delegated scope outside the approved envelope. Use the agreed policy and confirm relevant telemetry.
- Behavioral deviation. Corroborate unusual tool selection, new destinations or unexpected call-chain depth before escalating.
- Evidence integrity issue. Check digest drift, missing producers or an uncorroborated agent claim. Route to the policy owner when evidence is insufficient.
A policy deviation can be operational error or malicious activity. Preserve the evidence needed to determine intent and act on the immediate business risk.
Correlate signals into cases
Join endpoint process lineage, tool and agent traces, identity and delegation records, and data provenance where the evidence permits. Capture the policy boundary in question and the confidence in the conclusion.
Respond on a pre-authorized ladder
The ladder is a decision template for customer approval. Verify permissions, notice, reversal paths and stop conditions before enabling automatic actions.
Level Trigger Action Executed by Human role 0Observe Building-block or low-confidence signal Annotate the trace, raise sampling, add the event to the case graph Triage service None 1Constrain Behavioral deviation with corroboration Gate to default deny for mutative tools; narrow token scope; withdraw discoverability; cap the budget Authorized gate or broker, if configured Notified 2Interrupt Boundary violation, or deviation in a high consequence class Revoke the delegation chain including children; kill the process tree; disable the extension; isolate the host Authorized broker, dispatcher or endpoint response connector Approves irreversible or business-critical action under the agreed policy. 3Recover Stop state verified Restore configuration to the attested digest; purge memory and index entries by lineage; rotate secrets; rerun evaluation System and release owners Release authority signs the return to service 4Govern Every case at level 2 or above Decision record, notice and recourse assessment, baseline update, board reporting where material Case owner Accountable owner Verify containment and document recovery
Verify observed stop state across the components in scope, including active child agents, credentials and scheduled jobs. Check downstream outcomes and document recovery actions. Re-run the agreed evaluation before the release owner restores service.
Preserve evidence for review
Define capture, access, retention and chain-of-custody procedures. Where appropriate, record cryptographic digests and provide repeatable verification. Agree evidence handling with counsel and the system owner.
Worked example · child-process alert
Illustrative scenario adapted from the supplied map. An editor’s extension host starts a Python interpreter on a managed workstation. Elastic R8 provides a low-severity building-block signal for eligible child-process events.
- Observe. Check whether the process lineage matches the workstation’s approved workflow.
- Constrain if corroborated. If a related GenAI configuration change is flagged, check recorded approval and the tool registry.
- Consider an interrupt. If related alerts suggest access to credentials and suspicious outbound activity, reconcile with downstream evidence and the approved response policy before drawing a conclusion.
A building-block signal becomes actionable when the surrounding evidence and response threshold support it.
Measures that show the program works
- Components with at least one live detection, out of nineteen
- Share of agents running against an attested baseline
- Share of material actions with a gate decision record
- Producer coverage and recorded loss, by source
- Time to verified containment, measured to observed stop
- False-containment rate, by ladder level
What to report to the board
- Which material actions have verifiable decision and stop records.
- Which components lack a validated signal and who owns remediation.
- Which interventions have been approved and tested, and which still require case-by-case review.
- What evidence is available for counsel, auditors or insurers, subject to access and retention rules.
08 · ENGAGEMENT PATHShadow Mode, Pattern Capture, Agentic Cutover
A proposed three-stage engagement. Use the map to record starting coverage, validate new sources and approve the response scope.
See the action edge
Start in observe-only mode. Enable eligible rules, inventory agents and tools, and document which signals and permissions are actually available.
Workstream 2 v1.1
Light the dark components
Instrument the gate, dispatcher and broker first. Register the baseline and test proposed rules in alert-only mode before assigning any automated action.
Workstreams 1, 3, 4, 5 and 7 v1.1
Validate interruption and recovery
Enable the approved responses only after exercising permissions, stop-state verification, recovery and evidence handling. Reserve irreversible actions and return-to-service decisions for the designated owners.
Workstream 6 v1.1
09 · LIMITS AND VALIDATION NOTESDisclaimer and validation scope
- Source checks. On 20 November 2026, we compared the ten rule names, severity, default risk score, query type and published framework against the pinned Elastic files. We checked the displayed ATLAS tactic and technique identifiers and titles against MITRE ATLAS 2026.10. The mappings between rules, techniques and Harness components are davisAIgov’s analytical judgments.
- Meaning of coverage. Direct means a rule can observe matching activity associated with a component. Partial means only a related or proxy signal is shown. Dark means the analysis found no signal in the ten-rule subset. These grades do not measure detection probability or tested effectiveness. Tooling routes are implementation estimates for the reference design.
- Implementation status. The 39 Tier C detections, telemetry joins and response ladder are specifications. They have not been implemented or tested against production data for this document. Confirm data schemas and loss rates, validate rules on representative traffic and exercise rollback before enabling containment.
- Vendor scope. Linked product examples document relevant features. Verify version, region, licensing, configuration, data handling and response permissions for each deployment. Named vendors are examples. No sponsorship, partnership, certification or comparative ranking is represented by inclusion.
- Worked example and evidence. The example is illustrative. The underlying alert record was not independently available for verification. Evidence handling depends on the engagement, data sources, integrity controls and applicable requirements.
- Use. This document supports architecture and discussion. A deployment requires its own controls assessment, legal review when applicable, and written response authorization. Review primary sources.
10 · SOURCESPrimary sources and verification anchors
- Elastic detection-rules repository, commit c59ac34. Pinned records for R1–R10 and linked Tier B candidates. Each linked rule table entry opens its exact source.
- Elastic R8 reference. Building-block scope for the worked example.
- MITRE ATLAS 2026.10 release. Release with tactic and technique identifiers used here.
- MITRE ATLAS 2026.10 machine-readable dataset. Identifier and title reference.
- OpenTelemetry Generative AI semantic conventions. Development status and version-dependent telemetry vocabulary.
- davisAIgov AI Harness. Site companion. The supplied map uses the reference model’s v13.4 numbering.
- davisAIgov Regulatory Library. Companion threat catalogue and source references.
Each row in Reference tooling links to its primary source. External source and site links require internet access.