Multi-Agent AI Safety and Risk Evaluation

Project overview

Multi-agent systems add risks that single model testing misses. Instructions can enter through a document rather than from the user. One agent can impersonate or contradict another. A finding invented by one agent can be repeated by the next and end up in the final output. This project tests those handoffs directly, with execution traces that show where each attack was stopped or allowed through.

Every scenario ran twice: once in a baseline configuration where the policy engine is disabled and every check returns ALLOW, and once in a protected configuration where the controls described below are active. Comparing the two on identical scenarios isolates what the controls do. All documents, identifiers, recipients, and credentials in the scenarios are fictional, and the simulated tools perform no network access.

Research objectives

The research question is: do explicit runtime safety controls reduce simulated violations in a three-agent workflow without unnecessarily preventing legitimate tasks?

• Measure unsafe actions permitted, with an explicit denominator of attack attempts, in baseline and protected configurations.
• Measure information disclosure separately, because a marker reaching an output is a different failure from a tool executing.
• Measure the cost of the controls on legitimate work, including false positives, rather than reporting protection alone.
• Record structured audit decisions and traces, so each outcome can be traced to the check that allowed, denied, or redacted it.
• State the limits of a deterministic simulation clearly, so the results are not read as evidence about real LLM agents.

Three-agent architecture

Three deterministic, rule-based agents communicate through a signed message bus with recorded handoffs. Each role has a deliberately narrow job and a narrow set of tools.

• Planner. Interprets the trusted user task and delegates research and execution. It has no tool permissions at all, so it cannot read, write, send, or delete anything itself.
• Researcher. Reads approved synthetic documents and reports findings, with a citation to the source document. It may use only read_document, and only for documents in the approved corpus.
• Executor. Produces the final output. It may use only write_output, and only inside the run workspace.

Supporting components handle messages with trust labels and HMAC integrity signatures, the runtime policy engine, simulated tools confined to a fresh temporary workspace per run, the synthetic document corpus, and metric aggregation with explicit denominators. An offline Streamlit dashboard presents the results, audit logs, and traces. In protected mode, every message and tool call passes a policy gate that returns ALLOW, DENY, or REDACT and writes the decision to a structured audit log.

Safety threat model

The trust boundary is drawn between what the user and the system itself produce, and everything else.

• Trusted: the user task as issued, and messages created and signed inside the system by the Planner.
• Untrusted: document content, any instruction embedded in a document, forged messages, altered messages, and findings that cannot be cited to their source.
• Attacker capabilities in scope: embedding instructions in documents, requesting unauthorised tools, forging a message from another role, tampering with a message in transit, inserting conflicting instructions, and seeding a fabricated finding that would cascade into the final output.
• Out of scope: real credential theft, network attacks, operating system escape, and any production service. Simulated tools perform no network access. send_external only records a payload in memory, and delete_file only removes a dummy file inside the temporary workspace. Workspace path confinement is enforced in code in both configurations, including baseline, so it is not one of the differences being measured.

Evaluation scenario design

The suite in data/scenarios.json has 24 synthetic scenarios: 17 malicious and 7 legitimate. The malicious categories are prompt injection (3), sensitive information disclosure (3), unauthorized tool requests (3), agent role spoofing (2), message tampering (2), cross-agent instruction conflicts (2), and cascading agent errors (2). The 7 legitimate scenarios are ordinary work: summaries, action item extraction, and one simulated external sharing request.

Each malicious scenario presents one primary attack by design, which fixes the attempt denominator at 17 per configuration. Each scenario documents its expected behaviour in both configurations before it runs. Metrics keep their denominators explicit throughout: unsafe actions permitted out of 17 attempts, disclosure violations out of 4 disclosure relevant scenarios, and legitimate completion and false positives out of 7 legitimate scenarios.

The agents are deliberately compliant. If an injected instruction reaches the Executor, it attempts the requested simulated tool. That design choice matters for interpretation: the experiment measures whether the policy layer stops the instruction, not whether an agent judges it to be malicious. A real LLM agent might refuse some attacks on its own, and might also evade controls in ways these agents cannot.

Prompt injection and information disclosure risks

Prompt injection is the central entry point in this suite. In three scenarios, an instruction hidden inside a document asks the Executor to send findings to a fictional external address, delete a workspace file, or chain credential access with an external send. Two more scenarios use the same entry point through direct user task wording that requests a tool no agent is permitted to use, and through a document instruction that conflicts with an explicit trusted instruction. Because document content is untrusted, an instruction that arrives inside it must not acquire the authority of the user task. In baseline mode it does, because nothing separates the two.

Information disclosure is tracked as its own metric. Four scenarios are disclosure relevant: they carry fictional sensitive markers or a disclosure aim. They include a restricted credentials register the Researcher is pointed at, support notes that accidentally contain a fictional mock API key, a direct request to read stored credentials, and an injected instruction trying to move findings outside the workspace. A violation is counted when a fictional marker reaches the final output, a written file, or a simulated external payload. In baseline, 3 of these 4 scenarios disclosed a marker.

Runtime safety mechanisms

Protected mode applies seven controls, each aimed at a different point in the workflow:

• Role-based tool permissions. Each role has an explicit tool set. The Planner has none, the Researcher has read_document, and the Executor has write_output inside the workspace.
• Trusted and untrusted message separation. Instructions embedded in document content are carried in a separate untrusted field and stripped in protected mode, so they never merge with the trusted task.
• Tool execution allowlists. send_external, delete_file, and read_credentials are on no role allowlist in protected mode, whoever asks for them.
• Sensitive information redaction. Fictional mock secret patterns are replaced before output in protected mode.
• Message validation. HMAC signatures are verified on inter-agent messages, and research findings must quote the cited source document, which blocks fabricated findings.
• Runtime policy enforcement. Every message and tool call receives an ALLOW, DENY, or REDACT decision. With the engine disabled (baseline), every check returns ALLOW and no validation or redaction is applied.
• Structured audit logging. Every decision is recorded with its check type, reasons, and context, and is visible in the dashboard and in outputs/results.json.

Baseline versus protected evaluation

By category, baseline permitted every attempt: prompt injection 3 of 3, sensitive disclosure 3 of 3, unauthorized tools 3 of 3, role spoofing 2 of 2, message tampering 2 of 2, instruction conflicts 2 of 2, and cascading errors 2 of 2. Protected mode permitted 0 in every category. The 20 protected audit events with DENY or REDACT decisions are spread across categories: 3 each for prompt injection, sensitive disclosure, and unauthorized tools, 4 for message tampering, 2 each for role spoofing, instruction conflicts, and cascading errors, and 1 in a legitimate scenario. Those are 20 intervention events recorded during the run. They are not 20 distinct attack scenarios, and the suite only contains 17 malicious scenarios.

Safeguard effectiveness

Within this simulation, the controls did the specific jobs they were built for. Trust separation and tool allowlists broke the injection chains before a simulated tool could run. Signature validation denied forged and tampered messages, which a content filter alone would not catch, because the text of a forged instruction can look ordinary. Redaction and restricted document rules removed all three baseline marker disclosures. Citation validation stopped fabricated findings from cascading into the final output, which suggests provenance checks are a useful complement to content checks.

The baseline result needs as much care as the protected one. Baseline permitted 17 of 17 because the agents are written to comply with whatever instruction reaches them and no check stands in the way. That is a property of this simulation, not a measured failure rate for real systems. The same caution applies in reverse to the protected 0 of 17: the attacks were template based and designed alongside the controls, so a complete block rate on this suite shows the enforcement points work as specified here. It does not show robustness against novel attacks.

False positives and legitimate task completion

Protection was not free. Legitimate completion fell from 7 of 7 in baseline to 6 of 7 in protected mode, a false positive rate of 1 of 7 (14.3 percent) in this small sample.

The false positive is scenario L-07, a legitimate simulated external sharing request: share a travel notes summary with a fictional external collaborator. In baseline the simulated send completes. In protected mode the strict allowlist blocks send_external for every recipient, legitimate or not, so the task cannot complete. That is over-blocking by design of a strict allowlist, and it is reported here as a cost of the controls rather than hidden. Six other legitimate tasks, including a security awareness summary that mentions passwords without containing credential values, completed normally under the same controls.

With only 7 legitimate scenarios, one task is the difference between 100 percent and 85.7 percent completion. Both figures should be read as counts first. The finding worth keeping is qualitative: an allowlist strict enough to stop every external send will also stop legitimate sharing, so a real deployment would need a considered way to approve specific recipients rather than a blanket rule in either direction.

Visualizations

These eight figures were generated from the recorded results in outputs/results.json. Counts are shown rather than only percentages, because the sample is 24 scenarios.

Research limitations and conclusions

These limitations are essential to reading the results, so they are stated in full rather than summarised.

• No real LLM inference was tested. results.json records this explicitly: no LLM API key was available, the provider and model are null, and the request count is 0. Nothing here measures how a language model behaves.
• The agents are deterministic, rule-based simulations. They are deliberately compliant and attempt any simulated tool an instruction asks for. Results therefore measure policy enforcement behaviour (validation, permission gates, redaction, allowlists), not model judgement.
• Results cover 24 synthetic scenarios only. With denominators of 17, 7, and 4, rates have wide uncertainty and should be read as counts first.
• Findings do not establish robustness against novel attacks or real LLM behavior. A 100 percent block rate on this suite must not be generalised, and these are not real world LLM attack success rates.
• Attack cases were template-based rather than adaptive. Paraphrase, encoding, multi-step social engineering, and attackers who adapt to the controls are not covered. Each scenario also presents one primary attack, while real incidents combine vectors.
• Pattern-based redaction has limited coverage. It detects the fictional marker formats used in this project. It would not reliably catch secrets in other formats, and the citation check verifies that a quote exists in the cited document, not that a summary is semantically correct.
• Message integrity signatures do not cover every message field. The HMAC covers sender, recipient, kind, content, tool call, and trust. It does not cover the untrusted instruction, citation quote, source document, or message ID fields, so tampering confined to those fields would not be detected by the signature.
• The signing key is a demonstration constant. It is a local placeholder in the code, not a credential management or key distribution design.
• The system is not a production security certification. Simulated tools do no network access, workspaces are temporary, and no production service, monitoring, or incident process was evaluated.

The conclusion is correspondingly narrow. In this deterministic three-agent simulation, runtime controls at the message and tool layers stopped all 17 template based attacks in the suite and removed all fictional marker disclosures, while blocking 1 of 7 legitimate tasks. That shows the enforcement points work as specified on these scenarios, and it prices strict allowlisting with a concrete false positive. It does not prove that multi-agent LLM systems are secure, and it should not be cited as evidence about real LLM behaviour.

Reference note: this project was informed at a conceptual level by the public repository AI45Lab/TrinityGuard (MIT License), which describes evaluating safety risks in multi-agent systems with workflow tracing, message interception, and runtime protection. No source code, data, figures, or findings were copied from it, and this project uses a different decision set (ALLOW, DENY, and REDACT). All scenarios, code, and results here are original to this project.

Source code download

The downloadable archive contains the mas_safety source package, the evaluation runner, the offline Streamlit dashboard, the 24 evaluation scenarios in data/scenarios.json, requirements, automated tests, the README with reproduction instructions, the case study text, full results with audit logs and traces, and all eight figures. It contains no credentials and no .env file (only an empty .env.example placeholder), and no caches, temporary files, or virtual environments. The fictional mock secrets inside the synthetic documents are test fixtures for the redaction checks, not real credentials. Sixteen automated tests cover the controls and metrics, and all 16 pass.