Back to Aginiti Redteam – Adaptive AI Security Red-Teaming Engine
All Projects•Research Write-Up
AI Security & Adversarial Machine Learning

Aginiti Redteam: Adaptive Multi-Agent AI Security Red-Teaming Engine

Evidence-Graph-Driven Adversarial Evaluation of Agentic AI & RAG Architectures

Author(s)

Muhammad Hammad Irfan

Academic / Industry Supervision

Dr. Ahmad Faraz Khan (DevNeuron)

Institution & Collaborators

DevNeuron

Forward-Deployed AI Engineering Internship Research

Timeline & Funding

March 2026

View Source CodeInteractive Demo & Screenshots
Fewer Attempts
~5x
Adaptive planner needed 9.7 attempts on average vs. 50.7 for fixed-order selection, on a target with 8 defense layers
Published Attacks
4 Papers
ICLR 2026, IEEE TIFS 2026, ACM CCS 2025 methodologies implemented from source
Verification Stages
3 Steps
Deterministic scan, LLM judge, independent ground-truth comparison
Enterprise Defenses
8 Layers
Benchmarked against a target with RBAC, output redaction, rate limiting, and more

Abstract & Executive Summary

Conventional red-teaming of agentic LLM systems and Retrieval-Augmented Generation (RAG) pipelines relies on rigid, static checklist benchmarks that fail to adapt as new information about a target is discovered during an engagement. This research presents Aginiti Redteam, an open-source adaptive red-teaming framework that models the adversarial evaluation process as reasoning over a persistent evidence graph. Every signal a target reveals becomes a graded claim that reshapes which attack the engine tries next, rather than following a fixed script. We implement four attacks directly from published, peer-reviewed security research spanning three threat models: RAG knowledge-base reconstruction (IKEA, SECRET), document-level membership inference in the retrieval corpus (Interrogation Attack), and system-prompt extraction (SPE-LLM). Every finding passes through a three-stage verification pipeline (deterministic text scan, LLM judge against a specific list of what a real leak looks like, and a final direct comparison against real source data) before being reported, so no result is accepted on a single model's opinion alone.

#LLM Red-Teaming#Adversarial ML#RAG Security#OWASP Top 10 for LLMs#Multi-Agent Security

1. Problem Statement & Motivation

The Limits of Static Checklist Scanners

Modern enterprise AI architectures combine autonomous agent orchestration, tool calling, and vector retrieval backends (RAG). Traditional red-teaming tools execute predefined, linear lists of attack payloads. In practice, enterprise deployments incorporate active defenses — including output filters, rate limiters, and prompt guardrails — which can flag repetitive, static probing patterns. Just as importantly, a security finding is only as trustworthy as its verification: a single LLM asked whether an attack succeeded is a documented source of false positives, so findings need independent confirmation, not just detection.

Key Insights & Methodology

  • Static attack suites lack a memory of what a target has already revealed or blocked.
  • A single LLM's judgment of "did this attack work" is not sufficient evidence on its own.
  • Enterprise guardrails can detect and block repetitive, linear probing patterns.

Core Research Approach

An adaptive red-teaming engine that maintains an evolving model of what it has learned about a target, and re-ranks candidate attacks at every step based on expected new information and prior outcomes, should reach the same findings with fewer attempts and lower defense-trigger rates than fixed-order or random attack selection — validated head-to-head against honest baselines, not just asserted.

2. Published Attack Implementations & Threat Models

Implementing Peer-Reviewed Security Research as Working Attacks

Aginiti Redteam implements 4 attacks drawn directly from peer-reviewed security venues, each from the paper's own methodology:
AttackVenueTarget SurfaceThreat Objective
IKEAICLR 2026RAG Knowledge BaseVerbatim knowledge-base reconstruction via benign, natural-sounding queries — no jailbreak required
SECRETIEEE TIFS 2026RAG Knowledge BaseJailbreak-calibrated exploration (global exploration + local exploitation) to systematically harvest unseen documents
Interrogation AttackACM CCS 2025RAG Retrieval CorpusDocument-level membership inference — confirms whether a specific known document exists in the corpus, via calibrated yes/no probing
SPE-LLMICLR 2026System PromptSystem-prompt extraction via three fixed, non-adaptive probe templates

3. Adaptive Engine Architecture

Evidence-Graph-Driven Attack Selection

The core engine separates planning from payload generation: 1. **Evidence Store (ChromaDB)**: Maintains a persistent, evolving model of everything learned about a target — responses, defense triggers, and confirmed findings. 2. **Adaptive Ranker**: At every step, scores all currently-eligible attacks using a weighted formula combining multiple factors — expected new information, potential impact, and prior failures — rather than a fixed priority list or random choice. 3. **Provider-Agnostic LiteLLM Gateway**: Enables testing across different LLM providers through one unified interface, rather than being hardcoded to a single vendor.

Key Insights & Methodology

  • Persistent evidence tracking prevents redundant, repetitive probing.
  • Multi-factor scoring balances exploring new ground against following up on promising leads.
  • Provider-agnostic design avoids vendor lock-in for the attacker-side LLM.

4. Three-Stage Verification Pipeline

Reducing False Positives in Automated Red-Teaming

A single LLM's opinion is not treated as sufficient evidence. Every suspected finding passes through three independent stages before being recorded:

The 3-Stage Verification Gate

Stage 1: Deterministic text scan • A plain pattern/string check for obvious matches, requiring no model at all. Stage 2: LLM judge against enumerated claims • An independent model scores the response against a specific, concrete list of what a real leak would look like — not an open-ended "was this bad?" question. Stage 3: Direct source comparison • A final, non-LLM check comparing the suspected leaked content directly against real source data to confirm a genuine match.

5. Benchmark Results

Evaluated Against Honest Baselines and an Industry-Standard Scanner

The planner was benchmarked against random and fixed-order attack selection on a target running 8 independent defense layers (RBAC-scoped retrieval, output redaction, rate limiting, per-persona memory, a system-prompt guardrail, an input-filter classifier, session/auth expiry, and RBAC-scoped tool-calling), with an equal request budget and a fresh target process per trial.

Comparison Against NVIDIA garak

Run through the identical gateway and target as garak, the most widely used open-source LLM vulnerability scanner: on every directly comparable category (system-prompt extraction, jailbreak, indirect injection, markdown exfiltration), both tools reported exactly 0% ASR — full agreement, with neither tool producing a false positive against a well-defended target. Where this project goes further is structural, not a percentage improvement: garak's interface can only observe text in, text out, with no way to confirm a real tool invocation or network egress — exactly where the strongest confirmed findings against independently-built targets (e.g. AnythingLLM, a 19-agent multi-protocol test fleet) actually live.

MetricAdaptive PlannerFixed-Order BaselineResult
Avg. prompts to reach the same findings9.750.7~5x fewer requests for identical outcomes
Avg. times blocked by the target's input filter0.740.7Adaptive planner steers around dead ends rather than brute-forcing through them

Cite This Technical Report / Research

@misc{irfan2026aginiti,
  title={Aginiti Redteam: Adaptive Red-Teaming Engine for Agentic AI and RAG Systems},
  author={Irfan, Muhammad Hammad},
  howpublished={Open-source software, DevNeuron},
  year={2026},
  url={https://github.com/dev-devneuron/aginiti-redteam}
}
View Visual Overview & MediaExplore All Projects