🔘 About this critique
This assessment examines the Preprints.org version posted on 24 August 2026. It is an editorial critique, not peer review or an independent replication of the underlying agent experiments. Selected counts and retrieval percentages have been recalculated from the deposited review records. This verifies their arithmetic and consistency, not every inclusion decision, coding judgement or experimental result.
The assessment distinguishes limitations acknowledged by the authors from inconsistencies identified here. Exposure denotes the potential consequences of overstating a particular inference; it is not an overall rating of study quality. A subsequent withdrawal affecting one cited study is considered with its chronology.
The recommendation is to read the paper critically, together with the qualifications below.
Critique
| Audited version | Preprints.org v1 · DOI 10.20944/preprints202608.1600.v1 · posted 24 August 2026 · CC BY 4.0 · this version is not peer-reviewed. |
| Incorporated material | Complete preprint text, figures and classification tables; the deposited protocol, coding manual, study classification, screening and recovery records; and primary-source checks of the architectural examples discussed below. Consulted 10 November 2026. |
| Version distinction | The repository reports a subsequent publication in the Journal of Cybersecurity and Privacy, DOI 10.3390/jcp6050159. This assessment concerns the identified preprint and does not imply that the journal version has been separately reviewed. |
Unit 1 — A pivotal incident-response result has subsequently been withdrawn
Article data. Sections 4.6, 5.1 and 7.2 use MyAntFarm.ai to support a strong claim about the importance of orchestration: 100% actionable recommendations, compared with 1.7% for a single-agent configuration, across 348 trials. The review was posted on 24 August 2026. On 31 August 2026, the author of the cited study withdrew its Decision Quality results, including the associated actionability and zero-variance claims. The withdrawal notice explains that the evaluated action list had been fixed in the implementation rather than generated by the model; the scoring procedure assessed that list. The notice states that latency measurements are unaffected.
Reading. The withdrawal occurred after the selected preprint was posted. It therefore changes the evidence available to readers today without establishing that the review’s authors could have known about it when this version appeared. The affected measurements cannot now support the claim that orchestration determines incident-response quality more strongly than model capability.
Critical conclusion. The MyAntFarm.ai quality results should be removed from the review’s current evidential support. This correction does not invalidate the taxonomy or the results of other multi-agent studies; it specifically weakens the review’s strongest incident-response example and the broader inference drawn from it.
| Claim type | Subsequent correction to the evidence base |
| Exposure | High. A withdrawn result is used repeatedly to support a central architectural conclusion. |
Unit 2 — Architectural prevalence does not establish general superiority
Article data. Recalculation from the deposited classification confirms 18 single-agent tool-calling studies and 22 multi-agent studies: 11 centralised or hierarchical systems, eight pipeline systems and three debate systems. Centralised or hierarchical systems therefore constitute 18.6% of the 59-study corpus. Section 3.4 nevertheless describes this category as having the smallest evidence base within the multi-agent family. The review also highlights an improvement of up to 4.3-fold for HPTSA; the primary study reports this comparison on a benchmark of 14 vulnerabilities.
Reading. The classification makes centralised or hierarchical systems the largest of the three multi-agent categories. A claim about a smaller subset of controlled comparative experiments would require that subset to be defined. The HPTSA comparison remains relevant to its evaluated task, but its maximum relative improvement is not an average effect across architectures, security functions or deployment settings. The review itself acknowledges benchmark heterogeneity and does not pool effect sizes.
Critical conclusion. The prevalence statement requires correction. The comparative findings support further controlled evaluation of particular orchestration designs, while a general ranking of architectures remains unestablished. Architectural frequency and the number of convincing comparative evaluations should be reported separately.
| Claim type | Numerical consistency and comparative inference |
| Exposure | Medium. The error distorts the distribution of evidence; generalising selected performance maxima would overstate the comparative conclusion. |
Unit 3 — A documented subset cannot establish the absence of production evidence throughout the field
Article data. The review classifies 59 studies, with 30 assigned primarily to penetration testing (50.8%) and none assigned to production evaluation, E4. Its verification search recovers 44 of the 51 studies found through the primary search (86.3%), or 47 of all 59 included studies (79.7%). The authors also estimate that the eligible literature is substantially larger. Section 8 expressly limits frequency and absence claims to the selected corpus, whereas the abstract and Section 3.4 extend the absence of E4 studies to the entire field.
Reading. Recovery of previously included studies measures overlap between retrieval procedures. It does not establish the proportion of all eligible studies captured by the original search. The authors’ explicit acknowledgement that the corpus is a documented subset is therefore essential to interpreting both the frequencies and the absence of E4 evaluations.
Critical conclusion. The supported statement is that no E4 evaluation was identified among these 59 coded studies. A field-wide claim requires broader evidence than this corpus supplies. The qualification already present in the limitations should govern the abstract and the corresponding interpretative passages as well.
| Claim type | Retrieval coverage and generalisation |
| Exposure | High. A corpus-level absence could otherwise be mistaken for evidence that production evaluation does not exist anywhere in the field. |
Unit 4 — Disclosure of screening limitations does not restore missing exclusion records
Article data. The deposited screening file contains 211 records and records 30 full-text exclusions. For 29 of those exclusions, the individual study identities and reasons were not retained; the corresponding rows explicitly document this absence. The manuscript and archive acknowledge single-reviewer screening, the absence of a prospective registration and the lack of a formal risk-of-bias assessment. They also disclose that AI-assisted exclusion codings were adopted without record-by-record human re-verification.
Reading. These disclosures are substantive strengths of the reporting. They nevertheless distinguish what can be counted from what can be independently reassessed. Placeholder rows allow the totals to reconcile, but cannot establish whether each unidentified study satisfied an exclusion criterion. Similarly, the E1–E4 classification describes evaluation settings and does not assess all sources of bias in study conduct or measurement.
Critical conclusion. The archive supports an audit of the available classification and much of the selection record, but not a complete reconstruction of full-text eligibility decisions. This acknowledged limitation constrains confidence in selection completeness; it is not evidence that the retained studies were necessarily selected incorrectly.
| Claim type | Selection reliability and auditability |
| Exposure | Medium. Numerical reconciliation could be confused with a fully recoverable and independently verified selection process. |
Unit 5 — The supplementary files require reconciliation across corpus revisions
Article data. The classification file contains 59 studies. The recovery file contains 60, including PentestEval, which the manuscript and screening decisions exclude from the primary corpus. The verification-search documentation still reports 48 of 60 studies recovered and 45 of 52 primary-search studies recovered; its protocol also retains wording that treats PentestEval as included. Restricting the recovery file to the 59 studies actually classified gives 47 of 59 and 44 of 51, reproducing the manuscript’s 79.7% and 86.3%. These checks use the deposited review materials consulted on 10 November 2026.
Reading. The discrepancy is a traceable inconsistency between versions of the supplementary records. It does not refute the manuscript’s two recovery percentages, which can be reconstructed after the excluded record is removed. It does mean that readers cannot apply every supplied denominator or procedural description without first resolving which corpus version it describes.
Critical conclusion. The supplementary recovery file, protocol and explanatory text should be aligned with the final 59-study classification. Reproducibility requires consistent records as well as correct calculations. The revised percentages should remain explicitly tied to their distinct denominators.
| Claim type | Data consistency and reproducibility |
| Exposure | Medium. Unreconciled supplementary records can produce different, individually plausible recovery estimates. |
Unit 6 — Evaluated autonomy is not evidence that human oversight is unnecessary
Article data. The deposited classifications yield 52 fully automated systems, six human-guided systems and one system requiring human approval. None is classified as human-overseen. Among the 11 studies primarily concerned with DevSecOps, intrusion detection or incident response, ten are classified as fully automated. The coding manual concerns the evaluated configuration, rather than oversight proposed for a future deployment. Section 5.4 recognises that laboratory settings may explain the observed pattern.
Reading. A study can evaluate an automated component while leaving deployment decisions to people. Conversely, a proposed approval mechanism may not have been exercised in its experiment. The frequency of fully automated experimental configurations therefore describes the research designs in this corpus; it does not measure the comparative safety or operational adequacy of autonomous deployment.
Critical conclusion. The oversight findings identify an evaluation gap. They support testing approval mechanisms, intervention procedures and their effects under realistic conditions, rather than treating experimental autonomy as evidence that human oversight can be dispensed with.
| Claim type | Measurement scope and proposed application |
| Exposure | High. Confusing experimental configuration with demonstrated deployment suitability could distort decisions about consequential security actions. |
Overall assessment
The review offers a useful five-dimensional taxonomy and a publicly inspectable classification of 59 studies. Its principal descriptive counts can be reproduced from the deposited data. Its value as a structured account of selected research should nevertheless be distinguished from exhaustive coverage, a general ranking of architectures or evidence of production readiness. The later withdrawal of the MyAntFarm.ai quality results materially weakens one prominent argument; inconsistent supplementary records and the incomplete exclusion history further limit reconstruction of the synthesis. These qualifications preserve the contribution of the taxonomy while identifying where its interpretation and supporting records require correction.
The paper is recommended as a resource for studying how cybersecurity agents are classified and evaluated, with particular attention to the strength, currency and boundaries of the evidence supporting each conclusion.
🔘 Paper page: https://doi.org/10.20944/preprints202608.1600.v1
Abstract
The rapid evolution of Large Language Models (LLMs) has opened new frontiers in cybersecurity automation, enabling intelligent agents capable of multi-step reasoning, tool invocation, and autonomous decision-making across complex security tasks. While individual applications have emerged across threat intelligence, vulnerability assessment, penetration testing, and security operations center (SOC) automation, a systematic understanding of the LLM-based agent paradigm in cybersecurity — encompassing both single-agent and multi-agent architectures — remains lacking. This paper presents a systematic literature review following PRISMA guidelines, identifying records through 59 structured web-search queries whose results resolve predominantly to arXiv, Semantic Scholar, the ACM Digital Library, IEEE Xplore, USENIX, MDPI, SpringerLink, and Elsevier ScienceDirect, supplemented by citation chaining, for works published between January 2022 and April 2026; the full query record is published with the paper. We applied structured inclusion and exclusion criteria and classified 59 primary studies along five dimensions: security function, agent architecture pattern, knowledge augmentation strategy, human-in-the-loop posture, and evaluation rigor. Our analysis reveals that penetration testing and threat intelligence are the most extensively studied domains, while incident response and compliance verification remain critically underrepresented. Penetration testing alone accounts for over half the corpus (50.8%). Single-agent tool-calling remains the most prevalent architecture (30.5% of studies), whereas centralized multi-agent orchestration — present in 18.6% — yields the strongest reported performance gains, up to 4.3x on zero-day exploitation; prevalence and performance therefore point in opposite directions. No included study achieves production-grade (E4) evaluation: the entire field currently rests on controlled laboratory assessments. An independent search of six bibliographic databases recovers 86.3% of the studies the primary search had surfaced (79.7% of the full corpus) while indicating a total eligible literature of roughly 400 studies, so the corpus is reported as a documented subset rather than an exhaustive census. We propose a unifying taxonomy, identify cross-cutting challenges including hallucination, prompt injection, and benchmark fragmentation, and outline open research directions with particular emphasis on multi-agent orchestration design. Financial sector applicability under DORA and the EU AI Act is treated as a documented evidence gap rather than a synthesis: the corpus’s only compliance and risk assessment study is also its only banking-specific system.
Abstract reproduced from the authors’ Preprints.org v1 under the Creative Commons Attribution 4.0 International licence. Percentage signs and dash notation have been normalised for presentation; the wording is otherwise retained. The editorial critique above qualifies the abstract’s interpretation of the evidence. The subsequent withdrawal discussed in Unit 1 concerns a cited study, not this review.
Authors
George Fatouros, Konstantinos Mavrogiorgos, George Makridis, John Soldatos, Dimosthenis Kyriazis
Liked this post? Follow this blog to get more.