An independent study of two autonomous pentest platforms, what it establishes, what it does not, and one SQUR run on the same application.

Aikido and XBOW were given the same two open-source applications and the same source code. On Fider, XBOW confirmed 24 findings to Aikido's 17. On Photoview, Aikido confirmed 32 to XBOW's 7.

Same two products, reversed result. Buyers need both coverage and findings their engineers can act on, and this study says useful things about each.

Below: what the evidence supports, what it cannot settle, and a separate SQUR run on one of the same applications to show how we approach validation, including where that comparison is weaker than theirs.

The study

In spring 2026 the security firm Doyensec ran Aikido's and XBOW's products against Fider, a Go feedback portal, and Photoview, a Go and React photo library. Both platforms were given the source code. Doyensec then validated all 83 reported findings by hand, marked the false positives, re-scored the severities they disagreed with, and recorded the minutes each validation took. The report is Comparing AI Application Security Testing Platforms: Aikido vs. XBOW and it ships a findings file with every row.

• The evaluation was sponsored by Aikido, one of the two platforms tested. Doyensec state they controlled the execution and publication.
• The runs were not simultaneous. Aikido was tested on 23 March 2026. XBOW's Fider engagement ran from 8 April, when the order was placed, to 21 April, when the report arrived.
• The setups were not identical. The two platforms authenticated to the targets differently, which the study documents.

1. The result reverses between the two applications

Confirmed findings	Aikido	XBOW
Fider	17	24
Photoview	32	7
Both applications	49	31

Read the bottom row alone and one platform looks 58% ahead. The rows above show each winning one application, the second by more than four to one the other way.

A result from one application is evidence about that application. Two applications were enough to reverse the ranking here.

Both platforms found real, serious bugs on both targets, and both kept false positives low. A single engagement is simply thin evidence for a general ranking.

What this study does not establish, for any of the three tools discussed here: how many vulnerabilities each one missed, or how consistently it would perform across repeated runs. Nobody ran either platform twice, and no ground-truth list of every bug in either application exists. Those are larger unknowns than most of the caveats below.

2. What the reports cost to review

The findings file records the minutes Doyensec spent validating each finding, which is unusual and useful.

Across both applications	Aikido	XBOW
Findings reported	51	32
Confirmed real	49	31
Reviewer time, all reported findings	16.2 hours	9.1 hours
Per reported finding	19.0 min	17.1 min

Per finding the two are close, around 17 to 19 minutes, so volume explains much of the difference in the totals. It does not isolate the platform's contribution: reviewer familiarity and the complexity of individual findings are mixed into the same number, and the study was not designed to separate them.

Two things this figure is not. It is an independent researcher adversarially validating an unfamiliar tool's output, which is more thorough than routine triage, so it does not predict what your team would spend. And more review time is not waste: a larger set containing more consequential bugs can be worth every minute.

The question it sharpens is worth taking into any vendor call: how much work is it to get from a report to a distinct, actionable remediation decision? That depends on whether findings are deduplicated to root cause, whether the reproduction steps actually run, and whether the evidence tells an engineer what to change.

3. Severity moved under review, mostly downward

Of 83 reported findings, 80 were real. Severity was the less stable field. These are agreement rates with Doyensec's own severity assessment, which is expert judgement rather than a measurement:

Agreed with Doyensec's severity	Fider	Photoview
Aikido	13 of 17 (76%)	21 of 32 (66%)
XBOW	17 of 24 (71%)	4 of 7 (57%)

Two findings moved up: an Aikido finding from High to Critical, a mass-assignment flaw allowing unauthenticated account takeover, and an XBOW finding on Photoview from Medium to High. Every other disagreement went downward, and on Photoview eleven of Aikido's thirty-two were lowered, eight to informational.

The reviewers give a consistent reason: the platform did not know how sensitive the data behind a finding was. That is the hard part of severity and it does not go away. Impact depends on your data, your permissions and your business context.

What an executed exploit changes is the evidence a reviewer starts from. It establishes that the behaviour is real and reproducible, which removes one source of disagreement. The judgement about consequence still belongs to someone who knows the application.

4. What the study cannot settle

It publishes a chart of findings per vulnerability class, and the gaps in it look like blind spots. The study's own text suggests otherwise: the chart shows one platform with zero security-misconfiguration findings while the report lists "missing/misconfigured security headers" among what it found, and the other with zero insecure-design findings while listing "a logic error". Of both at once: "both platforms reported vulnerabilities due to missing and/or misconfigured headers." The most likely explanation is that those findings were filed under different headings, though the individual records are redacted so this cannot be reconciled row by row.

The overlap figures need the same care. On Fider the reviewers matched three findings across the two result sets, and on Photoview four. They described that analysis as a quick review done at the end and out of scope, and said in the same paragraph that they expected the real number was higher.

The buyer's lesson from both is short: compare distinct root causes and demonstrated impact, not category totals.

5. A SQUR run on the same application

We ran Fider at the same pinned version with the source code provided. Three limits before the numbers: one application not two, our findings validated by us rather than by an independent firm, and their figures are six months older than ours.

Fider, after review	Aikido	XBOW	SQUR
Critical	1	0	1
High	2	3	2
Medium	6	7	6
Low and informational	8	14	8
False positives in the delivered report	2	1	0
Who validated the report	Doyensec	Doyensec	SQUR
Tested	23 Mar 2026	10 to 16 Apr 2026	21 Sep 2026

All three reported three Critical or High findings on this application. That count alone does not establish equivalent coverage, and the spread between 17 and 24 findings is almost entirely low and informational.

Timing, with each measurement's boundaries

These are different clocks and are not a single number. Where the study does not record a stage, it is left blank rather than estimated.

Fider engagement	Aikido	XBOW	SQUR
Order and contract before testing	none, self-serve	8 to 10 Apr, DocuSign order form	none, self-serve
Setup to start a scan	under 20 min, both apps	not separately recorded	not recorded for this run
Test execution	8 h 40 min	6 days, with pauses and restarts	5 h 42 min
Report after testing finished	immediate	5 days	immediate

What the false-positive row is counting

All three counts concern delivered reports. Doyensec independently evaluated Aikido's and XBOW's; SQUR evaluated its own. We also disclose our internal candidate filtering below, for which equivalent competitor data is not available.

Stage	Count
Candidate findings the engine produced	25
Retracted by the engine's own verification, each with an executed disproof	−5
Merged into a parent by the deduplication pass, four rate-limit findings becoming one	−3
Delivered in the report	17
Failing our validation against the live application afterwards	0

No human edited the delivered report. Our validation afterwards changed three of our own assessments of the engine's findings, not the findings themselves, and one of those three was a finding we had wrongly called a false positive.

Two examples of what the engine did with its own output

It rejected a claim rather than passing it on. A published advisory said a JSON library could be crashed by deeply nested input. The engine generated tokens nested a thousand deep, sent them, watched the server return 401 and stay healthy, then explained why the vulnerable code path is unreachable in this application, and retracted its own finding with the test attached. For a customer that is an investigation that never lands on an engineer's desk.

It proved a consequence rather than describing one. The Critical is a mass-assignment account takeover. The engine read the untagged struct field in the Go source, supplied its own verification key, redeemed it, and confirmed the account's address had changed. It then ran the control, a key that was never submitted, and got the rejection it expected. That turns an abstract severity into something a team can see and re-run.

To answer the question that follows: we did not report it to the Fider maintainers, because they had already fixed it. The flaw was live in v0.33.0, the version the study used and we tested. Upstream closed it on 30 April 2026 in a commit that adds the missing serialisation tags across every affected struct. We checked our whole finding set against current Fider before writing any of this, and sent the maintainers the items that are still open.

This run illustrates how SQUR approaches report quality: test candidate findings, reject the unsupported ones, consolidate duplicates to root cause, and keep the reproduction evidence attached. The aim is to give engineers a clearer starting point for remediation, not to win a count.

What to ask, whoever you buy from

• Does every finding arrive with a reproduction I can run? It gives your engineers evidence to judge from and an assessor something reproducible for a DORA Article 24 or NIS2 file.
• How many of these share a root cause? Seven of one platform's findings on one application here shared one. One fix closed all seven.
• At what stage is the false-positive number counted, and who checked?
• How soon can I run it again? Coverage varies between tools and between runs. Frequency is the part you control.

Methodology

Aikido and XBOW figures come from the Doyensec report linked above and its published findings file, whose SHA-256 matches the paper's appendix. Severity columns are the reviewers' adjusted values. Validation minutes are their recorded totals divided by findings reported, false positives included, since those had to be reviewed too.

The SQUR run went against the same pinned Fider image, with the source supplied through the engine's source channel, and finished in 5 hours 42 minutes. It is a standard pentest, the same one we sell at €1,995.

• The rules were fixed before the run. The prediction, the counting rule and the failure condition were written down before the engagement started, so the result could not be graded to taste afterwards.
• What the engine was told. The target's name and stack, three seeded accounts, how the passwordless sign-in flow works, and where the source was. No vulnerability classes, no hints, no findings from the earlier study. The objective was to assess the application, confirm each issue against the running app, and record evidence and severity.
• Scope. The application origin plus its in-scope mailbox, exploitation permitted on the target only, and the three roles listed above.
• Could the model already have known these bugs? The honest test is dates, not assurances. Google documents the knowledge cutoff of the Gemini 3 Flash family, which the model doing the great majority of this run is built on, as January 2025. Fider v0.33.0, the version we tested, shipped on 5 March 2026. The upstream commit fixing the flaw behind our Critical landed on 30 April 2026, and the Doyensec study was published on 27 May 2026. Every public trace of this class of bug in this application postdates that cutoff by more than a year.
• And the study itself carries almost nothing to learn. 77 of the 83 findings in its published archive have their titles replaced by a hash. Six are disclosed, three of those are false positives, and fifteen of our seventeen findings have no counterpart anywhere in it.
• Public source is not a confound here. We gave the engine the source code, as the study gave it to both platforms. Prior exposure to a public repository cannot advantage a test in which the repository is an input.
• Artifacts retained: the full agent transcript, each finding at every revision, the engagement input, the validation log, and the offline image the run executed from with its digest.