Insights/Benchmarks

Benchmarks

Qlogue uses synthetic financial institutions and deliberately seeded assurance defects to test methods without exposing confidential client data. Results are published as they become available.

Research programme

The synthetic bank programme

Qlogue constructs synthetic regulated environments — institutions with realistic policy frameworks, model inventories, audit workpapers and control evidence — and seeds them with known assurance defects. This makes it possible to test whether a given method detects the defect, and to measure false-challenge rates and correlated errors across different assurers.

Research questions

Research question

Does trace-based assurance identify defects missed by component testing?

Testing in progress using synthetic Basel 3.1 implementation environments.

Research question

Does an independent challenger add genuinely new information?

Examining whether challenger outputs are correlated with or independent of primary assurer outputs.

Research question

Are multiple AI assurers making correlated errors?

Measuring the degree to which different AI models produce similar false positives and false negatives on the same synthetic defects.

Research question

Does apparently good outcome quality conceal weak reasoning?

Testing whether a correct conclusion can be reached through a reasoning chain that would not withstand challenge.

Measures

AEC

Method under testing

Assurance Evidence Coverage

A measure of the proportion of material claims in a judgement that are supported by traceable evidence.

ICY

Method under testing

Internal Consistency Index

A measure of whether the reasoning chain within a document is internally consistent — whether conclusions follow from the stated evidence and assumptions.

FCR

Method under testing

False Challenge Rate

The proportion of challenges raised by an AI assurer that do not identify a genuine defect.

Where empirical results are not yet published, measures are labelled as methods under testing. Qlogue does not present a named measure as validated merely because it has been defined.