RE LightGBM M3 and the retrospective audit of Italian public contracts

Open Models collection · Documentary assessment · Evidence reviewed on 15 November 2026

A procurement auditor may have time to examine only one file in a hundred. RE LightGBM M3 offers a way to choose that file once information about the award and execution of a contract is available. The Italian research team published the fitted trees and the scores assigned to held-out contracts. Together they allow a reader to examine what the ranking captures and where its evidential value ends. Fitted model · Held-out scores

Assessment. RE LightGBM M3 merits inclusion as an inspectable research model for retrospective audit selection. Its highest-ranked contracts concentrate a substantial share of the project’s documented positive outcomes. Across cases with known outcomes, however, its ability to distinguish the two labelled groups is weak. Whether it improves the yield of an audit authority’s existing selection practice remains an open empirical question.

A gold decision tree printed on a procurement file beside the title RE LightGBM M3.
© 2026 Juan Antonio Lloret Egea · ITVIA · The Bible of AI (BAI) · CC BY-NC-ND 4.0.

What the model learned from Italian public records

The project draws on the Italian National Anti-Corruption Authority’s procurement records for 2008–2025, administrative court decisions and territorial statistics. The broadest dataset contains 9,468,795 contracts. M3 concerns the 1,129,233 for which the project assembled an execution-stage dataset. Its 60 predictors include the value of the lot, the time to award and measures of cost overrun. An auditor considering an unfinished procurement would lack some of that information. Dataset counts · M3 predictors · Italian public contracts data

The study assigns a positive label when an administrative court upholds a challenge or when a recorded closure cites specified criminal or anti-mafia grounds. A negative label records a definitively dismissed challenge. M3 contains 210 known positives and 280 known negatives. The remaining contracts have unknown outcomes under these rules. An upheld challenge can establish a defect in a procurement procedure. A finding of corruption, responsibility for misconduct and the amount of any loss each require further evidence. A dismissed challenge likewise concerns the issues raised in that case. Judicial labelling procedure · Working paper

The model combines boosted decision trees with a risk estimator designed to learn from positive and unlabelled observations. Its reference configuration gives the labelled negatives their full weight (γ = 1) and uses an assumed positive-class proportion of 0.02. The latter is a modelling choice informed by the project’s prior estimation. It cannot establish the prevalence of corruption in Italian procurement. Estimation method · Reference configuration

Two different questions about accuracy

How many known positives reach the top of the list? In the project’s four validation groups, the top 1% of scores captured 11, 14, 12 and 9 known positives respectively. In total, 46 of 210 reached the threshold. Relative to random selection, the mean enrichment, or lift, was 21.89, with a standard deviation of 3.84 across groups. The ranking included sampled unlabelled contracts. These figures therefore describe the concentration of known positives in the evaluation sample. They do not give the proportion of all flagged contracts in which an auditor would substantiate an irregularity. Four-group results · Definition of lift

Can the model distinguish positive and negative legal outcomes? When the comparison is restricted to the two labelled groups, the mean area under the receiver operating characteristic curve (ROC AUC) is 0.531. On this scale, 0.5 corresponds to chance ordering of a randomly chosen positive–negative pair. Mean average precision is 0.555. The two results answer different questions. Proceedings may distinguish a case from the mass of unlabelled contracts even when the model has difficulty distinguishing the outcome of one proceeding from another. The study leaves this explanation untested. Per-group discrimination · Definitions of the measures

Does the pattern survive a different division of the records? The Bible of AI (BAI) recalculated the rankings from 12,490 published scores produced with a second division of the same source data. The file contains 210 positives, 280 negatives and 12,000 unlabelled contracts. The top 32 places in each group contained 10, 13, 8 and 12 positives, yielding 43 of 210 overall, a mean lift of 20.46 and a mean labelled-case ROC AUC of 0.535. This check uses scores generated by the research team. The tendency persists under the second division. Its value in another authority or a later period still requires an external test. Individual scores · Second validation design

What the published material allows a reader to inspect

The four-member research group developed the project in a statistics programme at the University of Padova. Italy’s National Anti-Corruption Authority supplied public records. The model file contains the learned split thresholds and leaf values. The accompanying material documents the construction of predictors, training settings, score calibration and the contribution of individual predictors. The project identifies a separate archive for larger data products. The public material reviewed does not establish a dedicated training budget, a hardware inventory for the final fit or operating costs for an audit authority. Project description · Learned decision trees · Construction of predictors

The project specifies the PolyForm Noncommercial License 1.0.0. Its terms expressly permit use by government institutions and set notice requirements for redistribution. A separately identified third-party component has its own terms. Readers assessing reuse should distinguish the conditions attached to the software from those governing procurement data and court records. Project licence · Third-party terms · Data provenance

What an audit authority would still need to establish

Legal challenges are selective. Differences in who has the resources or incentive to appeal may affect which contracts acquire labels. The published evaluation redistributes contract identifiers across groups, and gives no estimate of performance in a previously unseen authority or a later procurement period. The uncertainty analysis also matters. In the project’s conformal evaluation, both outcomes remained compatible with the model’s prediction for an average of 84.1% of the tested cases. Earlier Italian research on roadworks illustrates the importance of testing procurement indicators against separately defined judicial or investigative outcomes. Validation design · Prediction-set results · Independent Italian roadworks study

A meaningful test would draw on completed contracts from a later period or a different authority. The auditors would specify the finding they intend to detect, fix the number of files they can review and compare the model with their usual selection rule. Reviewers unaware of the selection method would assess files from both lists and a random sample outside them. Substantiated findings per hundred files, staff time per finding and missed findings estimated from the random sample would show whether the ranking improves the use of limited audit capacity.

Scope, methods and revision

This assessment covers the fitted M3 reference model in repository revision 6852e08, dated 19 October 2026. The recorded change to the model file dates from 10 June 2026. The documentary review included the working paper, fitted-tree format, data provenance, licence and published evaluation files. The numerical check used previously generated scores and involved no new model fit or inference. An independently adjudicated audit cohort, a prospective comparison or a change in the distributed trees would warrant a revised assessment. Revision examined · Model-file history

Liked this post? Follow this blog to get more.