Scaling Automated Database System Testing

Powerful automated tools can find the subtle bugs where a database silently returns a wrong answer, but each tool needs a hand-written SQL generator tuned to one database's exact dialect — weeks of work per system — so only a handful of the thousands of databases ever get tested; this paper (National University of Singapore) builds SQLancer++, which drops the per-database generator and instead fires SQL at any database, watches which statements it accepts or rejects, and learns each system's dialect on the fly, letting one tool test 18 databases and uncover 196 new bugs.

Why this problem is worth solving

The bugs at stake are the hardest kind to catch, and the tool that catches them hinges on a subtle idea.

The reason feeding the oracle is so costly is that SQL is not really one language.

What people did before, and where it broke

• Hand-write a dialect-specific generator per database. Encode each target system’s SQL dialect into a bespoke generator so it emits valid statements, as the leading tool does for a couple dozen systems. Where it broke. Each generator costs weeks and thousands of lines of code, so it scales to only a few of the thousands of databases, and most teams will not pay that setup cost.
• Reuse a generator across dialect-compatible systems. Point a generator built for one database at another that claims compatibility, hoping the dialects overlap enough. Where it broke. Even nominally-compatible systems differ enough that validity rates collapse — to under one percent in the paper’s example — without substantial manual fixing, so reuse does not actually avoid the per-system work.
• Grammar-free or generic fuzzing. Generate SQL without a hand-written grammar, or mutate seed queries, to sidestep dialect coding. Where it broke. Prior grammar-free and generic tools were limited — they did not adapt to a specific system’s supported features, so they either produce mostly invalid statements that never exercise the database deeply, or lack the oracle machinery to catch logic bugs.
• Just test the few systems that already have generators. Accept that only SQLite, MySQL, PostgreSQL, and a handful more get this powerful testing. Where it broke. It leaves the vast and growing population of databases — new specialized engines, accelerator- and learning-augmented systems — outside the reach of the methods most able to find their silent correctness bugs.
• The gap. No approach let the general, oracle-based logic-bug testing be applied to an arbitrary SQL database without first paying to hand-encode that database’s dialect. Making the generator learn the dialect itself, on the fly, so one tool works on any SQL system supporting a common subset, is what SQLancer++ provides.

How SQLancer++ works

The idea

Keep the general oracle, throw away the per-dialect generator. The insight that unlocks scale is separating what is genuinely general from what is not. The test oracle — rewrite a query into an equivalent one and check they agree — is already database-agnostic; it is only the generator of valid queries that was tied to a dialect. SQLancer++ keeps the oracle and replaces the hand-written generator with a learning one, so the whole power of oracle-based logic-bug finding becomes applicable to any SQL system without per-system code. Catching silent correctness bugs by checking that two computations which should agree actually do, rather than needing to know the right answer beforehand, is the same differential-oracle strategy the compiler-driven differential-testing work covered elsewhere on this site is built on, exposing a wrong result by disagreement between things that must be equal — and SQLancer++‘s contribution is to make feeding that oracle no longer require dialect-specific effort.

Generate against a superset of SQL, and let failures prune it down. Rather than trying to know a database’s dialect up front, SQLancer++ starts optimistic: it generates from a broad menu of common SQL features, fully expecting many statements to be rejected by any given system. Every rejection is information. The generator does not need a grammar for the target dialect because it discovers the dialect empirically, from which of its speculative statements the database accepts. This inverts the usual order — instead of encoding what is legal and generating only that, it generates broadly and learns what is legal — which is what removes the human from the loop. Producing structured test inputs from a general model and relying on feedback to steer generation toward inputs the system under test will actually accept is the same generate-then-adapt discipline the compiler-fuzzing work covered elsewhere on this site relies on, casting a wide net of candidate programs and letting the target’s responses shape which ones are worth producing.

Turn accept-or-reject feedback into a principled probability per feature. The learning has to be more than a blunt on-off switch, because a feature might fail for reasons unrelated to support, and a rarely-tried feature has little evidence either way. SQLancer++ therefore models each SQL feature’s chance of yielding a valid statement with Bayesian inference over the history of executions that used it, updating the estimate as evidence accumulates, and suppresses a feature only when its estimated success probability falls below a user-chosen threshold. This principled treatment lets the generator balance exploring uncertain features against exploiting known-good ones, and it is what drove the validity rate up sharply — for SQLite, nearly quadrupling it. Using Bayesian estimation over observed outcomes to decide which options to pursue, rather than a fixed rule, is the same probabilistic-feedback approach the Bayesian compiler-optimization work covered elsewhere on this site is built on, letting a model of success guide a search through a large space of choices toward the ones that pay off.

Following a database through SQLancer++

No dialect was written; the tool assumes only that the database supports some subset of common SQL.

Why it works

• Separating the general oracle from the dialect-bound generator is what scales. The oracle already worked on any database, but the generator did not → but replacing the hand-written generator with a learning one leaves the general oracle applicable everywhere → so the full power of logic-bug finding reaches any SQL system without per-system code.
• Generating broadly and pruning by failure needs no grammar. Encoding a dialect up front is the expensive step → but generating from a common-SQL superset and discarding what the database rejects discovers the dialect empirically → so no human has to write the grammar for each system.
• Bayesian feedback turns noisy accept-or-reject signals into good decisions. A single failure is weak evidence and a rare feature has little data → but estimating each feature’s success probability by Bayesian inference and thresholding it balances exploration and exploitation → so the generator converges to mostly-valid SQL, raising SQLite’s validity rate by about 292 percent.
• Feature-set deduplication keeps a broad sweep from drowning in duplicates. Testing many databases with a broad generator surfaces many bug reports → but recording the feature set behind each bug and suppressing subset-duplicates concentrates effort on distinct defects → so the 196 reported bugs are unique.
• Empirical validation across many systems proves the generality. A method that only worked on the usual few systems would not have solved the problem → but SQLancer++ ran on 18 databases with no per-system generator effort and found real, fixed bugs → so the approach genuinely scales beyond the handful prior tools covered.

What it costs (trade-offs and scope)

• It targets the common SQL subset, not every dialect feature. SQLancer++ works for databases that support a subset of common SQL and generates from that common catalog; deeply system-specific or exotic features outside the shared menu are not exercised, so a hand-written dialect generator can still probe corners of one system that the adaptive approach does not reach.
• Learning the dialect spends test budget on invalid statements. Especially early on, many generated statements are rejected while the tool learns what a database supports, so some of the testing effort goes into failed statements rather than deep query execution; the Bayesian pruning reduces this over time but does not eliminate the exploration cost.
• The oracle still bounds what can be found. SQLancer++ scales the generator, but finding a logic bug still depends on the test oracle applied on top; bugs no available oracle can detect, or behaviors outside what the oracles check, remain out of reach, so the contribution is broadening coverage of systems, not of bug types.
• Feature-based deduplication is a heuristic. Prioritizing bugs by their triggering feature sets, and treating subset-matches as duplicates, is an approximation; it could conflate genuinely distinct bugs that share features or fail to merge related ones, so the unique-bug count reflects that heuristic rather than a perfect deduplication.
• It is a first step, with follow-up challenges acknowledged. The authors frame SQLancer++ as the first major step toward scaling automated database testing and note remaining challenges; the results are strong across 18 systems, but the vision of covering the thousands of existing databases, and handling the full diversity of their features and setups, is not fully realized by this single tool.

The evaluation

Check yourself

Sources

The paper is “Scaling Automated Database System Testing” by Suyang Zhong and Manuel Rigger (National University of Singapore), published at ASPLOS 2026 (Volume 2), DOI 10.1145/3779212.3790215.

This explainer is written from the full text of the authors’ arXiv version (arXiv:2503.21424). Everything above comes from the paper’s own sections: the introduction on logic bugs (silent incorrect query results) and test oracles (transforming a query into a semantics-preserving equivalent and checking result equivalence), the scale of database management systems (MySQL over 5.5 million lines of code) and of the market (over 162 billion dollars growing at a compound annual rate of about 15 percent), and the trends toward more, increasingly specialized and accelerator- or learning-augmented databases; the core problem that existing tools need hand-written, dialect-specific generators (the state-of-the-art SQLancer supporting 22 databases at an average of 3,729 lines of code each, and a reused generator’s validity rate collapsing to under one percent on a nominally-compatible system without hundreds of lines of fixes) so they do not scale, with a cited project deciding the setup cost was too high; the design of SQLancer++‘s adaptive statement generator, which generates statements using a broad set of SQL features, maintains an internal schema model, executes statements and uses execution feedback to prioritize supported features and deprioritize unsupported ones, estimating each feature’s success probability by Bayesian inference against a user-set threshold to raise the validity rate (increasing SQLite’s by 292.5 percent); the notion of a SQL feature as an element or property of a statement used by both the generator and the bug prioritizer, which deduplicates bugs by treating a new bug whose feature set is a subset of a previously-reported one as a likely duplicate; the result validator that applies any existing oracle; and the evaluation across 18 database management systems discovering 196 unique, previously-unknown bugs of which 180 were fixed after being reported, framed as a first major step toward scaling automated database testing with follow-up challenges remaining.