Compositional AI Beyond LLMs: System Implications of Neuro-Symbolic-Probabilistic Architectures

Zishen Wan, Hanchen Yang, Jiayi Qian, Ritik Raj, Joongun Park, Chenyu Wang and 2 others

Large language models keep getting bigger and more capable, but they are expensive to run, easily fooled, and hard to explain, so researchers are building compositional AI that pairs a neural language model with old-fashioned symbolic reasoning and probabilistic inference to get robustness and interpretability; this work from Georgia Tech is a systems study of what that mixed style of computation actually does to a computer — it builds a taxonomy, measures these models running on CPUs and GPUs, and finds they run badly on today's hardware because they mix wildly different kinds of work, wander through memory irregularly, and move too much data, then shows software and system fixes that recover performance.

The argument in one glance

Problem

LLMs are costly, brittle, and opaque, so AI is turning compositional

Large language models drive AI progress but face unsustainable computation, limited robustness, and poor explainability. Compositional AI pairs a neural model with symbolic reasoning and probabilistic inference for interpretability, robustness, and data-efficient learning — but almost nothing is known about its hardware implications.
What people did

Assume it runs like an LLM on existing hardware

Treat neuro-symbolic-probabilistic systems as if they behave like a monolithic language model — a few dense-matrix operations — and run them on off-the-shelf CPUs and GPUs.
Why it fell short

The mixed computation runs badly on today's hardware

A compositional system stitches together fundamentally different neural, symbolic, and probabilistic computations with different data and control patterns. On off-the-shelf hardware it is constrained by heterogeneous compute kernels, irregular memory access, complex flow control, data dependencies, and high data movement — so it runs inefficiently, and no one had characterized where or why.
This paper's idea

Characterize the systems and optimize them (this work)

This work studies the system implications of neuro-symbolic-probabilistic architectures: it builds a taxonomy, analyzes their scaling efficiency, and characterizes them in detail across CPUs and GPUs — runtime, memory, operators, dataflow, scalability, communication. Guided by the revealed inefficiencies, it presents software- and system-level optimizations that improve performance, efficiency, and scalability.

Why this problem is worth solving

The systems community has spent enormous effort optimizing large language models, but the frontier of AI is starting to move toward something structurally different.

Large language models have driven remarkable progress, but their continued scaling faces real limits: the computation required grows unsustainably, the models remain brittle and easily fooled by inputs outside their training distribution, and their reasoning is opaque, which undermines trust in high-stakes settings. Compositional AI is an emerging response that does not treat a single neural network as the whole system. Instead it composes multiple modes of computation: a neural component (a language model) for perception and pattern-matching, a symbolic component for explicit logical reasoning over rules and structured knowledge, and a probabilistic component for reasoning under uncertainty. The premise is that combining them yields properties a pure neural model struggles with — interpretability (you can inspect the symbolic reasoning), robustness (logic and probability constrain the neural guesses), trustworthiness, and the ability to learn from far less data. These neuro-symbolic-probabilistic systems are showing strong promise for agentic applications and more human-like reasoning, which is why they are drawing growing attention as a direction beyond ever-larger language models.

The catch is that almost all of this work lives at the algorithm level, while the systems and hardware questions are wide open and non-obvious.

A monolithic large language model is, from a hardware perspective, relatively uniform: its computation is dominated by a handful of dense matrix operations that GPUs are exquisitely tuned for, so the systems challenge is largely about scaling those operations. A compositional neuro-symbolic-probabilistic system is a different beast. It stitches together fundamentally different kinds of computation — the dense linear algebra of a neural network, the discrete, branch-heavy operations of symbolic logic, and the sampling and graph-structured work of probabilistic inference — each with its own data access patterns, control flow, and performance profile. That mix does not map cleanly onto hardware built for dense neural computation: it involves heterogeneous compute kernels rather than one dominant operator, irregular memory access rather than streaming, complex flow control rather than straight-line execution, tight data dependencies between the components, and heavy data movement passing intermediate results among them. Because the excitement has been at the application level, there was no systematic understanding of how these systems actually behave on CPUs and GPUs, where their bottlenecks lie, or how to make them efficient — a gap that matters as these architectures move from research demos toward real deployment.

What people did before, and where it broke

Optimize the language model in isolation. Pour systems effort into making the neural component’s dense-matrix operations fast on GPUs. Where it broke. In a compositional system the neural part is only one of three modes, so optimizing it alone leaves the symbolic and probabilistic components, and the interactions among all three, unaddressed.
Assume compositional AI has an LLM-like hardware profile. Treat the whole system as dominated by dense linear algebra and schedule it accordingly. Where it broke. The symbolic and probabilistic parts have entirely different, irregular data and control patterns, so the LLM-shaped assumption misestimates where time and energy actually go.
Run neuro-symbolic-probabilistic systems on off-the-shelf hardware as-is. Deploy them on standard CPUs and GPUs without analysis. Where it broke. They run inefficiently, constrained by heterogeneous kernels, irregular memory access, complex flow control, data dependencies, and high data movement, wasting the hardware.
Study these systems only at the algorithm level. Advance the models’ reasoning and accuracy without measuring their system behavior. Where it broke. Without a hardware characterization, there is no basis for the systems and architecture work needed to run them efficiently at scale.
The gap. No one had built a taxonomy, characterized these systems across CPUs and GPUs, and identified the concrete inefficiencies to guide optimization. Providing that is what this work does.

How the study works

From an unstudied paradigm to a characterized, optimized one

↓ →
new
↓ →
new
↓ →
new
↓ →
new
↓ →
new

The idea

Take the symbolic and probabilistic parts as seriously as the neural one. The study’s premise is that compositional AI is not a language model with some extra steps but a genuinely mixed workload in which explicit logical reasoning and probabilistic inference stand alongside the neural network, each demanding its own kind of computation, and that understanding the system means characterizing all three rather than only the familiar neural part. Treating the non-neural reasoning components as first-class citizens of the workload, whose computation must be understood and accelerated in its own right rather than folded into the neural framing, echoes the logic-reasoning acceleration work covered elsewhere on this site, which takes the symbolic, rule-based computation at the heart of reasoning systems seriously as a demanding workload with its own hardware needs rather than an afterthought to neural inference.

Measure before you optimize. Rather than propose an accelerator on intuition, the work first builds a taxonomy and then characterizes these systems in detail across CPUs and GPUs — runtime, memory, operator mix, dataflow, scalability, communication — so that the inefficiencies (heterogeneous kernels, irregular memory, complex control, data dependencies, high data movement) are established by evidence and the optimizations are grounded in what the hardware actually does. Grounding systems design in a careful quantitative characterization of a workload before recommending how to build for it is the same discipline the quantitative accelerator-analysis work covered elsewhere on this site follows, measuring a real workload on real hardware to produce actionable guidelines rather than optimizing from assumption, so effort is aimed where the measurements show it matters.

A mixed workload wants mixed, coordinated resources. The characterization points toward the same conclusion for compositional AI that has emerged for other mixed workloads: because the neural, symbolic, and probabilistic components have such different compute and memory appetites and pass data among themselves, running them well is not about a single fast kernel but about coordinating heterogeneous resources and taming the data movement between them. Recognizing that a workload combining fundamentally different compute and memory profiles needs a heterogeneous, integrated system rather than one uniform engine is the same move the heterogeneous NPU-and-memory accelerator work covered elsewhere on this site makes, pairing different specialized units and a shared memory design so that each part of a mixed workload runs on resources suited to it and the costly data movement between them is minimized.

Following a compositional workload through the study

Characterizing a neuro-symbolic-probabilistic system

A conceptual trace based on the paper's abstract and standard neuro-symbolic and characterization background; the specific taxonomy, benchmarks, and optimization details were not available.

Step 1 of 5

A compositional model combines three modes

The system runs a neural language model for perception, a symbolic engine for explicit logical reasoning, and a probabilistic component for reasoning under uncertainty, passing intermediate results among them.

This is the structure that departs from a monolithic LLM.

Why it works

Characterization exposes the real bottlenecks. Because the study measures runtime, memory, operators, dataflow, and communication on real hardware → the actual sources of inefficiency are identified rather than guessed → so optimization effort targets what genuinely limits these systems.
A taxonomy makes a diverse space tractable. Because neuro-symbolic-probabilistic systems combine their components in many ways → organizing them into a taxonomy lets their behavior be reasoned about systematically → so findings generalize across the design space rather than one instance.
The mixed workload is the point, not an aberration. Because the inefficiencies come from combining heterogeneous kernels, irregular memory, and complex control → recognizing these as inherent to compositional AI reframes the systems challenge correctly → so the right optimizations address the mix, not just the neural part.
Scaling analysis anticipates deployment. Because the work analyzes how efficiency changes as models grow → it looks ahead to where these systems will strain as they scale up → so the guidance is relevant to real, larger deployments, not only small demos.
Evidence-guided optimization pays off. Because the software and system optimizations are chosen from the characterization → they attack the measured bottlenecks like data movement and irregular access → so they demonstrate concrete improvements in performance, efficiency, and scalability.

What it costs (trade-offs and scope)

This explainer is based on the abstract, not the full paper. The full text was not openly available at the time of writing, so the specific taxonomy, the benchmark suite, the exact quantitative findings, the magnitude of the inefficiencies, and the details and measured gains of the proposed optimizations are not reflected here beyond what the abstract states.
It is a characterization and optimization study, not a finished accelerator. The contribution is understanding and a set of software- and system-level optimizations, not a shipped hardware design; its value is in guiding future systems for compositional AI rather than delivering a single product.
Findings depend on the chosen models and hardware. The characterization is across particular CPUs and GPUs on particular neuro-symbolic-probabilistic systems; other models, implementations, or hardware could show different profiles, and compositional AI is a fast-moving and heterogeneous space.
Compositional AI itself is still maturing. The premise that combining neural, symbolic, and probabilistic reasoning delivers robustness and interpretability is promising but an active research area; the systems implications matter most if the paradigm continues to gain traction.
The optimizations are demonstrations of potential. The abstract frames the software and system optimizations as demonstrating the potential for improvement; the full extent, generality, and limits of those gains require the paper’s detailed evaluation, not available here.
Off-the-shelf hardware may remain the practical target. The study highlights inefficiencies on standard CPUs and GPUs, but specialized hardware for compositional AI is nascent, so near-term deployments will likely still run on general-purpose hardware, making the software-level insights especially relevant while architectural ones look further ahead.

The evaluation

Monolithic LLM versus compositional AI on today's hardware

Property	Monolithic LLM	Compositional AI, treated as an LLM
Computation profile
Memory access		Assumed regular
Control flow	Straight-line	Assumed simple
Fit to off-the-shelf hardware		Overestimated
What the work provides	Well studied	Untested assumption

Monolithic large language models versus compositional (neuro-symbolic-probabilistic) AI on today's hardware, as framed by the paper's abstract (the full paper was not openly available, so this reflects the abstract's claims and standard background rather than the paper's detailed evaluation). A monolithic LLM is dominated by dense matrix operations with mostly regular memory access and straight-line control, which off-the-shelf GPUs are tuned for. Compositional AI integrates a neural language model with symbolic reasoning and probabilistic inference to gain interpretability, robustness, trustworthiness, and data-efficient learning, but this makes it a fundamentally mixed workload — heterogeneous compute kernels rather than one dominant operator, irregular memory access, complex flow control, tight data dependencies between components, and high data movement. Treating it as if it were an LLM misjudges its hardware behavior. This work studies the system implications directly: it builds a taxonomy of neuro-symbolic-probabilistic models, analyzes their scaling efficiency, and characterizes them in detail across CPUs and GPUs (runtime, memory, operators, dataflow, scalability, communication, and system behavior), revealing that they run inefficiently on off-the-shelf hardware for exactly those reasons. Guided by these insights, it presents software- and system-level optimizations that demonstrate the potential to improve the performance, efficiency, and scalability of compositional AI.Underlined values explain themselves when clicked.

Where this stops working

What to keep in mind

This explainer is based on the paper’s abstract, because the full text was not openly accessible when it was written, so read it as a faithful account of the abstract’s claims plus standard, well-established background on large language models, neuro-symbolic and probabilistic AI, and hardware workload characterization — not as a report of the paper’s detailed taxonomy, measurements, or optimizations. With that caveat, the contribution is clear and timely. Large language models are powerful but costly, brittle, and opaque, and compositional AI — combining a neural model with symbolic reasoning and probabilistic inference — is an increasingly serious direction for getting interpretability, robustness, and data-efficient learning, especially for agentic systems. The paper’s observation is that essentially all the work on this paradigm has been at the algorithm and application level, while its systems implications are unstudied, even though a compositional system is a fundamentally different, mixed workload from a monolithic language model. Its contribution is to fill that gap: build a taxonomy of neuro-symbolic-probabilistic architectures, analyze their scaling efficiency, characterize them in detail on real CPUs and GPUs across runtime, memory, operators, dataflow, scalability, and communication, and show that they run inefficiently on off-the-shelf hardware because of heterogeneous kernels, irregular memory access, complex flow control, data dependencies, and high data movement — then use those insights to drive software- and system-level optimizations that demonstrate room for improvement. This kind of characterization work is valuable precisely because it turns a hand-wavy sense that a new workload is hard into concrete, measured bottlenecks that systems and architecture research can target. What is not available here is the specific taxonomy and its categories, the exact benchmarks and quantitative results, how large the inefficiencies are, and what the optimizations concretely do and how much they gain — the details needed to assess and build on it. The durable idea, clear even from the abstract, is that as AI moves beyond monolithic neural networks toward compositions of neural, symbolic, and probabilistic reasoning, the hardware and systems community must characterize and design for that heterogeneity rather than assuming the dense-neural playbook still applies.

Background

Why new AI workloads keep demanding fresh systems characterization

It helps to see this work within a recurring pattern in computer systems: whenever a new class of workload rises, someone must first characterize how it actually behaves on real hardware before the field can build for it well, because intuition transferred from the previous dominant workload is often wrong. Deep learning, then large language models, each triggered waves of measurement studies that reshaped accelerator and system design, and compositional neuro-symbolic-probabilistic AI is plausibly the next such shift, with a computation profile unlike the dense-neural one that current hardware was tuned for. The dynamic-parallelism LLM serving work covered elsewhere on this site reflects the maturity of the previous wave, refining how the now-well-understood language-model workload is served, whereas this paper is doing the earlier, foundational job for the next wave — mapping the terrain of a workload whose bottlenecks are not yet known. The common thread is that systems progress depends on honest characterization: you cannot build the right accelerator, scheduler, or memory system for a workload you have not measured, and the mixed, irregular nature of compositional AI makes that measurement both harder and more necessary. As AI diversifies beyond monolithic models into compositions of reasoning styles, studies like this one that establish the taxonomy and the empirical bottlenecks are likely to be the essential groundwork on which efficient systems for the next generation of AI are built.

If you remember only three things

1. Large language models have driven remarkable AI progress but face three limits: unsustainable computation as they scale, limited robustness (they are brittle and easily fooled), and poor explainability (opaque reasoning). Compositional AI is an emerging alternative that does not rely on a neural network alone but integrates a language model with symbolic reasoning (explicit logic and rules) and probabilistic inference (reasoning under uncertainty), aiming for interpretability, robustness, trustworthiness, and data-efficient learning, with strong promise for agentic applications and human-like reasoning. But almost all the work has been at the algorithm and application level, and the systems implications are unstudied. Unlike a monolithic LLM — dominated by a few dense matrix operations that GPUs are tuned for — a compositional neuro-symbolic-probabilistic system stitches together fundamentally different computations (dense neural linear algebra, discrete branch-heavy symbolic logic, and sampling and graph-structured probabilistic inference), each with its own data and control patterns, and it was unclear how well today's CPUs and GPUs handle that mix.
2. This work, according to its abstract, studies the system implications of neuro-symbolic-probabilistic architectures. It first builds a taxonomy of these models and analyzes their scaling efficiency, then performs a detailed characterization across real CPUs and GPUs, measuring runtime, memory, compute operators, dataflow, scalability, communication, and overall system behavior. The studies reveal that compositional models run inefficiently on off-the-shelf hardware, constrained by heterogeneous compute kernels (no single dominant operator), irregular memory access, complex flow control, tight data dependencies between the components, and high data movement passing intermediate results among the neural, symbolic, and probabilistic parts. These are the concrete, measured bottlenecks that a mixed workload incurs on hardware built for uniform dense-neural computation. Guided by these insights, the work presents several software- and system-level optimizations that demonstrate the potential to improve the performance, efficiency, and scalability of compositional AI.
3. Important caveat: this explainer is based on the abstract because the full paper was not openly available, so the specific taxonomy and its categories, the benchmark suite, the exact quantitative findings and magnitude of the inefficiencies, and the concrete details and measured gains of the proposed optimizations are not reflected here. The durable idea, clear even from the abstract, is that as AI moves beyond monolithic neural networks toward compositions of neural, symbolic, and probabilistic reasoning, the hardware and systems community must characterize and design for that heterogeneity rather than assuming the dense-neural playbook still applies — and that the foundational step is honest measurement on real hardware, turning a vague sense that a new workload is hard into the concrete bottlenecks (heterogeneous kernels, irregular memory, high data movement) that systems and architecture research can then target.

Check yourself

Check your understanding

1. What is compositional AI, and why is it emerging as an alternative to ever-larger LLMs?
2. Why is a compositional neuro-symbolic-probabilistic system a harder systems problem than a monolithic LLM?
3. What does this work actually do to understand these systems?
4. What should you keep in mind about the evidence behind this explainer, and what is the durable takeaway?

Sources

The paper is “Compositional AI Beyond LLMs: System Implications of Neuro-Symbolic-Probabilistic Architectures” by Zishen Wan, Hanchen Yang, Jiayi Qian, Ritik Raj, Joongun Park, Chenyu Wang, Arijit Raychowdhury, and Tushar Krishna, published at ASPLOS 2026, DOI 10.1145/3760250.3762235. The authors are affiliated with the Georgia Institute of Technology and collaborators.

This explainer is based on the paper’s abstract (Creative Commons Attribution / CC-BY, as indicated by the publisher’s open-access metadata). The full text was not openly accessible at the time of writing — the ACM Digital Library copy was gated behind a bot check, no open arXiv preprint for this specific paper was available (the authors’ pages link the ACM version), and the Semantic Scholar, OpenAlex, and Unpaywall records pointed only to the gated version — so this page reflects the abstract’s stated claims (that large language models face challenges of unsustainable computation, limited robustness, and poor explainability; that compositional AI integrating LLMs with symbolic reasoning and probabilistic inference is a promising paradigm for interpretability, robustness, trustworthiness, and data-efficient learning, with strong potential in agentic applications; that the work develops a taxonomy and analyzes the scaling efficiency of neuro-symbolic-probabilistic models; that it performs a detailed characterization across CPUs and GPUs evaluating runtime, memory, compute operators, dataflow, scalability, communication, and system behaviors; that compositional models suffer inefficiencies on off-the-shelf hardware constrained by heterogeneous compute kernels, irregular memory access, complex flow control, data dependencies, and high data movement; and that the work presents several software- and system-level optimizations to demonstrate the potential for improving performance, efficiency, and scalability) together with standard, well-established background on large language models, neuro-symbolic and probabilistic AI, and hardware workload characterization, provided to make the abstract’s contribution understandable. The specific taxonomy, benchmarks, quantitative characterization results, and optimization details are not reflected here beyond what the abstract states; the flow and step-through sections are conceptual readings of the abstract plus standard background rather than details drawn from the full paper. This page is marked as a draft for that reason and should be confirmed against the full paper when it becomes available.