Real-Time Decision Logic Needs Its Spreadsheet Moment

From SQL to code - and the missing authoring layer

In the previous article, I argued that Streaming SQL is not the right primary authoring tool for event-driven stream-based algorithms. Not because SQL is useless. It is useful, mature and natural for many streaming transformations. The problem starts when the logic is no longer mainly about deriving one relation from another, but about maintaining state, reacting to events, calling external systems, controlling when results are emitted and orchestrating decisions over time.

At that point, the usual answer is: use the Flink DataStream API.

Technically, that answer is correct. The DataStream API gives developers the expressive power missing from SQL. But it also moves the logic back into professional software development. The decision algorithm is no longer authored directly by the people who understand the business problem. It becomes Java or Scala code maintained by Flink developers.

That is the trap.

If the only choices are “SQL that cannot express the logic naturally” or “custom Flink code that requires specialized developers”, then real-time data will not get its spreadsheet moment. It will remain powerful, but narrow in adoption - much closer to the world before spreadsheets, where business logic lived inside programs written by professionals.

Nussknacker is our attempt to explore a different path: not SQL pretending to be enough, and not every decision algorithm implemented as bespoke Flink code. The idea is to expose real-time decision logic as reusable, composable and observable building blocks. Developers still matter: they create components, integrations and platform capabilities. But the everyday evolution of decision logic should not require dropping into the DataStream API.

Maybe this approach is right. Maybe it is incomplete. But the need for another authoring layer is real.

The rest of this article is a concrete test of that claim. I take ordinary building blocks of real-time decision systems - the kind used in fraud detection, recommendation, personalization, anomaly detection, streaming ML and real-time rating - and ask a simple question:

Can this be expressed naturally in Streaming SQL, or do we cross the boundary into custom Flink development?

Let’s start with the most basic operation of all: updating state.

1. Update State

Perhaps the most fundamental building block of a real-time decision system is surprisingly simple:

for each incoming event, read the current state for its key, combine that state with the event, and store the updated state.

A fraud detection algorithm updates the current risk profile. A recommendation engine updates user preferences. An anomaly detector updates its baseline. A real-time rating engine updates accumulated usage.

Different domains. The same operation.

In abstract form, it looks like this:

next_state(K) = f(current_state(K), event(K))

Streaming SQL naturally expresses transformations from input rows to output rows. It also expresses aggregations, especially over windows. But it does not provide a first-class way to describe explicit mutable keyed state evolving one event at a time.

This operation can be approximated with joins, changelog streams, upsert tables or lower-level APIs. But the simplicity of the original idea is lost. Instead of saying “update the state”, the author has to reconstruct a state machine from relational primitives.

If this most fundamental building block cannot be expressed naturally, it is hardly surprising that more sophisticated decision logic quickly reaches the expressive limits of streaming SQL.

2. External enrichment with OpenAPI

Nussknacker provides an OpenAPI Enricher component. From the author’s perspective, the idea is straightforward: call an external HTTP API for an event, use the API contract to understand the request and response structure, and continue processing with the enriched data.

In a SQL-centric approach, this usually becomes a function call inside a query:

SELECT call_api(x)
FROM stream

Technically, this can work. But the interesting part is no longer visible in SQL. Request execution, authentication, retries, timeouts, rate limiting and error handling move into externally developed code.

This matters because OpenAPI is designed to be consumed by tools. Services can be discovered automatically, their contracts are typed, and requests and responses can be validated without handwritten integration code. When this is hidden behind a UDF or external wrapper, a tool-friendly interface becomes handcrafted integration logic.

The problem is not that SQL cannot call a function. The problem is that the decision flow no longer shows what is actually happening.

3. Stateful merging of heterogeneous events

Many real-time decisions depend on information arriving from different sources at different times. A customer event, a transaction event and a device event may all describe different parts of the same situation. The system should accumulate partial information and emit a result when enough context is available.

In Nussknacker, this is the role of components such as Union Memo. They let the author model a shared keyed state that is gradually built from heterogeneous events.

The natural SQL instinct is to reach for joins:

SELECT ...
FROM A
JOIN B
JOIN C
ON key

But joins force an awkward choice. An inner join waits for all required inputs, which may delay or prevent decisions. An outer join emits incomplete, null-heavy results and pushes the responsibility for interpretation elsewhere.

What is missing is not the ability to combine tables. What is missing is a direct model of progressively building and updating state as events arrive.

To approximate this in SQL, the author has to reason about changelog streams, upserts, join semantics and watermark behavior. At that point, the logic starts to shift away from the decision problem and toward implementation mechanics.

4. Window lifecycle control

Streaming SQL supports standard time windows. Tumbling and hopping windows are natural when the lifecycle of a window is determined by time.

But real decision logic often needs more control than that.

A session may need to close because a particular event arrives. A result may need to be emitted when a specific event leaves a sliding context. A fraud or personalization scenario may need to react not only to what happened inside a time interval, but also to why that interval should be considered complete.

This requires explicit control over state and event lifecycle. SQL provides fixed window semantics, not programmable ones.

Again, the issue is not whether the behavior can be implemented somewhere. It can. The issue is whether the primary authoring abstraction lets the user express the logic naturally.

5. Machine learning: prediction vs decision

Streaming SQL can invoke a model and attach a prediction to a row. That is useful, but it is only part of the problem.

Real systems rarely stop at “row in, prediction out”. They need conditional model usage, fallback logic, model composition, version routing and experimentation. For example, if one model returns a low-confidence result, the system may need to call another model, apply a fallback rule or route the case for additional processing.

This is decision logic, not just model invocation.

SQL tends to treat a model as a function. Real systems treat models as elements of a broader decision process. As that process grows, orchestration moves outside SQL, and with it the control over how the system actually behaves.

Where this leaves us

These examples are not meant to prove that Streaming SQL is useless. It is not. SQL remains a good abstraction when the problem is naturally relational: filtering, projecting, joining and aggregating streams.

But real-time decision logic often has a different shape. It updates state event by event. It reacts to partial information. It calls external systems. It controls when results should be emitted. It treats ML models not just as functions, but as parts of a larger decision process.

When this logic cannot be expressed directly in SQL, the usual fallback is custom Flink code. That solves the technical problem, but it also changes who can work on the algorithm. The logic moves from an authoring environment into Java or Scala code maintained by specialized developers.

That is the point where the spreadsheet moment disappears.

Nussknacker is our attempt to avoid this forced choice. The goal is not to replace developers. Developers are still needed to create components, integrations and platform capabilities. The goal is to make everyday real-time decision logic visible, composable, testable and changeable without turning every change into a Flink development task.

So the practical distinction is not “SQL or code”. It is closer to this:

Use Streaming SQL when the logic is mainly relational and stable.

Consider Nussknacker when the logic is decision-oriented, domain-knowledge-heavy, frequently changing, or likely to require many iterations before it is right.

Use the DataStream API when you need something that cannot reasonably be expressed with existing higher-level building blocks: for example, a new reusable component, a custom connector, unusual state/timer behavior, or extreme performance tuning that must be hand-coded.