Ryan Metz writes about why synthetic data is becoming rehearsal infrastructure for Systems of Work, allowing enterprises to test consequential decisions before rare conditions arrive in production.
Most enterprise AI is tested on the past. Models are trained on historical records, evaluated against held-out examples, and released when they perform reliably on data drawn from broadly familiar conditions.
That approach works when the system is being asked to recognize another version of something the enterprise has already seen. It becomes less convincing when AI moves from classification and prediction into agentic systems that interpret context, coordinate workflows, and initiate action.
The most consequential question is no longer only whether an agent can perform well on yesterday’s data. It is how the agent will behave when it encounters a combination of conditions the enterprise has never recorded: a supply interruption that collides with a contractual exception, an unusual transaction that remains technically permissible, or a vessel whose movement appears ordinary in each individual system but becomes significant when radar, satellite imagery, and Automatic Identification System data are considered together.
These situations are difficult to test precisely because they are rare, novel, or have not happened at all. Yet waiting for production to supply the test case is a poor way to govern an intelligent system that can act. This creates a larger role for synthetic data: not merely producing more examples of the past, but helping an enterprise construct plausible versions of the future and rehearse how its Systems of Work will respond.
History is not a sufficient test suite
Synthetic data is commonly positioned as an answer to scarcity or sensitivity. If real records are limited, restricted, expensive, or imbalanced, a generator can create additional data with similar statistical properties for analytics, software testing, or model development.
That is useful, but it leaves the technology inside the same historical frame. The objective remains to reproduce the world already captured in the data.
Agentic systems create a different requirement. An agent participating in enterprise work may need to interpret a developing situation, apply policy, choose between competing objectives, requesting approval, call another system, or initiate a transaction. Its reliability depends not only on model accuracy, but on how the wider system behaves across context retrieval, reasoning, rules, permissions, escalation, and execution.
Historical records are incomplete evidence for that task. Systems of Record preserve what the enterprise transacted, approved, and observed. They do not necessarily preserve the alternatives considered, the near misses avoided, the judgment that remained in a person’s head, or conditions that have never coincided before. Rare but high-consequence events will be sparse by definition. New strategies have no performance history at all.
Testing an agent only against what has happened can therefore validate competence under familiar conditions while leaving its decision boundaries largely unexplored.
The deeper opportunity is to move from synthetic data generation to synthetic scenario generation. Instead of asking a generator to create more records that resemble the source data, the enterprise can ask it to construct a controlled situation that is absent from the record but still consistent with the rules of the world in which it could occur.
The purpose is not to predict that the invented event will happen. It is to determine what the enterprise and its intelligent systems would do if it did.
A plausible scenario needs more than realistic-looking data
This is where the technical standard becomes more demanding. A synthetic record can look convincing while being statistically distorted, physically impossible, operationally incoherent, or irrelevant to the decision under examination.
A credible scenario begins with real observations. In a maritime environment, that might include vessel movements, Automatic Identification System transmissions, radar returns, satellite data, weather, and specialized imagery. The initial synthesis layer needs to preserve the important distributions and relationships in those sources, including correlations that disappear when each dataset is modeled independently.
Real-world data establishes the baseline behavior of the domain. It does not need to contain the event the enterprise wants to examine, but it must provide enough evidence to represent the environment in which that event would unfold.
The next layer introduces intent. A scenario specification defines the condition to be explored: an unusual movement pattern, a search-and-rescue situation, an interdiction case, or a combination of signals that has not appeared in the historical record. Domain rules then constrain what the generator may produce. A vessel cannot exceed its physical capabilities. Locations, speeds, sensor readings, organizational responsibilities, and sequences of events must remain mutually consistent. Subject-matter experts contribute the operating knowledge that is rarely visible in a table of source data.
An ontology can give this process a shared representation of the relevant entities, relationships, states, responsibilities, and rules. It helps connect what a vessel is, which observations refer to it, what conditions are significant, and which authority owns the next decision. The ontology does not make the scenario true. It makes the assumptions and relationships required to interpret it explicit.
The output must then pass several tests. Statistical fidelity asks whether the relevant properties of the source environment have been preserved. Constraint validity asks whether the scenario is physically and operationally possible. Coverage asks whether it reaches the decision boundaries the test was designed to exercise. Task utility asks whether performance on the scenario reveals something useful about the system. Privacy and provenance controls establish which source information influenced the output and keep synthetic records distinct from authoritative ones.
These tests should not be collapsed into a single claim that the data is “realistic.” Realism is a set of claims that must be evaluated against the intended use.
Confidence also has a boundary. It is possible to begin with ranges, distributions, and expert descriptions when direct observations are limited. But the further a scenario moves from empirical evidence, the more its uncertainty should be made visible. Subject-matter expertise can constrain a plausible world; it cannot convert speculation into observation.
You do not need historical data for every scenario. You need enough evidence to understand which parts of the scenario are grounded, which are imposed by rules, and which remain assumptions.
Test the decision, not only the model
Once a scenario has been constructed and validated, it can do more than train a model. It can test the behavior of an entire System of Work.
Return to the maritime example. A synthetic sequence can be introduced into a controlled environment containing the same types of observations an operational agent would receive. The test is not merely whether the model labels the movement correctly. It is whether the system retrieves the right context, relates observations to the correct entities, recognizes the applicable rule, distinguishes uncertainty from evidence, and selects an action within its authority.
If the confidence threshold is not met, does it escalate? Does the operator receive the evidence needed to make a decision? If approval is given, does the right workflow begin? Are the recommendation, approval, and downstream actions preserved for audit? If the scenario changes midway through execution, does the system revise its judgment or continue acting on stale context?
This is decision rehearsal. It evaluates the chain from perception to accountable action.
The same principle applies beyond maritime operations. A bank can construct a permissible but unfamiliar transaction pattern to test whether an agent investigates rather than blocks it automatically. A manufacturer can combine supplier failure, demand movement, and capacity constraints to examine whether a planning agent protects the right commitments. A commercial organization can create a contract exception that places revenue, margin, legal precedent, and delivery capacity in tension, then observe whether its System of Work assembles the enterprise’s decision rather than optimizing one function.
Each scenario can be repeated with controlled variations. One variable can be moved across a decision threshold. Evidence can be withheld, delayed, or made contradictory. The same case can be run against different models, policies, or approval structures. The result is not a single benchmark score, but a map of where the system acts reliably, where it escalates appropriately, and where its operating design needs to change.
This is where our synthesis work and Agenix become complementary. The synthesis engine establishes a statistically credible foundation and can expand the available tabular data. Agenix can bring together that data with domain rules, scenario intent, and agentic capabilities to construct and evaluate situations that do not already exist as records. One layer helps preserve the shape of the observed world. The other helps determine which unobserved conditions the enterprise needs to examine.
Testing across tabular analytics workloads has also shown why the underlying quality matters. In our internal tests, models trained with the synthetic data improved on the real-data baseline in approximately 80 percent of cases. Most remaining results were broadly level, and the largest observed reduction on held-out test data was about one percent. This is an internal result, not a universal benchmark, but it demonstrates the standard required if synthetic data is to support real analytical work rather than serve as a convenient placeholder.
The more important test for agentic systems is broader still. A high-quality dataset can improve a component. A high-quality scenario can reveal whether the components form a trustworthy operating system for the work.
From institutional memory to institutional rehearsal
Scenario generation should not be treated as a one-time stage before deployment. The environment will change. New observations will alter the baseline. Policies, threats, products, and operational constraints will evolve. Subject-matter experts will identify conditions the original test library missed.
The synthetic environment can evolve with them. New data can trigger regeneration, rules can be revised when policy changes, and failed or disputed decisions can become the basis for new scenarios. Variants can be created around emerging risks before enough real cases exist to support conventional evaluation.
This produces a learning loop. Real outcomes improve the enterprise’s representation of the domain. That representation supports better scenarios. The scenarios expose weaknesses in agents, controls, and workflows. Corrections improve the System of Work, and its future decisions produce further evidence.
The governance requirement is equally important. Synthetic records used for testing should remain identifiable as synthetic, with their source data, scenario assumptions, generator version, validation results, and intended use preserved. They should not quietly enter a production knowledge layer as if they were observed facts. An ontology may define the states and relationships a scenario uses; synthetic instances should populate a governed test environment unless there is an explicit reason and control for using them elsewhere.
Over time, the valuable asset is not the volume of synthetic data produced. It is the enterprise’s growing library of consequential situations, decision boundaries, evaluation criteria, expert rules, and known failure modes. Models can be replaced. Agent frameworks will change. A well-governed scenario system preserves the organization’s accumulated understanding of what its intelligence must be able to handle.
This points to a larger transition. Systems of Record give the enterprise institutional memory. Systems of Work use that memory to decide and execute what happens next. Synthetic scenarios add a third capability: institutional rehearsal. They allow the enterprise to examine what its intelligent systems might do before an unfamiliar situation becomes an operational fact.
The strategic question is no longer only whether an agent performs well on the data the enterprise already has.
It is: which unobserved condition would expose the greatest weakness in this System of Work, and can we construct it before reality does?
About ToolShed
ToolShed is ConceptVines’ deep-technology series unpacking the tools, systems, and architectural patterns shaping Enterprise Agentic Platforms. Each edition takes one critical building block, including SuperAgents, ontologies, skills, synthetic data, Sovereign AI, code agents, and governance, and examines how it works, where it fits, and the architectural trade-offs that emerge at enterprise scale.
Led by ConceptVines’ engineering, product, and technology leaders, the series brings a practitioner’s lens to the technologies being assembled into Systems of Work today and the architecture of the increasingly autonomous enterprise.
Subscribe to ToolShed for a practitioner’s view of the technologies being assembled into Systems of Work and the decisions that will determine what enterprises can safely, intelligently, and autonomously do next.