Physical AI risks overlooked in safety debate, says investor

The current AI safety debate may be missing a key dimension: the risks of AI systems operating in physical environments. According to recent commentary, most discussions—whether around alignment, governance, or rogue agent containment—still focus on digital threats. This framing leaves out the more immediate dangers of AI deployed in warehouses, retail stores, and industrial settings, where poorly managed systems could lead to real-world harm.

The critique comes as enterprises increasingly embed AI into physical workflows, often without clear safeguards. One startup that recently raised funding, for example, uses video and sensor data to automate retail and restaurant operations, handling tasks like inventory tracking and staff scheduling. Another company’s funding round highlights demand for tools that debug AI agents—but these tools remain concentrated on digital environments, not the physical ones where errors could result in operational failures or safety violations.

Efforts like ServiceNow’s push to become an AI agent "control tower" reflect growing awareness of governance challenges. Yet even these initiatives tend to frame the issue as a balance between risk and business value, rather than a fundamental safety concern. This approach may work for digital agents, but physical AI introduces new variables: latency, hardware limitations, and the unpredictability of real-world conditions. A misaligned agent in a cloud environment might generate incorrect outputs; the same agent managing a robotic system could cause physical damage or injury.

The disconnect isn’t just theoretical. A recent overview of AI safety startups identified dozens of companies, but few appeared to specialize in physical AI risks. Similarly, an open-source platform announced by a major tech company targets digital agents, offering tools to manage autonomy within software ecosystems. While valuable, this leaves a gap where AI interacts with the physical world.

What’s missing isn’t just regulation or technical safeguards, but a shared understanding of what failure looks like in these contexts. A malfunctioning agent in a digital system might disrupt operations; the same agent operating machinery could lead to accidents. The latter scenario requires different safety protocols, yet the startup ecosystem—and the investors funding it—often treats them as the same problem.

The question now is whether the market will address this gap after incidents occur, or by building safety into physical AI from the start. For founders and operators, the takeaway is clear: if AI interacts with the physical world, safety isn’t just about alignment or governance—it’s about ensuring systems can fail without causing harm. For investors, it’s a signal to assess not just the technical sophistication of a startup’s AI, but its awareness of the environments where that AI will operate. The tools to manage these risks may not exist yet, but the need for them is already here.

Sources: sifted.eu