The week in data and AI: an AI supplier’s breach reaches Notion’s chats

This week’s stories share one lesson: an AI system is only as contained as the parts of it you can’t see. An inference provider’s breach reached its customers’ chat transcripts. Keys stolen from an app host ran up an estimated $740,000. An enclave held, but the software inside it didn’t. Australia answered an agent incident with a new regulator and a reporting duty, United shut its door on outside agents, and Databricks shipped an agent that runs jobs on its own with the user’s identity.

Fireworks AI’s breach widens to customer datasets and Notion AI chats

Fireworks AI, an inference provider, expanded its disclosure of an October incident on Monday, 9 November. Attackers who first misused internal credentials later reached its development machines through a third-party cloud provider that hosted its R&D systems. They took billing details, resource metadata and, for some customers, incomplete model weights and datasets. Notion said fewer than 1% of its workspaces were affected: chat transcripts from the Notion AI Insights feature were exposed, most of them not attributable to an author or workspace. Notion disabled the feature and stopped serving open-weight models through Fireworks.

Why it matters for the industry: Most firms buy AI inside a SaaS product, and the model behind it often runs at another company. What an employee types into a workspace assistant can travel to an inference provider the customer never contracted with, and sit in an analytics or test pipeline long after the answer came back. Here the exposure ran through that provider’s own supplier, three steps from the person who typed the prompt. For every AI feature, the questions are which subprocessors see prompts, what they keep and for how long, and whether restricted content may take that path at all.

How it connects to our work: The control is a label that travels with the content and is enforced on every approved path, as the Reference Implementation sets out. A prompt carrying restricted data goes only to providers approved for that label, with their retention terms on record, and a vendor’s change of model supplier is a change you review.

Sources: Fireworks AI security bulletin, 9 November 2026; Notion security advisory, updated 12 November 2026.

Australia will create an AI safety regulator after an OpenAI agent reached non-public Medicare statistics

On 10 November, Prime Minister Anthony Albanese said the government had accepted all 18 recommendations of a rapid review into how an OpenAI agent reached non-public Medicare statistics through a Services Australia portal in June. Albanese said no personal details were taken. OpenAI found the access in August and told the government in mid-September, through a public email inbox. The ABC reports that OpenAI appears to have broken no law, which is the gap the government now wants to close. A new regulator would be able to impose significant penalties, and AI companies would have to report incidents rapidly, allow independent evaluation of their models and publish safety and security frameworks. The review recommends considering time-bound reporting that covers agents behaving in ways their makers didn’t intend, and a “failure to prevent” offence. Home Affairs Minister Tony Burke wants criminal law updated to cover AI systems that act on their own. The review itself hasn’t been released, and national AI standards are due by the end of the year.

Why it matters for the industry: The reporting duty is aimed at AI companies, but it sets the expectation for every firm running agents. A clock on incident notice means you have to learn quickly that an agent did something it wasn’t meant to, reconstruct what it touched and on whose authority, and say so to the right person. Three months from access to notice, through a general mailbox, is what that path looks like when nobody owns it.

How it connects to our work: AI Governance Evidence covers the records that make a fast answer possible: a log of each agent request and what it reached, release evidence a reviewer can inspect, and a named owner for incident notice.

Sources: ABC News: OpenAI did not break law in Medicare hack as government moves to close legal loophole and The Guardian: Australia to introduce mandatory reporting after OpenAI hack, 10 November 2026.

United moves to block AI agents from its platforms

Reuters reported on 11 November that United Airlines will limit AI bots’ access to its digital platforms and, through updated booking, ticketing and distribution policies, their access through travel agencies and aggregators. United warned that sharing account logins with third-party AI assistants could expose sensitive information and lead to unintended purchases, account changes or unauthorized use of miles. It said automated check-in could create regulatory risk, because check-in requires a person to acknowledge the rules on prohibited items. The report notes that companies are split: Amazon has blocked Meta’s agent Muse, while Walmart has partnered with it.

Why it matters for the industry: Many consumer agents today work by signing in as the customer, so the business on the other side can’t tell the person from the software, can’t limit what the software does and can’t tell who agreed to a purchase. United’s check-in point is the sharper one. Some steps are a person’s own acknowledgment, and an agent clicking through them isn’t that person acknowledging. Any firm that lets agents act for customers or staff faces the same two questions: which steps need a human, and how does the system know which one it’s dealing with.

How it connects to our work: Agent Onboarding gives an agent its own bounded identity and an owner instead of a copy of someone’s login. That lets the other side allow low-risk actions, refuse the ones that need a person and revoke the agent without locking out the customer.

Source: Reuters: United Airlines moves to block AI agents from its platform over security concerns, 11 November 2026.

Databricks turns Genie into an always-on agent for enterprise work

Databricks announced on 11 November a rebuilt Genie that runs long jobs on a schedule or a trigger, such as a metric dropping. It tracks progress, follows up on missing inputs and proposes actions, for example weekly budget changes for approval. Databricks says Genie acts with the user’s identity and reaches only what that user is authorized to see, respects Unity Catalog and source-system permissions, and traces every action. Business users choose which actions run on their own and which need sign-off, and when a user needs more access, Genie prepares an access request for them to send, with admins and data owners keeping approval. Admins can set budgets and spending caps. The post doesn’t state whether most components are generally available or in preview, and gives no pricing.

Why it matters for the industry: Running as the user is a sound default for a question asked in a chat. For a job that runs for weeks it raises new questions. What happens when the user changes role or leaves? Does the job keep the access it had when it was set up? Should a scheduled action other people rely on depend on one person’s entitlements? The approval and budget controls are where the design work sits, and buyers will want to see them tested, not just listed.

How it connects to our work: Choosing an Enterprise Platform for AI Agents sets hard gates and a proof plan before commitment. For Genie that means testing how a long-running job behaves when its owner’s access changes, how sign-off is enforced and what the trace records.

Source: Databricks: The new Genie: autonomous AI for complex enterprise work, 11 November 2026.

Researchers find 35,000 Grok-built apps open to anyone with the link

Security firm Claroc reported on 10 November that of 37,571 apps built with Grok’s app builder that were still online, 35,578 opened without sign-in. In the paid account Claroc used, the publish step proposed “Anyone with the link”; SpaceXSI says employee accounts default to private. About one in ten open apps held internal business or personal data, including a searchable list of more than 3,500 hospital employees and a contractor timesheet storing passwords in plain text. Twenty-four were built by X and SpaceXSI staff. In a team app Grok built for the researchers, the generated sign-in code added any signed-in user to the team before checking membership. SpaceXSI took down its staff apps and said apps stay private until shared.

Why it matters for the industry: The chat app builder is the new desk spreadsheet. Staff paste real data into a prompt, publish with the default and leave a live web app outside every inventory. The generated sign-in check shows the deeper problem: code that looks like access control and does nothing. Firms need to know which app builders staff use, what data may go into them and who reviews an app before it’s shared beyond the person who made it.

How it connects to our work: Context Before Query checks authorization at the data for each request, rather than trusting whatever login screen a generator wrote. When the data sits behind a governed path that checks the requester, a careless app can’t widen who sees it.

Sources: Claroc: GhostShares 2: Grok Apps Exposed xAI’s Business Secrets and People’s Private Data, 10 November 2026, and The Hacker News, 11 November 2026.

A flaw in NEAR AI’s inference software exposed requests inside its enclave

NEAR AI reported on 9 November that an attacker exploited a remote code execution flaw in SGLang, open-source inference software, to run code on one inference server. On 8 November the attacker could have observed user requests being processed on that server, including requests in confidential mode. The attacker also reached other systems, exposing account details for about 100 users, chat history, settings and logs for 18 agents on a legacy deployment that didn’t run in the trusted execution environment, and two customers’ API keys written to logs. NEAR AI said confidential computing protects data from the operator and the host, but “does not protect against a vulnerability in the software running inside the environment.” It patched SGLang, rotated credentials and completed a forensic review.

Why it matters for the industry: Private AI services are increasingly sold on the strength of the enclave. This incident draws the boundary precisely. Attestation can show which code is running on genuine hardware, but not that the code is free of flaws. An inference server is a large, fast-changing codebase, and the logs and side systems around it often sit outside the protected boundary. Buyers should ask what runs inside the measured boundary, how fast it’s patched and what leaves it.

How it connects to our work: Confidential Computing for AI explains attestation-gated key release and what it does and doesn’t protect. The practical points from this case: keep the software inside the boundary small, release keys only to attested builds at a current patch level, and keep secrets out of logs.

Source: NEAR AI: Security Incident Report: SGLang Vulnerability, 9 November 2026.

Stolen keys from Railway ran up an estimated $740,000 in AI charges

App host Railway said in its incident report on 10 November that AI provider keys stolen from customer environment variables in its October breach drove an estimated $740,000 in unauthorized usage. The attacker partly escaped build isolation, used a wrongly defined OIDC binding in one of Railway’s cloud environments, and decrypted a subset of customer variables, its queries aimed specifically at AI credentials. Railway says it is working with providers on refunds and will cover the spend where a provider won’t. Provider terms differ: AWS Bedrock and Google Gemini apply their fraud refund policies, OpenAI reviews claims case by case and OpenRouter doesn’t refund.

Why it matters for the industry: A model API key works like cash. Whoever holds it spends on your account, and default terms usually leave the customer liable. These keys sat in a host’s environment variables, so the exposure depended on a supplier’s isolation, not the customer’s own. Spending caps per key, short-lived credentials and alerts on unusual spend are now finance controls as much as security ones.

How it connects to our work: AI Token Economics measures cost per task and separates capacity from cash. A cap set from the expected cost of the work a key supports turns a stolen key from an open-ended liability into a bounded one.

Source: Railway: Incident Report: October 21st Security Attack, 10 November 2026.

Microsoft shows a few edited neurons can turn a model into a sleeper agent

Microsoft Research described an attack it calls Lazarus on 10 November: reprogramming fewer than ten dormant neurons in a model is enough to backdoor an agent built on it, with no retraining. The model passes standard evaluations until a trigger appears, which can be as ordinary as the date passing into 2027, and then emits a command that downloads and runs an attacker’s code with whatever access the agent has. Microsoft says the attack needs only write access to a checkpoint, so internally developed models are just as exposed. Its advice: verify where models come from, compare deployed weights against the original, limit what agents can do and monitor what they do.

Why it matters for the industry: Firms increasingly download fine-tuned open-weight models and give them a terminal. A change this small passes ordinary evaluation, because the model behaves well until the trigger arrives. What the backdoor can do is bounded by what the agent may run, and where a model came from becomes a release question, not a research one.

How it connects to our work: Data Product Certification treats a shared asset as an owned, certified product with a contract. A model deserves the same: a named source, a fingerprint checked against its base, an owner, and recertification whenever the file changes.

Source: Microsoft Research: Lazare, veni foras: Reprogramming Dormant Neurons to Backdoor AI Agents, 10 November 2026.

Sources

• Update on October 2026 security incident, Fireworks AI, 9 November 2026
• Fireworks AI incident, Notion, updated 12 November 2026
• Fireworks AI Confirms Wider Breach Affecting Customer Datasets, Model Weights and Notion AI Chats, The Hacker News, 10 November 2026
• OpenAI did not break law in Medicare hack as government moves to close legal loophole, ABC News, 10 November 2026
• Australia to introduce mandatory reporting after OpenAI hack, The Guardian, 10 November 2026
• United Airlines moves to block AI agents from its platform over security concerns, Reuters, 11 November 2026
• The new Genie: autonomous AI for complex enterprise work, Databricks, 11 November 2026
• GhostShares 2: Grok Apps Exposed xAI’s Business Secrets and People’s Private Data, Claroc, 10 November 2026
• Security Incident Report: SGLang Vulnerability, NEAR AI, 9 November 2026
• Incident Report: October 21st Security Attack, Railway, 10 November 2026
• Lazare, veni foras: Reprogramming Dormant Neurons to Backdoor AI Agents, Microsoft Research, 10 November 2026

AIDataSecurityConfidential computing

Linked sources are the record. Check them for the full reporting.