Executive summary
Picture a bank's service team asking a new AI assistant about a customer's loan. The answer is quick and accurate. It also quotes a line from a credit committee note that the service team was never allowed to open. Nobody broke in. The assistant was connected to a document store through one broad service account, and it retrieved whatever matched the question.
That is how most AI systems leak data: through access nobody scoped. IBM's 2025 Cost of a Data Breach research found that 13% of organizations reported a breach of AI models or applications, and 97% of those lacked proper AI access controls. Our position is simple. An AI system should see only what the person it serves may see, send out only what the task needs, and keep nothing longer than your rules allow. Client data is never used to train models.
This is Part 3 of our series. It expands one of the six controls in our governance model for AI agents: data boundaries.
1. Four ways data leaves an AI system
Most AI data incidents follow one of four paths. Each needs its own boundary.
- Retrieval reaches too far. The system reads documents the user could not open. OWASP lists this under vector and embedding weaknesses: a shared index without access partitions leaks across user groups.
- Prompts carry too much. Names, account numbers and clinical notes go to an external model when the task needed none of them.
- Outputs and logs linger. Prompts, answers and traces sit in stores nobody reviews, for longer than anyone decided.
- Data crosses a line you did not draw. A request is processed in another country, or under terms that let the provider keep it.
Shadow AI makes all four worse, because nobody set any boundary at all. The same IBM research found that one in five organizations reported a breach due to shadow AI. The answer is a sanctioned tool with the boundaries built in, so people have less reason to paste data into one without them.
What it may read
RuleOnly what the person it serves may open.
BuildPermission-aware retrieval, with the search run as that user.
What it may send out
RuleOnly the fields the task needs.
BuildPersonal data masked at the gateway before any prompt leaves.
Where it runs
RuleIn a region and on terms you have approved.
BuildA model allow-list with region pinning and no training on your data.
How long it is kept
RulePrompts and outputs live for a named period.
BuildA short-lived content store, and an audit trail on your record rules.
HMR insight: An AI system should never see more than the person it is working for.
2. Classify data before you connect it
You can't set a boundary around data you haven't labelled. Most enterprises already have a classification scheme for documents. The work is to decide what an AI system may do with each class, then enforce it in the platform rather than in a policy.
| Class | Examples | What AI may do |
|---|---|---|
| Public | Product pages, published rates | Read, with any approved model |
| Internal | Policies, process notes | Read, using the user's own access |
| Confidential | Contracts, credit notes, pricing | Read with the user's rights, in-region models only |
| Restricted | Health records, Aadhaar, card data | Masked before any prompt, some fields never sent |
Start with the sources, not the whole estate. Label the three or four repositories your first use case needs. Leave everything else unconnected until it is labelled. An unlabelled shared drive is an incident waiting for a question.
3. Retrieval that respects permissions
Retrieval-augmented generation (RAG) is how most enterprise AI answers questions from your own documents. It is also where most over-sharing starts. The usual shortcut is to index everything with one service account and let the model sort it out. The model can't. It has no idea who is asking.
Permission-aware retrieval fixes this at the source. OWASP recommends permission-aware vector stores with strict partitions between user groups. In practice, that means three rules:
- Carry permissions into the index. Each chunk keeps the access list of its source document. The index updates when access changes.
- Filter before ranking. The search runs as the user, so documents they cannot open never become candidates. Filtering after the model has seen them is too late.
- Log what was retrieved. Record which documents fed each answer, so a reviewer can trace any output to its sources.
Agents follow the same rule for tools. An agent acting for a person gets that person's data scope, or a narrower one. It never gets the sum of every system it can reach. This is the scoped identity control from Part 2, applied to data.
- A question arrivesIt carries the identity of the person asking, not a shared account.
- Search runs as that personDocuments they cannot open are filtered out before ranking.
- Personal data is maskedNames, numbers and IDs become placeholders before the prompt is sent.
- Answer and sources are loggedThe trail records which documents fed the answer, and for whom.
4. Mask personal data before the prompt
Most tasks don't need to know who the customer is. A model can summarize a complaint, classify a claim or draft a reply with the name, phone number and account number replaced by placeholders. The gateway puts the real values back only where the output needs them, and only inside your systems.
Masking works best as a gateway step, not a habit you ask every developer to remember. Open-source tools such as Microsoft's Presidio detect common identifiers. You add rules for Indian formats: Aadhaar, PAN, mobile and account numbers. Test on your own documents. A detector tuned on English text can miss a name in a scanned Hindi form.
Some fields should not leave at all, even masked. Card data and full health records are the usual cases. For those, run the model inside your own environment or keep the field out of scope.
5. Residency and provider terms
Where a prompt is processed matters as much as where data is stored. Two questions settle most of it.
Where does the request run? Regional processing exists, but it is not always the default. Microsoft documents that a Standard deployment processes prompts in the geography you choose, while a Global deployment may process them in any geography where the model is deployed. OpenAI lists India among its data residency regions, with some controls needing approval.
For regulated data, the rules may decide for you. The RBI's 2018 direction on payment data requires payment system data to be stored only in India. The DPDP Act also lets the government restrict transfers to countries it notifies.
What does the provider keep, and can it train on it? Read the terms for the service you actually use, not the consumer app. OpenAI states that API data has not been used to train its models since March 2023 unless you opt in, and that abuse monitoring logs are kept for up to 30 days. Microsoft states that prompts and completions on its hosted models are not used to train models and are not shared with OpenAI.
| Ask every provider | What good looks like |
|---|---|
| Is our data used for training? | No, by contract, with no opt-out needed |
| Where are prompts processed? | A named region that you choose |
| How long are prompts kept? | Stated in days, with zero retention on request |
| Who can read them? | Human review only under named conditions |
| What happens when we leave? | Deletion confirmed in writing |
6. Decide how long prompts and outputs live
Every AI system creates new records: prompts, retrieved passages, answers, approvals and traces. They often hold the same personal data as the source systems, with none of the retention rules. Treat them as records in their own right.
We keep them in two stores. The audit trail records who did what, when and on whose authority. It holds references and masked values, and follows your audit retention. The content store keeps full prompts and outputs only where there is a reason, for a short, named period. The DPDP Rules require security logs to be kept for at least one year, so set audit retention with that floor in mind.
Erasure needs the same care. The Act requires personal data to be erased once its purpose is served or consent is withdrawn (section 8(7)). Search indexes, embeddings and caches are copies too. Build deletion into the index from the start, because retrofitting it is slow.
7. What the DPDP Act asks of an AI system
The DPDP Act 2023 doesn't mention AI. It doesn't need to. An AI system that handles personal data is processing it, and your duties as a data fiduciary apply. The DPDP Rules were notified on 14 November 2025, and most duties take effect eighteen months later, in May 2027. Rule 7 asks for a detailed breach report to the Data Protection Board within 72 hours of becoming aware. Penalties for failing to take reasonable security safeguards go up to ₹250 crore.
| Duty | Where it bites in AI | Control |
|---|---|---|
| Purpose limitation | Support chats reused for sales | A purpose tag on every source |
| Security safeguards | Broad retrieval, raw prompts | Scoped retrieval, masking, access logs |
| Processor contracts | The model provider holds your data | Terms on training, region and retention |
| Erasure | Copies in indexes, caches and logs | Deletion that reaches every copy |
| Breach reporting | A leak through an AI answer | Retrieval logs that show who saw what |
8. Illustrative scenario: a hospital group's clinical assistant
This scenario is illustrative. The company is fictional.
Narmada Health runs eight hospitals and wants an assistant to answer staff questions on clinical protocols and draft discharge summaries. Its first pilot indexed a shared drive, which turned out to hold HR files next to the protocols. The privacy officer stopped it.
The second design starts with classes. Protocols are internal, so any clinician can ask about them. Patient records are restricted. The assistant reads a record only for the doctor treating that patient, and names and hospital numbers become placeholders before the prompt is sent. The model runs in an Indian region, on terms that rule out training on the group's data. Every draft summary waits for the treating doctor to sign it, a tier 2 step in the Part 2 model.
Full prompts are kept for a short period agreed with the privacy officer. The audit trail keeps references for at least a year. What changes is what the privacy officer can answer. For any summary, she can see which records fed it, who asked and who signed it. That evidence is what lets the pilot go live.
9. Putting it in place
- Weeks 1 to 3AssessThe Readiness Audit maps the sources the first use case needs, their classes and who can open them.
- Weeks 4 to 6Label and connectLabel those sources, carry permissions into the index and agree provider terms.
- Weeks 7 to 10Build the gateway stepsMasking, region pinning, retrieval logs and retention rules, proven on one use case.
- After thatWiden one source at a timeConnect the next repository only once it is labelled and its access is clean.
- Assess. Our three-week Readiness Audit maps the sources your first use case needs, their classes, who can open them and where the data may go.
- Build. Label those sources, carry permissions into the index and add masking, region pinning and retrieval logs at the gateway. Agree provider terms before the first prompt leaves.
- Run and improve. Review retrieval logs with the data owner each month at first. Connect a new source only once it is labelled. Test masking again whenever a new document type arrives.
Data boundaries decide what an AI system can see. Part 4 of this series covers who approves what it then does, and Part 6 covers what to do when a boundary fails.
Moving forward
Pick one use case and list every source it touches. If you can't say who may open each one, that is where the work starts. To see how we handle data in our own delivery, read our AI governance policy. To have your first use case mapped, book a readiness audit.
Sources
- IBM, Cost of a Data Breach Report 2025: press release
- OWASP Gen AI Security Project, LLM08:2025 Vector and Embedding Weaknesses
- Microsoft, Presidio: data protection and de-identification SDK
- Microsoft Learn, Data, privacy and security for Azure-hosted models
- OpenAI, Data controls in the OpenAI platform
- Reserve Bank of India, Storage of Payment System Data, 6 April 2018
- MeitY, The Digital Personal Data Protection Act, 2023
- MeitY, Digital Personal Data Protection Rules, 2025
- Press Information Bureau, DPDP Rules, 2025 notified