The interesting question about an AI assistant inside a clinical system is not what it answers. It is what it refuses to answer.
Psicotime has an assistant called Alma, which helps psychologists with record-keeping and organizing their clinical work. Documents produced with that support need to comply with Resolução CFP 06/2019, the Brazilian federal psychology council's rules for documents produced by psychologists. And the most important product decision we made about it was not a feature. It was a list of things it does not do.
It does not offer a diagnostic hypothesis. It does not suggest a course of treatment. It does not produce, on its own, any document that goes into the record with the professional's name on it.
That is not product modesty. Resolução CFP 06/2019 holds the psychologist ethically and professionally responsible for the information included in psychological documents, alongside any applicable civil and criminal liability. A tool can support the drafting process. It does not replace the professional's review, decision and signature.
Why the prompt is not a guarantee
The most common way to impose a limit on a model is to write that limit into the prompt. "You must not provide diagnoses." It works most of the time, and it is precisely that "most" which stops it from being a guarantee.
An instruction in a prompt is one layer of protection, not a guarantee on its own. It competes with everything else that enters the conversation: accumulated context, the persistence of whoever is asking, a skilful rephrasing. The user does not even have to be trying to get around anything. They only have to ask, in good faith, "and what might that indicate?", and the model's path of least resistance may be to answer.
A limit that depends only on the model following an instruction at the right moment is not a sufficient barrier. It is a behavioral tendency.
Where the limit actually lives
A useful limit lives outside the model. In practice that means designing the system so the forbidden thing is structurally impossible, not merely discouraged.
The assistant has no free actions, it has a closed repertoire. It is not a chat window with access to the system; it executes a small, named set of operations. There is no "conclude diagnosis" operation available to call. The absence is the safeguard.
What it produces is a draft, never a record. Nothing enters the clinical record without a deliberate action by the professional. The difference between "the AI wrote in the record" and "the AI proposed text that the professional read, edited and signed" looks small in the interface, and it is the entire difference ethically and legally.
What it can see is limited by encryption, not policy alone. In Psicotime's current architecture, clinical record content is encrypted with a key derived from the professional's password. This limits Sothen's own operational access to the content in plain text. It does not remove the need to protect metadata, credentials, devices and backups, but it reduces what operating the platform can expose.
That last point is the most important and the least common. Almost every product treats access to customer data as a matter of internal policy: who has permission, who audited it, what training was given. Policy is revocable, and it depends on everyone behaving well, including three years from now, including after an acquisition. Architecture does not depend on behavior.
It costs something. You lose trivial password recovery, you lose a pile of aggregate analytics, you lose the ability to support a customer by looking at their data. It was a price worth paying in a product that carries the contents of psychotherapy sessions.
The question that holds for any regulated field
Health, law, accounting, credit, insurance. Whenever AI touches a field where professional responsibility is defined by law or by a professional body, the design starts from the same question:
What is the worst sentence this system could produce?
Not the most likely one, the worst. Write it down. Then make its emission impossible rather than improbable, by removing the capability instead of instructing against it.
The second question is the complement: who signs the result? If the answer is "the professional", then the whole system has to be built so that they read, edit and take ownership, with enough friction that this does not decay into a reflex click. An AI whose output is too good to be checked is a problem disguised as a feature.
What this does to the product
The most interesting effect of working this way is that the product gets better, not worse.
An assistant with a closed scope is more predictable, and predictability is what makes a professional trust a tool and use it every day. A clear limit is also a commercial argument: you can explain in one sentence what the tool does and what it will never do, to an audience with legitimate reasons to distrust AI handling sensitive data.
This is not AI as ornament, a generic chat bolted onto the side of the screen. It is AI embedded in a specific operation, with its boundary designed before its capability. And the boundary itself is part of what is being sold.