Secrets and KMS providers
Two resource products sit next to each other in the rail and solve different security problems. Treating them as one "crypto settings" screen is the usual source of wrong fixes and failed go-lives.
| Surface | Problem it solves | What operators handle | What applications use later |
|---|---|---|---|
| Secrets | Durable, opaque credentials and config secrets (passwords, API keys, vault tokens, integration material) | Which backend holds secrets, who may attach which backend, migration between backends | A Secret ID (sec_...) that never reveals the plaintext after write |
| Key Management (KMS) providers | Durable cryptographic key stores for sign, verify, encrypt, decrypt, MAC | Which key store exists, provider id, default store, share or enable platform offers | A provider id plus key alias (and often a DID verification method built on that key) |
Secrets answers: where does the SMTP password (or vault token) live, and how does a tenant get a safe handle to it?
KMS answers: where does the private key material for AS, issuer, verifier, status lists, and DIDs live, and under which providerId do services call generate/sign?
They connect in a few places only. A software KMS keystore may be unlocked with a password stored as a secret. A cloud KMS account key may be staged through write-only secret fields. That still does not make "Secrets" the place to look when a JWT signature fails. Signature failures belong to KMS health and key aliases; SMTP auth failures belong to the email account and the secret handle behind its password.
Audience: platform operators (platform storage, shared secret providers, tenant policy, optional KMS sharing) and tenant admins (tenant secret assignment, migrations, KMS providers for that tenant).
Prerequisites: a running platform with an active license that exposes Secrets and Key Management; at least one tenant from Onboard a tenant. Real Vault or cloud backends need network paths and credentials that exist outside the browser.
Reference: Secrets storage, Secrets provider, Shared providers, Tenant policy, Migrations, KMS providers.
APIs: Secret Management, Platform Config (tenant secret assignment and options, KMS resources), KMS.
What you see when you open these screens depends on onboarding and product defaults. A lab tenant may already have environment secret storage and a product-managed software KMS. A clean production tenant may show empty providers until you attach them. Empty is not always an error; it is often "not provisioned yet."
Mental model
Secrets: three layers
Secret management is built as three layers so platform credentials never leak into tenant UIs.
-
Provider definition (platform or tenant operated)
The real backend: Vault address, AWS account, Azure vault, environment, or KMS-backed secret store. Definitions have revisions. Credentials are staged write-only, tested, preflighted, then marked ready. Health is derived from probes, not from guessing. -
Offering (platform only)
A published, tenant-facing face of a platform definition. Tenants see display name, capabilities, isolation mode, and whether migration is allowed. They do not see the platform definition id, physical locator, or platform vault token. Isolation mode (for example tenant-key, tenant-namespace, tenant-vault) is the contract that keeps one tenant from reading another tenant's material on a shared backend. -
Assignment (exactly one active per tenant for secret storage)
The tenant's current secret backend. Source is either a platform offering or a tenant-managed provider. Product code that creates a secret handle or resolves SMTP credentials always goes through this assignment. If assignment is wrong or missing, features fail at send/use time even when forms still open.
Platform storage is the same idea for the platform itself (bootstrap secrets, platform-operated integrations). Do not point platform storage at a customer-controlled vault; keep platform and tenant trust boundaries separate.
KMS: two planes on one screen
Key Management Providers merges two planes so operators do not maintain two menus:
| Plane | API family | Answers |
|---|---|---|
| Management resource | Platform-config KMS resources | How the store is configured, credential ownership, share/detach/retire, version for optimistic concurrency |
| Runtime provider | Tenant KMS REST | What is live for generate/sign, providerId, type, default |
Ownership (this tenant vs platform provided) is a badge, not a technology. Technology is Software KMS, AWS KMS, Azure Key Vault, and so on. A platform-shared software KMS and a tenant-owned AWS KMS both appear as providers; only ownership and actions differ.
Provider id is permanent after create. Keys, DID verification methods, and service configs store that id. Renaming is not supported; pick default (common for product-managed software) or a stable business id once.
Secret ID vs key alias
| Identifier | Issued by | Reveals plaintext? | Typical use |
|---|---|---|---|
Secret ID (sec_...) | Secret management after create/rotate | Never after write | Email SMTP password, connector API key, keystore password reference |
| Key alias | KMS generate/import | No (private key stays in the store) | AS signing, issuer, verifier, status list, DID VM |
When an operator form says "use existing Secret ID" or "create new secret," it is writing a handle into config, not pasting a password into YAML for later display.
Why secret handling is strict
The console never shows a secret value after submit. That is intentional:
- Browser history, screen shares, and support screenshots must not recover production passwords.
- Rotation stages a new value; blank fields on rotate usually mean "keep current." Explicit clear is a separate, confirmed action.
- Mutations use strong ETag /
If-Match. Two operators editing the same assignment or policy get a conflict instead of a silent last-write-wins overwrite.
If you need the value again, you rotate or re-stage it. There is no "reveal" control by design.
Platform vs tenant: who does what
| Task | Platform tenant | Customer tenant |
|---|---|---|
| Platform secret storage (including environment) | Yes | No |
| Shared secret provider definitions and publish offerings | Yes | No |
| Global tenant secret policy and overrides | Yes | No (inherits or override applies) |
| Tenant secret assignment (active storage) | No (other tenants) | Yes for self |
| Migrate to another offering / manage tenant-owned secret providers | No | Yes, if policy allows |
| Add/share KMS resources for many tenants | Share from platform | Enable offer or add own KMS |
| Day-to-day keys and DIDs | Usually not | Yes; see Keys and DID |
Switch the tenant picker to platform for License, platform Secrets tabs, and platform KMS sharing. Work in the customer tenant for that organization's assignment, provider tab, and key stores.
Isolation modes (secrets)
When a platform offering is published, isolation mode is part of the contract. It describes how the backend separates tenants. Wrong isolation on a shared Vault or cloud account is a multi-tenant data-risk decision, not a UI preference.
| Mode (examples) | Intent |
|---|---|
| tenant-key | Per-tenant cryptographic separation or keying material |
| tenant-namespace / tenant-vault | Path or mount isolation inside one control plane |
| tenant-policy / tenant-role | Access controlled by IAM/policy attachment per tenant |
| read-only-deployment | Values come from deployment; console writes disabled |
Require backend-enforced isolation in tenant policy when your threat model forbids "shared credentials, soft path prefixes only." Policy cannot invent isolation the backend does not implement; it can only refuse offerings that do not match.
Secrets: tenant storage (active assignment)
In a customer tenant, Storage shows the single active assignment: display name, provider type, lifecycle state, and whether the source is a platform offering or tenant-managed.
Consequences:
- Healthy assignment means create/rotate secret handles and consumers (email, connectors) can resolve values through that backend.
- Missing or failed assignment means those consumers fail when they need a secret, while many forms still open and validate. Fix storage first.
- Switching assignment is not a display rename. It is a migration (copy, validate, cutover) when moving between backends. Use Provider and Migrations, not an ad-hoc config edit.
Navigation: Resources > Secrets > Storage
Deep link: #resource=secrets&rarea=storage
- Admin Console
- REST API
Tenant assignment (Platform Config, tenant path):
- Overview
- Request
- Response
Get tenant secret assignment
Endpoint: GET /api/platform/config/v1/tenants/acme/secrets/assignment
Captured response: 200 OK
This captured endpoint is shown from the E2E run; it is not mapped to one of the generated EDK REST API reference pages.
Connect an environment to rewrite this call to real service bases and run it.
Secrets: tenant Provider tab
The Provider tab is where a tenant understands current vs possible backends.
Current provider
States that this tenant has one active secret provider, and whether it came from a platform offering or is tenant-managed. That sentence matters for who can rotate credentials: platform offerings usually rotate on the platform side; tenant-managed providers rotate here with write-only fields.
Available platform offerings
Rows are migration targets, not a multi-select of concurrent stores. Available means the offering is permitted for this tenant under policy. It does not mean the tenant is already using it.
| Usage (server-derived) | Meaning |
|---|---|
| In use / selected | This offering backs the active assignment |
| Available | Allowed to migrate to, not currently selected |
| Migration target | Destination of an open migration |
Migrate to this offering runs preflight, then starts a migration that only becomes the active assignment after success. Blocked reasons are explicit: already selected, migration in progress, offering disabled, or policy denied. Do not force migration while another migration is open; finish, roll back, or purge according to product controls first.
Tenant-managed providers
When policy allows, the tenant can run its own Vault or cloud secret manager. New tenant provider walks the typed setup catalog (provider type, configuration, credentials, test, preflight, ready). That path gives the tenant operational control and also the duty to rotate credentials, monitor health, and keep network access working. Prefer platform offerings when the product standard is "platform operates the vault; tenants only assign."
Navigation: Resources > Secrets > Provider
Deep link: #resource=secrets&rarea=provider
- Admin Console
- REST API
Options bundle effective policy and permitted offerings:
- Overview
- Request
- Response
Get tenant secret options
Endpoint: GET /api/platform/config/v1/tenants/acme/secrets/options
Captured response: 200 OK
This captured endpoint is shown from the E2E run; it is not mapped to one of the generated EDK REST API reference pages.
Connect an environment to rewrite this call to real service bases and run it.
Secrets: migrations
Migrations move secret material between backends with a staged lifecycle so cutover is not a single unrecoverable write.
Typical phases (names may match product labels):
- Preflight - connectivity, capability, and isolation checks; produces a short-lived preflight token.
- Copy - material is written to the destination without yet switching readers.
- Validation - destination is checked before cutover.
- Cutover - assignment points at the new backend; traffic follows.
- Retention - old backend may be retained for rollback for retention days from policy.
- Rollback or purge - return to the retained source, or permanently drop it.
Resume continues a paused or recoverable step. Rollback is for failed or unwanted cutovers while retention still holds the source. Purge is permanent removal of the retained source after you accept that rollback is no longer possible.
Do not purge on the first green day of production cutover. Wait until consumers (email, connectors, KMS credential refs) have proven the new assignment under load.
Navigation: Resources > Secrets > Migrations
Deep link: #resource=secrets&rarea=migrations
Empty list means no migration is in flight. That is normal most of the time.
Secrets: platform storage
On the platform tenant, Storage is the platform's own secret assignment.
Environment storage is common in packaged deploys: values come from deployment configuration (env, mounted files, orchestrator secrets). The console often shows read-only and disables credential priming and writes. That is not a bug; it means "change these in the deployment, not in the browser."
Change storage starts a controlled workflow only when the setup catalog offers another platform backend. Use it when you deliberately move platform secrets into Vault or a cloud secret manager for operational reasons. Keep that backend under platform control.
Navigation: (platform tenant) Resources > Secrets > Storage
Deep link: #tenant=platform&resource=secrets&rarea=storage
- Admin Console
- REST API
- Overview
- Request
- Response
Get platform secret storage
Endpoint: GET /api/platform/admin/v1/application/secrets/storage
Captured response: 200 OK
This captured endpoint is shown from the E2E run; it is not mapped to one of the generated EDK REST API reference pages.
Connect an environment to rewrite this call to real service bases and run it.
Secrets: shared providers and offerings (platform)
Platform operators build the backends tenants will consume without ever showing tenants the platform credential.
Shared provider definitions
Each definition is a backend the platform operates: display name, type, role (platform-offering), revisions, health, credential state. Lifecycle moves through candidate, ready, active, and later suspended or retired shapes depending on product.
New shared provider is a multi-step workflow:
- Choose type and configuration from the setup catalog.
- Stage write-only credentials (token, client secret, access key).
- Test connectivity (authenticate and/or read-write probe as the server defines).
- Preflight capability and isolation checks.
- Mark ready and continue to publish.
Rotate credentials updates backend auth without making every tenant re-enter anything. Blank fields typically retain configured values; rotation is for the platform definition, not for each tenant assignment form.
Published offerings
An offering is what tenants list under "Available platform offerings." It carries isolation mode, capabilities, enable flag, assignment count, and revision. Disable stops new adoption and migration onto that offering according to product rules; existing assignments may remain until those tenants migrate away. Disabling is how you wind down a shared vault without deleting definitions mid-incident.
| If you... | Then... |
|---|---|
| Publish with weak isolation | Tenants can adopt a backend that does not meet your multi-tenant bar |
| Leave an old offering enabled forever | Tenants keep migrating onto a backend you meant to retire |
| Rotate platform credentials without testing | All assigned tenants can fail secret resolve at once |
| Skip preflight | Failures appear at cutover instead of before copy |
Navigation: (platform tenant) Resources > Secrets > Shared providers
Deep link: #tenant=platform&resource=secrets&rarea=sharedProviders
- Admin Console
- REST API
List published offerings:
- Overview
- Request
- Response
List platform secret offerings
Endpoint: GET /api/platform/admin/v1/application/secrets/offerings
Captured response: 200 OK
This captured endpoint is shown from the E2E run; it is not mapped to one of the generated EDK REST API reference pages.
Connect an environment to rewrite this call to real service bases and run it.
Secrets: tenant policy (platform)
Tenant policy is the rulebook every tenant inherits unless an override exists.
| Policy field | Why it exists |
|---|---|
| Allow tenant-managed providers | If false, tenants cannot attach their own Vault/cloud; only platform offerings. Use when you want one operated estate. |
| Allow platform offerings | If false, tenants cannot adopt shared offerings (unusual; mostly for lockdown or break-glass). |
| Require backend-enforced isolation | Refuse offerings that do not meet isolation expectations. |
| Allowed provider types | Limit types (vault, kms, aws-secrets-manager, ...). Stops surprise backends in regulated deployments. |
| Retention days | How long a migration may keep the old backend for rollback before purge is expected. |
Overrides attach a complete policy to one tenant when that customer has a different contract (for example they must run their own Vault). Prefer a tight global policy and few overrides. A global policy that allows every type and unrestricted tenant-managed providers maximizes flexibility and also maximizes incident blast radius and support load.
Search and pick tenants from the directory; do not invent free-text tenant ids that the product cannot resolve.
Navigation: (platform tenant) Resources > Secrets > Tenant policy
Deep link: #tenant=platform&resource=secrets&rarea=tenantPolicy
- Admin Console
- REST API
GET/PUT under Platform Admin secret-management tenant-policy paths. See Secret Management API.
Provider types (secrets backends)
| Type | Typical role | Notes |
|---|---|---|
| Environment | Deploy-time values | Often read-only in console; good for labs and fixed platform bootstrap |
| HashiCorp Vault | Shared or tenant-operated secret engine | Isolation mode and mounts matter for multi-tenant |
| AWS Secrets Manager / Azure Key Vault | Cloud-managed secrets | Credential ownership and IAM alignment with the tenant |
| Kubernetes mount | Cluster-injected material | Lifecycle tied to deployment, not browser rotation |
| KMS-backed storage | Secrets envelope-encrypted via KMS | Still a secrets assignment; not a substitute for Key Management providers used for AS/issuer signing keys |
Pick types your operators can actually monitor. A Vault nobody can reach from the product network is worse than environment storage that is known and backed up by deployment process.
KMS providers (key stores)
What the Providers table is for
Key Management > Providers is the place a KMS is seen, added, shared, enabled, defaulted, and retired. It is not the key inventory; keys live under Keys (see Keys and DID). Without at least one healthy provider, generate key, import key, and most DID creation paths have nowhere to put private material.
| Column / signal | Meaning |
|---|---|
| Name | Operator label (for example Software KMS) |
| Technology | Software, AWS KMS, Azure Key Vault, ... |
| Owner | This tenant vs platform provided |
| Status | Ready, needs credential, detached, and related health |
| Default | Where new keys go unless the operator picks another provider |
Empty list: onboarding did not attach a KMS, or your product expects operators to Add KMS here. That is common on clean production tenants.
Add KMS: decisions that stick
| Decision | Why it matters |
|---|---|
| Technology | Locks the backend class and field set (file keystore vs cloud region/vault URL). |
| Credential ownership | Product managed: product holds or provisions credentials. Tenant supplied: tenant enters cloud credentials (write-only). Wrong choice blocks validation forever or puts secrets in the wrong trust zone. |
| Provider id | Immutable. Services and keys store it. Prefer stable short ids (default, aws-prod). |
| Display name | Changeable operator label; does not replace provider id. |
| Technology-specific fields | Software: memory vs key store file (memory loses keys on restart). Cloud: region, vault URL, auth mode. |
After create, open the row for configuration, credential status (never displayed as plaintext), validate, and provider-scoped keys.
Platform offers vs share
- Available from the platform / Enable (tenant view): the platform already shared a KMS; the tenant deliberately enables a tenant-facing use of it without seeing platform credentials. Enabling is intentional so a shared store does not silently appear as a surprise default.
- Share (platform view): platform attaches selected tenants to a shareable resource. Sharing expands blast radius; only share stores designed for multi-tenant isolation.
Default, detach, retire
- Set default changes where new keys are created. Existing keys stay on their original provider id.
- Detach removes management association according to confirmations; runtime may still reference keys until cleaned up.
- Retire is a stronger lifecycle end. Confirm dialogs exist because retiring the only store can break AS, issuer, verifier, and DIDs that still point at its keys.
Never retire the only production provider to "clean up" before you have moved signing material and updated dependents.
Navigation: Resources > Key Management > Providers
Deep link: #resource=kms&rarea=providers
- Admin Console
- REST API
Offerings (what can be attached):
- Overview
- Request
- Response
List KMS offerings
Endpoint: GET /api/platform/config/v1/tenants/00000000-0000-4000-8000-000000000000/kms/offerings
Captured response: 200 OK
This captured endpoint is shown from the E2E run; it is not mapped to one of the generated EDK REST API reference pages.
Connect an environment to rewrite this call to real service bases and run it.
Management resources:
- Overview
- Request
- Response
List KMS resources
Endpoint: GET /api/platform/config/v1/tenants/00000000-0000-4000-8000-000000000000/kms/resources
Captured response: 200 OK
This captured endpoint is shown from the E2E run; it is not mapped to one of the generated EDK REST API reference pages.
Connect an environment to rewrite this call to real service bases and run it.
Runtime providers (what can sign):
- Overview
- Request
- Response
List runtime KMS providers
Endpoint: GET /api/kms/v1/providers
Captured response: 200 OK
This captured endpoint is shown from the E2E run; it is not mapped to one of the generated EDK REST API reference pages.
Connect an environment to rewrite this call to real service bases and run it.
For generate/import key, key detail, and DID binding, continue with Keys and DID.
How Secrets and KMS work in one deployment
A typical healthy platform looks like this:
- Platform secret storage is environment or a platform-operated Vault.
- Platform publishes one or more secret offerings with a real isolation mode.
- Tenant policy allows platform offerings (and only the provider types you support).
- Each tenant has a secret assignment (often the default offering after onboarding).
- Each tenant has at least one KMS provider (often product-managed software
default, or a cloud KMS). - Onboarding or operators create keys and DIDs on that KMS.
- Email accounts and connectors store Secret IDs, not passwords in clear config.
- Protocol instances reference KMS provider id + key alias for signing.
Failure patterns:
| Symptom | Check first | Not first |
|---|---|---|
| SMTP test-send fails auth | Email account From/host, then secret handle for password | KMS providers list |
| JWT or VC signature fails | KMS provider status, key alias, default provider | Secrets offerings table |
| "Could not load secrets" | Secret-management API health, tenant assignment, ETag conflicts | Branding |
| Tenant cannot migrate to offering | Tenant policy, offering enabled, isolation, open migration | License modules (unless Secrets product is locked) |
| Empty KMS providers after onboard | Product provisioning flags (keys and DIDs), then Add KMS | Secret migration history |
Design choices and consequences
| Choice | Upside | Downside if wrong |
|---|---|---|
| One shared Vault offering for all tenants | One ops team, one rotation process | Isolation or network failure affects everyone |
| Tenant-managed Vault per customer | Strong isolation, customer-owned credentials | Every tenant needs skilled operators; policy and support cost rise |
| Software KMS file keystore | Simple lab and many SaaS defaults | File backup, host affinity, and restart behavior must be understood |
| Software KMS memory only | Fast demos | Keys vanish on restart; never production |
| Platform-shared KMS | Fast tenant enable, no tenant cloud bill | Shared failure domain; share only when designed for it |
| Long migration retention | Safe rollback | Old backend still holds copies; purge discipline required |
| Short retention / immediate purge | Smaller residual risk | No rollback if cutover was wrong |
Recommended order of work
Platform first
- Confirm platform secret storage (environment or operated backend).
- Create shared provider definitions, test, ready, publish offerings with correct isolation.
- Set tenant policy (types, isolation, tenant-managed yes/no, retention).
- Optionally prepare shareable KMS resources if the product uses platform KMS offers.
Per tenant
- Confirm secret assignment on Storage (or complete onboarding that set it).
- Only then configure email SMTP and other secret consumers.
- Ensure a KMS provider exists (onboarding, enable platform offer, or Add KMS).
- Set default KMS if more than one provider exists.
- Keys and DID: generate or import keys, create DIDs, bind verification methods.
- Point AS, issuer, verifier, and status lists at the correct provider id and aliases.
When moving backends
- Publish or enable the destination offering (or tenant-managed provider).
- Preflight and start migration; watch Migrations until cutover.
- Exercise real consumers (test email, sign a token, issue a credential).
- Keep retention until confident; then purge if the product requires it.
After secrets and KMS providers
- Keys and DID for key and identifier lifecycle
- Branding and email for SMTP that depends on secret handles
- License and capabilities if Secrets or Key Management are missing from the rail
- Onboard a tenant capability flags (keys and DIDs, sample data) that pre-create stores