Scoping every search to the reader with classifications

Most organizations do not have one audience for their content. A policy library has public guidance, staff procedures, legal advice and personnel files. A case system has material that every analyst may read and material that only one team may read. Building a separate search index for each audience is expensive, and keeping those indexes in step is worse.

foundation4 takes a different route. One pipeline holds the content for every audience. Each search names who is reading, and the results are scoped to that reader in the same database query that ranks them. This post explains the mechanism behind that: classifications. It covers what a classification is, how a pipeline defines a hierarchy of them, how a search selects them, and where the responsibility of your application begins.

What a classification is

A classification is an attribute that a reader must hold to see a document. The docs describe it as following the attribute-based access control (ABAC) model: the document carries an attribute, the reader holds a set of attributes, and a match decides visibility.

Two rules make the model simple to reason about.

Every document carries exactly one classification. The classification field is required when you add a document, and it must name a classification that the pipeline defines. A document with an undefined classification is rejected and nothing is created. Every fragment inherits the classification of its document, so scoping applies to fragments without further setup.

foundation4 assigns no meaning to a classification name. Your organization decides what each one stands for: a clearance level, a team, a tenant, a customer account, or a combination. foundation4 only stores the names, the inheritance between them, and which document carries which.

The docs are also clear about what classifications are for. They suit coarse, stable divisions of content. Scoping that changes per request, such as the rooms or projects a user belongs to, belongs in metadata filters. We return to that split below.

Defining a hierarchy

Classifications are defined per pipeline, in the classifications field when the pipeline is created. Each entry is either a name, or a pair that records an inheritance: ["internal", "public"] means internal inherits public. A reader who holds a classification also sees documents that carry any classification it inherits, and inheritance is transitive.

Here is a hierarchy for a hypothetical policy library:

"classifications": [
  "public",
  ["staff", "public"],
  ["legal", "staff"],
  ["personnel", "staff"],
  ["executive", "legal"],
  ["executive", "personnel"]
]

Reading it from the top:

Reader holds	Sees documents classified
public	public
staff	staff, public
legal	legal, staff, public
personnel	personnel, staff, public
executive	executive, legal, personnel, staff, public

A classification that inherits more than one other classification is listed in one pair for each, as executive is here. Note what the table does not show. A reader holding legal does not see personnel documents, even though both sit under staff. Inheritance runs one way.

The hierarchy can change after the pipeline exists. POST /pipelines/{id}/classifications takes an add list and a remove list in the same entry forms, and applies additions before removals in one transaction. Removing an inheritance leaves both classifications and their documents in place. Removing a classification is different: it permanently deletes every document and fragment that carries it. Content that must survive is submitted again under its new classification first.

Naming the reader on every search

The classification field is required on every search request. It names the classifications the reader holds. It accepts three forms:

Form	Example	Matching
String	"classification": "legal"	Hierarchical
Array	"classification": ["legal", "public"]	Hierarchical
Object	"classification": {"classifications": ["legal"], "search_type": 1}	As set by search_type

In the object form, the array is named classifications, in the plural. The search_type value picks the matching mode:

• Hierarchical (2, the default). The named classifications plus everything they inherit, transitively. Names the pipeline does not define are ignored.
• Exact (1). Only the named classifications. The request fails with HTTP 404 if any name is not defined in the pipeline.
• Hierarchical with validation (3). Hierarchical matching, but the request fails if any name is not defined.

The choice matters more than it looks. Hierarchical matching that ignores unknown names is forgiving. An application can send every attribute a user holds and let the pipeline use the ones it knows. The cost is that a typo narrows the results silently instead of failing. Modes 1 and 3 turn that typo into an error. Exact matching suits a screen that should show one tier only, such as a review queue for legal documents that should not be crowded with staff material.

There is also a safe default at the edge. An empty array, or an array in which no name is defined under hierarchical matching, resolves no classification. The search then returns an empty array with status 200. Leaving the field out entirely returns HTTP 422.

A search request

Here is a search from the policy library for a member of the legal team, using the field names from the search request reference:

{
  "query": "notice period for contract termination",
  "type": "similarity",
  "classification": {"classifications": ["legal"], "search_type": 3},
  "params": {"k": 5},
  "filters": {"region": {"$in": ["north", "central"]}}
}

This reader sees legal, staff and public documents, because legal inherits the other two. The filter then keeps only documents whose region metadata is one of the two values. Each fragment in the response carries its own classification field, so the application can see which tier each result came from.

Scoping and ranking in one query

The order of operations is where this design earns its keep. For a vector search, the API server converts the query into a vector, then resolves the classifications named in the request, including inherited ones when matching is hierarchical. PostgreSQL then selects the matching fragments from the index, restricted to the resolved classifications, the metadata filter and current versions, in a single query. Only then does the API server decrypt the text of the selected fragments.

Two things follow from that.

First, scoping is part of selection, not a pass over results afterwards. A reader's top k are the best matches among the fragments that reader may see, not the survivors of a larger list. The same restriction applies identically to similarity, maximal marginal relevance (MMR) and full-text search.

Second, the classification and the content are read together. A fragment's classification, metadata and expiry live in the same database as its vector, and the search and retrieval page describes all three restrictions applied in that one query.

Classifications and metadata filters together

Classifications and metadata filters are two mechanisms that apply together on every search. A fragment must satisfy both to appear.

They do different jobs. Classifications are defined on the pipeline, change rarely, and form a hierarchy. Metadata is a JSON object on each document, validated against the pipeline's schema, and filters on it are built per request with operators such as $eq, $in and $and. Use classifications for the divisions that rarely move: clearance tiers, departments, tenants. Use metadata filters for the ones that change with each user and each request: the rooms, cases or projects a person is a member of today.

Put together, that gives results scoped to each reader by the classifications and metadata your application supplies.

Where your application fits

That phrase is deliberate. foundation4 applies the classifications that the request names. It does not know who your end user is. Your application holds the user's identity and entitlements, maps them to classifications, and names them on each request. Your application is therefore responsible for naming only the classifications the user actually holds.

The docs describe a working pattern from a chat integration. The application derives each reader's classifications from the roles in its own identity system. Every request names a base classification that every signed-in user holds, plus the name of each global role the user has, with hierarchical matching. Because names the pipeline does not define are ignored, the application can send every role without first checking which ones the pipeline knows. Messages classified with the base name are visible to every reader. Messages classified with a role name are visible only to holders of that role. Each request also carries a metadata filter on the rooms the reader belongs to.

That integration then adds one more step, and the docs recommend it for content with strict access rules: the application confirms the user's access to each result in its own system before displaying it. In the chat case, the application reads each message from its own database and drops any message in a room the reader does not belong to. Scoping in foundation4 narrows the results. The final check stays with the system that owns the identity.

API keys are a separate layer, and it is worth being precise about it. A key's permission on a pipeline can carry a classification allow-list. That list governs adding, expiring and deleting documents: the document's classification must be in the key's allow-list. It does not govern search. A search key needs read and execute permission on the pipeline, and the search request itself names the reader's classifications. So the service that ingests content can be limited to the tiers it is allowed to write, while the search path relies on the application naming the reader correctly.

Why a classification never changes between versions

A document keeps the classification it was given when first submitted. Every later version must carry the same one. A submission with a different classification returns an error, even when every earlier version has expired. Content that moves to a different classification is deleted and submitted again as a new document.

This can feel strict, but it makes the model easy to reason about. A document identifier always belongs to one audience, across its whole version history. Reclassification becomes an explicit act: delete, then add.

Classifications also carry the encryption model: stored text encrypted with a separate key for each classification, generated when the classification is created. A separate post covers what that encryption does and does not protect, and the data protection page gives the detail.

Designing your scheme

A few practical points from the docs:

• Keep classifications coarse and stable. If a division changes per request, it is metadata.
• Choose names that match how your organization already labels content, since foundation4 attaches no meaning to them.
• Decide the matching mode per screen. Validation modes catch mistakes. The default forgives unknown names.
• Plan reclassification as delete and re-add, and plan classification removal as permanent deletion.
• For strict content, check each result against your own system before showing it.

The classifications concept page and the access control page give the full reference.

If you are designing retrieval for several audiences over one body of content, request a briefing and we will walk through how a classification scheme would map to your organization.