Running retrieval on an isolated network: what works offline

foundation4 is built for isolated networks. Model weights and Python packages ship as container images, the Helm charts include every dependency chart, and foundation4 sends no telemetry to the vendor. There is one qualifier you need to plan around: some embedding models and the token splitter download files on first use, so they do not work without internet access.

This post is a planning checklist for a team installing foundation4 on a network with no route to the internet. The full reference is the Models and packages page of the documentation.

The outcome you are planning for

The goal is a deployment that makes no connection outside your network. The docs state the condition plainly. When the endpoints you configure are inside your network, and your pipelines use only the models and splitters that work without internet access, foundation4 makes no connection outside the network (Security at a glance).

Content, vectors, keys and logs stay in the deployment's PostgreSQL database, cache and cluster. No foundation4 cloud service takes part in any request. The only outbound connections are the ones an administrator configures: registered language model endpoints and, if configured, a remote embedding endpoint. So your content leaves your perimeter only if an administrator registers a model server outside it.

Planning therefore comes down to two things: choose components that work offline, and get every image into a registry the cluster can reach.

How models and packages reach the cluster

foundation4 does not fetch models at run time from a model hub. Model files and Python packages arrive as container images, and Kubernetes mounts them into the pods.

The API server and the workers run from the same image. Each API server pod and each worker pod also runs a gRPC service as a sidecar, a second container in the same pod. That Python service hosts the Python embedding providers and every text splitter.

Four kinds of image take part:

• The API server image. It carries the foundation4 binary and the files of two FastEmbed models.
• The gRPC service image. It carries the Python service and the LangChain libraries. It does not include the GPT4All, Sentence Transformers or NLTK packages.
• Package images. Each holds one Python package and its dependencies, such as GPT4All or Sentence Transformers. The gRPC service loads it at startup.
• Model images. Each holds the model files for one provider, mounted at the directory that provider reads.

Model and package images use the Kubernetes image volume source. The kubelet pulls an OCI image and mounts its content read-only, without running it. Your cluster therefore needs image volume support in the API server, the kubelet and the container runtime, and you confirm it yourself, because the charts declare no minimum Kubernetes version. An installation that uses only the models already in the API server image needs no image volumes at all.

Embedding models that work offline

The two models in the API server image

FastEmbed runs ONNX models inside the API server and worker processes. It makes no external calls when the model files are present. The API server image carries the files of two of the ten built-in FastEmbed models:

• Qdrant/all-MiniLM-L6-v2-onnx, the seeded model, 384 dimensions.
• nomic-ai/nomic-embed-text-v1.5, 768 dimensions.

These work offline with no extra images. Every installation seeds an embedding model named all-MiniLM-L6-v2 that uses the first of them, so a first pipeline can be created without configuring a provider.

The other eight built-in models

FastEmbed downloads the files of the other eight built-in models from Hugging Face the first time each process loads the model. On an isolated network that download fails. Plan as if these eight models are unavailable.

Custom ONNX models

A custom ONNX model is the supported way to add a FastEmbed model to an isolated deployment. FastEmbed reads custom models only from local files and never downloads them.

You package the model as a FastEmbed model image. The image starts from the FastEmbed files of the API server image, then adds your model directory in the Hugging Face cache layout: one snapshot that holds a single ONNX file and four tokenizer and configuration JSON files.

Keeping the API server's files in the new image is not optional. A model image mounted at the FastEmbed model directory replaces that directory. If your image leaves out the seeded model's files, the seeded model, and every pipeline that uses it, can no longer embed text.

After the image is mounted, you create the embedding model through the API. This request comes from the docs:

curl -X POST "$FOUNDATION4_URL/embedding-models" \
  -H "x-api-key: $FOUNDATION4_API_KEY" \
  -H "x-api-key-secret: $FOUNDATION4_API_SECRET" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "example-embed",
    "description": "Custom ONNX model from the FastEmbed model image",
    "provider": "FastEmbedEmbeddings",
    "model": "example-org/example-embed-onnx",
    "size": 0,
    "parameters": {"onnx": "onnx/model.onnx", "pooling": "mean"}
  }'

A size of 0 tells foundation4 to measure the dimension. foundation4 embeds a test string when it creates the model, so a missing file fails at creation, with HTTP 400 and Error reading <file> in details.error.

Use a single-file ONNX export. foundation4 reads only the ONNX file, so a model that stores weights in separate data files cannot be loaded.

OpenAI-compatible embedding servers inside the network

If you already run an embeddings server, the OpenAIEmbeddings provider calls any server that implements the OpenAI embeddings API. Its endpoint parameter defaults to the OpenAI service, so on an isolated network you set endpoint to your internal server. A server without authentication needs no api_key.

GPT4All and Sentence Transformers: test first

Both providers run in the gRPC service and need a package image and a model image. Both libraries can contact their own services while loading a model: the GPT4All service, and the Hugging Face Hub. The docs mark both as not verified offline. Before you rely on either one, create an embedding model with it in a test environment that has the same network restrictions as production.

A Hugging Face inference endpoint works only if the endpoint is inside your network and the gRPC service can reach it.

Text splitters that work offline

• Work offline: the recursive character, character, code and Markdown header splitters. They need no network access. The seeded recursive character splitter is one of them.
• Needs internet: the token splitter. Its tokenizer library downloads encoding files that no image provides.
• Not available: the NLTK sentence splitter. Every installation seeds one, but no image provides the nltk Python package, so it cannot split text.

Language models for generation

foundation4 includes no language model and has no default model server. Retrieval works without one. If you want generation, an administrator registers a model server inside your network with POST /llms. foundation4 works with any OpenAI-compatible model that the deployment can reach.

The API server makes the call, so your network policy must allow traffic from the API server pods to the model server. The OpenAI-compatible chat completions endpoint in foundation4 needs only chat completions on the model server. The direct query endpoint, which is the documented way to test a new registration, calls the OpenAI Responses API and needs an API key in the registration. Any non-empty value works for a server that does not check keys.

What works offline, in one table

Capability	Works with no internet access?	Why
FastEmbed, the two models in the API server image	Yes	Files are in the image, or in a FastEmbed model image that replaces the directory
FastEmbed, the other eight built-in models	No	Files download from Hugging Face on first use in each process
FastEmbed, custom ONNX models	Yes	Files come from your FastEmbed model image
OpenAI-compatible embeddings	Yes	When endpoint names a server inside the network
Sentence Transformers	Not verified	The library can contact the Hugging Face Hub while loading a model
GPT4All	Not verified	The library can contact the GPT4All service while loading a model
Hugging Face inference endpoint	Only with an internal endpoint	The gRPC service must reach it
Recursive character, character, code and Markdown header splitters	Yes	No network access needed
Token splitter	No	Encoding files download on first use
NLTK splitter	Not available	No image provides the package
Generation through a registered model server	Yes, with an internal server	The API server pods must reach it
The CLI command that downloads models	No	It fills only the pod that runs it and prepares no other pod

Getting images into the network

An isolated installation copies every image the releases use into a registry inside your network, then points each values key at the copy.

Both foundation4 charts include every dependency chart, including PostgreSQL, Valkey, NATS and Prometheus. Installation runs no helm dependency command and needs no access to public chart repositories.

The docs describe two ways to build the image list:

• From a staging installation with internet access. List every image and image volume reference in the namespace, save them to one archive, then load and push them inside the network. This gives the exact tags.
• From an installation bundle. The deployment files, image archives and example values can be supplied as one bundle. Verify it against its checksum file before you extract it. Do not use the example values unchanged: they point at an outside registry and can enable debug logging.

Three details catch teams out. The images are built for amd64 nodes. Each chart reads pull secrets from its own value, and the worker pods ignore the per-image pull secrets of model and package images, so those also go in the global list. And some dependency checks start public utility images, so mirror those too or run the equivalent pre-installation checks. Bring kubectl, Helm and a container runtime for the mirroring host into the network as well.

Pre-install checklist

Work through this before the first helm install:

1. Kubernetes version. 1.27 or later, with image volume support if you will add any model or package image (Pre-installation checks).
2. Node architecture. amd64 on the nodes that run foundation4.
3. Embedding models. Every model your pipelines use is one of the two in the API server image, a custom ONNX model in your own image, or an OpenAI-compatible server inside the network.
4. FastEmbed model image contents. If you build one, it includes the seeded model's files. After mounting it, compare the FastEmbed directory in the server and worker containers, as the docs describe.
5. GPT4All or Sentence Transformers. Tested in an environment with the same network restrictions, or left out.
6. Text splitters. Recursive character, character, code or Markdown header. No token or NLTK splitters.
7. Model servers. Registered endpoints are inside the network, reachable from the API server pods, and implement the OpenAI APIs your features call.
8. TLS to internal servers. A model or embedding server that uses TLS must present a certificate the API server image trusts. The charts provide no value to add a certificate authority. If your servers use a private certificate authority, raise it with us before installation.
9. Images mirrored. Every foundation4, core, model, package and utility image is in your registry, with pull secrets in each value the charts read.
10. Registry tokens. Tokens that do not expire, or a refresh process. An expired token stops new pulls, including the installation Jobs of the next upgrade.
11. Pre-installation checks. After the first five steps of Install on Kubernetes, run the installation test pod, then repeat the image test with the core release images.

Next steps

If you are planning a foundation4 installation on an isolated network and want to walk through your model and registry choices, request a briefing.