Skip to content

Packaging Model Images

An LLM deployment serves a model either from Hugging Face or from a model image in your private registry. A model image is an ordinary container image that holds the model files in /models. Your cluster pulls it through its in-cluster registry cache and vLLM serves the files from there, so private or fine-tuned weights never leave your registry project and no Hugging Face token is involved.

What goes into the image

  • Model files in /models, directly and not in a subfolder:
    • config.json and, if the model has one, generation_config.json
    • the tokenizer files, such as tokenizer.json, tokenizer_config.json, and vocab.json and merges.txt where the model uses them
    • the weights as .safetensors files, plus model.safetensors.index.json for a model split into several files
  • A model vLLM can serve: a transformers model architecture that vLLM supports.
  • A size that fits one GPU: every LLM deployment runs on one 24 GB GPU, which has room for at most 18 GB of weights.

1. Get the model files

Download a model from Hugging Face with the hf command line tool:

pip install -U huggingface_hub
hf download Qwen/Qwen2.5-3B-Instruct --local-dir model --exclude "*.md"

For a gated or private model, log in first with hf auth login. A model you trained or fine-tuned yourself works the same way once it is saved with save_pretrained(..., safe_serialization=True).

2. Write the Dockerfile

Put this Dockerfile next to the model folder:

FROM busybox:1.37
COPY model/ /models/

The small busybox base keeps the image usable in every way the cluster can attach a model image.

3. Build and push

Log in to the registry and push the image below models/ in your registry project. Your project name is shown in the dashboard on the Apps page, under New App.

docker login registry.alpcrunch.io
docker buildx build --platform linux/amd64 \
  -t registry.alpcrunch.io/<your project>/models/qwen2.5-3b-instruct:v1 --push .
  • models/: only repositories below models/ show up as model images in the dashboard, and the Apps page leaves them out.
  • --platform linux/amd64: the GPU nodes are x86_64 machines. On an Apple Silicon Mac, a build without this flag produces an image the cluster cannot use.
  • Tags: push each version of a model under a new tag. A deployment is pinned to the exact image its tag pointed to when you created it, so pushing a tag again changes no running deployment.

4. Deploy

In the dashboard, open Deployments, click New LLM Deployment and pick Your Registry. Then select the model image and its tag.

  • Model name: clients send the last part of the repository name as the model value, here qwen2.5-3b-instruct.
  • Endpoint: the model is served at https://llm-<deployment name>.<cluster domain>/v1, an OpenAI-compatible API protected by the API key the form shows once.
  • First start: it can take up to 45 minutes, while a GPU node starts and pulls the image.

Larger models

Docker pushes each COPY as one layer, and registries and nodes transfer layers in parallel. For a model with several weight files, one COPY per file makes pushes and pulls faster, and a failed transfer only repeats one file:

FROM busybox:1.37
COPY model/*.json model/*.txt /models/
COPY model/model-00001-of-00002.safetensors /models/
COPY model/model-00002-of-00002.safetensors /models/

Registries store layers compressed, and weights compress by about a fifth. The dashboard therefore refuses images above 18 GB, while a model whose weights exceed 18 GB in an image just below that size fails later, when vLLM loads it onto the GPU.

Troubleshooting

Message or symptom What to do
"has no linux/amd64 image" Build again with --platform linux/amd64.
"was not found in your registry" Check that the repository is below models/ in your project and that the tag exists.
"is too large" Pick a smaller model, or a quantized variant of it (for example AWQ or FP8).
The deployment fails and vLLM finds no model Check that the files sit directly in /models: docker run --rm --platform linux/amd64 <image> ls /models must list config.json.