Serving and scaling models

Section 4: online vs batch inference, where to serve (Agent Platform endpoints, Model Garden, Cloud Run, GKE), prebuilt and custom containers, custom prediction routines, Model Registry, canary and A/B rollouts, public, dedicated and private endpoints, autoscaling, serving hardware and model optimization. Includes a serving capacity calculator and a rollout lab.

1. Serving models

Online vs batch

Where to serve

Containers

Prebuilt prediction containers serve TensorFlow SavedModel, PyTorch, scikit-learn and XGBoost artifacts from Cloud Storage with no server code. A custom container is needed for other frameworks or custom logic. It must:

• Run an HTTP server on the port in AIP_HTTP_PORT (default 8080).
• Answer health checks on AIP_HEALTH_ROUTE and predictions on AIP_PREDICT_ROUTE.
• Accept {"instances": [...]} and return {"predictions": [...]}.
• Load model files from AIP_STORAGE_URI, and live in Artifact Registry.

Custom prediction routines (CPR) let you add preprocessing and postprocessing in Python (a Predictor class with preprocess, predict, postprocess) and build the container for you, so you don't write a web server.

Model Registry and versions

• Each upload is a model version under one model; aliases (default, champion, challenger) point to versions.
• Register models from custom training, AutoML and BigQuery ML (model_registry='VERTEX_AI'); attach evaluations; deploy any version to an endpoint.

Rollout strategies

# Deploy version 2 alongside version 1, sending it 10% of traffic
gcloud ai endpoints deploy-model ENDPOINT_ID --region=us-central1 \
  --model=MODEL_ID@2 --display-name=churn-v2 --machine-type=n1-standard-4 \
  --min-replica-count=1 --max-replica-count=5 \
  --traffic-split=0=10,DEPLOYED_MODEL_ID_V1=90     # 0 means "the model being deployed"

2. Scaling online serving

Endpoints and networking

Autoscaling and hardware

• min_replica_count keeps capacity warm (latency on spikes, but you pay for it). max_replica_count caps cost.
• Autoscaling follows CPU or accelerator duty cycle against a target (default about 60%), or request metrics on newer options. Scale-out takes minutes, so set the minimum for your baseline and expected surges.
• Some endpoint configurations can scale to zero; check current support. Otherwise the minimum replica bills continuously.
• GPUs for large deep models and LLMs; CPUs for trees and small models (cheaper per prediction at low volume); TPUs for high-throughput LLM serving.
• Several models can share one machine with deployment resource pools (co-hosting) to cut cost for low-traffic models.

Making models faster and cheaper to serve

Serving features

Online models need the same features they were trained with, fresh enough, in milliseconds. Fetch them from Feature Store online serving by entity ID (keyed on user, product, device), or compute request-time features in the request itself. Streaming features flow through Pub/Sub and Dataflow into the store.

3. Serving open LLMs efficiently

4. Serving capacity calculator

Estimate replicas for an online endpoint from peak traffic and what one replica can handle at your latency target (measure that with a load test). The calculator applies the autoscaling target so replicas aren't planned at 100%.

5. Lab: endpoint, canary rollout and rollback

1. Using version 1 from the training lab (or any registered model), create an endpoint and deploy with min 1, max 2 replicas.
2. Load-test with increasing request rates; note the rate one replica handles at your latency target and use it in the capacity calculator.
3. Train version 2, deploy it to the same endpoint with a 10% traffic split, and send 200 requests. Check the split in the endpoint metrics.
4. Roll back: set version 1 to 100%, then undeploy version 2.
5. Run batch inference for the same model over a BigQuery table and compare cost and effort with the endpoint.

Clean up: undeploy all models and delete the endpoint (gcloud ai endpoints undeploy-model, gcloud ai endpoints delete). Endpoints bill for every hour a replica is deployed, even with no traffic.

6. "Which serving option?" chooser

Pick an item to see what it means, why, and the tempting wrong choice.

7. Sample outputs

What the tools return. Expand an item to see a typical result and what to read in it.

deployedModels:
- id: '1111111111111111111'
  displayName: churn-v1
  model: projects/PROJECT_NUMBER/locations/us-central1/models/MODEL_ID@1
  dedicatedResources:
    machineSpec: {machineType: n1-standard-4}
    minReplicaCount: 1
    maxReplicaCount: 5
- id: '2222222222222222222'
  displayName: churn-v2
  model: projects/PROJECT_NUMBER/locations/us-central1/models/MODEL_ID@2
  dedicatedResources:
    machineSpec: {machineType: n1-standard-4}
    minReplicaCount: 1
    maxReplicaCount: 5
trafficSplit:
  '1111111111111111111': 90
  '2222222222222222222': 10

8. Scenarios

Exam-style situations: how to diagnose each one, the likely cause and the fix.

9. Searchable reference

10. Flashcards (click to flip)

11. Common gotchas in one list

12. Practice quiz (19 questions)