← All Google Cloud AI guidesSection 3 · 3.1–3.3

Training, hyperparameter tuning and hardware

Section 3: choosing the model type and product, organizing training data, managed custom training (prebuilt vs custom containers, worker pools, Spot and reserved capacity), troubleshooting failed training, hyperparameter tuning with Vizier, fine-tuning foundation models, CPU vs GPU vs TPU, and distributed training strategies.

1. Choosing the model, product and approach

Model type

Problem	Good first choice	Why
Tabular classification / regression	Gradient-boosted trees (XGBoost, BQML boosted trees, AutoML tabular)	Strong accuracy, little preprocessing, feature importance
Must explain decisions to regulators	Linear/logistic regression, small trees, or boosted trees with Explainable AI	Interpretable by design, or attributions per prediction
Univariate or many-series forecasting	ARIMA_PLUS (BQML), AutoML forecasting, TimesFM	Seasonality and holidays handled; fast
Images, audio, unstructured patterns	Deep neural networks (DNN, CNN, transformers), transfer learning	Learns representations; start from a pretrained model
Text generation, summarization, extraction, chat, multimodal reasoning	LLM (Gemini) with prompting, RAG or tuning	No task-specific training data needed to start
Recommendations	Matrix factorization, two-tower retrieval + ranking, or Recommendations AI	Collaborative signals at scale
Anomaly detection with no labels	k-means distance, autoencoder, ARIMA_PLUS anomalies	Unsupervised

Product

Product	Choose when
Prebuilt API	Generic task, standard classes
BigQuery ML	Data in BigQuery, SQL team, batch scoring
AutoML	Custom labels, best accuracy with no ML code
Gemini / Model Garden	Language, vision or multimodal tasks; quick prototypes
Managed (custom) training	Your code and framework, scaled and managed
Pipelines	Any of the above, repeatable and automated
GKE / Ray	Existing Kubernetes or Ray platform, special scheduling, multi-tenant clusters

Deployment strategy

Requirement	Strategy
Predictions needed per request in under ~100 ms	Online endpoint (Agent Platform, Cloud Run or GKE), features from the online Feature Store
Scores needed daily for all customers	Batch inference (or BQML `ML.PREDICT`) writing to BigQuery; no endpoint to pay for
Precompute is possible and inputs are known in advance	Batch precompute, then serve lookups from a low-latency store
No connectivity, privacy on device, ultra-low latency	Edge: TF Lite / LiteRT on device, exported AutoML Edge models

Interpretability is a requirement, not a nice-to-have, when the question mentions regulators, adverse action, credit, healthcare or "explain to the customer". Choose interpretable models or enable Explainable AI, and say so in the design.

2. Training and tuning on Google Cloud

Organizing and ingesting training data

Tabular: BigQuery tables, read by AutoML and pipelines directly, or by custom code through the BigQuery Storage Read API (fast parallel reads into TensorFlow, PyTorch or pandas).
Unstructured: files in Cloud Storage plus a manifest. Custom training jobs see buckets as local paths through Cloud Storage FUSE (/gcs/my-bucket/...).
Managed datasets on Agent Platform hold image, video and tabular datasets with labels and splits, used by AutoML and for lineage.
Ingest from other systems with Dataflow, Datastream (database change data capture into BigQuery), Storage Transfer Service, or BigQuery Data Transfer Service.

Managed training (custom training jobs)

Option	When
Prebuilt training container + your Python package or script	Your framework is supported (TensorFlow, PyTorch, scikit-learn, XGBoost); least work
Custom container (image in Artifact Registry)	Unusual dependencies, other languages, exact environment control
Worker pools	Pool 0 is the chief; further pools are workers, parameter servers or a reduction server
Serverless Training vs Managed Training Clusters	Pay per job with on-demand capacity, or reserved, persistent clusters for large, frequent training
Spot VMs, reservations, Dynamic Workload Scheduler	Cheaper interruptible capacity (checkpoint!), guaranteed capacity, or queued access to scarce GPUs
Kubeflow on GKE / Ray on Agent Platform	Existing Kubernetes skills, custom operators; Ray for Python-native distributed data and training
AutoML and Tabular Workflows	No training code; Tabular Workflows expose the AutoML pipeline for more control

gcloud ai custom-jobs create --region=us-central1 --display-name=churn-train \
  --python-package-uris=gs://BUCKET/trainer-0.1.tar.gz \
  --worker-pool-spec=machine-type=n1-standard-8,accelerator-type=NVIDIA_TESLA_T4,accelerator-count=1,replica-count=1,executor-image-uri=PREBUILT_PYTORCH_IMAGE,python-module=trainer.task \
  --args=--epochs=10,--data=bq://PROJECT_ID.shop.train
# The job writes its model to the directory in AIP_MODEL_DIR (a Cloud Storage path).

Troubleshooting training failures

Symptom	Likely cause	Fix
Job fails immediately with permission denied	The training service account can't read the bucket or BigQuery table	Grant the job's service account the roles it needs (it's not your user identity)
Out of memory (host or GPU)	Batch size too large, data loaded fully into memory	Smaller batches, streaming input (`tf.data`, DataLoader), larger machine or more GPUs
Loss becomes NaN	Learning rate too high, unscaled features, bad labels, log(0)	Lower learning rate, normalize, gradient clipping, check data
Quota exceeded / resources unavailable	GPU quota or capacity in the region	Request quota, another region or accelerator, reservations or Dynamic Workload Scheduler
Job restarted and lost hours of work	Spot preemption or maintenance without checkpoints	Checkpoint to Cloud Storage regularly; resume from the latest checkpoint
GPU utilization low	Input pipeline bottleneck	Prefetch, parallel reads, TFRecord shards, cache; profile with TensorBoard Profiler
Training accuracy high, validation poor	Overfitting	Regularization, dropout, more data, augmentation, early stopping
Both training and validation poor	Underfitting or bad features	Bigger model, better features, train longer

Hyperparameter tuning

Hyperparameter tuning jobs (powered by Vizier) run many trials of your training code with different values and optimize a metric your code reports (with the cloudml-hypertune helper or the metric reporting API).

Algorithms: Bayesian optimization (default), grid search, random search.
max_trial_count bounds cost; parallel_trial_count trades wall-clock time against quality: more parallel trials finish sooner, but Bayesian search learns less from earlier trials.
Use log scaling for learning rates, discrete values for layer counts, and early stopping of poor trials.
Simple alternatives: AutoML (tunes for you), BQML num_trials with hyperparameter ranges.

Fine-tuning foundation models

Option	Use
Supervised tuning of Gemini on Agent Platform (or from BigQuery ML)	Managed, parameter-efficient tuning on JSONL prompt/response pairs; the tuned model gets its own endpoint
Open models from Model Garden (Gemma, Llama) with LoRA / QLoRA notebooks or managed tuning	Full control of weights; deploy where you want
Reinforcement learning / preference tuning	Align to human preferences

Consider tuning only when prompting and grounding fall short and you have good labelled examples. Benefits: higher quality on a narrow task, shorter prompts (cheaper, faster), consistent format. Costs: data preparation, tuning jobs, re-tuning when the base model is updated, and evaluation work.

3. Hardware and distributed training

Hardware	Best for	Watch out
CPU	Classic ML (scikit-learn, XGBoost on modest data), preprocessing, small models, low-volume serving	Slow for deep learning
GPU (NVIDIA L4, A100, H100, H200, B200...)	Most deep learning; flexible across frameworks; custom CUDA ops; inference with TensorRT or vLLM	Quota and capacity; cost when idle
TPU (Cloud TPU v5e, v5p, Trillium, Ironwood...)	Large matrix-heavy models in JAX, TensorFlow or PyTorch/XLA; large-batch training; high-throughput LLM serving at low cost	Custom ops not supported by XLA, heavy dynamic shapes, high-precision (float64) math and very small models fit poorly
Edge (phones, Edge TPU, Jetson)	On-device inference, offline use	Needs quantized, small models

Accelerator generations change quickly. The exam tests the trade-off (CPU vs GPU vs TPU, and when TPUs don't fit), not the newest SKU.

Distributed training strategies

Strategy	How	When
Data parallelism	Copy the model to every device; split each batch; average gradients with all-reduce	The model fits on one device and you want speed. TensorFlow `MirroredStrategy` (one machine), `MultiWorkerMirroredStrategy` (many machines), `TPUStrategy`; PyTorch DDP
Parameter server	Workers push gradients to parameter servers asynchronously	Large sparse models (embeddings), many CPU workers
Model parallelism (tensor, pipeline)	Split layers or tensors across devices	The model doesn't fit on one device
Sharded data parallel (FSDP, ZeRO)	Shard parameters, gradients and optimizer states across data-parallel workers	Large models with data-parallel simplicity
Reduction Server (Agent Platform)	Extra reducer nodes speed up all-reduce for multi-node GPU data parallel jobs	Network-bound gradient synchronization

"Fits? Data. Doesn't fit? Model." If one copy fits on one device, scale out with data parallelism; if it doesn't, split the model.

More GPUs doesn't fix an input-bound job. If GPUs sit at 30% utilization, fix the data pipeline first. And with data parallelism, scale the learning rate or batch size deliberately: global batch = per-device batch × devices.

4. Accelerator memory calculator

A rough first estimate of whether a model fits on one accelerator, and therefore whether data parallelism is enough or the model must be sharded. Real usage also depends on activations, sequence length and batch size.

Parameters (billions)
Workload
Memory per accelerator (GB)
Overhead for activations / KV cache (%)

Bytes per parameter: 16-bit inference ≈ 2; int8 ≈ 1; int4 ≈ 0.5; full mixed-precision training with Adam ≈ 16 (weights, gradients, two optimizer states, master weights).

5. Lab: custom training, tuning and the registry

Placeholders: PROJECT_ID, BUCKET, region us-central1.

Write a scikit-learn or XGBoost script that reads from BigQuery, trains, reports a metric with cloudml-hypertune, and saves the model to AIP_MODEL_DIR.
Run it as a custom training job with a prebuilt container and a dedicated, least-privilege service account. Remove its BigQuery role once and read the 403 error: which principal does it name?
Run a hyperparameter tuning job with 12 trials, parallel 3, a log-scaled learning rate and a discrete max depth. Compare with parallel 12: what changes in wall-clock time and best metric?
Add checkpointing to Cloud Storage and rerun on Spot VMs; confirm it resumes after a restart.
Upload the best model to Model Registry as version 1 with a prebuilt prediction container. Continue in the serving lab.

Clean up: delete jobs' output you don't need and the staging bucket contents.

6. "Which training setup?" chooser

Pick an item to see what it means, why, and the tempting wrong choice.

Pick one above.

7. Sample outputs

What the tools return. Expand an item to see a typical result and what to read in it.

Illustrative output, not captured from a real project. IDs, numbers and some field names are invented or simplified, and formats change between releases. Learn the shape and the fields called out under each one.

gcloud ai hp-tuning-jobs describe JOB_ID --region=us-central1Hyperparameter tuning trials

state: JOB_STATE_SUCCEEDED
maxTrialCount: 20
parallelTrialCount: 4
studySpec:
  algorithm: ALGORITHM_UNSPECIFIED      # Bayesian optimization by default
  metrics: [{metricId: auc_pr, goal: MAXIMIZE}]
trials:
- id: '7'   parameters: [{parameterId: learning_rate, value: 0.042}, {parameterId: max_depth, value: 7}]
  finalMeasurement: {metrics: [{metricId: auc_pr, value: 0.734}]}
- id: '12'  state: INFEASIBLE
- id: '15'  parameters: [{parameterId: learning_rate, value: 0.31}, {parameterId: max_depth, value: 3}]
  finalMeasurement: {metrics: [{metricId: auc_pr, value: 0.611}]}

What to look at: The best trial (7) wins on the reported metric auc_pr, which the training code must report. Four trials ran in parallel: faster, but each wave learns less from the previous ones. An INFEASIBLE trial usually means the code crashed or didn't report the metric for those values.

Training job log (custom job)A failure you should recognize

INFO  Reading training data from bq://PROJECT_ID.shop.train
ERROR google.api_core.exceptions.Forbidden: 403 Access Denied: Table PROJECT_ID:shop.train:
      User does not have bigquery.tables.getData permission.
      principal: service-PROJECT_NUMBER@gcp-sa-aiplatform-cc.iam.gserviceaccount.com
Job failed: The replica workerpool0-0 exited with a non-zero status of 1.

What to look at: The job runs as a service account, not as you. Grant that identity (or, better, a custom service account you pass to the job) BigQuery Data Viewer on the dataset and BigQuery Job User on the project.

8. Scenarios

Exam-style situations: how to diagnose each one, the likely cause and the fix.

9. Searchable reference

Item	Meaning

10. Flashcards (click to flip)

11. Common gotchas in one list

The job runs as a service account, not as you: grant it the roles.
Save models to AIP_MODEL_DIR so the platform can find them.
Spot VMs need checkpoints or every preemption restarts training.
Fits on one device → data parallel; doesn't → shard or split the model.
Low GPU utilization is usually the input pipeline, not too few GPUs.
TPUs don't fit custom CUDA ops, float64 or heavy dynamic shapes.
More parallel trials finish sooner but search less adaptively.
Use log scale for learning rates in tuning.
Full fine-tuning needs ~8× the memory of 16-bit inference; LoRA and quantization cut it.
Prebuilt containers before custom containers when the framework is supported.
Interpretability requirements drive model choice in regulated domains.

12. Practice quiz (19 questions)

Score: 0 / 0