ML pipelines, retraining and CI/CD/CT

Section 5: Agent Platform Pipelines (Kubeflow Pipelines and TFX), Managed Service for Apache Airflow, Ray, data and model validation gates, keeping preprocessing consistent between training and serving, retraining policies, MLOps maturity levels and CI/CD/CT with Cloud Build. Includes a pipeline lab.

1. End-to-end ML pipelines

A pipeline turns notebook steps into a repeatable, versioned, automated workflow: ingest → validate data → transform → train → evaluate → validate the model → register → deploy → monitor. Each run records lineage, so you can answer "which data and code produced the model in production?"

Orchestrators

from kfp import dsl, compiler
from google_cloud_pipeline_components.v1.custom_job import CustomTrainingJobOp

@dsl.component(base_image="python:3.11", packages_to_install=["pandas", "google-cloud-bigquery"])
def validate_data(table: str) -> bool:
    ...  # row counts, schema, nulls, ranges; return False to stop the run

@dsl.component(base_image="python:3.11")
def evaluate(model: dsl.Input[dsl.Model], metrics: dsl.Output[dsl.Metrics]) -> float:
    ...

@dsl.pipeline(name="churn-training")
def pipeline(table: str = "bq://PROJECT_ID.shop.train", min_auc_pr: float = 0.65):
    ok = validate_data(table=table)
    with dsl.If(ok.output == True):
        train = CustomTrainingJobOp(display_name="train", worker_pool_specs=[...])
        score = evaluate(model=...)
        with dsl.If(score.output >= min_auc_pr):      # model validation gate
            ...  # ModelUploadOp → ModelDeployOp (or register as "challenger")

compiler.Compiler().compile(pipeline, "pipeline.yaml")
# aiplatform.PipelineJob(display_name="churn", template_path="pipeline.yaml",
#                        enable_caching=True, parameter_values={...}).submit()

TFX components (TensorFlow Extended)

Validating data and models

• Data validation: schema (types, required columns), ranges, null rates, category values, distribution shifts against training statistics. Stop the pipeline on failure instead of training on bad data.
• Model validation: compare the candidate with thresholds and with the current production model, overall and per slice (region, device, demographic group) to catch hidden regressions and fairness issues. Only then register or deploy.
• Infrastructure validation: the model loads in the serving container and answers within latency limits.

Consistent preprocessing between training and serving

2. Retraining and CI/CD/CT

Retraining policy

CI/CD/CT

Google's MLOps maturity levels: level 0 is manual notebooks and hand deployment; level 1 automates training with a pipeline (CT); level 2 automates the pipeline's own build, test and deployment (CI/CD of the pipeline).

# cloudbuild.yaml (sketch): test, build the trainer image, compile and upload the pipeline
steps:
- name: python:3.11
  entrypoint: bash
  args: ["-c", "pip install -r requirements.txt && pytest tests/"]
- name: gcr.io/cloud-builders/docker
  args: ["build", "-t", "us-docker.pkg.dev/PROJECT_ID/ml/trainer:$SHORT_SHA", "trainer/"]
- name: python:3.11
  entrypoint: bash
  args: ["-c", "pip install kfp && python compile_pipeline.py && python upload_template.py"]
images: ["us-docker.pkg.dev/PROJECT_ID/ml/trainer:$SHORT_SHA"]

3. MLOps maturity at a glance

4. Lab: a Kubeflow pipeline with a validation gate

1. Create three lightweight components: validate_data (checks row count and nulls), train, and evaluate (returns a metric).
2. Wire them with conditions so training only runs when validation passes, and registration happens only when the metric beats the current production model.
3. Compile to YAML, submit with PipelineJob(enable_caching=True), then re-run unchanged and watch cached steps skip.
4. Open the lineage view for the resulting model and trace it back to the dataset and run parameters.
5. Add a Cloud Build trigger that runs component unit tests and recompiles the pipeline on each commit.
6. Create a weekly schedule for the pipeline; then delete the schedule.

5. "Which orchestrator?" chooser

Pick an item to see what it means, why, and the tempting wrong choice.

6. Sample outputs

What the tools return. Expand an item to see a typical result and what to read in it.

Run: churn-training-20261114-0200   State: SUCCEEDED
  validate-data        SUCCEEDED   (cached)
  train                SUCCEEDED   41m
  evaluate             SUCCEEDED   3m     output: 0.612
  condition-1 (score >= 0.65)  SKIPPED
    model-upload       NOT_TRIGGERED
    model-deploy       NOT_TRIGGERED

7. Scenarios

Exam-style situations: how to diagnose each one, the likely cause and the fix.

8. Searchable reference

9. Flashcards (click to flip)

10. Common gotchas in one list

11. Practice quiz (18 questions)