Data tools, Feature Store, notebooks and experiment tracking

Section 2 on Google Cloud: where each kind of data lives, choosing BigQuery, Dataflow, Spark or pandas for preprocessing, the BigQuery-backed Feature Store, protecting PII, Workbench vs Colab Enterprise and their security, and tracking experiments, artifacts and lineage.

1. Where data lives

Reading data efficiently for training

• BigQuery Storage Read API: fast, parallel streaming of table rows into training jobs (used by BigQuery connectors in TensorFlow, PyTorch and Spark).
• Cloud Storage FUSE: custom training jobs see buckets as files under /gcs/BUCKET.
• Prefer fewer, larger files (TFRecord, Parquet shards of roughly 100 MB or more) over millions of tiny files.
• Keep data and compute in the same region to avoid egress cost and latency.
• Use managed datasets (tabular, image, text, video) on Agent Platform for AutoML and labelled data versioning.

2. Choosing a preprocessing tool

Data validation and quality

Labelling data

• Write clear labelling guidelines with examples; measure agreement between labellers.
• Use Gemini to pre-label and humans to review, or active learning to label the most informative examples first.
• Store labels with their source and version so models can be traced to the labels used.

3. Feature Store on Agent Platform

The Feature Store lets teams define features once and reuse them for training and serving, which removes a major source of training-serving skew. The current version is BigQuery-native:

• Feature data stays in BigQuery (tables or views). A feature group registers a BigQuery source and its features.
• Offline: training data comes from BigQuery with point-in-time correct lookups (feature values as they were at each label's timestamp), avoiding leakage.
• Online: an online store with feature views syncs selected features for low-latency serving. Bigtable online serving suits large data; optimized online serving gives the lowest latency and supports embedding (vector) retrieval.
• Sync on a schedule or continuously; serve with FetchFeatureValues by entity ID.

Offline (training)                          Online (serving)
BigQuery tables/views  ── feature groups ──► feature views ──sync──► online store
        │                                                                 │
point-in-time join with labels                               FetchFeatureValues(entity_id)
        ▼                                                                 ▼
  training dataset                                              endpoint / app

4. Privacy and sensitive data

5. Prototyping in notebooks

Security and collaboration best practices

• Run the notebook as a dedicated service account with least privilege; for personal instances use single-user access (end-user credentials) so actions are attributable.
• No public IP, a private VPC subnet, and VPC Service Controls around sensitive projects; reach Google APIs through Private Google Access.
• CMEK on disks when required; never store keys or passwords in notebooks: use Secret Manager.
• Idle shutdown to stop paying for forgotten GPUs.
• Keep code in Git (Workbench has a Git extension); move reusable code into packages and pipelines rather than sharing notebooks by email.
• Admins standardize runtimes with Colab Enterprise runtime templates or Workbench custom container images.

Frameworks and models

Both environments come with TensorFlow, PyTorch, scikit-learn, XGBoost and JAX images, plus the Agent Platform SDK. From a notebook you can call Gemini, deploy open models from Model Garden for experiments, or open a Model Garden model card's sample notebook directly in Colab Enterprise.

6. Experiments, lineage and the registry

from google.cloud import aiplatform
aiplatform.init(project="PROJECT_ID", location="us-central1",
                experiment="churn-xgb", experiment_tensorboard=True)
aiplatform.start_run("run-depth6-lr01")
aiplatform.log_params({"max_depth": 6, "learning_rate": 0.1})
# ... train ...
aiplatform.log_metrics({"auc_pr": 0.71, "recall_at_p90": 0.42})
aiplatform.end_run()
aiplatform.get_experiment_df()        # compare runs as a DataFrame

Lineage questions and where they're answered

7. Lab: from raw data to a tracked experiment

1. Create a Colab Enterprise notebook (or a Workbench instance with idle shutdown on, no public IP) in us-central1.
2. Run Sensitive Data Protection inspection on a sample table with a free-text column; create a de-identified copy (masking or tokenization).
3. Register the de-identified BigQuery table as a Feature Store feature group; create a feature view with two features and sync it to an online store; fetch values for one entity ID.
4. Build a point-in-time training set by joining labels with feature values as of each label timestamp.
5. Train two XGBoost variants in the notebook; log parameters and metrics to Experiments; compare them with get_experiment_df().

Clean up: delete the online store, feature view, de-identified tables and notebook runtime.

8. "Which data tool?" chooser

Pick an item to see what it means, why, and the tempting wrong choice.

9. Scenarios

Exam-style situations: how to diagnose each one, the likely cause and the fix.

10. Searchable reference

11. Flashcards (click to flip)

12. Common gotchas in one list

13. Practice quiz (20 questions)