Skip to content
Sophie Yin
← All work

ML platform optimization

2022 · Shipped · Built at Alt · Snowflake, Airflow, AWS

case study

Problem

Preprocessors, transformers, and the estimator ran as one six-hour task. Nothing in between was saved, so no other model could reuse the features, one changed adjustment meant retesting everything, and the whole task was sized for its heaviest step. Broken deploys surfaced only in production.

What it does

Splits training into five sections that run as separate, parallel tasks per category and per cross-validation fold. Intermediates persist as parquet with Snowflake views on top. Each task is sized for its own memory and watched in Datadog, and downstream DAGs fire in order through triggers and sensors. A pre-deployment gate catches serving failures first.

Fig. 01

One task, then five sections

PreprocessorsStandard steps grouped. Transaction, liquidity, time, and grade adjustments split out so each runs and tests on its own.

Tap a section to see what changed.
Fig. 02

Shipped in three iterations

Split the DAG. Change no tasks.

  • Three DAGs: training, data dumps, alt index(mid risk)
  • Rewire market momentum and live listings(low risk)
  • Per-step runtime logging(low risk)
  • Snowflake connection(low risk)
low riskmid riskhigh risk
The first iteration split the DAG without changing a task, so the risky refactors landed on a proven shape.

System

One lane per category · dashed = task group

One category's lane through core-model-trainingEach category's lane: training data, then model input, then preprocessor output. The lane splits into full-pipeline features (adjustments, then transformers, localized average, and triangulation in parallel) and five cross-validation folds, which rejoin as aggregated features. Then estimator, error model, metrics, promotion once every category's metrics are in, and a single API deploy.FULL-PIPELINE FEATURESCROSS-VALIDATIONWAITS ON EVERY CATEGORYTraining dataModel inputPreprocessor outputAdjustmentsTransformersLocalized averageTriangulationCV sets5 CV foldsSAME FAN-OUT, PER FOLDAggregated setsAggregated featuresEstimatorError modelMetricsPromoteDeploy APIONCE, FOR ALL CATEGORIESOne category's lane through core-model-trainingEach category's lane: training data, then model input, then preprocessor output. The lane splits into full-pipeline features (adjustments, then transformers, localized average, and triangulation in parallel) and five cross-validation folds, which rejoin as aggregated features. Then estimator, error model, metrics, promotion once every category's metrics are in, and a single API deploy.FULL PIPELINECROSS-VALWAITS ON EVERY CATEGORYTraining dataModel inputPreprocessor outputAdjustmentsTransformersLocalized averageTriangulationCV sets5 CV foldsAggregated setsAggregated featuresEstimatorError modelMetricsPromoteDeploy API

StartEvery day, core-model-training starts one lane per category, all the same shape.

Architecture

  1. 01Dump data assets from Postgres
  2. 02Prepare the training dataset per category
  3. 03Preprocessor output, then two branches in parallel: full-pipeline features and 5 CV folds
  4. 04Adjustments, then transformers, localized averages, and triangulation in parallel
  5. 05Aggregated features, estimator, error model, metrics
  6. 06Promote each category that passed; one API deploy
  7. 07Triggers + sensors: Snowflake upload, live listings, alt index, market momentum

Stack

  • OrchestrationAirflow DAGs, task groups, triggers, sensors, pools
  • ComputeECS + EC2 operators, sized per task
  • StorageS3 parquet intermediates
  • DataSnowflake views over each training run
  • MonitoringDatadog APM (ddtrace), logs, ECS metrics
  • TestingPre-deployment framework

Outcomes

  • Production pipeline runtime down 40%
  • Compute costs down 43% — about $95K/year
  • Pre-deployment gate ended the crash-and-revert cycle

Code is private — happy to walk through it.