Pricing model memory reduction

2026 · In Progress · Building at Alt · Valkey, S3, MLflow, AWS

Case study

Code is private

Problem

Each pricing model stores its lookup data inside the saved model file, so every model gets bigger as the data grows. That drives up serving memory and load time and makes the API less stable. Moving the data out is more than a storage change: training, error-model rebuilds, cross-validation, local downloads, experiments, staging, production, and rollback all need the exact data that belongs to a given model.

What it does

Each model becomes an immutable bundle: a slimmer pickle, a manifest that pins exact data IDs and checksums, and the separate lookup files. Training and offline work read indexed local files, and the API reads Valkey in batches. A deploy publishes and verifies a model's data before that model serves any traffic, and cleanup only removes bundles that nothing live, pending, or kept for rollback still uses.

Fig. 01

Where it has to hold up

Training

Components trained in separate steps lose their ratios
Error-model inputs rebuilt from another run's data
Cross-validation folds see full-dataset ratios
Training-only data pruned, blocking rebuilds
Retries overwrite files another job is reading

Experiments

A downloaded model.pkl no longer runs on its own
Reused experiment slots overwrite each other
One test release mixes runs across categories

Release

Model reaches staging before its data
Production reselects “latest” after staging tests
Code, model, and data deploy out of step
Partial retrains pair old models with new error models

Registry

Promotion in the UI doesn't publish or deploy
Demotion deletes data a live API still reads
CLI-only coordination misses UI changes
Rollback targets data that's already gone

Serving

Old and new instances both need their data mid-rollout
An outage read as “missing” silently returns a default price
Per-row network lookups slow bulk pricing
Unbounded local copies recreate the memory problem
Experiments overwrite or starve production data
Smaller pickle, no real ECS savings

Every case reduces to one question: can this consumer find the exact data that belongs to its model?

Moving lookup data out of the pickle touches every stage of the model lifecycle. These are the failure cases the design has to rule out, grouped by where they'd surface. Some are today's workflow constraints and some are new risks introduced by the move.

Fig. 02

One bundle, separate files

Today: one large pickle

Estimators + every lookup table, loaded into each API worker

Versioned model bundle · immutable ID

model.pkl

Estimators + lookup references

manifest.json

Exact data IDs, formats, checksums

Lookup files

Precomputed ratios, indexed

MLflow

Records the model version + the bundle it requires

Durable files to download, rebuild, or restore

Valkey

Serving copy, restorable from S3

Published files are never overwritten. Model v42 always reads bundles ABC + DEF; a new experiment mints new IDs and can't touch v42's data.

A bundle is a group of related files, not an archive. Immutability is what keeps concurrent experiments, retries, and rollbacks from ever reading each other's data.

Fig. 03

Same model, two ways to read its data

Model asks for ratios

Shared lookup interface

Bound to the model's exact bundle IDs · batched reads

Local indexed files

Training · offline · CV folds

Valkey

Pricing API serving

Durable bundle files in S3

A storage outage raises an explicit error. It must never read as “no matching ratio” and fall through to a default price.

The model only talks to a lookup interface. Training and offline analysis read indexed local files without rebuilding every table as Python dictionaries; the pricing API reads a serving cache. Both preserve values and missing-value behavior exactly.

Fig. 04

Save data during training, not at deploy

Fit a training component

Save an immutable component bundle

Component + ratios + manifest, published as soon as it's ready

Assemble final pricing model

Rebuild error-model inputs

Keeps exact component references

Runs and validation folds get separate bundles. Training reads S3 / local files and never depends on the serving cache.

Each fitted component publishes its own bundle the moment it's ready, so the error-model rebuild and final assembly can use it before any model version is registered.

Fig. 05

Publish first, activate second

Pin the release

Code + category models + error models

Resolve bundles

Read manifests for exact IDs

Publish

Private ECS job copies S3 → cache

Complete + verified?

Gate before activation

Start API

Pinned versions only

Serve traffic

After readiness checks

No → activation stops and the current release keeps serving.

Experiment API

Resolve each category separately, including production defaults

Staging → production

Promote the same pinned bundles; never reselect latest

Routine retraining

Validate every category's pricing / error / data pairing

One deployment coordinates code, pricing models, error models, and lookup data. They don't have to upload at the same instant; the data just has to be verified before the model serves a request.

Fig. 06

A dedicated cache per environment

S3 bundles + MLflow manifests

One source of truth for every environment

Dedicated cache

Primary + replica across AZs

Production pricing API

Dedicated cache

Tested with the exact pinned release

Staging pricing API

Dedicated cache

Distinct bundle IDs per experiment

Experiment pricing API

Each API gets an explicit cache endpoint. It's never inferred from ENV, which is how the experiment service can run production config without reaching production data.

Existing platform broker caches stay separate. They aren't spare pricing capacity.

Production, staging, and experiments each get their own pricing cache, so an experiment can never overwrite production data or consume its capacity.

Fig. 07

Keeping prediction fast

Prediction request

Collect + dedupe keys

Across the whole batch

Batch reads per stage

Pipelined, pooled connections

Reuse values + predict

Small, hot tables stay in memory, bounded per worker.

No network round trip per row, and no S3 on the request path.

Acceptance gate: end-to-end p95 / p99 latency against today's model, across cold and warm caches, bulk pricing, concurrency, and failover.

Moving data over the network only works if each request stays at a handful of round trips. Keys are deduplicated across the request and read in pipelined batches per model stage.

Fig. 08

Old bundles: retain briefly, then clean up

Publish + validate

Active release

Retired

Bounded rollback window

Still referenced?

Live, draining, pending, or protected

Delete cache copy

Bounded batches, async UNLINK

S3 copy retained

Longer, separate policy

Yes → the bundle stays in the window. If usage is uncertain, it's kept.

An older rollback republishes from S3 and verifies before activation.

MLflow demotion alone never authorizes deletion. There is no automatic expiry and the cache runs noeviction.

Several versions coexist only temporarily. The cache protects active releases and a bounded rollback window; S3 protects historical rebuilds and offline downloads for much longer.

Fig. 09

Cache capacity through a release

One release
4 categories × 4 GiB
16 GiB
Active + rollback
Two releases retained
32 GiB
Next upload lands
Peak — three releases present
48 GiB
Cleanup removes oldest
Back to steady state
32 GiB

Budgeted as full copies at measured cache footprint, not pickle size. A replica holds another copy and doesn't add usable capacity.

Sizing follows the release cycle, not one model: a rollback copy is always retained, and the next upload briefly adds a third. Cleanup frees capacity, but a fixed node bill only drops if the cache is resized.

Architecture

01Each training component publishes an immutable bundle
02Manifest pins exact lookup IDs in MLflow
03Durable bundle files land in S3
04Deploy publishes the pinned bundles to that environment's cache
05Data verified, then the API goes live
06Batched lookups at prediction time
07Scheduled cleanup of unused bundles

Stack

Serving cacheAmazon ElastiCache for Valkey
Durable storageVersioned bundles in S3
RegistryMLflow
OrchestrationAirflow training, GitHub Actions deploys
RuntimePricing API on ECS
Infra as codeTerraform

Outcomes

Target: serving memory stops tracking data size
Target: a more stable pricing API
Target: potential infrastructure savings once ECS is right-sized against measured memory savings
Each model, error model, and lookup snapshot identifiable as one compatible release

Code is private — happy to walk through it.

Curious how this would look on your problem?

sophie.fc.yin@gmail.com

Seattle, WA · Sophie YinLinkedIn