Writing on Ai/Ml Development

A client came to us in January with a $47,000/month OpenAI bill across three production systems. By the end of February, the same workload was running at $18,500/month with no measurable quality regression. The reduction was not from a single optimization but from four changes applied in a specific order — and one optimization we abandoned because it did not survive contact with real traffic.

Step one was instrumentation. The client had no per-feature token tracking. We added a thin middleware layer between application code and the OpenAI SDK that logged tokens-per-call, latency, and feature-attribution metadata to Postgres. Two weeks of data showed the cost distribution was extremely skewed: one feature (a long-form summarization endpoint hit by power users) accounted for 41% of total spend. Two other features were trivially cacheable but not cached.

Step two was model routing. We introduced a router that classified incoming requests by complexity proxy (input length, presence of structured outputs, expected output length) and routed cheap requests to gpt-4o-mini or Claude Haiku, expensive ones to the original gpt-4o. The router itself uses a small classifier we fine-tuned on 12,000 labeled requests. About 70% of traffic now goes to the cheaper models. Quality on the routed-down traffic was measured against held-out human-judged samples — no degradation detected.

Step three was prompt compression and template normalization. The summarization endpoint had grown a 1,400-token system prompt full of historical instructions. We reduced it to 480 tokens after measuring which sections actually changed outputs. We also normalized common phrases across templates to maximize KV-cache hits on Anthropic's prompt caching (~80% input cost reduction on cached portions).

Step four was semantic caching for the two trivially-cacheable endpoints. We deployed a Redis + sentence-transformers embedding layer that catches near-duplicate requests. Hit rate hovers around 34% in production. Each cache hit costs roughly $0.0001 vs. the original $0.025-$0.08 per call.

What we abandoned: aggressive fine-tuning of a smaller open-weight model to replace the routed-down workload entirely. The model trained successfully, the held-out evals looked competitive, but in shadow traffic against real production load we observed a 6% increase in escalations to the cheaper-but-still-prompted gpt-4o-mini path. Net cost savings disappeared after counting the GPU hosting cost on Modal. We kept the routing approach and dropped the custom model.

Total engagement: 7 weeks, two of our engineers part-time. Monthly savings: $28,500. The instrumentation layer we built is now permanent infrastructure on their side — every new LLM feature ships with cost attribution by default.

Fine-tuning has cultural cachet that exceeds its actual production utility. Teams ask for it because it sounds more substantive than prompt engineering, and vendors push it because the contracts are larger. We have completed 31 fine-tuning engagements since GPT-3.5 fine-tuning became viable in 2023. With honest hindsight: fewer than 10 of those genuinely needed to be fine-tuning engagements. The other 21 would have been better served by improved prompting, retrieval, or model routing.

Where fine-tuning genuinely won: structured output reliability for a high-traffic classification task where the cost of one mis-formatted JSON response was a downstream null pointer exception. A fine-tuned Mistral 7B at 99.4% format adherence beat gpt-4o at 97.1% — and ran at a fraction of the cost. The 2.3 percentage points mattered because the absolute traffic was 4M requests/day.

Where it surprisingly won: tone matching for a legal document drafting product. The client had 18,000 examples of their attorneys' final drafts and the corresponding initial drafts. Fine-tuning on draft-to-final pairs produced outputs that human reviewers preferred over base-model-with-detailed-prompt 73% of the time in blind comparison. This is the kind of style transfer prompting handles poorly.

Where it lost to prompting: any task where the input distribution is wide and the output expectations are flexible. Customer support response generation, document summarization, code explanation. In every case where we tested fine-tuned 7-13B models against prompted frontier models, the frontier model with a 800-1,500 token system prompt outperformed the fine-tuned model on held-out evaluations.

Where it lost to retrieval: any case where the goal is "the model should know about our internal documentation/policies/products." Fine-tuning was the wrong tool — the knowledge belongs in retrievable context, not baked into weights. Two engagements began as fine-tuning projects and pivoted to RAG mid-engagement once the data audit revealed the knowledge was changing weekly.

Our current rule of thumb: fine-tune when you can articulate the specific behavior the base model gets wrong, you can produce 1,000+ high-quality labeled examples, and you can describe what a successful eval looks like before training starts. If any of those three are missing, prompt engineering is faster, cheaper, and usually wins.

Over the past three years we have deployed production ML systems on top of every major MLOps platform our clients arrived with — sometimes because the platform was already in place, sometimes because we recommended it. This is a field-notes summary of what each one is actually like to operate against, with the perspective of an engineering team rather than a vendor demo.

Databricks. Strong for teams that already operate at Spark scale and have data engineering muscle. The MLflow integration is genuinely useful, Unity Catalog handles the governance most teams need, and the Mosaic AI training stack is competitive for fine-tuning. The friction points are notebook-driven culture (which fights production discipline) and per-DBU cost surprises. We have helped two clients consolidate Databricks usage by 35-40% by moving long-running pipelines off interactive clusters onto job clusters with appropriate auto-termination.

AWS SageMaker. The reasonable default for AWS-native organizations. Pipelines are solid, model registry works, autoscaling endpoints are reliable. The friction is volume of moving parts — SageMaker Studio, Pipelines, Inference, Endpoints, Feature Store, JumpStart, Canvas, Ground Truth, etc. — most teams use 10% of the surface and find the documentation maze exhausting. Cost is competitive but requires active management; idle endpoints are a common cost-leak we audit for routinely.

GCP Vertex AI. Often-overlooked alternative to SageMaker with cleaner UX. The Pipelines product is the strongest of the three hyperscalers in our opinion, and Gemini integration via Vertex is genuinely tight for teams already on Google models. Weaknesses: documentation gaps, slower release cadence than AWS, and the Workbench (notebooks) lags behind Databricks for collaborative work.

Self-hosted MLflow + Airflow + Postgres. The default for teams that prioritize cost and control over polish. We have built this stack for three clients in the last year — the operational burden is real (someone must own the cluster) but the cost savings vs. managed platforms is 60-80% at scale. Fits well in companies that already have strong DevOps and treat ML infrastructure as part of platform engineering.

Modal + W&B + Pinecone. The lean stack we recommend most for ML teams under 30 engineers building LLM-heavy products. Modal handles compute, W&B handles experiment tracking, Pinecone handles vector storage. Total platform cost for typical engagement is $4,000-$12,000/month at usage that would cost $40,000-$100,000 on Databricks or SageMaker.

Final note on platform choice: by month six of any engagement, the platform decision matters less than the team's operational discipline. We have seen teams ship excellent ML on subpar platforms, and seen teams produce broken systems on best-in-class platforms. Pick something defensible, then invest in the practices around it.