Throwing Tokens

Over the last few years I’ve had countless deterministic vs non-deterministic debates when building new features. There's no clear answer but my mental model is framed around product lifecycle. When a product is early and evolving quickly, I’d favor AI for speed and flexibility. Once features stabilize and customers expect reliability I’d look for opportunities to harden them with deterministic code.

There are two legitimate schools of thought in AI engineering:
1. AI-first - refactor to deterministic logic where AI ends up being unreliable
2. Deterministic-first - layer AI where its required

Both are valid. The mistake is treating either as universally correct. My opinion is that there’s a time and a place for both but it’s a moving target and choosing intentionally is key.

Many features are obvious beneficiaries of AI, they effectively wouldn't be possible without AI or would be too much effort, let’s ignore these. The more interesting situation is when there's an obvious way to implement it with classic deterministic logic but you could save time and effort by outsourcing to AI and replacing certainty with speed. In early-stage products, this often pushes teams toward an AI-first approach.

Just like throwing compute at inefficient code instead of fixing it, throwing tokens at a problem is a shortcut: you pay in runtime cost to save development time.

AI First

“We’re betting that the models will get smarter and cheaper over time”

This saying is repeated ad nauseam in the AI space but I don’t disagree with the premise. When building new products the bet is that AI first features will only improve and get more cost effective over time so it's smart to bias towards AI.

Eventually the AI starts to have weak performance on certain tasks and those tasks get pulled out to deterministic logic, usually in the form of tools.

An obvious benefit is development speed, if you can simply write a prompt with your logic and some generalized messy edge case reasoning instructions, it goes a long way to avoid writing complex code. The feature mostly works and sometimes that’s good enough.

The downside is cost and imperfection. The hardest part isn’t even always that it’s wrong but that the behavior isn’t reproducible. Failures are often silent, wrong tools are used, outputs drift over time, and debugging becomes guesswork. You’re throwing tokens at the problem and can’t guarantee it will work as designed in all situations.

Deterministic First

Jake Heller (co-founder Casetext) frames this well when distinguishing between prompts, workflows, and traditional software engineering.

That mindset encourages developers to be more intentional about using AI because it comes with hidden costs. Bias towards classic deterministic code and use AI as an escape hatch when you need it. Again, when I first heard this framing it clicked for me immediately but it has its own trade offs too.

The benefits are certainty, execution speed, and low costs. This approach has you think more critically about what AI is good at and what features are worth investing in. Raising the point that we all eventually learn from experience, that prompt based logic seems like an accelerator on the surface level but it has hidden costs. It’s slower, more difficult to test, debug, maintain, and monitor. Oh and it’s far more expensive to throw tokens at the problem.

When and Where

I’ve been working at startups for years and I’ve learned that development speed and product flexibility are more important than stability and reliability. Don’t get me wrong, as startups mature they must rebalance towards stability but in the beginning you don't know if even a single line of code will be around in a few months so taking as many short cuts as possible is a good bet.

Early-stage software is usually hacky by nature. If that's already the case, then keeping the same characteristics but allowing the opportunity for development time savings is an obvious choice. In that phase, AI first is a logical strategy but good teams know how to recognize when the balance flips and are willing to pay the cost to harden what matters.