Field Notes

It keeps coming up that LLMs can’t be trusted.

Sometimes they solve a problem (typically a coding problem) cleanly, other times they spin and spin trying to get it right. You can ship something 100x faster, or ship nothing after spending hundreds of dollars on tokens.

This is the main challenge we’ve been tackling since the ground suddenly shifted under software engineering, at the beginning of this year. On one hand, the tools are so powerful when they work that they change the game. On the other hand, when they don’t work, it’s incredibly wasteful and frustrating. And – at first – it can be really hard to predict when they will work.

With time and effort, we’re starting to understand it. And it has convinced me that engineers aren’t going to be replaced by LLMs; instead, using the LLMs requires its own kind of engineering.

LLMs charged with technical tasks appear to have a lot in common with human technical experts in many fields. Specifically, they thrive when they are given exactly the information they need to complete their tasks, and nothing extraneous to that.

In Big Tech companies, there were (I think not so much anymore after the recent purges) whole groups of knowledge engineers who specialized in this: making it so software engineers would get exactly the information they needed to work, without having to sift through tons of peripheral documentation to get to the relevant items. It was so important for people that these companies invested a lot of money in it.

Based on my own experiences, I’ve found it helpful to remember that I’ve already learned this from working with line cooks. They are the core of the restaurant, and technical experts who are usually geniuses of one kind or another. And if I got wrapped up in the details of the larger operation and then started giving them too much context, started calling attention to what was happening in service, or whatnot, I wasn’t helping them, I was distracting them and making it hard to focus on the task at hand. But when they received the exact specifications of what their output should be, and nothing more to worry about, that helped them thrive.

So, in terms of “the way we work now”, it seems that all of our new processes need to be grounded in this effort. To make sure every task comes with exactly the information needed to complete it, narrowly scoped, and nothing more. This was already needed in key places, and as the LLMs extend into more and more operations, it becomes a core effort.

When I first started working in Big Tech, I was surprised at how much of their development tooling was house-made. Version control wasn’t Github, internal messaging wasn’t Slack — except at Salesforce, which notably bought Slack the company — and project tracking wasn’t Jira. Instead the big companies used custom tools they made themselves.

At first I thought this was just because the big guys have so many resources (they do) and so much hubris (and who wouldn’t) that they preferred their own solutions as better than packaged ones. Although using these tools they didn’t necessarily feel like better products than the packaged ones, and they certainly weren’t as polished.

Over time, though, I started to see how the custom tools did exactly what the companies wanted done, in the exact way they wanted to do it. And, at scale, that made a huge difference. More than anything, this allowed the companies to create their own processes and culture, and know that the tools would serve that. Reinforce it, even. And given how unwieldy processes and culture will get across tens of thousands of people, this was worth the investment.

At smaller companies, of course, custom tooling was a luxury that rarely justified the ROI. The tradeoff was that, in return, the company subtly loses a little control of its contours. You use Slack for collaboration and you become a Slack-shaped org. You use Jira for tracking and, sure enough, over time you become a Jira-shaped org. Ultimately everybody thinks in terms that work well for Github, Jira, Slack, whatever the tools are. And it’s fine, it’s better than having your engineers tied up writing and maintaining code for a custom version control system.

Until coding is automated and cheap.

Suddenly the agentic engineering org has made two big shifts: 1) it is producing way more output, so the effects of tools on its culture is more pronounced. And 2) the cost of building custom tooling is way lower.

I believe that this will result in many more small-to-medium size orgs building a lot more tools. For instance, Slack is, at first glance, a convenient place to wire up your agents and collaborate with them. But Slack is probably not the best surface at every engineering org for humans to operate agents, and if your org would do better with something else, it’s a lot easier than it used to be to build and maintain what you want. In fact, it’s probably easier to build and maintain your own system than it is to constantly try to shoehorn what you need through Slack - particularly when engineers are producing many multiples of PRs, reviews, and discussion more than they used to.

I don’t think this shift will be monolithic or universal. Packaged tooling makes sense when a need is both very generalized across an industry, and also very complex. Slack proved to be able to address the specifics of team collaboration across many companies, in a deep and reliable way that made them almost impossible to beat in terms of functionality for many companies.

Similarly, it’s easy to imagine that certain needs of the agentic engineering org — perhaps the knowledge base or the memory/context system — will prove well-suited to complex yet general solutions.

That said, in the meantime I think you should just plan to build a lot of what you want. It’ll be cheaper, and most importantly, more conducive to creating the company you want and the culture you believe in.

The ramifications of scaling up coding via agents are pretty interesting.

I do think individual coders can attain massive increases in output of greenfield product work agentically. A single dev can manage, let’s say, 5-10 agentic crews working 24 hours a day in separate areas of a codebase. This makes a single engineer into a leader of ~25-50 coders, with output increases to match.

Any experienced dev reading the previous paragraph will point out that this creates an orchestration nightmare – a single person coordinating the work of 25-50 devs who work at different times and can’t possibly know what all the others are doing, is a recipe for disaster.

(It’s true.)

And then imagine an org of, say, 25 engineers, each of whom starts working agentically. At 25 engineers, you’ve got three teams, each team’s lead can coordinate with the others, that’s not a big deal. At 25 engineers working 50 coders each, that’s 1,250 coders to coordinate, which is a completely different type of problem.

The fun thing is, this problem has known solutions – because there are plenty of engineering organizations with a thousand devs. There’s even at least a handful of companies with ~10,000 devs. I’ve worked at 3 of them, and it’s interesting seeing the processes there suddenly become relevant for teams that number in the teens.

Big tech companies have whole groups dedicated to the problem of: how do we get engineers the documentation and information they need, narrowly scoped to exactly what they’re working on, at the time they need it, with minimal effort required by the engineer.

Of course you will recognize this as the exact challenge agentic systems face: help LLMs to work well without drowning them in cognitive overload, by surfacing only the context/tools/skills they need, at the moment they are materialized.

And there’s all these systems to do this. Mostly it’s a paper trail that looks like bureaucracy run amok: architecture documents, decision records, runbooks, everything versioned and logged and kept up to date. And these are the tools preferred by the LLMs when you start building software with them, which makes sense since they of course trained their systems from human ones.

What I’ve found from playing around with these, is that it works really well. It’s so important in agentic development to keep the robots on track, and nothing does a better job than a mountain of documentation. Of course, the robots also write the docs and maintain the docs (which, to be honest, they do a much better job at than people do). They don’t mind that job and I think it’s important to make sure they do it. Just as with people, when the project has hundreds of thousands or millions of lines of code, it’s sure nice to have all the old decisions and obscure architecture documented, when a significant change is needed or some unexpected outcome happens.

I mentioned in an earlier post that, as a developer working in their spare time to make production-grade software, agentic engineering increases my effectiveness around 50x over hand-coding.

Mature engineering organizations aren’t seeing those types of gains for a multitude of reasons which are mostly easy to suss out. I think the current rule of thumb is that mature organizations are seeing gains on the order of 25% as a first step if they can effectively introduce agentic development. Which seems like it might not be worth it, when you’re also, to some extent, giving up control of the your code to a robot.

Meanwhile, all sorts of new concerns arise.

LLMs are, in many instances, unreliable. Give them too much context, too much information, too many tools, and they bog down with cognitive overload. Give them too little, and they hallucinate the information they need. The technical requirements of a system that addresses this conundrum effectively can be extremely complex.

They’re also expensive, which the Claude subscription model tends to hide (these subscriptions are super subsidized). With a good tracking system like what we use in our dashboards, you can see what the API-equivalent costs are for subscription calls and it can be eye-opening – a $200/month subscription can easily absorb $10,000 worth of API calls in that month.

I like building things, and I’ve always taken pride in architecting elegant code. There’s something extremely satisfying about getting an object just right, where it maps perfectly to the problem domain, where it’s clearly reusable and maintainable and will serve the system quietly and beautifully. With agentic engineering, honestly, that all goes away.

Instead, the problem space shifts, from engineering software to engineering a software factory. How do we build this system – even a system for a single developer on a single computer – that runs more efficently, producing higer quality outputs at a lower cost? The good news is that it is real engineering, with a lot of levers that we can bring our expetise to, even though its a totally new problem space from where we recently spent our focus.

A colleague recently told me that they find it hard to learn about the state of agentic engineering through talking with friends in the industry, because many are reluctant to open such a converstaion for fear of being seen as an AI enthusiast or more extremely, a self-styled techno-futurist edgelord.

It’s a reasonable concern.

Personally, I’m pretty convinced that Generative AI as it exists at this moment is destructive to our ecology, harmful to our social fabric, and totally unnecessary for us as people to thrive. Of course I also think the same is true about cars. And for many folks, it is essential they know how to drive and have a basic understanding of how cars work.

My sense is that this is the state of software engineering right now, and will be for a while into the future. I think these tools are landing in our industry with some weight, and the time is coming when we will need to be skilled with them in order to do what we do. Acknowledging that doesn’t require losing sight of the real costs of this technology. If anything, an assessment of those costs has more credibility when it comes from someone who demonstrably understands the tools and how to use them.