What I learned building with AI coding agents this week
I build my own apps at sauerapple.com, including Tell Sophie (my newest) and bigbear.news, and I use coding agents for most of that work and for contract projects. Every week I read a lot about how other people work with agents. This is my first weekly writeup of what stuck with me and what I'm changing because of it.
TL;DR
- Agents now write cleaner code line by line, but they still rewrite library functions they could just call (Gabe Orlanski, LibraryUseBench: 73% to 100% of solutions).
- At incident.io, AI helped engineers who know the codebase much more than everyone else. Tenured engineers went from 11.1 to 19 PRs a week; the median went from 10.1 to 10.7.
- Basis treats its repo as context for agents, because an agent "onboards" every run. Cleaning that up drove a 5x jump in tokens per developer and 2.5x commit velocity in a quarter.
- Flo gets 40% of its subscriptions more than six months after install, which changed how I think about the Tell Sophie paywall.
Why do AI coding agents rewrite libraries instead of using them?
They tend to write their own version of something a library already does, even when that library is right there and they've read it. Gabe Orlanski measured this, and every model he tested did it in at least 73% of solutions.
In Slop Is Moving Up the Stack, Gabe follows up on SlopCodeBench and finds that line-level slop is going down. Newer models write fewer giant functions and use real exceptions instead of returning error tuples. The bad news is one level up. On his LibraryUseBench, models hand-wrote functionality the provided library already had in 73% to 100% of solutions, depending on the model. One example: a model was given BeautifulSoup, read its find_declared_encoding in dammit.py, and then wrote its own find_declared_encoding anyway.
A more prescriptive prompt helped, cutting reimplementation to about 40% on his LibraryDesignBench. So the problem can be reduced, but it doesn't go away on its own.
What I'm changing: when I review agent PRs, I'm going to look harder at any new helper function and ask whether a dependency already does it. I'm also going to name the libraries I want used in each repo's agent instructions, so the agent doesn't have to guess.
Do AI coding agents make engineers more productive?
They do, but mostly for people who already know the codebase. At incident.io, the median engineer barely moved while tenured engineers shipped about 70% more PRs a week.
Gergely Orosz's The Pulse: Details of incident.io's AI spend has the numbers. Compared with two years ago, the median engineer went from 10.1 to 10.7 PRs a week. Tenured engineers went from 11.1 to 19. Nearly 80% of PRs shipping to production now include agent-generated code.
The part I keep thinking about is how they onboard. New engineers hold off on AI for their first month and learn the codebase instead, because new joiners who leaned on AI heavily right away did worse over the medium and long term.
The takeaway for me is that the agent's speed depends on how well I know the code. If I know an app well, I can write a precise task and catch a wrong turn early. On a codebase I'm new to, I'm going to spend more time reading before I prompt.
How should you structure a codebase for AI coding agents?
Treat the repo as the agent's onboarding doc, because an agent starts from zero every run. Basis separates what's true today from plans and history, keeps instructions close to the code they apply to, and only auto-loads context that changes agent behavior.
Basis published How We Made Our Monorepo Ergonomic for Agents (also an X article) in May. Their framing: a company might onboard a few engineers a month, but an agent has to "onboard" to the codebase every single trajectory. Small contradictions in the repo get repeated across thousands of runs.
A few ideas I'm taking from it:
- Mark what's canon. AGENTS.md files, skills, docs and code comments describe the current system. Specs, tickets and historical notes are useful, but agents should read them as intent, not as truth.
- Put context where it's used. Basis has a root AGENTS.md of about 300 lines plus nested AGENTS.md files scoped to each directory.
- Default to leaving things out. In their words, tokens that earn no behavior are a tax on every session.
In three months, token usage per developer went up more than 5x and commit velocity went up 2.5x.
My site repo already has an AGENTS.md and a CLAUDE.md. My next step is to go through them with the default-no rule, cut anything an agent wouldn't need for most tasks, and do the same across my other apps.
When should a subscription app show its paywall?
It depends on how long your app stays useful. Flo gets 40% of its subscriptions more than six months after install, because the free app stays useful and the paid offer shows up when a user's needs change.
This one isn't about agents, but it's the thing I learned this week that most changed a product decision. In a RevenueCat Sub Club episode, Mike Charniatsou of Flo explains that core cycle tracking stays free, and paid value is offered when a user's life stage changes. He's clear that this is a luxury: a younger app that has to pay back acquisition from day one can't always wait six months.
His other point that stuck with me: "your paywall conversion is not your monetization strategy." Short-term conversion wins don't always turn into business wins.
For Tell Sophie, the better path is to stay useful long enough for parents to decide later, then re-offer the upgrade at natural moments instead of pushing hard on day one.
FAQ
Why do AI agents duplicate code that already exists in a library?
On Gabe Orlanski's LibraryUseBench, every model tested reimplemented functionality from the provided library in 73% to 100% of solutions. A more prescriptive prompt reduced this to about 40% in a related benchmark. Naming the libraries to use in your agent instructions and checking for new helper functions in review both help.
Does AI help junior or senior engineers more?
At incident.io, tenured engineers gained the most, going from 11.1 to 19 PRs a week, while the median engineer went from 10.1 to 10.7. New hires are asked to skip AI for their first month to learn the codebase.
What is an AGENTS.md file?
An AGENTS.md file is a plain-text instruction file that coding agents read when they work in a repo or folder. Basis uses a root file of about 300 lines plus nested files scoped to each directory.
How did Basis increase commit velocity with agents?
Basis cleaned up its repo's context for agents: it separated canon from plans and history, moved instructions next to the code they apply to, and kept auto-loaded context to a minimum. In a quarter, token usage per developer rose more than 5x and commit velocity rose 2.5x.
How many app subscriptions start long after install?
At Flo, 40% of subscriptions start more than six months after install. Flo keeps core tracking free and offers paid value when a user's needs change.
Sources
- Gabe Orlanski, Slop Is Moving Up the Stack
- Gergely Orosz, The Pragmatic Engineer, The Pulse: Details of incident.io's AI spend
- Basis, How We Made Our Monorepo Ergonomic for Agents (X article)
- RevenueCat Sub Club, Flo gets 40% of its subscriptions more than six months after install