AI.LOGJosh Byberg
04

Is Automating This Making You Worse At It?

Abstract art of a wall of falling glowing glyphs shattering into fragments of light mid-air.

There's a worry that shows up after a Claude workflow has been running long enough to be trusted, and it doesn't come from the output being wrong. It comes from the opposite — the output keeps being fine, week after week, and somewhere in there the person who used to do the task by hand notices they haven't done it by hand in months. The question that follows isn't "is this working," it's "am I still capable of doing this myself, or has that capability quietly gone away while I wasn't using it."

That worry deserves a real answer, not a reassurance. It's a fair concern, it applies to some of what you've automated and not to the rest, and the difference is worth being precise about.

Two different things get called "losing a skill"

Automating a task removes the repetition that used to happen every time you did it yourself. What that repetition was building — or just maintaining — is not the same for every task, and that's the whole issue.

Some of what repetition builds is pure mechanical fluency: the ability to type something out fast, to reformat a table by hand, to remember where a field lives in a form you fill in every week. Losing fluency at that is not a loss. Nobody's business is worse off because the person running it can no longer hand-type an invoice line as quickly as before. That skill's only value was in doing the task, and the task is now done. There's nothing underneath it to protect.

Some of what repetition builds is judgment, and judgment is a different animal. Reading enough contracts that you develop a feel for which clause is unusual before you can articulate why. Reading enough customer complaints that you can tell, from the first paragraph, when a complaint about a late delivery is actually about something else entirely. That kind of judgment isn't a byproduct of doing the task — the repetition of doing the task, badly at first and then less badly, is the mechanism that built it. There was no shortcut to it, and there's no way to have it without having gone through enough real instances to develop it.

The mistake is treating both the same way — dismissing every version of this worry because some of it is mechanical fluency nobody needed, or treating every automated task as a skill under threat because some of it genuinely is. Neither blanket answer is honest. The task decides which category it's in, and different tasks in the same business land in different places.

The test: could you still catch it wrong?

Here's the question that actually sorts a given task into one bucket or the other, and it's a question you can ask about a specific task in your business right now, not a general anxiety about AI.

If the automated output were subtly wrong — not obviously broken, just off in a way a careless reader would miss — would you catch it? Not "would the system flag it," would you, reading it cold, notice that something is off.

If yes, your judgment on that task is intact. You're not doing the task by hand anymore, but the thing the repetition built is still there and still working. The automation removed the labor, not the capability.

If you genuinely couldn't tell anymore — if a plausible-looking wrong answer would sail past you exactly as easily as a right one — that's the signal, and it's worth taking seriously rather than shrugging off. It means the judgment has eroded to the point where you can't actually supervise the workflow, whatever your approval process says on paper. A gate that has a human check the output only does anything if the human looking at it can still tell good from bad. That post is about deciding whether an output needs a human check at all, based on what a wrong answer would cost and how easily it would surface. This is a different, earlier question: assuming you've put a person in that seat, are they actually still equipped to do the job the seat requires? A check that exists on paper but is performed by someone who can no longer distinguish right from wrong isn't a safeguard. It's a formality that happens to look like one.

Run this test honestly and most of the worry sorts itself. The task you'd catch a subtle error on in your sleep was never at risk. The task you're not sure you'd catch is the one to actually do something about.

Keeping the judgment alive on purpose

Once you've found a task in the second bucket — real judgment, and you're not confident you'd still catch a subtle miss — the fix isn't to stop automating it. It's to keep exercising the judgment deliberately, on a schedule, instead of letting the automation be the only thing touching that task.

Two ways to do that, and neither requires giving up the time you got back.

Do the task by hand on a sample, periodically, without looking at what the automated version produced first. Not as a check on that run — as practice, the way the repetition worked before automation existed, just at a lower dose. Once a month, pull a real input and work it the old way, then compare. This is slower than reading the automated output and nodding, and that's the point — the slowness is where the judgment gets exercised.

Review a full, unfiltered batch of the automated output occasionally, rather than only the flagged exceptions. Most workflows surface the cases that look uncertain and let the rest through quietly, which is efficient and also exactly how pattern recognition atrophies — you stop seeing the ordinary cases at all, and the feel for what "ordinary" looks like fades with them. Reading twenty routine outputs in a row, including the ones nothing flagged, is what keeps that feel current.

Both cost real time, which is the tradeoff nobody wants to hear about after banking the time savings. It's worth paying on exactly the tasks that failed the catch-it test above, and nowhere else — there's no reason to spend it maintaining a skill that was never worth maintaining.

Who should actually worry about this

Not everyone automating a judgment task is at equal risk, and the difference is how deep the judgment was before the automation started.

Someone automating a task they already have years of hands-on expertise in is the low-risk case. The judgment is durable — built over a long enough stretch that it doesn't evaporate over a few months of not exercising it daily, the same way a skilled reader doesn't forget how to read after a few weeks away from a book. Periodic use is usually enough to keep it current.

Someone automating a task in an area they're still learning is the case to actually worry about. Hand off the repetition before the judgment exists and you're not preserving a skill you have — you're skipping the process that would have built one you don't have yet, and the automation is now permanently standing in for a capability that was never there. That's a structural risk, not a discipline problem, and it's why a junior hire and a twenty-year veteran should not automate the same task the same way, even when the task looks identical from the outside.

If that's your situation, the honest fix isn't to avoid automating the task — it's to do it by hand for real, enough times to build the judgment, before handing off the repetition that would have built it. Automating first and hoping to develop the feel for it later, on the side, rarely happens; the time pressure that made automation appealing doesn't go away once the workflow is running.

The concern this isn't

This isn't an argument for doing less. Most of what gets automated is closer to the mechanical end than people assume once they actually run the catch-it test against it, and worrying about losing fluency at typing something out by hand is worry spent on nothing. The point is narrower than "be careful with AI" — it's that one specific kind of task, where repetition was the only way judgment ever got built, deserves attention most workflow advice skips entirely. Ask the question honestly per task, keep exercising the ones that need it, and let the rest go.

Josh Byberg

NEXT.RUN

Have an idea worth automating?

I build the same kind of systems this blog writes about — multi-model pipelines, automated data entry, and tools that fix their own mistakes. Tell me what's eating your team's time and I'll tell you if AI can take it off your plate.

Bring Your Idea to Life →