We Hired Three AI Agents for a Weekend: Here's How They Did on Real GitHub Tickets

There's a weird experiment making the rounds in developer circles lately: instead of benchmarking AI agents on synthetic puzzles like LeetCode or SWE-bench, people are pointing them at real GitHub issues on real projects and watching what happens. No hand-holding, no curated test harness. Just a repo, a ticket, and an agent with something to prove.

We decided to try it ourselves. Over one chaotic weekend, we ran three popular autonomous coding agents against a handful of open issues on mid-sized open-source projects — the kind of work you'd hand a junior developer on day two. Think: fix a flaky test, patch a broken docstring, close a small bug in a CLI flag parser. Nothing glamorous. Exactly the kind of tasks that separate people who can code from people who can ship.

The Setup: Unpaid Interns With Infinite Patience

The rules were simple. Each agent got access to the repository, the issue text, and a compute budget. No human touched the code until a pull request appeared. We then graded the results the way any maintainer would: Does it run? Does it pass CI? Would I merge this, or would I close it with a sigh and the word 'wontfix'?

If you've ever onboarded a new hire, the experience was eerily familiar. Two of the three agents spent their opening moves doing exactly what a good junior dev does — reading the codebase, running the test suite, tracing where the bug actually lived. The third went straight to editing files like it was speedrunning a hackathon. Guess which one produced the PR we actually considered merging.

The Report Card

Agent A (the careful one): Cloned the repo, ran tests before writing a single line, and left a comment on the issue outlining its proposed approach. The resulting PR was clean, well-scoped, and included a test for the fix. It even updated the changelog unprompted. Verdict: merge with minor nitpicks. Honestly, better than some human PRs we've reviewed.

Agent B (the confident one): Produced a fix that worked — technically — but rewrote three unrelated files along the way and deleted a comment explaining a subtle race condition. Classic overachiever energy. The diff was a crime scene. Verdict: request changes, twice, then give up.

Agent C (the ghost): Opened the issue, posted a cheerful 'Working on this! 🚀' comment, and then... nothing. Twenty minutes of tool errors, a context window that apparently filled up reading the build config, and silence. Verdict: this is the coworker who updates their Jira ticket to 'In Progress' and logs off.

Across the full weekend, the agents closed roughly one in four assigned issues with a mergeable PR. For fully autonomous work on unfamiliar codebases, that's genuinely impressive. It's also nowhere near a hire.

What This Means for Actual Junior Developers

Here's the part that should make every engineering manager sit up. The tasks we gave these agents — small bug fixes, test coverage, doc cleanup — are precisely the on-ramp work that junior developers have always used to learn a codebase. If agents can do 25% of that work autonomously today, and the number is climbing every model release, the bottom rung of the career ladder is getting automated out from under the people standing on it.

But the weekend experiment also showed what agents can't do. Every successful PR came after the agent had done the unglamorous reading. Every failure came from skipping that step — charging in with a confident patch and no understanding of why the code looked the way it did. That's not a junior developer problem. That's a developer problem, and AI has just made it impossible to hide.

The teams that thrive in the next few years won't be the ones replacing juniors with agents. They'll be the ones whose juniors — and seniors — have learned the lesson the agents are teaching: read the codebase first, ship small, and never, ever delete the comment explaining the race condition.

The Takeaway

Running AI agents on real GitHub issues is the most honest interview process we've seen yet — for the agents, sure, but also for the tooling ecosystem around them. The scores are middling and the failures are instructive. And if your open-source project suddenly gets a burst of cheerful 'Working on this!' comments followed by abandoned branches, be gentle. Somebody's agent is out there, doing its best.

We'll be rerunning this experiment quarterly as the models improve. Based on the trajectory, the 3 a.m. interview might get a lot harder to fail — for humans and bots alike.