Grade an Agent Tool Before You Install It
Four checks that run before anything touches your machine: enumerate what you already have, subtract it from what the tool ships, query the issue tracker its users write, and confirm the uninstall works. Companion workshop to One Component I Didn't Already Have.
Nino Chavez
Product Architect
Prerequisites
- A specific agent tool, plugin, or extension you are considering installing
- The GitHub CLI (`gh`) installed and authenticated
- An enumerated list of your own setup — if you don't have one, run Audit Your AI Coding Harness in Four Exercises first
- Read the companion post: One Component I Didn't Already Have
What you'll build
- → Enumerate your current setup so an addition can actually be measured
- → Subtract that baseline from the tool's component list into three buckets
- → Query a project's issue tracker for the failure modes its documentation omits
- → Confirm the uninstall works before counting reversibility as a reason to try
Companion Post: One Component I Didn't Already Have
A Feature List Is Half a Subtraction
In One Component I Didn’t Already Have, a plugin advertising eleven agents and fifty-four hooks came out, measured against an existing setup, at a delta of one component.
Nothing in that result was about the plugin’s quality. It was about the other half of the subtraction — and that half lives on your machine, not on the project’s page.
This workshop runs the four checks against whatever tool is currently in your inbox. None of them requires installing it.
A few terms, defined once:
- Baseline — everything already wrapped around your model: instructions, skills, commands, hooks, memory files, tool connections.
- Delta — what the tool would add after subtracting your baseline. The only number that answers “should I install this.”
- Issue tracker — the part of a project’s repository the maintainer cannot curate without the gap being visible.
- Reversibility — whether the uninstall actually removes what the install added. Frequently assumed, rarely checked.
Enumerate your baseline
Why this matters
You cannot measure an addition without knowing the starting number. This is the check people skip, and skipping it is what makes every feature list look impressive — a list of fifty-four things is fifty-four gains only against zero.
It is also the check that pays off permanently. You do it once, and every future tool evaluation becomes a reading exercise.
The structure
Whatever your agent loads at startup, get it into one readable list: instruction files, skills, commands, hooks, tool connections, and any helper binaries on your path.
ls ~/.claude/skills/ ~/.claude/commands/ ~/.claude/hooks/ 2>/dev/null
ls -1 ~/.local/binIf you followed Audit Your AI Coding Harness, its report is your baseline and you can skip ahead.
The durable version of this is a repository rather than a command. Mine is public, which is the only reason the evaluation in the companion post took twenty minutes instead of an afternoon.
Your turn
Produce your list. Then answer one question about each entry: what does this do, in a sentence, without looking it up?
You should have a list where you can name the function of every item. Anything you can’t explain is a finding on its own — it’s loaded into every session and you don’t know why.
Subtract the baseline from the tool
Why this matters
The delta is the decision. Everything else in this workshop refines it.
The structure
Pull the tool’s component list from its reference documentation rather than its README — the first enumerates, the second sells. Many projects publish something like docs/reference/features.md:
curl -fsSL https://raw.githubusercontent.com/<owner>/<repo>/<branch>/docs/reference/features.mdThen walk it and mark each item one of three ways:
| Bucket | Meaning | What it does to the decision |
|---|---|---|
| Native | Your setup or your agent already does this | Removes it from the gain column entirely |
| New | Genuinely absent from your baseline | This is the delta. The only column that argues for installing |
| Blocked | New, but unusable for a reason unrelated to quality | Wrong platform, broken upstream, needs a subscription you don’t hold, requires patching something you don’t control |
Blocked is the bucket people collapse into the other two, and it distorts the answer in both directions. A component that is excellent and unusable is not a gain, and it is also not a criticism of the tool.
Then run it backwards: what does your baseline do that the tool’s list never mentions? That column is real and nobody publishes it for you.
Your turn
Produce the three-way split, then the reverse column. Count the New bucket.
You should be able to name your covering item for every Native row. “I think I have something like that” doesn’t count — if you can’t name it, it belongs in New.
If New is large, the tool is a genuine addition and the rest of this workshop is about risk. If New is one or two items, ask whether those items are available on their own before you adopt everything attached to them.
Query the issue tracker
Why this matters
A project’s documentation describes what it does when it works. Its issue tracker describes what it does on other people’s machines. The second is written by users, and unlike testimonials it can’t be curated without the absence being obvious.
This is also the fastest read of a project’s health that exists, and it’s entirely mechanical.
The structure
Three queries. Volume, recency, and your own failure modes.
# how much is open, and how fast is it arriving
gh api "search/issues?q=repo:<owner>/<repo>+is:issue+is:open&per_page=1" --jq '.total_count'
gh api "search/issues?q=repo:<owner>/<repo>+is:issue+created:>=<30-days-ago>&per_page=1" --jq '.total_count'
# what users argue about most
gh api "search/issues?q=repo:<owner>/<repo>+is:issue&sort=comments&order=desc&per_page=25" \
--jq '.items[] | "\(.comments)c \(.state) #\(.number) \(.title)"'Then search the tracker for the things that would specifically hurt you — your platform, your host tool’s version, your cost sensitivity, your workflow.
Read for patterns rather than counts. A high open count on a popular project mostly means it’s popular. What matters is shape:
- Does it break when its host releases a new version? Search the host’s version numbers.
- Do its own updates break things? Search
auto-updateandafter updating. - Are the most-discussed threads features or the same bug recurring?
Your turn
Run the three queries, then two searches specific to your situation. Write down the pattern in one sentence.
You should have one sentence of the form: “The recurring failure mode is ____, and it would/wouldn’t hit me because ____.”
If your searches return nothing, say that plainly rather than treating silence as a clean bill of health. A young project has a quiet tracker for reasons that have nothing to do with quality.
Confirm the exit before the entrance
Why this matters
“It’s cheap to try” is a claim about uninstalling, not installing. Every installer is one command. The exit is the part nobody tests, and it’s the part that decides whether trying costs you an afternoon or a weekend.
The structure
Read the install guide’s own account of what lands on disk, and check each location against your machine before running anything:
ls -1 ~/.local/bin # names it might shadow
grep -nE '^\[' <the-config-file-it-edits> # sections it might rewritePrefixed names and new named sections are additive and reversible. Bare generic names and rewrites of sections you already use are not.
Then do the thing almost nobody does: search the issue tracker for the uninstall itself.
gh api "search/issues?q=repo:<owner>/<repo>+is:issue+uninstall+OR+cleanup+OR+remove" \
--jq '.items[] | "\(.state) \(.created_at[0:10]) #\(.number) \(.title)"'Open reports that removal leaves things behind are worth more than the entire installation guide. They tell you the true cost of being wrong.
Your turn
Write a one-line verdict for each: name collisions, config writes, and whether uninstall is reported to work.
You should be able to say, without having installed anything: which files it creates, which existing file it modifies, whether any name shadows one of yours, and whether other people have successfully removed it.
If the install guide won’t tell you enough to answer those, that absence is your finding. Installers that won’t say what they write are the ones most worth not running.
What You Ran
Four checks, none of which required installing the thing:
- A baseline you can read, which turns every future evaluation into a lookup
- A three-bucket subtraction — native, new, blocked — run in both directions
- Three mechanical queries against the writing the vendor doesn’t control
- An exit check, before the entrance
The output is a number: how many components this tool would actually add to your setup. That number is the decision, and it is different for every person who asks you whether the tool is good.
Where This Is Heading
Exercise 1 is the one that hurts, and not because it’s difficult.
Most people cannot list what their agent loads at startup. The list has grown by accretion — a hook added during one bad afternoon, a rule added after one bad review, three skills installed and never invoked since. Every one of them still loads.
Which means the honest version of “should I install this” usually surfaces a second question underneath it, and it’s the less comfortable one: what is already in here that I would not install today?