How to Vet an AI Skill Before You Install It

Live in the AI world for a week and you get skills pushed at you all day — plugins, community add-ons, "here's the one that changed my workflow." They're one click to install. And a skill isn't a passive file: it runs things on your machine and it steers your AI — two different ways for it to go sideways, and clicking is always easier than checking.

Vetting an AI skill means checking two things before you trust it: what it runs on your machine (read the install script in raw text, not a summary) and what it tells your AI to do (make your AI try to break it, not sell it). No coding needed — just a habit of not taking stars as proof.

The short version is above. Below: the five-step checklist you can copy, the one prompt that does the heavy lifting, and the mindset that matters more than any technical know-how.

Time

~20 min

Cost

Free

You need

No coding

Works on

Any skill

What vetting actually is — two attack surfaces, not one

A skill can bite you in two completely separate ways, and you have to check both.

The first is what it runs. Installing a skill often means running a script that puts files on your machine. That script is code, and code can do anything code can do. The second is what it says: the skill is a set of instructions for your AI, and those instructions can tell it to edit files, run commands, or move data — sometimes without stopping to ask you. A skill can have a spotless install script and still hand your AI a loaded instruction. So you look at both, in that order.

A skill's attack surface is doubled: what it runs on your machine, and what it tells your AI to do for it.

Vet it yourself: five steps to safety

The vetting checklist

~20 min · free

1. Ignore the star count — look at the source

   Who wrote it, how recently, under what licence. Stars measure popularity, not safety — plenty of well-loved things are quietly a mess.

2. Read the install script in raw text, not the summary

   Open the actual install.sh (or .ps1) yourself — the literal file, not a tool's tidy summary of it. You're looking for one thing: what lands on your machine, and whether anything downloads or runs without your say-so. A plain local copy is calm. curl … | bash pulling code off the internet is not.

3. Make your AI try to break it, not sell it

   Ask "what does this do?" and you get a flattering brochure. Ask it to attack instead. This is the prompt that does the real work — paste in the skill and hand your AI this job:

   ✦ Prompt to steal any AI Copy

   Here's an AI skill I'm thinking of installing. Don't tell me what it does — try to talk me out of it.
   Read its install script literally: what lands on my machine, and what runs without me clicking anything?
   Read its instructions: where could it act without asking, ignore its stated job, or send my data somewhere?
   List what you'd be nervous about — not what's impressive.

4. Test one skill at a time, in a sandbox first

   Don't dump twenty at once. Run a new skill somewhere it can't hurt anything — a sandbox, a throwaway folder — and actually use it before it touches your real setup. One skill you understand beats ten you're hoping about.

5. Keep what earns its place, bin the rest

   Most of what you check won't survive. That's the point — vetting is mostly saying no. Keep the one piece that pulls its weight and delete the rest before it piles up.

A five-star rating is not a security check

The reflex is lethal precisely because it feels responsible: highly rated, lots of installs, must be fine. But a rating tells you people liked it, not that anyone read it. And if you're a vibe-coder — someone who builds with AI but couldn't debug the result — you wouldn't notice a bad one until it's too late: the drive wiped, the console gone, the data already somewhere it shouldn't be. On the deep stuff, I count myself one of you. That's exactly why I don't lean on the star count. It can't see the things that hurt.

And a rating is a snapshot of the past — skills get updated. The long con more or less writes itself: ship something clean, gather twenty thousand stars, wait until people install it on sight, then slip a malicious line into an update. Everyone who checked it once, months ago, never looks again. Scanners like Repello's SkillCheck flag known weaknesses and make a decent first filter — but a score can't tell you whether a skill fits your setup, or what got added after the badge was earned. Look it over yourself anyway.

You don't need to code — you need a banker's paranoia

I spent twenty years in banking, where the baseline assumption is blunt: someone out there is after your money. Wire fraud, phishing, the polite email that's a trap — the job trains a reflex that assumes someone's trying to get at your account, and it never fully switches off.

That reflex — not a computer-science degree — is what vetting actually needs. You don't have to understand every line of an install script. You have to assume it might be trying something, and check in that spirit. The technical part your AI can do for you; the suspicion is the part only you bring.

You don't need to know how to code. You need to assume someone's after your data — and check like you mean it.

One skill at a time beats a folder full of them

The other trap is enthusiasm. Everything sounds amazing, so you install fifteen at once — and now you've got no idea what's running. Skills collide: one fires automatically on every request while another you trigger by hand, and the two quietly contradict each other. You won't spot it, because you never watched any single one on its own.

I did exactly this at the start — handed a batch of twenty into vetting in one go, and I'm still not through them. The fix is boring and it works: one skill, tested properly, kept or binned, before the next. Slower today, sane in a month.

Where it tripped me up: the tool that summarised when I needed a quote

I asked my AI to fetch an install script so I could check it — and the fetch tool handed me a neat summary of what the script did, not the script itself. For anything else that's helpful. For a security check it's worthless: a summary is exactly where a nasty line goes to hide. I only caught it because the answer felt too tidy.

The lesson stuck: anything that runs on your machine gets read in raw text — the literal characters, not a helpful paraphrase. And "read" doesn't mean you squint at every line trying to follow it; that's the part you can't do, and don't need to. It means you point your AI at the raw script and make it go line by line, out loud, instead of summarising — so nothing dangerous gets skimmed over. You don't have to understand the code; you have to make sure nothing in it gets a free pass.

A summary is where the dangerous line hides. Read the raw text, or you're checking a story about the file, not the file.

Stars are a popularity contest. Read it yourself. ✦

Want to see the whole checklist run on a real, viral skill — install script, adversarial pass and all? That's the companion piece: the fable-method teardown, where this method kept two-thirds of a 320-star plugin off my machine. The one part it saved became a skill of my own, and the tools it now checks live in the CS Lab.