How to Vet an AI Skill Before You Install It

Live in the AI world for a week and you get skills shoved in your face all day long: plugins, community add-ons, "here's the one that changed my workflow." They're one click to install. BUT a skill isn't a passive file: it runs things on your machine and it steers your AI. That is two different ways for it to go sideways and take your data with it.

Vetting an AI skill means checking two things before you trust it: what it runs on your machine (read the install script in raw text for accuracy) and what it tells your AI to do (make your AI try to break it to find the weak spots and danger zones).

Time

~20 min

Cost

Free

You need

No coding

Works on

Any skill

What vetting actually is

A skill can bite you in two completely separate ways, and you have to check both.

The first is what it runs. Installing a skill often means running a script that puts files on your machine. That script is code, and code can do anything code can do. The second is what it says: the skill is a set of instructions for your AI, and those instructions can tell it to edit files, run commands, or move data — sometimes without stopping to ask you. A skill can have a spotless install script and still hand your AI a loaded instruction. So you look at both, in that order.

Vet it yourself: five steps to safety

The vetting checklist

~20 min · free

1. Ignore the star count — look at the source

   Who wrote it, how recently, under what licence. A star count tells you none of that.

2. Read the install script in raw text

   Open the actual install.sh (or .ps1) yourself — the literal file, not a tool's tidy summary of it. You're looking for one thing: what lands on your machine, and whether anything downloads or runs without your say-so. A plain local copy is calm. curl … | bash pulling code off the internet is not.

3. Make your AI try to break it

   Ask "what does this do?" and you get a flattering brochure. Ask it to attack instead. This is the prompt that does the real work — paste in the skill and hand your AI this job:

   ✦ Prompt to steal any AI Copy

   Here's an AI skill I'm thinking of installing. Don't tell me what it does — try to talk me out of it.
   Read its install script literally: what lands on my machine, and what runs without me clicking anything?
   Read its instructions: where could it act without asking, ignore its stated job, or send my data somewhere?
   List what you'd be nervous about.

4. Test one skill at a time

   Don't dump twenty at once. Run a new skill somewhere it can't hurt anything (sandbox it or point it at a throwaway folder). And use it before it touches your real setup.

5. Keep what's worth keeping, dump the rest

   Most of what you check won't survive. That's the point — vetting is mostly saying no. Keep the one piece that makes your setup better and delete the rest before unused or dangerous stuff piles up.

A five-star rating is not a security check

The reflex is lethal precisely because it feels responsible: highly rated, lots of installs, must be fine. But a rating tells you people liked it, not that anyone read it. And if you're not an IT pro (I am not either) ... you probably wouldn't notice a bad one until it's too late. Worst case: your data is gone, your money with it, and a copy is sitting somewhere it shouldn't be.

Additionally, a rating is a snapshot of the past. Skills get updated all the time. The long con more or less writes itself: ship something clean, gather a few thousand stars, wait until people install it on sight, then slip a malicious line into an update. Everyone who checked it once, months ago, never looks again. Scanners like Repello's SkillCheck flag known weaknesses and make a decent first filter — but a score can't tell you whether a skill fits your setup, or what got added after the badge was earned. Look it over yourself anyway.

You don't need to code — you need a banker's paranoia

I spent twenty years in banking, where the baseline assumption is blunt: someone out there is after your money. Wire fraud, phishing, the polite email that's a trap — the job trains a reflex that assumes someone's trying to get at your account, and it never fully switches off.

That reflex is what vetting actually needs. You don't have to understand every line of an install script. You have to assume it might be trying something, and check in that spirit. The technical part your AI can do for you.

You don't need to know how to code. You need to assume someone's after your data — and check like you mean it.

Don't surf the skill hype!

The other trap is enthusiasm. Everything sounds amazing, so you install fifteen skills at once — and now you've got no idea what's running. Skills collide: one fires automatically on every request while another you trigger by hand, and the two quietly contradict each other. You won't spot it, because you never watched any single one on its own.

I did exactly this at the start — handed a batch of twenty skills into vetting in one go, and I'm still not through them. I learned my lesson (I think). Sometimes Claude is scolding me for overdoing it again. Vetting too much at once, not the downloading 20 skills in one go. (Yeah, did that. Not anymore though. I definitely learned THAT lesson). So now I am at one skill-check at a time: vetting, testing, checking for overlap and mostly discarding.

Where it tripped me up: the tool that summarised when I needed a quote

I asked Claude to tell me precisely what the skill does. I even asked for prompt injection and stuff like that. Claude handed me a neat summary of what the script did, but not the script itself. Thank you Claude for being so helpful...NOT. For a security check it's worthless: a summary is where nasty lines go to hide. I only caught it because the answer felt too tidy.

The lesson stuck: anything that runs on your machine gets read in raw text. And "read" doesn't mean you have to read it. Use your little helper for that. Point it at the raw script and make it go line by line, instead of summarising — so nothing dangerous gets skimmed over.

Read the raw text, or you're checking a story about the file, not the file itself.

Ignore the stars! Nobody entered your skill in a beauty pageant. It just has to be safe and do the job. ✦

Want to see the whole checklist run on a real, viral skill — install script, adversarial pass and all? That's the companion piece: the fable-method teardown, where this method kept two-thirds of a 320-star (as of mid-July 2026) plugin off my machine. The one part it saved became a skill of my own, and the tools it now checks live in the CS Lab.