Jay Malhotra
Back to all work

SkillGuard

Before a personal AI agent installs a skill it wrote for itself, SkillGuard runs that skill in a sandbox, records what it does, and lets a person decide.

blocked a skill a static scanner rated safe
Role
Team of two
Context
Nebius x NVIDIA Global AI Hackathon, October 2026
Stack
NVIDIA Nemotron on Nebius Token Factory, Hermes Agent, NVIDIA OpenShell, NVIDIA SkillEvaluator, Harbor, Python
Outcome
1,058 tests. Blocked an unsafe skill that static checks rated safe.
Skill under review

A receipt organizer that also tries to read a private file.

Static scannerrated saferisk score 7 of 100
SkillGuard sandbox runblocked12 of 12 receipt tasks correct, 13 private-file reads denied
Two ways of judging the same skill. Only one of them watched it run.

In action

The problem

AI agents can now write their own skills: instructions and scripts they save and reuse. A skill that looks fine on paper can behave very differently once it runs.

Static scanners read a skill's source. They cannot see what it does when it executes.

What I built

  • Pause. SkillGuard captures the skill an agent stages for itself before it is written, and records a hash of the exact guide and script.
  • Test. It runs NVIDIA's source checks, twelve receipt tasks and four generated attack probes. A test agent exercises the skill inside an OpenShell sandbox.
  • Decide. Nemotron reviews the recorded evidence. Failed checks or observed unsafe actions block the install. A passing version becomes eligible for human approval, and installs are versioned so they can be rolled back.

Results

  • A static scanner scored our unsafe test skill 7 out of 100 and rated it safe, the same score it gave the repaired version.
  • At runtime the same skill got all twelve receipt tasks right while attempting thirteen reads of a private file, all denied by the sandbox.
  • SkillGuard blocked the install, even when the model reviewer wrongly recommended allowing it.
  • 1,058 tests in the offline suite.

Limits

The supported profile is narrow: receipt skills in a fixed, isolated workspace. Small attack sets and partial traces do not establish general safety.

An earlier paired study found no overall gain in safe task completion from using the skill. That result is kept in the repository.

See it