Skill Creator

A skill for creating new skills and iteratively improving them.

The Process

1. Decide what you want the skill to do and roughly how it should do it
2. Write a draft of the skill
3. Create a few test prompts and run claude-with-access-to-the-skill on them
4. Help the user evaluate the results both qualitatively and quantitatively
5. Rewrite the skill based on feedback
6. Repeat until satisfied
7. Expand the test set and try again at larger scale

Creating a Skill

Capture Intent

Start by understanding the user's intent:

1. What should this skill enable Claude to do?
2. When should this skill trigger?
3. What's the expected output format?
4. Should we set up test cases?

Write the SKILL.md

Anatomy of a Skill

skill-name/
├── SKILL.md (required)
│   ├── YAML frontmatter (name, description required)
│   └── Markdown instructions
└── Bundled Resources (optional)
    ├── scripts/    - Executable code
    ├── references/ - Docs loaded into context
    └── assets/     - Files used in output

Progressive Disclosure

Skills use a three-level loading system:

1. Metadata (name + description) - Always in context (~100 words)
2. SKILL.md body - In context whenever skill triggers (<500 lines ideal)
3. Bundled resources - As needed (unlimited)

Test Cases

After writing the skill draft, come up with 2-3 realistic test prompts. Save test cases to evals/evals.json.

{
  "skill_name": "example-skill",
  "evals": [
    {
      "id": 1,
      "prompt": "User's task prompt",
      "expected_output": "Description of expected result",
      "files": []
    }
  ]
}

Running and Evaluating Test Cases

Step 1: Spawn all runs (with-skill AND baseline)

For each test case, spawn two subagents:

• One with the skill
• One without (baseline)

Step 2: While runs are in progress, draft assertions

Draft quantitative assertions for each test case.

Step 3: As runs complete, capture timing data

{
  "total_tokens": 84852,
  "duration_ms": 23332,
  "total_duration_seconds": 23.3
}

Step 4: Grade, aggregate, and launch the viewer

1. Grade each run
2. Aggregate into benchmark
3. Do an analyst pass
4. Launch the viewer

Step 5: Read the feedback

Focus improvements on test cases with specific complaints.

Improving the Skill

1. Generalize from the feedback
2. Keep the prompt lean
3. Explain the why
4. Look for repeated work across test cases

Description Optimization

The description field determines whether Claude invokes a skill. After creating or improving a skill, optimize the description for better triggering accuracy.

Step 1: Generate trigger eval queries

Create 20 eval queries (mix of should-trigger and should-not-trigger).

Step 2: Review with user

Step 3: Run the optimization loop

python -m scripts.run_loop \
  --eval-set <path-to-trigger-eval.json> \
  --skill-path <path-to-skill> \
  --model <model-id> \
  --max-iterations 5

Step 4: Apply the result

Take best_description from the JSON output and update the skill's SKILL.md frontmatter.