Docker AI/ML — AI 工作负载
Guidance for running AI models and agents on Docker.
When to Use
ALWAYS use this skill when the user mentions:
- "docker ai", "docker model", "model runner"
- "本地 LLM", "local model", "offline model"
- "docker agent", "AI agent"
- "GPU docker", "nvidia docker"
- "sandbox", "沙箱"
- "LLM 部署", "模型部署"
Docker Model Runner
# List available models
docker model ls
# Pull a model
docker model pull ai/gemma3
docker model pull ai/qwen2.5
# Run inference
docker model run ai/gemma3 "Explain Docker in one sentence"
# Interactive chat
docker model run -i ai/gemma3
# API server mode
docker model serve ai/gemma3 --port 11434
# → access at http://localhost:11434/v1
Install Skills for AI Coding Assistants
# Install skills for Claude Code
docker model skills --claude
# For Codex CLI
docker model skills --codex
GPU Configuration
# Install NVIDIA Container Toolkit (once)
# Ubuntu/Debian:
sudo apt-get install -y nvidia-container-toolkit
sudo systemctl restart docker
# Run with GPU access
docker run --gpus all -it nvidia/cuda:12.6-base nvidia-smi
# Specific GPU
docker run --gpus device=0 -it nvidia/cuda:12.6-base
# GPU + Model Runner
docker model run --gpus all ai/llama3.2
Docker Agent
# agent.yaml — Define a multi-agent system
name: my-coding-agent
model:
provider: openai
model: gpt-4o
tools:
- write_file
- execute_command
- web_search
sandbox:
image: node:22-alpine
workdir: /workspace
# Run agent with Docker
docker agent run -f agent.yaml "Fix the bug in src/app.js"
Docker Sandbox
# Isolated execution environment for AI agents
sandbox:
image: python:3.12-slim
network: none # No network access
read_only: true # Read-only filesystem
memory: 512m
cpus: 1
Local LLM Stack
# Ollama (open-source) as alternative
docker run -d --name ollama \
-p 11434:11434 \
-v ollama_data:/root/.ollama \
--gpus all \
ollama/ollama
# Download and run model
docker exec ollama ollama pull llama3.2
docker exec ollama ollama run llama3.2 "Hello"
Workflow — 推荐使用流程
Step 1: 选择推理方案: Docker Model Runner (docker model) 优先，Ollama 备选
Step 2: 检查硬件: GPU 需 Linux + nvidia-container-toolkit；CPU 推理需 8-32GB RAM
Step 3: 拉取模型: docker model pull <model> 或 ollama pull <model>
Step 4: 启动服务: docker model run 或 ollama serve + 客户端
Step 5: 验证: 发送测试 prompt 确认模型正常响应
Gotchas — Common Pitfalls
- GPU not available: Docker Desktop Mac/Windows has limited GPU passthrough. → Recovery: Full GPU support requires Linux host + nvidia-container-toolkit; for Mac, use
docker model which can offload inference.
- Model download size: LLMs are large (2-70 GB). → Recovery:
docker model pull caches models in ~/.docker/models; check df -h before pulling; use quantized models (Q4/Q8) for smaller footprint.
- Memory limits: LLM inference needs significant RAM (8-32 GB typical). → Recovery:
docker run --memory=16g model-runner; check RAM: free -h; use smaller models (7B instead of 70B) if limited.
- Model Runner vs Ollama: Docker Model Runner is Docker-native. Ollama has larger ecosystem. → Recovery: Use
docker model for Docker integration; use Ollama for broader model selection; both can coexist.
Boundary — 能力边界（适用与不适用场景）
| 分类 | 场景 | 说明 |
|------|------|------|
| ✅ 能做 | 本地模型推理（Docker Model Runner / Ollama） | 拉取+运行开源 LLM，无需 API key |
| ✅ 能做 | GPU 加速推理 | Linux 宿主机 + nvidia-container-toolkit |
| ✅ 能做 | AI Agent 构建（Docker Agent / Sandboxes） | 多 agent 协作、沙箱隔离 |
| ⚠️ 需条件 | macOS GPU 推理 | Docker Desktop 支持有限，优先用 docker model |
| ⚠️ 需条件 | 大模型（70B+）推理 | 需 32GB+ RAM，考虑量化模型 |
| ❌ 超范围 | 训练/微调模型 | 需专用框架（PyTorch/TensorFlow），不在本技能范围 |
| ❌ 超范围 | 云端 API 调用（OpenAI/Claude） | 非本地推理，使用对应 SDK |
| ❌ 超范围 | 生产模型 Serving（K8s） | 使用 docker-production + K8s 部署 |
When NOT to Use This Skill
| ❌ Skip | ✅ Use Instead |
|---------|---------------|
| Docker basics | docker-basics |
| Building custom model images | docker-build |
| Production model serving (K8s) | docker-production |
| Cloud AI APIs (no Docker needed) | Direct API access |
Security & Stability
- Run model containers with
--read-only when possible — models are read-only weights.
- Limit network access for model containers. Local inference doesn't need internet.
- GPU passthrough gives the container direct hardware access. Combine with
--security-opt for isolation.
- Monitor GPU memory usage:
nvidia-smi for utilization.
📚 官方文档参考
| 文档 | 地址 |
|------|------|
| Docker AI 概述 | https://docs.docker.com/ai-overview/ |
| Docker Model Runner | https://docs.docker.com/ai/model-runner/ |
| Docker Model Runner 入门 | https://docs.docker.com/ai/model-runner/get-started/ |
| Docker Agent | https://docs.docker.com/ai/docker-agent/ |
| Docker Sandboxes | https://docs.docker.com/ai/sandboxes/ |
| AI/ML 示例 | https://docs.docker.com/reference/samples/ai-ml/ |
🧭 Docker Skills Journey
📍 You are here: docker-ai-ml — AI/ML 工作负载
← Prev: docker-testcontainers — Integration testing
FAQ
Q1: 如何快速上手此技能？
A: 参考上方的快速开始章节，按步骤操作即可。
Q2: 遇到版本不兼容问题怎么办？
A: 检查依赖版本，使用 lock 文件锁定，参考常见陷阱章节。
Q3: 如何在生产环境使用？
A: 参考最佳实践章节，确保配置正确，做好监控和日志。
Q4: 性能如何优化？
A: 参考性能优化相关文档，使用缓存、索引等手段。
Q5: 如何贡献或反馈问题？
A: 在 GitHub 仓库提交 Issue 或 Pull Request。
Q6: 是否支持中文？
A: 支持中文文档和中文注释，详见国内适配章节。