124 lines
4.6 KiB
Markdown
124 lines
4.6 KiB
Markdown
---
|
|
title: "Prefer Python over bash for multi-step pipeline scripts"
|
|
date: 2026-04-09
|
|
category: best-practices
|
|
module: "skill scripting / ce-demo-reel"
|
|
problem_type: tooling_decision
|
|
component: tooling
|
|
severity: medium
|
|
applies_when:
|
|
- Script orchestrates 2+ external CLI tools (ffmpeg, curl, silicon, vhs)
|
|
- Script needs retry logic or graceful degradation on tool failure
|
|
- Script will run on macOS where bash 3.2 is the default
|
|
- Script needs to be tested from a non-shell test runner (Bun, Jest, pytest)
|
|
- Script has conditional failure paths where some errors should be caught and others should abort
|
|
tags:
|
|
- bash-vs-python
|
|
- pipeline-scripts
|
|
- skill-scripting
|
|
- set-e-footguns
|
|
- error-handling
|
|
- ce-demo-reel
|
|
---
|
|
|
|
# Prefer Python over bash for multi-step pipeline scripts
|
|
|
|
## Context
|
|
|
|
When building the `ce-demo-reel` skill, the initial implementation used a bash script (`capture-evidence.sh`) to orchestrate ffmpeg stitching, frame normalization, and catbox.moe upload. Over 4 review rounds, the script hit 4 distinct bug classes that are inherent to bash's execution model rather than simple coding mistakes.
|
|
|
|
## Guidance
|
|
|
|
Use Python for agent pipeline scripts that chain multiple CLI tools with error handling. Bash `set -euo pipefail` works for simple sequential scripts but becomes a footgun when you need controlled failure paths.
|
|
|
|
**Python subprocess model (explicit error handling):**
|
|
```python
|
|
result = subprocess.run(
|
|
["curl", "-s", "-F", f"fileToUpload=@{file_path}", url],
|
|
capture_output=True, text=True, timeout=30, check=False
|
|
)
|
|
if result.returncode != 0:
|
|
# Retry logic runs normally
|
|
attempts += 1
|
|
continue
|
|
```
|
|
|
|
**Python timeout handling (explicit catch):**
|
|
```python
|
|
try:
|
|
result = subprocess.run(cmd, timeout=60)
|
|
except subprocess.TimeoutExpired:
|
|
# Controlled failure, not a crash
|
|
return subprocess.CompletedProcess(cmd, returncode=1, stdout="", stderr="Timed out")
|
|
```
|
|
|
|
**Bash equivalent (the footgun):**
|
|
```bash
|
|
set -euo pipefail
|
|
|
|
# Exits the entire script before retry logic runs
|
|
url=$(curl -s -F "fileToUpload=@${file}" "$endpoint")
|
|
# Never reaches here on curl failure
|
|
|
|
# Workaround: || true on every line that might fail
|
|
url=$(curl -s -F "fileToUpload=@${file}" "$endpoint") || true
|
|
# Works but fragile and easy to forget
|
|
```
|
|
|
|
## Why This Matters
|
|
|
|
Agent pipeline scripts run in environments the skill author does not control: different macOS versions (bash 3.2 vs 5.x), CI containers, worktrees. Each bash portability issue requires a non-obvious workaround that reviewers must catch. Python's subprocess model makes error handling explicit and testable rather than implicit and version-dependent.
|
|
|
|
The 4 bugs found were not unusual. They are the predictable consequence of using bash for scripts that exceed its sweet spot.
|
|
|
|
## When to Apply
|
|
|
|
Use Python when:
|
|
- The script orchestrates 2+ external CLI tools
|
|
- The script needs retry logic or graceful degradation on tool failure
|
|
- The script will run on macOS where bash 3.2 is the default
|
|
- The script needs to be tested from a non-shell test runner
|
|
- The script has more than ~3 subcommands
|
|
|
|
Bash is still the right choice when:
|
|
- Simple sequential scripts with no error recovery (set -e is fine)
|
|
- One-liner wrappers around a single tool
|
|
- Scripts using only POSIX features with no array manipulation
|
|
- Git hooks and CI steps where the only failure mode is "abort the pipeline"
|
|
|
|
## Examples
|
|
|
|
**Before (bash, 4 bugs across 4 review rounds):**
|
|
|
|
| Bug | Cause | Workaround needed |
|
|
|---|---|---|
|
|
| `url=$(curl ...)` exits on network failure | `set -e` + command substitution | `\|\| true` on every line |
|
|
| `${array[-1]}` fails | Bash 3.2 lacks negative indexing | `${array[${#array[@]}-1]}` |
|
|
| Frame reduction keeps all frames for n=3,4 | Integer math: `step=(n-1)/2` with min 1 | Minimum step of 2 |
|
|
| `command -v ffmpeg` in Bun tests | `command` is a shell builtin, not spawnable | Use `which` instead |
|
|
|
|
**After (Python, all 4 bug classes eliminated):**
|
|
|
|
```python
|
|
# Negative indexing just works
|
|
last = frames[-1]
|
|
|
|
# Timeout handling is explicit
|
|
try:
|
|
result = subprocess.run(cmd, timeout=30)
|
|
except subprocess.TimeoutExpired:
|
|
return None
|
|
|
|
# Tool detection is a regular function
|
|
if not shutil.which("ffmpeg"):
|
|
sys.exit("ffmpeg not found")
|
|
|
|
# Math is straightforward
|
|
step = max(2, (len(frames) - 1) // 2)
|
|
```
|
|
|
|
## Related
|
|
|
|
- `docs/solutions/skill-design/script-first-skill-architecture.md`: covers when to use scripts vs agent logic (complementary: that doc answers "should a script do this?", this doc answers "which language?")
|
|
- `docs/solutions/agent-friendly-cli-principles.md`: CLI design from the consumer side (overlaps on exit code and stderr patterns)
|