Introduction
# CI/CD Pipeline (GitHub Actions)
Set up and manage CI/CD pipelines using GitHub Actions. Covers workflow creation, testing, deployment, release automation, and debugging.
## When to Use
- Setting up automated testing on push/PR - Creating deployment pipelines (staging, production) - Automating releases with changelogs and tags - Debugging failing CI workflows - Setting up matrix builds for cross-platform testing - Managing secrets and environment variables in CI - Optimizing CI with caching and parallelism
## Quick Start: Add CI to a Project
### Node.js project
```yaml # .github/workflows/ci.yml name: CI
on: push: branches: [main] pull_request: branches: [main]
jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: 20 cache: npm - run: npm ci - run: npm test - run: npm run lint ```
### Python project
```yaml # .github/workflows/ci.yml name: CI
on: push: branches: [main] pull_request: branches: [main]
jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: python-version: "3.12" cache: pip - run: pip install -r requirements.txt - run: pytest - run: ruff check . ```
### Go project
```yaml # .github/workflows/ci.yml name: CI
on: push: branches: [main] pull_request: branches: [main]
jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-go@v5 with: go-version: "1.22" - run: go test ./... - run: go vet ./... ```
### Rust project
```yaml # .github/workflows/ci.yml name: CI
on: push: branches: [main] pull_request: branches: [main]
jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: dtolnay/rust-toolchain@stable - uses: Swatinem/rust-cache@v2 - run: cargo test - run: cargo clippy -- -D warnings ```
## Common Patterns
### Matrix builds (test across versions/OSes)
```yaml jobs: test: strategy: fail-fast: false matrix: os: [ubuntu-latest, macos-latest, windows-latest] node-version: [18, 20, 22] runs-on: ${{ matrix.os }} steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: ${{ matrix.node-version }} - run: npm ci - run: npm test ```
### Conditional jobs
```yaml jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - run: npm test
deploy: needs: test if: github.ref == 'refs/heads/main' && github.event_name == 'push' runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - run: ./deploy.sh ```
### Caching dependencies
```yaml # Node.js (automatic with setup-node) - uses: actions/setup-node@v4 with: node-version: 20 cache: npm # or yarn, pnpm
# Generic caching - uses: actions/cache@v4 with: path: | ~/.cache/pip ~/.cargo/registry node_modules key: ${{ runner.os }}-deps-${{ hashFiles('**/package-lock.json') }} restore-keys: | ${{ runner.os }}-deps- ```
### Artifacts (save build outputs)
```yaml - uses: actions/upload-artifact@v4 with: name: build-output path: dist/ retention-days: 7
# Download in another job - uses: actions/download-artifact@v4 with: name: build-output path: dist/ ```
### Run on schedule (cron)
```yaml on: schedule: - cron: "0 6 * * 1" # Every Monday at 6 AM UTC workflow_dispatch: # Also allow manual trigger ```
## Deployment Workflows
### Deploy to production on tag
```yaml name: Release
on: push: tags: - "v*"
jobs: release: runs-on: ubuntu-latest permissions: contents: write steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: 20 cache: npm - run: npm ci - run: npm run build - run: npm test
# Create GitHub release - uses: softprops/action-gh-release@v2 with: generate_release_notes: true files: | dist/*.js dist/*.css ```
### Deploy to multiple environments
```yaml name: Deploy
on: push: branches: [main, staging]
jobs: deploy: runs-on: ubuntu-latest environment: ${{ github.ref == 'refs/heads/main' && 'production' || 'staging' }} steps: - uses: actions/checkout@v4 - run: npm ci && npm run build - run: | if [ "${{ github.ref }}" = "refs/heads/main" ]; then ./deploy.sh production else ./deploy.sh staging fi env: DEPLOY_TOKEN: ${{ secrets.DEPLOY_TOKEN }} ```
### Docker build and push
```yaml name: Docker
on: push: branches: [main] tags: ["v*"]
jobs: build: runs-on: ubuntu-latest permissions: packages: write steps: - uses: actions/checkout@v4 - uses: docker/setup-buildx-action@v3 - uses: docker/login-action@v3 with: registry: ghcr.io username: ${{ github.actor }} password: ${{ secrets.GITHUB_TOKEN }} - uses: docker/build-push-action@v6 with: push: true tags: | ghcr.io/${{ github.repository }}:latest ghcr.io/${{ github.repository }}:${{ github.sha }} cache-from: type=gha cache-to: type=gha,mode=max ```
### npm publish on release
```yaml name: Publish
on: release: types: [published]
jobs: publish: runs-on: ubuntu-latest permissions: id-token: write steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: 20 registry-url: https://registry.npmjs.org - run: npm ci - run: npm test - run: npm publish --provenance env: NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }} ```
## Secrets Management
### Set secrets via CLI
```bash # Set a repository secret gh secret set DEPLOY_TOKEN --body "my-secret-value"
# Set from a file gh secret set SSH_KEY < ~/.ssh/deploy_key
# Set for a specific environment gh secret set DB_PASSWORD --env production --body "p@ssw0rd"
# List secrets gh secret list
# Delete a secret gh secret delete OLD_SECRET ```
### Use secrets in workflows
```yaml env: # Available to all steps in this job DATABASE_URL: ${{ secrets.DATABASE_URL }}
steps: - run: echo "Deploying..." env: # Available to this step only API_KEY: ${{ secrets.API_KEY }} ```
### Environment protection rules
Set up via GitHub UI or API: - Required reviewers before deployment - Wait timers - Branch restrictions - Custom deployment branch policies
```bash # View environments gh api repos/{owner}/{repo}/environments | jq '.environments[].name' ```
## Workflow Debugging
### Re-run failed jobs
```bash # List recent workflow runs gh run list --limit 10
# View a specific run gh run view <run-id>
# View failed job logs gh run view <run-id> --log-failed
# Re-run failed jobs only gh run rerun <run-id> --failed
# Re-run entire workflow gh run rerun <run-id> ```
### Debug with SSH (using tmate)
```yaml # Add this step before the failing step - uses: mxschmitt/action-tmate@v3 if: failure() with: limit-access-to-actor: true ```
### Common failures and fixes
**"Permission denied" on scripts** ```yaml - run: chmod +x ./scripts/deploy.sh && ./scripts/deploy.sh ```
**"Node modules not found"** ```yaml # Make sure npm ci runs before npm test - run: npm ci # Install exact lockfile versions - run: npm test # Now node_modules exists ```
**"Resource not accessible by integration"** ```yaml # Add permissions block permissions: contents: write packages: write pull-requests: write ```
**Cache not restoring** ```yaml # Check cache key matches - use hashFiles for lockfile key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }} # NOT: key: ${{ runner.os }}-node-${{ hashFiles('package.json') }} ```
**Workflow not triggering** - Check: is the workflow file on the default branch? - Check: does the trigger event match? (`push` vs `pull_request`) - Check: is the branch filter correct? ```bash # Manually trigger a workflow gh workflow run ci.yml --ref main ```
## Workflow Validation
### Validate locally before pushing
```bash # Check YAML syntax python3 -c "import yaml; yaml.safe_load(open('.github/workflows/ci.yml'))" && echo "Valid"
# Use actionlint (if installed) actionlint .github/workflows/ci.yml
# Or via Docker docker run --rm -v "$(pwd):/repo" -w /repo rhysd/actionlint:latest ```
### View workflow as graph
```bash # List all workflows gh workflow list
# View workflow definition gh workflow view ci.yml
# Watch a running workflow gh run watch ```
## Advanced Patterns
### Reusable workflows
```yaml # .github/workflows/reusable-test.yml name: Reusable Test on: workflow_call: inputs: node-version: required: false type: string default: "20" secrets: npm-token: required: false
jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-node@v4 with: node-version: ${{ inputs.node-version }} - run: npm ci - run: npm test ```
```yaml # .github/workflows/ci.yml - caller name: CI on: [push, pull_request] jobs: test: uses: ./.github/workflows/reusable-test.yml with: node-version: "20" ```
### Concurrency (prevent duplicate runs)
```yaml concurrency: group: ${{ github.workflow }}-${{ github.ref }} cancel-in-progress: true # Cancel previous runs for same branch ```
### Path filters (only run for relevant changes)
```yaml on: push: paths: - "src/**" - "package.json" - "package-lock.json" - ".github/workflows/ci.yml" paths-ignore: - "docs/**" - "*.md" ```
### Monorepo: only test changed packages
```yaml jobs: changes: runs-on: ubuntu-latest outputs: api: ${{ steps.filter.outputs.api }} web: ${{ steps.filter.outputs.web }} steps: - uses: actions/checkout@v4 - uses: dorny/paths-filter@v3 id: filter with: filters: | api: - 'packages/api/**' web: - 'packages/web/**'
test-api: needs: changes if: needs.changes.outputs.api == 'true' runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - run: cd packages/api && npm ci && npm test
test-web: needs: changes if: needs.changes.outputs.web == 'true' runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - run: cd packages/web && npm ci && npm test ```
## Tips
- Use `workflow_dispatch` on every workflow for manual triggering during debugging - Pin action versions to SHA for supply chain security: `uses: actions/checkout@b4ffde...` - Use `continue-on-error: true` for non-critical steps (like linting) - Set `timeout-minutes` on jobs to prevent runaway builds (default is 360 minutes) - Use job outputs to pass data between jobs: `outputs: result: ${{ steps.step-id.outputs.value }}` - For self-hosted runners: `runs-on: self-hosted` with labels for targeting specific machines