homelab-optimized/docs/runbooks/add-new-service.md

Add New Service Runbook

Overview

This runbook guides you through deploying a new containerized service to the homelab using GitOps with Portainer. The procedure ensures proper configuration, monitoring, and documentation.

Prerequisites

• Git access to the homelab repository
• Portainer access (https://192.168.0.200:9443) - Portainer EE v2.33.7
• Target host selected and available
• Service Docker Compose file prepared
• Required environment variables identified
• Network requirements understood (ports, domains, etc.)

Current GitOps Status

• Active Deployments: 18 compose stacks on Atlantis (verified Feb 14, 2026)
• Total Containers: 50+ containers across infrastructure
• GitOps Method: Automatic sync from Git repository via Portainer EE

Metadata

• Estimated Time: 30-60 minutes
• Risk Level: Low (if following proper testing)
• Requires Downtime: No (for new services)
• Reversible: Yes (can remove stack)
• Tested On: 2026-02-14

Decision: Which Host?

Choose the appropriate host based on service requirements:

Host	Best For	Available Resources	GitOps Status
Atlantis (DS1823xs+)	Media services, high I/O, primary storage	8 CPU, 31GB RAM, 50+ containers	✅ 18 Active Stacks
Calypso (DS723+)	Secondary media, backup services	4 CPU, 31GB RAM, 46 containers	✅ GitOps Ready
Concord NUC	Network services, DNS, VPN	4 CPU, 15.5GB RAM, 17 containers	✅ GitOps Ready
Homelab VM	Development, monitoring, testing	4 CPU, 28.7GB RAM, 23 containers	✅ GitOps Ready
Raspberry Pi 5	IoT, edge computing, lightweight services	4 CPU, 15.8GB RAM, 4 containers	✅ GitOps Ready

Procedure

Step 1: Create Docker Compose Configuration

Create a new compose file in the appropriate host directory:

cd ~/Documents/repos/homelab
# Choose the appropriate path:
# - hosts/synology/atlantis/
# - hosts/synology/calypso/
# - hosts/physical/concord-nuc/
# - hosts/vms/homelab-vm/
# - hosts/edge/raspberry-pi-5/

# Create new service file
nano hosts/[host]/[service-name].yaml

Example Docker Compose structure:

version: '3.8'

services:
  service-name:
    image: organization/image:tag
    container_name: service-name
    restart: unless-stopped

    environment:
      - PUID=1000
      - PGID=1000
      - TZ=America/Los_Angeles
      # Add service-specific variables

    volumes:
      - /path/to/config:/config
      - /path/to/data:/data

    ports:
      - "8080:8080"  # external:internal

    networks:
      - service-network

    # Optional: health check
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s

    # Optional: resource limits
    deploy:
      resources:
        limits:
          cpus: '2.0'
          memory: 4G
        reservations:
          cpus: '1.0'
          memory: 2G

networks:
  service-network:
    driver: bridge

# Optional: named volumes
volumes:
  service-data:
    driver: local

Step 2: Configure Environment Variables

If your service requires sensitive data, create an .env file (ensure it's in .gitignore):

# Create .env file (DO NOT commit to git)
nano .env.example  # Template for others

# Example .env content:
# SERVICE_API_KEY=REDACTED_API_KEY
# SERVICE_SECRET=your_secret_here
# DATABASE_PASSWORD="REDACTED_PASSWORD"

Step 3: Validate Configuration Locally (Optional but Recommended)

Test the compose file syntax:

# Validate syntax
docker-compose -f hosts/[host]/[service-name].yaml config

# Expected output: Valid YAML with no errors

Step 4: Commit and Push to Git Repository

# Add the new service file
git add hosts/[host]/[service-name].yaml

# If adding .env.example template
git add .env.example

# Commit with descriptive message
git commit -m "Add [service-name] deployment for [host]

- Add Docker Compose configuration
- Configure environment variables
- Set resource limits and health checks
- Documentation: [purpose of service]

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>"

# Push to remote
git push origin main

Expected output:

[main abc1234] Add service-name deployment for host
 1 file changed, 45 insertions(+)
 create mode 100644 hosts/host/service-name.yaml

Step 5: Deploy via Portainer GitOps

Option A: Auto-Sync (If GitOps is enabled)

1. Wait 5-10 minutes for automatic sync
2. Check Portainer dashboard for new stack
3. Skip to Step 6 (Verification)

Option B: Manual Deployment via Portainer UI

1. Open Portainer: http://vishinator.synology.me:10000
2. Navigate to the target endpoint (e.g., "Atlantis", "Calypso")
3. Click Stacks → Add stack
4. Configure stack:
   • Name: service-name
   • Build method: Git Repository
   • Repository URL: Your git repository URL
   • Repository reference: refs/heads/main
   • Compose path: hosts/[host]/[service-name].yaml
   • GitOps updates: ✅ Enable (optional)
5. Add environment variables if needed
6. Click Deploy the stack

Step 6: Verify Deployment

Check container status:

# SSH to the host
ssh atlantis  # or appropriate host

# Check container is running
docker ps | grep service-name

# Check logs for errors
docker logs service-name --tail 50

# Check resource usage
docker stats service-name --no-stream

Expected output:

CONTAINER ID   IMAGE                    STATUS          PORTS
abc123def456   org/image:tag           Up 2 minutes    0.0.0.0:8080->8080/tcp

Step 7: Configure Networking (If External Access Needed)

For Services Behind Reverse Proxy:

1. Add to Nginx Proxy Manager or configure reverse proxy
2. Create DNS record (Cloudflare or local DNS)
3. Configure SSL certificate (Let's Encrypt)

For Services Using Authentik SSO:

1. Add application in Authentik
2. Configure OAuth2/SAML provider
3. Update service with Authentik integration

See Authentik SSO Guide for details.

Step 8: Add to Monitoring (Optional but Recommended)

Add service to monitoring stack:

# Add to prometheus/prometheus.yml
scrape_configs:
  - job_name: 'service-name'
    static_configs:
      - targets: ['service-name:8080']

Update Grafana dashboards if needed.

Step 9: Document the Service

Update service inventory:

# Edit service documentation
nano docs/services/VERIFIED_SERVICE_INVENTORY.md

# Add entry:
# | Service Name | Host | Port | URL | Status |
# |--------------|------|------|-----|--------|
# | Service Name | Atlantis | 8080 | https://service.vish.gg | ✅ Active |

Step 10: Configure Backups (If Storing Important Data)

Add service to backup scripts:

# Edit backup configuration
nano backup.sh

# Add service data directory to backup list
BACKUP_DIRS=(
    # ... existing dirs ...
    "/path/to/service-name/data"
)

Test backup:

./backup.sh --test

Verification Checklist

After deployment, verify the following:

• Container is running: docker ps | grep service-name
• Logs show no critical errors: docker logs service-name
• Service responds on expected port: curl http://localhost:8080
• Health check passes (if configured): docker inspect service-name | grep Health
• Resource usage is reasonable: docker stats service-name --no-stream
• External access works (if configured): curl https://service.vish.gg
• SSO authentication works (if using Authentik)
• Service is added to documentation
• Monitoring is configured (if applicable)
• Backups include service data (if applicable)

Rollback Procedure

If the deployment fails or causes issues:

Via Portainer UI:

1. Go to Stacks → Select the problematic stack
2. Click Stop to stop the stack
3. Click Remove to delete the stack
4. Delete associated volumes if needed

Via Command Line:

# SSH to host
ssh [host]

# Stop and remove container
docker stop service-name
docker rm service-name

# Remove associated volumes (if needed)
docker volume ls | grep service-name
docker volume rm [volume-name]

# Remove from git
cd ~/Documents/repos/homelab
git rm hosts/[host]/[service-name].yaml
git commit -m "Rollback: Remove service-name deployment"
git push origin main

Troubleshooting

Issue: Container Fails to Start

Symptoms: Container status shows "Exited" or "Restarting"

Solution:

# Check logs for error messages
docker logs service-name --tail 100

# Common issues:
# - Port already in use: Change port mapping
# - Permission denied: Check PUID/PGID
# - Missing env variables: Add to compose file
# - Volume mount issues: Verify paths exist

Issue: Container Starts But Service Unreachable

Symptoms: Container running but can't access service

Solution:

# Check if service is listening on correct port
docker exec service-name netstat -tlnp

# Check container network
docker network inspect [network-name]

# Test from within container
docker exec service-name curl localhost:8080

# Check firewall rules on host
sudo ufw status

Issue: GitOps Auto-Sync Not Working

Symptoms: Pushed changes but Portainer doesn't update

Solution:

1. Check Portainer stack settings → GitOps
2. Verify webhook is configured correctly
3. Manually trigger sync: Stack → Pull and redeploy
4. Check Portainer logs for git sync errors

Issue: High Resource Usage

Symptoms: Container using too much CPU/RAM

Solution:

# Add resource limits to compose file
deploy:
  resources:
    limits:
      cpus: '1.0'
      memory: 2G

# Redeploy with limits
docker-compose up -d

Post-Deployment Tasks

After successful deployment:

1. Test the service thoroughly - Ensure all features work as expected
2. Set up monitoring alerts - Configure Grafana alerts for the service
3. Document usage - Add user guide if others will use the service
4. Schedule maintenance - Add to maintenance calendar for updates
5. Test backups - Verify backup includes service data
6. Update runbook - Note any deviations or improvements

Related Documentation

• GitOps Deployment Guide
• Deployment Documentation
• Infrastructure Overview
• Service Inventory
• Monitoring Setup

Examples

Example 1: Adding Uptime Kuma

version: '3.8'

services:
  uptime-kuma:
    image: louislam/uptime-kuma:1
    container_name: uptime-kuma
    restart: unless-stopped

    volumes:
      - /volume1/docker/uptime-kuma:/app/data

    ports:
      - "3001:3001"

    networks:
      - monitoring

networks:
  monitoring:
    external: true

Example 2: Adding a Service with Database

version: '3.8'

services:
  app:
    image: myapp:latest
    depends_on:
      - postgres
    environment:
      - DATABASE_URL=postgresql://user:REDACTED_PASSWORD@postgres:5432/dbname
    ports:
      - "8080:8080"
    networks:
      - app-network

  postgres:
    image: postgres:15
    environment:
      - POSTGRES_USER=user
      - POSTGRES_PASSWORD="REDACTED_PASSWORD"
      - POSTGRES_DB=dbname
    volumes:
      - postgres-data:/var/lib/postgresql/data
    networks:
      - app-network

networks:
  app-network:
    driver: bridge

volumes:
  postgres-data:

Change Log

• 2026-02-14 - Initial creation with GitOps workflow
• 2026-02-14 - Added examples and troubleshooting section