homelab-optimized/docs/admin/ANSIBLE_PLAYBOOK_GUIDE.md

Ansible Playbook Guide for Homelab

Last updated: 2026-02-17

Overview

This guide explains how to run Ansible playbooks in the homelab infrastructure. Ansible is used for automation, configuration management, and system maintenance across all hosts in the Tailscale network.

Directory Structure

/home/homelab/organized/repos/homelab/ansible/
├── automation/
│   ├── playbooks/          # Automation and maintenance playbooks
│   ├── hosts.ini           # Inventory file (defines all hosts)
│   ├── host_vars/          # Per-host variables
│   └── group_vars/         # Group-level variables
└── homelab/
    ├── playbooks/          # Deployment playbooks
    ├── inventory.yml       # Alternative inventory format
    └── roles/              # Reusable Ansible roles

Prerequisites

1. Ansible installed on the control node (homelab machine)
2. SSH access to target hosts (configured via Tailscale)
3. Proper working directory: Run playbooks from /home/homelab/organized/repos/homelab/ansible/automation/

Basic Ansible Concepts

• Inventory: List of hosts organized into groups (defined in hosts.ini)
• Playbook: YAML file containing automation tasks
• Host Groups: Logical grouping of hosts (e.g., debian_clients, synology)
• Tasks: Individual automation steps (e.g., "update packages")
• Become: Privilege escalation (sudo) for administrative tasks

Available Playbooks

Important Notes and Limitations

⚠️ TrueNAS SCALE: Cannot be updated via apt! Package management is disabled on TrueNAS appliances. Updates must be performed through the TrueNAS web interface only. Attempting to update via apt can result in a nonfunctional system.

# Exclude TrueNAS from apt updates
ansible-playbook -i hosts.ini playbooks/update_system.yml --limit "all:!truenas-scale"

⚠️ Raspberry Pi GPG Keys: If pi-5 fails with GPG signature errors for InfluxDB repository, fix with:

ansible -i hosts.ini pi-5 -m shell -a "curl -sL https://repos.influxdata.com/influxdata-archive_compat.key | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/influxdata.gpg" --become

⚠️ Home Assistant: Uses its own package management system and should be excluded from apt updates.

System Maintenance

1. update_system.yml

Updates apt cache and upgrades all packages on Debian-based systems.

Hosts: All hosts with Debian/Ubuntu (exclude TrueNAS and Home Assistant) Requires sudo: Yes Use case: Regular system updates

# Recommended: Exclude TrueNAS
ansible-playbook -i hosts.ini playbooks/update_system.yml --limit "all:!truenas-scale:!homeassistant"

# Or update specific hosts only
ansible-playbook -i hosts.ini playbooks/update_system.yml --limit "homelab,pve,pi-5,vish-concord-nuc"

2. update_ansible.yml

Updates apt cache and specifically upgrades Ansible on Linux hosts (excludes Synology).

Hosts: debian_clients (excluding Synology and Home Assistant) Requires sudo: Yes Use case: Keep Ansible up-to-date on managed hosts

ansible-playbook -i hosts.ini playbooks/update_ansible.yml

3. update_ansible_targeted.yml

Same as update_ansible.yml but allows targeting specific hosts or groups.

Hosts: Configurable via --limit Requires sudo: Yes Use case: Update Ansible on specific hosts only

# Update only on homelab and pi-5
ansible-playbook -i hosts.ini playbooks/update_ansible_targeted.yml --limit "homelab,pi-5"

# Update only on Raspberry Pis
ansible-playbook -i hosts.ini playbooks/update_ansible_targeted.yml --limit "rpi"

APT Cache / Proxy Management

4. check_apt_proxy.yml

Comprehensive health check for APT cache proxy configuration. Verifies that hosts are properly configured to use Calypso's apt-cacher-ng service.

Hosts: debian_clients Requires sudo: Partially (for some checks) Use case: Verify apt-cacher-ng is working correctly Expected proxy: calypso (100.103.48.78:3142)

ansible-playbook -i hosts.ini playbooks/check_apt_proxy.yml

What it checks:

• APT proxy configuration file exists (/etc/apt/apt.conf.d/01proxy)
• Proxy points to correct server (Calypso)
• Network connectivity to proxy server
• APT configuration is valid
• Provides recommendations for misconfigured hosts

5. configure_apt_proxy.yml

Configures hosts to use Calypso's APT cache proxy.

Hosts: debian_clients Requires sudo: Yes Use case: Set up apt-cacher-ng on new hosts

ansible-playbook -i hosts.ini playbooks/configure_apt_proxy.yml

Health Checks

6. ansible_status_check.yml

Checks Ansible installation and connectivity across all hosts.

Hosts: All Requires sudo: No Use case: Verify Ansible can communicate with all hosts

ansible-playbook -i hosts.ini playbooks/ansible_status_check.yml

7. synology_health.yml

Health check specific to Synology NAS devices.

Hosts: synology group Requires sudo: No Use case: Monitor Synology system health

ansible-playbook -i hosts.ini playbooks/synology_health.yml

8. tailscale_health.yml

Checks Tailscale connectivity and status.

Hosts: All Requires sudo: No Use case: Verify Tailscale VPN is working

ansible-playbook -i hosts.ini playbooks/tailscale_health.yml

Utility Playbooks

9. system_info.yml

Gathers and displays system information from all hosts.

Hosts: All Requires sudo: No Use case: Quick inventory of system specs

ansible-playbook -i hosts.ini playbooks/system_info.yml

10. add_ssh_keys.yml

Adds SSH keys to target hosts for passwordless authentication.

Hosts: Configurable Requires sudo: No Use case: Set up SSH access for new hosts

ansible-playbook -i hosts.ini playbooks/add_ssh_keys.yml

11. cleanup.yml

Performs system cleanup tasks (apt autoclean, autoremove, etc.).

Hosts: debian_clients Requires sudo: Yes Use case: Free up disk space

ansible-playbook -i hosts.ini playbooks/cleanup.yml

12. install_tools.yml

Installs common tools and utilities on hosts.

Hosts: Configurable Requires sudo: Yes Use case: Standardize tool installation

ansible-playbook -i hosts.ini playbooks/install_tools.yml

Host Groups Reference

From hosts.ini:

Group	Hosts	Purpose
homelab	homelab	Main management node
synology	atlantis, calypso, setillo	Synology NAS devices
rpi	pi-5, pi-5-kevin	Raspberry Pi nodes
hypervisors	pve, truenas-scale, homeassistant	Virtualization hosts
remote	vish-concord-nuc	Remote systems
debian_clients	homelab, pi-5, pi-5-kevin, vish-concord-nuc, pve, homeassistant, truenas-scale	All Debian/Ubuntu hosts using APT cache (⚠️ exclude truenas-scale and homeassistant from apt updates)
all	All hosts	Every host in inventory

Running Playbooks

Basic Syntax

cd /home/homelab/organized/repos/homelab/ansible/automation/

ansible-playbook -i hosts.ini playbooks/<playbook-name>.yml

Common Options

Target Specific Hosts

# Single host
ansible-playbook -i hosts.ini playbooks/update_system.yml --limit homelab

# Multiple hosts
ansible-playbook -i hosts.ini playbooks/update_system.yml --limit "homelab,pi-5"

# All hosts in a group
ansible-playbook -i hosts.ini playbooks/update_system.yml --limit "rpi"

# All except specific hosts
ansible-playbook -i hosts.ini playbooks/update_system.yml --limit "all:!synology"

Check Mode (Dry Run)

Preview what would change without actually making changes:

ansible-playbook -i hosts.ini playbooks/update_system.yml --check

Verbose Output

Get more detailed information about what Ansible is doing:

# Basic verbose
ansible-playbook -i hosts.ini playbooks/check_apt_proxy.yml -v

# More verbose (connection info)
ansible-playbook -i hosts.ini playbooks/check_apt_proxy.yml -vv

# Very verbose (includes module info)
ansible-playbook -i hosts.ini playbooks/check_apt_proxy.yml -vvv

# Debug level (everything)
ansible-playbook -i hosts.ini playbooks/check_apt_proxy.yml -vvvv

Ask for Sudo Password

If SSH user doesn't have passwordless sudo:

ansible-playbook -i hosts.ini playbooks/update_system.yml --ask-become-pass
# or short form:
ansible-playbook -i hosts.ini playbooks/update_system.yml -K

Ask for SSH Password

If using password authentication instead of SSH keys:

ansible-playbook -i hosts.ini playbooks/system_info.yml --ask-pass
# or short form:
ansible-playbook -i hosts.ini playbooks/system_info.yml -k

Common Workflows

Weekly Maintenance Routine

cd /home/homelab/organized/repos/homelab/ansible/automation/

# 1. Check that all hosts are reachable
ansible-playbook -i hosts.ini playbooks/ansible_status_check.yml

# 2. Verify APT cache proxy is working
ansible-playbook -i hosts.ini playbooks/check_apt_proxy.yml

# 3. Update all systems
ansible-playbook -i hosts.ini playbooks/update_system.yml

# 4. Clean up old packages
ansible-playbook -i hosts.ini playbooks/cleanup.yml

# 5. Check Tailscale connectivity
ansible-playbook -i hosts.ini playbooks/tailscale_health.yml

Adding a New Host

# 1. Edit hosts.ini and add the new host to appropriate groups
nano hosts.ini

# 2. Test connectivity
ansible -i hosts.ini <new-host> -m ping

# 3. Add SSH keys
ansible-playbook -i hosts.ini playbooks/add_ssh_keys.yml --limit <new-host>

# 4. Configure APT proxy
ansible-playbook -i hosts.ini playbooks/configure_apt_proxy.yml --limit <new-host>

# 5. Install standard tools
ansible-playbook -i hosts.ini playbooks/install_tools.yml --limit <new-host>

# 6. Update system
ansible-playbook -i hosts.ini playbooks/update_system.yml --limit <new-host>

Troubleshooting a Host

# 1. Get system info
ansible-playbook -i hosts.ini playbooks/system_info.yml --limit <host>

# 2. Check Ansible status
ansible-playbook -i hosts.ini playbooks/ansible_status_check.yml --limit <host>

# 3. Check Tailscale connectivity
ansible-playbook -i hosts.ini playbooks/tailscale_health.yml --limit <host>

# 4. Verify APT configuration
ansible-playbook -i hosts.ini playbooks/check_apt_proxy.yml --limit <host>

Ad-Hoc Commands

For quick one-off tasks, use ansible directly:

# Ping all hosts
ansible -i hosts.ini all -m ping

# Check disk space
ansible -i hosts.ini all -m shell -a "df -h" --become

# Restart a service
ansible -i hosts.ini homelab -m systemd -a "name=docker state=restarted" --become

# Check uptime
ansible -i hosts.ini all -m command -a "uptime"

# Get memory info
ansible -i hosts.ini all -m shell -a "free -h"

Troubleshooting

Connection Issues

Problem: "Connection timeout" or "Host unreachable"

# Test direct ping
ping <host-ip>

# Test SSH manually
ssh <user>@<host-ip>

# Check Tailscale status
tailscale status

Problem: "Permission denied (publickey)"

# Add your SSH key to the host
ssh-copy-id <user>@<host-ip>

# Or use password authentication
ansible-playbook -i hosts.ini playbooks/<playbook>.yml -k

Privilege Escalation Issues

Problem: "This command has to be run under the root user"

# Use --ask-become-pass
ansible-playbook -i hosts.ini playbooks/<playbook>.yml -K

# Or configure passwordless sudo on target host:
# sudo visudo
# Add: <user> ALL=(ALL) NOPASSWD:ALL

Playbook Failures

Problem: Task fails on some hosts

# Run in verbose mode to see detailed errors
ansible-playbook -i hosts.ini playbooks/<playbook>.yml -vvv

# Use --limit to retry only failed hosts
ansible-playbook -i hosts.ini playbooks/<playbook>.yml --limit @/tmp/retry_hosts.txt

Problem: "Module not found"

# Update Ansible on control node
sudo apt update && sudo apt upgrade ansible -y

# Check Ansible version
ansible --version

APT Update Failures

Problem: "Failed to update apt cache: unknown reason" (especially on Raspberry Pi)

# Often caused by missing GPG keys. Test manually:
ansible -i hosts.ini <host> -m shell -a "sudo apt-get update 2>&1" --become

# Fix missing GPG keys (InfluxDB example):
ansible -i hosts.ini <host> -m shell -a "curl -sL https://repos.influxdata.com/influxdata-archive_compat.key | sudo gpg --dearmor -o /etc/apt/trusted.gpg.d/influxdata.gpg" --become

# Workaround: Use shell commands instead of apt module
ansible -i hosts.ini <host> -m shell -a "sudo apt-get update && sudo apt-get upgrade -y" --become

Problem: TrueNAS apt update fails with "rc: -9" or package management disabled

# This is expected behavior - TrueNAS disables apt for system stability
# Solution: Update TrueNAS only through its web interface
# Exclude from playbooks:
ansible-playbook -i hosts.ini playbooks/update_system.yml --limit "all:!truenas-scale"

Problem: "Package lock" or "Unable to acquire dpkg lock"

# Check if another process is using apt
ansible -i hosts.ini <host> -m shell -a "sudo lsof /var/lib/dpkg/lock-frontend" --become

# Kill stuck apt processes (use with caution)
ansible -i hosts.ini <host> -m shell -a "sudo killall apt apt-get" --become

# Remove lock files if no process is running
ansible -i hosts.ini <host> -m shell -a "sudo rm /var/lib/dpkg/lock-frontend /var/lib/dpkg/lock" --become

Inventory Issues

Problem: "Could not match supplied host pattern"

# List all hosts in inventory
ansible -i hosts.ini all --list-hosts

# List hosts in a specific group
ansible -i hosts.ini debian_clients --list-hosts

# Verify inventory file syntax
ansible-inventory -i hosts.ini --list

Best Practices

1. Always use version control: Commit playbook changes to git
2. Test with --check first: Use dry-run mode for risky changes
3. Start small: Test on a single host before running on all hosts
4. Document changes: Add comments to playbooks explaining what they do
5. Use tags: Tag tasks for selective execution
6. Keep playbooks idempotent: Running multiple times should be safe
7. Monitor logs: Check /var/log/ansible.log on managed hosts
8. Backup before major changes: Create snapshots of important systems

Security Considerations

1. SSH Keys: Use SSH keys instead of passwords when possible
2. Vault: Use Ansible Vault for sensitive data (passwords, API keys)
3. Least Privilege: Don't run playbooks with more privileges than needed
4. Audit Trail: Keep git history of all playbook changes
5. Network Isolation: Use Tailscale for secure communication

Quick Reference Card

Task	Command
Update all systems	ansible-playbook -i hosts.ini playbooks/update_system.yml
Check APT proxy	ansible-playbook -i hosts.ini playbooks/check_apt_proxy.yml
Update Ansible	ansible-playbook -i hosts.ini playbooks/update_ansible.yml
Ping all hosts	ansible -i hosts.ini all -m ping
Get system info	ansible-playbook -i hosts.ini playbooks/system_info.yml
Clean up systems	ansible-playbook -i hosts.ini playbooks/cleanup.yml
Dry run (no changes)	ansible-playbook -i hosts.ini playbooks/<playbook>.yml --check
Verbose output	ansible-playbook -i hosts.ini playbooks/<playbook>.yml -vvv
Target one host	ansible-playbook -i hosts.ini playbooks/<playbook>.yml --limit <host>

Additional Resources

• Ansible Documentation
• Ansible Best Practices
• Ansible Galaxy - Community roles and playbooks
• Repository: /home/homelab/organized/repos/homelab/ansible/

Related Documentation

• Git Branches Guide - Version control for playbook changes
• Infrastructure Overview - Homelab infrastructure details
• Ansible host vars: /home/homelab/organized/repos/homelab/ansible/automation/host_vars/