homelab-optimized/docs/troubleshooting/authentik-sso-rebuild.md

# Authentik SSO Disaster Recovery & Rebuild Guide

**Last Updated**: 2026-01-31  
**Tested On**: Authentik 2024.12.x on Calypso (Synology DS723+)

This guide documents the complete process to rebuild Authentik SSO and reconfigure OAuth2 for all homelab services from scratch.

---

## Table of Contents

1. [Prerequisites](#prerequisites)
2. [Deploy Authentik](#deploy-authentik)
3. [Initial Configuration](#initial-configuration)
4. [Configure OAuth2 Providers](#configure-oauth2-providers)
5. [Configure Forward Auth Providers](#configure-forward-auth-providers)
6. [Service-Specific Configuration](#service-specific-configuration)
7. [NPM Integration](#npm-integration)
8. [Troubleshooting](#troubleshooting)
9. [Recovery Procedures](#recovery-procedures)

---

## Prerequisites

### Infrastructure Required
- Docker host (Calypso NAS or equivalent)
- PostgreSQL database
- Redis
- Nginx Proxy Manager (NPM) for reverse proxy
- Domain with SSL (e.g., sso.vish.gg via Cloudflare)

### Network Configuration
| Service | Host | Port |
|---------|------|------|
| Authentik Server | 192.168.0.250 | 9000 |
| Authentik Worker | 192.168.0.250 | (internal) |
| PostgreSQL | 192.168.0.250 | 5432 |
| Redis | 192.168.0.250 | 6379 |

### Credentials to Have Ready
- Admin email (e.g., admin@example.com)
- Strong admin password
- SMTP settings (optional, for email notifications)

---

## Deploy Authentik

### Docker Compose File

Location: `hosts/synology/calypso/authentik/docker-compose.yaml`

```yaml
version: '3.9'

services:
  postgresql:
    image: postgres:16-alpine
    container_name: Authentik-DB
    restart: unless-stopped
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -d $${POSTGRES_DB} -U $${POSTGRES_USER}"]
      start_period: 20s
      interval: 30s
      retries: 5
      timeout: 5s
    volumes:
      - database:/var/lib/postgresql/data
    environment:
      POSTGRES_PASSWORD: "REDACTED_PASSWORD" password required}
      POSTGRES_USER: ${PG_USER:-authentik}
      POSTGRES_DB: ${PG_DB:-authentik}
    networks:
      - authentik

  redis:
    image: redis:alpine
    container_name: Authentik-REDIS
    command: --save 60 1 --loglevel warning
    restart: unless-stopped
    healthcheck:
      test: ["CMD-SHELL", "redis-cli ping | grep PONG"]
      start_period: 20s
      interval: 30s
      retries: 5
      timeout: 3s
    volumes:
      - redis:/data
    networks:
      - authentik

  server:
    image: ghcr.io/goauthentik/server:2024.12
    container_name: Authentik-SERVER
    restart: unless-stopped
    command: server
    environment:
      AUTHENTIK_REDIS__HOST: redis
      AUTHENTIK_POSTGRESQL__HOST: postgresql
      AUTHENTIK_POSTGRESQL__USER: ${PG_USER:-authentik}
      AUTHENTIK_POSTGRESQL__NAME: ${PG_DB:-authentik}
      AUTHENTIK_POSTGRESQL__PASSWORD: "REDACTED_PASSWORD"
      AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY}
    volumes:
      - ./media:/media
      - ./custom-templates:/templates
    ports:
      - "9000:9000"
      - "9443:9443"
    depends_on:
      postgresql:
        condition: service_healthy
      redis:
        condition: service_healthy
    networks:
      - authentik

  worker:
    image: ghcr.io/goauthentik/server:2024.12
    container_name: Authentik-WORKER
    restart: unless-stopped
    command: worker
    environment:
      AUTHENTIK_REDIS__HOST: redis
      AUTHENTIK_POSTGRESQL__HOST: postgresql
      AUTHENTIK_POSTGRESQL__USER: ${PG_USER:-authentik}
      AUTHENTIK_POSTGRESQL__NAME: ${PG_DB:-authentik}
      AUTHENTIK_POSTGRESQL__PASSWORD: "REDACTED_PASSWORD"
      AUTHENTIK_SECRET_KEY: ${AUTHENTIK_SECRET_KEY}
    volumes:
      - ./media:/media
      - ./custom-templates:/templates
    depends_on:
      postgresql:
        condition: service_healthy
      redis:
        condition: service_healthy
    networks:
      - authentik

volumes:
  database:
  redis:

networks:
  authentik:
    driver: bridge
```

### Environment File (.env)

```bash
PG_PASS="REDACTED_PASSWORD"
PG_USER=authentik
PG_DB=authentik
AUTHENTIK_SECRET_KEY=<generate-with-openssl-rand-base64-60>
```

### Generate Secret Key

```bash
openssl rand -base64 60
```

### Deploy

```bash
cd /volume1/docker/authentik
docker-compose up -d
```

### Verify Deployment

```bash
docker ps | grep -i authentik
# Should show: Authentik-SERVER, Authentik-WORKER, Authentik-DB, Authentik-REDIS
```

---

## Initial Configuration

### First-Time Setup

1. Navigate to `https://sso.vish.gg/if/flow/initial-setup/`
2. Create admin account:
   - **Username**: `akadmin`
   - **Email**: `admin@example.com`
   - **Password**: (use password manager)

### Post-Setup Configuration

1. **Admin Interface**: `https://sso.vish.gg/if/admin/`
2. **User Portal**: `https://sso.vish.gg/if/user/`

### Create User Groups (Optional but Recommended)

Navigate to: Admin → Directory → Groups

| Group Name | Purpose |
|------------|---------|
| `Grafana Admins` | Admin access to Grafana |
| `Grafana Editors` | Editor access to Grafana |
| `Homelab Users` | General homelab access |

---

## Configure OAuth2 Providers

### Critical: Scope Mappings

**EVERY OAuth2 provider MUST have these scope mappings configured, or logins will fail with "InternalError":**

1. Go to: Admin → Customization → Property Mappings
2. Note these default mappings exist:
   - `authentik default OAuth Mapping: OpenID 'openid'`
   - `authentik default OAuth Mapping: OpenID 'email'`
   - `authentik default OAuth Mapping: OpenID 'profile'`

When creating providers, you MUST add these to the "Scopes" field.

### Provider 1: Grafana OAuth2

**Admin → Providers → Create → OAuth2/OpenID Provider**

| Setting | Value |
|---------|-------|
| Name | `Grafana OAuth2` |
| Authentication flow | default-authentication-flow |
| Authorization flow | default-provider-authorization-implicit-consent |
| Client type | Confidential |
| Client ID | (auto-generated, save this) |
| Client Secret | (auto-generated, save this) |
| Redirect URIs | `https://gf.vish.gg/login/generic_oauth` |
| Signing Key | authentik Self-signed Certificate |
| **Scopes** | Select: `openid`, `email`, `profile` ⚠️ CRITICAL |

**Create Application:**
- Admin → Applications → Create
- Name: `Grafana`
- Slug: `grafana`
- Provider: `Grafana OAuth2`
- Launch URL: `https://gf.vish.gg`

### Provider 2: Gitea OAuth2

**Admin → Providers → Create → OAuth2/OpenID Provider**

| Setting | Value |
|---------|-------|
| Name | `Gitea OAuth2` |
| Authorization flow | default-provider-authorization-implicit-consent |
| Client type | Confidential |
| Redirect URIs | `https://git.vish.gg/user/oauth2/authentik/callback` |
| **Scopes** | Select: `openid`, `email`, `profile` ⚠️ CRITICAL |

**Create Application:**
- Name: `Gitea`
- Slug: `gitea`
- Provider: `Gitea OAuth2`
- Launch URL: `https://git.vish.gg`

### Provider 3: Portainer OAuth2

**Admin → Providers → Create → OAuth2/OpenID Provider**

| Setting | Value |
|---------|-------|
| Name | `Portainer OAuth2` |
| Authorization flow | default-provider-authorization-implicit-consent |
| Client type | Confidential |
| Redirect URIs | `http://vishinator.synology.me:10000` |
| **Scopes** | Select: `openid`, `email`, `profile` ⚠️ CRITICAL |

**Create Application:**
- Name: `Portainer`
- Slug: `portainer`
- Provider: `Portainer OAuth2`
- Launch URL: `http://vishinator.synology.me:10000`

### Provider 4: Seafile OAuth2

**Admin → Providers → Create → OAuth2/OpenID Provider**

| Setting | Value |
|---------|-------|
| Name | `Seafile OAuth2` |
| Authorization flow | default-provider-authorization-implicit-consent |
| Client type | Confidential |
| Redirect URIs | `https://sf.vish.gg/oauth/callback/` |
| **Scopes** | Select: `openid`, `email`, `profile` ⚠️ CRITICAL |

**Create Application:**
- Name: `Seafile`
- Slug: `seafile`
- Launch URL: `https://sf.vish.gg`

---

## Configure Forward Auth Providers

Forward Auth is used for services that don't have native OAuth support. Authentik intercepts all requests and requires login first.

### Provider: vish.gg Domain Forward Auth

**Admin → Providers → Create → Proxy Provider**

| Setting | Value |
|---------|-------|
| Name | `vish.gg Domain Forward Auth` |
| Authorization flow | default-provider-authorization-implicit-consent |
| Mode | Forward auth (single application) |
| External host | `https://sso.vish.gg` |

**Create Application:**
- Name: `vish.gg Domain Auth`
- Slug: `vishgg-domain-auth`
- Provider: `vish.gg Domain Forward Auth`

### Create/Update Outpost

**Admin → Applications → Outposts**

1. Edit the embedded outpost (or create one)
2. Add all Forward Auth applications to it
3. The outpost will listen on port 9000

---

## Service-Specific Configuration

### Grafana Configuration

**Environment variables** (in docker-compose or Portainer):

```yaml
environment:
  # OAuth2 SSO
  - GF_AUTH_GENERIC_OAUTH_ENABLED=true
  - GF_AUTH_GENERIC_OAUTH_NAME=Authentik
  - GF_AUTH_GENERIC_OAUTH_CLIENT_ID=<client_id>
  - GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET=<client_secret>
  - GF_AUTH_GENERIC_OAUTH_SCOPES=openid profile email
  - GF_AUTH_GENERIC_OAUTH_AUTH_URL=https://sso.vish.gg/application/o/authorize/
  - GF_AUTH_GENERIC_OAUTH_TOKEN_URL=https://sso.vish.gg/application/o/token/
  - GF_AUTH_GENERIC_OAUTH_API_URL=https://sso.vish.gg/application/o/userinfo/
  - GF_AUTH_SIGNOUT_REDIRECT_URL=https://sso.vish.gg/application/o/grafana/end-session/
  
  # CRITICAL: Attribute paths
  - GF_AUTH_GENERIC_OAUTH_EMAIL_ATTRIBUTE_PATH=email
  - GF_AUTH_GENERIC_OAUTH_LOGIN_ATTRIBUTE_PATH=preferred_username
  - GF_AUTH_GENERIC_OAUTH_NAME_ATTRIBUTE_PATH=name
  
  # Role mapping
  - GF_AUTH_GENERIC_OAUTH_ROLE_ATTRIBUTE_PATH=contains(groups[*], 'Grafana Admins') && 'Admin' || contains(groups[*], 'Grafana Editors') && 'Editor' || 'Viewer'
  
  # Additional settings
  - GF_AUTH_GENERIC_OAUTH_USE_PKCE=true
  - GF_AUTH_GENERIC_OAUTH_ALLOW_ASSIGN_GRAFANA_ADMIN=true
  - GF_SERVER_ROOT_URL=https://gf.vish.gg
```

### Gitea Configuration

Configure via **Site Administration → Authentication Sources → Add OAuth2**:

| Setting | Value |
|---------|-------|
| Authentication Name | `authentik` |
| OAuth2 Provider | OpenID Connect |
| Client ID | (from Authentik) |
| Client Secret | (from Authentik) |
| OpenID Connect Auto Discovery URL | `https://sso.vish.gg/application/o/gitea/.well-known/openid-configuration` |

### Portainer Configuration

Configure via **Settings → Authentication → OAuth**:

| Setting | Value |
|---------|-------|
| Client ID | (from Authentik) |
| Client Secret | (from Authentik) |
| Authorization URL | `https://sso.vish.gg/application/o/authorize/` |
| Access Token URL | `https://sso.vish.gg/application/o/token/` |
| Resource URL | `https://sso.vish.gg/application/o/userinfo/` |
| Redirect URL | `http://vishinator.synology.me:10000` |
| User Identifier | `email` |
| Scopes | `openid profile email` |

### Seafile Configuration

Add to `/volume1/docker/seafile/data/seafile/conf/seahub_settings.py`:

```python
ENABLE_OAUTH = True
OAUTH_ENABLE_INSECURE_TRANSPORT = False
OAUTH_CLIENT_ID = "<client_id>"
OAUTH_CLIENT_SECRET = "<client_secret>"
OAUTH_REDIRECT_URL = "https://sf.vish.gg/oauth/callback/"
OAUTH_PROVIDER_DOMAIN = "sso.vish.gg"
OAUTH_AUTHORIZATION_URL = "https://sso.vish.gg/application/o/authorize/"
OAUTH_TOKEN_URL = "https://sso.vish.gg/application/o/token/"
OAUTH_USER_INFO_URL = "https://sso.vish.gg/application/o/userinfo/"
OAUTH_SCOPE = ["openid", "profile", "email"]
OAUTH_ATTRIBUTE_MAP = {
    "email": (True, "email"),
    "name": (False, "name"),
}
```

Then restart Seafile: `docker restart Seafile`

---

## NPM Integration

### For OAuth2 Services (Grafana, Gitea, etc.)

**DO NOT add Forward Auth config!** These services handle OAuth themselves.

NPM proxy host should be simple:
- Forward host: service IP
- Forward port: service port
- SSL: enabled
- Advanced config: **EMPTY**

### For Forward Auth Services (Paperless, Actual, etc.)

Add this to NPM Advanced Config:

```nginx
# Authentik Forward Auth Configuration
proxy_buffers 8 16k;
proxy_buffer_size 32k;

auth_request /outpost.goauthentik.io/auth/nginx;
error_page 401 = @goauthentik_proxy_signin;

auth_request_set $auth_cookie $upstream_http_set_cookie;
add_header Set-Cookie $auth_cookie;

auth_request_set $authentik_username $upstream_http_x_authentik_username;
auth_request_set $authentik_groups $upstream_http_x_authentik_groups;
auth_request_set $authentik_email $upstream_http_x_authentik_email;
auth_request_set $authentik_name $upstream_http_x_authentik_name;
auth_request_set $authentik_uid $upstream_http_x_authentik_uid;

proxy_set_header X-authentik-username $authentik_username;
proxy_set_header X-authentik-groups $authentik_groups;
proxy_set_header X-authentik-email $authentik_email;
proxy_set_header X-authentik-name $authentik_name;
proxy_set_header X-authentik-uid $authentik_uid;

location /outpost.goauthentik.io {
    proxy_pass http://192.168.0.250:9000/outpost.goauthentik.io;
    proxy_set_header Host $host;
    proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
    add_header Set-Cookie $auth_cookie;
    auth_request_set $auth_cookie $upstream_http_set_cookie;
    proxy_pass_request_body off;
    proxy_set_header Content-Length "";
}

location @goauthentik_proxy_signin {
    internal;
    add_header Set-Cookie $auth_cookie;
    return 302 https://sso.vish.gg/outpost.goauthentik.io/start?rd=$scheme://$http_host$request_uri;
}
```

### Services with Forward Auth Configured

| Domain | Backend | Port |
|--------|---------|------|
| paperless.vish.gg | 192.168.0.250 | 8777 |
| docs.vish.gg | 192.168.0.250 | 8777 |
| actual.vish.gg | 192.168.0.250 | 8304 |
| npm.vish.gg | 192.168.0.250 | 81 |

---

## Troubleshooting

### "InternalError" After OAuth Login

**Root Cause**: Missing scope mappings in Authentik provider.

**Fix**:
1. Admin → Providers → Edit the OAuth2 provider
2. Scroll to "Scopes" section
3. Add: `openid`, `email`, `profile`
4. Save

**Verify**:
```bash
curl https://sso.vish.gg/application/o/<app-slug>/.well-known/openid-configuration | jq '.scopes_supported'
```

### Redirect Loop Between Service and Authentik

**Root Cause**: Forward Auth configured in NPM for a service that uses native OAuth.

**Fix**:
1. NPM → Proxy Hosts → Edit the affected host
2. Go to Advanced tab
3. **Clear all content** from the Advanced Config box
4. Save

### "User not found" or "No email" Errors

**Root Cause**: Missing attribute paths in service config.

**Fix for Grafana**:
```
GF_AUTH_GENERIC_OAUTH_EMAIL_ATTRIBUTE_PATH=email
GF_AUTH_GENERIC_OAUTH_LOGIN_ATTRIBUTE_PATH=preferred_username
```

### OAuth Works But User Gets Wrong Permissions

**Root Cause**: Missing group claim or incorrect role mapping.

**Fix**:
1. Ensure user is in correct Authentik group
2. Verify `groups` scope is included
3. Check role mapping expression in service config

### Can't Access Authentik Admin

**Create recovery token via Portainer or SSH**:

```bash
docker exec -it Authentik-SERVER ak create_recovery_key 10 akadmin
```

This generates a one-time URL valid for 10 minutes.

---

## Recovery Procedures

### Scenario: Complete Authentik Loss

1. **Restore from backup** (if available):
   ```bash
   # Restore PostgreSQL database
   docker exec -i Authentik-DB psql -U authentik authentik < backup.sql
   
   # Restore media files
   rsync -av backup/media/ /volume1/docker/authentik/media/
   ```

2. **Or redeploy from scratch**:
   - Follow this entire guide from [Deploy Authentik](#deploy-authentik)
   - You'll need to reconfigure all OAuth providers
   - Services will need their OAuth credentials updated

### Scenario: Locked Out of Admin

```bash
# Via SSH to Calypso or Portainer exec
docker exec -it Authentik-SERVER ak create_recovery_key 10 akadmin
```

Navigate to the URL it outputs.

### Scenario: Service OAuth Broken After Authentik Rebuild

1. Create new OAuth2 provider in Authentik (same settings)
2. Note new Client ID and Secret
3. Update service configuration with new credentials
4. Restart service
5. Test login

### Scenario: Forward Auth Not Working

1. Verify Authentik outpost is running:
   ```bash
   docker logs Authentik-SERVER | grep -i outpost
   ```

2. Verify outpost includes the application:
   - Admin → Outposts → Edit → Check application is selected

3. Test outpost endpoint:
   ```bash
   curl -I http://192.168.0.250:9000/outpost.goauthentik.io/ping
   ```

4. Check NPM Advanced Config has correct Authentik IP

---

## Quick Reference

### Authentik Endpoints

| Endpoint | URL |
|----------|-----|
| Admin UI | `https://sso.vish.gg/if/admin/` |
| User Portal | `https://sso.vish.gg/if/user/` |
| Authorization | `https://sso.vish.gg/application/o/authorize/` |
| Token | `https://sso.vish.gg/application/o/token/` |
| User Info | `https://sso.vish.gg/application/o/userinfo/` |
| OpenID Config | `https://sso.vish.gg/application/o/<slug>/.well-known/openid-configuration` |
| End Session | `https://sso.vish.gg/application/o/<slug>/end-session/` |

### Service Status Checklist

After rebuilding, verify each service:

```bash
# OAuth2 Services
curl -sI https://gf.vish.gg | head -1      # Should be 302
curl -sI https://git.vish.gg | head -1     # Should be 200
curl -sI https://sf.vish.gg | head -1      # Should be 302

# Forward Auth Services
curl -sI https://paperless.vish.gg | head -1  # Should be 302 to SSO
curl -sI https://actual.vish.gg | head -1     # Should be 302 to SSO

# Authentik itself
curl -sI https://sso.vish.gg | head -1     # Should be 302
```

---

## Change Log

- **2026-01-31**: Initial creation based on live rebuild/verification session
- **2026-01-31**: Documented scope mappings fix (critical for OAuth2)
- **2026-01-31**: Added NPM Forward Auth vs OAuth2 distinction
- **2026-01-31**: Added all service-specific configurations