Creating Custom systemd Services in AlmaLinux

Introduction

Systemd has become the standard init system for modern Linux distributions, including AlmaLinux. It provides powerful capabilities for managing services, handling dependencies, and automating system tasks. This comprehensive guide explores creating custom systemd services, from basic unit files to advanced configurations with timers, dependencies, and security features.

Understanding systemd Architecture

Core Concepts

Systemd operates on several fundamental principles:

1. Units: Basic building blocks (services, sockets, timers, etc.)
2. Targets: Groups of units (similar to runlevels)
3. Dependencies: Relationships between units
4. Journal: Centralized logging system
5. Control Groups: Resource management
6. Socket Activation: On-demand service starting

Unit Types

# List all unit types
systemctl -t help

# Common unit types:
# .service - System services
# .socket  - Socket-based activation
# .timer   - Timer-based activation
# .mount   - Mount points
# .path    - Path-based activation
# .target  - Grouping of units
# .device  - Device units
# .scope   - External processes
# .slice   - Resource management

Basic Service Creation

Simple Service Example

# Create a simple application
cat > /usr/local/bin/myapp.sh << 'EOF'
#!/bin/bash
# Simple application for demonstration

LOG_FILE="/var/log/myapp.log"

log_message() {
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" >> "$LOG_FILE"
}

log_message "MyApp starting..."

# Main loop
while true; do
    log_message "MyApp is running - PID: $$"
    sleep 60
done
EOF

chmod +x /usr/local/bin/myapp.sh

# Create systemd service unit
cat > /etc/systemd/system/myapp.service << 'EOF'
[Unit]
Description=My Custom Application
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/myapp.sh
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

Service Management

# Reload systemd configuration
systemctl daemon-reload

# Start the service
systemctl start myapp.service

# Enable service at boot
systemctl enable myapp.service

# Check service status
systemctl status myapp.service

# View service logs
journalctl -u myapp.service -f

Advanced Service Configuration

Comprehensive Service Unit

# /etc/systemd/system/advanced-app.service
[Unit]
Description=Advanced Application Service
Documentation=https://example.com/docs
Documentation=man:advanced-app(8)
After=network-online.target postgresql.service
Wants=network-online.target
Requires=postgresql.service
Before=nginx.service
Conflicts=maintenance.target
ConditionPathExists=/etc/advanced-app/config.conf
ConditionMemory=>1G

[Service]
Type=notify
NotifyAccess=main
PIDFile=/run/advanced-app.pid
ExecStartPre=/usr/local/bin/advanced-app-check.sh
ExecStart=/usr/local/bin/advanced-app --config /etc/advanced-app/config.conf
ExecReload=/bin/kill -HUP $MAINPID
ExecStop=/usr/local/bin/advanced-app --shutdown
ExecStopPost=/usr/local/bin/advanced-app-cleanup.sh
TimeoutStartSec=120
TimeoutStopSec=60
Restart=on-failure
RestartSec=30
RestartPreventExitStatus=SIGTERM
SuccessExitStatus=0 2
RemainAfterExit=no
GuessMainPID=yes

# Resource Limits
LimitNOFILE=65536
LimitNPROC=4096
LimitCORE=infinity
TasksMax=512
CPUWeight=100
MemoryLimit=2G
IOWeight=50

# Security
User=appuser
Group=appgroup
DynamicUser=no
PrivateTmp=yes
PrivateDevices=yes
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/var/lib/advanced-app /var/log/advanced-app
NoNewPrivileges=yes
RestrictSUIDSGID=yes
RemoveIPC=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
RestrictRealtime=yes
RestrictNamespaces=yes
SystemCallFilter=@system-service
SystemCallErrorNumber=EPERM

# Environment
Environment="NODE_ENV=production"
Environment="APP_PORT=8080"
EnvironmentFile=-/etc/advanced-app/environment
WorkingDirectory=/var/lib/advanced-app
StandardOutput=journal
StandardError=journal
SyslogIdentifier=advanced-app

[Install]
WantedBy=multi-user.target
Alias=app.service
Also=advanced-app-worker.service

Service Types Explained

# Create examples for different service types

# Type=simple (default)
cat > /etc/systemd/system/simple-example.service << 'EOF'
[Unit]
Description=Simple Service Example

[Service]
Type=simple
ExecStart=/usr/bin/python3 /opt/app/server.py
# Process specified by ExecStart is main process

[Install]
WantedBy=multi-user.target
EOF

# Type=forking
cat > /etc/systemd/system/forking-example.service << 'EOF'
[Unit]
Description=Forking Service Example

[Service]
Type=forking
PIDFile=/var/run/myapp.pid
ExecStart=/usr/local/bin/myapp --daemonize
# Process forks and parent exits

[Install]
WantedBy=multi-user.target
EOF

# Type=oneshot
cat > /etc/systemd/system/oneshot-example.service << 'EOF'
[Unit]
Description=Oneshot Service Example

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/local/bin/setup-script.sh
ExecStop=/usr/local/bin/cleanup-script.sh
# Runs once and exits

[Install]
WantedBy=multi-user.target
EOF

# Type=notify
cat > /etc/systemd/system/notify-example.service << 'EOF'
[Unit]
Description=Notify Service Example

[Service]
Type=notify
NotifyAccess=main
ExecStart=/usr/local/bin/notify-app
# Service sends notification when ready

[Install]
WantedBy=multi-user.target
EOF

# Type=dbus
cat > /etc/systemd/system/dbus-example.service << 'EOF'
[Unit]
Description=D-Bus Service Example

[Service]
Type=dbus
BusName=com.example.MyApp
ExecStart=/usr/local/bin/dbus-app
# Service acquires D-Bus name when ready

[Install]
WantedBy=multi-user.target
EOF

Working with Dependencies

Dependency Configuration

# Create interdependent services
# Service A (Database)
cat > /etc/systemd/system/app-database.service << 'EOF'
[Unit]
Description=Application Database
Before=app-backend.service

[Service]
Type=simple
ExecStart=/usr/local/bin/database-server
Restart=always

[Install]
WantedBy=multi-user.target
EOF

# Service B (Backend)
cat > /etc/systemd/system/app-backend.service << 'EOF'
[Unit]
Description=Application Backend
After=app-database.service
Requires=app-database.service
Before=app-frontend.service

[Service]
Type=simple
ExecStart=/usr/local/bin/backend-server
Restart=always

[Install]
WantedBy=multi-user.target
EOF

# Service C (Frontend)
cat > /etc/systemd/system/app-frontend.service << 'EOF'
[Unit]
Description=Application Frontend
After=app-backend.service
Wants=app-backend.service

[Service]
Type=simple
ExecStart=/usr/local/bin/frontend-server
Restart=always

[Install]
WantedBy=multi-user.target
EOF

Dependency Types

# Strong Dependencies
Requires=postgresql.service
# If postgresql fails, this service fails

Requisite=postgresql.service
# Like Requires, but won't start postgresql

BindsTo=postgresql.service
# Stronger than Requires, stops when postgresql stops

# Weak Dependencies
Wants=redis.service
# Try to start redis, but continue if it fails

# Ordering Dependencies
After=network-online.target
# Start after network is online

Before=nginx.service
# Start before nginx

# Negative Dependencies
Conflicts=maintenance.service
# Cannot run at same time as maintenance

# Conditional Dependencies
ConditionPathExists=/etc/myapp/config.conf
ConditionPathIsDirectory=/var/lib/myapp
ConditionFileNotEmpty=/etc/myapp/license.key
ConditionMemory=>2G
ConditionCPUs=>2

Timer Units

Creating Timer-Based Services

# Create a backup service
cat > /usr/local/bin/backup-script.sh << 'EOF'
#!/bin/bash
BACKUP_DIR="/backup/system"
DATE=$(date +%Y%m%d_%H%M%S)

echo "Starting backup at $(date)"
mkdir -p "$BACKUP_DIR"

# Backup important directories
tar -czf "$BACKUP_DIR/etc_$DATE.tar.gz" /etc
tar -czf "$BACKUP_DIR/var_log_$DATE.tar.gz" /var/log

# Cleanup old backups (keep last 7 days)
find "$BACKUP_DIR" -name "*.tar.gz" -mtime +7 -delete

echo "Backup completed at $(date)"
EOF

chmod +x /usr/local/bin/backup-script.sh

# Create service unit
cat > /etc/systemd/system/system-backup.service << 'EOF'
[Unit]
Description=System Backup Service
ConditionACPower=true

[Service]
Type=oneshot
Nice=19
IOSchedulingClass=idle
ExecStart=/usr/local/bin/backup-script.sh
StandardOutput=journal
StandardError=journal
EOF

# Create timer unit
cat > /etc/systemd/system/system-backup.timer << 'EOF'
[Unit]
Description=Run System Backup daily
Requires=system-backup.service

[Timer]
OnCalendar=daily
OnCalendar=*-*-* 02:00:00
AccuracySec=1h
Persistent=true
WakeSystem=false
RandomizedDelaySec=30min

[Install]
WantedBy=timers.target
EOF

Advanced Timer Configuration

# Complex timer example
[Timer]
# Activation times
OnActiveSec=30min          # 30 minutes after timer activation
OnBootSec=10min           # 10 minutes after boot
OnStartupSec=15min        # 15 minutes after systemd startup
OnUnitActiveSec=1h        # 1 hour after unit was last activated
OnUnitInactiveSec=2h      # 2 hours after unit was last deactivated

# Calendar-based activation
OnCalendar=weekly         # Every Monday at 00:00
OnCalendar=mon..fri *-*-* 09:00:00  # Weekdays at 9 AM
OnCalendar=*-*-15 00:00:00          # 15th of every month
OnCalendar=quarterly                 # First day of each quarter
OnCalendar=*:0/15                   # Every 15 minutes

# Timer properties
AccuracySec=1min          # Timer accuracy
RandomizedDelaySec=5min   # Random delay to prevent thundering herd
Persistent=true           # Catch up missed runs
WakeSystem=true          # Wake system from suspend
RemainAfterElapse=yes    # Keep timer active after elapse

Socket Activation

Socket-Activated Service

# Create socket unit
cat > /etc/systemd/system/myapp.socket << 'EOF'
[Unit]
Description=MyApp Socket
PartOf=myapp.service

[Socket]
ListenStream=8080
ListenStream=/run/myapp.sock
Accept=no
Backlog=128
SocketMode=0660
SocketUser=appuser
SocketGroup=appgroup
NoDelay=true
ReusePort=true
KeepAlive=yes
KeepAliveTimeSec=60
MaxConnections=256

[Install]
WantedBy=sockets.target
EOF

# Create corresponding service
cat > /etc/systemd/system/myapp.service << 'EOF'
[Unit]
Description=MyApp Service
Requires=myapp.socket
After=myapp.socket

[Service]
Type=simple
ExecStart=/usr/local/bin/myapp
StandardInput=socket
StandardOutput=journal
StandardError=journal
User=appuser
Group=appgroup

[Install]
Also=myapp.socket
WantedBy=multi-user.target
EOF

Socket Types

# Stream socket (TCP)
ListenStream=8080
ListenStream=0.0.0.0:8080
ListenStream=[::]:8080

# Datagram socket (UDP)
ListenDatagram=8081

# Sequential packet socket
ListenSequentialPacket=/run/app.sock

# FIFO
ListenFIFO=/run/app.fifo

# Special socket
ListenSpecial=/dev/input/event0

# Netlink socket
ListenNetlink=route

# Message queue
ListenMessageQueue=/queue

Template Units

Creating Template Services

# Create template unit
cat > /etc/systemd/system/[email protected] << 'EOF'
[Unit]
Description=Worker Instance %i
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/worker --id %i --config /etc/worker/%i.conf
Restart=always
RestartSec=10
User=worker
Group=worker

# Instance-specific settings
Environment="WORKER_ID=%i"
Environment="WORKER_PORT=80%i"
WorkingDirectory=/var/lib/worker/%i
StandardOutput=journal
StandardError=journal
SyslogIdentifier=worker-%i

[Install]
WantedBy=multi-user.target
EOF

# Create worker script
cat > /usr/local/bin/worker << 'EOF'
#!/bin/bash
WORKER_ID=""
CONFIG_FILE=""

while [[ $# -gt 0 ]]; do
    case $1 in
        --id) WORKER_ID="$2"; shift 2 ;;
        --config) CONFIG_FILE="$2"; shift 2 ;;
        *) shift ;;
    esac
done

echo "Worker $WORKER_ID starting with config $CONFIG_FILE"
while true; do
    echo "Worker $WORKER_ID processing..."
    sleep 30
done
EOF

chmod +x /usr/local/bin/worker

# Create configurations
mkdir -p /etc/worker /var/lib/worker/{01,02,03}
echo "threads=4" > /etc/worker/01.conf
echo "threads=2" > /etc/worker/02.conf
echo "threads=8" > /etc/worker/03.conf

# Enable instances
systemctl daemon-reload
systemctl enable worker@{01,02,03}.service
systemctl start worker@{01,02,03}.service

Security Hardening

Secure Service Configuration

# Comprehensive security settings
[Service]
# User/Group
User=appuser
Group=appgroup
DynamicUser=yes
SupplementaryGroups=audio video

# Filesystem Protection
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
PrivateDevices=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
ReadOnlyPaths=/
ReadWritePaths=/var/lib/app /var/log/app
InaccessiblePaths=/home /root
TemporaryFileSystem=/var:ro
BindPaths=/var/lib/app:/data
BindReadOnlyPaths=/etc/app:/config:rbind

# Process Restrictions
NoNewPrivileges=yes
RestrictSUIDSGID=yes
RemoveIPC=yes
PrivateMounts=yes
MountFlags=slave

# Network Restrictions
PrivateNetwork=no
RestrictAddressFamilies=AF_INET AF_INET6
IPAddressDeny=any
IPAddressAllow=192.168.1.0/24

# Resource Restrictions
LimitNOFILE=1024
LimitNPROC=512
LimitCORE=0
TasksMax=100
CPUQuota=50%
MemoryMax=1G
IOReadBandwidthMax=/dev/sda 10M
IOWriteBandwidthMax=/dev/sda 5M

# Capability Restrictions
CapabilityBoundingSet=CAP_NET_BIND_SERVICE CAP_DAC_READ_SEARCH
AmbientCapabilities=CAP_NET_BIND_SERVICE

# System Call Filtering
SystemCallFilter=@system-service
SystemCallFilter=~@privileged @resources
SystemCallErrorNumber=EPERM
SystemCallArchitectures=native

# Namespace Isolation
PrivateUsers=yes
RestrictNamespaces=yes
ProcSubset=pid
ProtectProc=invisible
ProtectClock=yes
ProtectHostname=yes
LockPersonality=yes
RestrictRealtime=yes

# Execution Environment
UMask=0077
KeyringMode=private
ProtectKernelLogs=yes

SELinux Integration

# Create SELinux policy for custom service
cat > myapp.te << 'EOF'
policy_module(myapp, 1.0.0)

require {
    type init_t;
    type unconfined_t;
    class process transition;
}

# Define myapp domain
type myapp_t;
type myapp_exec_t;
init_daemon_domain(myapp_t, myapp_exec_t)

# Define myapp files
type myapp_etc_t;
files_config_file(myapp_etc_t)

type myapp_var_lib_t;
files_type(myapp_var_lib_t)

type myapp_log_t;
logging_log_file(myapp_log_t)

# Allow myapp to read config
allow myapp_t myapp_etc_t:file read_file_perms;
allow myapp_t myapp_etc_t:dir list_dir_perms;

# Allow myapp to write data
allow myapp_t myapp_var_lib_t:file manage_file_perms;
allow myapp_t myapp_var_lib_t:dir manage_dir_perms;

# Allow myapp to write logs
allow myapp_t myapp_log_t:file { create_file_perms append_file_perms };
allow myapp_t myapp_log_t:dir { add_entry_dir_perms };
EOF

# Compile and install policy
checkmodule -M -m -o myapp.mod myapp.te
semodule_package -o myapp.pp -m myapp.mod
semodule -i myapp.pp

# Label files
semanage fcontext -a -t myapp_exec_t '/usr/local/bin/myapp'
semanage fcontext -a -t myapp_etc_t '/etc/myapp(/.*)?'
semanage fcontext -a -t myapp_var_lib_t '/var/lib/myapp(/.*)?'
semanage fcontext -a -t myapp_log_t '/var/log/myapp(/.*)?'
restorecon -Rv /usr/local/bin/myapp /etc/myapp /var/lib/myapp /var/log/myapp

Resource Management

Control Groups Configuration

# Create resource-limited service
cat > /etc/systemd/system/resource-limited.service << 'EOF'
[Unit]
Description=Resource Limited Service
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/heavy-app

# CPU Limits
CPUAccounting=yes
CPUWeight=50
CPUQuota=200%
CPUQuotaPeriodSec=100ms
AllowedCPUs=0-3

# Memory Limits
MemoryAccounting=yes
MemoryMax=2G
MemoryHigh=1500M
MemoryLow=500M
MemorySwapMax=512M

# IO Limits
IOAccounting=yes
IOWeight=10
IOReadBandwidthMax=/dev/sda 10M
IOWriteBandwidthMax=/dev/sda 5M
IOReadIOPSMax=/dev/sda 1000
IOWriteIOPSMax=/dev/sda 1000

# Task Limits
TasksAccounting=yes
TasksMax=256

# Other Limits
IPAccounting=yes
IPAddressAllow=192.168.1.0/24
IPAddressDeny=any

[Install]
WantedBy=multi-user.target
EOF

Slice Units for Resource Hierarchy

# Create custom slice
cat > /etc/systemd/system/mycompany.slice << 'EOF'
[Unit]
Description=MyCompany Services Slice
Before=slices.target

[Slice]
CPUWeight=100
MemoryMax=8G
TasksMax=4096
EOF

# Create sub-slice
cat > /etc/systemd/system/mycompany-production.slice << 'EOF'
[Unit]
Description=Production Services
Before=slices.target

[Slice]
Slice=mycompany.slice
CPUWeight=80
MemoryMax=6G
EOF

# Assign service to slice
cat > /etc/systemd/system/production-app.service << 'EOF'
[Unit]
Description=Production Application

[Service]
Slice=mycompany-production.slice
Type=simple
ExecStart=/usr/local/bin/prod-app

[Install]
WantedBy=multi-user.target
EOF

Monitoring and Debugging

Service Monitoring

# Create monitoring wrapper
cat > /usr/local/bin/service-monitor.sh << 'EOF'
#!/bin/bash
SERVICE_NAME="$1"
WEBHOOK_URL="https://monitoring.example.com/webhook"

send_alert() {
    local status=$1
    local message=$2
    curl -X POST "$WEBHOOK_URL" \
        -H "Content-Type: application/json" \
        -d "{\"service\":\"$SERVICE_NAME\",\"status\":\"$status\",\"message\":\"$message\"}"
}

# Monitor service status
while true; do
    if ! systemctl is-active --quiet "$SERVICE_NAME"; then
        send_alert "down" "Service $SERVICE_NAME is not running"
        systemctl restart "$SERVICE_NAME"
        
        sleep 10
        
        if systemctl is-active --quiet "$SERVICE_NAME"; then
            send_alert "recovered" "Service $SERVICE_NAME has been restarted"
        else
            send_alert "critical" "Service $SERVICE_NAME failed to restart"
        fi
    fi
    
    sleep 60
done
EOF

chmod +x /usr/local/bin/service-monitor.sh

Debugging Tools

# Service debugging commands

# Detailed service status
systemctl status -l myapp.service

# Show service configuration
systemctl show myapp.service

# List dependencies
systemctl list-dependencies myapp.service

# Show service environment
systemctl show-environment

# Analyze boot time
systemd-analyze
systemd-analyze blame
systemd-analyze critical-chain myapp.service

# View service logs
journalctl -u myapp.service --since "1 hour ago"
journalctl -u myapp.service -p err
journalctl -u myapp.service -f

# Debug service startup
systemd-analyze verify myapp.service

# Check for masked services
systemctl list-unit-files --state=masked

Integration Examples

Database Service with Backup

# PostgreSQL service with automated backup
cat > /etc/systemd/system/postgresql-backup.service << 'EOF'
[Unit]
Description=PostgreSQL Backup
Requires=postgresql.service
After=postgresql.service

[Service]
Type=oneshot
User=postgres
ExecStart=/usr/local/bin/pg-backup.sh
StandardOutput=journal
StandardError=journal
EOF

cat > /etc/systemd/system/postgresql-backup.timer << 'EOF'
[Unit]
Description=PostgreSQL Backup Timer
Requires=postgresql-backup.service

[Timer]
OnCalendar=daily
OnCalendar=*-*-* 03:00:00
Persistent=true

[Install]
WantedBy=timers.target
EOF

cat > /usr/local/bin/pg-backup.sh << 'EOF'
#!/bin/bash
BACKUP_DIR="/backup/postgresql"
DATE=$(date +%Y%m%d_%H%M%S)
RETENTION_DAYS=7

mkdir -p "$BACKUP_DIR"

# Backup all databases
pg_dumpall | gzip > "$BACKUP_DIR/all_databases_$DATE.sql.gz"

# Cleanup old backups
find "$BACKUP_DIR" -name "*.sql.gz" -mtime +$RETENTION_DAYS -delete

# Verify backup
if [[ -f "$BACKUP_DIR/all_databases_$DATE.sql.gz" ]]; then
    echo "Backup successful: all_databases_$DATE.sql.gz"
else
    echo "Backup failed!" >&2
    exit 1
fi
EOF

chmod +x /usr/local/bin/pg-backup.sh

Web Application Stack

# Complete web application service set

# 1. Database service
cat > /etc/systemd/system/webapp-db.service << 'EOF'
[Unit]
Description=WebApp Database
After=network.target

[Service]
Type=notify
User=postgres
ExecStart=/usr/pgsql-14/bin/postgres -D /var/lib/pgsql/14/data
Restart=always

[Install]
WantedBy=multi-user.target
EOF

# 2. Redis cache service
cat > /etc/systemd/system/webapp-cache.service << 'EOF'
[Unit]
Description=WebApp Redis Cache
After=network.target

[Service]
Type=notify
ExecStart=/usr/bin/redis-server /etc/redis/webapp.conf
Restart=always
User=redis
Group=redis

[Install]
WantedBy=multi-user.target
EOF

# 3. Application service
cat > /etc/systemd/system/webapp.service << 'EOF'
[Unit]
Description=WebApp Application Server
After=webapp-db.service webapp-cache.service
Requires=webapp-db.service
Wants=webapp-cache.service

[Service]
Type=simple
User=webapp
Group=webapp
WorkingDirectory=/opt/webapp
Environment="NODE_ENV=production"
ExecStart=/usr/bin/node /opt/webapp/server.js
Restart=always
RestartSec=10

# Security
PrivateTmp=yes
NoNewPrivileges=yes
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/opt/webapp/uploads /var/log/webapp

[Install]
WantedBy=multi-user.target
EOF

# 4. Nginx frontend
cat > /etc/systemd/system/webapp-nginx.service << 'EOF'
[Unit]
Description=WebApp Nginx Frontend
After=webapp.service
Requires=webapp.service

[Service]
Type=forking
PIDFile=/run/nginx-webapp.pid
ExecStartPre=/usr/sbin/nginx -t -c /etc/nginx/webapp.conf
ExecStart=/usr/sbin/nginx -c /etc/nginx/webapp.conf
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s QUIT $MAINPID
Restart=always

[Install]
WantedBy=multi-user.target
EOF

# 5. Target unit to group all services
cat > /etc/systemd/system/webapp.target << 'EOF'
[Unit]
Description=WebApp Full Stack
Requires=webapp-db.service webapp.service webapp-nginx.service
Wants=webapp-cache.service
After=webapp-db.service webapp-cache.service webapp.service

[Install]
WantedBy=multi-user.target
EOF

Best Practices

Service Design Guidelines

# Service template with best practices
cat > /etc/systemd/system/best-practice.service << 'EOF'
[Unit]
Description=Best Practice Service Example
Documentation=https://example.com/docs
Documentation=man:best-practice(8)

# Dependencies
After=network-online.target
Wants=network-online.target

# Conditions
ConditionPathExists=/etc/best-practice/config.yml
AssertPathIsDirectory=/var/lib/best-practice

[Service]
# Service type
Type=notify
NotifyAccess=main

# Execution
ExecStartPre=/usr/local/bin/best-practice --check-config
ExecStart=/usr/local/bin/best-practice --daemon
ExecReload=/bin/kill -USR1 $MAINPID
ExecStop=/usr/local/bin/best-practice --shutdown

# Restart policy
Restart=on-failure
RestartSec=30s
RestartPreventExitStatus=0 1

# Timeouts
TimeoutStartSec=90s
TimeoutStopSec=90s
WatchdogSec=60s

# User/Group
User=svcuser
Group=svcgroup

# Security hardening
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
ReadWritePaths=/var/lib/best-practice /var/log/best-practice

# Resource limits
MemoryMax=1G
CPUQuota=50%
TasksMax=100

# Logging
StandardOutput=journal
StandardError=journal
SyslogIdentifier=best-practice

[Install]
WantedBy=multi-user.target
EOF

Service Checklist

# Create service validation script
cat > /usr/local/bin/validate-service.sh << 'EOF'
#!/bin/bash
SERVICE_FILE="$1"

if [[ ! -f "$SERVICE_FILE" ]]; then
    echo "Usage: $0 <service-file>"
    exit 1
fi

echo "Validating service file: $SERVICE_FILE"
echo "================================"

# Syntax check
systemd-analyze verify "$SERVICE_FILE"

# Security analysis
systemd-analyze security "$SERVICE_FILE"

# Check for common issues
echo -e "\nChecking best practices:"

# Check for description
grep -q "^Description=" "$SERVICE_FILE" || echo "⚠ Missing Description"

# Check for documentation
grep -q "^Documentation=" "$SERVICE_FILE" || echo "⚠ Missing Documentation"

# Check for restart policy
grep -q "^Restart=" "$SERVICE_FILE" || echo "⚠ Missing Restart policy"

# Check for user/group
grep -q "^User=" "$SERVICE_FILE" || echo "⚠ Running as root (no User specified)"

# Check for security settings
grep -q "NoNewPrivileges=yes" "$SERVICE_FILE" || echo "⚠ Consider adding NoNewPrivileges=yes"
grep -q "PrivateTmp=yes" "$SERVICE_FILE" || echo "⚠ Consider adding PrivateTmp=yes"

echo -e "\nValidation complete!"
EOF

chmod +x /usr/local/bin/validate-service.sh

Troubleshooting

Common Issues and Solutions

# Troubleshooting script
cat > /usr/local/bin/service-troubleshoot.sh << 'EOF'
#!/bin/bash
SERVICE="$1"

if [[ -z "$SERVICE" ]]; then
    echo "Usage: $0 <service-name>"
    exit 1
fi

echo "Troubleshooting service: $SERVICE"
echo "================================="

# Check if service exists
if ! systemctl list-unit-files | grep -q "^$SERVICE"; then
    echo "ERROR: Service $SERVICE not found"
    exit 1
fi

# Check service status
echo -e "\n1. Service Status:"
systemctl status "$SERVICE" --no-pager

# Check for failed state
if systemctl is-failed "$SERVICE" >/dev/null 2>&1; then
    echo -e "\n2. Service is in failed state. Recent logs:"
    journalctl -u "$SERVICE" -n 20 --no-pager
fi

# Check dependencies
echo -e "\n3. Dependencies:"
systemctl list-dependencies "$SERVICE" --no-pager

# Check for missing dependencies
echo -e "\n4. Checking for missing dependencies:"
systemctl list-dependencies "$SERVICE" | grep -E "●|○" | grep -v "$SERVICE"

# Check configuration
echo -e "\n5. Service configuration:"
systemctl show "$SERVICE" --no-pager | grep -E "ExecStart=|User=|WorkingDirectory=|Environment="

# Check for resource limits
echo -e "\n6. Resource limits:"
systemctl show "$SERVICE" --no-pager | grep -E "Limit|Memory|CPU|Tasks"

# Check SELinux denials
if command -v getenforce >/dev/null 2>&1 && [[ $(getenforce) != "Disabled" ]]; then
    echo -e "\n7. Recent SELinux denials:"
    ausearch -m avc -ts recent 2>/dev/null | grep "$SERVICE" | tail -5
fi

# Suggest fixes
echo -e "\n8. Common fixes to try:"
echo "  - systemctl daemon-reload"
echo "  - systemctl reset-failed $SERVICE"
echo "  - Check file permissions and SELinux contexts"
echo "  - Verify paths in ExecStart exist"
echo "  - Check journalctl -xe for detailed errors"
EOF

chmod +x /usr/local/bin/service-troubleshoot.sh

Conclusion

Creating custom systemd services in AlmaLinux provides powerful capabilities for system automation and service management. By mastering unit file configuration, dependencies, timers, and security features, administrators can build robust, secure, and efficient services.

Key takeaways:

• Understand different service types and their use cases
• Implement proper dependency management
• Use security hardening features
• Leverage timers for scheduled tasks
• Monitor and debug services effectively
• Follow best practices for production services

With systemd’s comprehensive feature set, you can create sophisticated service configurations that meet enterprise requirements while maintaining security and reliability.