Managing System Services with systemctl in AlmaLinux

Systemd is the init system and service manager for AlmaLinux, providing powerful tools for managing system services, dependencies, and the boot process. The systemctl command is your primary interface to systemd, offering comprehensive control over services, targets, and system states. This guide provides a deep dive into managing system services with systemctl, from basic operations to advanced configurations.

Understanding systemd and Services

What is systemd?

systemd is a suite of system management daemons, libraries, and utilities designed to centralize the management and configuration of Linux systems. It replaces the traditional SysV init system with a more powerful and flexible approach.

Key Concepts

• Units: Basic objects that systemd manages (services, sockets, devices, mounts, etc.)
• Services: Processes managed by systemd (.service units)
• Targets: Groups of units that define system states (similar to runlevels)
• Dependencies: Relationships between units
• Journal: Centralized logging system

Unit Types

# List all unit types
systemctl -t help

# Common unit types:
# .service - System services
# .socket - Network sockets
# .target - Unit groups
# .mount - Mount points
# .timer - Systemd timers
# .path - Path monitoring
# .device - Device units
# .scope - External processes
# .slice - Resource management

Basic systemctl Commands

System State Commands

# Check systemd system state
systemctl is-system-running

# Show system status
systemctl status

# List failed units
systemctl --failed

# Reset failed units
systemctl reset-failed

# Reload systemd manager configuration
systemctl daemon-reload

# Reexecute systemd manager
systemctl daemon-reexec

Listing Units

# List all loaded units
systemctl list-units

# List all units (including inactive)
systemctl list-units --all

# List specific unit type
systemctl list-units --type=service

# List all installed unit files
systemctl list-unit-files

# List services by state
systemctl list-units --type=service --state=running
systemctl list-units --type=service --state=failed

# Show unit dependencies
systemctl list-dependencies nginx.service

Getting Help

# Show systemctl help
systemctl --help

# Show help for specific command
systemctl help start

# Show man page for unit
systemctl help nginx.service

# Show systemd documentation
man systemd
man systemd.service
man systemd.unit

Service Management Operations

Starting and Stopping Services

# Start a service
sudo systemctl start nginx.service
# Note: .service extension is optional
sudo systemctl start nginx

# Stop a service
sudo systemctl stop nginx

# Restart a service
sudo systemctl restart nginx

# Reload service configuration
sudo systemctl reload nginx

# Reload or restart (reload if supported, otherwise restart)
sudo systemctl reload-or-restart nginx

# Try to restart (restart only if already running)
sudo systemctl try-restart nginx

Enabling and Disabling Services

# Enable service to start at boot
sudo systemctl enable nginx

# Disable service from starting at boot
sudo systemctl disable nginx

# Enable and start in one command
sudo systemctl enable --now nginx

# Disable and stop in one command
sudo systemctl disable --now nginx

# Check if service is enabled
systemctl is-enabled nginx

# Re-enable service (disable and enable)
sudo systemctl reenable nginx

Service Status and Information

# Show service status
systemctl status nginx

# Show full status (no ellipsis)
systemctl status nginx --full

# Show status without pager
systemctl status nginx --no-pager

# Check if service is active
systemctl is-active nginx

# Check if service failed
systemctl is-failed nginx

# Show service properties
systemctl show nginx

# Show specific property
systemctl show nginx -p MainPID
systemctl show nginx -p ActiveState,SubState

# Show service environment
systemctl show-environment

Service Control Options

# Send signal to service
sudo systemctl kill -s HUP nginx

# Set service properties temporarily
sudo systemctl set-property nginx CPUQuota=50%

# Mask service (prevent starting)
sudo systemctl mask nginx

# Unmask service
sudo systemctl unmask nginx

# Isolate target (start target and stop others)
sudo systemctl isolate multi-user.target

Working with Unit Files

Unit File Locations

# System unit files (do not edit)
/usr/lib/systemd/system/

# Local unit files (for customization)
/etc/systemd/system/

# Runtime unit files
/run/systemd/system/

# User unit files
~/.config/systemd/user/
/usr/lib/systemd/user/

Viewing Unit Files

# View unit file
systemctl cat nginx.service

# Edit unit file (creates override)
sudo systemctl edit nginx.service

# Edit full unit file (not recommended)
sudo systemctl edit --full nginx.service

# Show unit file path
systemctl show nginx.service -p FragmentPath

# List unit file properties
systemctl show nginx.service

Basic Service Unit File Structure

[Unit]
Description=The nginx HTTP and reverse proxy server
Documentation=man:nginx(8)
After=network-online.target remote-fs.target nss-lookup.target
Wants=network-online.target

[Service]
Type=forking
PIDFile=/run/nginx.pid
ExecStartPre=/usr/bin/rm -f /run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t
ExecStart=/usr/sbin/nginx
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s TERM $MAINPID
KillSignal=SIGQUIT
TimeoutStopSec=5
KillMode=mixed
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Service Types

# Simple (default) - process doesn't fork
Type=simple

# Forking - process forks and parent exits
Type=forking

# Oneshot - process exits after start
Type=oneshot

# Dbus - service acquires D-Bus name
Type=dbus

# Notify - service sends notification when ready
Type=notify

# Idle - service starts after all jobs dispatched
Type=idle

Service Dependencies and Ordering

Dependency Directives

[Unit]
# Strong dependencies (service fails if dependency fails)
Requires=network.target
Requisite=postgresql.service

# Weak dependencies (service continues if dependency fails)
Wants=redis.service
After=network.target postgresql.service
Before=nginx.service

# Conflict dependencies
Conflicts=apache2.service

# Binding dependencies
BindsTo=device.mount
PartOf=graphical.target

Viewing Dependencies

# Show dependencies
systemctl list-dependencies nginx

# Show reverse dependencies
systemctl list-dependencies nginx --reverse

# Show full dependency tree
systemctl list-dependencies nginx --all

# Show only specific dependency type
systemctl show nginx -p Wants
systemctl show nginx -p Requires
systemctl show nginx -p After
systemctl show nginx -p Before

Creating Service Dependencies

# Create dependency directory
sudo mkdir -p /etc/systemd/system/nginx.service.wants/

# Add dependency via symlink
sudo ln -s /usr/lib/systemd/system/php-fpm.service \
    /etc/systemd/system/nginx.service.wants/

# Or use systemctl
sudo systemctl add-wants nginx.service php-fpm.service
sudo systemctl add-requires nginx.service mariadb.service

System Targets and Runlevels

Understanding Targets

# List all targets
systemctl list-units --type=target

# Show current default target
systemctl get-default

# Set default target
sudo systemctl set-default multi-user.target

# Common targets:
# poweroff.target    - System shutdown
# rescue.target      - Single user mode
# multi-user.target  - Multi-user, non-graphical
# graphical.target   - Multi-user, graphical
# reboot.target      - System reboot
# emergency.target   - Emergency shell

Changing System State

# Switch to different target
sudo systemctl isolate multi-user.target

# Rescue mode
sudo systemctl rescue

# Emergency mode
sudo systemctl emergency

# Reboot system
sudo systemctl reboot

# Power off system
sudo systemctl poweroff

# Halt system
sudo systemctl halt

# Suspend system
sudo systemctl suspend

# Hibernate system
sudo systemctl hibernate

Target Dependencies

# Show what target pulls in
systemctl list-dependencies graphical.target

# Show target status
systemctl status graphical.target

# Check if target is active
systemctl is-active graphical.target

Creating Custom Services

Simple Service Example

Create /etc/systemd/system/myapp.service:

[Unit]
Description=My Custom Application
Documentation=https://example.com/docs
After=network.target

[Service]
Type=simple
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/bin/myapp
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Python Application Service

[Unit]
Description=Python Web Application
After=network.target

[Service]
Type=simple
User=webapp
Group=webapp
WorkingDirectory=/opt/webapp
Environment="PATH=/opt/webapp/venv/bin"
ExecStart=/opt/webapp/venv/bin/python app.py
Restart=on-failure
RestartSec=5s

# Security settings
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/opt/webapp/data

[Install]
WantedBy=multi-user.target

Node.js Application Service

[Unit]
Description=Node.js Application
Documentation=https://example.com
After=network.target

[Service]
Type=simple
User=nodeapp
Group=nodeapp
WorkingDirectory=/opt/nodeapp
ExecStart=/usr/bin/node /opt/nodeapp/server.js
Restart=always
RestartSec=10
StandardOutput=syslog
StandardError=syslog
SyslogIdentifier=nodeapp
Environment=NODE_ENV=production

[Install]
WantedBy=multi-user.target

Service with Pre/Post Scripts

[Unit]
Description=Application with Setup
After=network.target

[Service]
Type=forking
User=app
Group=app
PIDFile=/var/run/app.pid

# Pre-start commands
ExecStartPre=/opt/app/bin/check-config.sh
ExecStartPre=/opt/app/bin/init-database.sh

# Main service
ExecStart=/opt/app/bin/start.sh
ExecStop=/opt/app/bin/stop.sh

# Post-stop cleanup
ExecStopPost=/opt/app/bin/cleanup.sh

# Restart configuration
Restart=on-failure
RestartSec=30s
TimeoutStartSec=300
TimeoutStopSec=30

[Install]
WantedBy=multi-user.target

Service Security and Sandboxing

Security Directives

[Service]
# User/Group settings
User=nobody
Group=nogroup
DynamicUser=yes

# Filesystem protection
ProtectSystem=strict
ProtectHome=yes
PrivateTmp=yes
ReadWritePaths=/var/lib/myapp
ReadOnlyPaths=/etc/myapp
InaccessiblePaths=/home

# Device access
PrivateDevices=yes
DevicePolicy=closed
DeviceAllow=/dev/null rw

# Network isolation
PrivateNetwork=yes
RestrictAddressFamilies=AF_INET AF_INET6

# Capability restrictions
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE
NoNewPrivileges=yes

# System call filtering
SystemCallFilter=@system-service
SystemCallErrorNumber=EPERM

# Resource limits
LimitNOFILE=1024
LimitNPROC=512
MemoryLimit=1G
CPUQuota=50%

Sandboxing Example

[Service]
# Maximum sandboxing
User=myservice
DynamicUser=yes
PrivateDevices=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectControlGroups=yes
RestrictRealtime=yes
RestrictNamespaces=yes
LockPersonality=yes
MemoryDenyWriteExecute=yes
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6
SystemCallFilter=@system-service
SystemCallErrorNumber=EPERM
NoNewPrivileges=yes

Logging and Monitoring

Viewing Service Logs

# View service logs
journalctl -u nginx

# Follow logs in real-time
journalctl -u nginx -f

# Show logs since boot
journalctl -u nginx -b

# Show logs from previous boot
journalctl -u nginx -b -1

# Show logs with timestamp
journalctl -u nginx --since "2024-01-01 00:00:00"
journalctl -u nginx --since "1 hour ago"

# Show specific priority
journalctl -u nginx -p err
journalctl -u nginx -p warning..err

# Export logs
journalctl -u nginx --no-pager > nginx.log
journalctl -u nginx --output=json > nginx.json

Service Monitoring

# Monitor service status changes
systemctl status nginx --follow

# Watch service status
watch -n 1 systemctl status nginx

# Monitor all services
systemctl list-units --type=service --all

# Show service start times
systemd-analyze blame

# Show critical chain
systemd-analyze critical-chain nginx.service

# Plot service dependencies
systemd-analyze dot nginx.service | dot -Tsvg > nginx-deps.svg

Custom Logging Configuration

[Service]
# Logging configuration
StandardOutput=journal
StandardError=journal
SyslogIdentifier=myapp
SyslogFacility=local0
SyslogLevel=info

# Or redirect to file
StandardOutput=file:/var/log/myapp/output.log
StandardError=file:/var/log/myapp/error.log

# Or append to file
StandardOutput=append:/var/log/myapp/output.log
StandardError=append:/var/log/myapp/error.log

Troubleshooting Services

Common Issues and Solutions

Service Fails to Start

# Check status and logs
systemctl status myservice
journalctl -xeu myservice

# Verify unit file syntax
systemd-analyze verify myservice.service

# Check for missing dependencies
systemctl list-dependencies myservice

# Test configuration
ExecStartPre=/usr/bin/myapp --test-config

Service Keeps Restarting

# Check restart settings
systemctl show myservice -p Restart,RestartSec

# View recent logs
journalctl -u myservice --since "10 minutes ago"

# Temporarily disable restart
sudo systemctl edit myservice
# Add:
[Service]
Restart=no

Permission Issues

# Check service user
systemctl show myservice -p User,Group

# Verify file permissions
sudo -u serviceuser ls -la /path/to/files

# Check SELinux context
ls -Z /path/to/files
ausearch -m avc -ts recent

# Test as service user
sudo -u serviceuser /path/to/executable

Debugging Tools

# Analyze boot performance
systemd-analyze
systemd-analyze blame
systemd-analyze critical-chain

# Verify unit files
systemd-analyze verify

# Check service environment
systemctl show-environment
sudo systemctl set-environment VAR=value

# Debug systemd itself
SYSTEMD_LOG_LEVEL=debug systemctl status myservice

# Run service in foreground
sudo -u serviceuser /path/to/executable

Recovery Procedures

# Reset failed state
sudo systemctl reset-failed myservice

# Reload systemd if unit file changed
sudo systemctl daemon-reload

# Force service cleanup
sudo systemctl stop myservice
sudo rm -f /var/run/myservice.pid
sudo systemctl start myservice

# Emergency recovery
sudo systemctl isolate rescue.target
# Fix issues
sudo systemctl default

Performance Optimization

Service Startup Optimization

[Service]
# Parallel startup
Type=simple  # Instead of forking

# Delay service if not critical
Type=idle

# Socket activation
# Move to socket unit for on-demand start

# Resource limits to prevent overload
StartLimitInterval=300
StartLimitBurst=5

# Faster shutdown
TimeoutStopSec=10
KillMode=mixed

Resource Management

[Service]
# CPU limits
CPUQuota=50%
CPUShares=512
AllowedCPUs=0-3

# Memory limits
MemoryMax=1G
MemoryHigh=750M
MemoryLow=500M

# IO limits
IOWeight=10
IOReadBandwidthMax=/dev/sda 10M
IOWriteBandwidthMax=/dev/sda 5M

# Task limits
TasksMax=100

Boot Performance

# Analyze boot time
systemd-analyze

# Show blame (slowest services)
systemd-analyze blame

# Show critical chain
systemd-analyze critical-chain

# Plot boot chart
systemd-analyze plot > boot.svg

# Disable unnecessary services
systemctl disable --now unnecessary.service

# Mask slow services not needed
systemctl mask --now slow.service

Advanced systemctl Features

Template Units

Create template service /etc/systemd/system/[email protected]:

[Unit]
Description=Web Application %i
After=network.target

[Service]
Type=simple
User=webapp-%i
WorkingDirectory=/opt/webapp/%i
ExecStart=/opt/webapp/%i/start.sh
Restart=always

[Install]
WantedBy=multi-user.target

Use template:

# Enable instances
sudo systemctl enable [email protected]
sudo systemctl enable [email protected]

# Start instances
sudo systemctl start [email protected]
sudo systemctl start [email protected]

Socket Activation

Socket unit /etc/systemd/system/myapp.socket:

[Unit]
Description=MyApp Socket

[Socket]
ListenStream=8080
Accept=no

[Install]
WantedBy=sockets.target

Service unit /etc/systemd/system/myapp.service:

[Unit]
Description=MyApp Service
Requires=myapp.socket

[Service]
Type=simple
ExecStart=/opt/myapp/server
StandardInput=socket

[Install]
Also=myapp.socket

Timer Units

Timer unit /etc/systemd/system/backup.timer:

[Unit]
Description=Daily Backup Timer
Requires=backup.service

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target

Service unit /etc/systemd/system/backup.service:

[Unit]
Description=System Backup
ConditionACPower=true

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup.sh

Best Practices

Service Design

1. Use Type=simple when possible for easier management
2. Set proper dependencies to ensure correct startup order
3. Implement health checks with ExecStartPost or separate monitoring
4. Use security features to minimize attack surface
5. Set resource limits to prevent system overload

Configuration Management

1. Never edit system unit files in /usr/lib/systemd/system/
2. Use systemctl edit for modifications
3. Create drop-in directories for complex overrides
4. Version control your custom unit files
5. Document service dependencies and requirements

Monitoring and Maintenance

1. Monitor service logs regularly
2. Set up alerts for service failures
3. Test services after system updates
4. Review and optimize boot performance
5. Keep services minimal and focused

Security Considerations

1. Run services as non-root users
2. Use DynamicUser when possible
3. Enable sandboxing features
4. Restrict network access if not needed
5. Limit filesystem access to required paths

Common Service Examples

Web Server Management

# Apache/httpd
sudo systemctl start httpd
sudo systemctl enable httpd
systemctl status httpd

# Nginx
sudo systemctl start nginx
sudo systemctl enable nginx
systemctl status nginx

# Check configuration before reload
sudo nginx -t && sudo systemctl reload nginx

Database Services

# MariaDB/MySQL
sudo systemctl start mariadb
sudo systemctl enable mariadb
systemctl status mariadb

# PostgreSQL
sudo systemctl start postgresql
sudo systemctl enable postgresql
systemctl status postgresql

# MongoDB
sudo systemctl start mongod
sudo systemctl enable mongod
systemctl status mongod

Application Services

# Docker
sudo systemctl start docker
sudo systemctl enable docker
systemctl status docker

# Redis
sudo systemctl start redis
sudo systemctl enable redis
systemctl status redis

# Elasticsearch
sudo systemctl start elasticsearch
sudo systemctl enable elasticsearch
systemctl status elasticsearch

System Services

# SSH
sudo systemctl start sshd
sudo systemctl enable sshd
systemctl status sshd

# Firewall
sudo systemctl start firewalld
sudo systemctl enable firewalld
systemctl status firewalld

# NetworkManager
sudo systemctl start NetworkManager
sudo systemctl enable NetworkManager
systemctl status NetworkManager

Conclusion

systemctl is an essential tool for managing services in AlmaLinux. Understanding its capabilities enables efficient system administration, from basic service control to advanced features like socket activation and security sandboxing.

Key takeaways:

• Master basic service operations (start, stop, enable, status)
• Understand unit file structure and dependencies
• Implement security best practices in service configuration
• Use logging and monitoring for troubleshooting
• Leverage advanced features for complex deployments

Regular practice with systemctl commands and understanding of systemd concepts will make you proficient in managing modern Linux systems.