Working with Managed Servers

The Managed Server page is where you can add a new server, view the list of existing servers, and view monitoring data on each server. From here, you may also access server settings and details for each server. 

The Managed Server List Interface

The Managed Server Section List provides server and monitoring data for each managed DNS / DHCP server. 

The list provides the following information:

Server Name: The server display name, set by the user during server creation. You may click on the server name link to view server details

Requests: Requests are the heartbeat notifications received in the last 2 minutes. Requests are sent every 30 seconds, so a display of 4 or 5 requests represents a satisfactory request connection, and the request bar will show in green. For any lower value, the color of the bar will show in red.

DNS / Monitor / DHCP: The version of the component running. If the server is not running, no version will display.

State: State shows the overall state of the server - the server may be running, but without any components started. "Running" will show in green, whereas "Stopped" will display as an orange bar.

Action Menu (gear icon): Right click on the action menu (or anywhere on the row) to display available server actions

Working with the Managed Server List

The following actions may be performed when interacting with the server list:

Sort the list by clicking on the "Server Name" column, to view by Ascending / Descending order

Search / Filter the list for specific servers by entering a full or partial server name into the search box at the upper right of the list

Display more/fewer servers per page by clicking on the "Items per page" selector at the top right of the list, next to the search box

Refresh the list to check for updated information by clicking the "Refresh Table" button (rotating arrows)

Click on a server name to view / edit server details

Add New Managed Server (Setup Wizard)

To set up a new managed server, click "Add Server" at the top right of the Managed Server Tab. 

From there, proceed through each of the five steps of the Add New Server wizard, below:

After a new server has been created, you may edit it by clicking on the server name, or view monitoring data from the Managed Server list. 

View or Edit Managed Server Details

Click on a server name, or open the server action menu to view details for a managed server.

The details of a server are organized into tabs: Settings, Services, Monitoring, and RPS. 

Settings

By default, only "Display Name", "Server OS" and "FQDN or IP" fields are shown.

When any edits are complete, click "Save Changes". 

Services

Under Services, you can view the enabled components for this server.

When done, click "Save" to ensure any changes are saved, or click "Close" to exit without saving.

Monitoring

Under Monitoring, you can see the aggregated system statistics + "DNS aggregate":

RPS (requests per second)

Under RPS, you can search for data for a given dns record. 

Update / Install Managed Server (Existing Component)

The installation can be run more than once. If we already have running components, they will be stopped before the real installation continues.

If there is newer version of any component, it will be downloaded and used.

The overall process is:

Requirements

Python 3.6+ and Docker are the only requirements. If we run the installation with root (see below) and we have internet access, the installation process will install Docker.

The overall process is always the same - ProVision generates an install bundle, it is transferred to the server, extracted there and run some scripts from it. This can be done by ProVision (automatic install) or by the user (manual). In case of automatic install ProVision will automate the steps, which the user would do manually. All the components can run without root access, but we need it for user creation, folders creation, network and heartbeat service setup.

Automatic (Recommended)

This is the recommended and easiest setup method. If you have root access, ProVision can do all the necessary actions.

Go to the details page → Settings tab → "Run Actions" → "Push Install"

Confirmation dialog will be shown:

Popup with the progress will be shown.:

Each command/step is shown together with the overall progress.

Manual (from the command line)

This case is usually when the user doesn't want to input the root user in ProVision, but he has root access. The setup bundle should be downloaded and saved on your local computer:

The setup bundle name will be in format "setup-XXX.tar" where XXX is the resource ID of the managed server. The rest of the process is:

• copy the file to the managed server (with SCP or any other suitable way)
• extract it with "tar -xf setup-XXX.tar"
• execute "sudo /bin/bash install_python3.sh" - this should finish with "PVOK"

Without root

This is the used if root access to the server is not available. In this case, an admin with root access should first do the following:

• Install docker with the relevant commands for the OS. This can be done by running "install_docker.sh" from the setup bundle with root or any other suitable way

yum update -y
yum install -y yum-utils bind-utils

yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo

yum install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin

systemctl enable docker
systemctl start docker

docker run hello-world

sudo apt-get update

sudo apt-get install -y \
    ca-certificates \
    curl \
    gnupg \
    lsb-release

# default start
KEYRING_FILE=/usr/share/keyrings/docker-archive-keyring.gpg
[ -f "$KEYRING_FILE" ] && rm $KEYRING_FILE

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o $KEYRING_FILE

echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=$KEYRING_FILE] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt-get update
sudo apt-get -y install docker-ce docker-ce-cli containerd.io
sudo docker run hello-world

• Create the relevant user, add it to "docker" group, create "/provision" directory and make the created user owner of this directory. 

useradd -m <user>
passwd <user>
usermod -a -G docker <user>
mkdir -p /provision
chown -R <user>:<user> /provision

• Make the necessary network changes. This can be done by running "network_config.sh" from the install bundle 

yum update -y
yum install -y yum-utils bind-utils iptables-services

systemctl enable iptables
#iptables -I INPUT -p udp -m udp --dport 53 -j ACCEPT
#service iptables save

iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT
iptables -t nat -F
iptables -t mangle -F
iptables -F
iptables -X
ip6tables -P INPUT ACCEPT
ip6tables -P FORWARD ACCEPT
ip6tables -P OUTPUT ACCEPT
ip6tables -t nat -F
ip6tables -t mangle -F
ip6tables -F
ip6tables -X
service iptables save

#!/usr/bin/sh
if [ "$(readlink /etc/resolv.conf)" != "/run/systemd/resolve/resolv.conf" ]; then
    ln -sf /run/systemd/resolve/resolv.conf  /etc/resolv.conf
fi

sed -i 's/^#DNSStubListener/DNSStubListener/;s/DNSStubListener=yes/DNSStubListener=no/' /etc/systemd/resolved.conf

systemctl stop systemd-resolved
systemctl start systemd-resolved

• Create service file and edit ﹤user﹥ with the correct system user.

# https://www.cloudsavvyit.com/3092/how-to-add-your-own-services-to-systemd-for-easier-management/
# https://www.opentechguides.com/how-to/article/centos/169/systemd-custom-service.html
# https://computingforgeeks.com/how-to-run-systemd-service-without-root-sudo/
# https://unix.stackexchange.com/a/497011

[Unit]
Description=pv_heartbeat Service, which reports the current status.
After=network.target
# StartLimitIntervalSec=0

[Service]
Type=simple
Restart=always
RestartSec=1
User=<user>
ExecStart=/usr/bin/python3 /provision/libs/pv_heartbeat.py

[Install]
# WantedBy=multi-user.target
WantedBy=deault.target
Alias=pv_heartbeat.service

• Make symlink to this file (replace /provision/libs/pv_heartbeat.service with the correct file path):

ln -s /etc/systemd/system/pv_heartbeat.service /provision/libs/pv_heartbeat.service

• Create sudoers file (﹤user﹥ with the correct system user) and place it in /etc/sudoers.d:

# https://unix.stackexchange.com/a/497011

Cmnd_Alias MANAGE_PV_HEARTBEAT = \
    /bin/systemctl enable  pv_heartbeat, \
    /bin/systemctl disable pv_heartbeat, \
    /bin/systemctl start   pv_heartbeat, \
    /bin/systemctl stop    pv_heartbeat, \
    /bin/systemctl restart pv_heartbeat, \
    /bin/systemctl status  pv_heartbeat

<user> ALL = (root) NOPASSWD: MANAGE_PV_HEARTBEAT

# https://unix.stackexchange.com/a/497011

Cmnd_Alias MANAGE_PV_HEARTBEAT = \
    /usr/bin/systemctl enable  pv_heartbeat, \
    /usr/bin/systemctl disable pv_heartbeat, \
    /usr/bin/systemctl start   pv_heartbeat, \
    /usr/bin/systemctl stop    pv_heartbeat, \
    /usr/bin/systemctl restart pv_heartbeat, \
    /usr/bin/systemctl status  pv_heartbeat

<user> ALL = (root) NOPASSWD: MANAGE_PV_HEARTBEAT

• Reload the systemctl daemon:

systemctl daemon-reload

From this point automatic or manual setup can be done with the non-root user.

Update Managed Server

In ProVision's Managed Servers page, open the server details. Then, under the settings tab, click "Run Actions, and select "Push Update". ProVision will automatically create a backup file in /provision/backups/.

Managed Server Diagnostics /  Debug

Automatic - GUI

The user can run some basic diagnostics from the Managed Serve details page, under the Settings tab. Click "Run Actions", then "Run Diagnostics":

All the debug and diagnostic tools are in the monitoring component, thus it needs to be running in order to function.

After clicking the button, a confirmation dialog will be shown with the results:

We verify if we have docker running, if the monitoring component is running and the RabbitMQ connection.

You may download the results by clicking "Download Log File", and "Close" when done.

Manual - Command Line Scripts

All the scripts are in "/provision/libs/". 

All the scripts can be started with "-h" parameter to show relevant help.

• pv_install.py - used for the whole setup. The process itself is broken into separate steps, which can be executed individually with "-e/--exec-only=". The steps can be seen from the help (-h). This should be used by developers and for debugging purposes
• start_pv_*.sh - all these scripts are used to start given component. All of them support "-v/–version" to specify the version of the component to start and "-b/–bash" to start the container in debug mode. In debug mode the user will be logged into the container but without starting /start.sh (the entrypoint).
• stop_pv_*.sh - all these scripts are used to properly stop given component/container

Local

In addition to logging into any of the containers (starting the relevant script with -b/–bash) few other files are included into the monitoring component:

• mysql client for browsing the database
• dhtest (link) for DHCP testing
• dnsutils - dig, nslookup, nsupdate
• telnet

Delete Managed Server

To delete a managed server, right click on the action menu/ row for the server, and select "Delete".

Additional Information

• Managed Servers Settings