ppm/tutorial.md

The tutorial for ppm is so you fully understand what PPM is and what it does. We also provide an ansible role that will do everything that is described in this file but performing the steps in this tutorial will allow you to fully understand how to set up an application

Prepare Applications

To prepare the server, some dependencies must be installed. For our tutorial we use debian 12 and for now we did not yet test any other linux distribution, but our intention is to make ppm distribution independant - so long as it has recent enough versions of the packages we depend on, and that the system uses systemd.

PPM has the following dependencies:

• usermode podman. Most applications will be running as container, and everything will be running as a regular user. So to have usermode podman, the following packages are required: podman, uidmap, netavark, slirp4netns.

• podman compose. We have a few applications running as a docker-compose file. This dependency is not needed by ppm, but by several ppm applications.

• Restic. Restic allows us to do backups. We might in the future support more backup methods but for now, only restic is accepted

• Git. We distribute application definitions as git repositories, so we need git to clone and update these repositories

• PPM itself. PPM has been written in python, and is currently distributed as an executable zip file that includes all dependencies. However, python3-jsonschema is not written in pure python, so this dependency must be installed manually. Getting rid of non pure-python dependencies is welcome, help wanted.

So to install all these dependencies do the following as root:

apt install -y podman uidmap netavark slirp4netns podman-compose restic python3-jsonschema git
cd /usr/local/bin
wget https://ppm.pfoe.be/api/packages/ppm/generic/ppm/0.0.1/ppm
chmod 755 ppm

Prepare state directory

The state directory is the directory where applications put their state files. These state files are used to let applications communicate parameters to each other. We will see examples further in the tutorial. The state directory is by default /home/.ppmstate but can be changed in the ppm.yml config file. Make this directory and use the sticky bit to give each application the rights to their own files:

mkdir /home/.ppmstate
chmod 1777 /home/.ppmstate

Installing forgejo.

As a first application we will install forgejo. Forgejo is a github.com clone. We install it to explain how ppm works, but you are ofcourse not required to use forgejo, this is just an example.

To install a forgejo application, we need to create a new linux user. All applications run as regular users, so it is easy to understand where the files go, which tasks are running and how to remove an application. Well, easy to understand as long as you understand linux user namespaces, as processes will be using multiple userid's under usermode podman.

So to install forgejo, the first step is to create a forgejo user:

useradd -s /bin/bash -m forgejo

The second step is to enable lingering:

loginctl enable-linger forgejo

Lingering means that the user is active at all times, even if there is no one logged in as that user. This basically means that at boot time, systemd's management processes are started for this user. This also means that we can schedule user processes to be started at boot time as a regular user, therefore running applications without any involvement of the root user to schedule or start these daemons.

As of now, we will continue the installation of the forgejo application as the regular user forgejo as we do not need root for this application any further. However, note that if we just use su or sudo, we will not start a full user environment systemd style. Since we already use linger to make sure the complete login session has been set up, you can just place the XDG_RUNTIME_DIR="/run/user/$UID" environment variable, however, if you wish to switch users properly, ie create a brand new user session for said user you should use machinectl shell instead of su or sudo.

So using that knowledge, switch to the forgejo user, either with: (recommended, but might need extra package systemd-container)

machinectl shell forgejo@

or

su - forgejo
export XDG_RUNTIME_DIR="/run/user/$(id -u)"

We now need the configuration file for forgejo. This configuration file in general will contain the following components:

• The location of the backups
• The location of the application definition
• Extra config

For the backups we currently only support restic, but the configuration structure has been designed to allow easier addition of other backup methods in the future. Restic can take backups to various locations, including the local filesystem and a remote rest server. It can also backup against backends such as sftp, s3-compatible storage, various cloud providers, and all providers that are supported by rclone (https://rclone.org) but we have only tested and worked with the rest server and local files, patches are welcome if you can get others to work.

If you already have a restic rest server running you can configure your own repository here, but for this tutorial/example, we will quickly set up a local directory. We have application definitions to setup a restic rest server, if you wish to set up things decently later on.

So let's setup the restic repository under /home/forgejo/resticrepo:

restic -r /home/forgejo/resticrepo init

This will ask for a password, make one up and enter it twice.

Let's create the configuration for our ppm instance. Fire up your favorite editor and create config.yml in the home directory, with the following contents:

backup:
    type: restic
    restic:
        backupname: forgejo
        password: ThePasswordYouEnteredForYourResticRepository
        url: /home/forgejo/resticrepo
config:
    publicurl: forgejo.example.com
appinfo:
    url: https://ppm.pfoe.be/ppm/forgejo.git

Let's try ppm out to do the setup of our program. Execute ppm setup and let's see what it does.

ppm setup

After executing this command, you will see it has done 2 things: It has made a git clone of the appdefinition, so the directory /home/forgejo/appdefinition has appeared, and it attempted to restore a backup but failed.

Restoring of the backup is not possible at this time. We created a backup repository, but the repository is empty - it cannot restore a backup, so it gave us an error. We need to tell ppm that this is a clean install and no data is to be restored. To do this execute the following command:

mkdir data

For ppm, if the data directory does not exist, it will create it by attempting to restore the latest backup into the data directory. If the data directory does exist, it will do no restore. Whether a daily backup timer is also installed depends on the backup plugin loaded by the appinfo.yml: forgejo's appinfo loads the backups_scheduled plugin (offline backup by default), so a systemd timer will be installed to take a backup daily.

This way, you shouldn't worry about backups at all, restore and backup will be done whenever it is needed. However, should you ever need restoring data from an earlier backup, you will need to call restic manually, as ppm does not handle this case. Also ppm does not manage cleanups of old backups etc, but look at the backup manager (TODO, not yet publicly online) to see how to manage those.

Now let's try ppm setup again. This time setup should not give any errors, but instead write a bunch of files and reload systemd. (Note that if you have any errors here in the form of "Failed to connect to bus: No such file or directory" you probably missed the linger step above, or forgot to set up the XDG_RUNTIME_DIR environment variable by using machinectl shell or setting it up manually)

Now let's start forgejo. Use ppm start to do this:

ppm start

If everything goes well, forgejo will be starting. It will take some time to pull the container and start it, but you eventually should see the container in podman ps. With journalctl --user -u forgejo you can see the logs to investigate what is going wrong.

Let's go over everything that has happened:

• ppm has made a clone of the application definition, based on what was in config.yml. You can find it in /home/forgejo/appdefinition. You might want to have a look at appinfo.yml to see what is configured here.
• ppm has ensured that data/ has been created, by restoring the backup, or forcing us to create a new one
• ppm has written several configuration files, such as a forgejo.target, a forgejo.service and a backup.service and backup.timer (see /home/forgejo/.config/systemd/user). The forgejo.target is written by ppm itself: it represents the application as a whole. forgejo.service comes from the application definition and declares WantedBy=forgejo.target, so activating the target brings the service up. The backup.service and backup.timer come from the backups_scheduled plugin loaded by forgejo's appinfo.yml. The timer declares PartOf=forgejo.target and WantedBy=forgejo.target, so it starts together with the application and stops when the application is stopped. The backup.service itself is deliberately not tied to the target: when the timer fires and stop_target is enabled, the service stops the target, takes the backup, and starts the target again - without stopping itself in the process.
• ppm has enabled the target and every unit listed under enableunits in appinfo.yml, as well as backup.timer (contributed by the backup plugin), so they get started at boot time, but has not yet started them.

You might also have noticed that ppm start is basically executing systemctl --user start forgejo.target. We also have ppm stop and ppm status that are wrappers around systemctl, all operating on the target, they are for your convenience. You can use either syntax.

Let's have a look at what has been started. If we type ppm status we see that podman-compose has been used to start a forgejo container. If we use podman ps, we see that a container is running. We also see that the container is listening on localhost on a random port, redirecting that inside the container on port 3000.

We will also see that the data directory has been filled up by forgejo.

To finish off this installation, you might want to do a ppm stopandbackup command. This will stop your forgejo and take a backup.

But how do we use this forgejo instance? It is using a random port number and is only listening on localhost.

We want to be able to run many applications on one server. Many applications are web-based, so they all want to use port 80/443. So we need a way to manage all these applications. Ppm does this by using a basic system that will allow multiple ppm applications, different users to signal information to each other.

Let's have a look at the state file that we created for our forgejo instance. Have a look at /home/.ppmstate

ls /home/.ppmstate

We see a file forgejo.state has been created. Let's have a look at the file.

cat /home/.ppmstate/forgejo.state

This file is a json file, containing several entries. We won't cover everything, but let's look at 2 parts:

The ports entry allows ppm to allocate random ports for this application. The port is chosen randomly, by publishing it here we can ensure that other users will not use our already allocated ports.

Secondly, there are exports. We currently export a web entry, that defines the name of the website, and where to proxy the traffic for this address to.

So let's set up an nginx. We'll need to become root again and do the same as before:

exit (to become root again)
useradd -s /bin/bash -m nginx
loginctl enable-linger nginx

and let's switch to nginx, using either su or machinectl:

machinectl shell nginx@

Create the config.yml with the following content:

appinfo:
    url: https://ppm.pfoe.be/ppm/nginx.git

Currently, nginx requires an ssl certificate, which is outside the scope of this tutorial, so quickly create a self-signed certificate:

openssl req -x509 -newkey rsa:2048 -keyout ssl.key -out ssl.cert -sha256 -days 3650 -nodes -subj "/CN=Dummy"

and execute ppm setup followed by ppm start

Note that our config.yml did not include a backup statement: as our nginx does not contain any data we do not need backups. The nginx appinfo.yml loads the backups_none plugin so ppm knows backups have been intentionally opted out of. Also no other configuration is needed.

You can have a look at the nginx.conf which has been generated in the nginx home directory, you will see that it will contain the required proxypass and virtual host for the forgejo application.

The nginx listens by default on port 8443 and 8080. It uses host networking, so it can only open ports above 1024 as a regular user, but it is in the same namespace as the main host.

Redirecting ports is outside of the scope of ppm - ppm is only designed as non-root user.

Become root again, and use the following iptables rules to ensure that port 80 and 443 are redirected to the ports used by our userspace nginx:

# Redirect 8080 to 80
iptables -A INPUT -p tcp --dport 8080 -j ACCEPT
iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-ports 8080
# Redirect 8443 to 443
iptables -A INPUT -p tcp --dport 8443 -j ACCEPT
iptables -t nat -A PREROUTING -p tcp --dport 443 -j REDIRECT --to-ports 8443

Finally, you can try to surf to forgejo.example.com. DNS is not included so you must manually add forgejo.example.com (or whatever name you used) in your /etc/hosts. Surf to forgejo.example.com and admire the working installation of forgejo.

(note that forgejo might have shut down if you didn't complete the initial setup wizzard in time, you might need to restart it)