History

Manav Rathi e2c894f87f [infra] Limit the maximum size of the nginx stdout logs nginx logs to stdout, which is captured by docker and put into a file at /var/lib/docker/containers/<nginx-cont-id>/<id>-json.log By default, the size of this file is unbounded. Add a maximum limit of 1 GB to this. References: - https://docs.docker.com/config/containers/logging/local/ - https://stackoverflow.com/questions/31829587/docker-container-logs-taking-all-my-disk-space		2024-04-10 09:23:48 +05:30
..
listmonk	[infra] Fix the listmonk upgrade invocation	2024-04-05 16:51:08 +05:30
nginx	[infra] Limit the maximum size of the nginx stdout logs	2024-04-10 09:23:48 +05:30
prometheus	Update install instructions	2024-03-18 13:30:07 +05:30
promtail	Shorten	2024-03-18 13:31:10 +05:30
status	Initial cut of listmonk setup	2024-04-04 20:23:25 +05:30
README.md	[infra] Mention that logging to stdout is a good default for services	2024-03-20 11:28:40 +05:30

README.md

Services

"Services" are Docker images we run on our instances and manage using systemd.

All our services (including museum itself) follow the same pattern:

They're run on vanilla Ubuntu instances. The only expectation they have is for Docker to be installed.
They log to fixed, known, locations - /root/var/log/foo.log - so that these logs can get ingested by Promtail if needed.
Each service should consist of a Docker image (or a Docker compose file), and a systemd unit file.
To start / stop / schedule the service, we use systemd.
Each time the service runs it should pull the latest Docker image, so there is no separate installation/upgrade step needed. We can just restart the service, and it'll use the latest code.
Any credentials and/or configuration should be read by mounting the appropriate file from /root/service-name into the running Docker container.

Systemd cheatsheet

sudo systemctl status my-service
sudo systemctl start my-service
sudo systemctl stop my-service
sudo systemctl restart my-service
sudo journalctl --unit my-service

Adding a service

Create a systemd unit file (See the various *.service files in this repository for examples).

If we want the service to start on boot, add an [Install] section to its service file (note: starting on boot requires one more step later):

[Install]
WantedBy=multi-user.target

Copy the service file to the instance where we want to run the service. Services might also have some additional configuration or env files, also copy those to the instance.

scp services/example.service example.env <instance>:

SSH into the instance.

ssh <instance>

Move the service /etc/systemd/service, and any config files to their expected place. env and other config files that contain credentials are kept in /root.

sudo mv example.service /etc/systemd/system
sudo mv example.env /root

If you want to start the service on boot (as spoken of in the [Install] section above), then enable it (this only needs to be done once):

sudo systemctl enable service

Restarts systemd so that it gets to know of the service.

sudo systemctl daemon-reload

Now you can manage the service using standard systemd commands.

sudo systemctl start example

To view stdout/err, use:

sudo journalctl --follow --unit example

Logging

Simple services can log to their standard output: these are captured by Docker, and by default promtail is setup to injest Docker logs and send them to Grafana.

One issue with the above simple setup is that we cannot attach job names.

If the service needs to to attach a specific job name, or if the service wants more control over the log retention etc, then then services can log to to its own files.

Such files should be in /var/logs within the container, and this should be mounted to /root/var/logs on the instance (using the -v flag in the service file which launches the Docker container or the Docker compose cluster).
There should be entry for this log file in the promtail/promtail.yaml on that instance. The logs will then get scraped by Promtail and sent over to Grafana.