techne/mastodon.org

#+title: Mastodon
#+setupfile: ../org-templates/page.org

** Full-text search with elasticsearch
*** Install ElasticSearch
#+begin_src shell
sudo apt install -y openjdk-17-jre-headless

wget -O /usr/share/keyrings/elasticsearch.asc https://artifacts.elastic.co/GPG-KEY-elasticsearch

echo "deb [signed-by=/usr/share/keyrings/elasticsearch.asc] https://artifacts.elastic.co/packages/7.x/apt stable main" > /etc/apt/sources.list.d/elastic-7.x.list

sudo apt update

sudo apt install -y elasticsearch
#+end_src

*** Edit ~/etc/elasticsearch/elasticsearch.yaml~
#+begin_src yaml
xpack.security.enabled: true
discovery.type: single-node
#+end_src

*** Create passwords for built-in users
#+begin_src shell
sudo -u elasticsearch /usr/share/elasticsearch/bin/elasticsearch
#+end_src

In a separate shell:
#+begin_src shell
sudo -u elasticsearch /usr/share/elasticsearch/bin/elasticsearch-setup-passwords auto
#+end_src

Copy the generated password for the ~elastic~ user.

*** Create custom role for Mastodon to connect
As the mastodon user on the host:
#+begin_src shell
curl -X POST -u elastic:admin_password "localhost:9200/_security/role/mastodon_full_access?pretty" -H 'Content-Type: application/json' -d'
{
    "cluster": ["monitor"],
    "indices": [{
        "names": ["*"],
        "privileges": ["read", "monitor", "write", "manage"]
    }]
}
'
#+end_src

*** Create a user for Mastodon and assign it the custom role
#+begin_src shell
curl -X POST -u elastic:admin_password "localhost:9200/_security/user/mastodon?pretty" -H 'Content-Type: application/json' -d'
{
    "password": "l0ng-r4nd0m-p@ssw0rd",
    "roles": ["mastodon_full_access"]
}
'
#+end_src

*** Edit .env.production
#+begin_src shell
ES_ENABLED=true
ES_HOST=localhost
ES_PORT=9200
ES_PRESET=single_node_cluster
ES_USER=mastodon
ES_PASS=l0ng-r4ndom-p@ssw0rd
#+end_src

*** Populate the indices
#+begin_src shell
systemctl restart mastodon-sidekiq
systemctl reload mastodon-web
su - mastodon
cd live
RAILS_ENV=production bin/tootctl search deploy
#+end_src

** S3-compatible object storage with Minio
1. Install MinIO
2. Set the region for this instance to ~homelab~
3. Create 'mastodata' bucket
4. Setup Tailscale

Minio API endpoint: tailnet_ip_addr:9000

*** Caddy reverse proxy config
#+begin_quote
Ensure DNS resolves for assets.hyperreal.coffee
#+end_quote

#+begin_src caddy
assets.hyperreal.coffee {
    rewrite * /mastodata{path}
    reverse_proxy http://<tailnet_ip_addr>:9000 {
        header_up Host {upstream_hostport}
    }
}

fedi.hyperreal.coffee {
    @local {
        file
        not path /
    }
    @local_media {
        path_regexp /system/(.*)
    }

    redir @local_media https://assets.hyperreal.coffee/{http.regexp.1} permanent

    ...remainer of config
}
#+end_src

*** Set custom policy on mastodata bucket
#+begin_src json
{
   "Version": "2012-10-17",
   "Statement": [
      {
         "Effect": "Allow",
         "Principal": {
           "AWS": "*"
         },
         "Action": "s3:GetObject",
         "Resource": "arn:aws:s3:::mastodata/*"
      }
   ]
}
#+end_src

*** Create mastodon-readwrite policy
#+begin_src json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:*",
            "Resource": "arn:aws:s3:::mastodata/*"
        }
    ]
}
#+end_src

*** Setup .env.production
#+begin_src shell
S3_ENABLED=true
S3_BUCKET=mastodata
AWS_ACCESS_KEY=<access key>
AWS_SECRET_ACCESS_KEY=<secret access key>
S3_REGION=homelab
S3_PROTOCOL=http
S3_ENDPOINT=http://<tailnet_ip_addr>:9000
S3_FORCE_SINGLE_REQUEST=true
S3_ALIAS_HOST=assets.hyperreal.coffee
#+end_src

*** Restart Caddy and Mastodon services
#+begin_src shell
sudo systemctl restart caddy.service mastodon-web.service mastodon-streaming.service mastodon-sidekiq.service
#+end_src

** Prometheus metrics with statsd_exporter

On the host running Mastodon, download the latest binary from [[https://github.com/prometheus/statsd_exporter/releases][releases]] page.

#+BEGIN_SRC shell
tar xzvf statsd_exporter*.tar.gz
cd statsd_exporter*/
sudo cp -v statsd_exporter /usr/local/bin/
#+END_SRC

Install the statsd mapping file from IPng Networks:

#+BEGIN_SRC shell
curl -OL https://ipng.ch/assets/mastodon/statsd-mapping.yaml
sudo cp -v statsd-mapping.yml /etc/prometheus/
#+END_SRC

Create ~/etc/default/statsd_exporter~.

#+BEGIN_SRC shell
ARGS="--statsd.mapping-config=/etc/prometheus/statsd-mapping.yaml"
#+END_SRC

Create ~/etc/systemd/system/statsd_exporter.service~.

#+BEGIN_SRC systemd
[Unit]
Description=Statsd exporter
After=network.target

[Service]
Restart=always
User=prometheus
EnvironmentFile=/etc/default/statsd_exporter
ExecStart=/usr/local/bin/statsd_exporter $ARGS
ExecReload=/bin/kill -HUP $MAINPID
TimeoutStopSec=20s
SendSIGKILL=no

[Install]
WantedBy=multi-user.target
#+END_SRC

Ensure port 9102 is open in Firewalld's internal zone.

#+BEGIN_SRC shell
sudo firewall-cmd --permanent --zone=internal --add-port=9102/tcp
sudo firewall-cmd --reload
#+END_SRC

Edit ~/home/mastodon/live/.env.production~.

#+BEGIN_SRC shell
STATSD_ADDR=localhost:9125
#+END_SRC

Start and restart the daemons.

#+BEGIN_SRC shell
sudo systemctl daemon-reload
sudo systemctl start statsd_exporter.service
sudo systemctl restart mastodon-sidekiq.service mastodon-streaming.service mastodon-web.service
#+END_SRC

If using Tailscale, ensure the host running Prometheus can access port 9102 on the host running Mastodon.

On the host running Prometheus, add the statsd config.

#+BEGIN_SRC yaml
- job_name: "stats_exporter"
  static_configs:
  - targets: ["hyperreal:9102"]
#+END_SRC

Restart Prometheus.

#+BEGIN_SRC shell
sudo systemctl restart prometheus.service
#+END_SRC

To import the Grafana dashboard, use ID 17492.

Source: [[https://jaanus.com/mastodon-monitoring-prometheus-grafana/][How to set up monitoring for your Mastodon instance with Prometheus and Grafana]]