Add doc for Podman-in-Podman

.gitlab-ci.yml

@@ -13,7 +13,7 @@ compose-job:
     BUILD_REPO: "/build-repo"
     SOURCE_REPO: "/source-repo"
   before_script:
-    - podman login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
+    - podman login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"
   script:
     - mkdir -p "$(pwd)/.tmp"
     - |
@@ -35,10 +35,9 @@ nonfree-build-job:
   tags:
     - podman-in-podman
   variables:
-    RELVER: "39"
+    PODMAN_IMAGE_NAME: "${CI_REGISTRY_IMAGE}/vauxite-nonfree:39"
-    PODMAN_IMAGE_NAME: "${CI_REGISTRY_IMAGE}/vauxite-nonfree:${RELVER}"
   before_script:
-    - podman login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
+    - podman login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"
   script:
     - cd oci/nonfree
     - podman build -t "$PODMAN_IMAGE_NAME" .

README.org

@@ -7,12 +7,15 @@ The ~vauxite.json~ treefile and under the ~src~ directory contains my personal c
 ** How I use this repository
 *** Setup GitLab runner with Podman
 1. Install [[https://docs.gitlab.com/16.9/runner/install/linux-manually.html][GitLab Runner]].
+
 2. Create a new runner from the GitLab UI.
+
 3. Use the authentication token from the GitLab UI to register a new runner on the machine hosting the runner. Select the Docker executor.
    #+begin_src shell
    sudo systemctl enable --now gitlab-runner.service
    sudo gitlab-runner register --url https://git.hyperreal.coffee --token <TOKEN>
    #+end_src
+
 4. Add the following lines to ~/etc/gitlab-runner/config.toml~ for Podman:
    #+begin_quote
    We need to set ~privileged = true~ in order to create new namespaces inside the runner container. See https://github.com/coreos/rpm-ostree/pull/429
@@ -36,31 +39,99 @@ The ~vauxite.json~ treefile and under the ~src~ directory contains my personal c
 
    We should now be ready to use the Podman runner.
 
-*** Additional notes about ~.gitlab-ci.yml~
+*** Setup Podman-in-Podman runner
-- The package ~container-selinux~ is required for the vauxite-compose-job so SELinux works inside the runner container. I'm considering having a custom image built on a weekly basis from registry.fedoraproject.org/fedora:latest that contains updated packages and the required dependencies, which I would then just use as the runner's container image. The registry.fedoraproject.org/fedora:latest image doesn't seem to be updated at all.
+To get the [[https://gitlab.com/qontainers/pipglr][Podman-In-Podman]] runner working for a self-hosted GitLab instance, we need to clone the repository and build a custom image passing the GitLab instance URL as a build argument.
+
+1. Login to the right container registry.
+   #+begin_src shell
+   podman login -u hyperreal -p <registry api token> git.hyperreal.coffee:5050
+   #+end_src
+
+2. Clone, build, and push to registry.
+   #+begin_src shell
+   git clone https://gitlab.com/qontainers/pipglr
+   cd pipglr
+   podman build --env GITLAB_URL=https://git.hyperreal.coffee -t git.hyperreal.coffee:5050/fedora-atomic/containers/pipglr:latest .
+   podman push git.hyperreal.coffee:5050/fedora-atomic/containers/pipglr:latest
+   #+end_src
+
+3. Create a new runner from the GitLab UI.
+
+4. Use the authentication token from the GitLab UI to create a podman secret.
+   #+begin_src shell
+   echo '<token>' | podman secret create REGISTRATION_TOKEN -
+   #+end_src
+
+5. Ensure ~config.toml~ exists in the current working directory.
+   #+begin_src shell
+   touch ./config.toml
+   #+end_src
+
+6. Use the custom image we pushed to the container registry.
+   #+begin_src shell
+   IMAGE="git.hyperreal.coffee:5050/fedora-atomic/containers/pipglr:latest"
+   #+end_src
+
+7. Register the runner.
+   #+begin_src shell
+   podman container runlabel register $IMAGE
+   #+end_src
+
+8. Edit ~config.toml~ to use the custom Fedora image from this project.
+   #+begin_src toml
+   [runners.docker]
+     image = "git.hyperreal.coffee:5050/fedora-atomic/containers/fedora:latest"
+   #+end_src
+
+9. Setup the storage and cache volumes.
+   #+begin_src shell
+   podman container runlabel setupstorage $IMAGE
+   podman container runlabel setupcache $IMAGE
+   #+end_src
+
+10. Create a podman secret using ~config.toml~.
+    #+begin_src shell
+    podman secret create config.toml ./config.toml
+    #+end_src
+
+11. Run the runner.
+    #+begin_src shell
+    podman container runlabel run $IMAGE
+    #+end_src
+
+12. Enable the runner user to run services after logout:
+    #+begin_src shell
+    sudo loginctl enable-linger $(id -u)
+    #+end_src
+
+The new Podman-In-Podman runner should now appear in the GitLab UI.
+
+**** Expand user namespace
+1. pipglr excludes three UID/GIDs from being used by job-level containers. One for each of root, podman, and runner users. Since most distributions set ~65536~ as the default maximum number of IDs to allocate for user namespaces (via ~/etc/login.defs~), distribution images that assign essential users a high UID/GID will fail to setup the namespace for pipglr jobs. The workaround to this is increasing the UID/GID limit on the host by three. See [[https://gitlab.com/qontainers/pipglr/-/blob/main/root/setup.sh?ref_type=heads#L86]]
+ ~/etc/subuid~ and ~/etc/subgid~:
+   #+begin_src shell
+   jas:100000:65539
+   #+end_src
+
+2. Make the changes take effect.
+   #+begin_src shell
+   podman system migrate
+   #+end_src
+
+For debugging purposes:
+#+begin_src shell
+podman logs --since 0 pipglr
+#+end_src
+
+It may be necessary to build a custom pipglr image with more verbose logging. The ~runner.service~ and ~podman.service~ files have ~log-level~ options that can be set to "debug".
+
+*** Notes about ~.gitlab-ci.yml~
+- The package ~container-selinux~ is required for the vauxite-compose-job so SELinux works inside the runner container. I'm considering having a custom image built on a weekly basis from registry.fedoraproject.org/fedora:latest that contains updated packages and the required dependencies, which I would then just use as the runner's container image. The registry.fedoraproject.org/fedora:latest image doesn't seem to be updated at all. See [[https://git.hyperreal.coffee/fedora-atomic/containers]]
 - BUILD_REPO and SOURCE_REPO are the directories ~/build-repo~ and ~/source-repo~. If these values are changed, then we would need make equivalent changes to the ~volumes~ directive in ~/etc/gitlab-runner/config.toml~ if we want to keep persistent storage of those repos across pipeline runs. Eventually there will be a conditional in the ~.gitlab-ci.yml~ to clean these volumes if another variable (say CLEAN_BUILD) is set to true.
 
-** How I previously used this repository
+*** Rebase to the container image
-- I run Fedora Server in a VM on my homelab machine. This Fedora Server has the ~ostree~ and ~rpm-ostree~ commands installed.
+On PCs using Vauxite, rebase to the container image from the registry:
-- I clone this repository as root under ~/var/local/vauxite-build~.
+#+begin_src shell
-- I then run the ~ostree-engine~ script to build and deploy a new commit for the given reference branch (e.g. ~vauxite/f39/x86_64/main~). The ~rsync-repos~ command is used in the ~ostree-engine~ script to deploy the resulting OSTree repository to a web server root, which serves as the remote repository.
+rpm-ostree rebase ostree-unverified-registry:git.hyperreal.coffee:5050/fedora-atomic/vauxite-build/vauxite-nonfree:39
-
+sudo systemctl reboot
-  To add the repository as a remote on my local client machine:
+#+end_src
-  #+begin_src shell
-  sudo ostree remote add --no-gpg-verify vauxite <URL of web server>
-  #+end_src
-
-  I then rebase to the vauxite branch:
-  #+begin_src shell
-  rpm-ostree rebase vauxite:vauxite/f39/x86_64/main
-  #+end_src
-
-  To build new commits that contain updates, I put this in ~/etc/cron.d/ostree-engine~ on the Fedora Server:
-  #+begin_src shell
-  0 0 * * 2 root /var/local/vauxite-build/ostree-engine
-  #+end_src
-
-  This will run the ~ostree-engine~ script every Tuesday at midnight. Then I can update my local client machine like this:
-  #+begin_src shell
-  rpm-ostree update
-  #+end_src