Skip to content

Gitlab CE Behind Traefik

At my university’s CS department, we are trying to set up an internal GitLab CE instance for CI/CD of our projects. Based on our infrastructure, I decided to manage a Traefik instance and deploy GitLab Omnibus on Docker.

Nginx #

To have Traefik reverse-proxy the nginx instance, simply disable HTTPS and route the traffic to container port 80:

nginx['listen_port'] = 80
nginx['listen_https'] = false
nginx['real_ip_trusted_addresses'] = [ '172.19.0.2/32' ] # Traefik's internal addresss
nginx['real_ip_header'] = 'X-Real-Ip' # X-Real-Ip is set by Traefik
nginx['real_ip_recursive'] = 'on'

container labels:

traefik.http.services.gitlab.loadbalancer.server.port: "80"
traefik.http.routers.gitlab.service: "gitlab"
traefik.http.routers.gitlab.entrypoints: "web"
traefik.http.routers.gitlab.rule: "Host(`gitlab.example.com`)"
traefik.http.routers.gitlab.tls.certResolver: "http"

Grafana #

GitLab Omnibus has a built-in Grafana instance that authenticates user using GitLab OAuth. I ran into the issue where Grafana was not able to complete the OAuth process. Let’s say the machine’s public facing address is 1.2.3.4 and Traefik container IP address is 172.16.0.2. I looked into the logs and found this in /var/log/gitlab/grafana/current:

2021-04-19_18:15:05.40382 t=2021-04-19T18:15:05+0000 lvl=eror msg=login.OAuthLogin(NewTransportWithCode) logger=context userId=0 orgId=0 uname= error="Post \"https://gitlab.example.com/oauth/token\": dial tcp 1.2.3.4:443: connect: connection timed out"

Apparently the GitLab container is not able to connect to gitlab.example.com via 1.2.3.4. Double checking on a shell session inside the container confirms the case:

root@server:/srv/gitlab-ce# docker exec -it gitlab-ce_web_1 bash
root@96c275d00c01:/# curl -v gitlab.example.com
*   Trying 1.2.3.4:80...
(hangs)

It turns out ufw was blocking the traffic from all Docker network interfaces to the host machine (INPUT chain). I decided to disable ufw once and for all.

It is worth to mention that in GitLab’s official documentation on running with Docker Compose, they suggest specifying hostname option for the GitLab container. Actually, the option should not be used, because that would resolve gitlab.example.com to 127.0.0.1 inside the container.

At the beginning, you need access to the Grafana admin account to grant administrative permissions to other accounts (i.e. your Grafana account). To do that, first enable the login form, specify a secure password and log in.

grafana['disable_login_form'] = false
grafana['admin_password'] = 'foobar'

After you’ve done the administrative actions, do not forget to turn it back off:

grafana['disable_login_form'] = true

GitLab Pages #

In some of our CI pipelines, there are static websites built by Vue.js that we wish to be directly accessable in Jobs Artifacts, and that is handled by GitLab Pages daemon. To make it work with Traefik, disable pages_nginx and have Traefik route the requests to the Pages daemon directly:

pages_external_url "http://gitlab-pages.example.com"
pages_nginx['enable'] = false
gitlab_pages['external_http'] = ['0.0.0.0:8081']

Traefik labels:

traefik.http.services.gitlab-pages.loadbalancer.server.port: "8081"
traefik.http.routers.gitlab-pages.service: "gitlab-pages"
traefik.http.routers.gitlab-pages.entrypoints: "web"
traefik.http.routers.gitlab-pages.rule: "Host(`example.com`) || Host(`gitlab-pages.example.com`) || HostRegexp(`{subdomain:[a-z]+}.gitlab-pages.example.com`)"
traefik.http.routers.gitlab-pages.tls.certResolver: "http"

Don’t forget to expose port 8081 to make it reachable by Traefik.

In addition, our project visibility is set to internal. According to the docs, we needed to enable Access Control for GitLab Pages in gitlab.rb:

gitlab_pages['access_control'] = true

While the GitLab instance runs as a container, we also needed inplace_chroot:

gitlab_pages['inplace_chroot'] = true

It turned out that these two has conflicts to each other, which is explained in this official documentation.

The workaround was pretty much a hack. While the GitLab instance is for internal-only, I decided to go for the hack. Get a shell to the container, and run:

$ mkdir -p /var/opt/gitlab/gitlab-rails/shared/pages/etc/
$ cp /etc/resolv.conf /var/opt/gitlab/gitlab-rails/shared/pages/etc/
$ cp -rv /etc/ssl /var/opt/gitlab/gitlab-rails/shared/pages/etc/
$ gitlab-ctl restart gitlab-pages

After setting this up, I found out that Pages only serves .html, .htm, .txt, and .json. It does not serve .js or some images. So it ended up no use for us.

Git SSH #

In our setup, we have Traefik inside of an another Docker network named traefik. In the beginning, when I was trying to publish port 22 for git server, I forgot to add the GitLab container in the default network as well. So don’t forget to do that if you’re using similar setup.