HCL Nomad and TLS via CertMgr task
So I had this idea to test out Nomad Web on a brand-new v.14.5 Domino server as an alternative to the normal Notes client - but it had to fit into the "normal" way I give access to our servers...
Normally when I open up for access to an internal service I use an internal proxy server to orchestrate all the web sites/service. One majo benefit of this is to handle all TLS (using LetsEncrypt) in one central place. And this has worked and still works fine!
I had recently built an image from the community server to include e.g. Nomad on the Domino that I run in a container (on a QNAP NAS - but that is a different story!). So after having upgraded the server to use the new image I just needed to start the Nomad server task ("load nomad" - which I always handle in program documents) for Nomad to be ready on my server. Nomad will listen to requests on port 9443 "out of the box". But it also complained that it was not running on TLS. In the past I have lifted off the TLS cert on the proxy server and then continued to send the request on http (ie. not encrypted). However, I have also become aware that this may not be the best way to handle it (although I do think we have control over all access internally on our network - unless some foreign webcam, printer, or ... is listening to the internal traffic....???). Anyway, this was then the occasion to set up TLS on the inside.
First, I set up a website to be used from the outside called "nomad.domain.com" on the Nginx proxy server. Using the "certbot" code from LetsEncrypt it is just a matter of seconds to have a working TLS certificate on your Nginx server. So far so good - this is all wellknown ground.
Then I set up an internet site document on my Domino 14.5 server called "domino2.domain.com". However, this specific server is normally not accessible from the outside as there is no website on the proxy server to send anything that way. But as I wanted to use the "certmgr" task on Domino to automatically maintain the TLS certs for me I would need LetsEncrypt to be allowed to reach the server on port 80 (as you can see here). So how to do that? Well, after some friendly chats with some of the AI bots I ended up with this configuration on the nginx server:
server {
listen 80;
server_name domino2.domain.com;
location /.well-known/acme-challenge/ {
proxy_pass http://domino2.domain.com:80;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
location / {
return 404; # Normal operation – only .well-known is allowed
}
}
From the "outside" any subdomain of the "domain.com" will be sent to our public IP. And on the inside all traffic on port 80 is sent to the proxy server that then looks at the request. But internally "domino2.domain.com" will resolve to the Domino server - hence the proxy server can identify the request from the outside (well only from LetsEncrypt) and send it on to "itself".
Next I already had CertMgr running on our internal mail server (to maintain TLS for the domain name used for mail. I was not sure if I had to start certmgr on domino2 or just create a replica of certstore.nsf from the mail server. I looked at a couple of articles (here and here) without really becoming sure what to do. But it turns out that it doesn't matter. The certmgr task handles it either way. I ended up loading certmgr on the second server as well - and it nows that the other one is the "boss" do it just acts as a client (and creates certstore.nsf). Next, you set up TLS the "normal" way - and with the Nginx configuration in place (Ok, it took a couple of attempts) then a new certificate is received.
Then we want to make sure that the connection internally works to allow nomad to load on domino2. I started by first making sure I could open just the standard "homepage.nsf" on TLS. First you need to open up for port 443 in the server document under "Ports / Internet Ports / Web" - ensure TLS Port status is "Enabled". Next, you look at your internet site document where you need to have host names set to: "domino2.domain.com" - and you may as well add "nomad.domain.com" now (to ensure proper logging when the final setup is ready). And then you must go the Security tab and change the name of the Key filename to: domino2.domain.com (the exact same name that you used in the TLS Credentials document in the certstore.nsf. This will tell Domino which certificate to use.
With all this in place you should be able to open https://domino2.domain.com (when inside your local network - effectively what the nginx server will nee to be able to forward requests to).
One thing that turned out is that you have to tell Nomad how to find the right certificate from certstore.nsf (or really the TLS cache). It is not yet clever enough to use the same mechanism as the rest of Domino. You need to set a notes.ini variable (please do use a configuration document!). In our case:
NOMAD_WEB_HOST=domino2.domain.com
Restart the nomad task ("restart task nomad") and you should now be able to load nomad using: https://domino2.domain.com:9443
Having reached this point I wanted to be able to give access to nomad on the "normal" TLS port 443 from "the outside". So back to the Nginx proxy for the website "nomad.domain.com". I had to build a configuration based partly on what we normally do and partly on this article and before I finally had it working. This is the config:
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
# Copy this setup to the new server, set the server_name and name the virtual server the same
server {
server_name nomad.dalsgaard-data.dk;
location / {
proxy_pass https://domino2.domain.com:9443$request_uri;
# include /etc/nginx/default.d/*.settings;
proxy_set_header Host nomad.domain.com;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# Activate SNI, so Domino2 can select the right certificate
proxy_ssl_server_name on;
}
location /nrpc-wss {
proxy_pass https://domino2.domain.com:9443/nrpc-wss;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
}
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/nomad.domain.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/nomad.domain.com/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}
server {
if ($host = nomad.domain.com) {
return 301 https://$host$request_uri;
} # managed by Certbot
listen 80;
server_name nomad.nomad.domain.com;
return 404; # managed by Certbot
}
... and now I have a nomad set up where I route through our proxy for external connections and can use nomad internally with a direct url to Domino - but still using TLS.
Blog
