Traefik in Nomad using Consul and TLS
Using Traefik for routing containers running in Nomad, while using Consul for service discovery and TLS certificates for security
It may be arguable that microservices architecture is the best thing since the invention of the remote control.
Well, maybe not quite… but microservices have provided various solutions such as improved scalability, better fault isolation, separation of duties, programming language neutrality, security, and more.
After starting using microservices architecture, you will also face new challenges, such as service discovery, container orchestration, and container routing. Let’s look at solving some of these new challenges together!
Objective
To provide a quick guide on using Traefik to route requests to containers on HashiCorp Nomad, using HashiCorp Consul service discovery, and securing the connections with TLS certificates.
Assumptions
Before we continue, let’s get some assumptions out of the way. For you to get the most out of this tutorial, I am assuming that you have:
- Basic knowledge of Hashicorp Nomad and Consul
- Working instances of Hashicorp Nomad and Consul
- Basic experience with YAML file and HCL file configurations
- The Nomad CLI installed
- The Consul CLI installed (optional)
- A basic understanding of Traefik.
Recap
Let’s recap some relevant tools, concepts and terminology!
- Consul: Provides service discovery and health checks for microservices.
- Nomad: A container orchestrator (similar to Kubernetes).
- Traefik: Provides routing, load-balancing, and reverse proxying for microservices architectures.
- Certificates/PKI: Provides encryption and protection for web and other applications.
The problem
Before we start implementing Traefik, let’s review why we need it, and what we are trying to achieve.
Once you start working with microservices, you run into a different set of challenges, such as “Where do I run my containers?”, “How do microservices talk to each other?”, and “How do I reach those microservices?”
Nomad solves the where. Like Kubernetes, Nomad organizes containers into a set of servers, utilizing the best container and server combination to maximize resource utilization and uptime; however, container orchestrators do not necessarily map all the web containers to port 80 or 443. Because there is only one port 80 per server, you will quickly run out of port 80s.
BUT, customers and applications still expect to reach port 80 for HTTP and 443 for HTTPS. This means that we need to find a way to serve multiple ports 80 for various applications.
Let’s look at a scenario to solve these challenges. We will also use the same concept on a smaller scale for our implementation phase.
Let’s say you have five different web applications, A, B, C, D, E, and each application uses three containers each (for high availability, redundancy and performance). The more, the merrier, right?
Each application uses HTTPS on port 443 with TLS certificates for security.
Also, you have three virtual machines (VM) with the Nomad Client binary installed, up and running. You probably want to run all your applications in each Nomad Client to ensure maximum performance and high availability. Luckily, Nomad takes care of that for you. Here’s what your app would look like:
At this point, we encounter our first problem. If all the applications run on port 443 and the VM running Nomad Client only has a single port 443 available. Which application should take it?
To solve this, we let Nomad assign dynamic ports to each container.
In short, Nomad takes care of the mapping between the dynamic port and the application port inside the container.
Now that each container has a dynamic port, the apps can all live happily together, right? Unfortunately, another problem arises: how do I get to my containers?. You could say, “I will hardcode the new dynamic port assigned in Nomad”, right?
But the reality is that the Nomad dynamic ports will keep changing to a new port every time your containers are moved from host to host or the container restarts. So then the question becomes, how can different services communicate with each other if the IP and port combinations keep changing?
Consul takes care of this by using service discovery. This allows applications to connect to each other by querying Consul. App A can communicate with App B if needed, by doing a single API or DNS query to Consul.
Users expect to access the application via https://www.app1.com, which is on port 443 by default. And our new applications are running on dynamic ports. We can tell other applications to use consul for service discovery but it would be unrealistic to ask the user to query Consul every time they try to reach the application on a dynamic port.
Traefik is the solution. Traefik runs in the Nomad Client, and we associate ports 443 to Traefik. Traefik will receive all incoming requests on port 443 (or any assigned port) and routing them to the containers. Traefik follows a set of “rules” to determine which container should receive the proper request.
We will get into rules later but here is an example
“traefik.http.routers.hello-world.rule=Host(`hello-world.domain.com`)”,
Now you can get to your application via port 443 or HTTPS. Unfortunately, another problem arises: your browser presents you with a scary warning message saying that your certificates are invalid. We fix this by using TLS to ensure that the traffic stays encrypted and secure. It also removes that pesky warning message.
One final question: Why Traefik and Consul? Consul provides service discovery to allow communication between app A to app B. But why does Traefik care about it?
Two reasons:
- Traefik also needs to know where those dynamic ports are by querying Consul. Consul keeps track of them, and Traefik connects to Consul to obtain such information. Without it, it wouldn’t be able to route to the proper containers.
- Each container can push dynamically and automatically new configurations (rules) to Traefik. Consul provides the methods to deliver the dynamics rules to Traefik. The rules indicate how and when Traefik will route the request to each container. More on this below.
Ok, let’s dig into an example!
Sample Implementation
Before getting into the specifics, let’s review the bigger picture. We set up multiple applications in our problem section, each with various containers. Let’s do the same, but in a smaller scope to understand the basics.
We will set two applications, Traefik and hello-world. Traefik will query Consul for configuration and networking information. It will then forward the request to the hello-world application.
Running apps in Nomad
Nomad Jobs are used for application development. Basically, you define your application in an HCL file that contains the docker image, networking information, health checks, and other information. The Nomad server then takes the file and decides which Nomad Client to run it in.
Let’s review the Nomad jobs for Traefik and hello-world. We will break down each of them below in detail.
Traefik Jobspec
Hello-world Jobspec
Deploying the applications in Nomad
You can deploy both jobs to Nomad with the following commands
nomad job hello-world.hcl
nomad job traefik.hcl
Traefik configuration method
You can configure Traefik using the Traefik CLI flags, environment variables, or a configuration file (YAML or a TOML). All these do the same thing but in different ways. They are mutually exclusive, but the concepts are still the same.
--api.debug
TRAEFIK_API_DEBUG
File configuration (YMAL)
File configuration (TOML)
To configure Traefik in Nomad, you can use the CLI, Environment variables and or file:
As remainder here is the Traefik JobSpec
- You can place the Traefik CLI flags in the
argfield in the Nomad job.
2. Set the environment variables using Nomad env stanza
3. Use a YAML file yml, or a TOML toml file using the template stanza.
Whichever method you use, the results are the same. I will use a configuration YMAL file using the template stanza.
Nomad Job Explanation
For this hands-on example:
For this hands-on example:
We will:
- Use a YAML file configuration. Feel free to use other methods, such as CLI flags, Environment variables, or TOML.
- Configure Traefik routing, Consul Catalog, and TLS.
We will NOT:
- Cover all of the Traefik configurations; however, we will cover the relevant parts. For more, check out this article.
- Cover all the Nomad job configurations. Here is a great article that explains some of the basics. However, we will cover the relevant parts as well.
Traefik Nomad Job
Configuration (config) stanza
The config stanza explains where to obtain the image. In this case, it is being pulled directly from Docker hub. It also contains the arguments (args) that need to be passed to the container.
You can configure Traefik using Trafik CLI flags. If you decide to go this route, you will place all your CLI flags in the arg field with quotes and commas separated.
args = [“ — configFile”, “/local/Traefik.yml”]
In the snippet below we are configuring Traefik to use a configuration file using CLI flags
Template stanza
The template stanza creates files within the container. We have four template stanzas–the first three contain the details for Traefik to enable TLS certificates, and the fourth one contains the configuration file for Traefik.
We will cover the Traefik-specific configuration below.
Network stanza
Our Traefik Nomad job states that Traefik must use ports 80, 443 and 8082 using the network stanza. We statically assign the ports. When you use static in the port stanza, it means that Nomad won’t auto-assign a port. In this case, it means that container port 80 will map to port 80 on our browser.
Service stanza and Dynamic configuration
Consul looks for a service stanza in the Nomad jobspec, where health checks and tags (labels) are defined.
Traefik will read these tags out of Consul for its rules (dynamic configuration), We will see the tags in Consul CLI and the UI below, for now these tags will determine whether Traefik will forward the request to the containers. For example:
These tags are a set of rules or criteria that Traefik must meet to forward the traffic to your container instances. In this case, the container is Traefik itself. These rules can match multiple criteria. For example:
- The hostname or FQDN
- Protocol: HTTPS, HTTP, Any TCP or UDP port.
- Port Number: which frontend port number is used
- Is the front end responding to HTTPS, but the backend is using HTTP?
- And more
The tag is divided into a few parts. Let’s look at a specific example:
“traefik.http.routers.dashboard.entrypoints=websecure”
traefik: This is a text prefix. Traefik will look for any prefix with “traefik” to read configuration. If a tag contains anything but “traefik,” it will skip the tag. This prefix is configurable.http: Represents the protocol. In this case, we’re using the HTTP protocolrouters: Represents the service in Traefik. Other possible values include middleware, plugs-ing, etc. In this case, it is a router. More on this later.dashboard: The service identification. It’s usually the app name, but it can technically be anything you want, as long as it’s alphanumeric.entrypoint: This section represents the configuration of the service itself. In this case, our service is “Router,” and routing has the concept of endpoints. Endpoints are labels within the Traefik configuration pointing to network ports that will receive a package. In this case, the label is “websecure.” We will talk about what this means in detail later.
Hello-world Nomad Job
Network stanza
Using dynamic ports, we will allow Nomad to assign an externally-facing port. We just need to specify the application port.
Service Stanza
We will set our Traefik rules using Consul tags. We will explain this in detail below.
The critical thing to remember out of this section is Nomad sends service tags and network information (port and IP) of the containers to Consul. Meaning when you deploy a service stanza in Nomad, tags automatically are saved in Consul. Traefik dynamically configures itself by reading those tags and network information.
Step by step configuration
Now that we understand the basics of the Nomad job and the dynamic configuration. Let’s review the Traefik configuration in detail.
The default locations that Traefik looks for the configuration file are:
/etc/Traefik/$XDG_CONFIG_HOME/$HOME/.config/.(the working directory)
It looks for a file named Traefik.yml (or Traefik.yaml or Traefik.toml), and you can override it with
traefik — configFile=foo/bar/myconfigfile.yml
We override the configuration location to /local/traefik.yml
Configuration
Here’s the full code for the Traefik.yml file. We’ll break it down below.
Traefik is meant for routing, so let’s look at the router and load balancer components. Let’s start with the endpoints
Endpoints state which ports Traefik should listen to and assign a label to each. It also enables HTTP or TCP routers for each port.
In this case, it listens for ports 80, 443 and 8082 for incoming requests. The web, websecure and metrics labels are used in the dynamic configuration in the Consul tag. The tags look like this.
In the Nomad job
“traefik.http.routers.demo.entrypoints=web”
This is what it looks like in the Consul UI:
This is what it looks like in the Consul CLI:
Endpoint rule explanation
traefik= The prefix named Traefikhttp= use the HTTP protocol for the routerrouters= use the router’s servicedemo= it the service identificationentrypoint= which entry point label to use
The rule above states that the container only accepts requests that match the label entry point web map to port 80. In a few words, requests can only reach this container in port 80.
Router
A router is in charge of handling incoming requests to the containers using rules. While there is no router-specific configuration in Traefik, it uses the entry point to configure the routers.
Traefik will automatically route to 80 (“web” label), 443 (“websecure” label) and 8082 (“metrics” label) ports using HTTP or TCP, as per the configs above. The labels defined here come into play in our Traefik configs in our hello-world job, which we’ll see later.
Consul tags
Containers can use Consul tags (rules) to specify how Traefik should route to the containers. For example:
“traefik.http.routers.hello-world.rule=Host(`hello-world.domain.com`)”,
The Consul tag for a TCP router
“traefik.tcp.routers.otlp.rule=HostSNI(`*`)”,
By default, TCP and HTTP routers are enabled with entry points. TCP or HTTP routers do not require changes to entrypoint configuration. On the other hand, UDP routers require additional configuration.
If UDP routers configuration is required, it requires to include /udp as part of the port definition in the entrypoint
Avoid routing issues
Remember that HTTP, TCP, and UDP routers are exclusive to their ports and by default Traefik allows requests to route TCP and HTTP simultaneously. I highly discourage you from using both for the same port.
Routes can only support one protocol (HTTP, TCP or UDP) per port. The TCP or UDP layer understands port numbers only and cannot process requests based on FQDN or DNS.
Let’s look at an example with two containers.
Container 1. It uses an HTTP router, as HTTP routers are enabled by default without additional Traefik configuration. The HTTP router will read the DNS or FQDN and route the request to the container that contains “app1.domain.com” on port 80. Example rule:
“traefik.http.routers.app1.rule=Host(`app1.domain.com`)”,
“traefik.http.routers.app1.entrypoints=web”
webis just a label for the entry point and can therefore be called whatever, but it’s good to use an intuitive name.
Container 2. It uses a TCP router, TCP routers are also enabled by default without additional Traefik configuration, just like the HTTP routers. The TCP router will forward ANY requests that meet the port and TCP protocol.
“traefik.tcp.routers.app2.rule=HostSNI(`*`)”
“traefik.tcp.routers.app2.entrypoints=web”
HostSNI checks if the server name indication matches a domain. This case we are using a wildcard for all domains.
A request for app1.domain.com satisfies both rules: the app1.domain.com FQDN for container 1 and the ANY TCP for container 2. Because an HTTP request is also a TCP request, it will cause issues (conflicts) with Traefik and will result in unexpected behaviour if both containers use the same port 80.
UI/GUI
Traefik has a Web UI, which looks like this:
Enable the UI/GUI. We will touch on API configuration later on. For now, just know that enabling the API and the dashboard configuration. Enable the UI interface.
When configuring Traefik for the first time, you probably want HTTP instead of HTTPS for the Traefik dashboard. The insecure: true configuration allows for it. For production, however, you should set insecure: false and add a TLS certificate.
Ping
You should also enable Ping. Ping is always useful for network troubleshooting.
Logs and access logs
They are not the same thing. Access logs tell you “who calls whom,” — that is routing calls. Logs, on the other hand, cover well…everything else.
You can send the logs to a file or specify the format
To finalize the monitoring part, we will add Prometheus metrics. Metrics are used to record and transmit readings from your infrastructure over a period of time. They adhere to a specific format. In this case, we will use a well-known standard: Prometheus (which is both a data format and a product).
A significant amount of metrics can be extracted from Traefik and consumed for alerting and/or dashboards. The metrics require a port from which to export this data to your monitoring tool, the monitoring tool can scrape the data as frequently as needed.
We use the label metrics on the entryPoint definition. That way, anything hitting the metrics endpoint will be able to consume the Prometheus data emitted by Traefik: application status, how many requests are hitting the Traefik dashboard or a particular container, etc.
Ok, we explored the basic routing, troubleshooting and monitoring configuration. Let’s make it easy to use and a little more complex to configure. Let’s review the consul configuration
Providers
Here is the official explanation
The providers are infrastructure components, whether orchestrators, container engines, cloud providers, or key-value stores. The idea is that Traefik queries the provider APIs in order to find relevant information about routing, and when Traefik detects a change, it dynamically updates the routes.
You can think of them as plug-ins (though, Traefik plug-ins are different). Traefik has two Hashicorp Consul providers. It can be confusing because one of the providers is just called “consul” and another one is called “Consul Catalog”. They are not the same, but they both work with Hashicorp Consul.
Consul providers, allow Traefik access to a Hashicorp Consul server KV storage and ConsulCatalog exposes the services in a Hashicorp Consul server. We are interested in the ConsulCatalog.
Although we won’t use the consul provider, here what it does
Consul Provider
Hashicorp Consul has a storage backend called K/V (key/value) storage. It is a simple yet powerful secure storage that Traefik can use to save and consume data. Most of the time, you can save certs or sensitive data there that Traefik can use. The “rootKey” is a fancy name for a root folder. In this case, it’s just called Traefik.
Put simply, anything in the Consul KV Traefik folder is for Traefik to use. You need to provide an endpoint at which Consul can be reached. If you are using Nomad with the Consul (like I am) locahost:8500 is your Consul endpoint.
Ok, enough consul, let’s review what we’re really interested in, consulCatalog.
Consul Catalog Provider
Remember how we are using Consul tags to set our Traefik rules? This is because of the Consul Catalog provider. It queries Consul tags to dynamically configure container-specific rules
First, we need to go back to the API configuration
Enables the Traefik API engine. First, it allows the API to handle and use different provider APIs to interact with Traefik to set the rules and more. Some providers that use the API are Docker, Kubernetes or Consul Catalog and even the Traefik Dashboard.
To use the Consul tags, we need to configure the Consul Catalog provider.
Entrypoint defines how to communicate with Consul
scheme: Which protocol to be to connect to Consul. In this case, HTTP.address: I am using Nomad to run Traefik and Consul integrate nicely with Nomad on locahost:8500token: If you need authentication against Consul, you need a Consul token. You can create a new token using Consul UI, CLI or API call.
cache
To reduce Traefik requests to Consul, Trafik uses a local agent for caching catalogue reads and updates when something changes. To enable cache:
Prefix
This tells Traefik what to look for when reviewing the tags.
“traefik.tcp.routers.waypoint-internal.rule=HostSNI(`*`)”
For example, it only knows that this rule is meant for Traefik usage because it starts with traefik.
Expose by default
exposeByDefault. By default, Traefik will be enabled, read all tags and apply to route as soon as the tag is available in Consul. This can be problematic for production environments as you can run into routing issues by accident. exposeByDefault: false states that Traefik must have a Traefik.enable: true tag set to true before Traefik does anything. Once set to true Traefik will activate that service and its routing.
This configuration is great for production workloads. We don’t want to expose tags automatically. We want to be very intentional when enabling the Traefik configuration. This configuration requires a particular tag that enables Traefik
The tag in the container looks like this:
“traefik.enable=true”,
TLS configuration
The Traefik TLS configuration can be confusing at best. Hopefully, this saves you some trouble!
To start working with TLS, sometimes you want your front-end configuration to show the certificate, but want to skip the verification of your servers or containers certificates themselves.
This helps troubleshoot if your certificate issue is on Traefik or your application. Let’s start to disable the TLS check on the application. We will fix this at the end of this guide.
You need a file provider that points to a dynamic file. Dynamic files are files that Trafiek keeps an eye out for changes after starting the Trafik configure.
The content of the dynamic file can again be toml or yml. I use yml. The file must be saved inside the Traefik container.
Here is the breakdown of the dynamic.yml configuration
tls: start the configuration for TLScertificate: In Traefik, you can have different certs for different endpoints, services or containers.- The
certFileandkeyFilecombination can be repeated as many times as needed. A pair for each certificate. Traefik will match the host request and provide the certificate for the appropriate request.
But what do these files should look like
The lb.crt content must be included in the template stanza for lb.crt
The lb.key content must be included in the template stanza for lb.key
lb.keyIf you are wondering why you need to apply for the private key in both certFile and certKey, the only reason I can give you is that “it works.”
If you run into another configuration that works, please share it with me to update this guide.
TLS stores. You can save the certificate as the default certificate using the stores. You need to specify in both the store and the certificate sections for this to work.
After your TLS is up and running, I recommend dropping the insecure sections of your configuration, remove this from the configuration
The final configuration looks like this
Final thoughts
Let’s recap what we learned from this journey.
- Traefik allows to route to containers in Nomad by querying Consul
- Traefik dynamic configuration can be set using Nomad service tags via Consul
- Disable the TLS checks when testing and deployment but enable for production workloads.
- TLS certificates can be set via dynamic in Traefik in pem bundle format
After all this, the above configurations should make your Traefik work with Consul and TLS. There are many possible configurations of Traefik, and I encourage you to review them to fit your specific production needs. Traefik Configuration
Now that you have completed this guide, enjoy this field with a cow. Well earned. Because you know “a whole cow” of Traefik, Nomad, Consul and TLS now.
