Automating DevOps Workflows with GitLab and Terraform

Overview

GitLab is a complete DevOps platform delivered as a single application. GitLab provides a single UI for development and operational teams within organizations to work concurrently throughout the application development to delivery life cycle. As a result of applications becoming more and more complex, organizations turn to automation, like CI/CD, in hopes to streamline processes. However, to increase reliability and consistency in automation beyond software development, more and more teams are resorting to declarative automation and their respective workflows in code and storing them in code repositories. Often and dynamically triggering automation as code to quickly iterate and deliver value into production is becoming more and more popular. This phenomenon is commonly known as GitOps, where tools such as Terraform and GitLab shine.

Google Cloud provides scalable cloud infrastructure with managed services for various forms of compute, storage, networking, etc. With extensive open APIs, almost any and every aspect of it can be automated and managed via code the proper tools. Organizations looking to adopt cloud native solutions in an agile way need a better way to manage the inherent complexities when elastically scaling their services, breaking up monolithic applications, and securely operating at speed through cross functional teams. All of the prior has been the foundation of the DevOps transformation as we recognize today.

The fundamental challenge DevOps attempts to address is the unification of Developer and Operator workflow. When automating the application development process, Continuous Integration / Continuous Delivery (CI/CD) is crucial in iteratively deploying code reliably, in a secure fashion from source code into production. When automating Infrastructure administration and provisioning, Infrastructure as Code (IaC) is an emerging method to quickly and reliably build deployment targets for applications. IaC enables practitioners to deploy infrastructure through declarative means and maintain the desired state and lifecycle of cloud resources.

GitLab is the single application that provides all of the necessary functionality for DevOps teams out of the box for a seamless, low maintenance, just-commit-code software development, and delivery experience. GitLab enables teams to plan sprints and projects, organize code in git repositories, operationalize application code with CI/CD pipelines, secure source code through various vulnerability scanning, and much more. GitLab provides means to construct flexible automation workflows that are complementary to other tools like Terraform. Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently. The infrastructure Terraform can manage includes low-level components such as compute instances, storage, and networking, as well as high-level components such as DNS entries, SaaS features, etc. Together, GitLab and Terraform can be configured together to provide DevOps teams the capability to manage their cloud through IaC, continuously and reliably.

Sample Application Overview

We will be using a sample application that we have called Vote-App which is representative of a N-Tier microservice architecture, containerized application that will be deployed to a Kubernetes cluster provided by GKE. Below is a high level overview of the sample application architecture:

Vote UI: Is a python containerized microservice using flask to generate a front end UI for end users to place a ‘vote’. This service will be running on GKE.

Results UI: Is a NodeJs containerized microservice used to generate a front end UI for end users to see voting results. This service will be running on GKE.

Redis: This will be a managed Redis service powered by Cloud Memorystore which will serve as a cached queuing system for votes. This managed service will store aggregated votes inserted by the *Vote UI *service.

Postgres: This will be a managed Postgres service powered by Cloud SQL which will serve as a central database for all votes casted and accounted for. This managed service will be queried by the Results UI to show results.

Worker: This Is a .Net Core containerized microservice that queries votes in the que stored in Redis cache and enters them into the Postgres database. This service will be running on GKE.

DevOps Workflow Overview

You will have access to two GitLab users, Developer and Operator. After initial GitLab configuration and post importing of all relevant projects, you will role play deploying microservices and managing Infrastructure via IaC automated through CI/CD from code commit to running in production.

You will start with only a single, pre-provisioned GKE cluster as a deployment target for your microservices. In this example, we will simply be committing to the Master branch to deploy into a single staging environment. The high level workflow is as follows:

• The user will commit code to the application code repositories to scan, package and deploy the service.
• This will kick off a series of CI/CD pipelines to
• Execute code security scanning for vulnerabilities
• Package the code into a docker container
• Trigger a child pipeline to provision dependent infrastructure.
• Deploy all necessary Kubernetes resources (pods, services, secrets, etc.) to GKE.

The Terraform child CI/CD pipeline stored will:

• Use Terraform to init, plan, and apply to provision a Memstore instance and a Cloud SQL Postgres Instance on GCP.
• The CI/CD pipeline will also update and deploy a ConfigMap and Secrets resource to Kubernetes that the microservices will reference for appropriate DB endpoints and credentials.
• Lastly, the services will then be deployed to Kubernetes and be accessible through a web endpoints by the end user.

The workflow will be concerned with a single GitLab subgroup spanning multiple GitLab Projects outlined below. Each project listed below is a git repository with a .gitlab-ci.yml file already defined for an automated pipeline to execute upon code commit.

Objectives

We will:

• Configure relevant CI environment variables within GitLab projects to provide appropriate runtime information for Terraform CLI
• Add an existing Kuberenetes cluster as a deployment target in GitLab via GitLab’s Kubernetes integration
• Trigger parent-child CI/CD pipelines through new Merge Requests that call on pre-defined CI templates and evoke an Infrastructure as Code workflow via Terraform
• Learn how to organize GitLab Groups with multiple projects per microservice when developing and operating an N-Tier cloud native application

You can list the active account name with this command:

gcloud auth list

You can list the project ID with this command:

gcloud config list project

Login and Configure GitLab

In this section, you will login to GitLab for the first time using credentials provided to you.

1. First, let’s identify the region and zone the lab has provisioned resources in. In the Cloud Console, navigate to Kubernetes Engine > Clusters.

You will see 1 cluster provisioned in a specific zone. The cloud Region in this example is us-west1, while the cloud Zone is us-west1-c. It should look something like this:

2. Using your location information from your GKE cluster, enter in your region and zone values in the Cloud Shell with the following commands:

gcloud config set compute/region <GCP_REGION>
content_copy
gcloud config set compute/zone <GCP_ZONE>
content_copy

Additionally, we will also set the context for kubectl in Cloud Shell for future use. In this lab, a GKE cluster by the name of gitlab-gke-cl has been pre-provisioned. You can optionally verify that the cluster has been created and ready with gcloud container clusters list command.

3. Set the kubectl context for the gitlab-gke-cl cluster by entering the following command:

gcloud container clusters get-credentials gitlab-gke-cl
content_copy

Login to GitLab

1. In Google Cloud Console, navigate to Storage > Browser in the left navigation pane to view pre-provisioned GCS buckets.

2. In the Storage browser, you will notice a few GCS buckets have been created. There is one in particular with the following naming convention: qwiklabs-gcp-XX-XXXXXXXXXX-lab-info which will have a .txt file with the relevant login information to authenticate to GitLab. Click the first bucket with lab-info suffix.

3. Click on the file lab-info.txt.

4. Right click to ‘open in a new tab’ on the Link URL to view the contents of the file.

In the file you will see the following values that will be needed in the subsequent steps:

• GitLab Login URL
• GitLab User Name
• GitLab Password
• Vault Login URL
• Vault Token

Here is what it should look like:

5. In a new browser tab, navigate to your GitLab Login URL, login with GitLab Username root with the corresponding provisioned GitLab Password.

6. Navigate to Groups > Explore Groups from the top menu bar.

You will see the Voting-App group in the list.

7. Click it to enter the group and see the 4 projects (code repositories) within.

Add a GKE environment to GitLab

In this step, we will associate a GKE cluster to the GitLab subgroup thereby enabling projects within the subgroup to inherit the cluster as a deployment target within CI/CD pipelines. Configuring this integration between Gitlab and GKE will automatically provide the context necessary for any kubectl commands with the proper configs within the CI/CD runtime environments for jobs where the environment is specified.

1. In the left hand navigation pane, at the voting-app group level, click on Kuberentes and then Add Kubernetes Cluster.

2. We will now add an existing cluster that came pre-provisioned with this lab environment. Click on Add existing cluster tab.

Note: If you did not already have a K8s cluster to add, you can click on Google GKE and follow the wizard to provision a new GKE cluster to deploy to. In this lab, a cluster has already been provided to add.

3. Fill out the following information:

• For Kubernetes Cluster Name use Production
• For Environment scope use *
• For API URL, navigate to the Cloud Shell and enter the following command to retrieve the URL:

kubectl cluster-info | grep 'Kubernetes master' | awk '/http/ {print $NF}'
content_copy

The output of this command is the value that needs to be entered into GitLab for the cluster’s API URL parameter. It should look something similar to https://104.154.20.176

• For CA Certificate, first list the secrets with:

kubectl get secrets
content_copy

The output of this command should be something similar to default-token-xxxxx. Copy that token name for use below. Now, get the certificate by running this command, replacing <SECRET_NAME> with the name of the token you just got above:

kubectl get secret <SECRET_NAME> -o jsonpath="{['data']['ca\.crt']}" | base64 --decode
content_copy

Here’s a sample output:

> -----BEGIN CERTIFICATE-----
MIIDCzCCAfOgAwIBAgIQKk3UV4TSri/Q9rvQ1xoY7zANBgkqhkiG9w0BAQsFADAvMS0wKwYDVQQDEyQwYTU1NjY3Ny1kZDZkLTQ5NzYtOGM5OC03N2FlZDRjMWJlY2MwHhcNMjAwMzA1MjEwNjA1WhcNMjUwMzA0MjIwNjA1WjAvMS0wKwYDVQQDEyQwYTU1NjY3Ny1kZDZkLTQ5NzYtOGM5OC03N2FlZDRjMWJlY2MwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDEYryvktxuzKhvHJ8p0ktwJC3wnLLSv9H9S/MXWYa49otck7/wsUtXG7kIT0pDO/Xh8ipxR80zBh2fjJyuyHe6QDxo9lfzBy0ip32oFnOCRclnwFrHUxsJWCrh/TgWO35X/3xtQTeglsH/9+LbTurOESw/EMCxH3/55OBlOpF7bHDTzZ0cD1/eRM5ywj9t2XIwmySRCUGOoUdPb7FY3Y7x++TizgM07Y9Vdvm7C/FWh/0XHKEtOTUcBOS+IFlELuGkTwAiOIH88nDPmXCILzOsmyC8KKPV7t0N1uK7MRShaoxYkL1+L5N+R07xrcueqln4lCEmcm7v8eEuinoDJxZ3AgMBAAGjIzAhMA4GA1UdDwEB/wQEAwICBDAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IBAQASNnA1XJg06V5nBXHsIpyXZ8VvGAFNZuIi+J01396Iyrbk3JAYqsdxo7fENyK16AZIy/rq8FJ3MbIQm+PqeX1iSZ/mW2dLZ7B29VVcu+OuWq8/avG3SiiKuBwMYdgUbYCvPYs7hTKkigkbb4TH1ra2LsMLDTBDKz5nGQU4ron00HJUofBk+JY0KG+q7m10X2I2za2P4w/+HCpPRAS+aPxVQDRb/EExoy5SKwAH4YarQDFzfNLEljdw1FEy4MiYRN7HabYD/SWkjVfU/Bk5PWyATl6FBdVgIB440A+9vxNZX6aFPA9v6gD0gWj6ivlX/UMGifmPpj7VvVczYTO1mBq/
-----END CERTIFICATE-----
content_copy

Copy the certificate from your terminal and paste it into the CA Certificate field. Make sure include both the ---BEGIN CERTIFICATE--- and the ---END CERTIFICATE--- lines as shown in the expanded view.

• Service Token: GitLab authenticates against Kubernetes using service tokens, which are scoped to a particular namespace. The token used should belong to a service account with cluster-admin privileges. A service account gitlab-admin has already been created for you in this lab environment. To retrieve the token for the gitlab-admin service account, enter the following commands in Cloud Shell:

SERVICE_NAME=$(kubectl -n kube-system get secret | grep gitlab-admin | awk '{print $1}')
kubectl -n kube-system describe secret $SERVICE_NAME | grep token: | awk '{print$2}'
content_copy

The output of the command will look something like this:

eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVyb
mV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJnaXRsYWItYWRtaW4tdG9rZW4tanF2Z2ciLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoiZ2l0bGFiLWFkbWluIiw
ia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQudWlkIjoiNjAyMzU3ZGMtNjFjZC0xMWVhLTk3N2MtNDIwMTBhODAwMWFlIiwic3ViIjoic3lzdGVtOnNlcnZpY2VhY2NvdW50Omt1YmUtc3lzdGVtO
mdpdGxhYi1hZG1pbiJ9.WMYyWqbK89QqZcyogilMHTyD5kmsxpD9DmZaWWZF919bvc4yGpypaumuV7WCnuz91oYcavqH6VYUZcys7SUKT1h7yIxM6qbOyyT4DHtQEBOuq7zAxB0UoEujDiyIyxmXFexH3qSGjmetvH4y4a8-C-N2uE
MEgEZ0vpOKRMhaTec-jBX3H0s6_oLDmrUAfRQkIm-rUYz7Z3QYaSIrBIJwLnbl4ZNw032nKDynQw_ZQtkmTOp6H9vXM_m8YWQgEkIfaABLXSx5qzIiJNWa3BB2A-Hkx8guBPudEIt8LH95rTPr7PnsqEJc5r3j15R1ln2CZhL37e8i
DnPK9wk5oxD6dQcontent_copy

Copy your resulting token string and paste it into the Service Token field within GitLab's UI. BE CAREFUL. Copying from the cloud shell inserts spaces at the lines breaks. You can use option + left arrow/right arrow to look through the string for spaces. When you see them, delete them, but realize that this will also stop on other word separators like - and . and _ which should NOT be deleted. Note that the token will not work if there are whitespaces anywhere in the token.

Make sure that the RBAC-enabled cluster and GitLab-managed cluster boxes are checked, then click Add Kubernetes cluster. Below is a representative snapshot of GitLab UI.

In the subsequent steps, you’ll see additional options to deploy Gitlab Managed Apps to the cluster.

4. First, go into the applications tab. Continue by installing Helm by clicking the Install button in the GitLab Applications wizard. This will take a moment to complete.

5. Once Helm is installed, install Ingress. This will allow dynamic FQDN endpoints for our deployments later in the lab. This too may take a few moments to install. When it’s done, it should look like the following:

Now you will notice the Ingress Endpoint IP address. We will use that to create a Base Domain for the cluster using qlencrypt.com IP DNS services.

6. Copy the Ingress Endpoint IP address. Go into the details tab above.

7. Next, paste it into the Base domain box, using the format <INGRESS_IP>.qlencrypt.com.

“Make sure to replace the periods (.) with dashes (-) in the IP address. So if the IP is: 34.83.80.30, then the Base Domain value must be: 34–83–80–30.qlencrypt.com.”

After you’ve modified the IP address, click Save changes. Here’s how it should look:

This concludes adding and configuring Kubernetes within the GitLab group. This will allow all subsequent child project repositories access to this environment during CI/CD jobs.

Configure Infrastructure Project for Terraform

In this section, we will configure the pre-requisite configurations necessary for our IaC pipeline to execute successfully. Below is the high level overview of the CI/CD pipeline that has been pre-defined in the Infrastructure project.

Terraform validates a plan to provision a Memstore instance, Postgres Cloud SQL instance and save a config map of key/values in Vault of the respective database endpoints as well as login credentials. This config map will then eventually be deployed along with the microservices vote-app, worker-app, and result-app so the services can query the databases when deployed to GKE. Terraform will also leverage GCS as the backend to store the state file which will correlate running state vs desired state of the provisioned infrastructure on every CI/CD run.

In the subsequent steps, we will define prerequisite environment variables necessary for Terraform to call Vault Google Cloud APIs. Traditionally, the Infrastructure project will be managed though your Operations teams that may enforce templetaization of key GitLab CI/CD jobs/steps along with ops managed Terraform modules that can be referenced and called by developers during AppDev CI/CD pipelines.

Configure CI/CD Environment Variables for Infrastructure Project

The Infrastructure project holds Terraform code which leverages Google credentials to provision in a given GCP zone a Memstore and CloudSQL instance as backend databases. Terraform saves those provisioned database endpoints into Vault as Config Map and associated secrets. The ConfigMap and Secrets will be consumed from Vault by application code during CI/CD pipelines within the deployment stage. Terraform also uses a GCS bucket to store it’s statefile to correlate desired state against running state when Terraform plan is evoked within the CI/CD pipeline. To enable the workflow, we need to configure environment variables so Terraform has the correct context to execute.

1. Navigate to the Infrastructure project via searching from the top navigation UI pane.

2. Once in the Infrastructure project, navigate to Settings UI pane. Click on Settings > CI/CD in the left navigation menu pane.

3. Expand the Variables section.

In this pane, we will define Key / Value pairs that define Environment Variables and respective values that will be consumed by code that is being executed in the CI/CD runtime environment. Below are the Variables (type) that will be defined. All variables for this tutorial will be not protected, not masked and the scope will be defined for all environments.

• Click Add Varible

• Click For first key, use GC_BUCKET. This is the lab bucket for remote storage of state.

The value of this variable will be defined as the name of the GCS bucket which will host the Terraform state file. The following command will save the output locally in the shell for use in subsequent steps as well. In the Cloud Shell, enter the following command:

echo "$(gcloud config list --format 'value(core.project)' 2>/dev/null)-terraform-backend"
content_copy

Use this output for your value. Leave the rest of the fields blank and click Add Variable.

• Click Add Variable again. The second key will be VAULT_ADDR. This is the address to Vault instance.

This value is defined in the lab-info.txt file that was created during lab provisioning in a GCS bucket which also contains GitLab root password. Navigate to the open tab with the file contents.

In this example, the value for VAULT_ADDR would be http://vault-35-184-80-200.qlencrypt.com.

Use your Vault Login URL provided in your file for the value and click Add Variable.

• The third key will be VAULT_TOKEN. This is the vault token with access to write to generic secret

This value is also defined in the lab-info.txt. Navigate to the open tab with the file contents.

Use your Vault Token provided in the file for your value and add the variable.

• The next key will be VAULT_SKIP_VERIFY. This skips SSL certification validation on Vault's end point. Set the value to be 1 to skip cert validation for this lab and add the variable.
• For the next key, use TF_VAR_project. This is the GCP project for lab.

This value is set as the GCP project resources are being deployed to.

Run the following command in the Cloud Shell to retrieve the correct value:

gcloud config list --format 'value(core.project)'
content_copy

It should be something similar to: “qwiklabs-gcp-04-dd8231f8cda3”. Use the output for your value and add the variable.

• The next key will be TF_VAR_zone. This is the GCP zone for lab

Here we will use the same zone the GKE cluster is deployed to. Enter the following command to retrieve the appropriate GCP zone.

gcloud container clusters list
content_copy

The output will be similar to:

NAME           LOCATION       MASTER_VERSION  MASTER_IP    MACHINE_TYPE   NODE_VERSION    NUM_NODES  STATUS
gitlab-gke-cl  us-central1-a  1.14.10-gke.17  104.154.20.176  n1-standard-4  1.14.10-gke.17  3          RUNNINGcontent_copy

You can find the Zone under the Location column in the output. In this example above, the value for TF_VAR_zone is: us-central1-a. Use your zone for the value and add the variable.

• Next, use the key TF_VAR_region. This is the GCP region for lab.

The GCP Region is the first two segments of the zone string delimited by a hyphen of the TF_VAR_zone value.

So in this example above, the value for TF_VAR_region is: us-central1. Enter in your zone for the value and add the variable.

• The next key is TF_VAR_password. This is the administrator password to the PostgreSQL database. Enter the value as Testing!123 and add the variable.
• The final key is GOOGLE_CREDENTIALS. This is the JSON payload with service account key for infrastructure. Must have minimum access to Kubernetes, Cloud SQL, and MemoryStore as administrator.

To obtain the value, we must follow a couple steps:

• Enter the following commands in Google Cloud Shell to see a list of existing service accounts. We will be recycling the Qwiklab service account that comes pre-provisioned with this lab.

gcloud iam service-accounts list
content_copy

The output will be similar to:

NAME                                    EMAIL                                                                              DISABLED
Compute Engine default service account  748659956488-compute@developer.gserviceaccount.com                                 False
Qwiklabs User Service Account           qwiklabs-gcp-04-dd8231f8cda3@qwiklabs-gcp-04-dd8231f8cda3.iam.gserviceaccount.com  False
App Engine default service account      qwiklabs-gcp-04-dd8231f8cda3@appspot.gserviceaccount.com                           False
vault service account                   vault-server@qwiklabs-gcp-04-dd8231f8cda3.iam.gserviceaccount.com                  False
content_copy

From the list, note your Qwiklabs User Service Account email value It will be used in the subsequent command.

In this example, the value is “qwiklabs-gcp-04-dd8231f8cda3@qwiklabs-gcp-04-dd8231f8cda3.iam.gserviceaccount.com”.

• Next, create a key which will enable Terraform to authenticate via the service account during CI jobs. We will store the key in a GCS bucket to view and copy into GitLab. The contents of that file will be set as the value of GOOGLE_CREDENTIALS.

In the command below, you will have to replace the <QWIKLABS_USER_SERVICE_ACCOUNT> string with your own Qwiklabs User Service Account email value retrieved from the prior step.

gcloud iam service-accounts keys create ./key.json --iam-account <QWIKLABS_USER_SERVICE_ACCOUNT>
content_copy

• Now, enter the following command to compact the JSON credentials into a string literal and copy it to the GCS bucket.

cat key.json | jq -c . > creds.json | gsutil cp creds.json gs://$(gcloud config list --format 'value(core.project)')-terraform-backend
content_copy

Next, navigate to the <PROJECT_ID>-terraform-backend bucket and click on creds.json. Open the Link URL file. Copy the entire JSON payload to paste into the GOOGLE_CREDENTIALS variable within GitLab.

Once completed, your GitLab configurations should look like the following screenshot:

Configure CI/CD Environment Variables for Application Code Projects

The vote-app, results-app, and worker-app projects hold application source code that are containerized and deployed as microservices. Each service depends on one or more of the backend databases to function properly. Each microservice is configured to rely on environment variables defined within the container at runtime and Kuberentes secrets to function properly. All relevant information is stored in Vault that will be called and used during the CI/CD pipeline. To enable the workflow, we need to configure environment variables so the applications can be deployed to Kubernetes correctly with the right resources.

Note: The environment variables being configured will be the same for all the code repositories. In subsequent steps, you will set values for VAULT_ADDR, VAULT_TOKEN, and VAULT_SKIP_VERIFY for each project.

Configure CI/CD Environment Variables for Vote App

1. Navigate to the vote-app project via searching from the top navigation UI pane.

2. Once in the vote-app project, navigate to Settings > CI/CD in the left navigation menu pane.

3. Expand the Variables section.

4. In this pane, we will define Key / Value pairs that define Environment Variables and respective values that will be consumed by code that is being executed in the CI/CD runtime environment. Below are the Variables (type) that will be defined. All variables for this tutorial will be not protected, not masked and the scope will be defined for All environments.

• Click Add Varible

• The first key will be VAULT_ADDR. This is the address to Vault instance.

This value is defined in the lab-info.txt file that was created during lab provisioning in a GCS bucket which also contains GitLab root password. Navigate to the open tab with the file contents.

Use your Vault Login URL provided in your file for the value.

• The second key will be VAULT_TOKEN. This is the vault token with access to write to generic secret.

This value is also defined in the lab-info.txt. Navigate to the open tab with the file contents.

Use your Vault Token provided in the file for your value.

• The next key will be VAULT_SKIP_VERIFY. This skips SSL certification validation on Vault's end point. Set the value to be 1 to skip cert validation for this lab.

5. Once completed, your GitLab configurations should look like the following screenshot:

Configure CI/CD Environment Variables for Result App

1. Navigate to the result-app project via searching from the top navigation UI pane.

2. Once in the result-app project, navigate to Settings > CI/CD in the left navigation menu pane.

3. Expand the Variables section.

4. In this pane, we will define Key / Value pairs that define Environment Variables and respective values that will be consumed by code that is being executed in the CI/CD runtime environment. Below are the Variables (type) that will be defined. All variables for this tutorial will be not protected, not masked and the scope will be defined for All environments.

• Click Add Varible

• The first key will be VAULT_ADDR. This is the address to Vault instance.

This value is defined in the lab-info.txt file that was created during lab provisioning in a GCS bucket which also contains GitLab root password. Navigate to the open tab with the file contents.

Use your Vault Login URL provided in your file for the value.

• The second key will be VAULT_TOKEN. This is the vault token with access to write to generic secret.

This value is also defined in the lab-info.txt. Navigate to the open tab with the file contents.

Use your Vault Token provided in the file for your value.

• The next key will be VAULT_SKIP_VERIFY. This skips SSL certification validation on Vault's end point. Set the value to be 1 to skip cert validation for this lab.

5. Once completed, your GitLab configurations should look like the following screenshot:

Configure CI/CD Environment Variables for Worker App

1. Navigate to the worker-app project via searching from the top navigation UI pane.

2. Once in the worker-app project, navigate to Settings > CI/CD in the left navigation menu pane.

3. Expand the Variables section.

4. In this pane, we will define Key / Value pairs that define Environment Variables and respective values that will be consumed by code that is being executed in the CI/CD runtime environment. Below are the Variables (type) that will be defined. All variables for this tutorial will be not protected, not masked and the scope will be defined for All environments.

• Click Add Varible

• The first key will be VAULT_ADDR. This is the address to Vault instance.

This value is defined in the lab-info.txt file that was created during lab provisioning in a GCS bucket which also contains GitLab root password. Navigate to the open tab with the file contents.

Use your Vault Login URL provided in your file for the value.

• The second key will be VAULT_TOKEN. This is the vault token with access to write to generic secret.

This value is also defined in the lab-info.txt. Navigate to the open tab with the file contents.

Use your Vault Token provided in the file for your value.

• The next key will be VAULT_SKIP_VERIFY. This skips SSL certification validation on Vault's end point. Set the value to be 1 to skip cert validation for this lab.

5. Once completed, your GitLab configurations should look like the following screenshot:

Trigger a Parent-Child Pipeline

Functionally, when adopting DevOps, organizations need to bridge the gap between Operational teams and Application Development teams through tighter communication, better visibility, and overall with intimate collaboration. The Voting-App GitLab group has been organized in such a way that it has dedicated projects for application code and the infrastructure code. With GitLab’s Parent-Child Pipelines, an application developer can focus writing code while defining their CI/CD pipeline using templates that are pre-defined by operations team which adhere to internal policies. Infrastructure teams therefore can take ownership of defining GitLab CI templates to execute reliable, approved, and consistently automated terraform workflows that gets triggered when code is committed and ready to be pushed out into a live environment.

Below is a high level of the workflow you will be executing.

• Create an MR in the worker-app project which will kick off an MR pipeline.
• You will then merge the MR branch with master to kick off the Master branch pipeline that will evoke terraform to provision dependent infrastructure and deploy the service into production
• Create an MR in the vote-app project which will kick off an MR pipeline. In this MR pipeline you will have a Review App deployed to visually inspect the web UI of the service.
• You will then merge the MR branch with master to kick off the Master branch pipeline that will evoke terraform to provision dependent infrastructure and deploy the service into production. You will then be able to access the application running in Production.
• Create an MR in the result-app project which will kick off an MR pipeline. In this MR pipeline you will have a Review App deployed to visually inspect the web UI of the service.
• You will then merge the MR branch with master to kick off the Master branch pipeline that will evoke terraform to provision dependent infrastructure and deploy the service into production. You will then be able to access the application running in Production.

Triggering worker-app CI/CD Pipeline

1. First, we will trigger the CI/CD pipeline for the worker-app. Navigate to the worker-app project via searching from the top navigation UI pane.

2. Next, navigate to Merge Requests from the side menu pane and click on the New Merge Request UI button.

3. Select mr-flow-testing branch as the Source Branch.

4. Select master branch as the Target Branch.

5. Click Compare Branches and Continue UI button. Provide the Title as Testing MR Flow for GitOps and leave the description as is.

6. Click Submit Merge Request at the bottom of the page.

7. Next, click on Open Web IDE UI button to open the editor to simulate code change and trigger a CI/CD pipeline for the MR Branch.

8. Navigate and click on the .gitlab-ci.yml file. This file outlines the logic for the CI/CD pipelines for this particular project. Once selected, take your cursor and add a space to the end of the file after the work terraform. This simple space character doesn't affect the function of the pipeline but induces a change in the file (marked by the orange circle) that will be committed to the repository.

9. Click the Commit... UI button at the bottom left of the page to continue.

10. Make sure that Commit to mr-flow-testing is selected and click the Commit UI button.

11. Next, navigate to the pipeline that was just kicked off. At the top right of the page, click the Rocket Ship icon UI button to extend the side pane. Next click the #1 to view the pipeline.

• You should see an executing pipeline which consists of the Security stage and the Package stage. At this point, since this is an MR branch, nothing has been deployed to production. Over time, both jobs will pass successfully. This pipeline simply tests that all jobs prior to deploying in the working MR branch are successful.

12. Next we will merge this branch into master to kick off the broader pipeline and push latest code into production. Navigate back to the MR by clicking the !1 Testing MR For for GitOps hyperlink in the middle of the page indicated by 1 related merge request text.

Note: to explore a bit, you can browse the Security tab to see the outputs of the Security scanning for additional details on any detected vulnerabilities.

13. Click on the green Merge UI button to merge with master.

14. You will notice a new pipeline being created. Navigate to that pipeline by clicking on the #2 hyperlink text.

• Here you will notice a slightly different pipeline being triggered which evokes the Terraform job triggering the Infrastructure child pipeline downstream. Additionally, there is a Deploy stage which deploys the worker-app microservice to kubernetes.

15. Since the worker app does not have a front end UI, there is not a service endpoint to query to visualize the deployment. Instead, you can validate the deployment is running in GKE via the Cloud Shell. Once the pipeline is finished, enter the following command to confirm the deployment has successfully started:

kubectl get deployments -n worker-app-1-production
content_copy

Output:

NAME     READY   UP-TO-DATE   AVAILABLE   AGE
worker   1/1     1            1           2mcontent_copy

Triggering vote-app CI/CD Pipeline

1. Now we will trigger the CI/CD pipeline for the vote-app. Navigate to the vote-app project via searching from the top navigation UI pane.

2. Next, navigate to Merge Requests from the side menu pane and click on the New Merge Request UI button.

3. Select mr-flow-testing branch as the Source Branch.

1. Select master branch as the Target Branch.

4. Click Compare Branches and Continue UI button. Provide the Title as Testing MR Flow for GitOps and leave the description as is.

5. Click Submit Merge Request at the bottom of the page.

6. Next, click on Open Web IDE UI button to open the editor to simulate code change and trigger a CI/CD pipeline for the MR Branch.

7. Navigate and click on the .gitlab-ci.yml file. This file outlines the logic for the CI/CD pipelines for this particular project. Once selected, take your cursor and add a space to the end of the file after the work terraform. This simple space character doesn't affect the function of the pipeline but induces a change in the file (marked by the orange circle) that will be committed to the repository.

8. Click the Commit… UI button at the bottom left of the page to continue.

9. Make sure that Commit to mr-flow-testing is selected and click the Commit UI button.

10. Next, navigate to the pipeline that was just kicked off. At the top right of the page, click the Rocket Ship icon UI button to extend the side pane. Next click the #<number> to view the pipeline. Note: your pipeline number may differ than the one showed in this screenshot.

You should see an executing pipeline which consists of the Security stage and the Package stage and the Review stage. At this point, since this is an MR branch, nothing has been deployed to production per-say, but the application containers have been deployed to the cluster and exposed for visual inspection. This is mimicking a common stage in the CI/CD pipeline where applications are deployed to a staging environment or to a local developer environment for UI testing. Over time, all the jobs will pass successfully.

11. Next we will visually review the application’s UI and then merge this branch into master to kick off the broader pipeline to push latest code into production. Navigate back to the MR by clicking the !1 Testing MR For for GitOps hyperlink in the middle of the page indicated by the related merge requests text.

In the MR UI, you’ll notice there is a new button: View App available. This will redirect you to the running application to visually inspect it at run time. This is a mock "staging" environment where only the vote-app microservice is deployed along with an ephemeral Redis container for it's back end. This is simply for the "Developer" working in the feature branch to access code at runtime.

12. Click the View App button to open the front end service in a new tab.

You’ll notice a webpage where you can cast a vote: “Cat” or “Dog” at the following URL http://vote-review.<ip-address>.qlencrypt.com.

13. Assuming that the visual inspection of the front end service checks out, we’re ready to push this microservice into production. Navigate back to the Merge Request UI and continue by clicking the green Merge button in the center of the screen.

14. This will kick off a master branch pipeline. Click the #<number> hyperlik under the Merged UI description shown below to see additional details regarding the pipeline.

In this pipeline, you will notice that there are many more stages and jobs being executed. The code is being scanned for security vulnerabilities, packaged into a container, deployed for visual inspection, calling a child pipeline to trigger terraform to validate desired state of infrastructure and finally deploying the app into production.

15. Near the top of the page under the Merge Branch description, click the hyperlink text !1, which will redirect you back into the Merge Request UI. Note, your number here may differ than the screenshot.

16. In the Merge request, you’ll notice additional review apps to investigate. Click on the view app for the production environment.

17. You’ll notice a webpage where you can cast a vote: “Cat” or “Dog” at the following URL http://vote<ip-address>.qlencrypt.com. Note that the "review" word missing from the FQDN compared to the app when previously deployed. Go ahead and caste a vote for CATS.

Note: keep this window open in a separate tab for subsequent section.

This concludes the deployment of the vote-app microservice into production though a fully executed pipeline.

Triggering results-app CI/CD Pipeline

1. Lastly, we will trigger the CI/CD pipeline for the result-app. In GitLab, navigate to the result-app project via searching from the top navigation UI pane.

2. Next, navigate to Merge Requests from the side menu pane and click on the New Merge Request UI button.

1. Select mr-flow-testing branch as the Source Branch.

3. Select master branch as the Target Branch.

4. Click Compare Branches and Continue UI button. Provide the Title as Testing MR Flow for GitOps and leave the description as is.

5. Click Submit Merge Request at the bottom of the page.

6. Next, click on Open Web IDE UI button to open the editor to simulate code change and trigger a CI/CD pipeline for the MR Branch.

7. Navigate and click on the .gitlab-ci.yml file. This file outlines the logic for the CI/CD pipelines for this particular project. Once selected, take your cursor and add a space to the end of the file after the work terraform. This simple space character doesn't affect the function of the pipeline but induces a change in the file (marked by the orange circle) that will be committed to the repository.

8. Click the Commit... UI button at the bottom left of the page to continue.

9. Make sure that Commit to mr-flow-testing is selected and click the Commit UI button.

10. Next, navigate to the pipeline that was just kicked off. At the top right of the page, click the Rocket Ship icon UI button to extend the side pane. Next click the #<number> to view the pipeline.

You should see an executing pipeline which consists of the Security stage and the Package stage and the Review stage. At this point, since this is an MR branch, nothing has been deployed to production per-say, but the application containers have been deployed to the cluster and exposed for visual inspection. This is mimicking a common stage in the CI/CD pipeline where applications are deployed to a staging environment or to a local development environment for testing / inspection. Over time, all the jobs will pass successfully.

11. Next we will visually review the application’s UI and then merge this branch into master to kick off the broader pipeline to push latest code into production. Navigate back to the MR by clicking the !1 Testing MR For for GitOps hyperlink in the middle of the page indicated by the related merge requests text.

In the MR UI, you’ll notice there is a new button: View App available. This will redirect you to the running application to visually inspect it at run time. This is a mock "staging" environment where only the vote-app microservice is deployed along with an ephemeral Redis container for it's back end. This is simply for the "Developer" working in the feature branch to access code at runtime.

12. Click the View App button to open the front end service in a new tab.

You’ll notice a webpage where you can see the vote results between “Cat” or “Dog” at the following URL http://result-review.<ip-address>.qlencrypt.com.

13. Assuming that the visual inspection of the front end service checks out, we’re ready to push this microservice into production. Navigate back to the Merge Request UI and continue by clicking the green Merge button in the center of the screen.

14. This will kick off a master branch pipeline. Click the #<number> hyperlink under the Merged UI description shown below to see additional details regarding the pipeline.

In this pipeline, you will notice that there are many more stages and jobs being executed. The code is being scanned for security vulnerabilities, packaged into a container, deployed for visual inspection, calling a child pipeline to trigger terraform to validate desired state of infrastructure and finally deploying the app into production.

15. Near the top of the page under the Merge Branch description, you will see hyperlink text !1 which will redirect you back into the Merge Request UI.

16. In the Merge request, you’ll notice additional review apps to investigate. Click on the view app for the production environment.

17. You’ll notice a webpage where you can see the results of casted votes between “Cat” or “Dog” at the following URL http://result-[ip-address].qlencrypt.com. Note that the ‘review' word missing from the FQDN compared to the app when previously deployed. You will also notice that CATS is showing to be at 100% with 1 votes. This is because in the previous section, we casted the vote in production which is now being reflected by the recently deployed result-app microservice.

Optional: you can navigate to “http://result-[ip-address].qlencrypt.com" in a separate tab and cast a vote for “DOGS” to investigate the effect on the ‘results’ front end service.

This concludes the deployment of the results-app microservice into production though a fully executed pipeline.

Summary

We easily configured Terraform plans in GitLab SCM and executed plans via GitLab CI/CD for a Infrastructure as Code (IaC) workflow. Then leveraged Parent-Child pipelines in GitLab CI/CD to trigger IaC pipelines to validate deployment targets and/or provision prerequisites. Lastly, leveraged GitLab with Terraform (and Vault) for an end-to-end DevOps workflow focused around CI/CD automation.

Refer:

GitLab CI/CD

Documentation - Terraform by HashiCorp