Deploy Mage on GCP in a Flash with Terraform

Want to harness the power of Mage, the versatile pipeline orchestration tool, on Google Cloud Platform (GCP)? Buckle up, data enthusiasts, because deploying Mage with Terraform can be quicker than you think!

What is Mage?

Mage is a user-friendly tool that helps you build, schedule, and manage complex data pipelines with ease. It simplifies the orchestration process, allowing you to focus on the logic of your pipelines rather than tedious infrastructure management.

What is Terraform?

Terraform is an Infrastructure as Code (IaC) tool that enables you to define and manage your cloud infrastructure through code. By using Terraform, you can automate the creation and deletion of resources, ensuring consistency and repeatability.

Why deploy Mage on GCP with Terraform?

Combining Mage with Terraform offers several advantages:

• Fast and efficient deployment: Terraform automates the infrastructure provisioning, saving you time and effort.
• Simplified management: Manage your Mage deployment alongside your other GCP resources using Terraform.
• Scalability: Easily scale your Mage setup up or down as your needs change.
• Repeatable deployments: Ensure consistent deployments across different environments using Terraform code.

Prerequisites:

Before we dive in, gather your essentials:

• Terraform: Ensure you have Terraform installed and configured on your local machine.
• gcloud CLI: Set up the gcloud CLI and authenticate with your GCP account.
• Google Cloud Permissions: Make sure your service account has the necessary permissions:

1. Artifact Registry Reader
2. Artifact Registry Writer
3. Cloud Run Developer
4. Cloud SQL Administrator
5. Service Account Token Creator

The Deployment Dance:

Now, let’s waltz through the steps:

1. Verify gcloud: Check gcloud’s functionality by running:

gcloud storage ls

A list of your buckets should appear, confirming gcloud is ready to rock.

2. Fetch the Mage Terraform Templates: Clone the official repository from

https://github.com/mage-ai/mage-ai-terraform-templates.git

Remember, Terraform automates infrastructure creation and management through code — a valuable tool in our deployment arsenal.

3. Navigate and Choose: Dive into the list_of_cloud_providers directory and select the GCP template.

4. Customize Your Configuration: In the variable.tf file, tailor the variables (like project ID and region) to match your GCP environment.

5. Initialize Terraform: Run terraform init to download and install any required dependencies.

6. Preview and Deploy: Double-check everything with

terraform plan

Satisfied?

Proceed with

terraform apply

to deploy the infrastructure according to your configuration.

After successful creation of resources.

7. Mage Materializes: Once the apply command completes, resources are successfully created, and Mage is up and running on GCP!

8. Cloud Run Rendezvous: Head to Google Cloud Run. You’ll find an app named after your specified name in the Terraform template.

Security Note: Initially, access to the service is restricted. While we’ll enable access for demonstration purposes, remember that granting public access can pose security risks. In production environments, implement appropriate access controls.

9. Unlocking Access: In the Cloud Run console, navigate to the “Networking” tab and select “Allow direct access to your service from the internet.” Click “Save” to unlock the Mage app.

10. Welcome Home: Hit the provided URL, and voila! You’ll be greeted by the Mage homepage.

Cloud Storage Advantage: With files stored in Cloud Storage, your pipelines keep running as long as the service is active.

Wrapping Up:

Terraform simplifies Mage deployment on GCP, providing a structured and efficient approach. Remember to destroy resources with

terraform destroy

when you're done to avoid unnecessary charges.