Provisioning Analytics Engine — Serverless Spark using Terraform

Terraform — Overview

Terraform is an open source software from Hashicorp that enables you to easily provision and manage resources of your cloud provider using a simple declarative json like configuration files to define your requirements.

Terraform simplifies the provisioning aspects especially when you have to deal with multi clouds and multiple environments. Cloud providers have the predefined templates and plugins that makes it easy to be able to provision, modify and delete infrastructure resources.

Step1:- Install the Terraform plugin

Based on your platform — you can install the right version of Terraform plugin. Choose one from here.

On my linux based system, these are the steps, I executed:

wget https://releases.hashicorp.com/terraform/1.1.7/terraform_1.1.7_linux_amd64.zipunzip terraform_1.1.7_linux_amd64.zipexport PATH=$PATH:$HOME/terraform

For more details see here.

Once you install, you can execute the terraform binary and check if it is working fine:

Step2:- Configure the IBM Cloud Provider Plugin

To access the IBM Cloud and to be able to provision resources — you need to provide the required credentials, i.e the IAM APIKey. Additionally you need to provide the IBM Cloud Provider Plugin files.
The relavent files are provider.tf, versions.tf and terraform.tfvars.

provider.tf

variable "ibmcloud_api_key" {}
variable "region" {}
provider "ibm" {
ibmcloud_api_key = var.ibmcloud_api_key
region = var.region
}

versions.tf

terraform {
required_version = ">=1.0.0, <2.0"
required_providers {
ibm = {
source = "IBM-Cloud/ibm"
}
}
}

terraform.tfvars
This contains the APIKey of cloud.ibm.com account where you want to create the instance of Analytics Engine

ibmcloud_api_key = "blahablahblahblahblah"
region = "us-south"

Step3:- Analytics Engine Serverless Terraform provision payload

Depending on the IBM Cloud resource that you are trying to create, you the terraform file will change. In this case, we are creating the Analytics Engine Serverless instance.

ae.tf

data "ibm_resource_group" "group" {
name = "test"
}
resource "ibm_resource_instance" "resource_instance" {
name = "my-demo-terraform-test"
service = "ibmanalyticsengine"
plan = "standard-serverless-spark"
location = "us-south"
resource_group_id = data.ibm_resource_group.group.id
tags = ["test"]
parameters_json = <<PARAMETERS_JSON
{
"default_runtime": {
"spark_version": "3.1"
},
"instance_home": {
"region": "us-south",
"endpoint": "https://s3.us-south.cloud-object-storage.appdomain.cloud",
"hmac_access_key": "9dasdfasdfasdfasdfasdfasdf9e4b1",
"hmac_secret_key": "9dasdfasdfasdfasdfasdfasdf9e4b1"
}
}
PARAMETERS_JSON
}

Step4:- terraform init

Initializing of the plugins here..

Step 5:- terraform plan
The plan action shows what will happen when you actually execute

Step 5:- terraform apply

This actually executes the action — provision of the service instance happens here.

Step6: terraform show

You can do a terraform show to check the instance details.

Done! Your instance is ready for Spark application submissions!

Click here to learn more.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store