How to Push Data Events to a Kafka Topic, Read Them Using SparkStreaming and Persist to ObjectStore — Contents Terminology Basics Brush Up Create “Topic” in IBM Event Streams Run a DataProducer Application That Generates Events Create and Run Spark Streaming Application Check Data Persisted on Bucket in IBM Cloud Object Storage End Notes Terminology Basics Brush Up Event Streaming: “Event streaming is the practice of capturing data in real-time from event sources…

Terraform — Overview Terraform is an open source software from Hashicorp that enables you to easily provision and manage resources of your cloud provider using a simple declarative json like configuration files to define your requirements. Terraform simplifies the provisioning aspects especially when you have to deal with multi clouds and multiple environments…

Export Logs for each application Overview In Part1 and Part2 of this series, we saw some fundamental usage of logging in Analytics Engine Serverless Spark. For simple applications, you can look at the logDNA interface for troubleshooting or analyzing how it performed. …

Learn how to tune logging : Executor vs Driver logs, Log Levels and Turning off Logs Overview In Part1 of this series we saw how the hooking up Log DNA with Analytics Engine service enables you to do simple troubleshooting and see the output of your print() or show() commands in…

What can logs tell you about user errors (and output)? Introduction Analytics Engine in cloud.ibm.com is a managed service that gives you a way to quickly submit spark applications without having to bother setting up and installing Spark. When you submit a Python, Scala or R application, a Spark cluster gets launched in the background, executes your code against the given…

A fun take, drawing parallels between Pizza baking and running a Spark application — Even if you aren’t into pizzas, I am sure you have a vague idea of what goes into making them. This quick (and hopefully) easy read is a fun take at how you can draw parallels between baking a pizza and running of an Apache Spark application! Okay! So here…

Working With External Metastore — Overview Spark SQL uses Hive Metastore to manage the metadata of user’s applications tables, columns, partition information. By default, the database powering this meta store is an embedded Derby instance that comes with Spark cluster. You could choose to externalize this meta store DB to an external relational database instance outside…

Run Pay-Per-Second, On-Demand, Spark Workloads — Overview IBM Analytics Engine’s recently launched new plan Standard Serverless for Apache Spark, comes with a host of exciting new features for teams that need to run BigData, Machine Learning, Spark based workloads. You, as a Developer, will dig: Running on latest and greatest versions :- Spark on Scala, Python, R …

How to Provision IBM Cloud Analytics Engine instance Terraform — Overview Terraform is an open source software from Hashicorp that enables you to easily provision and manage resources of your cloud provider using a simple declarative json like configuration files to define your requirements. In general, using Terraform simplifies the provisioning aspects especially…