Logging in Serverless Spark (Part2)

Overview

stderr: Executor print() and Driver print()

Log4j Logging

Setting Log Level to DEBUG

log4j = sc._jvm.org.apache.logging.log4j
log4j.core.config.Configurator.setLevel("com.ibm.stocator",log4j.Level.DEBUG)

Turning off Spark Logs

log4j = sc._jvm.org.apache.logging.log4j
log4j.core.config.Configurator.setRootLevel(log4j.Level.OFF)

Conclusion

--

--

--

Senior Consultant, IBM Cloud. Sharing titbits of epiphanies...

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Looting GraphQL Endpoints for Fun and Profit

Klingon… or Python args and kwargs

My Path To Custom Web App Development

Linux Filesystem and User Permissions

Configure and Deploy Your Service with Helm

What’s new in Boost Note in Sep: Increased the free plan, Page load speed improvement, Desktop…

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Mrudula Madiraju

Mrudula Madiraju

Senior Consultant, IBM Cloud. Sharing titbits of epiphanies...

More from Medium

Logging in Serverless Spark (Part3)

Create Presto Views in Glue and query using Athena

Interacting Redshift using EMR on EKS

from the above table we can easily understand that 1st offset will process the 24 rows(4+12+08)…