Logging in Serverless Spark (Part2)

Overview

stderr: Executor print() and Driver print()

Log4j Logging

Setting Log Level to DEBUG

log4j = sc._jvm.org.apache.logging.log4j
log4j.core.config.Configurator.setLevel("com.ibm.stocator",log4j.Level.DEBUG)

Turning off Spark Logs

log4j = sc._jvm.org.apache.logging.log4j
log4j.core.config.Configurator.setRootLevel(log4j.Level.OFF)

Conclusion

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store