Logging in Serverless Spark (Part2)

3 min readFeb 14, 2022

Learn how to tune logging : Executor vs Driver logs, Log Levels and Turning off Logs

Overview

In Part1 of this series we saw how the hooking up Log DNA with Analytics Engine service enables you to do simple troubleshooting and see the output of your print() or show() commands in spark applications.

In this part, we shall delve a bit more and see a few more aspects of logging — namely the executor vs driver user logs, executor vs driver Spark logs and also how you can set log levels.

stderr: Executor print() and Driver print()

When you do a print() or show() — the output goes to stderr or console output. In this simple application that calculates the squares of a given set of numbers, we print some info that gets into the executor output and some info that gets into the driver output

Executing this will result in both of the executor application stderr output

As well as the driver application stderr output

Log4j Logging

In the same example as above, we shall now define the log4j based logger object and use that to log the application output. (And yes we use log4j 2.x in Spark :)

Notice how it takes the logger name given — “squares” and logs it as an INFO line

Now if we change the code to user logger.warn() — this is how it would appear:

Setting Log Level to DEBUG

If you need to set log level to DEBUG, you can use the following code to set the level for a certain package.

log4j = sc._jvm.org.apache.logging.log4j
log4j.core.config.Configurator.setLevel("com.ibm.stocator",log4j.Level.DEBUG)

And you would see the DEBUG logs as below:

Turning off Spark Logs

Sometimes, you may want to turn off the Spark Logs to cut down and troubleshoot a specific area of code.

The way you can do this is to use the following lines of code and you will see that considerable amount of logs on the application are reduced, especially for large applications.

log4j = sc._jvm.org.apache.logging.log4j
log4j.core.config.Configurator.setRootLevel(log4j.Level.OFF)

Conclusion

Have a play around with these settings and let me know what other examples you would like to see here. Thanks for reading and watch out for Part 3.
(Thanks to Oliver Koeth for inputs)