Explore Courses Blog Tutorials Interview Questions
0 votes
in Big Data Hadoop & Spark by (11.4k points)

I've been trying to find a reasonable way to test SparkSession with the JUnit testing framework. While there seem to be good examples for SparkContext, I couldn't figure out how to get a corresponding example working for SparkSession, even though it is used in several places internally in spark-testing-base. I'd be happy to try a solution that doesn't use spark-testing-base as well if it isn't really the right way to go here.

Simple test case (complete MWE project with build.sbt):

import com.holdenkarau.spark.testing.DataFrameSuiteBase
import org.junit.Test
import org.scalatest.FunSuite

import org.apache.spark.sql.SparkSession

class SessionTest extends FunSuite with DataFrameSuiteBase {

  implicit val sparkImpl: SparkSession = spark

  def simpleLookupTest {

    val homeDir = System.getProperty("user.home")
    val training ="libsvm")
    println("completed simple lookup test")


The result of running this with JUnit is an NPE at the load line:

    at SessionTest.simpleLookupTest(SessionTest.scala:16)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(
    at java.lang.reflect.Method.invoke(
    at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(
    at org.junit.runners.model.FrameworkMethod.invokeExplosively(
    at org.junit.internal.runners.statements.InvokeMethod.evaluate(
    at org.junit.runners.ParentRunner.runLeaf(
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(
    at org.junit.runners.BlockJUnit4ClassRunner.runChild(
    at org.junit.runners.ParentRunner$



1 Answer

0 votes
by (32.3k points)

You can write a simple test with FunSuite and BeforeAndAfterEach like below

class Tests extends FunSuite with BeforeAndAfterEach {

  var sparkSession : SparkSession = _

  override def beforeEach() {

    sparkSession = SparkSession.builder().appName("udf testings")


      .config("", "")



  test("your test name here"){

    //your unit test assert here like below

    assert("True".toLowerCase == "true")


  override def afterEach() {




You don't need to create a functions in test you can simply write as

test ("test name") {//implementation and assert}


You need to check out below is a simple example

class TestSharedSparkContext extends FunSuite with SharedSparkContext {

  val expectedResult = List(("a", 3),("b", 2),("c", 4))

  test("Word counts should be equal to expected") {

    verifyWordCount(Seq("c a a b a c b c c"))


  def verifyWordCount(seq: Seq[String]): Unit = {

    assertResult(expectedResult)(new WordCount().transform(sc.makeRDD(seq)).collect().toList)



Browse Categories