Analyse, Understanding, and Using Spark

Just some notes when I am walking through HelloSlick project.

Admittedly, I am still on my way to learn more about Scala. As a beginner of Scala language, I found HelloSlick was not easy to understand. After I figure it out, I think there could be a better way to explain Slick to impatient developers like me.

Versions

I found Slick 3.1.x is quite different from Slick 3.0.x. For example, DBIO is an alias of DBIOAction in Slick 3.1.x, but it was not in 3.0.x (not very sure though, didn't dig into it).

Future, onSuccess, flatMap, and map

To understand HelloSlick, we have to understand the concept and usage of Scala's Futures. The official document of Future can be found here:

Futures and Promises - Scala Documentation

Not sure this is the best doc for beginners who has Java background but want to start his project in a quick and dirty way.

Here are some code I played in scala REPL to help understand the usage of Future:

Create a Future instance

scala> val s = "Hello"
s: String = Hello

scala> import concurrent.Future
import concurrent.Future

scala> import concurrent._
import concurrent._

scala> import concurrent.ExecutionContext.Implicits.global
import concurrent.ExecutionContext.Implicits.global

scala> val f = Future { s + " future!" }
f: scala.concurrent.Future[String] = scala.concurrent.impl.Promise$DefaultPromise@75cd8043

Use onSuccess method

In my experience, I found onSuccess is the most easy way to deal with Future.

defonSuccess[U](pf: PartialFunction[T, U])(implicit executor: ExecutionContext): Unit

When this future is completed successfully (i.e. with a value), apply the provided partial function to the value if the partial function is defined at that value.

If the future has already been completed with a value, this will either be applied immediately or be scheduled asynchronously.

Multiple callbacks may be registered; there is no guarantee that they will be executed in a particular order.

The provided callback always runs in the provided implicit ExecutionContext, though there is no guarantee that the execute() method on the ExecutionContextwill be called once per callback or that execute() will be called in the current thread. That is, the implementation may run multiple callbacks in a batch within a singleexecute() and it may run execute() either immediately or asynchronously.

But here I met the concept of PartialFunction. There is a link to understand PartialFunctions:

Scala School - Pattern matching & functional composition

To beginners, we can just consider "case" here.

Await.result(f, Duration.Inf)

If we don't care about the result, but just want to have the application/thread wait there until Future finishes its job, then we can use Await.result(f, Duration.Inf). It is specially useful for chaining those Futures together as a batch job, and just need to wait for it finishes.

flatMap, and map

defmap[S](f: (T) ⇒ S)(implicit executor: ExecutionContext): Future[S]

Creates a new future by applying a function to the successful result of this future. If this future is completed with an exception then the new future will also contain this exception.

Example:

val f = Future { 5 }
val g = Future { 3 }
val h = for {
  x: Int <- f // returns Future(5)
  y: Int <- g // returns Future(3)
} yield x + y

is translated to:

f flatMap { (x: Int) => g map { (y: Int) => x + y } }

Let's take the implicit currying part out to simplify the signature:

def map[S](f: (T) ⇒ S): Future[S]

Side notes of <- (left-arrow, generator):

Reference to Scala By Example, 10.5 Generalizing For

www.scala-lang.org/docu/files/ScalaByExample.pdf
We have seen that the translation of for-comprehensions only relies on the presence of methods map, flatMap, and filter. Therefore it is possible to apply the same notation to generators that produce objects other than lists; these objects only have to support the three key functions map, flatMap, and filter.
So to put is simple, notation like x: Int <- f was meant to translate to use flatMap, map, and filter. And <- is a keyword in Scala (session 5.5 Language Elements Seen So Far)

It looks quite straightforward, although transform is not a concrete implementation in Future.scala. From the code, we can tell that it takes the result of T, and map it to use function f. The process of calculating the next result will be transformed and put into the input executor.

So far, the underscore operator still confuses me. Let's get back to it later.

defflatMap[S](f: (T) ⇒ Future[S])(implicit executor: ExecutionContext): Future[S]

Creates a new future by applying a function to the successful result of this future, and returns the result of the function as the new future. If this future is completed with an exception then the new future will also contain this exception.

Example:

val f = Future { 5 }
val g = Future { 3 }
val h = for {
  x: Int <- f // returns Future(5)
  y: Int <- g // returns Future(3)
} yield x + y

is translated to:

f flatMap { (x: Int) => g map { (y: Int) => x + y } }

If we take the currying part out, and somehow simplify the signature as this:

def flatMap[S](f: (T) ⇒ Future[S]): Future[S]

This is another version of map. In map method, we don't need to care about building the Future instance for the chain, but for flatMap, we need to do that.

It makes more sense when we read the implementation of scala:

The input function only works when the previous Future finishes successfully.

The plus sign

We can see the plus sign everywhere when we try to read Scaladoc regarding this example:

traitFuture[+T] extends Awaitable[T]

sealed traitDBIOAction[+R, +S <: NoStream, -E <: Effect] extends Dumpable

final def>>[R2, S2 <: NoStream, E2 <: Effect](a: DBIOAction[R2, S2, E2]): DBIOAction[R2, S2, E with E2]

A shortcut for andThen.

Here is a reference for understanding the plus sign and the minors sign:

+- Signs in Generic Declaration in Scala - Stack Overflow

And here is a reference to understand more about co-, contra-, and in- variances in Scala:

generics - Why doesn't the example compile, aka how does (co-, contra-, and in-) variance work? - Stack Overflow