2013-02-12

Component Based Dependency Injection in Scala

I recently wrote up my experiences using the "cake pattern" for dependency injection in Scala. It grew into quite a lengthy write-up, so I wanted to share it as a PDF in case reading in this blog format is inconvenient:
The same content appears below, but please be warned that the internal links are broken, and the code formatting is not to my liking:

Component Based Dependency Injection in Scala

Initial version by John Sullivan, February 2013

Introduction

As a software engineer at The Broad Institute, I’ve been very fortunate to be able to program in Scala for the past 2+ years. I got to start one major project from scratch, and was able to select the best Scala tools and libraries, and really take the time to work through a solid software design process. One of the earliest design decisions I needed to make on this project is what to use for dependency injection. I knew I could always fall back on using Java-based tools such as Spring or Guice, but I wanted to look around for a solution that might be a more natural approach for Scala. I happened upon the “cake pattern”, as described in Scalable Component Abstractions by Martin Odersky, and Real-World Scala: Dependency Injection (DI) by Jonas Bonér.

I ended up adopting the cake pattern in the manner described in the Jonas Bonér paper, with some modifications. I’ve gained a lot of practical experience in using the cake pattern, (which I’ve come to call “component based dependency injection”, or “CBDI” for short), and wanted to share some of those lessons here. Two of the most interesting extensions of the cake pattern I discovered are hierarchical components, and encapsulating the details of a composite component, both discussed below. I use the examples from the Bonér paper as a launching point, and I focus my comparisons to Spring when considering Java based DI frameworks, since I am much more familiar with Spring than with Guice.

One final note before I get going: Much of the source code presented here is available in GitHub project CBDI. I tend to use unit tests to exercise my code, so be sure to look in src/test/scala for running examples.

The Bonér Approach

I’ll use the approach presented in the Bonér paper as a launching point for describing how I employ the cake pattern. I start from this paper instead of the Odersky paper because the syntax is a little out-dated in the Odersky paper, and because the Bonér paper is a little easier to understand. This is the final UserRepository/UserService example that Bonér comes up with, with slight modifications from me to fill out some details of the example he leaves out:

case class User(username: String, password: String)

trait UserRepositoryComponent {
   val userRepository: UserRepository
   class UserRepository {
     def create(user: User): Unit = println("create " + user)
   }
}

trait UserServiceComponent {
   self: UserRepositoryComponent =>
   val userService: UserService
   class UserService {
     def create(username: String, password: String): Unit =
       userRepository.create(User(username, password))
   }
}

object ComponentRegistry extends UserServiceComponent with UserRepositoryComponent {
   val userRepository = new UserRepository
   val userService = new UserService
}

Bonér does not suggest any terminology for the “service” classes defined within the components, such as UserRepository and UserService in the above example. This makes the discussion below a little awkward. I’m going to call them injectables, for lack of a better term. In Spring, the injectable itself is called a component. By contrast, here, the component consists of both the injectable’s API, and the point of access for the injectable.

Separating Interface from Implementation

One drawback to the Bonér approach is that there is no interface defined for the UserRepository and UserService implementations; We haven’t really separated out interface and implementation for these injectables. Bonér chooses not to make this separation for the purpose of simplicity, but it is unclear if that choice is just for the sake of presentation in the paper, or if it applies to his use in production code as well.

Without separation of interface and implementation, the only way to really get an alternative implementation of one of our injectables - including mock objects - is by subclassing the original and overriding its methods. I wasn’t satisfied with this; I needed a genuine interface for these injectables. So I took the above concept one step further, as follows:

case class User(username: String, password: String)

trait UserRepositoryComponent {
   val userRepository: UserRepository
   trait UserRepository {
     def create(user: User): Unit
   }
}

trait UserRepositoryComponentImpl extends UserRepositoryComponent {
   override val userRepository: UserRepository = new UserRepositoryImpl
   private class UserRepositoryImpl extends UserRepository {
     override def create(user: User) = println("create " + user)
   }
}

trait UserServiceComponent {
   val userService: UserService
   trait UserService {
     def create(username: String, password: String): Unit
   }
}

trait UserServiceComponentImpl extends UserServiceComponent {
   self: UserRepositoryComponent =>
   override val userService: UserService = new UserServiceImpl
   private class UserServiceImpl extends UserService {
     override def create(username: String, password: String) =
       userRepository.create(User(username, password))
   }
}

trait ComponentRegistry
extends UserServiceComponent
with UserRepositoryComponent

trait ComponentRegistryImpl
extends ComponentRegistry
with UserServiceComponentImpl
with UserRepositoryComponentImpl

I’ve applied the separation of interface and implementation to the component registry itself, which is going to seem a little pointless until I introduce hierarchical components below.

The above separation of interface and implementation creates an annoying plethora of seemingly meaningless repeated code, but this seems to be a general problem with separating implementation from interface. I could imagine a language having features to make this kind of pattern less verbose - for instance, a way to define an interface and a default implementation in a single pass - but I don’t know of any languages that support this. I’ve made the choice of putting the interface and default implementation in the same source file. This makes it easier to scan the contents of a directory, at the expense of having to skip some extra “preamble” towards the top of the file to get to the real content.

Unit Testing with Mock Objects

Jonas Bonér uses specs and jMock for unit testing DI components with mock objects. For the sake of variety, I will demonstrate a unit test with ScalaTest and EasyMock. First, let’s create our component registry of mock objects:

// libraryDependencies += "org.scalatest" %% "scalatest" % "1.9.1" % "test"
//
// libraryDependencies += "org.easymock" % "easymockclassextension" % "3.1" % "test"

import org.scalatest.mock.EasyMockSugar
trait ComponentRegistryMock extends ComponentRegistry with EasyMockSugar {
   val userService = mock[UserService]
   val userRepository = mock[UserRepository]
}

Next, we’ll write a unit test for UserServiceImpl.create:

import org.easymock.EasyMock.reset
import org.scalatest.FlatSpec
class UserServiceSpec
extends FlatSpec
with ComponentRegistryMock
with UserServiceComponentImpl {
 
  behavior of "UserServiceImpl.create"
  it should "delegate to UserRepository.create" in {
     val user = User("charlie", "swordfish")
     expecting {
       userRepository.create(user)
     }
     whenExecuting(userRepository) {
       userService.create(user.username, user.password)
     }
     reset(userRepository)
   }
}

We make sure to inherit from UserServiceComponentImpl last, so that we get a UserServiceImpl as the userService injected dependency, hiding the mock UserService.

Using Hierarchical Components for Expressing Design Constraints

Let’s take the above example a step further, and add in multiple “service” and “repository” components. We could easily just add them in to the ComponentRegistry and ComponentRegistryImpl traits described above. But if we do this, we are building out a large flat space of objects with no organization to them. As the number of components grows, things will get more and more disorganized. We also want to be able to express design constraints such as “service components should be able to access repository components, but not vice versa.” In the following example, we show how to accomplish these goals using the existing user components, plus service and repository components for a new “project” entity:

trait RepositoryComponent
extends ProjectRepositoryComponent
with UserRepositoryComponent

trait RepositoryComponentImpl
extends RepositoryComponent
with ProjectRepositoryComponentImpl
with UserRepositoryComponentImpl

trait ServiceComponent
extends ProjectServiceComponent
with UserServiceComponent {
   self: RepositoryComponent =>
}

trait ServiceComponentImpl
extends ServiceComponent
with ProjectServiceComponentImpl
with UserServiceComponentImpl {
   self: RepositoryComponent =>
}

trait TopComponent
extends ServiceComponent
with RepositoryComponent

trait TopComponentImpl
extends TopComponent
with ServiceComponentImpl
with RepositoryComponentImpl

This prevents component access that does not conform to design constraint. For instance, suppose we tried to have the ProjectRepository access the UserService, as follows:

trait ProjectRepositoryComponent {
   val projectRepository: ProjectRepository
   trait ProjectRepository {
     def create(project: Project): Unit
   }
}

trait ProjectRepositoryComponentImpl extends ProjectRepositoryComponent {
   self: UserServiceComponent => // <= does not compile!
   override val projectRepository: ProjectRepository = new ProjectRepositoryImpl
   private class ProjectRepositoryImpl extends ProjectRepository {
     override def create(project: Project) = println("create " + project)
   }
}

This design constraint violation produces the following compiler error:

[error] RepositoryComponent.scala:9: illegal inheritance;
[error] self-type RepositoryComponentImpl does not conform to ProjectRepositoryComponentImpl's selftype ProjectRepositoryComponentImpl with UserServiceComponent
[error] with ProjectRepositoryComponentImpl
[error] ^

Unfortunately, it’s not completely obvious what the problem is from the error message. Furthermore, the error message directs the developer’s attention to RepositoryComponentImpl, instead of ProjectRepositoryComponentImpl. This could easily lead a naive developer into attempting to resolve the error like so:

trait RepositoryComponentImpl
extends RepositoryComponent
with ProjectRepositoryComponentImpl
with UserRepositoryComponentImpl {
   self: ServiceComponent =>
}

This fixes the compiler error, but it breaks our design constraint! I’ve developed the habit of adding warnings in comments to protect against this kind of mistake. For example:

trait RepositoryComponentImpl
extends RepositoryComponent
with ProjectRepositoryComponentImpl
with UserRepositoryComponentImpl {
   // do not self-type to ServiceComponent here as it breaks design constraint!
}

Examples of Application Design Constraints

The particular design constraints that you will want to enforce will depend largely on the architecture or your particular application. However, I would like to demonstrate a couple of standard application architectures, to give you a sense of the possibilities. The components for a standard web application might be organized like so:


A typical desktop GUI application with a service layer for accessing external resources might have the following organization:

Mock Objects with Hierarchical Components

For the sake of completeness, here is a suite of mock objects for the new, hierarchical design:

import org.scalatest.mock.EasyMockSugar
trait RepositoryComponentMock
extends RepositoryComponent
with EasyMockSugar {
   val projectRepository = mock[ProjectRepository]
   val userRepository = mock[UserRepository]
}

trait ServiceComponentMock
extends ServiceComponent
with EasyMockSugar {
   self: RepositoryComponent =>
   val projectService = mock[ProjectService]
   val userService = mock[UserService]
}

trait TopComponentMock
extends TopComponent
with ServiceComponentMock
with RepositoryComponentMock

Encapsulating the Details of a Composite Component

One advantage to using hierarchical components for dependency injection over frameworks like Spring is that we can actually hide the implementation details of a composite component within its implementation. For instance, consider the following ChartViewFactory, which constructs an appropriate ChartView for the supplied Chart. The ChartViewFactoryImpl accomplishes this by delegating to another factory based on the particular type of chart:

trait ChartViewFactoryComponent {
   val chartViewFactory: ChartViewFactory
   trait ChartViewFactory {
     def create(chart: Chart): ChartView
   }
}

trait ChartViewFactoryComponentImpl extends ChartViewFactoryComponent {
   self: HistogramViewFactoryComponent with ScatterPlotViewFactoryComponent =>
   override val chartViewFactory: ChartViewFactory = new ChartViewFactoryImpl
   private class ChartViewFactoryImpl extends ChartViewFactory {
     override def create(chart: Chart) = {
       chart match {
         case c: Histogram => histogramViewFactory.create(c)
         case c: ScatterPlot => scatterPlotViewFactory.create(c)
       }
     }
   }
}

The ChartViewFactoryComponent is a sub-component of the ChartViewComponent, which in turn is a sub-component of the ViewComponent. But notice that the ChartViewComponentImpl contains extra components for the factories that we delegate to:

trait ChartViewComponent
extends ChartViewFactoryComponent

trait ChartViewComponentImpl
extends ChartViewComponent
with ChartViewFactoryComponentImpl
with HistogramViewFactoryComponentImpl
with ScatterPlotViewFactoryComponentImpl

trait ViewComponent
extends TopViewComponent
with ChartViewComponent

trait ViewComponentImpl
extends ViewComponent
with TopViewComponentImpl
with ChartViewComponentImpl

This tactic of including the sub-components in the parent component implementation, but excluding them from the parent component interface, limits their visibility to within that parent component. For instance, it would be illegal for TopViewComponentImpl to self-type on HistogramViewFactoryComponent.

Prototyping Components

Spring provides support for injectables with different life cycle scopes, such as singleton, prototype, request, and session. The typical life cycle used in dependency injection is singleton, where a single instance is created for use throughout the entire application. Clearly, the dependency injection system described here is not going to support request and session scopes out of the box. But it is possible to mimic a prototyping Spring component. Here is how we would define and use it in Java and Spring:

@Component @Scope("prototype")
public class StatefulService {
     // ...
}

@Component
public class ReferencingService {

     // every referencing component will get its own instance of the service
     @Autowired
     private StatefulService statefulService;
}

To mimic this with component based dependency injection in Scala, we can simply use a def instead of a val to define the injectable:

trait StatefulServiceComponent {
   def statefulService: StatefulService
   trait StatefulService {
     // ...
   }
}

trait StatefulServiceComponentImpl extends StatefulServiceComponent {
   override def statefulService: StatefulService = new StatefulServiceImpl
   private class StatefulServiceImpl extends StatefulService {
     // ...
   }
}

The major drawback here is that each time statefulService is referenced, a new instance of the service will be created. With Spring prototypes, you get a single instance of StatefulService per referencing component. So to make this work, you need to reference the StatefulService exactly one in the referencing component:

trait ReferencingServiceComponent {
   def referencingService: ReferencingService
   trait ReferencingService {
     // ...
   }
}

trait ReferencingServiceComponentImpl extends ReferencingServiceComponent {
   self: StatefulServiceComponent =>

   override def statefulService: ReferencingService = new ReferencingServiceImpl
   private class ReferencingServiceImpl extends ReferencingService {

     // create and store a single instance of the stateful service for use here
     private val myStatefulService = statefulService

     // ...
   }
}

In the end, it will probably be more clear and manageable to create a StatefulServiceFactoryComponent, and inject the StatefulServiceFactory. The bottom line here is that component based dependency injection is not going to do any life cycle management for you.

Injecting the Top Component into the Application

There are many ways to make the component hierarchy accessible to the application. There is no right answer, but to a certain extent, it depends on the nature of the application. For instance, I’ve had to take different approaches for standalone applications, and for applications that are built with a framework such as Lift or Play.

Standalone Application

In a standalone client application, injecting the TopComponent is relatively straightforward. The top-level application class is conceptually defined as follows:

abstract class MyApplication extends TopComponent {
   def start = {
     // ...
   }
   // ...
}

The only thing that makes the MyApplication class abstract is the fact that it has not been provided with an implementation of the TopComponent. This is easily done in the main method of the program:

object MyApplication {
   def main(args: Array[String]) =
     (new MyApplication with TopComponentImpl).start
}

This approach allows us to write unit tests on the methods in the MyApplication class that use a mock implementation of the TopComponent. It also provides for an easy way to switch in alternate application contexts. For instance, maybe I want to stub out some of my service classes in test mode, so I don’t need to rely on external resources to test the user interface. I could easily accomplish this like so:

object MyApplication {
   def main(args: Array[String]) = {
     val app =
       if (testMode)
         (new MyApplication with TopComponentTestImpl)
       else
         (new MyApplication with TopComponentImpl)
     app.start
   }
}

Framework Application

We will need a different approach on an application built in a framework such as Lift or Play, as these frameworks normally take over the responsibilities of implementing the method main, and/or instantiating an object that represents the overarching application. For simplicity, we supply a singleton object that maintains a standard implementation of the TopComponent, like so:

trait TopComponent
extends RepositoryComponent
with ServiceComponent
object TopComponentImpl extends TopComponentImpl
trait TopComponentImpl
extends TopComponent
with RepositoryComponentImpl
with ServiceComponentImpl

Now, in any of the hooks that the framework provides, we can use the TopComponentImpl companion object to access the server-side dependency injection framework. For example, here is some code for installing our own exception handler into the Lift framework. The call into the TopComponent singleton is highlighted in red:

class BootLoader extends Bootable {
   def boot = {
     LiftRules.exceptionHandler.prepend {
       case (_, _, throwable) => {
         val errorReport = ErrorReport(
           throwable,
           Authenticater.userRequestVar.is.get.username)
         TopComponentImpl.errorReportService.handleErrorReport(errorReport)
         InternalServerErrorResponse()
       }
     }
   }
}

Gotchas and Edgy Cases

Initialization and Initialization Order

It’s important to know that your injectables are going to be initialized in Scala class linearization order. The linearization order for the components that belong to your TopComponentImpl will be initialized differently, depending on the order that the components are listed in the definition of your TopComponentImpl. Scala class linearization rules are complicated, so it’s probably not a good idea to make this component ordering have any special meaning. For readability, I prefer to order them lexicographically. It’s important to understand that the order does matter, but only if your components are accessing their dependencies at initialization time. For instance, consider the following example:

trait Service1Component {
   val service1: Service1
   trait Service1 {
     def announce(): Unit
   }
}

trait Service1ComponentImpl extends Service1Component {
   override val service1: Service1 = new Service1Impl
   private class Service1Impl extends Service1 {
     override def announce() = println("hi from Service1!")
   }
}

trait Service2Component {
   val service2: Service2
   trait Service2
}

trait Service2ComponentImpl extends Service2Component {
   self: Service1Component =>
   override val service2: Service2 = new Service2Impl
   private class Service2Impl extends Service2 {
     service1.announce() // has service1 been initialized yet?
   }
}

trait TopComponent1
extends Service1Component
with Service2Component

// this works
trait TopComponent1Impl
extends TopComponent1
with Service1ComponentImpl
with Service2ComponentImpl

trait TopComponent2
extends Service2Component
with Service1Component

// this throws NullPointerException when initializing Service2Impl
trait TopComponent2Impl
extends TopComponent2
with Service2ComponentImpl
with Service1ComponentImpl

There are two ways around this problem. The simplest is to just agree that a component should not access any of its dependencies until after initialization is complete. Alternatively, you could always declare your injectables as lazy, and make sure there are no cyclic references between injectables during initialization. This latter approach may sound a little dubious, but there are instances where accessing dependencies during initialization comes in handy. For instance, it’s standard practice in Scala-Swing to lay out the child components of a panel when that panel is being initialized. If both the panel and some of its sub-components are injectables, then it would be quite natural for the panel to access an injected dependency at initialization time.

It’s worth noting that, unlike in Spring, cyclic references between injectables is not in and of itself a problem. However, this is not a recommended practice, as cyclical dependencies between components is a sign of tight coupling between these components. These components should probably be refactored to remove the cyclic references.

Does the Self Type Belong on the API or the Implementation?

One question that often came up for me was, should I declare my dependencies just at the implementation level, or at the API level as well? This question applies to both simple, leaf-node components, as well as container components. It is only strictly necessary on the implementation class, but where to put it is a matter of style. Generally speaking, I try to ask myself, “is this dependency an implementation detail of the service?” Normally it is, but sometimes it makes sense at the interface level. Consider, for example, a ControllerComponent in a typical MVC application. It’s a central part of the MVC design pattern that the ControllerComponent references the ModelComponent and the ViewComponent. In this case, I would choose to add the self types on the ControllerComponent, and not just the ControllerComponentImpl.

Two Injectables with the Same Name

Even though we are able to organize our components into hierarchies, and restrict access to components in various ways, we still end up with all our injectables in a single name space. Any name collisions are going to produce a compiler error about incompatible types, even if the injectables with the conflicting names are never available from any single component.

Why Overriding the Injectable in the Implementation is So Verbose

It may seem overly verbose to repeat the type of the injectable when overriding it in the implementation class. For instance, here I have to redeclare exampleService to have type ExampleService:

trait ExampleServiceComponent {
   val exampleService: ExampleService
   trait ExampleService
}

trait ExampleServiceComponentImpl extends ExampleServiceComponent {

   // why do I have to redeclare the type of exampleService??
   override val exampleService: ExampleService = new ExampleServiceImpl

   private class ExampleServiceImpl extends ExampleService
}

If I leave out this type redeclaration, I am actually narrowing the type of exampleService to ExampleServiceImpl, since this is the inferred type from the right-hand side of the assignment. But this is illegal, since I have declared ExampleServiceImpl to be private:

[error] ExampleServiceComponent.scala:12: private class ExampleServiceImpl escapes its defining scope as part of type ExampleServiceComponentImpl.this.ExampleServiceImpl
[error] override val exampleService = new ExampleServiceImpl
[error] ^

Admittedly, making the implementation class private does not accomplish much, since dependent classes will self type on ExampleServiceComponent, and not ExampleServiceComponentImpl. The only place this type could actually be accessed would be in a containing component implementation, such as TopComponentImpl. It would be highly unusual to be making any use of ExampleServiceImpl there. In the end, I chose to make these implementation classes private based on the principle of information hiding. You might choose to not bother making the implementation classes private. Indeed, if we were to take this thinking to its logical conclusion, the interface elements would be protected, and the implementation would be private[this], like so:

trait UserRepositoryComponent {
   protected val userRepository: UserRepository
   protected trait UserRepository {
     def create(user: User): Unit
   }
}

trait UserRepositoryComponentImpl extends UserRepositoryComponent {
   override protected val userRepository: UserRepository = new UserRepositoryImpl
   private[this] class UserRepositoryImpl extends UserRepository {
     override def create(user: User) = println("create " + user)
   }
}

Conclusions

There are two major drawbacks to component based dependency injection as I see it. One major problem is that the code constructs can be quite difficult to understand by those unfamiliar with the pattern. I would recommend developer-level documentation describing your particular usage patterns to overcome this obstacle. As with many new programming techniques, it can seem confusing and strange, but I expect most developers would get used to it pretty quickly. The other major problem with this approach is that it is a little verbose at the source code level. There’s certainly more boilerplate involved than we are used to with Scala, or that we would need to generate when using a library like Spring.

Component based injection dependency also has many benefits. Some are described in this paper, particularly hierarchical components and component encapsulation. Hierarchical components allow users to express their component structure in a rich, tree-like fashion instead of as a single flat component space. Design constraints (i.e., which components can see which other components) become enforceable at compile time. Finally, component encapsulation allows the details of a composite component to be hidden from other components in the hierarchy.

Tailoring a solution around a language-level feature allows for much greater flexibility than using a library or framework, and other people may find themselves solving problems with a hand-rolled component based dependency injection system that would not be possible with a pre-rolled solution. A language-level solution can potentially reduce the application footprint, which can be an important issue for desktop and mobile applications. Also, the costs of learning and maintaining knowledge of a complex framework like Spring, and integrating it into your application, can often be underestimated. I would imagine that, in the end, these costs would be similar to the costs of using the component based dependency injection described here.

But my intent in this paper is not to evangelize the cake pattern approach to dependency injection. Instead, I try to provide practical examples for using this approach, and to discuss and flesh out some of the issues that you might come across that are not presented in earlier work.

at
Labels: , , , , ,

2 comments:

  1. Morgan CreightonFebruary 14, 2013 at 10:18 AM

    I really appreciate you sharing this. Your observation that cake is difficult at first for those unfamiliar with the pattern is spot-on.

    Regarding your second observation, that the code is too boilerplate-y, I wonder whether there is an opportunity here for a thin "Scala-Guice" that can cut down on that.

    ReplyDelete
  2. John SullivanFebruary 14, 2013 at 11:33 AM

    Thanks for the feedback Morgan. In regards to Scala & Guice, I noticed this post that came by quite recently, but I haven't had time to look at it closely yet:

    http://eng.42go.com/play-framework-dependency-injection-guice/

    In regards to reducing boilerplate, there is a discussion about this now on scala-user Google group.

    This may even be possible with the new macros language feature in Scala 2.10.

    ReplyDelete

Note: Only a member of this blog may post a comment.

Subscribe to: Post Comments (Atom)