Longevity Now Supports Querying with Akka Streams

I'm excited to announce longevity release 0.8, which features streaming queries with Akka Streams! Prior to this release, longevity queries returned a list of results wrapped in a future, which was sort of only half-way reactive, and definitely wouldn't work in the face of query results that were too large to fit into memory. I was never particularly happy with this, but I had enough to do at the time with designing and implementing the query DSL.


There were a variety of choices for underlying technologies for streaming queries. I could have gone with RxScala, RxJava, or Project Reactor. One thing that all of these libraries have in common is they all implement the Reactive Streams API. As I see it, the reactive streams project has two major goals. First, to provide a common underlying interface for streams libraries, so they can better interoperate. Second, to provide a library-level approach to handle back pressure, which is a means for the stream consumer to tell the producer that it needs to slow down, because the consumer can't keep up with the flow.

Users no longer have to worry about throttling stream producers to prevent the consumer from getting overwhelmed, and eventually overrunning buffer. This is a good thing for two main reasons. It makes the library users' lives easier, and it also provides an elegant solution to a problem that is normally handled by heuristic approaches that don't always give the best performance. As an example of what I mean, consider a scenario where the stream producer is going too fast for the consumer. The most obvious approach to solve this problem is to throttle, or slow down, the producer. So the programmer chooses some throttle technique - for instance, forcing a 100 millisecond wait between elements produced. This prevents the consumer from being overwhelmed in most cases, and can be tuned to be as performant as possible in whatever testing environment is being used. But it does not prevent the consumer from being overwhelmed in all circumstances, and in some environments, the consumer may be able to handle more throughput. With reactive streams, the throttling is done dynamically, to match the capacity of the consumer.

I chose Akka Streams over the alternatives for a handful of reasons. For one, I mildly prefer to provide a Scala-based API for my users. I also expect better integration between Akka and the Scala language and library, due to the close relationship between the Akka and Scala teams. But more importantly, longevity is a tool designed for enterprise application development, and I expect many of my users will already be using Akka for other aspects of their development. This will help them limit the number of dependencies their projects have, and it will make for easier integration with other parts of their applications.

I also foresee more integration points with Akka in future longevity development. These features are far off, and haven't even really made it on my planning board yet. I'm really focusing on getting the core persistence story done well first. Once that is in place, there is room for further application support. For instance, automatically generated REST APIs for longevity persistence. I would probably go with Akka HTTP for that. I've also tossed around the idea of having an actor-based API for the longevity repositories. Finally, I would like to add support for popular DDD approaches such as CQRS, event driven architecture and event sourcing. I would want to take a close look at Akka Persistence here, finding possible integration points with that project, or perhaps borrowing ideas from it.

In any event, I am so glad that this feature is complete, and that it was so easy to integrate with Akka Streams! I am really looking forward to my next longevity task, which is to write a full-fledged getting started guide. I put together a somewhat hasty quick start guide when I was doing the initial draft of the user manual. When writing the user manual, I realized that while it was a very important and useful document to have, I needed a better tool to help people get started with longevity. But the first draft of the manual was a major piece of work, and I didn't have time to write a proper getting started guide at the time. Now is the time! I'm excited to work on a sample application to use in the getting started guide, and to get this important piece of user support material out there. I'm also looking forward to taking a look at Lightbend Activator again. I dabbled in it on another project a couple years back, and I'm looking forward to seeing how it's evolved since then.

For more information on streaming queries in longevity, please see the section on Repo.streamByQuery in the user manual.