Two Major Design Flaws

Throughout the time I spent developing longevity, I've made any number of major design flaws. The vision I started out with has changed drastically over time. It's evolved as I experimented with developing the longevity API. While my core guiding principles have stayed in place, the early versions of the library are nearly unrecognizable when compared to the latest version.

Since the beginning, I've had a comprehensive test suite, that has allowed me to correct mistakes and simplify my API with ease and assurance of correctness. And working with the API as it has grown has allowed me to spot problems as I moved forward. But there were two major design flaws that remain in place. Both of them have been nagging me for many months now, until they have become clear in my head. Now I'm ready to admit and address these two errors: the failure to use shapeless, and the lack of column-oriented storage of persistent entities with column-oriented back ends such as Cassandra and SQLite.

Longevity 0.27 - Stepping into Shapeless

After over 8 months since it's previous release, longevity version 0.27 is out! This release is very exciting to me, as it represents longevity's first steps into shapeless. I was able to incorporate shapeless.Generics into the domain model for all of the persistent objects, which will allow me to replace home-grown reflection based code with shapeless going forward.

The externally facing changes are minor. I've replaced a home-grown test data generation tool with ScalaCheck and scalacheck-shapeless. However, this minor change has already reaped benefits, as it replaces a particularly ugly part of the old longevity API - custom generation of test data - with a much more elegant process of supplying the right implicit org.scalacheck.Arbitrary.

This minor feature is really a proof of concept that I would be able to replace my homegrown reflective code with shapeless features. I feel confident that I can migrate the entire architecture over. Although of course, it will take some time. I will write more soon on how I plan to move forward from here. Stay tuned!

Longevity 0.26 - Migrations Framework

Longevity release 0.26 is out, and features the initial version of a schema migrations framework. You might ask, "Why would longevity - a persistence framework for Scala and NoSQL - require its own migrations framework? Couldn't it use any number of existing tools for doing schema migrations?"

Because longevity handles persistence for you, allowing you to write you application in terms of your domain model, the traditional approach to migrating schema with database scripts does not always work out. At present, every back end stores your persistent objects in JSON. Back ends such as SQLite and Cassandra store the JSON in regular text columns. MongoDB offers update commands that allow you to modify your BSON documents dynamically, so a more traditional approach to schema migrations may work for you. It’s also possible that we will be adding column-based back ends for SQLite and Cassandra in the future, which will also be amenable to more traditional approaches.

But even if a database-oriented tool such as FlywayDB works for you, we still want to provide you with an option that raises schema migrations up to the domain model abstraction layer. With longevity migrations, you define your domain model evolution in terms of Scala functions that map from the old version of the domain to the new version. So no database scripts are needed.

I'm excited about this release because the lack of tools to help migrate a schema was, in my mind, the last major showstopper for using longevity in a production environment. Domain models do evolve, and we need to be able to bring existing data along with the changes. It's a pretty bare bones implementation of a migrations framework, but it's entirely functional, and I have a long list of ideas on ways to improve it.

My favorite feature by far on the wish list, I've been calling soft-stop migrations. If our migrations are defined in terms of Scala, that gives us the opportunity to keep both versions of the schema alive in the database simultaneously. While we migrate from one version to another, applications running the old version can use those Scala functions to keep the new version up to date with ongoing changes. This way, the application can continue running throughout the process of a schema migration. This kind of thing is very difficult to do when your migration scripts are written in a database query language. And limiting downtime is highly desirable. It would also be relatively easy to implement, and I'm tempted to do it. But now that I have a basic schema migrations tool in place, I'd like to focus on replacing home-grown reflective code with shapeless for a while.

Check out the chapter on migrations in the user manual to get a feel for how it works.

longevity IDEA plugin

I'm so excited to announce the longevity IDEA plugin, written by Mardo Del Cid, that helps IntelliJ understand the longevity macro annotations! If you are using IDEA with your longevity project, installing this plugin will remove your red squiggles! It's in the JetBrains plugins repository, so you can install it just like you would any other plugin for IntelliJ. Just search for “longevity” under Plugins in the Settings / Preferences Dialog.

I haven't been using IDEA much at all these last couple years, so I hadn't really realized that IDEA was showing false errors for projects using longevity. I'm grateful to have a user that discovered and reported the problem. I wonder how many people saw this and never even mentioned it! But the best part is that Mardo didn't just tell me about it, he went ahead and fixed the problem as well! I really, really appreciate it. Thank you.

Summer Update

Hi there!

I'll be taking a month-long vacation starting this weekend (family road trip), and I wanted to drop a quick line before I head off, and give a status update on longevity.

First off, I made a video tutorial last week that's pretty nice. It's about 70 minutes, but it's probably a comfortable watch for most people at 1.5x speed.

I've been playing around with early steps in migrating to shapeless on a feature branch. The first bite I broke off for this is to rewrite my test data generation tools using Arbitrary from ScalaCheck. It's not finished yet, but I got through the most worrisome part, which was, how am I going to generate all the implicit stuff used by shapeless libraries? In this case, I need a bunch of implicits to generate an Arbitrary[P] using scalacheck-shapeless. The worrisome part is that I have to generate the class for a user-provided type P. This is problematic in general, because in order to get things like a Generic[P], you have to have the actual type P right there in your hands. Nothing else will do - not a TypeTag, nothing else along those lines. Thankfully, I already have macros that produce scaffolding around the type P, so I can generate the code to produce the Arbitrary[P], and it ends up getting inserted right there in the user code, where P is well known. I'm sure happy about that because it means the major worry I had about implementing things with shapeless is now solved.

So that's on a branch, and I put that aside because I decided I really need to start focusing in on providing users a schema migration tool. It's really the last "absolutely must have" feature for longevity, and while working on making longevity more functional is very enticing, I need to keep my priorities straight. The last two functionalization tasks - the ModelEv, PEv, and Key type classes, and the finally tagless, were both super important, but most importantly, user-facing changes. Most of the rest of the functionalization work I want to do is more internal, and shouldn't affect the user API too much. So I can definitely de-prioritize it. Only thing is, I want to support options in queries, and I've sort of been delaying that until after shapeless is in place. I could actually do it before then, but it just isn't the most efficient course, as I would have to rewrite most of it in shapeless.

So when I come back in September, I'll definitely be focusing on schema migration tool first. Once I have a rudimentary working tool out, I'll probably start splitting my time between that and migrating to shapeless.

Have a great August everybody!

Longevity Goes Finally Tagless

Longevity - your persistence framework for Scala and NoSQL - has always sported a future-based API. But now in release 0.24, longevity has gone finally tagless, replacing the hardcoded references to scala.concurrent.Future with a generic effect F. We currently support three effects: Scala futures, the cats-effect IO monad, and an old-fashioned blocking API for the purists out there. We plan to support a variety of Task monads as effects in the near future, and you can always write your own effect as well.

For more information on longevity effects, see the user manual.

For examples on how longevity works with different effects, check out this demo code.

This release is exciting for me for many reasons. Perhaps most importantly, longevity now sports a truly functional API with support for IO monads. But the flexibility of the finally tagless approach is truly freeing. We can satisfy the purist functional programmers, and continue to support people who want to do reactive programming with futures, with a minimum of cognitive overhead. It's also exciting to me because it's a major step on my personal path to becoming a better functional programmer.

But the best part for me is just how beautiful the repository API has become. Just look at it. In a sense, this is how this API was always meant to be. Perfectly intuitive, fully typesafe, just the right level of abstraction without a hint of leaks. I couldn't imagine doing persistence in Scala any other way.

Longevity Release 0.23 - Use Type Classes to Improve Type Safety of Persistent API

This longevity release has been a long time in the making. The main improvement here, and the one that has taken by far the most effort, is instituting the use of type classes to provide type safety for the persistence API. I've done a technical writeup on this work in my previous blog post. It's actually pretty interesting because I've never seen type classes quite this way before, so I think it's worth the read. The TLDR is that the repository API is now fully type safe, and the last instance of forcing users to inherit from a longevity trait has been removed.

Here are some of the other major improvements in this release:

An Interesting Use for Type Classes in Scala

This is a technical writeup of the major new feature in the longevity 0.23 release.

There are more than enough writeups on using type classes in Scala already. I normally wouldn't bother to write about it, but I will now, for three reasons. First, the usage of type classes in longevity seems unique to me; I haven't seen type classes used quite this way before. Second, I want to write this up as documentation for any future contributors to the longevity project. Third, I'd really like to get your feedback into what I've done here, and hear your advice about how I might further improve on things.

Longevity now has Better Support for Alternative JDBC Back Ends

Longevity release 0.21 features a new back end called JDBC. It should work in a lot of situations where using SQLite back end with a non-SQLite driver would fail. There is also a much easier development path for creating your own back end, should the generic JDBC back end not be sufficient.

Ten Ways You Can Contribute To An Open Source Project

Okay, so it's really about my open source project. But that's the whole point. I don't want it to be my project. I want it to be your project too! While this article is written to specifically address contributing to longevity, much of the material here will probably apply to other projects as well.

This is a complete rewrite of the contributing page on the longevity web site. You can read it over there if you like. I've also pasted the contents below for your convenience:

New SQLite Back End for Longevity

Longevity release 0.20.0 is out, featuring the following changes:

• Added a SQLite back end.
• Trying to be a bit more formal, changed occurrences of "Mongo" with "MongoDB" in the public API.
• Replaced "partition key" terminology with "primary key". I was undecided about this for a long time leading up to implementing the feature, and I think I made the wrong choice when the time came. "Partition key" is a little over specified for distributed database scenario. It's a little more general to posit that a database table can have multiple keys, but can more powerfully optimize a single one of those keys at your choosing. Now that I have three very different back ends, it's getting harder and harder to overfit longevity to a single database technology!

Keep reading to learn more about the new back end:

More Longevity Awesomeness with Macro Annotations!

I just got longevity release 0.18 out and wow, is it awesome! If you've looked at longevity before, you will be amazed at how easy it has become to start persisting your domain objects. And the best part is that everything persistence related is tucked away in the annotations. Your domain classes are completely free of persistence concerns, expressing your domain model perfectly, and ready for use in all portions of your application. Of course, we also provide a complete persistence layer for you, saving you countless hours of development effort!

See for yourself, here is a simple example demonstrating how easy it is to get started:

More API Simplifications in Longevity 0.17

Longevity is the persistence framework to use with Scala and NoSQL. Release 0.17 simplifies the user API in some very pleasant ways. Empty marker traits Persistent and Embeddable are removed. Also, KeyVal now takes a single type parameter instead of two. In brief, this will make defining the classes you want to persist so much easier and more natural, as you no longer have to extend your persistent classes with empty marker traits.

Longevity Artifacts for Scala 2.11 and 2.12

So I've finally got the longevity build set up to publish artifacts for Scala versions 2.11 and 2.12! I'm a little behind on the game, but, I've been busy ;-). Please use longevity version 0.16.1 for 2.12 artifacts. I'll be publishing artifacts for both Scala versions moving forward.

I'd like to shout out to the wonderful people at ScalaTest - Bill Venners and Chua Chee Seng - for helping me resolve an interoperability issue that made the longevity 2.12 artifacts possible. Thanks again!

Partition Keys in Longevity

I'm so excited to announce the release of longevity version 0.16.0! Longevity now supports partition keys - a key where, given a key value, we can determine the node in a distributed database where the data resides (or would reside, if it existed). This is a critical feature for NoSQL databases in a distributed context. A partition key is implemented with sharding in MongoDB, and with partitioning in Cassandra.

I'll provide a brief example of a partition key here after the jump, but to learn more about it, please take a look at the user manual.

Longevity 0.15.0 - Query Enhancements

Longevity - a persistence framework for Scala and NoSQL - is now up to version 0.15.0! The latest release focuses on query enhancements. We have added orderBy, offset, and limit clauses to our queries and query DSL.

Sample Application for Longevity and Play Framework

I've ported my sample longevity application - Simple Blogging - from Akka HTTP to Play. The Play version is a Lightbend Activator tutorial, which you can find here. Or you can take a look at the source code here on GitHub.

Longevity is a persistence framework for Scala and NoSQL.

Longevity 0.13.0 - Odds and Ends

In the past few weeks longevity has had three minor releases (0.11, 0.12, and 0.13) that clean up a variety of odds and ends that had been building up in the backlog. There was nothing major or exciting in these releases, but they are nonetheless important improvements. I haven't really publicized them much, but I wanted to let you know that longevity development is still proceeding.

Here are some changes that are worth mentioning:

• You no longer have to specify polyType and polyPType in your DerivedTypes and DerivedPTypes, respectively.
• Instead of just creating schema every time, I've added RepoPool.createSchema(), which you can call as you like. Schema creation is still non-destructive. And you can turn on the old behavior with configuration flag longevity.autocreateSchema.
• I added RepoPool.closeSession(), so you can clean up resources held by the underlying database drivers.
• I added some logging output.
• I added JSON marshallers and unmarshallers to the longevity context.

Next on the list is query enhancements! Although I have a couple of things to work on other than longevity proper that may precede it. I need to debug and fix a problem that has come up with longevity tests in ScalaTest version 3.0.0. And I'm thinking I'll port my sample application simbl from Akka HTTP to Play.

Configuration-level Optimistic Locking with Longevity

Longevity - a persistence framework for Scala and NoSQL - has been built from day one to isolate persistence concerns behind a clear, consistent, and easy to use API. The power of this design principle is demonstrated by how easy it is for you to bring optimistic locking to your application. All you have to do is set a configuration flag.

Longevity is growing to full maturity, and most features you would expect from a persistence framework are in place. Our 0.10.0 release brings many incremental improvements - including support for controlled vocabularies. But the central feature of this release is configuration-level optimistic locking.

Longevity Framework Cares About Code Quality

We care a lot about code quality here at longevity framework. You can rest assured that longevity is well-designed, has clean code, and a robust test suite. As of now, code coverage comes in at 95.02% statement, and 93.44% branch, as measured by sbt-scoverage. What software projects do you know of that have that kind of coverage?