Longevity 0.26 - Migrations Framework

Longevity release 0.26 is out, and features the initial version of a schema migrations framework. You might ask, "Why would longevity - a persistence framework for Scala and NoSQL - require its own migrations framework? Couldn't it use any number of existing tools for doing schema migrations?"

Because longevity handles persistence for you, allowing you to write you application in terms of your domain model, the traditional approach to migrating schema with database scripts does not always work out. At present, every back end stores your persistent objects in JSON. Back ends such as SQLite and Cassandra store the JSON in regular text columns. MongoDB offers update commands that allow you to modify your BSON documents dynamically, so a more traditional approach to schema migrations may work for you. It’s also possible that we will be adding column-based back ends for SQLite and Cassandra in the future, which will also be amenable to more traditional approaches.

But even if a database-oriented tool such as FlywayDB works for you, we still want to provide you with an option that raises schema migrations up to the domain model abstraction layer. With longevity migrations, you define your domain model evolution in terms of Scala functions that map from the old version of the domain to the new version. So no database scripts are needed.

I'm excited about this release because the lack of tools to help migrate a schema was, in my mind, the last major showstopper for using longevity in a production environment. Domain models do evolve, and we need to be able to bring existing data along with the changes. It's a pretty bare bones implementation of a migrations framework, but it's entirely functional, and I have a long list of ideas on ways to improve it.

My favorite feature by far on the wish list, I've been calling soft-stop migrations. If our migrations are defined in terms of Scala, that gives us the opportunity to keep both versions of the schema alive in the database simultaneously. While we migrate from one version to another, applications running the old version can use those Scala functions to keep the new version up to date with ongoing changes. This way, the application can continue running throughout the process of a schema migration. This kind of thing is very difficult to do when your migration scripts are written in a database query language. And limiting downtime is highly desirable. It would also be relatively easy to implement, and I'm tempted to do it. But now that I have a basic schema migrations tool in place, I'd like to focus on replacing home-grown reflective code with shapeless for a while.

Check out the chapter on migrations in the user manual to get a feel for how it works.

longevity IDEA plugin

I'm so excited to announce the longevity IDEA plugin, written by Mardo Del Cid, that helps IntelliJ understand the longevity macro annotations! If you are using IDEA with your longevity project, installing this plugin will remove your red squiggles! It's in the JetBrains plugins repository, so you can install it just like you would any other plugin for IntelliJ. Just search for “longevity” under Plugins in the Settings / Preferences Dialog.

I haven't been using IDEA much at all these last couple years, so I hadn't really realized that IDEA was showing false errors for projects using longevity. I'm grateful to have a user that discovered and reported the problem. I wonder how many people saw this and never even mentioned it! But the best part is that Mardo didn't just tell me about it, he went ahead and fixed the problem as well! I really, really appreciate it. Thank you.

Summer Update

Hi there!

I'll be taking a month-long vacation starting this weekend (family road trip), and I wanted to drop a quick line before I head off, and give a status update on longevity.

First off, I made a video tutorial last week that's pretty nice. It's about 70 minutes, but it's probably a comfortable watch for most people at 1.5x speed.

I've been playing around with early steps in migrating to shapeless on a feature branch. The first bite I broke off for this is to rewrite my test data generation tools using Arbitrary from ScalaCheck. It's not finished yet, but I got through the most worrisome part, which was, how am I going to generate all the implicit stuff used by shapeless libraries? In this case, I need a bunch of implicits to generate an Arbitrary[P] using scalacheck-shapeless. The worrisome part is that I have to generate the class for a user-provided type P. This is problematic in general, because in order to get things like a Generic[P], you have to have the actual type P right there in your hands. Nothing else will do - not a TypeTag, nothing else along those lines. Thankfully, I already have macros that produce scaffolding around the type P, so I can generate the code to produce the Arbitrary[P], and it ends up getting inserted right there in the user code, where P is well known. I'm sure happy about that because it means the major worry I had about implementing things with shapeless is now solved.

So that's on a branch, and I put that aside because I decided I really need to start focusing in on providing users a schema migration tool. It's really the last "absolutely must have" feature for longevity, and while working on making longevity more functional is very enticing, I need to keep my priorities straight. The last two functionalization tasks - the ModelEv, PEv, and Key type classes, and the finally tagless, were both super important, but most importantly, user-facing changes. Most of the rest of the functionalization work I want to do is more internal, and shouldn't affect the user API too much. So I can definitely de-prioritize it. Only thing is, I want to support options in queries, and I've sort of been delaying that until after shapeless is in place. I could actually do it before then, but it just isn't the most efficient course, as I would have to rewrite most of it in shapeless.

So when I come back in September, I'll definitely be focusing on schema migration tool first. Once I have a rudimentary working tool out, I'll probably start splitting my time between that and migrating to shapeless.

Have a great August everybody!

Longevity Goes Finally Tagless

Longevity - your persistence framework for Scala and NoSQL - has always sported a future-based API. But now in release 0.24, longevity has gone finally tagless, replacing the hardcoded references to scala.concurrent.Future with a generic effect F. We currently support three effects: Scala futures, the cats-effect IO monad, and an old-fashioned blocking API for the purists out there. We plan to support a variety of Task monads as effects in the near future, and you can always write your own effect as well.

For more information on longevity effects, see the user manual.

For examples on how longevity works with different effects, check out this demo code.

This release is exciting for me for many reasons. Perhaps most importantly, longevity now sports a truly functional API with support for IO monads. But the flexibility of the finally tagless approach is truly freeing. We can satisfy the purist functional programmers, and continue to support people who want to do reactive programming with futures, with a minimum of cognitive overhead. It's also exciting to me because it's a major step on my personal path to becoming a better functional programmer.

But the best part for me is just how beautiful the repository API has become. Just look at it. In a sense, this is how this API was always meant to be. Perfectly intuitive, fully typesafe, just the right level of abstraction without a hint of leaks. I couldn't imagine doing persistence in Scala any other way.

Longevity Release 0.23 - Use Type Classes to Improve Type Safety of Persistent API

This longevity release has been a long time in the making. The main improvement here, and the one that has taken by far the most effort, is instituting the use of type classes to provide type safety for the persistence API. I've done a technical writeup on this work in my previous blog post. It's actually pretty interesting because I've never seen type classes quite this way before, so I think it's worth the read. The TLDR is that the repository API is now fully type safe, and the last instance of forcing users to inherit from a longevity trait has been removed.

Here are some of the other major improvements in this release:

An Interesting Use for Type Classes in Scala

This is a technical writeup of the major new feature in the longevity 0.23 release.

There are more than enough writeups on using type classes in Scala already. I normally wouldn't bother to write about it, but I will now, for three reasons. First, the usage of type classes in longevity seems unique to me; I haven't seen type classes used quite this way before. Second, I want to write this up as documentation for any future contributors to the longevity project. Third, I'd really like to get your feedback into what I've done here, and hear your advice about how I might further improve on things.

Longevity now has Better Support for Alternative JDBC Back Ends

Longevity release 0.21 features a new back end called JDBC. It should work in a lot of situations where using SQLite back end with a non-SQLite driver would fail. There is also a much easier development path for creating your own back end, should the generic JDBC back end not be sufficient.

Ten Ways You Can Contribute To An Open Source Project

Okay, so it's really about my open source project. But that's the whole point. I don't want it to be my project. I want it to be your project too! While this article is written to specifically address contributing to longevity, much of the material here will probably apply to other projects as well.

This is a complete rewrite of the contributing page on the longevity web site. You can read it over there if you like. I've also pasted the contents below for your convenience:

New SQLite Back End for Longevity

Longevity release 0.20.0 is out, featuring the following changes:

• Added a SQLite back end.
• Trying to be a bit more formal, changed occurrences of "Mongo" with "MongoDB" in the public API.
• Replaced "partition key" terminology with "primary key". I was undecided about this for a long time leading up to implementing the feature, and I think I made the wrong choice when the time came. "Partition key" is a little over specified for distributed database scenario. It's a little more general to posit that a database table can have multiple keys, but can more powerfully optimize a single one of those keys at your choosing. Now that I have three very different back ends, it's getting harder and harder to overfit longevity to a single database technology!

Keep reading to learn more about the new back end: