Throughout the time I spent developing longevity, I've made any number of major design flaws. The vision I started out with has changed drastically over time. It's evolved as I experimented with developing the longevity API. While my core guiding principles have stayed in place, the early versions of the library are nearly unrecognizable when compared to the latest version.
Since the beginning, I've had a comprehensive test suite, that has allowed me to correct mistakes and simplify my API with ease and assurance of correctness. And working with the API as it has grown has allowed me to spot problems as I moved forward. But there were two major design flaws that remain in place. Both of them have been nagging me for many months now, until they have become clear in my head. Now I'm ready to admit and address these two errors: the failure to use shapeless, and the lack of column-oriented storage of persistent entities with column-oriented back ends such as Cassandra and SQLite.
2018-07-04
2018-07-03
Longevity 0.27 - Stepping into Shapeless
After over 8 months since it's previous release, longevity version 0.27 is out! This release is very exciting to me, as it represents longevity's first steps into shapeless. I was able to incorporate shapeless.Generics into the domain model for all of the persistent objects, which will allow me to replace home-grown reflection based code with shapeless going forward.
The externally facing changes are minor. I've replaced a home-grown test data generation tool with ScalaCheck and scalacheck-shapeless. However, this minor change has already reaped benefits, as it replaces a particularly ugly part of the old longevity API - custom generation of test data - with a much more elegant process of supplying the right implicit org.scalacheck.Arbitrary.
This minor feature is really a proof of concept that I would be able to replace my homegrown reflective code with shapeless features. I feel confident that I can migrate the entire architecture over. Although of course, it will take some time. I will write more soon on how I plan to move forward from here. Stay tuned!
The externally facing changes are minor. I've replaced a home-grown test data generation tool with ScalaCheck and scalacheck-shapeless. However, this minor change has already reaped benefits, as it replaces a particularly ugly part of the old longevity API - custom generation of test data - with a much more elegant process of supplying the right implicit org.scalacheck.Arbitrary.
This minor feature is really a proof of concept that I would be able to replace my homegrown reflective code with shapeless features. I feel confident that I can migrate the entire architecture over. Although of course, it will take some time. I will write more soon on how I plan to move forward from here. Stay tuned!
2017-10-26
Longevity 0.26 - Migrations Framework
Longevity release 0.26 is out, and features the initial version of a schema migrations framework. You might ask, "Why would longevity - a persistence framework for Scala and NoSQL - require its own migrations framework? Couldn't it use any number of existing tools for doing schema migrations?"
Because longevity handles persistence for you, allowing you to write you application in terms of your domain model, the traditional approach to migrating schema with database scripts does not always work out. At present, every back end stores your persistent objects in JSON. Back ends such as SQLite and Cassandra store the JSON in regular text columns. MongoDB offers update commands that allow you to modify your BSON documents dynamically, so a more traditional approach to schema migrations may work for you. It’s also possible that we will be adding column-based back ends for SQLite and Cassandra in the future, which will also be amenable to more traditional approaches.
But even if a database-oriented tool such as FlywayDB works for you, we still want to provide you with an option that raises schema migrations up to the domain model abstraction layer. With longevity migrations, you define your domain model evolution in terms of Scala functions that map from the old version of the domain to the new version. So no database scripts are needed.
I'm excited about this release because the lack of tools to help migrate a schema was, in my mind, the last major showstopper for using longevity in a production environment. Domain models do evolve, and we need to be able to bring existing data along with the changes. It's a pretty bare bones implementation of a migrations framework, but it's entirely functional, and I have a long list of ideas on ways to improve it.
My favorite feature by far on the wish list, I've been calling soft-stop migrations. If our migrations are defined in terms of Scala, that gives us the opportunity to keep both versions of the schema alive in the database simultaneously. While we migrate from one version to another, applications running the old version can use those Scala functions to keep the new version up to date with ongoing changes. This way, the application can continue running throughout the process of a schema migration. This kind of thing is very difficult to do when your migration scripts are written in a database query language. And limiting downtime is highly desirable. It would also be relatively easy to implement, and I'm tempted to do it. But now that I have a basic schema migrations tool in place, I'd like to focus on replacing home-grown reflective code with shapeless for a while.
Check out the chapter on migrations in the user manual to get a feel for how it works.
Because longevity handles persistence for you, allowing you to write you application in terms of your domain model, the traditional approach to migrating schema with database scripts does not always work out. At present, every back end stores your persistent objects in JSON. Back ends such as SQLite and Cassandra store the JSON in regular text columns. MongoDB offers update commands that allow you to modify your BSON documents dynamically, so a more traditional approach to schema migrations may work for you. It’s also possible that we will be adding column-based back ends for SQLite and Cassandra in the future, which will also be amenable to more traditional approaches.
But even if a database-oriented tool such as FlywayDB works for you, we still want to provide you with an option that raises schema migrations up to the domain model abstraction layer. With longevity migrations, you define your domain model evolution in terms of Scala functions that map from the old version of the domain to the new version. So no database scripts are needed.
I'm excited about this release because the lack of tools to help migrate a schema was, in my mind, the last major showstopper for using longevity in a production environment. Domain models do evolve, and we need to be able to bring existing data along with the changes. It's a pretty bare bones implementation of a migrations framework, but it's entirely functional, and I have a long list of ideas on ways to improve it.
My favorite feature by far on the wish list, I've been calling soft-stop migrations. If our migrations are defined in terms of Scala, that gives us the opportunity to keep both versions of the schema alive in the database simultaneously. While we migrate from one version to another, applications running the old version can use those Scala functions to keep the new version up to date with ongoing changes. This way, the application can continue running throughout the process of a schema migration. This kind of thing is very difficult to do when your migration scripts are written in a database query language. And limiting downtime is highly desirable. It would also be relatively easy to implement, and I'm tempted to do it. But now that I have a basic schema migrations tool in place, I'd like to focus on replacing home-grown reflective code with shapeless for a while.
Check out the chapter on migrations in the user manual to get a feel for how it works.
2017-09-06
longevity IDEA plugin
I'm so excited to announce the longevity IDEA plugin, written by Mardo Del Cid, that helps IntelliJ understand the longevity macro annotations! If you are using IDEA with your longevity project, installing this plugin will remove your red squiggles! It's in the JetBrains plugins repository, so you can install it just like you would any other plugin for IntelliJ. Just search for “longevity” under Plugins in the Settings / Preferences Dialog.
I haven't been using IDEA much at all these last couple years, so I hadn't really realized that IDEA was showing false errors for projects using longevity. I'm grateful to have a user that discovered and reported the problem. I wonder how many people saw this and never even mentioned it! But the best part is that Mardo didn't just tell me about it, he went ahead and fixed the problem as well! I really, really appreciate it. Thank you.
I haven't been using IDEA much at all these last couple years, so I hadn't really realized that IDEA was showing false errors for projects using longevity. I'm grateful to have a user that discovered and reported the problem. I wonder how many people saw this and never even mentioned it! But the best part is that Mardo didn't just tell me about it, he went ahead and fixed the problem as well! I really, really appreciate it. Thank you.
2017-08-02
Summer Update
Hi there!
I'll be taking a month-long vacation starting this weekend (family road trip), and I wanted to drop a quick line before I head off, and give a status update on longevity.
First off, I made a video tutorial last week that's pretty nice. It's about 70 minutes, but it's probably a comfortable watch for most people at 1.5x speed.
I've been playing around with early steps in migrating to shapeless on a feature branch. The first bite I broke off for this is to rewrite my test data generation tools using Arbitrary from ScalaCheck. It's not finished yet, but I got through the most worrisome part, which was, how am I going to generate all the implicit stuff used by shapeless libraries? In this case, I need a bunch of implicits to generate an Arbitrary[P] using scalacheck-shapeless. The worrisome part is that I have to generate the class for a user-provided type P. This is problematic in general, because in order to get things like a Generic[P], you have to have the actual type P right there in your hands. Nothing else will do - not a TypeTag, nothing else along those lines. Thankfully, I already have macros that produce scaffolding around the type P, so I can generate the code to produce the Arbitrary[P], and it ends up getting inserted right there in the user code, where P is well known. I'm sure happy about that because it means the major worry I had about implementing things with shapeless is now solved.
So that's on a branch, and I put that aside because I decided I really need to start focusing in on providing users a schema migration tool. It's really the last "absolutely must have" feature for longevity, and while working on making longevity more functional is very enticing, I need to keep my priorities straight. The last two functionalization tasks - the ModelEv, PEv, and Key type classes, and the finally tagless, were both super important, but most importantly, user-facing changes. Most of the rest of the functionalization work I want to do is more internal, and shouldn't affect the user API too much. So I can definitely de-prioritize it. Only thing is, I want to support options in queries, and I've sort of been delaying that until after shapeless is in place. I could actually do it before then, but it just isn't the most efficient course, as I would have to rewrite most of it in shapeless.
So when I come back in September, I'll definitely be focusing on schema migration tool first. Once I have a rudimentary working tool out, I'll probably start splitting my time between that and migrating to shapeless.
Have a great August everybody!
I'll be taking a month-long vacation starting this weekend (family road trip), and I wanted to drop a quick line before I head off, and give a status update on longevity.
First off, I made a video tutorial last week that's pretty nice. It's about 70 minutes, but it's probably a comfortable watch for most people at 1.5x speed.
I've been playing around with early steps in migrating to shapeless on a feature branch. The first bite I broke off for this is to rewrite my test data generation tools using Arbitrary from ScalaCheck. It's not finished yet, but I got through the most worrisome part, which was, how am I going to generate all the implicit stuff used by shapeless libraries? In this case, I need a bunch of implicits to generate an Arbitrary[P] using scalacheck-shapeless. The worrisome part is that I have to generate the class for a user-provided type P. This is problematic in general, because in order to get things like a Generic[P], you have to have the actual type P right there in your hands. Nothing else will do - not a TypeTag, nothing else along those lines. Thankfully, I already have macros that produce scaffolding around the type P, so I can generate the code to produce the Arbitrary[P], and it ends up getting inserted right there in the user code, where P is well known. I'm sure happy about that because it means the major worry I had about implementing things with shapeless is now solved.
So that's on a branch, and I put that aside because I decided I really need to start focusing in on providing users a schema migration tool. It's really the last "absolutely must have" feature for longevity, and while working on making longevity more functional is very enticing, I need to keep my priorities straight. The last two functionalization tasks - the ModelEv, PEv, and Key type classes, and the finally tagless, were both super important, but most importantly, user-facing changes. Most of the rest of the functionalization work I want to do is more internal, and shouldn't affect the user API too much. So I can definitely de-prioritize it. Only thing is, I want to support options in queries, and I've sort of been delaying that until after shapeless is in place. I could actually do it before then, but it just isn't the most efficient course, as I would have to rewrite most of it in shapeless.
So when I come back in September, I'll definitely be focusing on schema migration tool first. Once I have a rudimentary working tool out, I'll probably start splitting my time between that and migrating to shapeless.
Have a great August everybody!
2017-07-06
Longevity Goes Finally Tagless
Longevity - your persistence framework for Scala and NoSQL - has always sported a future-based API. But now in release 0.24, longevity has gone finally tagless, replacing the hardcoded references to scala.concurrent.Future with a generic effect F. We currently support three effects: Scala futures, the cats-effect IO monad, and an old-fashioned blocking API for the purists out there. We plan to support a variety of Task monads as effects in the near future, and you can always write your own effect as well.
For more information on longevity effects, see the user manual.
For examples on how longevity works with different effects, check out this demo code.
This release is exciting for me for many reasons. Perhaps most importantly, longevity now sports a truly functional API with support for IO monads. But the flexibility of the finally tagless approach is truly freeing. We can satisfy the purist functional programmers, and continue to support people who want to do reactive programming with futures, with a minimum of cognitive overhead. It's also exciting to me because it's a major step on my personal path to becoming a better functional programmer.
But the best part for me is just how beautiful the repository API has become. Just look at it. In a sense, this is how this API was always meant to be. Perfectly intuitive, fully typesafe, just the right level of abstraction without a hint of leaks. I couldn't imagine doing persistence in Scala any other way.
For more information on longevity effects, see the user manual.
For examples on how longevity works with different effects, check out this demo code.
This release is exciting for me for many reasons. Perhaps most importantly, longevity now sports a truly functional API with support for IO monads. But the flexibility of the finally tagless approach is truly freeing. We can satisfy the purist functional programmers, and continue to support people who want to do reactive programming with futures, with a minimum of cognitive overhead. It's also exciting to me because it's a major step on my personal path to becoming a better functional programmer.
But the best part for me is just how beautiful the repository API has become. Just look at it. In a sense, this is how this API was always meant to be. Perfectly intuitive, fully typesafe, just the right level of abstraction without a hint of leaks. I couldn't imagine doing persistence in Scala any other way.
2017-06-12
Longevity Release 0.23 - Use Type Classes to Improve Type Safety of Persistent API
This longevity release has been a long time in the making. The main improvement here, and the one that has taken by far the most effort, is instituting the use of type classes to provide type safety for the persistence API. I've done a technical writeup on this work in my previous blog post. It's actually pretty interesting because I've never seen type classes quite this way before, so I think it's worth the read. The TLDR is that the repository API is now fully type safe, and the last instance of forcing users to inherit from a longevity trait has been removed.
Here are some of the other major improvements in this release:
Here are some of the other major improvements in this release:
Subscribe to:
Posts (Atom)