Blog of Gavin King

Unique approach to observer/observable pattern in Ceylon

The essence of the famous observer/observable pattern is that you have an observable object that produces events of various kinds, and one or more observer objects that register themselves as interested in notification when these events occur.

Of course, we represent each kind of event as a type, usually a class, though nothing prevents us from using an interface type as an event type.

For example:

class Started() {}
class Stopped() {}

An event type may even be generic:

class Created<out Entity>
        (shared Entity entity) 
        given Entity satisfies Object {
    string => "Created[``entity``]";
}

class Updated<out Entity>
        (shared Entity entity) 
        given Entity satisfies Object {
    string => "Updated[``entity``]";
}

class Deleted<out Entity>
        (shared Entity entity) 
        given Entity satisfies Object {
    string => "Deleted[``entity``]";
}

Of course, we have powerful mechanisms for abstracting over event types, for example:

alias Lifecycle<Entity> 
        given Entity satisfies Object
        => Created<Entity> |
           Updated<Entity> |
           Deleted<Entity>;

An observer, usually, is in essence nothing more than a function that accepts a certain type of event as a parameter.

For example, this anonymous function observes the creation of Users:

(Created<User> userCreated) 
        => print("new user created: " + userCreated.entity.name)

This anonymous function observes lifecycle events of any kind of entity:

(Lifecycle<Object> event) 
        => print("something happened: " + event)

Union and intersection types give us a nice way to express conjunction and disjunction of event types:

void (Created<User>|Deleted<User> userEvent) {
    switch (userEvent)
    case (is Created<User>) {
        print("user created: " + userEvent.entity.name);
    }
    case (is Deleted<User>) {
        print("user deleted: " + userEvent.entity.name);
    }
}

Now here's where we can do something really cute. Typically, in other languages, the observable object provides various observer registration operations, one for each kind of event the object produces. We're going to define a generic class Observable that works for any event type, and uses reified generics to map events to observer functions.

shared class Observable<in Event>() 
        given Event satisfies Object {
    ...
}

The type parameter Event captures the various kinds of events that this object produces, for example, an Observable<Lifecycle<User>> produces events of type Created<User>, Updated<User>, and Deleted<User>.

We need a list to store observers in:

value listeners = ArrayList<Anything(Nothing)>();

Here, Anything(Nothing) is the supertype of any function with one parameter.

The addObserver() method registers an observer function with the Observable:

shared void addObserver<ObservedEvent>
        (void handle(ObservedEvent event))
        given ObservedEvent satisfies Event
        => listeners.add(handle);

This method only accepts observer functions for some subset of the events actually produced by the Observable. This constraint is enforced by the upper bound given ObservedEvent satisfies Event.

The raise() method produces an event:

shared void raise<RaisedEvent>(RaisedEvent event)
        given RaisedEvent satisfies Event
        => listeners.narrow<Anything(RaisedEvent)>()
            .each((handle) => handle(event));

Again, the upper bound enforces that this method only accepts event objects that are of an event type produced by the Observable.

This method uses the new narrow() method of Iterable in Ceylon 1.2 to filter out observer functions that don't accept the raised event type. This method is implemented using reified generics. Here's its definition in Iterable<Element>:

shared default {Element&Type*} narrow<Type>() 
        => { for (elem in this) if (is Type elem) elem };

That is, if we have a stream of Elements, and we call narrow<Type>(), explicitly passing an arbitrary type Type, then we get back a stream of all elements of the original stream which are instances of Type. This is, naturally, a stream of Element&Types.

Now, finally, if we define an instance of Observable:

object userPersistence 
        extends Observable<Lifecycle<User>>() {

    shared void create(User user) {
        ...
        //raise an event
        raise(Created(user));
    }

    ...
}

Then we can register observers for this object like this:

//observe User creation events
userPersistence.addObserver(
        (Created<User> userCreated) 
        => print("new user created: " + userCreated.entity.name));

//observe User creation and deletion events
userPersistence.addObserver(
        void (Created<User>|Deleted<User> userEvent) {
    switch (userEvent)
    case (is Created<User>) {
        print("user created: " + userEvent.entity.name);
    }
    case (is Deleted<User>) {
        print("user deleted: " + userEvent.entity.name);
    }
});

Notice how with union and intersection types, subtyping, and variance, we find ourselves with a powerful expression language for specifying exactly which kinds of events we're interested in, in a typesafe way, right in the parameter list of the observer function.

For the record, here's the complete code of Observable:

shared class Observable<in Event>() 
        given Event satisfies Object {
    value listeners = ArrayList<Anything(Nothing)>();

    shared void addObserver<ObservedEvent>
            (void handle(ObservedEvent event))
            given ObservedEvent satisfies Event
            => listeners.add(handle);

    shared void raise<RaisedEvent>(RaisedEvent event)
            given RaisedEvent satisfies Event
            => listeners.narrow<Anything(RaisedEvent)>()
                .each((handle) => handle(event));

}

Finally, a caveat: the precise code above does not compile in Ceylon 1.1, because the narrow() method is new, and because of a fixed bug in the typechecker. But it will work in the upcoming 1.2 release of Ceylon.

Tuple and entry destructuring

The next release of Ceylon features an interesting range of new language features, including constructors, if and switch expression, let and object expressions, and destructuring of tuples and entries. In this post, I'm going to describe our new syntax for destructuring.

A destructuring statement looks a lot like a normal value declaration, except that where we would expect to see the value name, a pattern occurs instead.

An entry pattern is indicated using the skinny arrow -> we use to construct entries:

String->Integer entry = "one"->1;
value key->item = entry;    //destructure the Entry

A tuple pattern is indicated with brackets:

[String,Integer] pair = ["one",1];
value [first,second] = pair;    //destructure the Tuple

The pattern variables, key, item, first, and second are just regular local values.

We can nest tuple and entry patterns:

String->[String,Integer] entry = "one"->["one",1];
value key->[first,second] = entry;

A tuple pattern may have a tail variable, indicated with a *:

[String+] ints = 1..100;
value [first,*rest] = ints;    //destructure the Sequence

(This syntax resembles the spread operator.)

Patterns may optionally indicate an explicit type:

value String key->[String first, Integer second] = entry;

Pattern-based destructuring can occur in a number of other places in the language. A pattern can occur in a for loop:

for ([x, y] in points) { ... }

for (key->item in map) { ... }

Or in an exists or nonempty condition:

if (exists index->item = stream.indexed.first) { ... }

if (nonempty [first,*rest] = sequence) { ... }

Or in a let expression:

value dist = let ([x,y] = point) sqrt(x^2+y^2);

You might wonder why we decided to introduce this syntax, or at least, why we decided to do it now. Well, I suppose the simple answer is that it always felt a bit incomplete or unfinished to have a language with tuples but no convenient destructuring syntax for them. Especially when we did already have destructuring for entries, but only in for, as a special case.

But looking into the future, you could also choose to see this as us dipping our toes in the water of eventual support for pattern matching. I remain ambivalent about pattern matching, and it's certainly not something we find that the language is missing or needs, but lots of folks tell us they like it in other languages, so we're keeping our options open. Fortunately, the syntax described above will scale nicely to more complex patterns in a full pattern matching facility.

This functionality is already implemented and available in github.

Useless lying version ranges

A frequent request from the Ceylon community is support for version ranges in expressing module dependencies. There's no doubt that our current module system is too inflexible in terms of dependency resolution in the face of version conflicts, and I have some reasonable ideas about how to address that problem without needing version ranges. But I would like to document precisely why I think version ranges are strictly-speaking useless at best, and harmful at worst.

First, a philosophical point: version ranges encourage module authors to make untested or untestable claims about their modules, such as:

my.module is compatible with other.dependency version 2.x.

Yeah, right, 'cos you've actually tested my.module with every single minor version and point release of other.dependency, including all the versions that have not even been released yet! Sorry, but I simply don't believe you and I have to assume you're lying to me. Almost nobody tests their program or library with many different versions of its dependencies, and it's easy to see why they don't: as soon as we have a program with several dependencies, we face a factorial explosion of dependency version combinations.

OK, sure, you might argue, but version ranges are still better than nothing. Alright, alright. I'm not the kind of guy who much buys into the notion that something broken is better than nothing, but I realize I'm in the minority on that one, 'cos, y'know, worse is better, as the neckbeards keep telling me.

So let's see what we could do to make the most of version ranges. Let's consider the problem first from the point of view of two library authors assigning version ranges to their modules, and then from the point of view of the program or person assembling these modules.

Let's suppose my library x.y depends on org.hibernate. I tested and released x.y with the then-current version of org.hibernate, which was 4.1.3. What version range would I have chosen when declaring this dependency?

  • Well, for the lower bound I decided to lie and write 4.1.0. Typically, I hadn't actually tested x.y with versions 4.1.0, 4.1.1 and 4.1.2, but I had been using 4.1.2 in development at one stage and it seemed to work, and I didn't see any particular reason it wouldn't also work with 4.1.0 and 4.1.1. On the other hand, maybe I could have just picked 4.1.3. (It's not going to matter for the rest of this argument.)
  • For the upper bound, I had no clue. How could I possibly have known at the time which future unreleased version of org.hibernate would break my library? Assigning the upper bound of 4.1.3 would have seemed much too restrictive, so what I did was assume that org.hibernate is following the completely untestable semantic versioning standard, and that future versions of Hibernate 4.x would not have any bugs.

Thus, I arrived at the version range 4.[1.0-] using some imaginary syntax I just pulled out of my ahem, excuse me, invented for the sake of argument.

A critical thing to notice here is that, from the point of view of the library authors, there is no reasonable way to determine an accurate upper bound to the version range. This is utterly typical and normal and is the case for almost any library author!

Now, sometime later, you released your library a.b, which also depends on org.hibernate. At this point, the current release of org.hibernate was 4.2.0. Quite atypically, you actually do test your library with a previous version of its dependency, and so you know that it is actually compatible with 4.1.5 (the latest release of 4.1.x). Thus, you arrive at the version range 4.[1.5-].

Now suppose some poor soul wants to use both our libraries together in their program, thus taking advantage of the bugs in both of them. So now, when assembling the program, what version of org.hibernate should the module system choose. Let's consider the reasonable options:

  • Pick the latest release that fits the version ranges, that is, the latest release of 4.x. This approach means that a new release of org.hibernate can break our application. We're picking a release which hasn't been tested with either x.y or a.b. Not acceptable.
  • Pick the earliest release that fits the version ranges, in this case 4.1.5. This is better. At least there's a chance that one of the libraries (in this case, a.b) has actually been tested with this version. Still, according to this strategy the system could in general pick a dependency version that's earlier than all the current versions when the libraries were developed. That seems quite suboptimal.
  • Pick 4.2.0, since that was the current version of org.hibernate when one of the libraries was developed, so we know for a fact that it works with at least one of the libraries, and it's newer that the version that the other library was developed with. This seems to me like it's by far the most robust and natural strategy.

There are some variations on this scenario, which raise other possible choices, and I'm going to let you experiment with the variations yourself, and see how it affects the conclusion. But as far as I can tell, there's essentially no common scenario in which the third strategy isn't at least as good as any other possible strategy.

And now note that this third strategy doesn't actually use version ranges at all! We can write down this strategy without reference to version ranges. It just says: pick the latest version of the dependency among the versions with which the libraries were developed. Version ranges don't really add any useful additional information to that, especially in light of the fact that upper bounds are essentially impossible to determine, and even lower bounds often lie. Why inject additional inaccurate information into the mix when we already have an algorithm that produces a result without depending on guesses and lies and unverifiable assumptions? (I apologize for going all logical positivist on your arse.)

What do you think? Am I wrong? Is there some reasonably common scenario where the module system can be expected to produce a better outcome with the addition of version range information? Is there a scenario in which upper bound information doesn't lie? Is there a strategy involving version ranges that is unambiguously better than my admittedly unsophisticated "pick the latest version" approach?

New features coming soon

We've been spending time discussing the priorities for development of Ceylon 1.1.5 and 1.2, including soliciting community feedback. The plan is still suprisingly fluid right now, but there are a number of things that we've already started working on, or have decided to start working on, and so in the interest of transparency, I thought I would share them.

Warning: we're not committing to a timeframe or release version for most of these features. It's merely a summary of what we're working on now, or plan to start work on soon.

Serialization

As already announced, ceylon.language 1.1.5 will feature an API for Serialization. Note that this API does not itself specify a serialization format. Rather, it's a general-purpose and platform-neutral facility for marshalling objects to and from a serialized stream. Serialization libraries founded on this API may serialize to text-based formats like JSON or XML, to binary formats, or even to a database via ORM.

Work on this API is already well-advanced. Tom has already done the Java implementation, and Enrique has got it working in JavaScript.

Type argument inference for function references

In Ceylon 1.1, we made it possible to leave off the type of a parameter of an anonymous function that occurs in an argument list, letting the type be inferred by the compiler, for example:

{Float*} measurements = ... ;
Float product = measurements.fold(1.0)((x,y)=>x*y);

In Ceylon 1.1.5, I've extended this approach to cover references to generic functions. So now, instead of this:

{Float*} measurements = ... ;
Float product = measurements.fold(1.0)(times<Float>);

You can write this:

{Float*} measurements = ... ;
Float product = measurements.fold(1.0)(times);

This even works for static value references, so instead of this:

{[Float+]*} sequences = ... ;
{Float*} heads = sequences.map(Iterable<Float>.first);

You can write simply this:

{[Float+]*} sequences = ... ;
{Float*} heads = sequences.map(Iterable.first);

This is already implemented, and you can try it out in git. It will be released in Ceylon 1.1.5.

Named constructors

In Ceylon 1.1, there is only one "constructor" of a class, the body of the class itself. For the vast majority of classes this is far more elegant and convenient. But in a minority of cases, there is a true need to have multiple initialization paths, and so we've designed a new syntax to support that. It took us a while to come up with something elegant and regular that didn't break the block structure of the language or the rules about definite initialization, but I'm very happy with the final outcome.

Since Ceylon doesn't have overloading (except for Java interop), constructors have distinct names.

class Point {
    shared Float x;
    shared Float y;

    //the "default" constructor
    shared new Point(Float x, Float y) {
        this.x = x;
        this.y = y;
    }

    //an additional constructor
    shared new Diagonal(Float d) {
        x = (d^2/2)^0.5 * d.sign;
        y = x;
    }
}

Every constructor must initialize all members which are left uninitialized by the body of the class, and must delegate to a constructor of the superclass (in this case, they delegate to Basic() by default). Now we can create a Point in two different ways:

Point p1 = Point(2.0, 3.0);  //call the default constructor
Point p2 = Point.Diagonal(1.0);

The typechecker already supports constructors, and Tom has made good progress on implementing this feature for the Java backend. I'm not sure if this will make it into 1.1.5, but if it does then we might actually need to rename 1.1.5 to 1.2, given that this is a pretty significant enhancement to the language itself.

Extensions to the expression syntax

We're making several extensions to the expression syntax. These features are already supported in the typechecker, but not yet by the backends. Note that these features are especially useful when combined with certain other features of the language, like comprehensions, anonymous functions, named argument lists, and fat arrow function definitions.

Inline object expressions

An inline anonymous object expression is very similar to an anonymous class in Java, and is useful in essentially the same cases. For example:

printAll(object satisfies {Integer+} {
    iterator() =>
        object satisfies Iterator<Integer> {
            variable value current = 0;
            next() => current++;
        };
});

let expressions

A let expression allows the definition of new values within an expression. For example:

Float d = ... ;
value ptl = let (x = (d^2/2)^0.5 * d.sign) Point(x,x);

Inline if and switch expressions

Ceylon's then and else operators are nice, but they don't do anything special in terms of flow-sensitive typing, so we quite often run into cases where we're forced to use a whole if or switch statement in a block. To alleviate that minor source of discomfort, we're now going to let you use if and switch within expressions. For example:

String string(Object it)
        => if (is Person it) 
           then it.name 
           else it.string;

Or:

String name(Person|Org it)
        => switch (it) 
           case (is Org) it.tradingName 
           case (is Person) it.firstName + " " + it.lastName;

Cayla web framework

Frameworks for developing web applications are a top request from the community. After some discussion, we've decided to focus first on the server side, and come back later to the problem of client-side web frameworks. Note that there's no problem at all with using a native JS client-side web framework to call a Ceylon module compiled to JavaScript.

Julien is going to work on getting Cayla, a web framework for use on Vert.x, ready for release.

To showcase Cayla, Ceylon, and Vert.x, Julien is going to do a partial port of Ceylon Herd from Java/Play to Ceylon/Cayla. That should make for a great demo.

SDK modules ceylon.html and ceylon.promise

Cayla will offer a choice of templating technologies, but one of the options we obviously want to offer is templates written in Ceylon. In order to avoid the cost of rebuilding the template from scratch on each request, ceylon.html needs to be enhanced to support a mix of static nodes and nodes which are created or rendered dynamically.

Work on Cayla will also likely necessitate improvements to ceylon.promise, and, in particular, we need to make this module cross-platform (right now it is only available on the JVM).

Java EE integration

Toby Crawley has started work on integration with Java EE. The first order of business here is to make it easy to write a servlet in Ceylon and package it into a war archive. After that, we'll need to make sure Ceylon works well with CDI, JPA, JAX-RS, etc.

Improved debugging in Ceylon IDE

David is going to work on making Eclipse's debugger work better with Ceylon. This is now the only really major feature missing from Ceylon IDE, so when he's done with that, he's going to move onto the #1 requested feature from the Ceylon community, which is...

IntelliJ-based IDE for Ceylon

The IntelliJ plugin for Ceylon is still rudimentary, and not yet ready for release. But now that the Eclipse-based IDE is feature complete, we're going to refocus our tooling development efforts on IntelliJ.

Note that this doesn't really represent a change of direction for us; I'm an Eclipse user, I prefer Eclipse, and I see no good reason to change to IntelliJ. That's especially true since whenever I discover a nice feature of IntelliJ, I just go ahead and reimplement it in Ceylon IDE ;-) However, we recognize that there are plenty of folks on the other side of the fence, who, preferring IntelliJ, and likewise seeing no reason to change, deserve a great plugin for Ceylon. So I hereby promise that we will have absolutely awesome tooling for both these IDEs.

Source maps

To make it easy to debug Ceylon code running on a JavaScript virtual machine, Enrique is going to add support for source maps to ceylon compile-js.

Consume Typescript interface definitions

Microsoft's Typescript project (which recently took inspiration from Ceylon by adopting our approach to union types and flow sensitive typing) has put a whole lot of work into defining statically typed definitions of important APIs in the JavaScript world. Now that Ceylon 1.1 has dynamic interfaces it's at least in principle possible to have a well-defined transformation from a Typescript API definition to a Ceylon type. This could take the form of a mechanical source translator, or even a "model loader" for the Ceylon compiler. Stef is going to investigate this.

More

The above is an incomplete list. If the thing you're waiting for (Android!!) isn't on that list, that doesn't mean we don't want to work on it, it just means I don't yet have a concrete plan for actually starting work on it right now. Feel very welcome to bug us about it in comments or on the mailing list or IRC :-)

Planning the future of Ceylon 1.x

With the release of Ceylon 1.1, we've reached a point where we need to do some serious thinking about what are our priorities for the development of Ceylon 1.1.5, 1.2, and beyond. I definitely don't yet have a crystal clear vision of what is going to be in 1.2, so we're also looking for community feedback on this.

I do know of one item which is the top priority right now, and will be the main feature of Ceylon 1.1.5:

This was a feature that slipped from Ceylon 1.0, and which again narrowly missed out on inclusion in Ceylon 1.1. The concept behind serialization in Ceylon is to have an API responsible for assembling and disassembling objects that is agnostic as to the actual format of the serialized stream. Of course, this API also has to be platform neutral, in order to allow serialization between programs running on the JVM and programs running on a JavaScript VM. Tom Bentley already has a working prototype implementation. Once this feature is done, we can start working on serialization libraries supporting JSON and whatever else.

I also count the following as a high priority areas of work:

  • Java EE integration, and support for technologies like JPA and CDI.
  • Adding properties to the language, that is, a new syntax for attribute references, allowing easy MVC UI bindings.
  • Improving the Cayla web framework, and ceylon.html.

Beyond that, we're not sure where else we should concentrate development effort. Here are some things that stick out to me:

  • Addition of named constructors, allowing multiple ways to instantiate and initialize a class.
  • AST transformers—a system of compiler plugins, based around ceylon.ast, enabling advanced compile-time metaprogramming, which would form the foundation for LINQ-style queries, interceptors and proxies, and autogeneration of equals(), hash, and string, and more.
  • Addition of a syntax for expressing patterns in BNF.
  • The Ceylon plugin for IntelliJ IDEA.
  • Android support.
  • Assemblies—a facility for packaging multiple modules into a deployable "application".
  • New platform modules defining dynamic interfaces for typesafe interaction with JavaScript APIs such as the DOM, jQuery, etc.
  • Interoperation with dynamic languages on the JVM, via Ceylon's dynamic blocks and dynamic interfaces.
  • Enabling the use of Ceylon for scripting.

We can't do all of this in Ceylon 1.2. Therefore, we're looking for feedback from the community. Let us know, here in comments, or on the mailing list, what you feel is missing from Ceylon, either from the above list, or whatever else you think is important.