Team blog

Status Report: Ceylon TypeScript Loader GSoC project

TL;DR: coming along, won’t be done in time for official GSoC deadline, I’ll just keep working.

(Note: this is a GSoC status report. The entire project is work in progress, and if you’re looking for a something finished, you’ll have to wait until the project is released, at which point there will be a proper announcement.)

For this year’s Google Summer of Code, I am working on adding TypeScript support to the Ceylon JavaScript backend. The goal is to have a tool that, given a TypeScript module (one or more files), produces a Ceylon module for the JavaScript backend. The JS code of that module will just be the TypeScript compiler’s output (plus possibly some metamodel stuff), but the tool will also add model information that will allow you to use the TypeScript module from Ceylon like any other Ceylon module, without needing to use dynamic blocks or having to declare your own dynamic interfaces.

I actually started work on this tool a few months before GSoC, in January this year. I wrote the first iteration of the program in TypeScript in order to be able to interact with the TypeScript compiler (which is also written in TypeScript). The goal of that first iteration is just to be able to load the TypeScript compiler itself, so that I can then use that module (tsc) from Ceylon and write the second (and probably final) iteration of the program in Ceylon.

This first iteration of the program was mostly finished, at least good enough that I could start writing the second iteration, shortly after the GSoC work period officially started. I had hoped to start with the second iteration exactly at the start of GSoC, but while there had been good progress with my own simple test module, tsc held a couple of nasty surprises that I hadn’t anticipated and that I had to work around. It also turned out that I had made a couple of very bad decisions early on:

  • I chose to operate on the parsed syntax tree instead of the typechecker model. How I ever expected this to work, I have no idea, but problems quickly became obvious:

    • When I see a type reference, like Element, I have to know if this refers to a type parameter, a type from the same module, or perhaps one from another module. The type I store in the model must be fully qualified.
    • To represent a type with type arguments, the JS model lists the type arguments along with their type parameter name. So to transform a type like Array<string>, I have to know that Array’s first type parameter is called Element.
    • TypeScript features type inference. Of course, I have to know the inferred type so that I can put it into the model.
  • To get a first version out quickly, I just wrote directly to the -model.js file. The model is mostly JSON, so this meant some ugly manual comma management when it turned out there are some declarations where I don’t emit anything (such as index signatures).

There are some lovely commit messages in the git log, with phrases like “totally hacky”, “should be enough for now”, “this is where it all falls apart”, “add fake support”, “horrible hack”, and “How could I possibly ever think that it’s possible to write this loader without access to tsc’s model, purely based on the AST?” I am very glad that I get to learn from these mistakes in the second iteration :)

Now, writing the second iteration also turned out to be trickier than I’d thought. I had (still have) a Ceylon version of the TypeScript compiler, the tsc module, but it’s difficult to work with from Ceylon. Two problems mainly held me up:

  1. TypeScript supports optional members: { name?: string } is the type of a value that might have a member name (containing a string), but might also have no such member (undefined). The Ceylon runtime doesn’t like this at all and breaks in many different and difficult-to-debug ways; the only solution I found was to not emit such members at all. To access them without “attribute does not exist” typechecker errors, I use eval inside dynamic blocks, which leads us to the second problem.
  2. Whenever a dynamic value (such as the return value of an eval call) is assigned to a Ceylon type, the runtime “dresses” the value with the type that it’s being assigned to (adds RTTI). If this is not the most precise type of the value (say, you dressed the value with Node and later want to use it as a VariableDeclaration), you have to jump through some hoops to fix this – a simple assert (is VariableDeclaration node) won’t work, since Ceylon doesn’t like to narrow these dressed types.

I now have workarounds for both problems, but they took a while to find, and of course I shouldn’t have to work around them in the first place, so I’ll have to teach the Ceylon JS backend to “do the right thing” eventually. (I also have to do that in many other cases, since the Ceylon JS “ABI” differs significantly from plain JS – Ceylon classes aren’t instantiated with new, toplevel values compile to functions, arrays aren’t arrays, etc. etc.)

Right now, the second iteration supports:

  • toplevel values,
  • toplevel functions (no parameters), and
  • the string type.

That’s it. But, unlike the first iteration, it supports this without needing to add any JS code (I’ve already taught the compiler to access the toplevel values as values and not functions). Currently, some JS code is added (metamodel stuff), but I’m not sure if I’ll actually keep that. We could say that what, are you crazy, why would TypeScript modules support the metamodel?, and then the JS file would just be the unaltered tsc output, and the tool would only add the -model.js file. This would probably make working with declaration files easier (where the JS isn’t generated by tsc – one of many problems I haven’t even begun to think about yet).

I’m fairly confident that I’ll be able to add support for most “basic” features – type references and other primitive types, type parameters and arguments, parameters, classes, interfaces, methods, attributes – without too much trouble. After that, all bets are off. TypeScript supports some crazy features, like string types ("foo" | "bar" | "baz" is a valid type) and type guards (pet is Fish is a valid return type, a weird inversion of Ceylon’s is Fish pet conditions), and I have no idea how well we can support them. I will also need to make more changes to the JS backend and runtime, and I don’t know how difficult those will be.

There are about four weeks left before the official GSoC deadline, and I’ll be on vacation for one of them. I hope that the “basic” support will be done until then, but there’s zero chance that the project will be in shippable state. I’m not too worried about that – I’ve worked on the project before GSoC started, I’ll continue to work on it after GSoC ends, just like I did with ceylon.ast two years ago. And, just like two years ago, I should have more time after GSoC ends, because GSoC is actually scheduled pretty weirdly for me, where I still have lectures for most of it and it barely overlaps with my semester break. I should be free to work on this project for most of August, September, and the first half of October, and hope to arrive at something that can actually be released somewhere before the end of that period.

Modularity Changes

TL;DR: This article describes the modularity changes in the Ceylon run-time and distribution, in order to make them lighter at run-time. Skip to the Final runtime dependencies section if you just want the outcome.

As we initially expected most Ceylon users to run their code using the ceylon run command, we figured that since they have the Ceylon distribution installed, it does not matter if they depend on more modules from that distribution than strictly necessary. Those modules had to be there anyway, so it would not save any bandwidth to reduce those dependencies.

Naturally, we were wrong, and between the Ceylon Eclipse or IntelliJ IDEs, running Ceylon on OpenShift, WildFly, Vert.x or on Android, people started running Ceylon without the distribution installed, using just the standard java runner. It became soon apparent that we had to untangle those dependencies to make the runtime requirements lighter.

Historically we had the following modules:

  • common module used by other modules
  • typechecker (the shared compiler front-end)
  • Java compiler back-end
  • JavaScript compiler back-end
  • module repository system
  • JBoss modules runtime

The model module

When we implemented reified generics, we had to add subtyping to the runtime, so that we'd be able to figure out if is T bar was true or not. The easiest thing at the time was to "just" depend on the typechecker (compiler front-end) which dealt with the language model and subtyping, and the Java compiler back-end, which had infrastructure to load a language model from JVM information such as class files, or in this case reflection.

This essentially made the runtime depend on the compiler front-end and back-end, which we realised was not ideal, so during the Ceylon 1.2 development, we extracted all model description, loading and subtyping to a new ceylon-model module, but we did not have enough time to do more and so these dependencies remained due to other causes.

Supporting Java 9

During our work on supporting Java 9 / Jigsaw modules in Ceylon, it became clear that having kept our "fork" of javalang tools (that we use for javac) under its original package name would not work anymore, we renamed its package and used the opportunity to prune away parts of the java tools we did not use. We also extracted the class-file reader part to its own module so we could use it outside of the compiler to remove our dependency to jandex (a class-file scanner).

Finally, when we created the ceylon jigsaw tool (which populates a folder with the jar files required by a Ceylon module, to run it on a Java 9 VM) it became evident that the runtime still depended not just on the compiler front-end and Java back-end, but even on the JavaScript back-end, which frankly made little sense in most JVM executions.

These dependencies were due to the Ceylon Tool Provider API having snuck into the ceylon.language module as a convenience (at the time). Since that allowed you to compile and run Ceylon programmatically for both Java and JavaScript back-ends, it had to depend on the tools.

We decided to split the Ceylon Tool Provider into its own module and got rid of the final dependencies from the language module to the compilers and typechecker, but had no more time to get rid of further dependencies such as JBoss Modules and Aether in time for Ceylon 1.2.2.

Supporting Android

Initial work on running Ceylon on Android revealed that what passes for small dependencies on ordinary JVM executions, or even on Java EE deployments, was not an option on Android where every method counts.

At this point we had to bite the bullet and make every non-required transitive dependency go.

We noticed that the old common module had grew to include the Command-Line Tooling API that makes the ceylon command and its subcommands and plugins work. That in turn depended on a Markdown renderer used by ceylon doc. It was pretty trivial to extract it to its own module because this was never used in Ceylon user programs.

Next in line was our Shrinkwrap Resolver dependency, which our module repository system uses to interoperate with Maven repositories. This was a fat-jar with all its dependencies included, including some Apache Commons modules, and an outdated version of Eclipse Aether. That fat-jar had already been problematic in our Maven module, which already had its version of Aether, so getting rid of the fat-jar was a good idea. We also realised that some of its Apache Commons dependencies were already included outside the fat-jar in our distribution repository, so there was that duplication to fix too.

So what we did was remove the Shrinkwrap Resolver dependency and use Aether directly, by incorporating all its subcomponents into our distribution. It turns out that because the latest version of Aether requires Google Guava, our distribution grew in size rather than shrink (that jar is huge). But to offset that, we made the Aether dependency optional, and made sure it was possible to run Ceylon without it as long as there was some compilation step beforehand that provided all the Maven dependencies that you may use in interop. ceylon fat-jar or ceylon jigsaw would do that for you, for example.

Our module repository system also provided support for writing to WebDAV or Herd repositories, which required some dependencies on Apache Http Client or Sardine, and we made these dependencies optional as well, because at runtime your Ceylon program is very unlikely to write to HTTP repositories. This is something only the compiler and other tools do.

We also removed a dependency to JBoss Modules from the language module using abstraction, since that platform was optional and never used on Android or other flat-classpath runtimes.

Finally, the language module only had one dependency left on the (much slimmer) module repository system via the presence of the Main API in there, and we moved that class to its own module.

Final runtime dependencies

After all this pruning, the language module on the JVM is back down to requiring the following set of transitive dependencies:

  • common (small and free of tooling and dependencies)
  • model (which depends only on the class-file reader)
  • class-file reader

So your Ceylon module will only depend on four jars (these three and the language module), the sum size of which is 2.4 Mb, which is much smaller than initially, and has dramatically less methods, at around 17148 methods. This is still too much, but can be brought down by tooling such as ProGuard to remove unused classes. Remember this includes a runtime for an entire language, so it's not that big, all things considered.

SDK changes

In order to be able to use Ceylon's HTTP client on Android, we also split up the module from the Ceylon SDK into client and server modules. Otherwise the HTTP server and its dependencies were too much drag for Android's method count.

Ceylon Fat Jars

Ceylon 1.2.3 is looming closer, and so we should start talking about some of the new features that it will contain. One of those new features is the new ceylon fat-jar command.

ceylon fat-jar my.module/1 lets you generate a jar which contains the my.module/1 module and every required dependency. Once you have that jar, you can execute it with java -jar my.module-1.jar and it will execute the module.

Naturally, you can customise the function or class to execute with the --run option, but the default is the run method in your module (my.module::run in our case).

The modules are executed in a "flat classpath" mode, without JBoss Modules or any sort of module isolation, and with the metamodel already set up statically when the fat jar was created, so it's a little bit different to the older alternative of running Ceylon modules using the Main API and ceylon classpath tool, which did not deal with packing all the dependencies. It is also different to the ceylon run --flat-classpath in that it does not require a module repository at run-time, since ceylon fat-jar packs it all together.

Try it out, it's pretty useful to distribute Ceylon programs and run them in places without having to install the Ceylon distribution.

Ceylon on Android

In my last post, I explained how you can use Ceylon in Apache Cordova to write applications for every mobile platform, including iOS and Android. This time, with many apologies for writing it late (“next week” turned into next month), I will explain how to use the Ceylon IntelliJ plugin to write native Android applications in Ceylon in Android Studio.

Getting started with Ceylon on Android Studio

To start writing Ceylon applications in Android Studio, follow these steps:

  • Download Android Studio
  • Start it
  • Create a new application by clicking on Start a new Android Studio Project
  • You can use these values for Application name: CeylonDemo
  • And for Company domain:
  • Next, select an Empty activity, with Activity Name: MainActivity
  • Click Finish and wait for the project to be created

At this point you have an Android project open, but we still haven't had time to install the Ceylon plugin, so let's do this right now:

  • Click on File > Settings > Plugins > Browse Repositories
  • Then on Manage Repositories > +
  • Add this repository:
  • Now click on Install Ceylon IDE

You will likely need to restart Android Studio, so do that.

Next we're going to convert our Android project to a Ceylon Android project:

  • In the Android view, Right-click on app > Configure Ceylon in this Module
  • Click OK on the resulting configuration dialog

This will set up the Ceylon plugin, and will add most of what you need in your Gradle build to build Ceylon Android applications. You now have your Ceylon sources in app/src/main/ceylon and it includes a module descriptor and an empty activity:

At the moment, this requires a Ceylon 1.2.3 distribution to build, and since it's not released yet you're going to have to either build one yourself (just the Getting the source part), or download a nightly build. Once you have it, edit app/build.gradle near then end to add ceylon > ceylonLocation and make it point to where you installed your distribution (it needs to point to the Ceylon binary, not just the distribution root):

ceylon {
    // ...
    ceylonLocation ".../ceylon/dist/dist/bin/ceylon"

Make sure you click on Sync now to sync your Gradle build.

Now, there's a bug we're in the process of fixing which fails to detect the exact version of the Android SDK tooling and modules, and so depending on which version of the Android Tools you're using you may have to sync the imports of in app/build.gradle (in dependencies) and in the Ceylon module descriptor in module.ceylon. Make sure the Ceylon import version is the same as the Gradle import version, because the Gradle build is what makes it available to Ceylon, due to Android's peculiarities.

In order to finish the conversion, make sure you delete the Java activity (since we're going to keep the Ceylon one), in Project Files, delete app/src/main/java.

Due to another pending plugin fix, you may have to click on Tools > Ceylon > Reset Ceylon Model at this point so that the Ceylon plugin gets synchronised with all these past changes (don't worry we're fixing this at the moment).

The good news is we're already able to click on Run app and try this in the emulator, but we're going to make it a little more interesting.

Customising your Ceylon Android activity

We're going to be displaying a list of Ceylon modules published on Ceylon Herd, so we will make use of the Ceylon SDK, and in particular you will have to edit module.ceylon to add the following imports:

import ceylon.http.client "1.2.3";
import ceylon.uri "1.2.3";
import ceylon.json "1.2.3";
import ceylon.collection "1.2.3";
import "1.2.3";

Next, we're going to turn our MainActivity into a ListActivity and run an asynchronous task to connect to the Herd REST endpoint, so edit MainActivity.ceylon with this:

import android.os { Bundle, AsyncTask }
import { ListActivity }
import android.widget { ArrayAdapter, ListAdapter }
import { AppCompatActivity }
import { createJavaStringArray }
import java.lang { JString = String }
import android { AndroidR = R }
import ceylon.language.meta { modules }
import ceylon.uri { parseUri = parse }
import ceylon.http.client { httpGet = get }
import ceylon.json { parseJson = parse, JsonObject = Object, JsonArray = Array }
import ceylon.collection { MutableList, LinkedList }

shared class MainActivity() extends ListActivity() {

    class LoadModules() extends AsyncTask<String, Nothing, List<String>>() {
        shared actual List<String> doInBackground(String?* uris){
            assert(exists uri = uris.first);
            value response = httpGet(parseUri(uri)).execute();
            value modules = LinkedList<String>();
            assert(is JsonObject json = parseJson(response.contents),
                    is JsonArray results = json["results"]);
            // Iterate modules
            for(res in results) {
                assert (is JsonObject res); // Get the list of versions
                assert (is String name = res["module"],
                        is JsonArray versions = res["versions"]);
            return modules;
        shared actual void onPostExecute(List<String> result){
            print("Got result: ``result``");

            ListAdapter adapter = ArrayAdapter<JString>(outer, AndroidR.Layout.simple_list_item_1,
            listAdapter = adapter;

    shared actual void onCreate(Bundle savedInstanceState) {


Now edit app/src/main/res/layout/activity_main.xml to change the activity type to a list activity:


And lastly request the network permission for your app, since we're hitting a web service, by adding this to app/src/main/res/AndroidManifest.xml:

<uses-permission android:name="android.permission.INTERNET"/>

That's all you need, now just click on Run > Run app and watch your Ceylon application display the list of modules in the emulator:

Some technical info

The Ceylon IntelliJ plugin has not been released yet, but a preview is forthcoming really soon. You will see it's already quite advanced when you try this out. Don't hesitate to report any bugs, or better yet, contribute fixes :)

Most of it works well enough for Android, except the caveats noted above, and the fact that Android Studio does not yet recognize Ceylon classes, so they will be marked as errors in the .xml files that refer to them, and when you run your application it will report an error:

Could not identify launch activity: Default Activity not found. Error while Launching activity

It only means it could not start your application, you will have to click on it to start it in the emulator. But the deployment worked. We're fixing this at the moment, so it will only improve.

If you want to revert to the Ceylon Eclipse IDE to edit your Ceylon Android application, you can, it will work once you have your project set up with Android Studio. It's much easier to use it to set it up so all the Gradle config is just right. Once that is done, you can use Eclipse if you want, and use $ ./gradlew assembleDebug to build your APK.

This work depends on changes we've made in Ceylon 1.2.3 (to be released soon) which adds support for jars which provide alternate smaller JDKs (such as the Android jar), improvements in modularity so that the created applications depend on much fewer runtime Ceylon jars than before, fixes in the runtime metamodel to support Android runtimes, and several other tweaks. I will probably write an account of all that in a future blog entry.

This work also depends on the Ceylon Gradle plugin written by Renato Athaydes, and on a new Ceylon Gradle Android plugin which adds support for Ceylon in Android applications. This plugin is by no means finished, and in particular does not yet support incremental compilation (even though the Ceylon IDE and compiler do). It also does not yet support the latest Android Instant Run feature. Again, please report issues or better, contribute pull-requests :)

Ceylon on mobile devices

Ceylon already runs on the JVM, whether bare-bones, via JBoss Modules, Vert.x, Java EE Servlet containers such as WildFly, or OSGi containers, as well as on JavaScript VMs such as Node.js and the browser. But today we're going to explain how to run Ceylon on mobile devices, not just in the browser (though it does play a part in it), but as applications, via Apache Cordova.

Apache Cordova allows you to write applications for every mobile platform, including Android and iOS, using nothing but HTML, CSS and JavaScript. Since Ceylon compiles to JavaScript this is perfect as it allows us to run our Ceylon applications on iOS, via the JavaScript compiler backend.

Note that this article is using Ceylon 1.2.3 which is not yet released, because the JavaScript runtime in Cordova on Android had one peculiarity we had to work around in the language module JavaScript implementation. Luckily you can get nightly builds of Ceylon 1.2.3 and the Ceylon 1.2.3 SDK.

Writing your Ceylon Cordova application

Installing Apache Cordova

First, install Apache Cordova and add two platforms. I haven't been able to test the iOS platform since it requires an OSX platform to build and an iOS device to test, and I lack both, so I will explain how to package for Android and the browser, and let you guys try it out for iOS, but I have enough faith in Apache Cordova that it will Just Work™.

# Install npm, the Node.js package manager
$ sudo apt-get install npm
# Then install Apache Cordova
$ npm install cordova

Small note: for me this installed things in ./node_modules/cordova and the Apache Cordova command in ./node_modules/cordova/bin/cordova, so adapt your path as you must.

# Create your application
$ cordova create ceylon-cordova-demo
$ cd ceylon-cordova-demo
# Now add the browser and Android platforms
$ cordova platform add browser
$ cordova platform add android

At this point you have your application ready to be checked in your browser:

$ cordova platform run browser

Or in an Android emulator, provided you have downloaded the Android SDK already:

$ ANDROID_HOME=.../Android/Sdk cordova platform build android
$ ANDROID_HOME=.../Android/Sdk cordova platform run android

Getting a little side-tracked about styling

Writing an application using just HTML and CSS means you have to make some effort for it to look good, and instead I decided to delegate to use Polymer so that my application would have the look and feel of Android Material Design applications to feel even more like a native application on Android. No doubt a similar look and feel exists for iOS.

So let's download Polymer in our application's HTML sources:

$ npm install bower
$ cd www
$ bower init
# At this point just hit enter/Yes/No until it's set up 
$ bower install --save Polymer/polymer
$ bower install --save Polymerelements/paper-item

And now edit www/index.html to use Polymer:

<script src="bower_components/webcomponentsjs/webcomponents.js"></script>
<link rel="import" href="bower_components/paper-item/paper-item.html">
<link rel="import" href="bower_components/paper-item/paper-item-body.html">

You should also remove the default CSS:

<link rel="stylesheet" type="text/css" href="css/index.css">

Getting require.js and jQuery

Ceylon compiles to JavaScript modules by way of require.js, so we're going to have to download it too:

$ cd www/js
$ wget

Our Ceylon demo will use jQuery to add elements to the HTML page, so we also need it:

$ cd www
$ bower install --save jquery

Now edit www/index.html to use both:

<script type="text/javascript" src="js/require.js"></script>
<script src="bower_components/jquery/dist/jquery.js"></script>

Writing the Ceylon application

We're going to write a trivial application that queries Ceylon Herd for the list of modules, to display them in a list.

Let's start by creating a Ceylon module in source/cordova/demo/module.ceylon:

module cordova.demo "1.0.0" {
    import ceylon.json "1.2.3";

And our application's main method in source/cordova/demo/run.ceylon:

import ceylon.json { parseJson = parse, JsonObject = Object, JsonArray = Array }

shared void run() {
    dynamic {
        // The HTML element where we'll add our items
        dynamic target = jQuery("#target");
        // The function called when we get data from the server
        void success(dynamic data){
            // Parse the JSON
            assert(is JsonObject json = parseJson(data),
                   is JsonArray results = json["results"]);
            // Iterate modules
            for(res in results){
              assert(is JsonObject res);
              // Get the list of versions
              assert(is String name = res["module"],
                     is JsonArray versions = res["versions"]);
              // Join them
              value versionText = ", ".join(versions.narrow<String>());
              // Now add the HTML items
              dynamic item = jQuery("<paper-item/>");
              dynamic body = jQuery("<paper-item-body two-line/>").appendTo(item);
              jQuery("<div secondary/>").text(versionText).appendTo(body);
        // Query Herd for the list of modules
        jQuery.get("", null, success, "text");

Now, obviously using jQuery to add HTML is far from ideal, so I can't wait for someone to extend ceylon.html to allow Polymer web components!

We can now compile our application for JavaScript:

$ ceylon compile-js

And copy our compiled module and all its dependencies to where the Apache Cordova application will find them in www/modules:

$ ceylon copy --with-dependencies --js --out www/modules cordova.demo/1.0.0

Invoking the Ceylon module from the Cordova application

Because we're going to use require.js inline and connect to Ceylon Herd, we have to adjust the Apache Cordova permissions in www/index.html, so find that line and edit it as such:

<meta http-equiv="Content-Security-Policy" 
      content="default-src 'self' 'unsafe-inline' 
               connect-src *">

We're left with just invoking our Ceylon function in www/index.html:

<script type="text/javascript">
  // tell require.js where our Ceylon modules are 
  // when the document is ready
    // load our Ceylon module
    require(['cordova/demo/1.0.0/cordova.demo-1.0.0'], function(client) {
      // and call our run method;

And setting up the target HTML elements where we're going to add every loaded module (in the same file):

<body id="app" unresolved>
  <app-shell class="fit">
    <div id="target" role="listbox"></div>

Trying it

And that's it, try it out in your browser:

$ cordova platform run browser

Or in an Android emulator:

$ ANDROID_HOME=.../Android/Sdk cordova platform build android
$ ANDROID_HOME=.../Android/Sdk cordova platform run android

If you have OSX and iOS dev tools, please try this with the iOS Cordova platform and let me know how it works :)

In the future, we would benefit from having a type-safe API in front of the Cordova JavaScript API that lets you access native mobile APIs such as the camera, GPS, contacts, but even without it you can already use them using dynamic blocks.

And now for the teaser… this is only one method to run Ceylon on Android, because obviously it may be more desirable to use the JVM compiler backend and integrate with the Android Tools to run Ceylon on Android using only type-safe APIs. Don't worry, it's coming, and next week I will show you how :)