Blog tagged modularity

Modularity Changes

TL;DR: This article describes the modularity changes in the Ceylon run-time and distribution, in order to make them lighter at run-time. Skip to the Final runtime dependencies section if you just want the outcome.

As we initially expected most Ceylon users to run their code using the ceylon run command, we figured that since they have the Ceylon distribution installed, it does not matter if they depend on more modules from that distribution than strictly necessary. Those modules had to be there anyway, so it would not save any bandwidth to reduce those dependencies.

Naturally, we were wrong, and between the Ceylon Eclipse or IntelliJ IDEs, running Ceylon on OpenShift, WildFly, Vert.x or on Android, people started running Ceylon without the distribution installed, using just the standard java runner. It became soon apparent that we had to untangle those dependencies to make the runtime requirements lighter.

Historically we had the following modules:

  • common module used by other modules
  • typechecker (the shared compiler front-end)
  • Java compiler back-end
  • JavaScript compiler back-end
  • module repository system
  • JBoss modules runtime

The model module

When we implemented reified generics, we had to add subtyping to the runtime, so that we'd be able to figure out if is T bar was true or not. The easiest thing at the time was to "just" depend on the typechecker (compiler front-end) which dealt with the language model and subtyping, and the Java compiler back-end, which had infrastructure to load a language model from JVM information such as class files, or in this case reflection.

This essentially made the runtime depend on the compiler front-end and back-end, which we realised was not ideal, so during the Ceylon 1.2 development, we extracted all model description, loading and subtyping to a new ceylon-model module, but we did not have enough time to do more and so these dependencies remained due to other causes.

Supporting Java 9

During our work on supporting Java 9 / Jigsaw modules in Ceylon, it became clear that having kept our "fork" of javalang tools (that we use for javac) under its original package name would not work anymore, we renamed its package and used the opportunity to prune away parts of the java tools we did not use. We also extracted the class-file reader part to its own module so we could use it outside of the compiler to remove our dependency to jandex (a class-file scanner).

Finally, when we created the ceylon jigsaw tool (which populates a folder with the jar files required by a Ceylon module, to run it on a Java 9 VM) it became evident that the runtime still depended not just on the compiler front-end and Java back-end, but even on the JavaScript back-end, which frankly made little sense in most JVM executions.

These dependencies were due to the Ceylon Tool Provider API having snuck into the ceylon.language module as a convenience (at the time). Since that allowed you to compile and run Ceylon programmatically for both Java and JavaScript back-ends, it had to depend on the tools.

We decided to split the Ceylon Tool Provider into its own module and got rid of the final dependencies from the language module to the compilers and typechecker, but had no more time to get rid of further dependencies such as JBoss Modules and Aether in time for Ceylon 1.2.2.

Supporting Android

Initial work on running Ceylon on Android revealed that what passes for small dependencies on ordinary JVM executions, or even on Java EE deployments, was not an option on Android where every method counts.

At this point we had to bite the bullet and make every non-required transitive dependency go.

We noticed that the old common module had grew to include the Command-Line Tooling API that makes the ceylon command and its subcommands and plugins work. That in turn depended on a Markdown renderer used by ceylon doc. It was pretty trivial to extract it to its own module because this was never used in Ceylon user programs.

Next in line was our Shrinkwrap Resolver dependency, which our module repository system uses to interoperate with Maven repositories. This was a fat-jar with all its dependencies included, including some Apache Commons modules, and an outdated version of Eclipse Aether. That fat-jar had already been problematic in our Maven module, which already had its version of Aether, so getting rid of the fat-jar was a good idea. We also realised that some of its Apache Commons dependencies were already included outside the fat-jar in our distribution repository, so there was that duplication to fix too.

So what we did was remove the Shrinkwrap Resolver dependency and use Aether directly, by incorporating all its subcomponents into our distribution. It turns out that because the latest version of Aether requires Google Guava, our distribution grew in size rather than shrink (that jar is huge). But to offset that, we made the Aether dependency optional, and made sure it was possible to run Ceylon without it as long as there was some compilation step beforehand that provided all the Maven dependencies that you may use in interop. ceylon fat-jar or ceylon jigsaw would do that for you, for example.

Our module repository system also provided support for writing to WebDAV or Herd repositories, which required some dependencies on Apache Http Client or Sardine, and we made these dependencies optional as well, because at runtime your Ceylon program is very unlikely to write to HTTP repositories. This is something only the compiler and other tools do.

We also removed a dependency to JBoss Modules from the language module using abstraction, since that platform was optional and never used on Android or other flat-classpath runtimes.

Finally, the language module only had one dependency left on the (much slimmer) module repository system via the presence of the Main API in there, and we moved that class to its own module.

Final runtime dependencies

After all this pruning, the language module on the JVM is back down to requiring the following set of transitive dependencies:

  • common (small and free of tooling and dependencies)
  • model (which depends only on the class-file reader)
  • class-file reader

So your Ceylon module will only depend on four jars (these three and the language module), the sum size of which is 2.4 Mb, which is much smaller than initially, and has dramatically less methods, at around 17148 methods. This is still too much, but can be brought down by tooling such as ProGuard to remove unused classes. Remember this includes a runtime for an entire language, so it's not that big, all things considered.

SDK changes

In order to be able to use Ceylon's HTTP client on Android, we also split up the module from the Ceylon SDK into client and server modules. Otherwise the HTTP server and its dependencies were too much drag for Android's method count.

Ceylon Fat Jars

Ceylon 1.2.3 is looming closer, and so we should start talking about some of the new features that it will contain. One of those new features is the new ceylon fat-jar command.

ceylon fat-jar my.module/1 lets you generate a jar which contains the my.module/1 module and every required dependency. Once you have that jar, you can execute it with java -jar my.module-1.jar and it will execute the module.

Naturally, you can customise the function or class to execute with the --run option, but the default is the run method in your module (my.module::run in our case).

The modules are executed in a "flat classpath" mode, without JBoss Modules or any sort of module isolation, and with the metamodel already set up statically when the fat jar was created, so it's a little bit different to the older alternative of running Ceylon modules using the Main API and ceylon classpath tool, which did not deal with packing all the dependencies. It is also different to the ceylon run --flat-classpath in that it does not require a module repository at run-time, since ceylon fat-jar packs it all together.

Try it out, it's pretty useful to distribute Ceylon programs and run them in places without having to install the Ceylon distribution.

Ceylon on Android

In my last post, I explained how you can use Ceylon in Apache Cordova to write applications for every mobile platform, including iOS and Android. This time, with many apologies for writing it late (“next week” turned into next month), I will explain how to use the Ceylon IntelliJ plugin to write native Android applications in Ceylon in Android Studio.

Getting started with Ceylon on Android Studio

To start writing Ceylon applications in Android Studio, follow these steps:

  • Download Android Studio
  • Start it
  • Create a new application by clicking on Start a new Android Studio Project
  • You can use these values for Application name: CeylonDemo
  • And for Company domain:
  • Next, select an Empty activity, with Activity Name: MainActivity
  • Click Finish and wait for the project to be created

At this point you have an Android project open, but we still haven't had time to install the Ceylon plugin, so let's do this right now:

  • Click on File > Settings > Plugins > Browse Repositories
  • Then on Manage Repositories > +
  • Add this repository:
  • Now click on Install Ceylon IDE

You will likely need to restart Android Studio, so do that.

Next we're going to convert our Android project to a Ceylon Android project:

  • In the Android view, Right-click on app > Configure Ceylon in this Module
  • Click OK on the resulting configuration dialog

This will set up the Ceylon plugin, and will add most of what you need in your Gradle build to build Ceylon Android applications. You now have your Ceylon sources in app/src/main/ceylon and it includes a module descriptor and an empty activity:

At the moment, this requires a Ceylon 1.2.3 distribution to build, and since it's not released yet you're going to have to either build one yourself (just the Getting the source part), or download a nightly build. Once you have it, edit app/build.gradle near then end to add ceylon > ceylonLocation and make it point to where you installed your distribution (it needs to point to the Ceylon binary, not just the distribution root):

ceylon {
    // ...
    ceylonLocation ".../ceylon/dist/dist/bin/ceylon"

Make sure you click on Sync now to sync your Gradle build.

Now, there's a bug we're in the process of fixing which fails to detect the exact version of the Android SDK tooling and modules, and so depending on which version of the Android Tools you're using you may have to sync the imports of in app/build.gradle (in dependencies) and in the Ceylon module descriptor in module.ceylon. Make sure the Ceylon import version is the same as the Gradle import version, because the Gradle build is what makes it available to Ceylon, due to Android's peculiarities.

In order to finish the conversion, make sure you delete the Java activity (since we're going to keep the Ceylon one), in Project Files, delete app/src/main/java.

Due to another pending plugin fix, you may have to click on Tools > Ceylon > Reset Ceylon Model at this point so that the Ceylon plugin gets synchronised with all these past changes (don't worry we're fixing this at the moment).

The good news is we're already able to click on Run app and try this in the emulator, but we're going to make it a little more interesting.

Customising your Ceylon Android activity

We're going to be displaying a list of Ceylon modules published on Ceylon Herd, so we will make use of the Ceylon SDK, and in particular you will have to edit module.ceylon to add the following imports:

import ceylon.http.client "1.2.3";
import ceylon.uri "1.2.3";
import ceylon.json "1.2.3";
import ceylon.collection "1.2.3";
import "1.2.3";

Next, we're going to turn our MainActivity into a ListActivity and run an asynchronous task to connect to the Herd REST endpoint, so edit MainActivity.ceylon with this:

import android.os { Bundle, AsyncTask }
import { ListActivity }
import android.widget { ArrayAdapter, ListAdapter }
import { AppCompatActivity }
import { createJavaStringArray }
import java.lang { JString = String }
import android { AndroidR = R }
import ceylon.language.meta { modules }
import ceylon.uri { parseUri = parse }
import ceylon.http.client { httpGet = get }
import ceylon.json { parseJson = parse, JsonObject = Object, JsonArray = Array }
import ceylon.collection { MutableList, LinkedList }

shared class MainActivity() extends ListActivity() {

    class LoadModules() extends AsyncTask<String, Nothing, List<String>>() {
        shared actual List<String> doInBackground(String?* uris){
            assert(exists uri = uris.first);
            value response = httpGet(parseUri(uri)).execute();
            value modules = LinkedList<String>();
            assert(is JsonObject json = parseJson(response.contents),
                    is JsonArray results = json["results"]);
            // Iterate modules
            for(res in results) {
                assert (is JsonObject res); // Get the list of versions
                assert (is String name = res["module"],
                        is JsonArray versions = res["versions"]);
            return modules;
        shared actual void onPostExecute(List<String> result){
            print("Got result: ``result``");

            ListAdapter adapter = ArrayAdapter<JString>(outer, AndroidR.Layout.simple_list_item_1,
            listAdapter = adapter;

    shared actual void onCreate(Bundle savedInstanceState) {


Now edit app/src/main/res/layout/activity_main.xml to change the activity type to a list activity:


And lastly request the network permission for your app, since we're hitting a web service, by adding this to app/src/main/res/AndroidManifest.xml:

<uses-permission android:name="android.permission.INTERNET"/>

That's all you need, now just click on Run > Run app and watch your Ceylon application display the list of modules in the emulator:

Some technical info

The Ceylon IntelliJ plugin has not been released yet, but a preview is forthcoming really soon. You will see it's already quite advanced when you try this out. Don't hesitate to report any bugs, or better yet, contribute fixes :)

Most of it works well enough for Android, except the caveats noted above, and the fact that Android Studio does not yet recognize Ceylon classes, so they will be marked as errors in the .xml files that refer to them, and when you run your application it will report an error:

Could not identify launch activity: Default Activity not found. Error while Launching activity

It only means it could not start your application, you will have to click on it to start it in the emulator. But the deployment worked. We're fixing this at the moment, so it will only improve.

If you want to revert to the Ceylon Eclipse IDE to edit your Ceylon Android application, you can, it will work once you have your project set up with Android Studio. It's much easier to use it to set it up so all the Gradle config is just right. Once that is done, you can use Eclipse if you want, and use $ ./gradlew assembleDebug to build your APK.

This work depends on changes we've made in Ceylon 1.2.3 (to be released soon) which adds support for jars which provide alternate smaller JDKs (such as the Android jar), improvements in modularity so that the created applications depend on much fewer runtime Ceylon jars than before, fixes in the runtime metamodel to support Android runtimes, and several other tweaks. I will probably write an account of all that in a future blog entry.

This work also depends on the Ceylon Gradle plugin written by Renato Athaydes, and on a new Ceylon Gradle Android plugin which adds support for Ceylon in Android applications. This plugin is by no means finished, and in particular does not yet support incremental compilation (even though the Ceylon IDE and compiler do). It also does not yet support the latest Android Instant Run feature. Again, please report issues or better, contribute pull-requests :)

Ceylon on mobile devices

Ceylon already runs on the JVM, whether bare-bones, via JBoss Modules, Vert.x, Java EE Servlet containers such as WildFly, or OSGi containers, as well as on JavaScript VMs such as Node.js and the browser. But today we're going to explain how to run Ceylon on mobile devices, not just in the browser (though it does play a part in it), but as applications, via Apache Cordova.

Apache Cordova allows you to write applications for every mobile platform, including Android and iOS, using nothing but HTML, CSS and JavaScript. Since Ceylon compiles to JavaScript this is perfect as it allows us to run our Ceylon applications on iOS, via the JavaScript compiler backend.

Note that this article is using Ceylon 1.2.3 which is not yet released, because the JavaScript runtime in Cordova on Android had one peculiarity we had to work around in the language module JavaScript implementation. Luckily you can get nightly builds of Ceylon 1.2.3 and the Ceylon 1.2.3 SDK.

Writing your Ceylon Cordova application

Installing Apache Cordova

First, install Apache Cordova and add two platforms. I haven't been able to test the iOS platform since it requires an OSX platform to build and an iOS device to test, and I lack both, so I will explain how to package for Android and the browser, and let you guys try it out for iOS, but I have enough faith in Apache Cordova that it will Just Work™.

# Install npm, the Node.js package manager
$ sudo apt-get install npm
# Then install Apache Cordova
$ npm install cordova

Small note: for me this installed things in ./node_modules/cordova and the Apache Cordova command in ./node_modules/cordova/bin/cordova, so adapt your path as you must.

# Create your application
$ cordova create ceylon-cordova-demo
$ cd ceylon-cordova-demo
# Now add the browser and Android platforms
$ cordova platform add browser
$ cordova platform add android

At this point you have your application ready to be checked in your browser:

$ cordova platform run browser

Or in an Android emulator, provided you have downloaded the Android SDK already:

$ ANDROID_HOME=.../Android/Sdk cordova platform build android
$ ANDROID_HOME=.../Android/Sdk cordova platform run android

Getting a little side-tracked about styling

Writing an application using just HTML and CSS means you have to make some effort for it to look good, and instead I decided to delegate to use Polymer so that my application would have the look and feel of Android Material Design applications to feel even more like a native application on Android. No doubt a similar look and feel exists for iOS.

So let's download Polymer in our application's HTML sources:

$ npm install bower
$ cd www
$ bower init
# At this point just hit enter/Yes/No until it's set up 
$ bower install --save Polymer/polymer
$ bower install --save Polymerelements/paper-item

And now edit www/index.html to use Polymer:

<script src="bower_components/webcomponentsjs/webcomponents.js"></script>
<link rel="import" href="bower_components/paper-item/paper-item.html">
<link rel="import" href="bower_components/paper-item/paper-item-body.html">

You should also remove the default CSS:

<link rel="stylesheet" type="text/css" href="css/index.css">

Getting require.js and jQuery

Ceylon compiles to JavaScript modules by way of require.js, so we're going to have to download it too:

$ cd www/js
$ wget

Our Ceylon demo will use jQuery to add elements to the HTML page, so we also need it:

$ cd www
$ bower install --save jquery

Now edit www/index.html to use both:

<script type="text/javascript" src="js/require.js"></script>
<script src="bower_components/jquery/dist/jquery.js"></script>

Writing the Ceylon application

We're going to write a trivial application that queries Ceylon Herd for the list of modules, to display them in a list.

Let's start by creating a Ceylon module in source/cordova/demo/module.ceylon:

module cordova.demo "1.0.0" {
    import ceylon.json "1.2.3";

And our application's main method in source/cordova/demo/run.ceylon:

import ceylon.json { parseJson = parse, JsonObject = Object, JsonArray = Array }

shared void run() {
    dynamic {
        // The HTML element where we'll add our items
        dynamic target = jQuery("#target");
        // The function called when we get data from the server
        void success(dynamic data){
            // Parse the JSON
            assert(is JsonObject json = parseJson(data),
                   is JsonArray results = json["results"]);
            // Iterate modules
            for(res in results){
              assert(is JsonObject res);
              // Get the list of versions
              assert(is String name = res["module"],
                     is JsonArray versions = res["versions"]);
              // Join them
              value versionText = ", ".join(versions.narrow<String>());
              // Now add the HTML items
              dynamic item = jQuery("<paper-item/>");
              dynamic body = jQuery("<paper-item-body two-line/>").appendTo(item);
              jQuery("<div secondary/>").text(versionText).appendTo(body);
        // Query Herd for the list of modules
        jQuery.get("", null, success, "text");

Now, obviously using jQuery to add HTML is far from ideal, so I can't wait for someone to extend ceylon.html to allow Polymer web components!

We can now compile our application for JavaScript:

$ ceylon compile-js

And copy our compiled module and all its dependencies to where the Apache Cordova application will find them in www/modules:

$ ceylon copy --with-dependencies --js --out www/modules cordova.demo/1.0.0

Invoking the Ceylon module from the Cordova application

Because we're going to use require.js inline and connect to Ceylon Herd, we have to adjust the Apache Cordova permissions in www/index.html, so find that line and edit it as such:

<meta http-equiv="Content-Security-Policy" 
      content="default-src 'self' 'unsafe-inline' 
               connect-src *">

We're left with just invoking our Ceylon function in www/index.html:

<script type="text/javascript">
  // tell require.js where our Ceylon modules are 
  // when the document is ready
    // load our Ceylon module
    require(['cordova/demo/1.0.0/cordova.demo-1.0.0'], function(client) {
      // and call our run method;

And setting up the target HTML elements where we're going to add every loaded module (in the same file):

<body id="app" unresolved>
  <app-shell class="fit">
    <div id="target" role="listbox"></div>

Trying it

And that's it, try it out in your browser:

$ cordova platform run browser

Or in an Android emulator:

$ ANDROID_HOME=.../Android/Sdk cordova platform build android
$ ANDROID_HOME=.../Android/Sdk cordova platform run android

If you have OSX and iOS dev tools, please try this with the iOS Cordova platform and let me know how it works :)

In the future, we would benefit from having a type-safe API in front of the Cordova JavaScript API that lets you access native mobile APIs such as the camera, GPS, contacts, but even without it you can already use them using dynamic blocks.

And now for the teaser… this is only one method to run Ceylon on Android, because obviously it may be more desirable to use the JVM compiler backend and integrate with the Android Tools to run Ceylon on Android using only type-safe APIs. Don't worry, it's coming, and next week I will show you how :)

Ceylon on Java 9 + Jigsaw

Everyone is talking about modules these days. New languages try to incorporate them, and older languages try to retrofit them in. Which is great news, because modules are essential. Java 9 is around the corner, because it's supposed to come out next year, and the really big new feature is modularity, which it calls the Jigsaw project.

Ceylon is a language that featured modularity from the start, as part of the language and not as an afterthought requiring complex third-party tool integration. In fact, at the time we designed our Java JDK integration (at the time of Java 7), we went as far as using the Jigsaw modularity plans for the JDK (yes Jigsaw got delayed a few times) from the start, requiring JDK users to import Jigsaw modules as they were planned at the time, rather than import the whole JDK in one go. So perhaps we were the first ones with a modular JDK, in some sense :)

Java 9’s Jigsaw

Jigsaw is a very large project, which includes the following changes:

  • Modularisation of the JDK into smaller units, such as java.base, java.xml that Ceylon users of the JDK are already familiar with.
  • This modularisation means removal of rt.jar that contained every JDK class. In fact it's been replaced by a bootmodules.jimage file which is not a jar, but whose contents can be accessed by a virtual NIO FileSystem at jrt:/.
  • You can write your own modules. To turn your Java code into a Java 9 module, you simply add a module descriptor in a file called (much like Ceylon module descriptors, or Java package descriptors), which describes your module and the Java 9 compiler and jar tools will then generate a jar with a module-info.class descriptor at the root of the jar.
  • That module descriptor allows you to specify the module name, the packages it exports, the name of the modules it imports and a few other things. But not versions, unfortunately, which are currently "out of scope" in Java 9.
  • You can run your code as previously from the classpath, or as modules from the module path. The module path is just a folder in which you can place your modules and the JRE will look them up for you based on module name alone.

Ceylon and Jigsaw

Java 9 has two early-access (EA) downloads for users to try the module system. Only one of them includes user modules. Make sure you use that one if you want to try out Ceylon running on Java 9.

Over the past weeks I've worked on getting Ceylon compiling and running on Java 9. This involved (among other details) the following things:

  • Generating module-info.class files from Ceylon module descriptors.
  • Generating module-info.class files for the Ceylon distribution modules which are not written in Ceylon (like the compilers or runtime system).
  • Making use of the Java 9 module descriptors for the shared packages information it contains (something supported by Ceylon since the beginning, but which was lacking for plain Java jars).
  • Backporting Java 9 code that deals with modules to the javac fork we use to compile Java files and generate bytecode.
  • Dealing with the removal of rt.jar and the boot classpath.
  • Creating a new tool ceylon jigsaw which allows for the creation of a Java 9 module path.
  • Making sure we can run Ceylon modules as Java 9 modules as an alternative to the four existing JVM runtimes which are the JBoss Modules, classpath, OSGi or Java EE.
  • Make sure we can build and run on any of Java 7,8,9. This means that by default we do not generate Java 9 module descriptors, because several tools have problems dealing with them at this time.
  • We have split some things out of the ceylon.language module so that it no longer depends on the compilers and type-checker, which means a lighter minimal runtime, which will be even further improved in the next weeks with more dependency removals :)

Just tell me how to try this!

I will spare you the many details of this work, but with help from the Java 9 team, this is how you can run your Ceylon modules on a Java 9 runtime:

  • Download the Java 9 EA with Jigsaw.
  • Get the Ceylon distribution code, and compile it with ant -Djigsaw=true clean dist to get the Java 9 module descriptors.
  • Write your Ceylon module normally, but compile it with .../ceylon/dist/dist/bin/ceylon compile --generate-module-info to generate the Java 9 module descriptors.
  • Create your Java 9 module path in an mlib folder with .../ceylon/dist/dist/bin/ceylon jigsaw create-mlib my.module/1.
  • Run your Ceylon module on Java 9 with .../jdk1.9.0-jigsaw/bin/java -mp mlib -m ceylon.language my.module/1. At the moment, the ceylon.language module acts as main module and does the required setting up of the Ceylon runtime before loading and invoking your Ceylon module.

That's all there is to it!


Java 9 is not complete yet, and our support for Java 9 is also not complete. There will be issues and bugs, and in fact we already know of several limitations, such as the following:

  • While you can import a pure Java 9 module from Ceylon, we will respect its exported packages, but we will not respect its dependencies, because Java 9 modules do not include dependency versions. In fact, even the module's version is not stored in the source module descriptor, but added by an optional flag to the Java 9 jar tool. Ceylon requires module dependencies to describe a version, so we have to combine the Java 9 module descriptor with another descriptor such as an OSGi descriptor or a Maven pom.xml descriptor. This merging of information is not currently done.
  • Java 9 does not currently support optional modules or module cycles. It is not clear if they will support them at this time, unfortunately.
  • The ceylon import-jar tool may complain about module visibility artifacts. We intend to fix this in time, but for now you can use --force.
  • The JDK module list we used in Ceylon has slightly changed in Java 9. This is what we get for being the first to support Jigsaw ;) For example, the javax.xml module has been renamed to java.xml. We have set up aliases so that it "just" works, but there are modules that have been merged, and packages that have changed module, so it will not always work.
  • The Java 9 runtime has been tested, but not as thoroughly as the existing JBoss Modules, classpath, OSGi or Java EE runtimes. We expect a few issues in the Ceylon metamodel.