Code by Martin

Binary Search

2010-04-21T20:41:00+02:00

Mike Taylor noted in a blog post that in the book Programming Pearls it is mentioned that no more than 10% of developers can correctly implement a binary search. Appalling!

Let’s see if I am in that 10% or not…

As I’ve found myself using Ruby more and more lately, I thought I’d try to implement the binary search using that language.

Here’s my stab:

The script includes code to run the binary search through the test cases for the exercise created by Steve Witham. And once the file reading worked - all tests actually passed (or the code incorrectly shows them as passed… ;).

Good training this!

Migrated To Octopress

2010-03-19T21:23:00+01:00

Like all cool kids blogging nowadays, as part of the attempt to rejuvinate the blog, I’ve switched to Jekyll. Or, rather, the extended fork Octopress.

As part of the migration I wanted to extract all the old posts from Wordpress into the proper markdown format used by Jekyll.

Jekyll includes a migration script for Wordpress, but this used a direct database connection to the Wordpress MySQL server to extract a minimum amount of information for each blog post. To avoid missing information that I later might want after having shut down the Wordpress server, I wanted to base the migration on a full XML export from Wordpress. This way I could go back and extract more information from the old posts, if, and when, needed.

Seeing this as a chance to play with Ruby - a language I’ve only dabbled in previously - I set out to build a simple Wordpress XML->Jekyll converter. The converter reads the Wordpress export and converts all blog posts into separate files with a proper Yaml frontmatter. It also converts the syntax highlighting markup, code markup and headings into the corresponding markdown conventions. The rest of the HTML is left alone.

For the images I just copied the files in the old Wordpress structure into the new blog directory layout so that the URLs still match. Not the nicest way, but I was too lazy to do anything about it just yet.

Anywho - the source for the Wordpress XML converter is available in the blog branch of my fork of Octopress. The source is also given below. To use, just run with the filename of the Wordpress XML export as the first argument.

It’s my first publicly available Ruby program - so be careful. Improvements welcome!

Nginx

2010-03-16T00:00:00+01:00

For that fresh new feeling I switched this site from Apache2 to Nginx. A quick compile of the source package and following the instructions on hooking it up with Wordpress over fastcgi using php-cgi and we’re done!

Feels good with a server handling the C10K problem - and it has a fairly decent configuration language IMHO.

Also looking forward to use Nginx more in the days to come as a the web frontend for development tools server at work hosting, among some things:

Gitorious
Redmine
Hudson
Nexus

Will post more about this later!

New Direction?

2010-03-16T00:00:00+01:00

More than a full year since last post - this might not be working as well as I had hoped…!

Finding the time and inspiration to write between the day job and private commitments turns out to be a challenge. Instead of seeing this as a failure, however, I will try to learn from the experience and attempt to find a new way of relating to the blog. Perhaps by being less long winded (!) in each post I could get a few more out there. There are still a lot of things going on I would like to capture, but the previous form is clearly not working.

So - let’s try to shake things up and see how that works!

The Hitchhiker’s Guide to an Ioke Dev Env From Source (part 5: Ioke and the REPL)

2009-01-23T00:00:00+01:00

This is the fifth part in a series of posts for non-experts about setting up an Ioke development environment on Linux. Please see the previous posts to start at the beginning:

This time we will finally get the Ioke source code, build it, and test the Ioke REPL!

Ioke

Ioke uses Git as the source code versioning tool, with a public repository set up on GitHub (soon I’ll be linking like Jeff Atwood).

So - let’s get Ioke using Git by cloning Ola’s repository:

~/bin$ cd ~/work
      ~/work$ git clone git://github.com/olabini/ioke
      Initialized empty Git repository in /home/melwin/work/ioke/.git/
      remote: Counting objects: 9221, done.
      remote: Compressing objects: 100% (2941/2941), done.
      remote: Total 9221 (delta 5782), reused 8656 (delta 5351)
      Receiving objects: 100% (9221/9221), 45.95 MiB | 399 KiB/s, done.
      Resolving deltas: 100% (5782/5782), done.

Great - let’s quickly move on to the build step:

~/work$ cd ioke
      ~/work/ioke$ ant
      ...
           [java] 2606 examples, 0 failures
      jar:
            [jar] Building jar: /home/melwin/work/ioke/lib/ioke.jar
      
      BUILD SUCCESSFUL
      Total time: 22 seconds

Running just ant will execute the default target in the build.xml file. This target will make sure Ioke is compiled and the test suite is run.

At the end is a summary of the test results - if these are not all successful you might have a problem with the environment or you might have checked out a broken build from the repository.

The Ioke REPL

Now all is installed and Ioke is compiled, so let’s fire up the REPL. This is done by just running the main Ioke binary:

~/work/ioke/$ bin/ioke
      iik>

When doing this we get the iik> REPL prompt. From here we can execute most Ioke code. Let’s try to print a string:

Here we create a Text (Ioke’s string type) literal and send it the message println, which will cause the text to be printed to the console. The return type of println is nil (similar to null), which is printed by the REPL itself. After that we get the prompt back and can execute another line.

Let’s try a little more complex example - the factorial:

Note how the REPL prints the method definition in a canonical form - it makes it obvious how Ioke parses the code, what are messages and what are arguments.

Let’s run our new factorial function:

Ioke handles arbitrarily big numbers, so we don’t need to think about what fits in a certain number of bits.

The Iik REPL also comes with a simple debugger, which helps with handling conditions. For instance, let’s try to run the fact method, passing a message which doesn’t mean anything (yet):

The above output shows that when we try to reference something that doesn’t exist we get a chance to provide that value. Here I decided to provide a value using the useValue restart and entered the value 5. This let the method continue executing and the result was printed.

To exit the REPL, just type: exit

Next up are the Emacs modes for Ioke - stay tuned!

The Hitchhiker’s Guide to an Ioke Dev Env From Source (part 4: Java+Ant)

2009-01-22T00:00:00+01:00

This is the fourth part in a series of posts for non-experts about setting up an Ioke development environment on Linux. Please see the previous posts to start at the beginning:

It took a bit longer than anticipated to get to this point, as I got side tracked with the Ioke type checking activities. But now when Ioke S is out we can continue with the series. So! We’re getting closer… Only a few more things to go.

Installing Java

As the current version of Ioke is implemented to run on a Java Virtual Machine, we need to get hold of one of those to be able to continue. The JVM brand we will install here is the latest stable one produced by Sun.

What we need is Java SE Development Kit 6u11 for Linux, Multi-language, and the the filename for this is jdk-6u11-linux-i586.bin. Note that this is not the .rpm.bin, which is also available.

Download the file from http://java.sun.com/javase/downloads/index.jsp (I’m sure you can find it) and put it in the ~/work directory.

To unpack the file, let’s make it executable and then run it:

~$ cd ~/work
      ~/work$ cd ~/work                   
      ~/work$ chmod a+x jdk-6u11-linux-i586.bin 
      ~/work$ ./jdk-6u11-linux-i586.bin         
      ... lots of legalese here...

Once the license agreement is accepted, the JVM will unpack itself into the directory ~/work/jdk1.6.0_11. Let’s move it to where we want it and add java and javac to the ~/bin directory as before:

~/work$ mv jdk1.6.0_11 ../opt
      ~/work$ cd ~/bin
      ~/bin$ ln -s ../opt/jdk1.6.0_11/bin/java
      ~/bin$ ln -s ../opt/jdk1.6.0_11/bin/javac

And test it:

~/bin$ java -version
      java version "1.6.0_11"
      Java(TM) SE Runtime Environment (build 1.6.0_11-b03)
      Java HotSpot(TM) Client VM (build 11.0-b16, mixed mode, sharing)

That done - let’s move on to Ant.

Ant

Ioke uses Apache Ant to handle the build process, so to build Ioke we need to get Ant installed.

Let’s download the Ant distributable package and unpack it in one go - as we did with Git:

~/bin$ cd ~/opt
      ~/opt$ wget -O - http://www.apache.org/dist/ant/binaries/apache-ant-1.7.1-bin.tar.bz2 | tar xjv
      
      
      And add it to the ~/bin directory again so it's on the path:
      
      lang="sh">
      ~/opt$ cd ~/bin
      ~/bin$ ln -s ../opt/apache-ant-1.7.1/bin/ant

Let’s try it out:

~/bin$ ant -version
      Apache Ant version 1.7.1 compiled on June 27 2008

Looking good!

Soon to come: How to get the Ioke source and build it.

Update: Part 5: Ioke and the REPL

Type Checking in Ioke Java Methods

2009-01-19T00:00:00+01:00

Ola Bini recently issued a call to arms to help with the receiver and argument validation for Java methods in Ioke. These are the Ioke methods that are implemented in Java, instead of in Ioke itself.

The Java methods usually operate on the specific data contained within Ioke objects. This data corresponds to different Java classes, depending on what the Ioke object should hold. For instance, an Ioke List contains inside the data area a Java List. To operate on this List the Java code needs to get hold of the List reference - and it does this in a lot of cases by assuming the data object is of a certain type and casts to this type. If the data object is of a different type then a Java ClassCastException is thrown, which makes the interpreter quit.

Since Ola mailed the call to arms a number of things have happened. A few people (including myself) have volunteered to help out with adding the validation code. Additionally, a few supporting methods and additions have been added to the Ioke code base to simplify adding type checks to the existing code. And it’s these additions that I will talk about here.

Current Code

As an example what the new type validations can look like I will use the List + method that Ola also mentioned in the example.

This is the previous code from IokeList.java:

obj.registerMethod(runtime.newJavaMethod("returns a new list that contains the receivers elements and the elements of the list sent in as the argument.", new JavaMethod("+") {
              private final DefaultArgumentsDefinition ARGUMENTS = DefaultArgumentsDefinition
                  .builder()
                  .withRequiredPositional("otherList")
                  .getArguments();
              
              @Override
              public DefaultArgumentsDefinition getArguments() {
                  return ARGUMENTS;
              }
              
              @Override
              public Object activate(IokeObject method, IokeObject context, IokeObject message, Object on) throws ControlFlow {
                  List<Object> args = new ArrayList<Object>();
                  getArguments().getEvaluatedArguments(context, message, on, args, new HashMap<String, Object>());
                  List<Object> newList = new ArrayList<Object>();
                  newList.addAll(((IokeList)IokeObject.data(on)).getList());
                  newList.addAll(((IokeList)IokeObject.data(args.get(0))).getList());
                  return context.runtime.newList(newList, IokeObject.as(on));
              }
          }));

New Type Checks

The key to the new validation are two new classes, which extend the functionality of the normal JavaMethod class with type checking functionality. These are:

When using these to define the Java method we can “annotate” the arguments definition with types for both the arguments and the receiver. By then implementing the appropriate activate(...) the super class takes care of evaluating and validating (converting as appropriate) the receiver and arguments.

For instance, for the List + example, the above code gets changed to the following:

obj.registerMethod(runtime.newJavaMethod("returns a new list that contains the receivers elements and the elements of the list sent in as the argument.", new TypeCheckingJavaMethod("+") {
              private final TypeCheckingArgumentsDefinition ARGUMENTS = TypeCheckingArgumentsDefinition
                  .builder()
                  .receiverMustMimic(runtime.list)
                  .withRequiredPositional("otherList").whichMustMimic(runtime.list)
                  .getArguments();
              
              @Override
              public TypeCheckingArgumentsDefinition getArguments() {
                  return ARGUMENTS;
              }
              
              @Override
              public Object activate(IokeObject self, Object on, List<Object> args, Map<String, Object> keywords, IokeObject context, IokeObject message) throws ControlFlow {
                  List<Object> newList = new ArrayList<Object>();
                  newList.addAll(((IokeList)IokeObject.data(on)).getList());
                  newList.addAll(((IokeList)IokeObject.data(args.get(0))).getList());
                  return context.runtime.newList(newList, IokeObject.as(on));
              }
          }));

Note that most of the boiler plate code for arguments handling (which most JavaMethods do) is removed, leaving a clean implementation of the necessary logic - to concatenate two lists in this case.

A few things are done:

The JavaMethod is changed to TypeCheckingJavaMethod
The arguments definition is changed to a TypeCheckingArgumentsDefinition
The appropriate types are added to the arguments definition
Return type of getArguments is changed to TypeCheckingArgumentsDefinition
The active method is changed to the one which gets the arguments list and keywords
The manual call to getArguments().getEvaluatedArguments() is removed as we now get the arguments passed

The relevant tests for this, which should be added to the list_spec.ik test file, could look like:

So for you to do:

Fork Ioke
Pick a `IokeData` subclass which no one has started on yet (check with Ola/naeu/me in the IRC channel #ioke on FreeNode)
Identify a method which makes faulty assumptions on receiver or arguments
Add a test as per the above to validate the arguments and receiver
Change `JavaMethod` implementation as per the above to fix the test
Commit test and fix in togther to your fork
Rinse and repeat until the whole file is done
Convince Ola to pull your fork

Let’s get crackin’!

(but first - back to normal work… :)

The Hitchhiker’s Guide to an Ioke Dev Env From Source (part 2: Emacs)

2009-01-13T00:00:00+01:00

This is the second part in a series of posts for non-experts about setting up an Ioke development environment on Linux. Please see the previous post to start at the beginning:

Part 1: Git

In this post we will install GNU Emacs from source and also get a basic configuration set up using the emacs-starter-kit.

If you already have Emacs installed and have an old configuration laying around you probably want to make a backup of this before following the below instructions. This post assumes that there is no Emacs installed and that the user doesn’t have any Emacs configuration in the home directory.

Installing Emacs

Git is an efficient Distributed Version Control System. Although the primary VCS for Emacs is still CVS - and it looks like they are moving towards Bazaar - we’ll get the Emacs sources from the Emacs Git mirror. We want to be cool, right?

~$ cd work
      ~/work$ git clone --depth 1 git://git.sv.gnu.org/emacs.git
      Initialized empty Git repository in /home/melwin/work/emacs/.git/
      remote: Counting objects: 46400, done.
      remote: Compressing objects: 100% (24410/24410), done.
      remote: Total 46400 (delta 41204), reused 25501 (delta 21836)
      Receiving objects: 100% (46400/46400), 74.38 MiB | 213 KiB/s, done.
      Resolving deltas: 100% (41204/41204), done.
      Checking out files: 100% (2837/2837), done.

With the --depth 1 parameter we limit the history so that we only get the latest version of the files - in this case we’re not interested in the full history.

Note: As we’re getting the bleeding edge source code of Emacs, it could happen that the build is broken. I’ve never had this happen on me, but in case you get strange errors when building Emacs, this might be the reason. Usually such problems are fixed quickly, so try to do a git pull a bit later to update the downloaded source.

Previously, to get fun things like multi-tty support and smooth fonts, we had to get specific feature branches of Emacs. Nowadays, however, all the things we want are merged into the Emacs master branch. Before building, the only thing we need to do is to make sure the necessary development libraries are installed. In case you wonder how I came up with this list: the good ol’ method of trial and error. I simply ran the configure command and checked the error messages. This helped me identify the needed libraries. Now you can reap the benefits by just doing the following:

~/work$ sudo apt-get install libgtk2.0-dev libxpm-dev libjpeg-dev libgif-dev libtiff-dev
      ...

… and then run the classical configure, make and make install:

~/work$ cd emacs
      ~/work/emacs$ ./configure --prefix=$HOME/opt/emacs-23.0.60
      ...
      ~/work/emacs$ make
      ...
      ~/work/emacs$ make install
      ...

Phew. That burned som CPU cycles…! Once done - let’s add it to our bin directory, just as we did with Git.

~/work/emacs$ cd ~/bin
      ~/bin$ ln -s ../opt/emacs-23.0.60/bin/emacs
      ~/bin$ ln -s ../opt/emacs-23.0.60/bin/emacsclient
      ~/bin$ ls
      emacs  emacsclient  git

Time to test. Just run emacs:

~/bin% emacs

This should show you an Emacs window with a pretty(?) GNU.

If you’re completely new to Emacs, this might be a good opportunity to run the Emacs tutorial. Emacs is a self-documenting editor, which means that most things that you might want to learn about Emacs can be found inside the editor itself - including information about internal functions and libraries.

To run the tutorial, just press Ctrl+h t (that is, control and h, release control, then press t), or, in Emacs lingo, C-h t (which is the convention I’ll use from now on).

Once you’ve learned enough (you did complete the whole thing, right?), just quit using C-x c.

Note: To update the source code and rebuild, do a git pull in the emacs directory, then make distclean (in case some build files changed) and then perform the compile and installation as per the above again.

Next up is emacs-starter-kit to get more functionality in Emacs and a decent default configuration to help us get going - stay tuned!

Update: Part 3: emacs-starter-kit

The Hitchhiker’s Guide to an Ioke Dev Env From Source (part 3: emacs-starter-kit)

2009-01-13T00:00:00+01:00

This is the third part in a series of posts for non-experts about setting up an Ioke development environment on Linux. Please see the previous posts to start at the beginning:

The emacs-starter-kit is a set of base configuration for Emacs. It contains a number of useful elisp libraries, with a slight focus on dynamic languages.

To install it, perform the following steps (note that we move any existing Emacs configuration out of the way first to avoid stomping what you currently have):

~/work/emacs$ cd
      ~$ mv .emacs.d .emacs.d.old
      ~$ mv .emacsrc .emacsrc.old
      ~$ git clone git://github.com/technomancy/emacs-starter-kit.git .emacs.d

If you now start Emacs again you’ll see that the menu bar and the toolbar is gone. This is the default in emacs-starter-kit as most Emacs users don’t find them useful. For new users the menu bar can sometimes come in handy, to get it back temporarily, just press F1.

Configuring Emacs

If you want to add your own customizations to Emacs when using emacs-starter-kit, just add an Emacs LISP file called username.el, or hostname.el, in the ~/.emacs.d directory. For instance, to make the menu bar always visible:

Open Emacs, if it’s not open already.
Press C-x C-f and type in: ~/.emacs.d/username.el
Where username is the name you log in with (for instance, in my case the complete filename is melwin.el).
Type in the following in the file:
```
 (menu-bar-mode 1)
      
```
And save the file with C-x s.
Now quit (C-x c) and restart and you’ll see that the menu bar is shown.

Working with Magit

emacs-starter-kit includes, among many other things, the very nice Magit Git mode for Emacs, which gives you a nice interface for working with a Git repository.

Let’s use this mode to commit our recent changes to the configuration file to our local clone of the emacs-starter-kit repository. This helps us track changes we make and also makes a backup of the file in case we screw (sorry, mess) something up.

Note 1: To move easily between Emacs windows using the keyboard, just press Shift and the arrow key pointing in the direction you want to move. Note 2: To only show the current Emacs window - press: C-x 1

Inside Emacs, press `C-x g` to run `magit-status` and enter the directory (note that `Tab` auto-completes): `~/.emacs.d`
Put the cursor over the `username.el` in the list of `Untracked files`.
Press `s` to Stage the new file - this adds the file to the Git staging area, from which all files are committed.
Press `d` and then accept to diff against HEAD - this will show you a diff view of the changes we have staged - just the add of a single file.
Press `c` to perform the commit. This opens a new buffer into which a commit message can be added.
Write something like: `Add personal configuration file.`
Now press `C-c C-c` to commit the file.

That’s it - now the change has been committed to the local clone of the emacs-starter-kit Git repository.

To see the log of all commits, press l (lowercase L) in the magit-status buffer:

To look at a certain commit - just press Enter on it and a view of the diff will be shown. This makes it quite easy to browse through commits in a repository. At the top of the log is the most recent commit, which in this case is the file we just added.

To update emacs-starter-kit with the latest changes in the GitHub repository, just press F in the magit-status buffer or run git pull in the ~/.emacs.d directory.

Summary

Now that we have Git and Emacs set up we can finally move to Ioke. In the next post we’ll go through installing the latest Java JDK and getting and compiling the Ioke source code.

Join me then!

Update: Part 4: Java and Ant

The Hitchhiker’s Guide to an Ioke Dev Env From Source (part 1: Git)

2009-01-12T00:00:00+01:00

In this series of posts I will guide you, the humble reader, through the install procedure to get a development environment for Ioke up and running. All the cool kids want to develop in Ioke nowadays, so let’s make you can as well!

The components we will be installing are:

These instructions are written for a non-expert who might not have too much experience with compiling things, using Git or Emacs. If you’re an expert you will find nothing new or exciting here! The environment I use is a freshly installed Kubuntu Intrepid Ibex box, although most of it should be similar, if not identical, to other Ubuntu/Debian based distributions.

The components listed above will all be installed manually - not using the package system. This is to allow us to get the freshest versions of what we need without depending on the packages in the distribution to be updated (or hunted down in alternative repositories). The only thing we will install from the distribution are the compilers, tools and development libraries used to compile the software.

Why do we not use the package system? Well, that’s a valid question. Most of the above can be installed from packages in custom repositories. It’s simple and convenient. However here we will not use if for two reasons:

If packages are used we will depend on the repositories to be updated to use the latest version of some software, which can be annoying if we want to develop on the bleeding edge.
It’s good to know how to compile things yourself - so by not using prepared packages we might learn something!

To keep control over our custom built software we’ll install it in dedicated directories under $HOME/opt instead of in the normal locations like /usr/bin. This allows us to get everything installed without needing root access (except to install the compilers etc from the distribution).

First Step

Don’t Panic!

Always sound advice to start with - especially if you are hitchhiking. I’m going to try to cover each step in detail, but if you think something is unclear - just leave a comment and I’ll try to clarify. So - grab your towel and let’s get going!

Install Git

For Emacs, emacs-starter-kit and Ioke we will use Git to retrieve the sources. Therefore the first thing we need to install is Git.

In the command line instructions below, all lines prefixed by the prompt <dir>$ are commands to be typed in. The rest are output or my comments.

Open a terminal window and create a new work directory under your home using the following instructions. In this directory we will work with downloaded source files and build the software.

~$ mkdir ~/work
      ~/work$ cd ~/work
      ~/work$ wget -O - http://kernel.org/pub/software/scm/git/git-1.6.1.tar.bz2 | tar xjv
      ... many lines...

The last command downloads the git source package and expands it in one go by piping it directly from wget to tar, which is told to expand it using bz2 with the j parameter. This is a convenient way to get packages off the net and unpacked, especially when there is no need to keep the package itself around.

The result is that we now have a git-1.6.1 directory in the work directory. Let’s see how we compile it:

~/work$ cd git-1.6.1/
      ~/work/git-1.6.1$ head INSTALL
      
                      Git installation
      
      Normally you can just do "make" followed by "make install", and that
      will install the git programs in your own ~/bin/ directory.  If you want
      to do a global install, you can do
      
              $ make prefix=/usr all doc info ;# as yourself
              # make prefix=/usr install install-doc install-html install-info ;# as root
      
      ~/work/git-1.6.1$

So - two commands. Simple enough! However, before we compile - we need to make sure all the relevant packages are installed:

~/work/git-1.6.1$ sudo apt-get install libcurl4-openssl-dev zlib1g-dev libexpat-dev tk8.5 asciidoc docbook2x

Depending on your internet connection, this could take quite a while, as we need to download the texlive distribution, among other things, to build all of the Git documentation. This is not strictly necessary, but here we’ll just do it for completeness’ sake.

Let’s build git and make sure it’s installed under our $HOME/opt directory as we said in the beginning:

~/work/git-1.6.1$ make prefix=~/opt/git-1.6.1 all doc info
      ... lots of lines...

Compiling Git will take some time as well. Go get a coffee (or perhaps a Pan Galactic Gargle Blaster - sweet like nectar).

Once the compile is done - install it. Note that we don’t need to do this as root as we’re installing under the user home directory:

~/work/git-1.6.1$ make prefix=~/opt/git-1.6.1 install install-doc install-html install-info

Let’s add it to the user’s path by linking it into the private bin directory. This depends on the standard Ubuntu bash shell profile script which adds ~/bin to the PATH variable. If a different shell is used you need to perform the appropriate steps yourself.

~/work/git-1.6.1$ mkdir ~/bin
      ~/work/git-1.6.1$ cd !$
      ~/bin$ ln -s ../opt/git-1.6.1/bin/git

Now close the shell/terminal, open a new one - and try the git command:

~$ git --version
      git version 1.6.1

If you get the above output - great! You’re don! Sit back and relax for a bit before moving on to the next section.

However, if you don’t get the version output, but instead see the following message, review the previous instructions and make sure it works ok before continuing.

#INCORRECT OUTPUT - SOMETHING WAS MISSED!
      #GO BACK AND REVIEW
      ~$ git                                                                                                                                                                                        
      The program 'git' is currently not installed.  You can install it by typing:                                                                                                                  
      sudo apt-get install git-core                                                                                                                                                                 
      -bash: git: command not found

If you still can’t get it to work - drop me a comment!

Git in 30 Seconds

There are loads of good Git tutorials and information. You can find several on the official Git page and at GitHub (go sign up if you haven’t already, and fork me!).

Here is a quick run through of a few common Git commands you could try out with your freshly brewed cup of Git:

~$ cd
      ~$ mkdir gittest
      ~$ cd gittest/
      ~/gittest$ git init
      Initialized empty Git repository in /home/melwin/gittest/.git/
      ~/gittest$ echo Test file! > test.txt
      ~/gittest$ git add test.txt
      ~/gittest$ git commit -m "Initial import."
      [master (root-commit)]: created 79c091d: "Initial import."
       1 files changed, 1 insertions(+), 0 deletions(-)
       create mode 100644 test.txt
      ~/gittest$ echo Add line. >> test.txt
      ~/gittest$ git diff test.txt
      diff --git a/test.txt b/test.txt
      index 1cbaf90..3746f9e 100644
      --- a/test.txt
      +++ b/test.txt
      @@ -1 +1,2 @@
       Test file!
      +Add line.
      ~/gittest$ git commit -a -m "Add new line."
      [master]: created b20f9a4: "Add new line."
       1 files changed, 1 insertions(+), 0 deletions(-)

If you can follow the above - great! First part finished. Next up is to install GNU Emacs from source and the emacs-starter-kit, which provides a decent default a set of configuration for Emacs.

Stay tuned!

Update: Part 2: Emacs

Ioke Syntax Highlighter for GeSHi

2009-01-10T00:00:00+01:00

Speaking of Ioke - just added a syntax highlighter definition for GeSHi in my Ioke fork repository. The commit can be found here:

http://github.com/melwin/ioke/commit/54cfd7e54de0be910385c6ec805693fd3ed4e294

The keyword definitions I borrowed (read stole) from Sam Aaron’s TextMate bundle (also in the Ioke repository). As Sam just said in the #ioke on freenode: “what goes around comes around”. :)

Highlighting test:

Simple JSON Parser in Ioke

2009-01-10T00:00:00+01:00

After having enjoyed most of the days off over Christmas my fingers started itching - time to do some programming. Luckily around the same time I stumbled across Ola Bini’s posts on the Ioke language. As the Ioke guide begins:

Ioke is a general purpose language. It is a strongly typed, extremely dynamic, prototype object oriented language. It is homoiconic and it’s closest ancestors is Io, Smalltalk, Ruby and Lisp - but it’s quite a distance from all of them.

So what does this all mean? It means it’s fun! The language syntax is very regular (although not quite as regular as lisp) - code is data, data is code and everything is a message. It’s also one of relatively few languages that provide macros similar to those in lisp. In Ioke this means that a message stream can be operated on (modified, transformed, etc) before it’s evaluated, using normal Ioke methods. Anyone who groks lisp should be familiar with the power this gives.

In Ioke, a chain of messages is separated by space, with the first message being sent to the current receiver and subsequent messages are sent to the result of the previous one. For instance, in the code

the text literal “ioke rocks” results in an internal message which creates a Text object. To this object the upper messages is sent, which results in an upper case copy of the Text object is created. To this object the println message is sent, which prints the upper case text to the console.

As the Ioke reader handles space separated tokens, it’s quite easy to create macros to process non Ioke code. As a learning exercise I decided to try to implement a JSON parser. Nothing too fancy - and doesn’t even have to be secure - just enough to parse simple JSON into Ioke objects.

JSON is usually parsed in one of two ways - into custom objects/classes or into collections like dictionaries and arrays. For this exercise I decided to create Ioke dictionaries and arrays and I want the result to work like this:

Now, looking at JSON, it’s suspiciously similar to Ioke - [] is used for arrays, {} is used for “dictionaries”, comma separates entries, ” quotes strings, etc. The main problem is the colon “:” used for separating keys and values in a dictionary. As Ioke is a dynamic language we can redefine how messages are handled. Instead of using colon to define symbols, we can redefine it to instead create pairs suitable for a dict:

”:” is originally defined on DefaultBehavior Literals, but in the JSON the “:” message is always sent to Text, so it’s enough to redefine it there.

This single redefinition allows us to parse JSON directly with the Ioke reader - very nice.

But… Redefining “:” for all Text objects is not really good (especially not when Ola adds concurrency constructs!). Another alternative would be to redefine how the text objects are created and create a new “:” cell just for our JSON text objects - but I never got this to work, as the internal:createText message works with a raw Java string, which I couldn’t work with in a macro/method…

Update: With Ola’s recent commit, the previously described workaround can now be simplified to the following code.

Now the above redefinition of the “:” message can be used, but is visible only in the scope of a let:

As can be seen, the Ioke macro functionality gives us nice control over how messages are handled, and the environment can be changed before the arguments are evaluated. I’m looking forward to the day when we have full Java interop and can use these powerful constructs to control Java code!

Source code to the above example is committed to my Ioke fork at: git://github.com/melwin/ioke.git

PS: The thing that confuses me the most: how the heck are you supposed to pronounce Ioke?

Update: Sam Aaron shared the fact that Ola pronounces it eye-oh-key in the following interview: http://www.akitaonrails.com/2008/11/22/rails-podcast-brasil-qcon-special-ola-bini-jruby-ioke

Caching Filter Queries with Coherence

2008-09-21T00:00:00+02:00

A pretty nice thing that I recently tried with a customer was storing a query result in a query cache.

For instance, consider the following method:

public Collection getByFilter1(String cacheName, Filter f) {
          NamedCache c = CacheFactory.getCache(cacheName);
          return c.entrySet(f);
      }

A query is executed across all nodes containing cache data in a cluster. The filter acts on the values in the cache, not the keys. And as all values are usually not available locally on every node the query needs to execute on the separate nodes.

The query above is correct, but not very efficient. In the above case we query and retrieve the data as well from the separate nodes. A more efficient way would be to only get the keys for the matching values from the nodes and then retrieve the values from the local near cache:

public Collection getByFilter2(String cacheName, Filter f) {
          NamedCache c = CacheFactory.getCache(cacheName);
          Set keys = c.keySet(f);
          return c.getAll(keys).values();
      }

However, in this case we still perform the query every time.

Now - the clever part. Not so much on my side, but the Coherence engineers have thought things through and made the Filter implementations have good hashCode and equals implementations. This together with the fact that they are serializable makes them possible to use as keys in a cache! Sweet! Without changing our method’s interface we can add a query cache so that each query only is performed once.

public Collection getByFilter3(String cacheName, Filter f) {
          NamedCache c = CacheFactory.getCache(cacheName);
      
          NamedCache queryC = CacheFactory.getCache(cacheName + ".querycache");
          Set keys = (Set)queryC.get(f);
      
          if(keys == null) {
              keys = c.keySet(f);
              queryC.put(f, keys);
          }
      
          return c.getAll(keys).values();
      }

Note that we only save the keys in the query cache. This to avoid having several caches with the same data. When near caching is used, getting the data for the keys can still be a local only operation. Compared to getting the data from the separate nodes it’s several orders of magnitude faster - depending on the usage patterns.

Of course if queries are different every time, the query cache will not help much. But in most high load applications the same data tends to be needed several times.

Additionally, the properties queried on should in most cases be indexed. This is important to avoid too much overhead when searching for an entry in a cache.

One thing to think about when adding query caches is: how is data updated?

As part of updating the value caches the query caches should preferably be emptied of the outdated cached queries. This could be done programmatically “manually” when the data is updated, or by hooking in code to clean the entries from the query cache using the map listener mechanism. The relevant code could do a query using the ContainsFilter.

To summarize: a little code can go a long way to improve performance without affecting the interface used by an application. Good when query heavy applications are adapted to use a distributed cache like Coherence.

Scala Syntax Highlighting for WP

2008-06-30T00:00:00+02:00

Writing the previous post I realized that the Wordpress plugin for syntax highligting I was using (Highlight Source Pro) didn’t support Scala.

Instead I tried the WP-Syntax plugin, but this didn’t support Symbol literals:

val sym = 'foobar
      println("Symbol is: " + sym)
      val other = 'barfoo

This seems to be because of single quote being interpreted as a quotation character. A fix seems to be removing the single quote from the Geshi language definition file scala.php in the wp-syntax/geshi/geshi directory. While there one can also specify a regex for symbols and give them their own color:

val sym = 'foobar
      println("Symbol is: " + sym)
      val other = 'barfoo

Also check out the scala.php in the LAMP repository - it seems to add some additional keywords and other colors. I used this and made the changes described above.

Full new scala.php for Geshi after the break.


      /*************************************************************************************
       * scala.php
       * --------
       * Author: Geoffrey Washburn (washburn@acm.ogr)
       * Copyright: (c) 2004 Nigel McNie (http://qbnz.com/highlighter/)
       * Release Version: ???
       * Date Started: 2008/01/03
       *
       * Scala language file for GeSHi.
       *
       * CHANGES
       * -------
       * 2007/01/03
       *   -  Created by copying the Java highlighter
       *
       * TODO
       * -------------------------
       * * Finish
       *
       *************************************************************************************
       *
       *     This file is part of GeSHi.
       *
       *   GeSHi is free software; you can redistribute it and/or modify
       *   it under the terms of the GNU General Public License as published by
       *   the Free Software Foundation; either version 2 of the License, or
       *   (at your option) any later version.
       *
       *   GeSHi is distributed in the hope that it will be useful,
       *   but WITHOUT ANY WARRANTY; without even the implied warranty of
       *   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
       *   GNU General Public License for more details.
       *
       *   You should have received a copy of the GNU General Public License
       *   along with GeSHi; if not, write to the Free Software
       *   Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
       *
       ************************************************************************************/
      
      $language_data = array (
              'LANG_NAME' => 'Scala',
              'COMMENT_SINGLE' => array(1 => '//'),   /* import statements are not comments! */
              'COMMENT_MULTI' => array('/*' => '*/'),
              'CASE_KEYWORDS' => GESHI_CAPS_NO_CHANGE,
              'QUOTEMARKS' => array('"', '"""'),
              'ESCAPE_CHAR' => '\\',
              'KEYWORDS' => array(
                      1 => array(
                              /* Scala keywords, part 1: control flow */
                              'case', 'default', 'do', 'else', 'for',
                              'if', 'match', 'while'
                              ),
                      2 => array(
                              /* Scala keywords, part 2 */
                              'return', 'throw',
                              'try', 'catch', 'finally',
                              'abstract', 'class', 'def', 'extends',
                              'final', 'forSome', 'implicit', 'import',
                              'lazy', 'new', 'object', 'override', 'package',
                              'private', 'protected',
                              'requires', 'sealed', 'super', 'this', 'trait', 'type',
                              'val', 'var', 'with', 'yield'
                              ),
                      3 => array(
                              /* Scala keywords, part 3: standard value types */
                              'unit', 'Unit', 'boolean', 'Boolean', 'int', 'Int', 'Any', 'AnyVal', 'Nothing',
                              ),
                      4 => array(
                              /* other reserved words in Scala: literals */
                              /* should be styled to look similar to numbers and Strings */
                              'false', 'null', 'true'
                              ),
                      5 => array(
                              /* Scala reference types */
                              'AnyRef', 'Null', 'List', 'String', 'Integer', 'Option', 'Array'
                              )
      
                      ),
              'SYMBOLS' => array(
                      ':', '*', '&', '%', '!', ';', '<', '>', '?', '_', '=', '=>',
                      '<-', '<:', '<%', '>:', '#', '@'
                      ),
              'CASE_SENSITIVE' => array(
                      GESHI_COMMENTS => true,
                      /* all Scala keywords are case sensitive */
                      1 => true, 2 => true, 3 => true, 4 => true, 5 => true ),
              'STYLES' => array(
                      'KEYWORDS' => array(
                              1 => 'color: #b1b100;',
                              2 => 'color: #000000; font-weight: bold;',
                              3 => 'color: #993333;',
                              4 => 'color: #b13366;',
                              5 => 'color: #aaaadd;'
                              ),
                      'SYMBOLS' => array(
                              0 => 'color: #FFAA00;'
                              ),
                      'COMMENTS' => array(
                              1 => 'color: #808080; font-style: italic;',
                              'MULTI' => 'color: #808080; font-style: italic;'
                              ),
                      'ESCAPE_CHAR' => array(
                              0 => 'color: #000099; font-weight: bold;'
                              ),
                      'BRACKETS' => array(
                              0 => 'color: #66cc66;'
                              ),
                      'STRINGS' => array(
                              0 => 'color: #ff0000;'
                              ),
                      'NUMBERS' => array(
                              0 => 'color: #cc66cc;'
                              ),
                      'METHODS' => array(
                              1 => 'color: #006600;',
                              2 => 'color: #006600;'
                              ),
                      'SCRIPT' => array(
                              ),
                      'REGEXPS' => array(
                              0 => 'color: #008000;'
                              )
                      ),
              'URLS' => array(
                      1 => '',
                      2 => '',
                      3 => '',
                      4 => ''
                      ),
              'OOLANG' => true,
              'OBJECT_SPLITTERS' => array(
                      1 => '.'
                      ),
              'REGEXPS' => array(
                     0 => "'[a-zA-Z_][a-zA-Z0-9_]*"
                      ),
              'STRICT_MODE_APPLIES' => GESHI_NEVER,
              'SCRIPT_DELIMITERS' => array(
                      ),
              'HIGHLIGHT_STRICT_BLOCK' => array(
                      )
      );
      
      ?>

Clustering Scala Actors with Oracle Coherence for Fun and Profit

2008-06-30T00:00:00+02:00

[Disclaimer: I work for Oracle.]

Although I haven’t used it too much yet, Scala is definitely one of the languages I find most interesting right now. Many customers I work with are heavily Java focused, and getting a more flexible and powerful language with superb Java interoperability to use on the JVM feels very liberating. Now I just need to convince the customers that Scala is the future… :] But if Gosling likes it it must be good, right?

A few months ago Jonas Bonér wrote about how Scala actors can be clustered with Terracotta. I really enjoyed the article and I think the idea of distributed, redundant actors is very appealing. The actor paradigm is a nice way of developing concurrent applications (see the intro in Jonas’ blog entry for more info) and if we can liberate the actors from the confines of a single JVM and easily distribute them over multiple hosts - all the better.

Clustering with Coherence

I’m not going to compare Coherence and Terracotta here. In short, Coherence provides, among other things, a distributed caching and code execution mechanism without single points of failure. Coherence can be downloaded for evaluation purposes from the Oracle web site.

The idea I wanted to try was to have Scala actors that store their state in the distributed Coherence cache and run as normal Scala actors on a single node in the cluster at a time. The node the actor runs on should be allocated by Coherence and if the node fails, the actor should be automatically started on another node with maintained state and without any lost messages.

Also, I wanted this to work as similarly to normal Scala actors as possible with compatibility between the two.

The Result

Before investigating the proof of concept solution, let’s look at the result and what it gives us.

Here’s a simple test application with a normal Scala actor. It uses the recommended way of creating actors with actor { ... }:

package coherencetest1
      
      import java.util.Date
      
      import scala.actors.Actor._
      
      object ActorTest {
        def main(args : Array[String]) : Unit = {
      
          actor {
            var pings : Int = 0
            
            println("Actor started.")
            
            self ! ('ping, 1)
            
            loop {
              react {
              case ('ping, i : Int) =>
                pings = pings + 1
                println(new Date + " - Got ping: " + i + " Total pings: " + pings)
                
                Thread.sleep(1000)
      
                self ! ('ping, i+1)
              }
            }
          }
      
        }
      }

When this code is run, a simple actor that sends a message to itself is created and started. It sleeps for 1 second to pace the execution and to simulate a task that takes time to perform (of course, normally you shouldn’t sleep in real actors as you tie up the thread).

When the code is run, the following is displayed:

Actor started.
      Sun Jun 29 15:57:16 CEST 2008 - Got ping: 1 Total pings: 1
      Sun Jun 29 15:57:17 CEST 2008 - Got ping: 2 Total pings: 2
      Sun Jun 29 15:57:18 CEST 2008 - Got ping: 3 Total pings: 3
      Sun Jun 29 15:57:19 CEST 2008 - Got ping: 4 Total pings: 4
      Sun Jun 29 15:57:20 CEST 2008 - Got ping: 5 Total pings: 5
      Sun Jun 29 15:57:21 CEST 2008 - Got ping: 6 Total pings: 6
      ...

Nothing too fancy, but a decent test case for our actor distribution. An important aspect of this actor is that it defines a local variable pings and prints a message in the initialization part, before the loop and react. The value of the local var must be maintained and the initialization code must only be run once, and not when an actor is started on a new node after a failure.

Let’s make it a distributed Actor:

package coherencetest1
      
      import java.util.Date
      
      import scala.actors.coherence.CoActor._
      
      @serializable
      object DactorTest {
        def main(args : Array[String]) : Unit = {
      
          dactor {
            var pings : Int = 0
            
            println("Actor started.")
            
            self ! ('ping, 1)
            
            loop {
              react {
              case ('ping, i : Int) =>
                pings = pings + 1
                println(new Date + " - Got ping: " + i + " Total pings: " + pings)
                
                Thread.sleep(1000)
      
                self ! ('ping, i+1)
              }
            }
          }
      
        }
      }

What have we done here? Three things:

Import `scala.actors.coherence.CoActor._` instead of `scala.actors.Actor._`
Made the application object serializable
Create the actor using `dactor { … }` instead of `actor { … }`

The first point is simple - we need access to the new functionality, so we import the new CoActor object instead of the standard Actor object.

For number two - this is slightly nasty. If I interpret things correctly; as the code block created as a parameter to react needs to be serializable (so that the actor can be distributed over the network), all enclosing types needs to be serializable. I struggled with this for a while and the only option seems to be creating a proper named serializable type… But since I want to be able to create an actor in-line, we need to do it this way.

For the last point - dactor { ... } is simply the function used to create a distributed actor instead of a normal actor.

Let’s run it:

2008-06-29 16:11:18.779 Oracle Coherence 3.3.1/389  (thread=main, member=n/a): Loaded operational configuration from resource "jar:file:/opt/coherence-3.3.1/lib/coherence.jar!/tangosol-coherence.xml"
      2008-06-29 16:11:18.785 Oracle Coherence 3.3.1/389  (thread=main, member=n/a): Loaded operational overrides from resource "jar:file:/opt/coherence-3.3.1/lib/coherence.jar!/tangosol-coherence-override-dev.xml"
      2008-06-29 16:11:18.786 Oracle Coherence 3.3.1/389  (thread=main, member=n/a): Optional configuration override "/tangosol-coherence-override.xml" is not specified
      
      Oracle Coherence Version 3.3.1/389
       Grid Edition: Development mode
      Copyright (c) 2000-2007 Oracle. All rights reserved.
      
      2008-06-29 16:11:19.042 Oracle Coherence GE 3.3.1/389  (thread=main, member=n/a): Loaded cache configuration from resource "file:/crypt/dev/scala/CoherenceTest1/config/scalacoherence.xml"
      2008-06-29 16:11:19.331 Oracle Coherence GE 3.3.1/389  (thread=main, member=n/a): UnicastUdpSocket failed to set receive buffer size to 1428 packets (2096304 bytes); actual size is 714 packets (1048576 bytes). Consult your OS documentation regarding increasing the maximum socket buffer size. Proceeding with the actual value may cause sub-optimal performance.
      2008-06-29 16:11:19.459 Oracle Coherence GE 3.3.1/389  (thread=Cluster, member=n/a): Service Cluster joined the cluster with senior service member n/a
      2008-06-29 16:11:22.662 Oracle Coherence GE 3.3.1/389  (thread=Cluster, member=n/a): Created a new cluster with Member(Id=1, Timestamp=2008-06-29 16:11:19.343, Address=192.168.54.1:8088, MachineId=24065, Location=process:31397@dellicious, Edition=Grid Edition, Mode=Development, CpuCount=2, SocketCount=1) UID=0xC0A836010000011AD4A9DCAF5E011F98
      2008-06-29 16:11:22.834 Oracle Coherence GE 3.3.1/389  (thread=DistributedCache, member=1): Service DistributedCache joined the cluster with senior service member 1
      Actor started.
      Sun Jun 29 16:11:23 CEST 2008 - Got ping: 1 Total pings: 1
      Sun Jun 29 16:11:24 CEST 2008 - Got ping: 2 Total pings: 2
      Sun Jun 29 16:11:25 CEST 2008 - Got ping: 3 Total pings: 3
      Sun Jun 29 16:11:26 CEST 2008 - Got ping: 4 Total pings: 4
      Sun Jun 29 16:11:27 CEST 2008 - Got ping: 5 Total pings: 5
      Sun Jun 29 16:11:28 CEST 2008 - Got ping: 6 Total pings: 6
      ...

After the Coherence initialization (which happens automatically and which I’ve disabled in the outputs below) the actor starts up as expected. However, if we start this on two nodes - there will be two actors created, and no way for a new JVM to get hold of a reference to a specific existing actor. To handle this, let’s specify a name for the actor that we create using the dactor(name : Symbol) { ... } function:

...
      object DactorTest {
        def main(args : Array[String]) : Unit = {
      
          dactor('pingActor) {
            var pings : Int = 0
      ...

This simply means: Give me a reference to pingActor, but if it doesn’t exist - create it with the following body. This mechanism makes it easy to have a single instance of an actor even if the same application is running on multiple nodes, without having to explicitly check if an actor has already been created or not.

Now we can run the program on two different nodes. After the actor has started and is running on one node, I’ll kill that node:

Node 1

Node 2

Actor started.
      Sun Jun 29 16:30:41 CEST 2008 - Got ping: 1 Total pings: 1
      Sun Jun 29 16:30:42 CEST 2008 - Got ping: 2 Total pings: 2
      Sun Jun 29 16:30:43 CEST 2008 - Got ping: 3 Total pings: 3
      Sun Jun 29 16:30:44 CEST 2008 - Got ping: 4 Total pings: 4
      Sun Jun 29 16:30:45 CEST 2008 - Got ping: 5 Total pings: 5
      Sun Jun 29 16:30:46 CEST 2008 - Got ping: 6 Total pings: 6
      Sun Jun 29 16:31:02 CEST 2008 - Got ping: 19 Total pings: 19
      Sun Jun 29 16:31:03 CEST 2008 - Got ping: 20 Total pings: 20
      Sun Jun 29 16:31:04 CEST 2008 - Got ping: 21 Total pings: 21
      Sun Jun 29 16:31:05 CEST 2008 - Got ping: 22 Total pings: 22
      Sun Jun 29 16:31:06 CEST 2008 - Got ping: 23 Total pings: 23
      Sun Jun 29 16:31:07 CEST 2008 - Got ping: 24 Total pings: 24
      Sun Jun 29 16:31:08 CEST 2008 - Got ping: 25 Total pings: 25
      Sun Jun 29 16:31:09 CEST 2008 - Got ping: 26 Total pings: 26
      Sun Jun 29 16:31:10 CEST 2008 - Got ping: 27 Total pings: 27
      Sun Jun 29 16:31:11 CEST 2008 - Got ping: 28 Total pings: 28
      ...

Sun Jun 29 16:30:47 CEST 2008 - Got ping: 6 Total pings: 6
      Sun Jun 29 16:30:48 CEST 2008 - Got ping: 7 Total pings: 7
      Sun Jun 29 16:30:49 CEST 2008 - Got ping: 8 Total pings: 8
      Sun Jun 29 16:30:50 CEST 2008 - Got ping: 9 Total pings: 9
      Sun Jun 29 16:30:51 CEST 2008 - Got ping: 10 Total pings: 10
      Sun Jun 29 16:30:53 CEST 2008 - Got ping: 11 Total pings: 11
      Sun Jun 29 16:30:54 CEST 2008 - Got ping: 12 Total pings: 12
      Sun Jun 29 16:30:55 CEST 2008 - Got ping: 13 Total pings: 13
      Sun Jun 29 16:30:56 CEST 2008 - Got ping: 14 Total pings: 14
      Sun Jun 29 16:30:57 CEST 2008 - Got ping: 15 Total pings: 15
      Sun Jun 29 16:30:58 CEST 2008 - Got ping: 16 Total pings: 16
      Sun Jun 29 16:30:59 CEST 2008 - Got ping: 17 Total pings: 17
      Sun Jun 29 16:31:00 CEST 2008 - Got ping: 18 Total pings: 18
      Sun Jun 29 16:31:01 CEST 2008 - Got ping: 19 Total pings: 19^C

First Node 1 started up and ran the actor until Node 2 started. At this point the actor was distributed to Node 2 (determined by the automatic cache partitioning done by Coherence) and started there. As can be seen, the local state (the total pings) was persisted and transferred over. When Node 2 was killed the actor was migrated back and started on Node 1. Note that the state of the actor is persisted for each message, so a sudden shutdown of a JVM is not a problem.

One might wonder why the message for ping number 6 and 19 can be seen in both outputs - this happens as the actor was migrated while the actor thread was sleeping - before the react body was complete. This causes the new node to rerun the message (since the processing of the message didn’t complete on the old node) and the support code in the old node makes sure all messages sent by the old actor are discarded as it’s been terminated. It’s a bit tricky coding actors to be fully idempotent as not everything is handled in a transaction, but limiting side effects to sending messages at the end of the processing makes it fairly reliable.

Here’s a slightly more complex example:

package coherencetest1
      
      import java.util.Date
      
      import scala.actors.coherence.CoActor._
      
      @serializable
      object DactorTest2 {
        def main(args : Array[String]) : Unit = {
          init
          
          readLine
          
          val numActors = 80
      
          val actors = for(id <- 1 to numActors)
            yield dactor {
              loop {
                react {
                case 'ping =>
                  println(new Date + " - Actor " + id + " got ping.")
                  reply(('pong, id))
                }
              }
            }
          
          actors.map(_ ! 'ping).force
          
          var pongs = 0
          
          while(pongs < numActors) {
            receive {
            case ('pong, x : Int) =>
              pongs = pongs + 1
            }
          }
          
          println("Got " + pongs + " pongs.")
          
          readLine
        }
      }

In this example 80 distributed actors are created and sent a ping message. After that the main thread receives all the pong replies. The output of this, when run on 4 nodes look like so:

Node 1	Node 2
Sun Jun 29 09:19:34 CEST 2008 - Actor 1 got ping. Sun Jun 29 15:19:34 CEST 2008 - Actor 12 got ping. Sun Jun 29 15:19:34 CEST 2008 - Actor 15 got ping. Sun Jun 29 15:19:34 CEST 2008 - Actor 18 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 22 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 25 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 27 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 29 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 41 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 42 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 47 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 50 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 54 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 56 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 61 got ping. Sun Jun 29 15:19:36 CEST 2008 - Actor 75 got ping. Sun Jun 29 15:19:36 CEST 2008 - Actor 79 got ping. Sun Jun 29 15:19:36 CEST 2008 - Actor 80 got ping. Got 80 pongs.	Sun Jun 29 15:19:34 CEST 2008 - Actor 9 got ping. Sun Jun 29 15:19:34 CEST 2008 - Actor 13 got ping. Sun Jun 29 15:19:34 CEST 2008 - Actor 19 got ping. Sun Jun 29 15:19:34 CEST 2008 - Actor 20 got ping. Sun Jun 29 15:19:34 CEST 2008 - Actor 21 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 23 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 24 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 28 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 33 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 36 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 46 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 52 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 57 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 58 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 62 got ping. Sun Jun 29 15:19:36 CEST 2008 - Actor 67 got ping. Sun Jun 29 15:19:36 CEST 2008 - Actor 68 got ping. Sun Jun 29 15:19:36 CEST 2008 - Actor 71 got ping. Sun Jun 29 15:19:36 CEST 2008 - Actor 77 got ping.
Node 3	Node 4
Sun Jun 29 15:19:34 CEST 2008 - Actor 2 got ping. Sun Jun 29 15:19:34 CEST 2008 - Actor 4 got ping. Sun Jun 29 15:19:34 CEST 2008 - Actor 5 got ping. Sun Jun 29 15:19:34 CEST 2008 - Actor 6 got ping. Sun Jun 29 15:19:34 CEST 2008 - Actor 7 got ping. Sun Jun 29 15:19:34 CEST 2008 - Actor 8 got ping. Sun Jun 29 15:19:34 CEST 2008 - Actor 10 got ping. Sun Jun 29 15:19:34 CEST 2008 - Actor 11 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 30 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 31 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 32 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 34 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 35 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 37 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 39 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 40 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 43 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 44 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 45 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 53 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 59 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 60 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 63 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 64 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 66 got ping. Sun Jun 29 15:19:36 CEST 2008 - Actor 69 got ping. Sun Jun 29 15:19:36 CEST 2008 - Actor 70 got ping. Sun Jun 29 15:19:36 CEST 2008 - Actor 72 got ping. Sun Jun 29 15:19:36 CEST 2008 - Actor 74 got ping.	Sun Jun 29 15:19:34 CEST 2008 - Actor 3 got ping. Sun Jun 29 15:19:34 CEST 2008 - Actor 14 got ping. Sun Jun 29 15:19:34 CEST 2008 - Actor 16 got ping. Sun Jun 29 15:19:34 CEST 2008 - Actor 17 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 26 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 38 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 48 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 49 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 51 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 55 got ping. Sun Jun 29 15:19:35 CEST 2008 - Actor 65 got ping. Sun Jun 29 15:19:36 CEST 2008 - Actor 73 got ping. Sun Jun 29 15:19:36 CEST 2008 - Actor 76 got ping. Sun Jun 29 15:19:36 CEST 2008 - Actor 78 got ping.

The program was started on all 4 nodes, but only the first node passes the first readLine. The other nodes just init the distributed actor framework and wait. As can be seen, the actors were distributed over the running nodes as expected - however, with small numbers like this the distribution can be a bit uneven (compare Node 3 and Node 4).

The Solution

The solution to allow the distribution of serializable actors is based on the Coherence backing map listeners. These can be used to get notifications of which specific node is the master for a certain object. As there only is one master for an object at any point, and a new master is allocated automatically if a master fails, we can use this to determine where the actor should run.

The object returned from dactor { ... } is a Proxy object - very similar to the Proxy used in the standard Scala remote actors. In fact, this whole thing is built in a way similar to the standard Scala remote actors, with Proxy objects acting on behalf of the sender on the receiver side and receiver on the sender side.

Additionally, the Coherence grid invocation mechanism allows us to deliver messages to running actors directly to the node where it is running.

When a new dactor is created, the following happens:

`dactor { … }` creates a new anonymous `CoActor` class with the dactor body set as the `init` method.
The CoActor.distribute method is called which in turn saves the anonymous class instance to the Coherence cache. The object gets serialized when this happens. The key to the object in the cache is either the passed in symbol name, or a name created from a `java.util.UUID`
The `dactor` function returns a Proxy object with the name of the newly created distributed actor.
Meanwhile, in the node which Coherence designates the master of the object, the backing map listener gets an insert event and starts the actor.
The default `act()` method is called, which in a CoActor calls the `init` method first.
The `init` method contains the `dactor { … }` body, which gets executed.
The body executes as normally up until the `loop` block. `loop` in the CoActor first saves the body of the loop into a variable in CoActor so that it can continue executing just this body after it’s started in a new node, without running the initialization code again. After this the loop body is run as normal.
When `react` is hit, the thread breaks and the actor waits for incoming message using the normal actor scheduling mechanism.

When a message is sent to a distributed actor using a Proxy object the following happens:

The message is sent using the Coherence invocation mechanism which transports the message to the master node where the cached actor runs.
In the master node, a Proxy actor representing the sender (which does not need to be a distributed actor) is created - this is because the actor framework always needs a sending actor which the receiver can reply to.
The sender Proxy is asked to send the actual message to the real distributed actor receiver.
The normal Scala actor framework handles the passing of the message and scheduling of the actors.
An overriden `send` in CoActor locks the actor and stores the message and a Save message in the mailbox.
The Save message gets the actor to first persist itself to the Coherence cache with the real message still in the mailbox. This is to ensure the message isn’t lost in case of a node failure.
After this the real message is processed by the actor and an overridden `react` saves the actor again after the message has been processed. This to update the cache with the new state of the actor.

If the distributed actor does a reply or sends a message to the sender, the Proxy which represents the non-distributed actor gets called as follows:

As the recevier (the original sender) isn’t a distributed actor handled by Coherence we cannot use the invocation mechanism. Instead the message is just put into a message cache.
A `MapListener` on every node checks to see if the newly inserted message in the message cache is intended for an actor running on that specific node.
If so, the message is deleted from the cache and delivered to the local actor through a Proxy representing the sender - just as in the previous case.

The Limitations

The distributed actors are a bit limited in what they can do, as they always needs to be serializable and I didn’t want to change any of the standard Scala code. For instance - when a synchronous invocation is made the return channel is stored in the Scala actor. The return channel implementation used in Scala isn’t serializable, so I decided to not implement this feature for now.

Basically, only message sending (!), reply, sender, loop and react are allowed in the distributed actors. However, they can interoperate with normal actors as can be seen in this example:

    val distActor = dactor('distActor) {
            loop {
              react {
              case ('ping) =>
                reply('pong)
              }
            }
          }
          
          actor {
            distActor ! 'ping
            loop {
              react {
              case x =>
                println("Actor got message: " + x)
              }
            }
          }

The Proxy objects can be used by actors as they serialize correctly.

The Conclusion

Distributing (or GridEnabling(tm) or whatever the word du jour is) actors to easily use the processing power and resilience multiple computers give but at the same time hiding the complexity from the developer is a nice way to fairly easily scale up an actors based application. To add more processing power or redundancy - just fire up new nodes.

The proof of concept I made here just scratches the surface, but it’s interesting to see that it can be done with Coherence while maintaining the syntax and behavior expected by Scala developers.

The Source

The highly experimental and hackish source code for the proof of concept is available in the git repository at:

      http://martin.elwin.com/git/scala-coherence.git

Dependencies are Scala (2.7.1 recommended) and Oracle Coherence. There are some scripts for Linux which are trivial to adapt to other operating environments.