Code by Martin

The Hitchhiker’s Guide to an Ioke Dev Env From Source (part 5: Ioke and the REPL)

Martin — Fri, 23 Jan 2009 06:45:45 +0000

This is the fifth part in a series of posts for non-experts about setting up an Ioke development environment on Linux. Please see the previous posts to start at the beginning:

This time we will finally get the Ioke source code, build it, and test the Ioke REPL!

Ioke

Ioke uses Git as the source code versioning tool, with a public repository set up on GitHub (soon I’ll be linking like Jeff Atwood).

So – let’s get Ioke using Git by cloning Ola’s repository:

~/bin$ cd ~/work
~/work$ git clone git://github.com/olabini/ioke
Initialized empty Git repository in /home/melwin/work/ioke/.git/
remote: Counting objects: 9221, done.
remote: Compressing objects: 100% (2941/2941), done.
remote: Total 9221 (delta 5782), reused 8656 (delta 5351)
Receiving objects: 100% (9221/9221), 45.95 MiB | 399 KiB/s, done.
Resolving deltas: 100% (5782/5782), done.

Great – let’s quickly move on to the build step:

~/work$ cd ioke
~/work/ioke$ ant
...
     [java] 2606 examples, 0 failures
jar:
      [jar] Building jar: /home/melwin/work/ioke/lib/ioke.jar
 
BUILD SUCCESSFUL
Total time: 22 seconds

Running just ant will execute the default target in the build.xml file. This target will make sure Ioke is compiled and the test suite is run.

At the end is a summary of the test results – if these are not all successful you might have a problem with the environment or you might have checked out a broken build from the repository.

The Ioke REPL

Now all is installed and Ioke is compiled, so let’s fire up the REPL. This is done by just running the main Ioke binary:

~/work/ioke/$ bin/ioke
iik>

When doing this we get the iik> REPL prompt. From here we can execute most Ioke code. Let’s try to print a string:

iik> "Hello World!" println
Hello World!
+> nil
 
iik>

Here we create a Text (Ioke’s string type) literal and send it the message println, which will cause the text to be printed to the console. The return type of println is nil (similar to null), which is printed by the REPL itself. After that we get the prompt back and can execute another line.

Let’s try a little more complex example – the factorial:

iik> fact = method(n, if(n > 1, n * fact(n pred), 1))
+> fact:method(n, if(n >(1), n *(fact(n pred)), 1))

Note how the REPL prints the method definition in a canonical form – it makes it obvious how Ioke parses the code, what are messages and what are arguments.

Let’s run our new factorial function:

iik> fact(3)
+> 6
 
iik> fact(10)
+> 3628800
 
iik> fact(50)
+> 30414093201713378043612608166064768844377641568960512000000000000

Ioke handles arbitrarily big numbers, so we don’t need to think about what fits in a certain number of bits.

The Iik REPL also comes with a simple debugger, which helps with handling conditions. For instance, let’s try to run the fact method, passing a message which doesn’t mean anything (yet):

iik> fact(f)
*** - couldn't find cell 'f' on 'Ground_0xC360E7' (Condition Error NoSuchCell)
 f                                                [<init>:1:5]
The following restarts are available:
 0: storeValue           (Store value for: f)
 1: useValue             (Use value for: f)
 2: abort                (restart: abort)
 3: quit                 (restart: quit)
 
 dbg:1> 1
  dbg:1:newValue> 5
  +> 5
 
+> 120

The above output shows that when we try to reference something that doesn’t exist we get a chance to provide that value. Here I decided to provide a value using the useValue restart and entered the value 5. This let the method continue executing and the result was printed.

To exit the REPL, just type: exit

Next up are the Emacs modes for Ioke – stay tuned!

The Hitchhiker’s Guide to an Ioke Dev Env From Source (part 4: Java+Ant)

Martin — Thu, 22 Jan 2009 22:08:43 +0000

This is the fourth part in a series of posts for non-experts about setting up an Ioke development environment on Linux. Please see the previous posts to start at the beginning:

It took a bit longer than anticipated to get to this point, as I got side tracked with the Ioke type checking activities. But now when Ioke S is out we can continue with the series. So! We’re getting closer… Only a few more things to go.

Installing Java

As the current version of Ioke is implemented to run on a Java Virtual Machine, we need to get hold of one of those to be able to continue. The JVM brand we will install here is the latest stable one produced by Sun.

What we need is Java SE Development Kit 6u11 for Linux, Multi-language, and the the filename for this is jdk-6u11-linux-i586.bin. Note that this is not the .rpm.bin, which is also available.

Download the file from http://java.sun.com/javase/downloads/index.jsp (I’m sure you can find it) and put it in the ~/work directory.

To unpack the file, let’s make it executable and then run it:

~$ cd ~/work
~/work$ cd ~/work                   
~/work$ chmod a+x jdk-6u11-linux-i586.bin 
~/work$ ./jdk-6u11-linux-i586.bin         
... lots of legalese here...

Once the license agreement is accepted, the JVM will unpack itself into the directory ~/work/jdk1.6.0_11. Let’s move it to where we want it and add java and javac to the ~/bin directory as before:

~/work$ mv jdk1.6.0_11 ../opt
~/work$ cd ~/bin
~/bin$ ln -s ../opt/jdk1.6.0_11/bin/java
~/bin$ ln -s ../opt/jdk1.6.0_11/bin/javac

And test it:

That done – let’s move on to Ant.
Ant
Ioke uses Apache Ant to handle the build process, so to build Ioke we need to get Ant installed.
Let’s download the Ant distributable package and unpack it in one go – as we did with Git:

~/bin$ cd ~/opt
~/opt$ wget -O - http://www.apache.org/dist/ant/binaries/apache-ant-1.7.1-bin.tar.bz2 | tar xjv

 
And add it to the ~/bin directory again so it's on the path:
 
~/opt$ cd ~/bin
~/bin$ ln -s ../opt/apache-ant-1.7.1/bin/ant

Let’s try it out:

~/bin$ ant -version
Apache Ant version 1.7.1 compiled on June 27 2008

Looking good!
Soon to come: How to get the Ioke source and build it.
/M
Update: Part 5: Ioke and the REPL

Type Checking in Ioke Java Methods

Martin — Mon, 19 Jan 2009 07:15:06 +0000

Ola Bini recently issued a call to arms to help with the receiver and argument validation for Java methods in Ioke. These are the Ioke methods that are implemented in Java, instead of in Ioke itself.

The Java methods usually operate on the specific data contained within Ioke objects. This data corresponds to different Java classes, depending on what the Ioke object should hold. For instance, an Ioke List contains inside the data area a Java List. To operate on this List the Java code needs to get hold of the List reference – and it does this in a lot of cases by assuming the data object is of a certain type and casts to this type. If the data object is of a different type then a Java ClassCastException is thrown, which makes the interpreter quit.

Since Ola mailed the call to arms a number of things have happened. A few people (including myself) have volunteered to help out with adding the validation code. Additionally, a few supporting methods and additions have been added to the Ioke code base to simplify adding type checks to the existing code. And it’s these additions that I will talk about here.

Current Code

As an example what the new type validations can look like I will use the List + method that Ola also mentioned in the example.

This is the previous code from IokeList.java:

obj.registerMethod(runtime.newJavaMethod("returns a new list that contains the receivers elements and the elements of the list sent in as the argument.", new JavaMethod("+") {
        private final DefaultArgumentsDefinition ARGUMENTS = DefaultArgumentsDefinition
            .builder()
            .withRequiredPositional("otherList")
            .getArguments();
 
        @Override
        public DefaultArgumentsDefinition getArguments() {
            return ARGUMENTS;
        }
 
        @Override
        public Object activate(IokeObject method, IokeObject context, IokeObject message, Object on) throws ControlFlow {
            List<Object> args = new ArrayList<Object>();
            getArguments().getEvaluatedArguments(context, message, on, args, new HashMap<String, Object>());
            List<Object> newList = new ArrayList<Object>();
            newList.addAll(((IokeList)IokeObject.data(on)).getList());
            newList.addAll(((IokeList)IokeObject.data(args.get(0))).getList());
            return context.runtime.newList(newList, IokeObject.as(on));
        }
    }));

New Type Checks

The key to the new validation are two new classes, which extend the functionality of the normal JavaMethod class with type checking functionality. These are:

When using these to define the Java method we can “annotate” the arguments definition with types for both the arguments and the receiver. By then implementing the appropriate activate(...) the super class takes care of evaluating and validating (converting as appropriate) the receiver and arguments.

For instance, for the List + example, the above code gets changed to the following:

obj.registerMethod(runtime.newJavaMethod("returns a new list that contains the receivers elements and the elements of the list sent in as the argument.", new TypeCheckingJavaMethod("+") {
        private final TypeCheckingArgumentsDefinition ARGUMENTS = TypeCheckingArgumentsDefinition
            .builder()
            .receiverMustMimic(runtime.list)
            .withRequiredPositional("otherList").whichMustMimic(runtime.list)
            .getArguments();
 
        @Override
        public TypeCheckingArgumentsDefinition getArguments() {
            return ARGUMENTS;
        }
 
        @Override
        public Object activate(IokeObject self, Object on, List<Object> args, Map<String, Object> keywords, IokeObject context, IokeObject message) throws ControlFlow {
            List<Object> newList = new ArrayList<Object>();
            newList.addAll(((IokeList)IokeObject.data(on)).getList());
            newList.addAll(((IokeList)IokeObject.data(args.get(0))).getList());
            return context.runtime.newList(newList, IokeObject.as(on));
        }
    }));

Note that most of the boiler plate code for arguments handling (which most JavaMethods do) is removed, leaving a clean implementation of the necessary logic – to concatenate two lists in this case.

A few things are done:

The JavaMethod is changed to TypeCheckingJavaMethod
The arguments definition is changed to a TypeCheckingArgumentsDefinition
The appropriate types are added to the arguments definition
Return type of getArguments is changed to TypeCheckingArgumentsDefinition
The active method is changed to the one which gets the arguments list and keywords
The manual call to getArguments().getEvaluatedArguments() is removed as we now get the arguments passed

The relevant tests for this, which should be added to the list_spec.ik test file, could look like:

describe(List,
  describe("+", 
    it("should validate type of receiver",
      x = Origin mimic
      x cell("+") = List cell("+")
      fn(x + [3]) should signal(Condition Error Type IncorrectType)
    )
 
    it("should validate type of argument",
      fn([1,2,3] + 3) should signal(Condition Error Type IncorrectType)
    )
  )
)

So for you to do:

Fork Ioke
Pick a IokeData subclass which no one has started on yet (check with Ola/naeu/me in the IRC channel #ioke on FreeNode)
Identify a method which makes faulty assumptions on receiver or arguments
Add a test as per the above to validate the arguments and receiver
Change JavaMethod implementation as per the above to fix the test
Commit test and fix in togther to your fork
Rinse and repeat until the whole file is done
Convince Ola to pull your fork

Let’s get crackin’!

(but first – back to normal work… :)

The Hitchhiker’s Guide to an Ioke Dev Env From Source (part 3: emacs-starter-kit)

Martin — Tue, 13 Jan 2009 21:46:57 +0000

This is the third part in a series of posts for non-experts about setting up an Ioke development environment on Linux. Please see the previous posts to start at the beginning:

The emacs-starter-kit is a set of base configuration for Emacs. It contains a number of useful elisp libraries, with a slight focus on dynamic languages.

To install it, perform the following steps (note that we move any existing Emacs configuration out of the way first to avoid stomping what you currently have):

~/work/emacs$ cd
~$ mv .emacs.d .emacs.d.old
~$ mv .emacsrc .emacsrc.old
~$ git clone git://github.com/technomancy/emacs-starter-kit.git .emacs.d

If you now start Emacs again you’ll see that the menu bar and the toolbar is gone. This is the default in emacs-starter-kit as most Emacs users don’t find them useful. For new users the menu bar can sometimes come in handy, to get it back temporarily, just press F1.

Configuring Emacs

If you want to add your own customizations to Emacs when using emacs-starter-kit, just add an Emacs LISP file called username.el, or hostname.el, in the ~/.emacs.d directory. For instance, to make the menu bar always visible:

Open Emacs, if it’s not open already.
Press C-x C-f and type in: ~/.emacs.d/username.el
Where username is the name you log in with (for instance, in my case the complete filename is melwin.el).
Type in the following in the file:
```
(menu-bar-mode 1)
```
And save the file with C-x s.
Now quit (C-x c) and restart and you’ll see that the menu bar is shown.

Working with Magit

emacs-starter-kit includes, among many other things, the very nice Magit Git mode for Emacs, which gives you a nice interface for working with a Git repository.

Let’s use this mode to commit our recent changes to the configuration file to our local clone of the emacs-starter-kit repository. This helps us track changes we make and also makes a backup of the file in case we screw (sorry, mess) something up.

Note 1: To move easily between Emacs windows using the keyboard, just press Shift and the arrow key pointing in the direction you want to move.
Note 2: To only show the current Emacs window – press: C-x 1

Inside Emacs, press C-x g to run magit-status and enter the directory (note that Tab auto-completes): ~/.emacs.d
Put the cursor over the username.el in the list of Untracked files.
Press s to Stage the new file – this adds the file to the Git staging area, from which all files are committed.
Press d and then accept to diff against HEAD – this will show you a diff view of the changes we have staged – just the add of a single file.
Press c to perform the commit. This opens a new buffer into which a commit message can be added.
Write something like: Add personal configuration file.
Now press C-c C-c to commit the file.

That’s it – now the change has been committed to the local clone of the emacs-starter-kit Git repository.

To see the log of all commits, press l (lowercase L) in the magit-status buffer:

To look at a certain commit – just press Enter on it and a view of the diff will be shown. This makes it quite easy to browse through commits in a repository. At the top of the log is the most recent commit, which in this case is the file we just added.

To update emacs-starter-kit with the latest changes in the GitHub repository, just press F in the magit-status buffer or run git pull in the ~/.emacs.d directory.

Summary

Now that we have Git and Emacs set up we can finally move to Ioke. In the next post we’ll go through installing the latest Java JDK and getting and compiling the Ioke source code.

Join me then!

Update: Part 4: Java and Ant

The Hitchhiker’s Guide to an Ioke Dev Env From Source (part 2: Emacs)

Martin — Mon, 12 Jan 2009 23:10:59 +0000

This is the second part in a series of posts for non-experts about setting up an Ioke development environment on Linux. Please see the previous post to start at the beginning:

Part 1: Git

In this post we will install GNU Emacs from source and also get a basic configuration set up using the emacs-starter-kit.

If you already have Emacs installed and have an old configuration laying around you probably want to make a backup of this before following the below instructions. This post assumes that there is no Emacs installed and that the user doesn’t have any Emacs configuration in the home directory.

Installing Emacs

Git is an efficient Distributed Version Control System. Although the primary VCS for Emacs is still CVS – and it looks like they are moving towards Bazaar – we’ll get the Emacs sources from the Emacs Git mirror. We want to be cool, right?

~$ cd work
~/work$ git clone --depth 1 git://git.sv.gnu.org/emacs.git
Initialized empty Git repository in /home/melwin/work/emacs/.git/
remote: Counting objects: 46400, done.
remote: Compressing objects: 100% (24410/24410), done.
remote: Total 46400 (delta 41204), reused 25501 (delta 21836)
Receiving objects: 100% (46400/46400), 74.38 MiB | 213 KiB/s, done.
Resolving deltas: 100% (41204/41204), done.
Checking out files: 100% (2837/2837), done.

With the --depth 1 parameter we limit the history so that we only get the latest version of the files – in this case we’re not interested in the full history.

Note: As we’re getting the bleeding edge source code of Emacs, it could happen that the build is broken. I’ve never had this happen on me, but in case you get strange errors when building Emacs, this might be the reason. Usually such problems are fixed quickly, so try to do a git pull a bit later to update the downloaded source.

Previously, to get fun things like multi-tty support and smooth fonts, we had to get specific feature branches of Emacs. Nowadays, however, all the things we want are merged into the Emacs master branch. Before building, the only thing we need to do is to make sure the necessary development libraries are installed. In case you wonder how I came up with this list: the good ol’ method of trial and error. I simply ran the configure command and checked the error messages. This helped me identify the needed libraries. Now you can reap the benefits by just doing the following:

~/work$ sudo apt-get install libgtk2.0-dev libxpm-dev libjpeg-dev libgif-dev libtiff-dev
...

… and then run the classical configure, make and make install:

~/work$ cd emacs
~/work/emacs$ ./configure --prefix=$HOME/opt/emacs-23.0.60
...
~/work/emacs$ make
...
~/work/emacs$ make install
...

Phew. That burned som CPU cycles…! Once done – let’s add it to our bin directory, just as we did with Git.

~/work/emacs$ cd ~/bin
~/bin$ ln -s ../opt/emacs-23.0.60/bin/emacs
~/bin$ ln -s ../opt/emacs-23.0.60/bin/emacsclient
~/bin$ ls
emacs  emacsclient  git

Time to test. Just run emacs:

~/bin% emacs

This should show you an Emacs window with a pretty(?) GNU.

If you’re completely new to Emacs, this might be a good opportunity to run the Emacs tutorial. Emacs is a self-documenting editor, which means that most things that you might want to learn about Emacs can be found inside the editor itself – including information about internal functions and libraries.

To run the tutorial, just press Ctrl+h t (that is, control and h, release control, then press t), or, in Emacs lingo, C-h t (which is the convention I’ll use from now on).

Once you’ve learned enough (you did complete the whole thing, right?), just quit using C-x c.

Note: To update the source code and rebuild, do a git pull in the emacs directory, then make distclean (in case some build files changed) and then perform the compile and installation as per the above again.

Next up is emacs-starter-kit to get more functionality in Emacs and a decent default configuration to help us get going – stay tuned!

Update: Part 3: emacs-starter-kit

The Hitchhiker’s Guide to an Ioke Dev Env From Source (part 1: Git)

Martin — Mon, 12 Jan 2009 19:35:36 +0000

In this series of posts I will guide you, the humble reader, through the install procedure to get a development environment for Ioke up and running. All the cool kids want to develop in Ioke nowadays, so let’s make you can as well!

The components we will be installing are:

These instructions are written for a non-expert who might not have too much experience with compiling things, using Git or Emacs. If you’re an expert you will find nothing new or exciting here! The environment I use is a freshly installed Kubuntu Intrepid Ibex box, although most of it should be similar, if not identical, to other Ubuntu/Debian based distributions.

The components listed above will all be installed manually – not using the package system. This is to allow us to get the freshest versions of what we need without depending on the packages in the distribution to be updated (or hunted down in alternative repositories). The only thing we will install from the distribution are the compilers, tools and development libraries used to compile the software.

Why do we not use the package system? Well, that’s a valid question. Most of the above can be installed from packages in custom repositories. It’s simple and convenient. However here we will not use if for two reasons:

If packages are used we will depend on the repositories to be updated to use the latest version of some software, which can be annoying if we want to develop on the bleeding edge.
It’s good to know how to compile things yourself – so by not using prepared packages we might learn something!

To keep control over our custom built software we’ll install it in dedicated directories under $HOME/opt instead of in the normal locations like /usr/bin. This allows us to get everything installed without needing root access (except to install the compilers etc from the distribution).

First Step

Don’t Panic!

Always sound advice to start with – especially if you are hitchhiking. I’m going to try to cover each step in detail, but if you think something is unclear – just leave a comment and I’ll try to clarify. So – grab your towel and let’s get going!

Install Git

For Emacs, emacs-starter-kit and Ioke we will use Git to retrieve the sources. Therefore the first thing we need to install is Git.

In the command line instructions below, all lines prefixed by the prompt

$ are commands to be typed in. The rest are output or my comments.

Open a terminal window and create a new work directory under your home using the following instructions. In this directory we will work with downloaded source files and build the software.

~$ mkdir ~/work
~/work$ cd ~/work
~/work$ wget -O - http://kernel.org/pub/software/scm/git/git-1.6.1.tar.bz2 | tar xjv
... many lines...

The last command downloads the git source package and expands it in one go by piping it directly from wget to tar, which is told to expand it using bz2 with the j parameter. This is a convenient way to get packages off the net and unpacked, especially when there is no need to keep the package itself around.

The result is that we now have a git-1.6.1 directory in the work directory. Let’s see how we compile it:

~/work$ cd git-1.6.1/
~/work/git-1.6.1$ head INSTALL
 
                Git installation
 
Normally you can just do "make" followed by "make install", and that
will install the git programs in your own ~/bin/ directory.  If you want
to do a global install, you can do
 
        $ make prefix=/usr all doc info ;# as yourself
        # make prefix=/usr install install-doc install-html install-info ;# as root
 
~/work/git-1.6.1$

So – two commands. Simple enough! However, before we compile – we need to make sure all the relevant packages are installed:

~/work/git-1.6.1$ sudo apt-get install libcurl4-openssl-dev zlib1g-dev libexpat-dev tk8.5 asciidoc docbook2x

Depending on your internet connection, this could take quite a while, as we need to download the texlive distribution, among other things, to build all of the Git documentation. This is not strictly necessary, but here we’ll just do it for completeness’ sake.

Let’s build git and make sure it’s installed under our $HOME/opt directory as we said in the beginning:

~/work/git-1.6.1$ make prefix=~/opt/git-1.6.1 all doc info
... lots of lines...

Compiling Git will take some time as well. Go get a coffee (or perhaps a Pan Galactic Gargle Blaster – sweet like nectar).

Once the compile is done – install it. Note that we don’t need to do this as root as we’re installing under the user home directory:

~/work/git-1.6.1$ make prefix=~/opt/git-1.6.1 install install-doc install-html install-info

Let’s add it to the user’s path by linking it into the private bin directory. This depends on the standard Ubuntu bash shell profile script which adds ~/bin to the PATH variable. If a different shell is used you need to perform the appropriate steps yourself.

~/work/git-1.6.1$ mkdir ~/bin
~/work/git-1.6.1$ cd !$
~/bin$ ln -s ../opt/git-1.6.1/bin/git

Now close the shell/terminal, open a new one – and try the git command:

~$ git --version
git version 1.6.1

If you get the above output – great! You’re don! Sit back and relax for a bit before moving on to the next section.

However, if you don’t get the version output, but instead see the following message, review the previous instructions and make sure it works ok before continuing.

#INCORRECT OUTPUT - SOMETHING WAS MISSED!
#GO BACK AND REVIEW
~$ git                                                                                                                                                                                        
The program 'git' is currently not installed.  You can install it by typing:                                                                                                                  
sudo apt-get install git-core                                                                                                                                                                 
-bash: git: command not found

If you still can’t get it to work – drop me a comment!

Git in 30 Seconds

There are loads of good Git tutorials and information. You can find several on the official Git page and at GitHub (go sign up if you haven’t already, and fork me!).

Here is a quick run through of a few common Git commands you could try out with your freshly brewed cup of Git:

~$ cd
~$ mkdir gittest
~$ cd gittest/
~/gittest$ git init
Initialized empty Git repository in /home/melwin/gittest/.git/
~/gittest$ echo Test file! > test.txt
~/gittest$ git add test.txt
~/gittest$ git commit -m "Initial import."
[master (root-commit)]: created 79c091d: "Initial import."
 1 files changed, 1 insertions(+), 0 deletions(-)
 create mode 100644 test.txt
~/gittest$ echo Add line. >> test.txt
~/gittest$ git diff test.txt
diff --git a/test.txt b/test.txt
index 1cbaf90..3746f9e 100644
--- a/test.txt
+++ b/test.txt
@@ -1 +1,2 @@
 Test file!
+Add line.
~/gittest$ git commit -a -m "Add new line."
[master]: created b20f9a4: "Add new line."
 1 files changed, 1 insertions(+), 0 deletions(-)

If you can follow the above – great! First part finished. Next up is to install GNU Emacs from source and the emacs-starter-kit, which provides a decent default a set of configuration for Emacs.

Stay tuned!

Update: Part 2: Emacs

Ioke Syntax Highlighter for GeSHi

Martin — Fri, 09 Jan 2009 22:53:07 +0000

Speaking of Ioke – just added a syntax highlighter definition for GeSHi in my Ioke fork repository. The commit can be found here:

http://github.com/melwin/ioke/commit/54cfd7e54de0be910385c6ec805693fd3ed4e294

The keyword definitions I borrowed (read stole) from Sam Aaron’s TextMate bundle (also in the Ioke repository). As Sam just said in the #ioke on freenode: “what goes around comes around”. :)

Highlighting test:

    m = #/({areaCode}\d{3})-({localNumber}\d{5})/ =~ number
 
    describe("start",
      it("should return the start index of group zero, which is the whole group",
        (#/foo/ =~ "foobar") start should == 0
        (#/foo/ =~ "abcfoobar") start should == 3
 
        (#/foo/ =~ "foobar") start(0) should == 0
        (#/foo/ =~ "abcfoobar") start(0) should == 3
      )
 
      it("should return the start index of another group",
        (#/(..) (..) (..)/ =~ "fooab cd efbar") start(2) should == 6
      )
 
      it("should return the start index from the name of a named group",
        (#/({one}..) ({two}..) ({three}..)/ =~ "fooab cd efbar") start(:two) should == 6
      )
 
      it("should return -1 for a group that wasn't matched",
        (#/(..)((..))?/ =~ "ab") start(2) should == -1
        (#/({no}..)(({way}..))?/ =~ "ab") start(:way) should == -1
 
        (#/(..)((..))?/ =~ "ab") start(10) should == -1
        (#/({no}..)(({way}..))?/ =~ "ab") start(:blarg) should == -1
      )
 
      it("should validate type of receiver",
        Regexp Match should checkReceiverTypeOn(:start)
      )
    )
 
    x = #/bla #{"foo"} bar/

Simple JSON Parser in Ioke

Martin — Fri, 09 Jan 2009 22:35:47 +0000

After having enjoyed most of the days off over Christmas my fingers started itching – time to do some programming. Luckily around the same time I stumbled across Ola Bini’s posts on the Ioke language. As the Ioke guide begins:

Ioke is a general purpose language. It is a strongly typed, extremely dynamic, prototype object oriented language. It is homoiconic and it’s closest ancestors is Io, Smalltalk, Ruby and Lisp – but it’s quite a distance from all of them.

So what does this all mean? It means it’s fun! The language syntax is very regular (although not quite as regular as lisp) – code is data, data is code and everything is a message. It’s also one of relatively few languages that provide macros similar to those in lisp. In Ioke this means that a message stream can be operated on (modified, transformed, etc) before it’s evaluated, using normal Ioke methods. Anyone who groks lisp should be familiar with the power this gives.

In Ioke, a chain of messages is separated by space, with the first message being sent to the current receiver and subsequent messages are sent to the result of the previous one. For instance, in the code

"ioke rocks" upper println

the text literal “ioke rocks” results in an internal message which creates a Text object. To this object the upper messages is sent, which results in an upper case copy of the Text object is created. To this object the println message is sent, which prints the upper case text to the console.

As the Ioke reader handles space separated tokens, it’s quite easy to create macros to process non Ioke code. As a learning exercise I decided to try to implement a JSON parser. Nothing too fancy – and doesn’t even have to be secure – just enough to parse simple JSON into Ioke objects.

JSON is usually parsed in one of two ways – into custom objects/classes or into collections like dictionaries and arrays. For this exercise I decided to create Ioke dictionaries and arrays and I want the result to work like this:

json({
  "string" : "string1",
  "int" : 1234,
  "arr" : ["item1", "item2"],
  "dict" : {"key1":"value1"}
}) println  ;; => {dict={key1=value1}, arr=[item1, item2], int=1234, string=string1}
 
json(["string",
  1234,
  ["item1", "item2"],
  {"key1": "val1"}
]) println ;; => [string, 1234, [item1, item2], {key1=val1}]

Now, looking at JSON, it’s suspiciously similar to Ioke – [] is used for arrays, {} is used for “dictionaries”, comma separates entries, ” quotes strings, etc. The main problem is the colon “:” used for separating keys and values in a dictionary. As Ioke is a dynamic language we can redefine how messages are handled. Instead of using colon to define symbols, we can redefine it to instead create pairs suitable for a dict:

Text cell(":") = macro(call resendToMethod("=>"))

“:” is originally defined on DefaultBehavior Literals, but in the JSON the “:” message is always sent to Text, so it’s enough to redefine it there.

This single redefinition allows us to parse JSON directly with the Ioke reader – very nice.

But… Redefining “:” for all Text objects is not really good (especially not when Ola adds concurrency constructs!). Another alternative would be to redefine how the text objects are created and create a new “:” cell just for our JSON text objects – but I never got this to work, as the internal:createText message works with a raw Java string, which I couldn’t work with in a macro/method…

Update: With Ola’s recent commit, the previously described workaround can now be simplified to the following code.

Now the above redefinition of the “:” message can be used, but is visible only in the scope of a let:

json = macro(let(Text cell(":"), DefaultBehavior cell("=>"),
    call argAt(0)
))
 
;; Used as:
 
json({
  "string" : "string1",
  "int" : 1234,
  "arr" : ["item1", "item2"],
  "dict" : {"key1":"value1"}
  }) println
 
json(["string",
  1234,
  ["item1", "item2"],
  {"key1": "val1"}
  ]) println

As can be seen, the Ioke macro functionality gives us nice control over how messages are handled, and the environment can be changed before the arguments are evaluated. I’m looking forward to the day when we have full Java interop and can use these powerful constructs to control Java code!

Source code to the above example is committed to my Ioke fork at: git://github.com/melwin/ioke.git

PS: The thing that confuses me the most: how the heck are you supposed to pronounce Ioke?

Update: Sam Aaron shared the fact that Ola pronounces it eye-oh-key in the following interview: http://www.akitaonrails.com/2008/11/22/rails-podcast-brasil-qcon-special-ola-bini-jruby-ioke

Caching Filter Queries with Coherence

Martin — Sun, 21 Sep 2008 20:56:59 +0000

A pretty nice thing that I recently tried with a customer was storing a query result in a query cache.

For instance, consider the following method:

public Collection getByFilter1(String cacheName, Filter f) {
    NamedCache c = CacheFactory.getCache(cacheName);
    return c.entrySet(f);
}

A query is executed across all nodes containing cache data in a cluster. The filter acts on the values in the cache, not the keys. And as all values are usually not available locally on every node the query needs to execute on the separate nodes.

The query above is correct, but not very efficient. In the above case we query and retrieve the data as well from the separate nodes. A more efficient way would be to only get the keys for the matching values from the nodes and then retrieve the values from the local near cache:

public Collection getByFilter2(String cacheName, Filter f) {
    NamedCache c = CacheFactory.getCache(cacheName);
    Set keys = c.keySet(f);
    return c.getAll(keys).values();
}

However, in this case we still perform the query every time.

Now – the clever part. Not so much on my side, but the Coherence engineers have thought things through and made the Filter implementations have good hashCode and equals implementations. This together with the fact that they are serializable makes them possible to use as keys in a cache! Sweet! Without changing our method’s interface we can add a query cache so that each query only is performed once.

public Collection getByFilter3(String cacheName, Filter f) {
    NamedCache c = CacheFactory.getCache(cacheName);
 
    NamedCache queryC = CacheFactory.getCache(cacheName + ".querycache");
    Set keys = (Set)queryC.get(f);
 
    if(keys == null) {
        keys = c.keySet(f);
        queryC.put(f, keys);
    }
 
    return c.getAll(keys).values();
}

Note that we only save the keys in the query cache. This to avoid having several caches with the same data. When near caching is used, getting the data for the keys can still be a local only operation. Compared to getting the data from the separate nodes it’s several orders of magnitude faster – depending on the usage patterns.

Of course if queries are different every time, the query cache will not help much. But in most high load applications the same data tends to be needed several times.

Additionally, the properties queried on should in most cases be indexed. This is important to avoid too much overhead when searching for an entry in a cache.

One thing to think about when adding query caches is: how is data updated?

As part of updating the value caches the query caches should preferably be emptied of the outdated cached queries. This could be done programmatically “manually” when the data is updated, or by hooking in code to clean the entries from the query cache using the map listener mechanism. The relevant code could do a query using the ContainsFilter.

To summarize: a little code can go a long way to improve performance without affecting the interface used by an application. Good when query heavy applications are adapted to use a distributed cache like Coherence.

Scala Syntax Highlighting for WP

Martin — Mon, 30 Jun 2008 07:56:52 +0000

Writing the previous post I realized that the Wordpress plugin for syntax highligting I was using (Highlight Source Pro) didn’t support Scala.

Instead I tried the WP-Syntax plugin, but this didn’t support Symbol literals:

val sym = 'foobar
println("Symbol is: " + sym)
val other = 'barfoo

This seems to be because of single quote being interpreted as a quotation character. A fix seems to be removing the single quote from the Geshi language definition file scala.php in the wp-syntax/geshi/geshi directory. While there one can also specify a regex for symbols and give them their own color:

val sym = 'foobar
println("Symbol is: " + sym)
val other = 'barfoo

Also check out the scala.php in the LAMP repository – it seems to add some additional keywords and other colors. I used this and made the changes described above.

Full new scala.php for Geshi after the break.


/*************************************************************************************
 * scala.php
 * --------
 * Author: Geoffrey Washburn (washburn@acm.ogr)
 * Copyright: (c) 2004 Nigel McNie (http://qbnz.com/highlighter/)
 * Release Version: ???
 * Date Started: 2008/01/03
 *
 * Scala language file for GeSHi.
 *
 * CHANGES
 * -------
 * 2007/01/03
 *   -  Created by copying the Java highlighter
 *
 * TODO
 * -------------------------
 * * Finish
 *
 *************************************************************************************
 *
 *     This file is part of GeSHi.
 *
 *   GeSHi is free software; you can redistribute it and/or modify
 *   it under the terms of the GNU General Public License as published by
 *   the Free Software Foundation; either version 2 of the License, or
 *   (at your option) any later version.
 *
 *   GeSHi is distributed in the hope that it will be useful,
 *   but WITHOUT ANY WARRANTY; without even the implied warranty of
 *   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 *   GNU General Public License for more details.
 *
 *   You should have received a copy of the GNU General Public License
 *   along with GeSHi; if not, write to the Free Software
 *   Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
 *
 ************************************************************************************/
 
$language_data = array (
        'LANG_NAME' => 'Scala',
        'COMMENT_SINGLE' => array(1 => '//'),   /* import statements are not comments! */
        'COMMENT_MULTI' => array('/*' => '*/'),
        'CASE_KEYWORDS' => GESHI_CAPS_NO_CHANGE,
        'QUOTEMARKS' => array('"', '"""'),
        'ESCAPE_CHAR' => '\\',
        'KEYWORDS' => array(
                1 => array(
                        /* Scala keywords, part 1: control flow */
                        'case', 'default', 'do', 'else', 'for',
                        'if', 'match', 'while'
                        ),
                2 => array(
                        /* Scala keywords, part 2 */
                        'return', 'throw',
                        'try', 'catch', 'finally',
                        'abstract', 'class', 'def', 'extends',
                        'final', 'forSome', 'implicit', 'import',
                        'lazy', 'new', 'object', 'override', 'package',
                        'private', 'protected',
                        'requires', 'sealed', 'super', 'this', 'trait', 'type',
                        'val', 'var', 'with', 'yield'
                        ),
                3 => array(
                        /* Scala keywords, part 3: standard value types */
                        'unit', 'Unit', 'boolean', 'Boolean', 'int', 'Int', 'Any', 'AnyVal', 'Nothing',
                        ),
                4 => array(
                        /* other reserved words in Scala: literals */
                        /* should be styled to look similar to numbers and Strings */
                        'false', 'null', 'true'
                        ),
                5 => array(
                        /* Scala reference types */
                        'AnyRef', 'Null', 'List', 'String', 'Integer', 'Option', 'Array'
                        )
 
                ),
        'SYMBOLS' => array(
                ':', '*', '&', '%', '!', ';', '<', '>', '?', '_', '=', '=>',
                '<-', '<:', '<%', '>:', '#', '@'
                ),
        'CASE_SENSITIVE' => array(
                GESHI_COMMENTS => true,
                /* all Scala keywords are case sensitive */
                1 => true, 2 => true, 3 => true, 4 => true, 5 => true ),
        'STYLES' => array(
                'KEYWORDS' => array(
                        1 => 'color: #b1b100;',
                        2 => 'color: #000000; font-weight: bold;',
                        3 => 'color: #993333;',
                        4 => 'color: #b13366;',
                        5 => 'color: #aaaadd;'
                        ),
                'SYMBOLS' => array(
                        0 => 'color: #FFAA00;'
                        ),
                'COMMENTS' => array(
                        1 => 'color: #808080; font-style: italic;',
                        'MULTI' => 'color: #808080; font-style: italic;'
                        ),
                'ESCAPE_CHAR' => array(
                        0 => 'color: #000099; font-weight: bold;'
                        ),
                'BRACKETS' => array(
                        0 => 'color: #66cc66;'
                        ),
                'STRINGS' => array(
                        0 => 'color: #ff0000;'
                        ),
                'NUMBERS' => array(
                        0 => 'color: #cc66cc;'
                        ),
                'METHODS' => array(
                        1 => 'color: #006600;',
                        2 => 'color: #006600;'
                        ),
                'SCRIPT' => array(
                        ),
                'REGEXPS' => array(
                        0 => 'color: #008000;'
                        )
                ),
        'URLS' => array(
                1 => '',
                2 => '',
                3 => '',
                4 => ''
                ),
        'OOLANG' => true,
        'OBJECT_SPLITTERS' => array(
                1 => '.'
                ),
        'REGEXPS' => array(
               0 => "'[a-zA-Z_][a-zA-Z0-9_]*"
                ),
        'STRICT_MODE_APPLIES' => GESHI_NEVER,
        'SCRIPT_DELIMITERS' => array(
                ),
        'HIGHLIGHT_STRICT_BLOCK' => array(
                )
);
 
?>