Code by Martin

This is the fifth part in a series of posts for non-experts about setting up an Ioke development environment on Linux. Please see the previous posts to start at the beginning:

• Part 1: Git
• Part 2: Emacs
• Part 3: emacs-starter-kit
• Part 4: Java and Ant

This time we will finally get the Ioke source code, build it, and test the Ioke REPL!

Ioke

Ioke uses Git as the source code versioning tool, with a public repository set up on GitHub (soon I’ll be linking like Jeff Atwood).

So – let’s get Ioke using Git by cloning Ola’s repository:

~/bin$ cd ~/work
~/work$ git clone git://github.com/olabini/ioke
Initialized empty Git repository in /home/melwin/work/ioke/.git/
remote: Counting objects: 9221, done.
remote: Compressing objects: 100% (2941/2941), done.
remote: Total 9221 (delta 5782), reused 8656 (delta 5351)
Receiving objects: 100% (9221/9221), 45.95 MiB | 399 KiB/s, done.
Resolving deltas: 100% (5782/5782), done.

Great – let’s quickly move on to the build step:

~/work$ cd ioke
~/work/ioke$ ant
...
     [java] 2606 examples, 0 failures
jar:
      [jar] Building jar: /home/melwin/work/ioke/lib/ioke.jar
 
BUILD SUCCESSFUL
Total time: 22 seconds

Running just ant will execute the default target in the build.xml file. This target will make sure Ioke is compiled and the test suite is run.

At the end is a summary of the test results – if these are not all successful you might have a problem with the environment or you might have checked out a broken build from the repository.

The Ioke REPL

Now all is installed and Ioke is compiled, so let’s fire up the REPL. This is done by just running the main Ioke binary:

~/work/ioke/$ bin/ioke
iik>

When doing this we get the iik> REPL prompt. From here we can execute most Ioke code. Let’s try to print a string:

iik> "Hello World!" println
Hello World!
+> nil
 
iik>

Here we create a Text (Ioke’s string type) literal and send it the message println, which will cause the text to be printed to the console. The return type of println is nil (similar to null), which is printed by the REPL itself. After that we get the prompt back and can execute another line.

Let’s try a little more complex example – the factorial:

iik> fact = method(n, if(n > 1, n * fact(n pred), 1))
+> fact:method(n, if(n >(1), n *(fact(n pred)), 1))

Note how the REPL prints the method definition in a canonical form – it makes it obvious how Ioke parses the code, what are messages and what are arguments.

Let’s run our new factorial function:

iik> fact(3)
+> 6
 
iik> fact(10)
+> 3628800
 
iik> fact(50)
+> 30414093201713378043612608166064768844377641568960512000000000000

Ioke handles arbitrarily big numbers, so we don’t need to think about what fits in a certain number of bits.

The Iik REPL also comes with a simple debugger, which helps with handling conditions. For instance, let’s try to run the fact method, passing a message which doesn’t mean anything (yet):

iik> fact(f)
*** - couldn't find cell 'f' on 'Ground_0xC360E7' (Condition Error NoSuchCell)
 f                                                [<init>:1:5]
The following restarts are available:
 0: storeValue           (Store value for: f)
 1: useValue             (Use value for: f)
 2: abort                (restart: abort)
 3: quit                 (restart: quit)
 
 dbg:1> 1
  dbg:1:newValue> 5
  +> 5
 
+> 120

The above output shows that when we try to reference something that doesn’t exist we get a chance to provide that value. Here I decided to provide a value using the useValue restart and entered the value 5. This let the method continue executing and the result was printed.

To exit the REPL, just type: exit

Next up are the Emacs modes for Ioke – stay tuned!

/M

This is the fourth part in a series of posts for non-experts about setting up an Ioke development environment on Linux. Please see the previous posts to start at the beginning:

• Part 1: Git
• Part 2: Emacs
• Part 3: emacs-starter-kit

It took a bit longer than anticipated to get to this point, as I got side tracked with the Ioke type checking activities. But now when Ioke S is out we can continue with the series. So! We’re getting closer… Only a few more things to go.

Installing Java

As the current version of Ioke is implemented to run on a Java Virtual Machine, we need to get hold of one of those to be able to continue. The JVM brand we will install here is the latest stable one produced by Sun.

What we need is Java SE Development Kit 6u11 for Linux, Multi-language, and the the filename for this is jdk-6u11-linux-i586.bin. Note that this is not the .rpm.bin, which is also available.

Download the file from http://java.sun.com/javase/downloads/index.jsp (I’m sure you can find it) and put it in the ~/work directory.

To unpack the file, let’s make it executable and then run it:

~$ cd ~/work
~/work$ cd ~/work                   
~/work$ chmod a+x jdk-6u11-linux-i586.bin 
~/work$ ./jdk-6u11-linux-i586.bin         
... lots of legalese here...

Once the license agreement is accepted, the JVM will unpack itself into the directory ~/work/jdk1.6.0_11. Let’s move it to where we want it and add java and javac to the ~/bin directory as before:

~/work$ mv jdk1.6.0_11 ../opt
~/work$ cd ~/bin
~/bin$ ln -s ../opt/jdk1.6.0_11/bin/java
~/bin$ ln -s ../opt/jdk1.6.0_11/bin/javac

And test it:

That done – let’s move on to Ant.
Ant
Ioke uses Apache Ant to handle the build process, so to build Ioke we need to get Ant installed.
Let’s download the Ant distributable package and unpack it in one go – as we did with Git:

~/bin$ cd ~/opt
~/opt$ wget -O - http://www.apache.org/dist/ant/binaries/apache-ant-1.7.1-bin.tar.bz2 | tar xjv
</lang>
 
And add it to the ~/bin directory again so it's on the path:
 
<pre lang="sh">
~/opt$ cd ~/bin
~/bin$ ln -s ../opt/apache-ant-1.7.1/bin/ant

Let’s try it out:

~/bin$ ant -version
Apache Ant version 1.7.1 compiled on June 27 2008

Looking good!
Soon to come: How to get the Ioke source and build it.
/M
Update: Part 5: Ioke and the REPL
				

Ola Bini recently issued a call to arms to help with the receiver and argument validation for Java methods in Ioke. These are the Ioke methods that are implemented in Java, instead of in Ioke itself.

The Java methods usually operate on the specific data contained within Ioke objects. This data corresponds to different Java classes, depending on what the Ioke object should hold. For instance, an Ioke List contains inside the data area a Java List. To operate on this List the Java code needs to get hold of the List reference – and it does this in a lot of cases by assuming the data object is of a certain type and casts to this type. If the data object is of a different type then a Java ClassCastException is thrown, which makes the interpreter quit.

Since Ola mailed the call to arms a number of things have happened. A few people (including myself) have volunteered to help out with adding the validation code. Additionally, a few supporting methods and additions have been added to the Ioke code base to simplify adding type checks to the existing code. And it’s these additions that I will talk about here.

Current Code

As an example what the new type validations can look like I will use the List + method that Ola also mentioned in the example.

This is the previous code from IokeList.java:

obj.registerMethod(runtime.newJavaMethod("returns a new list that contains the receivers elements and the elements of the list sent in as the argument.", new JavaMethod("+") {
        private final DefaultArgumentsDefinition ARGUMENTS = DefaultArgumentsDefinition
            .builder()
            .withRequiredPositional("otherList")
            .getArguments();
 
        @Override
        public DefaultArgumentsDefinition getArguments() {
            return ARGUMENTS;
        }
 
        @Override
        public Object activate(IokeObject method, IokeObject context, IokeObject message, Object on) throws ControlFlow {
            List<Object> args = new ArrayList<Object>();
            getArguments().getEvaluatedArguments(context, message, on, args, new HashMap<String, Object>());
            List<Object> newList = new ArrayList<Object>();
            newList.addAll(((IokeList)IokeObject.data(on)).getList());
            newList.addAll(((IokeList)IokeObject.data(args.get(0))).getList());
            return context.runtime.newList(newList, IokeObject.as(on));
        }
    }));

New Type Checks

The key to the new validation are two new classes, which extend the functionality of the normal JavaMethod class with type checking functionality. These are:

• TypeCheckingArgumentsDefinition
• TypeCheckingJavaMethod

When using these to define the Java method we can “annotate” the arguments definition with types for both the arguments and the receiver. By then implementing the appropriate activate(...) the super class takes care of evaluating and validating (converting as appropriate) the receiver and arguments.

For instance, for the List + example, the above code gets changed to the following:

obj.registerMethod(runtime.newJavaMethod("returns a new list that contains the receivers elements and the elements of the list sent in as the argument.", new TypeCheckingJavaMethod("+") {
        private final TypeCheckingArgumentsDefinition ARGUMENTS = TypeCheckingArgumentsDefinition
            .builder()
            .receiverMustMimic(runtime.list)
            .withRequiredPositional("otherList").whichMustMimic(runtime.list)
            .getArguments();
 
        @Override
        public TypeCheckingArgumentsDefinition getArguments() {
            return ARGUMENTS;
        }
 
        @Override
        public Object activate(IokeObject self, Object on, List<Object> args, Map<String, Object> keywords, IokeObject context, IokeObject message) throws ControlFlow {
            List<Object> newList = new ArrayList<Object>();
            newList.addAll(((IokeList)IokeObject.data(on)).getList());
            newList.addAll(((IokeList)IokeObject.data(args.get(0))).getList());
            return context.runtime.newList(newList, IokeObject.as(on));
        }
    }));

Note that most of the boiler plate code for arguments handling (which most JavaMethods do) is removed, leaving a clean implementation of the necessary logic – to concatenate two lists in this case.

A few things are done:

1. The JavaMethod is changed to TypeCheckingJavaMethod
2. The arguments definition is changed to a TypeCheckingArgumentsDefinition
3. The appropriate types are added to the arguments definition
4. Return type of getArguments is changed to TypeCheckingArgumentsDefinition
5. The active method is changed to the one which gets the arguments list and keywords
6. The manual call to getArguments().getEvaluatedArguments() is removed as we now get the arguments passed

The relevant tests for this, which should be added to the list_spec.ik test file, could look like:

describe(List,
  describe("+", 
    it("should validate type of receiver",
      x = Origin mimic
      x cell("+") = List cell("+")
      fn(x + [3]) should signal(Condition Error Type IncorrectType)
    )
 
    it("should validate type of argument",
      fn([1,2,3] + 3) should signal(Condition Error Type IncorrectType)
    )
  )
)

So for you to do:

1. Fork Ioke
2. Pick a IokeData subclass which no one has started on yet (check with Ola/naeu/me in the IRC channel #ioke on FreeNode)
3. Identify a method which makes faulty assumptions on receiver or arguments
4. Add a test as per the above to validate the arguments and receiver
5. Change JavaMethod implementation as per the above to fix the test
6. Commit test and fix in togther to your fork
7. Rinse and repeat until the whole file is done
8. Convince Ola to pull your fork

Let’s get crackin’!

(but first – back to normal work… :)

/M

A pretty nice thing that I recently tried with a customer was storing a query result in a query cache.

For instance, consider the following method:

public Collection getByFilter1(String cacheName, Filter f) {
    NamedCache c = CacheFactory.getCache(cacheName);
    return c.entrySet(f);
}

A query is executed across all nodes containing cache data in a cluster. The filter acts on the values in the cache, not the keys. And as all values are usually not available locally on every node the query needs to execute on the separate nodes.

The query above is correct, but not very efficient. In the above case we query and retrieve the data as well from the separate nodes. A more efficient way would be to only get the keys for the matching values from the nodes and then retrieve the values from the local near cache:

public Collection getByFilter2(String cacheName, Filter f) {
    NamedCache c = CacheFactory.getCache(cacheName);
    Set keys = c.keySet(f);
    return c.getAll(keys).values();
}

However, in this case we still perform the query every time.

Now – the clever part. Not so much on my side, but the Coherence engineers have thought things through and made the Filter implementations have good hashCode and equals implementations. This together with the fact that they are serializable makes them possible to use as keys in a cache! Sweet! Without changing our method’s interface we can add a query cache so that each query only is performed once.

public Collection getByFilter3(String cacheName, Filter f) {
    NamedCache c = CacheFactory.getCache(cacheName);
 
    NamedCache queryC = CacheFactory.getCache(cacheName + ".querycache");
    Set keys = (Set)queryC.get(f);
 
    if(keys == null) {
        keys = c.keySet(f);
        queryC.put(f, keys);
    }
 
    return c.getAll(keys).values();
}

Note that we only save the keys in the query cache. This to avoid having several caches with the same data. When near caching is used, getting the data for the keys can still be a local only operation. Compared to getting the data from the separate nodes it’s several orders of magnitude faster – depending on the usage patterns.

Of course if queries are different every time, the query cache will not help much. But in most high load applications the same data tends to be needed several times.

Additionally, the properties queried on should in most cases be indexed. This is important to avoid too much overhead when searching for an entry in a cache.

One thing to think about when adding query caches is: how is data updated?

As part of updating the value caches the query caches should preferably be emptied of the outdated cached queries. This could be done programmatically “manually” when the data is updated, or by hooking in code to clean the entries from the query cache using the map listener mechanism. The relevant code could do a query using the ContainsFilter.

To summarize: a little code can go a long way to improve performance without affecting the interface used by an application. Good when query heavy applications are adapted to use a distributed cache like Coherence.

/M

How do we go from and DOM to Scala XML the most efficiently? Well… That’s what I asked myself the other day. The Scala API scala.xml.XML object has helper functions to create Scala XML structures from the usual suspects: Strings, InputStream, Reader, File, etc. But DOM Document or Element is missing.

Going via a Byte array, Char array or String is simple. For instance, the following outputs the DOM to a char array, which is then used as the source of the Scala XML creation:

Using Char array
//dom is the DOM Element
val charWriter = new CharArrayWriter()
TransformerFactory.newInstance.newTransformer.transform(new DOMSource(dom), new StreamResult(charWriter))
val xml = XML.load(new CharArrayReader(charWriter.toCharArray))

This works fine, and is reasonably fast. However, it does allocate some unnecessary memory (the char array) and performs some unnecessary parsing (of the char array) – both of which we’d really like to avoid.

How do we do this? Well, here’s one option that I came up with:

Using SAX
val saxHandler = new NoBindingFactoryAdapter()
saxHandler.scopeStack.push(TopScope)
TransformerFactory.newInstance.newTransformer.transform(new DOMSource(dom), new SAXResult(saxHandler))
saxHandler.scopeStack.pop
val xml = factoryHandler.rootElem

What’s going on here? Well, the Scala XML library uses SAX to parse XML and create the XML structure. One way of generating SAX events is to walk a DOM tree, which is handled by the javax.xml.transform.Transformer with a DOMSource as input and a SAXResult as output. The extension of the DefaultHandler needed for handling the SAX events is implemented by the scala.xml.parsing.FactoryAdapter, which is extended by the NoBindingFactoryAdapter used to construct the XML structure. Because of this, we can do violence on the API and use the NoBindingFactoryAdapter directly as a SAX DefaultHandler – nice! The scopeStack calls are done to maintain the scope information, which I stole from the loadXML method in the AdapterFactory class.

However, let’s take a moment to reflect on this. Using the Scala XML library in this way is not really good. Even if it’s possible to do it this way, I’ve not seen it described as a supported way of using it and therefore it should be done only after considering that the next release of Scala might remove this possibility.

[Update 2008-07-02: Burak Emir kindly added a comment; "Don't worry, the SAX factory adapter is not going to go away." - good to know!]

That said – let’s consider this an exploration of possibilities which could potentially lead to an update of the Scala XML API to allow a DOM to be used as a source instead…!

A quick test gives the following result.

Test data is a 6897 bytes XML file containing 118 elements with some 4 different namespaces.

I ran each test in 1000 iterations, with a full garbage collection before the first iteration. For every 100 iterations I printed the delta of the free memory and then timed the time for the complete 1000 iterations.

Char array: 100 iterations use around 28 MB, full test: 1414 ms
SAX: 100 iterations use around 18 MB, full test: 970 ms

So, in conclusion, not overwhelming difference, but around 1/3 faster and 1/3 less memory consumed. Can we do better? I’m not sure. :)

The next step is to do Scala XML to DOM… This could be more interesting. I see two options:

1. Implement the DOM API wrapping Scala XML
2. Generate SAX events based on the Scala XML and use that to build a DOM

Option 1 would be more efficient – but the DOM API isn’t fun implementing. Option 2 would be much simpler, but probably would be less efficient and require more allocations. Gotta think about this one…

/M