Java Article

BeanShell, Rhino and Java -- Performance Comparison

Abstract

This article compares the execution speed of a simple program under three different Java-based execution environments: BeanShell, JavaScript engine Rhino and Java. The performance test program consists of two parts: genration of an array of 20 character long random strings and arranging the strings in lexicographycally descending order using bubble sort. The performance numbers, though not unexpected, are interesting to anyone who wants to leverage the power of dynamic languages without compromising too much on performance.

8:00pm, 31 Aug 2005: JavaLobby listed my blog entry in their newsletter's "pick of the week" section today, resulting in a fair amount of traffic. I also got a jython version of my program from Jim Adrig, which I have ran on my box and updated the article with the observed numbers. Jim's program used Java classes java.util.Random and java.util.Date (perhaps for a fair comparison) and hence couldn't be run as a python program. I modified it to use python native modules random and time and executed under both python and jython. Look at the table for the observed figures. The measurements were made with Python-2.4.1 and Jython-2.1 on the same machine and JVM.

BTW, if you want me to include performance figures for your favorite scripting environment then just send me the bsort program for that environment.

Why Care?

I have been using BeanShell for exploratory programming and developing simple utilities for quite sometime. Lately, some of these utilities have become popular with my colleagues and are finding much wider use. Given the nature of these utilities, I never worried about their runtime execution speed, but this article on Java scripting reminded me that there are better alternatives available, at least with respect to raw execution speed. This got me thinking. Although most of my current programs are not performance sensitive, what I plan to write in next few months would perhaps benefit from a well performing execution environment.

This realization coincided with my frustration with BeanShell on a number of other, somewhat unrelated fronts: (a) not able to write derivable scripted objects; (b) not able to construct lists and hashes with a much more compact syntax ( basically, able to write { a => avalue, b => bvalue } than writing "h = new HashMap(); h.put(a, avalue); h.put(b, bvalue);"); (c) inability to write methods that take variable number of arguments; and so on.

I am sure, some of these limitations will go away with time. But I seem to be losing patience, especially when alternatives offer most of these. And that got me to look at the alternatives. The first one to excite me is Rhino - JavaScript for Java. Granted that JavaScript is not Java and Rhino doesn't support use of compiled Java classes as seamlessly as BeanShell, but it does have a number of other advantages.

The above mentioned article on Java scripting talks about some performance tests, but these tests are not representative of the kind of programs that I write. My programs usually do a lot of string processing and random number generation. So I decided to compare Rhino and BeanShell with such a program.

Test Program

This program generates a number of strings, places them in an array and then sorts them in-pace using bubble sort. It has two distinct phases: genration of 20 character long random strings and arranging the strings in lexicographycally descending order using bubble sort algorithm. Execution time is reported for both these phases, generation and sorting, separately. The number of strings can be changed for a given run by changing a variable named num in the source file. For measurements, I used three different values for this variable: 1000, 5000 and 10000.

I wrote four versions of this program:

1. js/bsort1.js: A JavaScript program that uses native JavaScript String type to hold generated strings and Math.random() function to generate random characters for strings.
2. js/bsort2.js: A JavaScript program that uses Java class java.lang.String to hold generated strings and class java.util.Random as random number generator.
3. bsh/bsort.bsh: A BeanShell program.
4. java/bsort.java: A Java program. I added this one just a benchmark.
5. jython/bsort.py: A Jython program sent by Jim Adrig. It uses a couple of Java classes.
6. python/bsort.py: A pure python program based on Jim Adrig's version.

Note: You can download the source files by clicking on the above links. You will find the execution instructions as part of the source comments. For executing JavaScript programs you will need js.jar, which is part of the Rhino download. Similarly, for executing the BeanShell program you will need bsh-2.0b1.jar (or the latest).

I felt it important to write two versions for Rhino -- one that uses native JavaScript types and functions and another one that uses corresponding Java classes and functions. This turned out to be a wise thing to do, for the performance numbers for these two versions are quite different and throw light on an aspect that I hadn't considered before -- use of Java objects from a JavaScript program incurs an additional cost.

In fact, I ran the JavaScript programs in two different ways: once under the interpreted mode and then after compiling with the JavaScript compiler. Afterall, availability of a compiler is considered a plus for Rhino and hence its effectiveness must be evaluated.

Performance Numbers

Enough talk. Let us now look at the numbers.

No. of strings	num: 1000		num: 5000		num: 10000
Program	gen.	sort	gen.	sort	gen.	sort
bsh/bsort (BeanShell, interpreted)	688	15500	2515	402000	4780	1623000
js/bsort1 (intepreted)	140	450	344	11050	600	46330
js/bsort1 (compiled)	140	440	344	10940	580	46020
js/bsort2 (intepreted)	440	1660	1360	41580	2516	172000
js/bsort2 (compiled)	440	1625	1345	40390	2470	168000
js/bsort (java, compiled)	32	47	63	1093	109	4625
jython/bsort (jython-2.1)	172	328	391	8860	688	44360
python/bsort (python-2.4.1)	204	370	1020	9200	2040	37100
python/bsort (jython-2.1)	515	300	1500	8400	2800	43400

These numbers are for elapsed time in milli seconds for string generation and bubble sort, shown separately, my Windows XP machine with 2.93 MHz Pentium 4, 512 MB RAM, JDK 1.5.0, BeanShell-2.0b2 and Rhino-1.6R1.

Conclusions

For those who don't like looking at raw numbers, here are my conclusions:

1. Rhino is significantly faster than BeanShell and the difference become more pronounced as the data size increases.
2. There doesn't seem to be much difference in speed between interpreted and compiled execution of JavaScript programs. This matches with the observation in the above-mentioned article regarding speed of compiled JavaScript programs. So, there is not much benefit of a compiler, after all.
3. Java classes are a second class citizen in a JavaScript program (compared to native types). Given a choice, choose a native type.
4. Even pure JavaScript is no match for a compiled Java program, which is almost 10 times faster.

The performance advantage of Rhino alone may not sufficient for me to switch, for most of my programs spend less than 10 percent of the total execution time running BeanShell scripts (rest is spent running Java byte code from existing class libraries), but it certainly is going to be a factor.

© 2005 by Pankaj Kumar. All Rights Reserved. You may use the content of this article, including the source code, in your programs and documentation. In general, you do not need to contact me for permission unless you are reproducing a significant portion of the article or code for commercial purposes. For example, writing a program that uses the code for personal or company use does not require permission. Republishing this article, even with proper attribution, does require permission. Similarly, incorporating the code in a program that you sell for profit requires permission.

You are welcome to send me comments, suggestions or bug fixes. I will look into those as and when time permits and may even publish the good ones on this page.

You can contact me by sending e-mail to pankaj.kumar@gmail.com.