There is no easy solution to this problem, and we don't pretend to offer one. What we do offer is a very simple process that will guarantee that once a problem is identified, a compiler developer will be able to reproduce the problem and show that a proposed fix both solves the original problem and does not cause others. An automated regression testing system like Jacks gives the compiler developer peace of mind in knowing that a change will not break anything, and it makes the user happy because the problem will get fixed and stay fixed.
The only catch is that the effectiveness of a regression testing system depends on the scope and completeness of the tests. If the regression system does not test everything, then you can't really be sure that everything works. This is why we need your help. With enough community involvement, we can reach a critical mass of tests that will make Java compiler development significantly easier.
// File SynchronizedInterface.java
public synchronized interface SynchronizedInterface {}
% jikes SynchronizedInterface.java
Found 1 semantic error compiling "SynchronizedInterface.java":
3. public synchronized interface SynchronizedInterface {}
<---------->
*** Error: synchronized is not a valid interface modifier.
% javac SynchronizedInterface.java
jacks/tests
subdirectory. Within that subdirectory, there are additional
top level directories that contain related sets of tests.
For example, the jacks/tests/jls
top level directory
stores all of the compiler tests that can be mapped to
a JLS section. The structure of subdirectories within
this top level directory mimic the chapter layout of the
JLS. One would find that all tests related to chapter 7
of the JLS would be found in the subdirectory
jacks/tests/jls/packages
.
Take a look at the Test Index for the most up to date JLS to Jacks directory mappings.
The test case is covered by section 9.1.1 of the JLS. Chapter 9 is titled "Interfaces", section 9.1 is titled "Interface Declarations", and section 9.1.1 is titled "Interface Modifiers", so we would expect to put our test in a directory named:
tests/jls/interfaces/interface-declarations/interface-modifiers
tests.tcl
.
If a tests.tcl
file does not exist already, don't worry,
just create the regression test and save it with the file name tests.tcl.
And now for the real fun, the writing of the test. The format of
a regression test in the tcltest framework is as follows:
tcltest::test NAME DESCRIPTION {
COMMANDS
} EXPECTED_RESULT
Don't worry if any of that looks confusing at first. It is really simple.
Keep in mind the Jacks convention for determining the name of a given
test case. This naming convention is required so that tests case
documentation can be generated automatically.
The NAME
of the regression test should be the JLS
section number that corresponds to the directory where the test
case is located, followed by a dash and a unique test number.
This test falls under section 9.1.1 of the JLS and it is the
first test in section 9.1.1, so the NAME
is 9.1.1-1.
The DESCRIPTION
can be anything you want it to be. The
DESCRIPTION
is just a user provided description of the
test, it is not actually used by the system. The only thing to note
is that {} characters should be used around the description string.
The COMMANDS
section could contain any Tcl commands, but
most of the time you will just want to use the saveas
and
compile
methods supplied as part of the Jacks framework.
The saveas
command takes two arguments, the file name
and the data to be saved in the file, and returns the name of the
file that was just saved. Like so:
The
saveas SynchronizedInterface.java {public synchronized interface SynchronizedInterface {}}
compile
command takes a number of command
line arguments and passes them to the actual java compiler.
The compile
command will return
PASS
, FAIL
, or WARN
,
to indicate the exit status of the compiler.
The EXPECTED_RESULT
is where the result one expects
from the compile
command should be placed.
In our interface example, we expect the compile to fail. Wrapping all of this together, one could write the test case like so:
One could even combine the invocations of saveas and compile
to simplify things a bit. Remember that the saveas
command returns the name of the saved file, which is then passed to
the compile command.
tcltest::test 9.1.1-1 {should generate error on synchronized interface} {
saveas SynchronizedInterface.java \
{synchronized interface SynchronizedInterface {}}
compile SynchronizedInterface.java
} FAIL
tcltest::test 9.1.1-1 {should generate error on synchronized interface} {
compile [saveas SynchronizedInterface.java \
{synchronized interface SynchronizedInterface {}}]
} FAIL
After saving the tests.tcl file, the regression test can now be run in the jacks framework.
% cd tests/jls/interfaces/interface-declarations/interface-modifiers
% jacks javac
Testing javac
Working directory is /usr/local/project/jacks/tests/jls/interfaces/interface-declarations/i
nterface-modifiers
compile = /usr/local/jdk118_v1/bin/javac ...
run = /usr/local/jdk118_v1/bin/java ...
tests/jls/interfaces/interface-declarations/interface-modifiers
==== 9.1.1-2 adding the public keyword to an interface declared
as public triggers a bug in the javac shipped with jdk 1.1 FAILED
==== Contents of test case:
compile [saveas PublicSynchronizedInterface.java "public synchronized interface Public
SynchronizedInterface {}"]
---- Result was:
PASS
---- Result should have been:
FAIL
==== 9.1.1-2 FAILED
javac: Total 1 Passed 0 Skipped 0 Failed 1
You probably will not care about most of this output, just note that the test ran and it failed with javac. Now lets run the same test with the jikes compiler.
% jacks jikes
Testing jikes
Working directory is /usr/local/project/jacks/tests/jls/interfaces/interface-declarations/i
nterface-modifiers
CLASSPATH = /usr/local/jdk118_v1/lib/classes.zip
compile = /home/mo/project/build/jikes/src/jikes ...
run = /usr/local/jdk118_v1/bin/java ...
tests/jls/interfaces/interface-declarations/interface-modifiers
jikes: Total 1 Passed 1 Skipped 0 Failed 0
Thats all there is to it! The test case has now been integrated into
the Jacks regression testing system and it has been double checked to
ensure that it is working as expected. Take a moment to reflect
on how easy it easy it was to add the test and then run it under
multiple compiler environments. All that is left to do is send this
new test case in to the jacks mailing list. You need to subscribe
to the list named "jacks" as described on the
Jikes Mailing List Page.
Don't forget to look at the Test Index. It provides a nice web interface to the existing set of tests.