Assuming you only really need to test  output from a desired set of
input, and only using CLI … you could probably just hack it together in
bash script in a few hours. If you really had to do it in java, you can
hijack a programs input  and output with ByteArrayInputStream and
ByteArrayOutputStream … it is pretty effective for that kind of testing.

The moment you want to do any kind of introspection on the class  file
(to see what it is really doing), or testing GUI components then … well
you’re not getting paid enough :)

So if all you really want to do is compare the desired output to the
actual output … not hard. If you want to compare the magical stuff
happening inside a class (include GUIs), then that just filled the too
hard basket.

Be warned though, students can break automatic testing like it is going
out of style, even when it works perfectly everywhere else but in the
automatic testing script. Also strict adherence to automatic output
testing doesn’t actually test what the programmer has actually written

Point in case in one of my programming assignments in first year
(program a linked list) got 80/100 even though he didn’t make a
linkedlist (he used the JFC one). His output was perfect and he followed
the coding standards properly. Of course that made his assignment about
5 hours of work, while every body else beat their heads against linked
lists for about 20 hours :) (I got 87% and I worked my arse off).