the software you are trying to make to help
beginners making small programs.
Well, I have made my own using VB 6 and I have faced the same problem concerning
the compilation and running the program. But I had an idea of making the
compilation process and running process in two different steps. For each one to
be made, I make a batch file (.bat) to compile and run the class.
Well, I hope this would help you.

I faced the same problem months ago, but i lost the code! But i
can say, yes you got it: you must save the program the user have
entered in the textarea to a file .java with the right name (the class
you are defining: look! it's java  so file name must match the class
name. Case sensitive, uh ? ) Then you must lunch the javac.exe
application as you were in a dos console. You can use Process class
maybe. Get the output to check if compilation process did right. And
when you must execute  just launch as before the process java.exe with
the name of the file mentioned above (without .class extension, for
sure!). That's all. Problems may rise with the classpath. I think that
if you search on the net you can find the docs/tutorials i found times
ago.