1 ) If u want to write a Mail server  by java there are
some ways :
a) Socekt Programming
b) using Java Mail API
c) ...
in first item , u must know somethings about SMTP or
ESMTP and POP3 or IMAP
and then implements them on ur server.
but sencond item is better for u.
2 ) JSP Technology is not suitable for writing Mail
Server(it is better i tell that is not possible)
but u can use it for N-tier Mail web based application
in bussines layer.
HTML page <----> JSP page <-----> Mail Server
User Interface Layer <-----> Bussines Layer <------>
Data Store Layer