So where exactly is your Grub installed to? From the behaviour you
described, I'm guessing it is installed on the external hard drive. Which
means if you unplug the drive you are left with an unbootable system? I
don't know if this is a security feature I would want. There are two
options to make a bootable pen drive with Ubuntu, and maybe one of thos are
the way to go to boot to an external HD. Also, this site apears to include
the proper procedure to make a full install of Ubuntu to external HD.

www.pendrivelinux.com/.../

For all of these options, it is not a true dual boot. Your prefered OS is
based on the boot options in your BIOS. You will either set preference to
boot from HD or boot from USB. Selecting the non-default OS to boot is as
simple as entering the BIOS boot menu when you first turn your PC on.

btw, if you have the space on your board, you can get an internal HD at 40GB
from newegg.com for around 30-40 bucks... may be worth looking into. The
good thing about the external HD is that you could just take it with you and
use it from any PC though.