I don't know what the problem is. I use usb keys with installed Linux all of the
time. I use unetbootin almost daily to make bootable usb keys from which to
install Linux. I have also used pendrivelinux and other methods.

There are a few things to remember. The pendrivelinux approach is easier to do
it from Windows if that is the environment that you are most familiar with. If
you use unetbootin, you need to have the usb empty and mounted before you open
the programme of your key will not appear in the device list.. If you want
persistence then need to follow those instructions separately as a post
installation step.

Your BIOS settings come into play. If you use grub, it needs to be written to
the key itself and not a hard drive. Why you need to disconnect your hard drive
is because grub is naming its devices differently. I am thinking that you don't
have a boot manager and when your BIOS has your drives enabled this changes the
order and the grub settings don't work.

An explanation of what is happening:
Let's say you have two hard drives. One is sda and the other is sdb. The usb key
then becomes sdc. When you write grub to the usb key it should be saved in grub
as hd(0,0) as grub uses an ordinal system with zero being the first drive in the
chain and the second zero is the first partition on that drive. This will be
written into grub and can only be changed by editing the file. A boot manager
does what you are physically doing when you disconnect your drives. Grub will
work and Ubuntu will load because you will have the hd(0,0) situation. However
when the drives are attached and you don't have a boot manager sda becomes
(hd0,0), sdb becomes (hd1,0) and the usb key will be had(2,0) and grub will fail
to load because the settings are pointing to hd(0,0) and it can't find the
kernel on your first hard drive because it isn't there, but on the key.

To remedy the situation you need a boot manager. My HP BIOS comes with one. I
just press ESC and then I can choose to boot from the usb key instead of my hard
drives. Another option would be to edit grub on your usb key to change it to be
hd(3,0), but this would not be useful because it would fail if you tried to use
it on a different computer. That is why hd(0,0) is so important. It is neutral,
provided you can choose to boot from the usb key before the hard drives. You can
get third party boot managers. There is even and open source one. I am not sure
if they would work because it is written to the first sector of your first hard
drive. It certainly would not help if you put your key into another computer
without a boot manager. You would be stuck once again.

You can try changing the boot order in your BIOS, so that the usbkey is first in
order. I am not sure if this would work as both of my computers have a boot
manager and I have never had to worry about it not working. Perhaps someone who
has no boot manager and uses usb keys to install a distro can jump in.