You are right - 'su' is short for 'superuser' and gives you root
privileges within the terminal. However, Ubuntu does not have a
'proper' root and relies on 'sudo' instead (which, IIRC, is short for
'superuser do' and gives you root privileges on the specified program only).

When you typed in 'su', did the prompt change from '...$' to '...#'?
If not, then you are not in superuser mode. I suspect that you put in
your own password when prompted, which won't work; you need to enter
root's password. (To set up a root password, click on
System>Administration>Users and Groups, enter your password and then
highlight root in the dialog that appears. Click on Properties and it
should be obvious where you need to set the password. However, I don't
actually recommend doing this.)

As to sudo not working, did you type in the command exactly as given?
This is an Ubuntu default and is set up so that all users have sudo
rights. 'sudo nautilus' should be a single command
(sudo nautilus). The only way I can replicate your error is by
making a typo.