The biggest issue with 11.04 is that the Unity user-interface is not as polished
as it will be. At logon you can choose "Ubuntu classic" if you don't like it.

For people with new hardware, there are big improvements in (especially laptop)
audio and (especially USB) wireless hardware support. If your audio and wireless
work fine in 10.10, and you don't want to see how Unity works, there's no real
reason to change.

In another month, 11.10 will be available. Unity will be more mature in that
release.

I have never done a version upgrade, as a clean install is less likely to cause
problems. It's a lot easier if you have separate root and home partitions, which
is set up at install time.