You don't have to stop X if you install the drivers in the repositories, only if
you use Nvidia's. The changes will not take affect until you re-boot, but you
should be able to change the driver without problems. Ctrl+Alt+backspace kills X
and restarts it. But it is better to exit normally and re-boot.

The Nvidia one's won't allow you to change while X is running, but I have an
Nvidia card and it has never required any driver other than the restricted ones
in the repositories and I have full 3D effects.

If you insist on using Nvidia's drivers then you can try these:
As root (sudo -i)
service dm stop (to stop X)
telinit 3 (this changes the runlevel to init 3 which is textmode and will stay
in effect after you re-boot until the state is changed back)
telinit 5 (this changes the runlevel back to init 5 which is graphical mode)
service dm start (to restart X)

You can also edit the file manually. It is: /etc/inittab
First as root (sudo -i) change it from id:5:initdefault: to id:3:initdefault:
and what you need to get the graphical display back is to change the line:
id:3:initdefault: back to id:5:initdefault:
with 3 meaning text mode and 5 meaning graphical.