Apt-get will not work because you did not install it with apt. When you click on
a .deb file it launches Gdebi which is the graphical interface for dpkg.

Try to use gdebi or dpkg.
sudo dpkg --remove <name of package>

There is a lesson to be learned here. Always use the repositories unless you
have no alternative. Adobe reader is available in the repositories and the open
source readers such as Evince are usually faster anyway.