I am running Edgy on a 64bit system with 4GB ram. Even with such
high ram capacity when reading and plotting grids on Matlab the memory
goes to the max. What I can suggest you do is to place the applet System
Monitor on your panel and see how the memory varies as you run
several applications. It may very well be that you are occupying something
that is choking your machine. When I accidentally choked mine a couple
of days ago, it went completely frozen and after a while it automatically
killed Matlab and memory went minimal. I suspect that Matlab (being mostly
developed for Windows) has a bug that doesn't know how to clear its memory.
Other than that my system is working perfectly.

Otherwise, check that all your hardware is compatible with each other. Failure
to comply would be responsible for such behaviour.