It took a while but I got it to work by doing this:
https://wiki.archlinux.org/index.php/PulseAudio/Examples#When_everything_else_se\
ems_to_fail


When everything else seems to fail

The following is a quickfix and NOT a permanent solution

On the Server:

$ paprefs

Go to Network Access -> Enable access to local sound devices (Also check
both 'Allow discover' and 'Don't require authentication').

On the Client:

$ export PULSE_SERVER=server.ip && mplayer test.mp3