Adding the -v option to ssh when trying to log in will give a lot of debug information which might give a clue to exactly what the problem is, like for instance
debug1: Remote: No xauth program; cannot forward with spoofing.
which in my case installing xauth on the server fixed the issue.
I had to edit the sshd config file on the remote server to fix the issue. It worked on Ubuntu 16.04 Server:
$ sudo vim /etc/ssh/sshd_config
Set `X11UseLocalhost no`
Save the file.
$ sudo service sshd restart
$ exit
Now it works!
$ ssh -X user@remotehost
$ xclock