Let us take an example, where 1234 is the SSH port being used by the host (which is a different SSH port), now if I want to add a remote origin server, I would have to do this:
git remote add origin ssh://user@host:1234/srv/git/example.git
Here, 1234 is the ssh port being used
Also, use absolute path, not a relative path to user home directory