PhantomJS runs perfectly on the command-line, it is suitable as the first layer of smoke testing, whether as part of development workflow or in a continuous integration server.
Selenium targets multiple browsers and hence it is very useful to ensure cross-browser consistency and carry out extensive testing across different operating systems.
Running the UI testing only with PhantomJS will not yield the most test coverage.
But you can exercise some basic sanity checks before doing the in-depth tests using PhantomJS
The line between the two gets slightly blurred with the recent WebDriver support in the latest PhantomJS
Now using PhantomJS you can run tests and confirm that there are no serious threats then execute the same tests thoroughly using selenium setup.