You could use Selenium Grid so that the browser is opened on a completely different machine (or virtual machine) that you can then connect to via VNC or Remote Desktop Connection if you wanted to see the browser. Also, another option: if you run a Jenkins foreground process on that remote server, it can execute your test project on the desktop.
You can run Selenium 'headless' on Linux in XVFB. I've never tried doing this and doubt it's really worth the effort.
You can wrap Selenium RC in a Windows service. Except that permissions constraints on later versions of windows will probably prevent Selenium from accessing the desktop like Windows 2000 used to allow us to do.