For single file use this command
git rm --cached mylogfile.log
For single directory
git rm --cached -r mydirectory
As you are using
git rm mylogfile.log
use the above code that will help you to delete the file from a git repository without removing it in a local system.
--cached will remove from just the index but not allow you to delete the file on the local system.
For more information please go through the following tutorial to get more info about git: