Make sure to remove VSTS agent using
config.cmd remove command for which, PAT is required, you don’t need to use original PAT, you can apply a new PAT with Agent Pools (read, manage) scope and use it to remove agent.
In case you try it without the PAT:
Try deleting the agent from the server:
Deleting agent service in local system through sc command if it is running as service:
sc delete [service name].
After that, you can delete the agent files.