They ask the NSA.
No, they use different systems to detect it. Its a lot to describe. They scan accounts for certain things, like how many times you have killed one mob a certain ammount of time, how many ores and herbs you have collected in a certain ammount of time, how many auctions you have up and stuff like that.
And of course they investigate player reports and take a look into the logfiles. If they see that your char was running against a tree for 8 hours they know you bot.
And they have some security features but there aren`t many informations around. Their tripwire system should be able to scan your computer for programms that are running in the background. But using stuff like that is not legal in every country, so I am not sure if they really use it. If they do, we all would already be banned.