Well, I should have included a disclaimer that this developer tip requires the use of IIS 7 and the rewrite module. If you're unfamiliar with the rewrite module, you can find a brief introduction here.
I run into this problem when creating multiple websites within a single Kentico installation. I had a tough time finding the right solution for my application. I did find a solution in the Kentico forums that suggested adding a robots.txt file as a file type in the CMS Desk and using custom url extensions. More details of this solution can be found here. But, I also found that if your website isn't running under Integrated Mode, using the custom url extensions gets a bit messy. Therefore, I decided to look further into using the rewrite module.