| Why a robot defender went necessary: |
BTTR is hosted on a single 400MHz computer, handling all our services. This one machine is used by thousands of people each day, so this CPU never gets bored.
Since this box runs Linux, it can stand the load well, and the site is still pretty fast. It is because humans surfing a site click through it in a very server friendly way and frequency.
Whenever the server is slow then someone is hammering our site with a download util, fetching pages and files in a frequency which slows down our server horribly! These tools try and hog all the CPU and bandwidth for themselves. This is not very social!
But the main prob is, most people don't know what they do when they start a Download Accelerator or MSIE's Make available for offline usage function with such a complex site.
BTTR is just too big to be downloaded totally, accept that. Nobody in the team even has the full collection at home! It's just too much. Besides that, the site changes hourly and you just can't fetch the site in an hour, you get invalid parts.
We were nearly shutdown by our former host because of the many robot attacks we get per day and the CPU that consumed and the problems that made. Asking people through the newspage to stop this behaviour was of no help, they just continued to hammer down our work.
That was when the robot defender was coded as a last resort to avoid a site being shutdown due to egoistic usage. And it really was not our interest to code defense weapons instead of something more useful and topic related... |
|