Recently I have been working with some NGFW tools to automatically detect and block when someone is scraping, brute forcing or “load testing” your website. I quickly ran into a problem where none of the tools I use would allow me to quickly change user agents so I put together a couple of quick scripts that call one of 7500 valid user agents from this file.
First I went with the old standby of CURL which does the job but I was only able to do 10 requests in 4 seconds.
Here is what the output of curl.sh looks like:
That was not going to be fast enough for my testing needs so I switch to Apache Bench and am able to do 1,000 requests in 2 seconds. Which was what I need to do proper testing.
Here is what the output of ab.sh looks like:
All the scripts are in this GitHub Repo.
As always: Use these for good, not bad.
WAF Testing With Random User Agents.
