The next version of Acunetix Web Vulnerability Scanner (version 7), will contain a much more improved HTTP stack.   While testing, we wanted to test the new HTTP stack on as many sites as possible to make sure we didn’t introduce any bugs.

Alexa, a web information company, maintains a CSV file containing the top 1,000,000 sites on the internet.  We used this database to test the new HTTP Stack.  While testing, we also thought that it might be useful to gather all the information and come up with some statistics. After all, who does not like statistics and graphs?

From this list, 951990 records were valid and we got a response. The other records were removed either because they were invalid (some entries from the Alexa list might be invalid), or because the hostname could not be resolved, or they were unresponsive during the testing phase. Also, some other entries were removed because there were multiple domains redirecting to the same domain that was already in the database.  These results were stored in a Microsoft SQL Server 2008 database. In the database, we stored the complete HTTP headers and the body.  The information was split in several tables like headers, links, metas and responses, so to be queried more easily. We’ve also enabled Full-Text Search on the database. In the end, the database size was around 50Gb.

Top Web Servers

To start off with, I wanted to see which are the most popular web servers running the top 1m sites on the internet.  I’ve used the information returned in the ‘Server’ header to compute this information.  Not all sites do return a Server header, and as a matter of a fact, 11,100 (1.16 %) sites didn’t return a Server header.  This was quite surprising, as I was expecting a much higher percentage of websites which do not report the running web server version and make.

Read more > www.acunetix.com

To create link towards this article on your website,
copy and paste the text below in your page.

<style type="text/css"><!--.quote {width:350px; padding: 6px; border: solid 1px #456B8F; font: 10px helvetica, verdana, sans-serif; color: #222222; background-color: #ffffff}.quote a {font: 13px arial, serif; color: #003399; text-decoration: underline}.quote a:hover {color: #FF9900; }//--></style><div style="float:left;padding:5px;"><img src="http://www.acunetix.com/blog/wp-content/uploads/2010/01/tld_top_50_distribution.png" width="120" alt="" /></div><div class="quote"><a href="http://www.onlyineurope.com/technology/statistics-from-the-top-1-000-000-websites" target="_blank">Statistics from the top 1,000,000 websites</a><br />Friday, 15 January 2010<br /> The next version of Acunetix Web Vulnerability Scanner (version 7), will contain a much more improved HTTP stack.   While testing, we wanted to test the new HTTP stack on as many sites as...<div align="right" style="width:350px"><p style="text-align:right;">© 2010 - <a href="http://www.onlyineurope.com/" target="_blank">onlyineurope</a></p></div></div>

Preview :

Statistics from the top 1,000,000 websites
Friday, 15 January 2010
The next version of Acunetix Web Vulnerability Scanner (version 7), will contain a much more improved HTTP stack.   While testing, we wanted to test the new HTTP stack on as many sites as...

© 2010 - onlyineurope

Powered by QuoteThis © 2008