Comments on: Can I get list of all websites running in world? domain names? http://www.atozhosting.net/can-i-get-list-of-all-websites-running-in-world-domain-names.html A to Z Hosting Review Sat, 18 May 2013 09:40:02 +0000 hourly 1 http://wordpress.org/?v=3.4.1 By: rainofkayos http://www.atozhosting.net/can-i-get-list-of-all-websites-running-in-world-domain-names.html#comment-4160 rainofkayos Thu, 22 Mar 2012 15:27:47 +0000 http://www.atozhosting.net/can-i-get-list-of-all-websites-running-in-world-domain-names.html#comment-4160 well u could do u have the resources and cycles for this.... nslookup is a defunct tool,,, i would use the host command,, ur best bet would be to work out a way to recursively query dns from root down,, actually ur best bet would be to iterate through every logical IP address in a loop,,, doing reverse dns queries for each $ IP in that loop (friggin huge loop),,,u would then for intergrity (&& every site will not have a reverse entry so this is not even accurate) need to probably actively query a dns server or run a whois on the sites,,, u will need to format this data in some way like use awk or cut,, or most likely both and this will be such a wopping amount of data ur only chance of reviewing it logically would be to have it stored to a database,,, this all is actually very easy one thing to note is this type query on root dns will certainly bring government authority attention,,, another thing u can do is crawl the internet,,, or even crawl the crawlers,,, u can maybe write a perl script or bash that crawls googles search pages,,, and maybe uses some regex to produce websites allinurl:<autogeneratedword>.`com|net|etc|etc` ,, this even u will need to watch as u dont want to be crawling google to much either,,, BUT heres a really cool site i know about;; this is rumored to contain a copy of almost every site that ever existed IN THE PAST and no longer exists try it out http://www.archive.org/index.php well u could do u have the resources and cycles for this…. nslookup is a defunct tool,,, i would use the host command,, ur best bet would be to work out a way to recursively query dns from root down,, actually ur best bet would be to iterate through every logical IP address in a loop,,, doing reverse dns queries for each $ IP in that loop (friggin huge loop),,,u would then for intergrity (&& every site will not have a reverse entry so this is not even accurate) need to probably actively query a dns server or run a whois on the sites,,, u will need to format this data in some way like use awk or cut,, or most likely both and this will be such a wopping amount of data ur only chance of reviewing it logically would be to have it stored to a database,,, this all is actually very easy one thing to note is this type query on root dns will certainly bring government authority attention,,, another thing u can do is crawl the internet,,, or even crawl the crawlers,,, u can maybe write a perl script or bash that crawls googles search pages,,, and maybe uses some regex to produce websites allinurl:.`com|net|etc|etc` ,, this even u will need to watch as u dont want to be crawling google to much either,,, BUT heres a really cool site i know about;; this is rumored to contain a copy of almost every site that ever existed IN THE PAST and no longer exists try it out

http://www.archive.org/index.php

]]>