help me create the web crawler that is described ,
January 24, 2010 at 3:58 AM
please if possible send me some info in the next e-mail address about how to build a web crawler that the only thing that will do is to recieve a string(an html address: included those that ends in: .html, .htm, .asp, .aspx, .php , or slash character). second an address of destination in the local system that the java application runs.
the pages must be downloaded and stored in the destination file. then all the links that the specific pages contains must be extracted. from all of those links we are interesting only for those that belong to the same host. for each one of these pages the proccess must be repeated.
in the end all the downloaded pages must be stored in the destination folder . thusly user could be able to see the continent of those pages in the destination folder on his pc, in the hard disk . attention in order this to be accomplished the addresses that might be in the pages that are downloaded must be transformed accordingly.
please if anyone could help me on that project as soon as possible do send me info or the program in the next mail address: stavfot@hotmail.com. thanks in advance
View All Comments
| View Tutorial