Spider Sites

Overview

The spider sites tools will spider any number of websites you enter and grab its meta  title, description, and keywords tags besides searching the site homepage for valid email then submit this fetched site automatically to the system pending spidered database for manual approve later and review.

Spider Sites

Enter the website address you want to spider to your engine database in the text box. Please enter one URL or website address on each line and make sure to press enter at the end of each line.

It is very important not to enter many URL at once to spider since this may lead your server to time out. You may try between five to ten URLs at the same time, some sites may fail due to being offline or not available and some may respond slowly therefore this process may take long time. The spider will open a new window for displaying the progress of sites being processed.

Extract Web Page Links

This tools will extract all the links of a web pages that you enter its full URL and export these links to the spider so you can spider these sites.

For example if you want to spider all the sites on a any search engine like yahoo® category, just enter the URL of this category page on yahoo. The program will extract all the sites links on this category pages and export it to the spider so you can review and spider them.

This tools is very useful in bulk spidering pages and web directories to your search engine database.

Please enter only on URL.

The links extractor uses filters also to allow you filter only the links you want exactly.

Links must contain these keywords:
This will only extract the links that contain these keywords. If you do not want to filter by keywords, just leave this field empty.

Links must not contain these keywords:
This will NOT extract the links that contain these keywords. If you do not want to filter by keywords, just leave this field empty.

Exclude Local Links:
This options if checked will not extract the links that belongs to the same URL that is being extracted. This option is very useful to exclude the links that belongs to the same page you are spidering. For example, if you are spdering the website http://ww.mewsoft.com , and this option is enabled, then it will exclude all the index pages, pages that has URL starting with www.mewsoft.com , and so on. Normally these pages are not necessary and it will save you a lot of time that you filter these links.

Figure (1) Sites Spider.