You can use the builtin webs spider and the Dmoz plugin to spider from the dmoz.org which is the largest

directory database all search engines even google use its database.

If you want to build your own spider, then you have to do programming, see the cgi file :

PendingMgr.pm

this pice of code shows how your links should be stored in the file name submissions.txt

under the folder data/submissions

Code: Perl    Select All    Expand All

sub Split_Spidered_Link{ my ($Link) = @_; my (%URL);        undef %URL;     (   $URL{ID},        $URL{Cat_ID},       $URL{Title},       $URL{URL},       $URL{Description},       $URL{Keywords},       $URL{User_ID},       $URL{Language},       $URL{Location},       $URL{Email},       $URL{FName},       $URL{LName},       $URL{Company},       $URL{Job_Title},       $URL{Street},       $URL{City},       $URL{State},       $URL{Zip},       $URL{Country},       $URL{Phone},       $URL{Fax},       $URL{Website},       $URL{URL1},       $URL{URL2},       $URL{URL_Image},       $URL{URL_Icon},       $URL{Rank},       $URL{Score},       $URL{Price},       $URL{Extra1},       $URL{Extra2},       $URL{Extra3},       $URL{Extra4},       $URL{Extra5},       $URL{Extra6},       $URL{Extra7},       $URL{Extra8},       $URL{Extra9},       $URL{Extra10},       $URL{Extra11},       $URL{Extra12},       $URL{Extra13},       $URL{Extra14},       $URL{Extra15},       $URL{Extra16},       $URL{Extra17},       $URL{Extra18},       $URL{Extra19},       $URL{Extra20}    ) = split(/\~\|\|\~/, $Link);

Mewsoft Support
www.mewsoft.com