Forums Index > Mewsoft Products > Searchawy > spidering query
gofindit4me First Post Posted on: 10-23-11 05:14 PM
spidering query
Hi\ \ I have searchawy and dmoz crawler installed.\ \ Is there another way of re-spidering all the links in the database to make sure they are all live and to update the site's keywords? When you retrieve sites from Dmoz they just have very basic descriptions from the Dmoz category and so are not so easily found when searching from my site.\ \ Thanks\ \ Mark
mewsoft Reply #: 1 Posted on: 10-23-11 06:10 PM
Re: spidering query
No, this can not be done, you can not respider sites because if your users have accounts and have entered their own keywords for bidding these will be lost. We selected dmoz specially because it is very well reviewed my thounsands of editors which you or anyone else can not afford even google uses dmoz and almost all search engine useres dmoz for their database.

Mewsoft Support
www.mewsoft.com
gofindit4me Reply #: 2 Posted on: 10-23-11 09:19 PM
re spidering
OK. Then is there a way with this search engine of actually using the data retrieved by the DMOZ crawler? because most results come up without any keywords that allow the sites to be found using the search. Some dont even have a brief description.\ Each link needs to be spidered to get all of the relevant keywords for it to be found.\ With the cost per click advertisers, doesnt their position depend on the amount they bid for those keywords? so other sites having similar keywords wouldnt matter?
mewsoft Reply #: 3 Posted on: 10-23-11 11:25 PM
Re: spidering query

It seems you are talking about the web spider not the dmoz plugin, the dmoz plugin will get the links and their description as you see them on the dmoz directory, the Web spider if you used it it can get links yes without description if it did not find metag tags or some header text on the spidered websites.

As for indexing, these options you set in your admin center, you can set the engine to index sites Title, Desciption, Keywords, ifyou disabled these then it will not index any.


Mewsoft Support
www.mewsoft.com
gofindit4me Reply #: 4 Posted on: 10-24-11 01:53 PM
spidering
Right I see. That must be where I am going wrong.\ \ It is a brilliant script. The site I am building is looking great and offers bags more than my old one. Its just a bit complicated for a novice like me to work out - so many feattures!!\ \ So in the admin section I can set it to search descriptions, keywords and meta tags?\ \ I'll go and have another play ..\ \ Thanks
gofindit4me Reply #: 5 Posted on: 10-24-11 02:07 PM
Re: spidering query
OK, Still not got it sorted yet. I have selected search by title, description, keywords etc but still only seems to be going by title and description. I must be missing something.\ \ Another quick question. I cant find the answer in your documentation pages.\ \ in the directory manager there is a featured + and featured ++ option for the listings. What is this for??\ \ Also I have added a DJ's website to my directory. For some reason the search will find every other word he mentions in his description ecxept for DJ, it completely ignores it even though its not on my ignored words list. Any ideas? That is a major search keyword for him.\ \ regards\ \ Mark
gofindit4me Reply #: 6 Posted on: 10-24-11 05:25 PM
Re: spidering query
Ive spent a bit of time on this now and still cant get it to do what I want. Also the search methods listed in the 'tips' link dont seem to make any difference to the search results.\ \ Any suggestions?\ \ Thanks