Web Crawler

Web crawler is small piece of code which is used by search engine for inventory the web sites. Crawler collect the information about sites and report back to the search engine.

When search engine find a good website, it can include the site in database for search engine result pages for given search or keyword phrase.

Web crawler neglects the script. So your site making easy to crawl by web crawler as possible as by limiting the amount of scripting navigation to per page. Web crawler cannot read Flash component and pages.

If you want to crawl you pages to web crawler, you should be make your pages web page crawler friendly. Your web page content should be original, easy to understand and free to scripting.