Request quote

script to create a php site crawl robot easily

Posted on: August 30th, 2012 by Mohammed Aqeel 1 Comment

The following tutorial is a simple example of how to create a site crawl robot with php. This can be used for search engines, creating caches of your sites at regular intervals etc. The crawler uses php’s CURL library to fetch the URL contents. It will then look for an anchor tag inside the page contents and will crawl those pages also. Here is the script.

If you have any other easier or better way of doing it, dont forget to post it below in the comments section.

Tags: , , , ,

One Response

  1. iMi says:

    It fails if a site uses absolute links.