The following tutorial is a simple example of how to create a site crawl robot with php. This can be used for search engines, creating caches of your sites at regular intervals etc. The crawler uses php’s CURL library to fetch the URL contents. It will then look for an anchor tag inside the page contents and will crawl those pages also. Here is the script.
If you have any other easier or better way of doing it, dont forget to post it below in the comments section.Tags: create a php crawler, php crawl robot, php crawler script, php robot script, php site robot