Search engine robots spider your web site all of the time, so they can find your new and updated web pages for their search engines. This is a good thing. However, if you don’t have a robots.txt file, your server will report errors and the spiders may not understand what to crawl. There are many tutorials online regarding how to create these files and it is very easy to do. You could also save the line of text below into a text file, name it robots.txt and put it in your main file directory, in case you want them to spider everything.
User-agent: *
That's it! However, if you have some private folders and don't want the spiders to locate them, you'll need to do something slightly more complicated. Simply Google the solution and you can find what you need.
Subscribe to:
Post Comments (Atom)
No comments:
Post a Comment