1 reply [Last post]
Alan
Offline
Enthusiast
Last seen: 17 years 5 weeks ago
Joined: 2003-10-20
Posts: 72
Points: 0

Hi there,

I have a question about robots.txt files. I understand that you can use a robots text file to stop Search Engines from indexing specific folders and images on your webspace. For example:

User-agent: *
Disallow: /photos/
Disallow: /thumbnails/

But what I'd like to know is, can Search Engines really index files that aren't linked to your website? What I mean by that is, suppose you've got a "secret" folder on your webspace, in which you put personal files that aren't linked or used by your website.... will a Seach Engine still be able to find and index this folder and all the files in it?

Tags:
polarmate
polarmate's picture
Offline
newbie
Last seen: 16 years 6 weeks ago
Timezone: GMT-5
Joined: 2004-10-13
Posts: 1
Points: 0

Search Engines Indexing Folders

If you use toolbars like the Google toolbar when you visit that page, you might see the spider hit that page. But if the page has been excluded through the robots.txt file, then unless it's a rogue spider, the file should not be indexed.