1 reply [Last post]
Last seen: 17 years 30 weeks ago
Joined: 2003-10-20
Posts: 72
Points: 0

Hi there,

I have a question about robots.txt files. I understand that you can use a robots text file to stop Search Engines from indexing specific folders and images on your webspace. For example:

User-agent: *
Disallow: /photos/
Disallow: /thumbnails/

But what I'd like to know is, can Search Engines really index files that aren't linked to your website? What I mean by that is, suppose you've got a "secret" folder on your webspace, in which you put personal files that aren't linked or used by your website.... will a Seach Engine still be able to find and index this folder and all the files in it?

polarmate's picture
Last seen: 16 years 31 weeks ago
Timezone: GMT-5
Joined: 2004-10-13
Posts: 1
Points: 0

Search Engines Indexing Folders

If you use toolbars like the Google toolbar when you visit that page, you might see the spider hit that page. But if the page has been excluded through the robots.txt file, then unless it's a rogue spider, the file should not be indexed.