I have a question about robots.txt files. I understand that you can use a robots text file to stop Search Engines from indexing specific folders and images on your webspace. For example:
But what I'd like to know is, can Search Engines really index files that aren't linked to your website? What I mean by that is, suppose you've got a "secret" folder on your webspace, in which you put personal files that aren't linked or used by your website.... will a Seach Engine still be able to find and index this folder and all the files in it?
Search Engines Indexing Folders
If you use toolbars like the Google toolbar when you visit that page, you might see the spider hit that page. But if the page has been excluded through the robots.txt file, then unless it's a rogue spider, the file should not be indexed.