Sat, 2004-07-24 16:46
Hi there,
I was wondering if anyone could tell me how to do something with a robots.txt. Basically all I want to know is how to stop ANY Search Engine from indexing ANY image file (jpeg, gif) on my website. What code do you need to put in your robots.txt file, to do this?
Alan
Tue, 2004-07-27 16:40
#1
Robots.txt - Don't Index JPEGs
So long as you're a neat developer this should be easy:
User-agent: *
Disallow: /images/
if, on the other hand, you have images scattered all over the place....
User-agent: googlebot
Disallow: *.jpg
Disallow: *.jpeg
Disallow: *.gif
But that only works for google.
Tue, 2004-07-27 17:51
#2
Robots.txt - Don't Index JPEGs
Thanks for your reply Seb,
The code I'm using now is:
User-agent: googlebot
Disallow: *.gif
Disallow: *.jpg
Do you know how to also stop other Search Engines, such as Yahoo! and AltaVista, from indexing images on your website? Are their other User-agent names for these Search Engines? If so, do you know where I can find them?
Tue, 2004-07-27 18:26
#3
Robots.txt - Don't Index JPEGs
Alan wrote:
Do you know how to also stop other Search Engines, such as Yahoo! and AltaVista, from indexing images on your website? Are their other User-agent names for these Search Engines? If so, do you know where I can find them?
AFAIK, google is the only webcrawler that supports the wildcard notation. Trying to use it with any other user agent may cause them to ignore your entire robots.txt in the worst case.
Why don't you just ensure that all your images are in one subdirectory, and use my first suggestion.
Tue, 2004-07-27 18:35
#4
Robots.txt - Don't Index JPEGs
Hi Seb,
Yes, your first suggestion, putting all the images in one folder, makes sense. Its just, I'd have to make a lot of changes to my existing site to do that. I'll keep it in mind though, for next time I build a website. So, thanks.
Tue, 2004-07-27 21:19
#5
Robots.txt - Don't Index JPEGs
Well you could of course specify multiple folders if it's the case that you have images in x different folders. But if your site is even more horribly messy or dynamic than that then the google filter is the best option I know of.
Wed, 2004-07-28 15:28
#6
Robots.txt - Don't Index JPEGs
Hi Seb,
I have just one last question about robots.txt. If I put the following code in my robots.txt file, would it stop all search bots (including googlebot) from indexing any files (JPEGs and GIFs) within these folders?
User-agent: *
Disallow: /photos1/
Disallow: /photos2/
Disallow: /photos3/
Disallow: /iphotos4/
Disallow: /photos5/
Wed, 2004-07-28 15:41
#7
Robots.txt - Don't Index JPEGs
yes