7 replies [Last post]
Alan
Offline
Enthusiast
Last seen: 17 years 5 weeks ago
Joined: 2003-10-20
Posts: 72
Points: 0

Hi there,

I was wondering if anyone could tell me how to do something with a robots.txt. Basically all I want to know is how to stop ANY Search Engine from indexing ANY image file (jpeg, gif) on my website. What code do you need to put in your robots.txt file, to do this?

Alan

Tags:
seb
Offline
Enthusiast
Last seen: 13 years 43 weeks ago
Joined: 2003-09-20
Posts: 208
Points: 0

Robots.txt - Don't Index JPEGs

So long as you're a neat developer this should be easy:

User-agent: *
Disallow: /images/

if, on the other hand, you have images scattered all over the place....

User-agent: googlebot
Disallow: *.jpg
Disallow: *.jpeg
Disallow: *.gif

But that only works for google.

Alan
Offline
Enthusiast
Last seen: 17 years 5 weeks ago
Joined: 2003-10-20
Posts: 72
Points: 0

Robots.txt - Don't Index JPEGs

Thanks for your reply Seb,

The code I'm using now is:

User-agent: googlebot
Disallow: *.gif
Disallow: *.jpg

Do you know how to also stop other Search Engines, such as Yahoo! and AltaVista, from indexing images on your website? Are their other User-agent names for these Search Engines? If so, do you know where I can find them?

seb
Offline
Enthusiast
Last seen: 13 years 43 weeks ago
Joined: 2003-09-20
Posts: 208
Points: 0

Robots.txt - Don't Index JPEGs

Alan wrote:
Do you know how to also stop other Search Engines, such as Yahoo! and AltaVista, from indexing images on your website? Are their other User-agent names for these Search Engines? If so, do you know where I can find them?

AFAIK, google is the only webcrawler that supports the wildcard notation. Trying to use it with any other user agent may cause them to ignore your entire robots.txt in the worst case.

Why don't you just ensure that all your images are in one subdirectory, and use my first suggestion.

Alan
Offline
Enthusiast
Last seen: 17 years 5 weeks ago
Joined: 2003-10-20
Posts: 72
Points: 0

Robots.txt - Don't Index JPEGs

Hi Seb,

Yes, your first suggestion, putting all the images in one folder, makes sense. Its just, I'd have to make a lot of changes to my existing site to do that. I'll keep it in mind though, for next time I build a website. So, thanks.

seb
Offline
Enthusiast
Last seen: 13 years 43 weeks ago
Joined: 2003-09-20
Posts: 208
Points: 0

Robots.txt - Don't Index JPEGs

Well you could of course specify multiple folders if it's the case that you have images in x different folders. But if your site is even more horribly messy or dynamic than that then the google filter is the best option I know of.

Alan
Offline
Enthusiast
Last seen: 17 years 5 weeks ago
Joined: 2003-10-20
Posts: 72
Points: 0

Robots.txt - Don't Index JPEGs

Hi Seb,

I have just one last question about robots.txt. If I put the following code in my robots.txt file, would it stop all search bots (including googlebot) from indexing any files (JPEGs and GIFs) within these folders?

User-agent: *
Disallow: /photos1/
Disallow: /photos2/
Disallow: /photos3/
Disallow: /iphotos4/
Disallow: /photos5/

seb
Offline
Enthusiast
Last seen: 13 years 43 weeks ago
Joined: 2003-09-20
Posts: 208
Points: 0

Robots.txt - Don't Index JPEGs

yes