An example of robots.txt

User-agent: *
Crawl-delay: 31
Disallow: /folder1
Disallow: /folder2
Disallow: /*.gif$
Disallow: /*.Gif$
Disallow: /*.GIF$
Disallow: /*.jpg$
Disallow: /*.Jpg$
Disallow: /*.JPG$
Disallow: /*.jpeg$
Disallow: /*.Jpeg$
Disallow: /*.JPEG$
Disallow: /*.pdf$
Disallow: /*.Pdf$
Disallow: /*.PDF$
Disallow: /*.zip$
Disallow: /*.Zip$
Disallow: /*.ZIP$

This will suggest web crawlers to not index folder1, folder2, and GIF, JPEG, PDF, and ZIP files.

References

• http://www.robotstxt.org/
• Robots exclusion standard