User-agent: SEMrushBot
Crawl-delay: 5

User-agent: YandexBot
Crawl-delay: 5

User-agent: Gigabot
Crawl-delay: 5

User-agent: MSNBot
Crawl-delay: 5

# Yahoo
User-agent: Slurp
Crawl-delay: 5 


User-agent: *
Disallow: /*?*pageSize=
Disallow: /*?*ordering=
Disallow: /*?*noindex=true
Disallow: /*/concepts/*