about summary refs log tree commit diff
path: root/public
diff options
context:
space:
mode:
authorEugen Rochko <eugen@zeonfederated.com>2019-02-13 21:28:18 +0100
committerGitHub <noreply@github.com>2019-02-13 21:28:18 +0100
commit80161f43510ad9316c60c9b50dd5c09c2dae4d54 (patch)
tree460e50d7b81024e30a7ca42c003690b80b6c0ace /public
parent7750416597e4d172b1d4751215aac2a50cf528e3 (diff)
Change robots.txt to exclude some URLs (#10037)
- Exclude static assets
- Exclude uploaded files
- Exclude alternate versions of the profile page
- Exclude media proxy URLs
Diffstat (limited to 'public')
-rw-r--r--public/robots.txt18
1 files changed, 13 insertions, 5 deletions
diff --git a/public/robots.txt b/public/robots.txt
index 3c9c7c01f..36afc85ef 100644
--- a/public/robots.txt
+++ b/public/robots.txt
@@ -1,5 +1,13 @@
-# See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
-#
-# To ban all spiders from the entire site uncomment the next two lines:
-# User-agent: *
-# Disallow: /
+User-Agent: *
+Disallow: /users/*/followers
+Disallow: /users/*/following
+Disallow: /@*/media
+Disallow: /@*/with_replies
+Disallow: /@*/tagged/*
+Disallow: /media_proxy/*
+Disallow: /emoji/*
+Disallow: /packs/*
+Disallow: /sounds/*
+Disallow: /system/*
+Disallow: /avatars/*
+Disallow: /headers/*