script to automate updates to robots.txt #11199

Rewritten the script to work with a 'robots.txt' template file and we
generate the 'robots.txt' from the template when we execute the script.
It's found to be a better alternative to editing the 'robots.txt' file
in-place.

scripts/docs.sh

@@ -27,6 +27,7 @@
 #------------------------------------------------------------------------------
 
 set -e
+scripts/update_robotstxt.sh
 cd docs
 pip3 install -r requirements.txt
 sphinx-build -nW -b html -d _build/doctrees . _build/html