feat(pages): standardize web crawler policy (#939)

* feat: standardize web crawler policies

* js sitemap gen core

* route package

* robots.txt modifications & deploy & test

* chore: issues for OCR

* remove sitemap.xml

* feat: Apply suggestion from @lizhengfeng101

---------

Co-authored-by: Kite <254839944+lizhengfeng101@users.noreply.github.com>