The XMLUI and it's out-of-the-box themes do not come with a robots.txt file. It's also unclear where such globally static content can be placed so that it's accessible.
Recommendation: Globally static content (*.html or *.txt) should be placed in a /src/main/webapp/static/ directory, so that it is accessible to all XMLUI themes. So, the following should be distributed in XMLUI:
robots.txt <-- Default robots.txt based on DSpace recommendations
*.html <-- Other static HTML content may be placed here (but none would be there by default)
[Default themes are here, as they already are.]