Consider moving robots.txt checks into this module

[smulube]

Currently we check this in the consuming application, but it might make sense to move that functionality into this module