Restrict URL

[kristofferwiklund]

I would be nice with some settings for entering URL patterns that should not send to Blippex. Because we have some development server with some low level security by obscurity, it would be nice to make sure that it will not send those URLs to Blippex.

[blippex]

Good idea, but maybe what we are doing already fixes this: we crawl the URL with our own crawler, at least an hour delayed and when we cannot access the domain, get redirected or find in the metatags the "noindex"-tag we don't index the page, just to be on the safe side,
But with security by obscurity is is a little bit complicated, the noindex-tag maybe helps you (and protect you from other search engines too, if someone goes to your side and then to Google maybe Google uses the Referer-link to crawl your page too!)

But we will think about it how we can implement this, thank you for the input!

[gubatron]

@kristofferwiklund @blippex what about the standard robots.txt? (first time I hear about the "noindex" tag, what if the folder has no html?)

[blippex]

Right now we don't use the robots.txt because we are only getting the content of one page, we don't crawl the whole server, and we get a page that was seen by a real person, but we will think about it!

[ProMcTagonist]

The user should still be able to abstain from sending certain websites to Blippex if this is not already possible. Also, is this project dead?