Duplicate URLs? #134
Description
Hello! I was reading the implementation and got a bit surprised in finding that there seems to be no check for duplicate URLs before adding them to the queue. Is it not needed naturally, or am I missing something? Thank you so much!