Hello .
Why if the entry point is zaledia.com, the crawler does not find all the links and gets stuck on zenalio.ch? Maybe it depends on the number of threads? That's how the crawler was launched:
./grawler -ilogfile ilog.txt -crawlers 50
The grawler.dot content after the scan is complete:
strict strict digraph {
"zaledia.com:70" [alive=true]
"zaledia.com:70" -> "zaledia.com:70"
"zaledia.com:70" -> "zenalio.ch:9999"
"zenalio.ch:9999" [alive=true]
"zenalio.ch:9999" -> "zenalio.ch:9999."
}
Would it be more efficient to modify the main.go file by adding reading from an array?
Example:
// Bootstrap the crawling.
var lnks1 [6]string
lnks1[0] = "sdf.org"
lnks1[1] = "gopher.quux.org."
lnks1[2] = "gopher.floodgap.com."
lnks1[3] = "bitreich.org."
lnks1[4] = "uninformativ.de."
lnks1[5] = "gopher.viste.fr."
.....
If there are a lot of references?
In 2 hours
Crawling without parmeters took longer, but the content of the grawler.dot file remained unchanged.
Hello .
Why if the entry point is zaledia.com, the crawler does not find all the links and gets stuck on zenalio.ch? Maybe it depends on the number of threads? That's how the crawler was launched:
The grawler.dot content after the scan is complete:
Would it be more efficient to modify the main.go file by adding reading from an array?
Example:
.....
If there are a lot of references?
In 2 hours
Crawling without parmeters took longer, but the content of the grawler.dot file remained unchanged.