buffer overflow detected

[andrejj]

Hi.

In case you have an easy way to fix this or let me know how to fix it this would be much appreciated.
If you remove one symbol and no matter which one, this thing works, otherwise it fails.

I know this was meant for English and is probably pointless to use it on Chinese, but in any case this is just a string and should not fail with a buffer overflow error :)

I'm using: ruby-1.9.3-p547

TermExtract.extract("画像を、デスクトップ画像と共有ソフト（winnyなど）のアイコンをそれぞれ合成して、デスクトップのスクショを捏造され、「ファイルを違法共有するクソ作者！」みたいな感じで叩かれた")

*** buffer overflow detected ***

[rrphotosoft]

@andrejj , I faced the same problem, even with english. The problem can be traced to the .so file used by rb-tagger, there is nothing primarily wrong with term-extract. What happens is that it being a so file, and given the way in which rb-tagger is initialized and held in memory by term-extract, you land up with too many open files(ulimit).
There is no way to solve this without using epoll or poll instead of select, at the IO level.
In case you need to use this gem for English, please use my fork:

https://github.com/rrphotosoft/term-extract

It uses the engtagger gem for POS recognition and I have tested it on 1.3 million docs being processed in parallel on a sidekiq driven ruby process. There were no problems and it works like a charm.

As for why you encounter this problem with one line of Chinese characters, I guess the POS tagging is to blame, the rb-tagger gem doesn't know what to do with these characters and probably converts them to a low level format which becomes equal to parsing 1000's of english documents, and then you hit the same error that I had.

Good luck with the Chinese!