I am getting an error from the encode function. I am using it in this way: encode("Some string text").length. But maybe for any value I am getting the error: TypeError: bpe(...).split is not a function from this line: const new_tokens = bpe(token).split(' ').map(x => encoder[x]).
Note: I have tried with some random values instead of string. Like: [], 112, [''] etc. But for these values another error comes which is TypeError: text.matchAll is not a function from this line: const matches = Array.from(text.matchAll(pat)).map(x => x[0]).
Please solve these issues and release another version of it.
Thanks in advance
I am getting an error from the
encodefunction. I am using it in this way:encode("Some string text").length. But maybe for any value I am getting the error:TypeError: bpe(...).split is not a functionfrom this line:const new_tokens = bpe(token).split(' ').map(x => encoder[x]).Note: I have tried with some random values instead of string. Like:
[],112,['']etc. But for these values another error comes which isTypeError: text.matchAll is not a functionfrom this line:const matches = Array.from(text.matchAll(pat)).map(x => x[0]).Please solve these issues and release another version of it.
Thanks in advance