Trying to build this repo (in order to get NeMo running) I get the following error:
$ pip install "git+https://github.com/LahiLuk/YouTokenToMe"
Collecting git+https://github.com/LahiLuk/YouTokenToMe
Cloning https://github.com/LahiLuk/YouTokenToMe to /tmp/pip-req-build-pbvabuz3
Running command git clone --filter=blob:none --quiet https://github.com/LahiLuk/YouTokenToMe /tmp/pip-req-build-pbvabuz3
Resolved https://github.com/LahiLuk/YouTokenToMe to commit f9fe56e198e22d552d821a5f432d4e5ada1e81e8
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Collecting Click>=7.0 (from youtokentome==1.0.6)
Obtaining dependency information for Click>=7.0 from https://files.pythonhosted.org/packages/00/2e/d53fa4befbf2cfa713304affc7ca780ce4fc1fd8710527771b58311a3229/click-8.1.7-py3-none-any.whl.metadata
Downloading click-8.1.7-py3-none-any.whl.metadata (3.0 kB)
Using cached click-8.1.7-py3-none-any.whl (97 kB)
Building wheels for collected packages: youtokentome
Building wheel for youtokentome (pyproject.toml) ... error
error: subprocess-exited-with-error
× Building wheel for youtokentome (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [121 lines of output]
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-cpython-311
creating build/lib.linux-x86_64-cpython-311/youtokentome
copying youtokentome/youtokentome.py -> build/lib.linux-x86_64-cpython-311/youtokentome
copying youtokentome/__init__.py -> build/lib.linux-x86_64-cpython-311/youtokentome
copying youtokentome/yttm_cli.py -> build/lib.linux-x86_64-cpython-311/youtokentome
running build_ext
building '_youtokentome_cython' extension
creating build/temp.linux-x86_64-cpython-311
creating build/temp.linux-x86_64-cpython-311/youtokentome
creating build/temp.linux-x86_64-cpython-311/youtokentome/cpp
x86_64-linux-gnu-gcc -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -g -fwrapv -O2 -fPIC -Iyoutokentome/cpp -I/home/tejes/work/inmoment/parakeet/venv/include -I/usr/include/python3.11 -c youtokentome/cpp/bpe.cpp -o build/temp.linux-x86_64-cpython-311/youtokentome/cpp/bpe.o -std=c++11 -pthread -O3
x86_64-linux-gnu-gcc -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -fstack-clash-protection -Wformat -Werror=format-security -fcf-protection -g -fwrapv -O2 -fPIC -Iyoutokentome/cpp -I/home/tejes/work/inmoment/parakeet/venv/include -I/usr/include/python3.11 -c youtokentome/cpp/utf8.cpp -o build/temp.linux-x86_64-cpython-311/youtokentome/cpp/utf8.o -std=c++11 -pthread -O3
In file included from youtokentome/cpp/utf8.cpp:1:
youtokentome/cpp/utf8.h:9:18: error: ‘uint32_t’ does not name a type
9 | constexpr static uint32_t INVALID_UNICODE = 0x0fffffff;
| ^~~~~~~~
youtokentome/cpp/utf8.h:6:1: note: ‘uint32_t’ is defined in header ‘<cstdint>’; did you forget to ‘#include <cstdint>’?
5 | #include <cassert>
+++ |+#include <cstdint>
6 |
youtokentome/cpp/utf8.h:11:1: error: ‘uint32_t’ does not name a type
11 | uint32_t chars_to_utf8(const char* begin, uint64_t size, uint64_t* utf8_len);
| ^~~~~~~~
youtokentome/cpp/utf8.h:11:1: note: ‘uint32_t’ is defined in header ‘<cstdint>’; did you forget to ‘#include <cstdint>’?
youtokentome/cpp/utf8.h:13:6: error: variable or field ‘utf8_to_chars’ declared void
13 | void utf8_to_chars(uint32_t x, std::back_insert_iterator<std::string> it);
| ^~~~~~~~~~~~~
youtokentome/cpp/utf8.h:13:20: error: ‘uint32_t’ was not declared in this scope
13 | void utf8_to_chars(uint32_t x, std::back_insert_iterator<std::string> it);
| ^~~~~~~~
youtokentome/cpp/utf8.h:13:20: note: ‘uint32_t’ is defined in header ‘<cstdint>’; did you forget to ‘#include <cstdint>’?
youtokentome/cpp/utf8.h:13:71: error: expected primary-expression before ‘it’
13 | void utf8_to_chars(uint32_t x, std::back_insert_iterator<std::string> it);
| ^~
youtokentome/cpp/utf8.h:15:43: error: ‘uint32_t’ was not declared in this scope
15 | std::string encode_utf8(const std::vector<uint32_t> &utext);
| ^~~~~~~~
youtokentome/cpp/utf8.h:15:43: note: ‘uint32_t’ is defined in header ‘<cstdint>’; did you forget to ‘#include <cstdint>’?
youtokentome/cpp/utf8.h:15:51: error: template argument 1 is invalid
15 | std::string encode_utf8(const std::vector<uint32_t> &utext);
| ^
youtokentome/cpp/utf8.h:15:51: error: template argument 2 is invalid
youtokentome/cpp/utf8.h:17:13: error: ‘uint32_t’ was not declared in this scope
17 | std::vector<uint32_t> decode_utf8(const char *begin, const char *end);
| ^~~~~~~~
youtokentome/cpp/utf8.h:17:13: note: ‘uint32_t’ is defined in header ‘<cstdint>’; did you forget to ‘#include <cstdint>’?
youtokentome/cpp/utf8.h:17:21: error: template argument 1 is invalid
17 | std::vector<uint32_t> decode_utf8(const char *begin, const char *end);
| ^
youtokentome/cpp/utf8.h:17:21: error: template argument 2 is invalid
youtokentome/cpp/utf8.h:19:13: error: ‘uint32_t’ was not declared in this scope
19 | std::vector<uint32_t> decode_utf8(const std::string &utf8_text);
| ^~~~~~~~
youtokentome/cpp/utf8.h:19:13: note: ‘uint32_t’ is defined in header ‘<cstdint>’; did you forget to ‘#include <cstdint>’?
youtokentome/cpp/utf8.h:19:21: error: template argument 1 is invalid
19 | std::vector<uint32_t> decode_utf8(const std::string &utf8_text);
| ^
youtokentome/cpp/utf8.h:19:21: error: template argument 2 is invalid
youtokentome/cpp/utf8.h:33:3: error: ‘uint32_t’ does not name a type
33 | uint32_t operator*() {
| ^~~~~~~~
youtokentome/cpp/utf8.h:33:3: note: ‘uint32_t’ is defined in header ‘<cstdint>’; did you forget to ‘#include <cstdint>’?
youtokentome/cpp/utf8.h:43:3: error: ‘uint64_t’ does not name a type
43 | uint64_t get_utf8_len() {
| ^~~~~~~~
youtokentome/cpp/utf8.h:43:3: note: ‘uint64_t’ is defined in header ‘<cstdint>’; did you forget to ‘#include <cstdint>’?
youtokentome/cpp/utf8.h:53:3: error: ‘uint32_t’ does not name a type
53 | uint32_t code_point = 0;
| ^~~~~~~~
youtokentome/cpp/utf8.h:53:3: note: ‘uint32_t’ is defined in header ‘<cstdint>’; did you forget to ‘#include <cstdint>’?
youtokentome/cpp/utf8.h:54:3: error: ‘uint64_t’ does not name a type
54 | uint64_t utf8_len = 0;
| ^~~~~~~~
youtokentome/cpp/utf8.h:54:3: note: ‘uint64_t’ is defined in header ‘<cstdint>’; did you forget to ‘#include <cstdint>’?
youtokentome/cpp/utf8.h: In member function ‘vkcom::UTF8Iterator vkcom::UTF8Iterator::operator++()’:
youtokentome/cpp/utf8.h:28:14: error: ‘utf8_len’ was not declared in this scope
28 | begin += utf8_len;
| ^~~~~~~~
youtokentome/cpp/utf8.h: In member function ‘void vkcom::UTF8Iterator::parse()’:
youtokentome/cpp/utf8.h:61:5: error: ‘code_point’ was not declared in this scope
61 | code_point = chars_to_utf8(begin, end - begin, &utf8_len);
| ^~~~~~~~~~
youtokentome/cpp/utf8.h:61:53: error: ‘utf8_len’ was not declared in this scope
61 | code_point = chars_to_utf8(begin, end - begin, &utf8_len);
| ^~~~~~~~
youtokentome/cpp/utf8.h:61:18: error: ‘chars_to_utf8’ was not declared in this scope
61 | code_point = chars_to_utf8(begin, end - begin, &utf8_len);
| ^~~~~~~~~~~~~
youtokentome/cpp/utf8.cpp: In function ‘uint32_t vkcom::chars_to_utf8(const char*, uint64_t, uint64_t*)’:
youtokentome/cpp/utf8.cpp:73:10: error: ‘INVALID_UNICODE’ was not declared in this scope
73 | return INVALID_UNICODE;
| ^~~~~~~~~~~~~~~
youtokentome/cpp/utf8.cpp: At global scope:
youtokentome/cpp/utf8.cpp:111:18: error: ambiguating new declaration of ‘std::vector<unsigned int> vkcom::decode_utf8(const char*, const char*)’
111 | vector<uint32_t> decode_utf8(const char* begin, const char* end) {
| ^~~~~~~~~~~
youtokentome/cpp/utf8.h:17:23: note: old declaration ‘int vkcom::decode_utf8(const char*, const char*)’
17 | std::vector<uint32_t> decode_utf8(const char *begin, const char *end);
| ^~~~~~~~~~~
youtokentome/cpp/utf8.cpp: In function ‘std::vector<unsigned int> vkcom::decode_utf8(const char*, const char*)’:
youtokentome/cpp/utf8.cpp:117:23: error: ‘INVALID_UNICODE’ was not declared in this scope
117 | if (code_point != INVALID_UNICODE) {
| ^~~~~~~~~~~~~~~
youtokentome/cpp/utf8.cpp: At global scope:
youtokentome/cpp/utf8.cpp:130:18: error: ambiguating new declaration of ‘std::vector<unsigned int> vkcom::decode_utf8(const std::string&)’
130 | vector<uint32_t> decode_utf8(const string& utf8_text) {
| ^~~~~~~~~~~
youtokentome/cpp/utf8.h:19:23: note: old declaration ‘int vkcom::decode_utf8(const std::string&)’
19 | std::vector<uint32_t> decode_utf8(const std::string &utf8_text);
| ^~~~~~~~~~~
youtokentome/cpp/utf8.cpp: In function ‘std::vector<unsigned int> vkcom::decode_utf8(const std::string&)’:
youtokentome/cpp/utf8.cpp:131:21: error: could not convert ‘vkcom::decode_utf8((& utf8_text)->std::__cxx11::basic_string<char>::data(), ((& utf8_text)->std::__cxx11::basic_string<char>::data() + ((sizetype)(& utf8_text)->std::__cxx11::basic_string<char>::size())))’ from ‘int’ to ‘std::vector<unsigned int>’
131 | return decode_utf8(utf8_text.data(), utf8_text.data() + utf8_text.size());
| ~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| |
| int
error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for youtokentome
Failed to build youtokentome
ERROR: Could not build wheels for youtokentome, which is required to install pyproject.toml-based projects
Trying to build this repo (in order to get NeMo running) I get the following error:
OS: Ubuntu 23.10 x64
Python: 3.11.6
GCC: 13.2.0
Cython: 3.0.10 already installed in the venv