Possibility of 'official' python bindings

[LLukas22]

I really like this project, especially the plans to not only support llama based models but to create a more general llm runtime.

I was asking myself if there are any plans to create python bindings for this project similar to e.g. llama-cpp-python.

A huge part of the ML community uses python as their primary language and could probably profit from the ability of performing model inference on cpu based nodes with this package. This also would simplify the integration of llama-rs run models into already existing solution like LangChain.

[philpax]

Yeah, sure, I don't see why not! I would suggest someone creates a repo that wraps llama-rs with PyO3, and once it's good to go we'll move it into rustformers.

[LLukas22]

@philpax Alright i cooked something up as i had problems with avx2/avx512 cross-compilation with llama-cpp-python and needed it for a project.

Its probably not good as i never touched rust before but i found the codebase surprisingly easy to read and could implement model loading and inference without any problems. I uploaded the project here: llama-rs-python.

Maybe someone who actually knows what he's doing could use it as a starting point 🤔

[philpax]

Nice work! You may also want to chat to @hlhr202, who works on https://github.com/Atome-FE/llama-node

[LLukas22]

Thanks, maybe i will chat a bit with him, i still need to find a way to stream a response from the rust side to the python side without a callback to circumvent the GIL. Maybe he knows something.

When i'm finished with my current project i will probably also try to support GPT-J based models in llama-rs, if they aren't supported yet.