2023-03-24 00:06:24 -04:00
# 🦙 Python Bindings for `llama.cpp`
2023-03-27 18:30:12 -04:00
[](https://abetlen.github.io/llama-cpp-python)
2023-04-05 04:41:24 -04:00
[](https://github.com/abetlen/llama-cpp-python/actions/workflows/test.yaml)
2023-03-24 00:06:24 -04:00
[](https://pypi.org/project/llama-cpp-python/)
[](https://pypi.org/project/llama-cpp-python/)
[](https://pypi.org/project/llama-cpp-python/)
[](https://pypi.org/project/llama-cpp-python/)
2023-03-23 05:33:06 -04:00
2023-03-23 23:55:42 -04:00
Simple Python bindings for * * @ggerganov 's** [`llama.cpp` ](https://github.com/ggerganov/llama.cpp ) library.
2023-03-23 16:00:10 -04:00
This package provides:
2023-03-23 05:33:06 -04:00
2023-03-23 23:55:42 -04:00
- Low-level access to C API via `ctypes` interface.
- High-level Python API for text completion
- OpenAI-like API
2023-03-24 00:06:24 -04:00
- LangChain compatibility
# Installation
2023-03-23 05:33:06 -04:00
2023-03-24 00:06:24 -04:00
Install from PyPI:
2023-03-23 05:33:06 -04:00
``` bash
2023-03-23 14:24:34 -04:00
pip install llama-cpp-python
2023-03-23 05:33:06 -04:00
```
# Usage
``` python
>> > from llama_cpp import Llama
>> > llm = Llama ( model_path = " models/7B/... " )
>> > output = llm ( " Q: Name the planets in the solar system? A: " , max_tokens = 32 , stop = [ " Q: " , " \n " ] , echo = True )
>> > print ( output )
{
" id " : " cmpl-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx " ,
" object " : " text_completion " ,
" created " : 1679561337 ,
" model " : " models/7B/... " ,
" choices " : [
{
" text " : " Q: Name the planets in the solar system? A: Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune and Pluto. " ,
" index " : 0 ,
" logprobs " : None ,
" finish_reason " : " stop "
}
] ,
" usage " : {
" prompt_tokens " : 14 ,
" completion_tokens " : 28 ,
" total_tokens " : 42
}
}
```
2023-03-24 00:06:24 -04:00
2023-04-02 21:03:39 -04:00
# Documentation
Documentation is available at [https://abetlen.github.io/llama-cpp-python ](https://abetlen.github.io/llama-cpp-python ).
If you find any issues with the documentation, please open an issue or submit a PR.
# Development
This package is under active development and I welcome any contributions.
To get started, clone the repository and install the package in development mode:
``` bash
git clone git@github.com:abetlen/llama-cpp-python.git
git submodule update --init --recursive
# Will need to be re-run any time vendor/llama.cpp is updated
python3 setup.py develop
```
# How does this compare to other Python bindings of `llama.cpp`?
2023-04-04 10:57:22 -04:00
I originally wrote this package for my own use with two goals in mind:
2023-04-02 21:03:39 -04:00
- Provide a simple process to install `llama.cpp` and access the full C API in `llama.h` from Python
- Provide a high-level Python API that can be used as a drop-in replacement for the OpenAI API so existing apps can be easily ported to use `llama.cpp`
Any contributions and changes to this package will be made with these goals in mind.
2023-03-24 00:06:24 -04:00
# License
This project is licensed under the terms of the MIT license.
