github/oneuptime
1
1
Fork 0
mirror of https://github.com/OneUptime/oneuptime.git synced 2026-04-06 00:32:12 +02:00
Code Issues 193 Packages Projects Releases Wiki Activity

enable gpu on llama docker

Browse Source
This commit is contained in:
Simon Larsen
2023-10-18 12:07:37 +01:00
parent c629921d01
commit b3cfdbf45a
2 changed files with 10 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
Llama/Readme.md
View File

@@ -13,6 +13,14 @@ docker build -t llama . -f ./Llama/Dockerfile
## Run
### For Linux
```
docker run --gpus all -p 8547:8547 -it -v ./Llama/Models:/app/Models llama
```
### For MacOS
```
docker run -p 8547:8547 -it -v ./Llama/Models:/app/Models llama
```

3
Llama/app.py
View File

@@ -15,7 +15,8 @@ tokenizer = AutoTokenizer.from_pretrained(model_path, local_files_only=True)
pipeline = transformers.pipeline(
"text-generation",
model=model_path,
torch_dtype=torch.float32,
# torch_dtype=torch.float32, # for CPU
torch_dtype=torch.float16, # for GPU
device_map="auto",
)