Wiki » Konfiguracja serwerów » Hardware » GPU » GPU Nvidia »
GPU Nvidia RTX 40xx¶
laama.cpp + Nvidia CUDA¶
- Instalacja zależności(cmake itd.) PokażUkryj
apt-get install -y cmake ccache libssl-dev;
- Kompilacja llama.cpp PokażUkryj
git clone https://github.com/ggml-org/llama.cpp `date +%Y%m%d-cuda`; \ cd `date +%Y%m%d-cuda`; \ \ cmake -B build \ -DGGML_CUDA_FA=ON \ -DGGML_CUDA_USE_FLASH_ATTENTION=ON \ -DGGML_CUDA_FA_ALL_QUANTS=ON \ -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON \ -DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc \ -DCMAKE_CUDA_ARCHITECTURES=89; \ \ cmake --build build --config Release -j$(nproc);
Pobranie modeli i uruchomienie¶
- Pobranie modeli PokażUkryj
apt-get install -y pipx; \ \ pipx install huggingface_hub; \ pipx ensurepath; \ source ~/.bashrc; \ \ export HF_XET_HIGH_PERFORMANCE=1; # FIM/NextLine dla Auto Complete IDE hf download unsloth/Qwen2.5-Coder-3B-Instruct-128K-GGUF \ Qwen2.5-Coder-3B-Instruct-Q8_0.gguf \ --cache-dir ./models/Qwen2.5-Coder-3B # Embedding hf download unsloth/embeddinggemma-300m-GGUF \ embeddinggemma-300M-Q8_0.gguf \ --cache-dir ./models/embeddinggemma-300M # ReRanker hf download sinjab/jina-reranker-v2-base-multilingual-Q4_K_M-GGUF \ jina-reranker-v2-base-multilingual-Q4_K_M.gguf \ --cache-dir ./models/jina-reranker-v2-base-multilingual # 16GB GB GPU #hf download unsloth/Qwen3.6-27B-MTP-GGUF \ # Qwen3.6-27B-UD-IQ3_XXS.gguf \ # --cache-dir ./models/qwen36-27b-MTP; #hf download michaelw9999/Qwen3.6-27B-NVFP4-SMALL-MTP-GGUF \ # --cache-dir ./models/qwen36-27b # Mniejszy context tylko 32K, zjada wiecej RAM (porownanie z unsloth/Qwen2.5-Coder-3B-Instruct-128K-GGUF Qwen2.5-Coder-3B-Instruct-Q8_0.gguf) #hf download mradermacher/AceCoder-Qwen2.5-Coder-7B-Ins-V1.1-i1-GGUF \ # AceCoder-Qwen2.5-Coder-7B-Ins-V1.1.i1-Q4_K_M.gguf \ # --cache-dir ./models/Qwen2.5-Coder-7B
- Uruchomienie:
- Model kodujacy PokażUkryj
TREE=20260723 MODEL=/home/storage/stripe/llm/models/Qwen2.5-Coder-3B/models--unsloth--Qwen2.5-Coder-3B-Instruct-128K-GGUF/snapshots/5326551926d06f7f9cab53c9b9b552e3bedfe8ba/Qwen2.5-Coder-3B-Instruct-Q8_0.gguf CTX=131072 RAM=12288 IP=x.x.x.x PORT=8081 ./${TREE}/build/bin/llama-server \ -m ${MODEL} \ \ -t 4 \ -fa 1 \ \ --parallel 2 \ \ --ctx-size ${CTX} \ --cache-ram ${RAM} \ --cache-type-v q8_0 \ --cache-type-k q8_0 \ \ --jinja \ --reasoning off\ \ --temperature 0.6 \ --top-k 20 \ --top-p 0.95 \ --min-p 0.0 \ --repeat-penalty 1.0 \ --presence-penalty 0.0 \ \ --batch-size 2048 \ --ubatch-size 2048 \ \ --host ${IP} \ --port ${PORT} - Model Embedding PokażUkryj
TREE=20260723 MODEL=/home/storage/stripe/llm/models/embeddinggemma-300M/models--unsloth--embeddinggemma-300m-GGUF/snapshots/6661a6504c30d8304af13455cb4a5d4f5bc6011f/embeddinggemma-300M-Q8_0.gguf CTX=8192 # maksymalny rozmiar jaki był trenowany dla czterech wątków RAM=2048 IP=x.x.x.x PORT=8082 ./${TREE}/build/bin/llama-server \ --embedding \ -m ${MODEL} \ \ -t 4 \ -fa 1 \ \ --parallel 4 \ \ --ctx-size ${CTX} \ --cache-ram ${RAM} \ --cache-type-v q8_0 \ --cache-type-k q8_0 \ \ --jinja \ --reasoning off\ \ --batch-size 2048 \ --ubatch-size 2048 \ \ --host ${IP} \ --port ${PORT} - Model Embedding PokażUkryj
TREE=20260723 MODEL=/home/storage/stripe/llm/models/jina-reranker-v2-base-multilingual/models--sinjab--jina-reranker-v2-base-multilingual-Q4_K_M-GGUF/snapshots/c1b17df2ce6b810e31e2cae9491b98eecf05947b/jina-reranker-v2-base-multilingual-Q4_K_M.gguf CTX=4096 # Maksymalny jaki był trenowany RAM=2048 IP=x.x.x.x PORT=8083 ./${TREE}/build/bin/llama-server \ --reranking \ -m ${MODEL} \ --alias jina-reranker-v2 \ \ -fa 1 \ \ --parallel 4 \ \ --ctx-size ${CTX} \ --cache-ram ${RAM} \ --cache-type-v q8_0 \ --cache-type-k q8_0 \ \ --jinja \ --reasoning off\ \ --batch-size 2048 \ --ubatch-size 2048 \ \ --host ${IP} \ --port ${PORT}
- Model kodujacy PokażUkryj