roadmap: mmap for keeping Model in VRAM when Flash Attention is used

[dan-menlo]

Goal

Feedback from WiseFarAI:

I am also wondering if you guys use Mmap and other ways of keeping the model in Vram memory and if/when Flash Attention is used. Since these parameters cannot easily be observed (their current setting) or changed from within the UI. Sometimes it seems like it tries to reload the model mid-conversation and the generation speed drops from 6 tokens to 2 per second.

[gabrielle-ong]

@vansangpfiev assigning to you to investigate, cc @dan-homebrew - will have a call with Sang next week

[vansangpfiev]

llama.cpp optimizes memory usage by strategically distributing model components across VRAM and RAM:

1. VRAM allocation:

   • Model weights
   • Key-value (KV) cache
   • Preprocessed prompt buffer

2. RAM allocation:

   • Embedding lookup table
   • Auxiliary buffers for data transfer between VRAM and CPU.
   • Model

The potential issue of reloading the model mid-conversation typically arises if the model in memory is swapped out due to system memory pressure or OS schedule.

The model is kept in VRAM during inference, for example in case of CUDA, the cudaHostRegister() API is used to pin memory, which ensures it stays in GPU memory and isn't swapped out. https://github.com/ggerganov/llama.cpp/blob/8d59d911711b8f1ba9ec57c4b192ccd2628af033/ggml/src/ggml-cuda/ggml-cuda.cu#L2685

llama.cpp provides mlock parameter which ensures to lock model into RAM, prevent swapping out.

cc: @dan-menlo @nguyenhoangthuan99