bug: Jan is not using GPU

[majsterkovic]

Jan version

0.5.4

Describe the Bug

Jan is not using GPU

Steps to Reproduce

Model: Bielik-11B-v2.3-Instruct.Q8_0.gguf
GPU: NVIDIA RTX 4070 SUPER

Screenshots / Logs

What is your OS?

• MacOS
• Windows
• Linux

[chlimouj]

I'm having a similar issue. I can only use my 4090 through Vulcan, not CUDA.

I also have a Tesla P40 installed, and Jan sees it but can't use it. Of course, the P40 doesn't support Vulcan.

In the past, I've been able to use the P40 and 4090 simultaneously to load large models. I did have some driver issues with a recent upgrade to Win11 (BSOD every few hours). Possibly not fully resolved? Not sure what to look at or what to try.

[imtuyethan]

@louis-jan is this related to #3558

[chlimouj]

@louis-jan is this related to #3558

In my case, it happens with models I've imported manually as well.

And the models load and run fine on CPU (just slower, obviously)

[louis-jan]

Hi @chlimouj Could you please share the log file, and what NGL settings are you using?

[mtomas7]

Same here. Log file attached.
app - Copy.log

• First I noticed that GPU Acceleration was OFF in Settings/Advanced. I turned it ON.
• After that AI Model would not load at all, but I got a message to install additional components for CUDA:
• Additional Dependencies
  This extension embeds cortex.cpp, a lightweight inference engine written in C++. See https://jan.ai/. Additional dependencies could be installed to run without Cuda Toolkit installation.

NGL was set to default 33 layers.

After that AI model was loaded into VRAM and worked.

[louis-jan]

Hi @chlimouj Yeah, with the fix, you'll need to re-import it for it to work properly. Which means it will retrieve the correct metadata from the model when importing now.

[louis-jan]

Same here. Log file attached. app - Copy.log

• First I noticed that GPU Acceleration was OFF in Settings/Advanced. I turned it ON.
• After that AI Model would not load at all, but I got a message to install additional components for CUDA:
• Additional Dependencies
  This extension embeds cortex.cpp, a lightweight inference engine written in C++. See https://jan.ai/. Additional dependencies could be installed to run without Cuda Toolkit installation.

NGL was set to default 33 layers.

After that AI model was loaded into VRAM and worked.

Yeah, we're working on bundling these dependencies into the app by default. It will increase the app size a bit, but it won't require any additional actions from the user.

[freelerobot]

Will be fixed with #3690

[Fuckingnameless]

how do you install this fucking cortex extension?

[Fuckingnameless]

also how to change the models folder?