Qwen3-32B is a powerful and thoughtful language model designed to handle everything from casual chats to deep reasoning. It shines in areas like solving logic puzzles, writing stories, walking through math problems step by step, and even following complex instructions over long conversations.

What sets it apart is its ability to switch modes — it can think carefully when needed or keep things light and efficient during everyday conversations. It also understands over a hundred languages and can switch between them smoothly.

This version comes in the GGUF format, which means it’s easy to run efficiently on a well-configured system without needing complicated setups. Whether you’re building tools, writing creative content, or exploring new ideas, Qwen3-32B is ready to help — and it does it with a touch of clarity and purpose that’s rare.

Resource

Hugging Face

Link:https://huggingface.co/Qwen/Qwen3-32B-GGUF

GitHub

Link: https://github.com/QwenLM/Qwen3

Step-by-Step Process to Install Qwen3-32B-GGUF Locally

For the purpose of this tutorial, we will use a GPU-powered Virtual Machine offered by NodeShift; however, you can replicate the same steps with any other cloud provider of your choice. NodeShift provides the most affordable Virtual Machines at a scale that meets GDPR, SOC2, and ISO27001 requirements.

Step 1: Sign Up and Set Up a NodeShift Cloud Account

Visit the NodeShift Platform and create an account. Once you’ve signed up, log into your account.

Follow the account setup process and provide the necessary details and information.

Step 2: Create a GPU Node (Virtual Machine)

GPU Nodes are NodeShift’s GPU Virtual Machines, on-demand resources equipped with diverse GPUs ranging from H100s to A100s. These GPU-powered VMs provide enhanced environmental control, allowing configuration adjustments for GPUs, CPUs, RAM, and Storage based on specific requirements.

Navigate to the menu on the left side. Select the GPU Nodes option, create a GPU Node in the Dashboard, click the Create GPU Node button, and create your first Virtual Machine deploy

Step 3: Select a Model, Region, and Storage

In the “GPU Nodes” tab, select a GPU Model and Storage according to your needs and the geographical region where you want to launch your model.

We will use 1 x H100 SXM GPU for this tutorial to achieve the fastest performance. However, you can choose a more affordable GPU with less VRAM if that better suits your requirements.

Step 4: Select Authentication Method

There are two authentication methods available: Password and SSH Key. SSH keys are a more secure option. To create them, please refer to our official documentation.

Step 5: Choose an Image

Next, you will need to choose an image for your Virtual Machine. We will deploy Qwen3-32B-GGUF on an NVIDIA Cuda Virtual Machine. This proprietary, closed-source parallel computing platform will allow you to install Qwen3-32B-GGUF on your GPU Node.

After choosing the image, click the ‘Create’ button, and your Virtual Machine will be deployed.

Step 6: Virtual Machine Successfully Deployed

You will get visual confirmation that your node is up and running.

Step 7: Connect to GPUs using SSH

NodeShift GPUs can be connected to and controlled through a terminal using the SSH key provided during GPU creation.

Once your GPU Node deployment is successfully created and has reached the ‘RUNNING’ status, you can navigate to the page of your GPU Deployment Instance. Then, click the ‘Connect’ button in the top right corner.

Now open your terminal and paste the proxy SSH IP or direct SSH IP.

Next, if you want to check the GPU details, run the command below:
nvidia-smi


Step 8: Install Required Dependencies

Run the following command to install required dependencies:
sudo apt install git build-essential cmake libopenblas-dev libsqlite3-dev -y


Step 9: Clone the llama.cpp Repository

Run the following command to clone the llama.cpp repository:
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp


Step 10: Install libcurl Development Package

Run the following command to install the missing dependency:
sudo apt install libcurl4-openssl-dev -y


Step 11: Build llama.cpp

Run the following command to build llama.cpp:
mkdir build && cd build
cmake ..
cmake --build . --config Release


Step 12: Download the Model from Hugging Face

Run the following command to download the model from hugging face:
mkdir -p ~/models/qwen3-32b
cd ~/models/qwen3-32b
wget https://huggingface.co/Qwen/Qwen3-32B-GGUF/resolve/main/qwen3-32b.Q8_0.gguf


Step 13: Run the Model Using llama-cli

Execute the following command to run the model using llama-cli:

~/llama.cpp/build/bin/llama-cli \
  -m ~/models/qwen3-32b/Qwen3-32B-Q8_0.gguf \
  --temp 0.6 \
  --top-k 20 \
  --top-p 0.95 \
  --presence-penalty 1.5 \
  -n 2048 \
  -c 40960 \
  --color \
  --jinja \
  --no-context-shift

Step 14: Run Prompts and Generate Response in Terminal

Run the prompts in terminal and generate response:

Thinking mode (math, logic, code):
What is 129 * 42? /think


Non-thinking mode (casual talk):

Tell me a bedtime story /no_think


Conclusion

Setting up and running Qwen3-32B-GGUF on a GPU-powered virtual machine might seem like a long journey at first, but once you’re through it, the possibilities open up wide. Whether you’re solving tricky math problems, writing creative stories, or testing multilingual dialogue, this model delivers with clarity and flexibility.

The best part? It doesn’t require heavyweight frameworks or tangled toolchains. With just a solid GPU, a clean Ubuntu setup, and the simplicity of llama.cpp, you’re ready to go — no unnecessary noise, no bloated dependencies.

If you’re someone who enjoys working close to the metal, tuning performance, and having full control over your environment, this guide should serve as your launchpad.