Make two RTX or more together works in P2P. The step will be, update GRUB, remove all the NVIDIA drivers, re-install with the latest version, patch and run samples.
First thing, make sure to enable Resize BAR and disable IOMMU in BIOS (I’m using ASUS WRX80SAGE).
Install required packages
sudo update-alternatives --config gcc
sudo apt install -y pkg-config libglvnd-dev git cmake gcc-13 g++-13 gcc-13 g++-13
- Update GRUB
sudo vim /etc/default/grub and add amd_iommu=on iommu=pt
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash pci=nommconf amd_iommu=on iommu=pt"
Then, sudo update-grub
Install NVIDIA Driver
Next, download the NVIDIA drivers and install it
wget -c https://us.download.nvidia.com/XFree86/Linux-x86_64/590.44.01/NVIDIA-Linux-x86_64-590.44.01.run
chmod +x NVIDIA-Linux-x86_64-590.44.01.run
sudo ./NVIDIA-Linux-x86_64-590.44.01.run
Next, install the patch
git clone https://github.com/aikitoria/open-gpu-kernel-modules.git
cd open-gpu-kernel-modules
# IMPORTANT! Check into branch with same driver
# Otherwise you will get error nvidia-smi is returning "NVML: Driver/library version mismatch"
git checkout 590.44.01-p2p
./install.sh
sudo reboot
Last, download cuda samples. Make sure to have NVCC installed with you, I’m using 12.8 :
https://developer.nvidia.com/cuda-12-8-0-download-archive
Then, I download the cuda samples with 12.8 version
https://github.com/NVIDIA/cuda-samples/tree/v12.8
sudo apt install gcc-10 g++-10 gcc-10 g++-10 -y
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-10 10 --slave /usr/bin/g++ g++ /usr/bin/g++-10
# to change others
sudo update-alternatives --config gcc
wget -c https://github.com/NVIDIA/cuda-samples/archive/refs/tags/v12.8.zip
unzip v12.8.zip
cd cuda-samples-12.8
mkdir build && cd build
cmake ..
cmake --build . --config Release
./Samples/1_Utilities/deviceQuery/deviceQuery
The result will be
Supports Cooperative Kernel Launch: Yes
Supports MultiDevice Co-op Kernel Launch: Yes
Device PCI Domain ID / Bus ID / location ID: 0 / 66 / 0
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
> Peer access from NVIDIA GeForce RTX 4090 (GPU0) -> NVIDIA GeForce RTX 4090 (GPU1) : Yes
> Peer access from NVIDIA GeForce RTX 4090 (GPU1) -> NVIDIA GeForce RTX 4090 (GPU0) : Yes
deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 13.2, CUDA Runtime Version = 12.8, NumDevs = 2
Result = PASS
If you need to remove NVIDIA drivers entirely, follow the step below
Remove NVIDIA
sudo systemctl stop gdm3
sudo apt-get --purge remove "*nvidia*"
sudo apt-get --purge remove "*cuda*" "*cudnn*" "*cublas*" "*cufft*" "*cufile*" "*curand*" "*cusolver*" "*cusparse*" "*gds-tools*" "*npp*" "*nvjpeg*" "nsight*" "*nvvm*" "*libnccl*"
sudo /usr/bin/nvidia-uninstall
# disable iommu
ls -l /sys/class/iommu/
# reboot
sudo reboot
To check if everything is done completely. Run this command, if return result like this, then we need to remove it
cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX Open Kernel Module for x86_64 570.148.08 Release Build
Sat Jan 17 05:57:24 PM WIB 2026 GCC version: gcc version 13.3.0 (Ubuntu 13.3.0-6ubuntu2~24.04)
To remove it
sudo modprobe -r nvidia_drm nvidia_modeset nvidia_uvm nvidia || true
sudo rm -rf /lib/modules/$(uname -r)/kernel/drivers/video/nvidia*
sudo rm -rf /lib/modules/$(uname -r)/updates/dkms/nvidia*
sudo depmod -a
sudo update-initramfs -u
sudo reboot
Check if everything is cleared
nvidia-smi
command not found: nvidia-smi
cat /proc/driver/nvidia/version
cat: /proc/driver/nvidia/version: No such file or directory
Disable Novoeau and reboot
sudo bash -c "echo blacklist nouveau > /etc/modprobe.d/blacklist-nouveau.conf"
sudo bash -c "echo options nouveau modeset=0 >> /etc/modprobe.d/blacklist-nouveau.conf"
sudo update-initramfs -u
sudo reboot
