Skip to content

libcuda.so is not in found but libcuda.so.1 is.  #8587

Closed

Description

Version

Microsoft Windows [Version 10.0.22000.778]

WSL Version

  • WSL 2
  • WSL 1

Kernel Version

5.10.102.1

Distro Version

Ubuntu 22.04

Other Software

cuda 10.7 , miniconda , pytorch

Repro Steps

Install CUDA as described on nvidia website using option 1:
https://docs.nvidia.com/cuda/wsl-user-guide/index.html

then install python3.9 and try the following code that shows the issue :

$ python
Python 3.9.12 (main, Apr  5 2022, 06:56:58)
[GCC 7.5.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from ctypes import cdll
>>> cdll.LoadLibrary('libcuda.so.1')
<CDLL 'libcuda.so.1', handle 562f22ae4160 at 0x7f74fb662c10>
>>> cdll.LoadLibrary('libcuda.so')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/piotr/miniconda3/lib/python3.9/ctypes/__init__.py", line 460, in LoadLibrary
    return self._dlltype(name)
  File "/home/piotr/miniconda3/lib/python3.9/ctypes/__init__.py", line 382, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libcuda.so: cannot open shared object file: No such file or directory

I'm loading the library manually so that you don't have to install pytorch and timm, and then try to run training of 'resnet26d', as this triggers the loading of libcudnn_cnn_infer.so.8 which is linked against libcuda.so instead of libcuda.so.1 .

Expected Behavior

It should be possible to load libcuda.so without specifing /usr/lib/wsl/lib in the LD_LIBRARY_PATH.

>>> cdll.LoadLibrary('libcuda.so')
should return a library handle

Actual Behavior

Unfortunately somehow the ldconfig fails to cache the libcuda.so and subsequently it does not find it unless /usr/lib/wsl/lib is in the LD_LIBRARY_PATH.

Diagnostic Logs

libcuda.so is in the /usr/lib/wsl/lib along with libcuda.so.1 and libcuda.so.1.1 . All files are the same.

It looks like ldconfig somehow does not save the path to libcuda.so, you can observe this by running:

$  ldconfig  /usr/lib/wsl/lib -v  -n
libnvwgf2umx.so -> libnvwgf2umx.so
        libnvidia-opticalflow.so.1 -> libnvidia-opticalflow.so.1
        libnvidia-ml.so.1 -> libnvidia-ml.so.1
        libnvidia-encode.so.1 -> libnvidia-encode.so.1
        libnvdxdlkernels.so -> libnvdxdlkernels.so
        libnvcuvid.so.1 -> libnvcuvid.so.1
        libdxcore.so -> libdxcore.so
        libd3d12core.so -> libd3d12core.so
        libd3d12.so -> libd3d12.so
/sbin/ldconfig.real: /usr/lib/wsl/lib/libcuda.so.1 is not a symbolic link

        libcuda.so.1 -> libcuda.so.1.1

The libcuda.so is not being picked up, even though strace shows that ldconfig is reading that file:
strace ldconfig /usr/lib/wsl/lib -n

newfstatat(AT_FDCWD, "/usr/lib/wsl/lib", {st_mode=S_IFDIR|0555, st_size=4096, ...}, 0) = 0
openat(AT_FDCWD, "/usr/lib/wsl/lib/glibc-hwcaps", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/var/cache/ldconfig/aux-cache", O_RDONLY) = -1 EACCES (Permission denied)
openat(AT_FDCWD, "/usr/lib/wsl/lib", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
newfstatat(3, "", {st_mode=S_IFDIR|0555, st_size=4096, ...}, AT_EMPTY_PATH) = 0
getdents64(3, 0x555556da4ee0 /* 18 entries */, 32768) = 672
newfstatat(AT_FDCWD, "/usr/lib/wsl/lib/libcuda.so", {st_mode=S_IFREG|0555, st_size=141464, ...}, AT_SYMLINK_NOFOLLOW) = 0
openat(AT_FDCWD, "/usr/lib/wsl/lib/libcuda.so", O_RDONLY) = 4
newfstatat(4, "", {st_mode=S_IFREG|0555, st_size=141464, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 141464, PROT_READ, MAP_SHARED, 4, 0) = 0x7fe867b6e000
munmap(0x7fe867b6e000, 141464)          = 0
close(4)                                = 0
newfstatat(AT_FDCWD, "/usr/lib/wsl/lib/libcuda.so.1", {st_mode=S_IFREG|0555, st_size=141464, ...}, AT_SYMLINK_NOFOLLOW) = 0
openat(AT_FDCWD, "/usr/lib/wsl/lib/libcuda.so.1", O_RDONLY) = 4
newfstatat(4, "", {st_mode=S_IFREG|0555, st_size=141464, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 141464, PROT_READ, MAP_SHARED, 4, 0) = 0x7fe867b6e000
munmap(0x7fe867b6e000, 141464)          = 0
close(4)                                = 0
newfstatat(AT_FDCWD, "/usr/lib/wsl/lib/libcuda.so.1.1", {st_mode=S_IFREG|0555, st_size=141464, ...}, AT_SYMLINK_NOFOLLOW) = 0
openat(AT_FDCWD, "/usr/lib/wsl/lib/libcuda.so.1.1", O_RDONLY) = 4
newfstatat(4, "", {st_mode=S_IFREG|0555, st_size=141464, ...}, AT_EMPTY_PATH) = 0
mmap(NULL, 141464, PROT_READ, MAP_SHARED, 4, 0) = 0x7fe867b6e000
munmap(0x7fe867b6e000, 141464)          = 0
close(4)                                = 0
newfstatat(AT_FDCWD, "/usr/lib/wsl/lib/libd3d12.so"

Activity

  1. PiotrCzapla commented on Jul 7, 2022

    @PiotrCzapla
    Author

    I've found a way to get the content of lib cache and the libcuda.so and libcuda.so.1.1 is not there, even though it is in the wsl folder.

    $ ldconfig -p| grep libcuda
            libcudart.so.11.0 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcudart.so.11.0
            libcudart.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcudart.so
            libcuda.so.1 (libc6,x86-64) => /usr/lib/wsl/lib/libcuda.so.1
    $ ls -la /usr/lib/wsl/lib/libcuda*
    -r-xr-xr-x 1 root root 141464 Jun 24 05:05 /usr/lib/wsl/lib/libcuda.so
    -r-xr-xr-x 1 root root 141464 Jun 24 05:05 /usr/lib/wsl/lib/libcuda.so.1
    -r-xr-xr-x 1 root root 141464 Jun 24 05:05 /usr/lib/wsl/lib/libcuda.so.1.1
    

    And here is how this works if you specify LD_LIBRARY_PATH

    $ LD_LIBRARY_PATH=/usr/lib/wsl/lib/ python
    Python 3.9.12 (main, Apr  5 2022, 06:56:58)
    [GCC 7.5.0] :: Anaconda, Inc. on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> from ctypes import cdll
    >>> cdll.LoadLibrary('libcuda.so.1')
    <CDLL 'libcuda.so.1', handle 55f038a352a0 at 0x7ff7bcdc68e0>
    >>> cdll.LoadLibrary('libcuda.so')
    <CDLL 'libcuda.so', handle 55f038a46510 at 0x7ff7bcd2eac0>
    >>> cdll.LoadLibrary('libcuda.so.1.1')
    <CDLL 'libcuda.so.1.1', handle 55f038a470a0 at 0x7ff7bcd2dc70>
    
  2. PiotrCzapla commented on Jul 7, 2022

    @PiotrCzapla
    Author

    For anyone facing the following issue:

    Could not load library libcudnn_cnn_infer.so.8. Error: libcuda.so: cannot open shared object file: No such file or directory130 00:00<00:00]
    Please make sure libcudnn_cnn_infer.so.8 is in your library path!

    when traing put LD_LIBARAR_PATH=/usr/lib/wsl/lib/:$LD_LIBARAR_PATH to your .bashrc . It should fix the problem.

  3. kryptan commented on Aug 27, 2022

    @kryptan

    I think the correct line to add to the .bashrc file is:

    export LD_LIBRARY_PATH=/usr/lib/wsl/lib:$LD_LIBRARY_PATH
  4. matsuu commented on Dec 2, 2022

    @matsuu

    try this:

    cd /usr/lib/wsl/lib
    # backup
    sudo mv libcuda.so.1 libcuda.so.1.backup
    sudo mv libcuda.so libcuda.so.backup
    # create symlink manually
    sudo ln -s libcuda.so.1.1 libcuda.so.1
    sudo ln -s libcuda.so.1 libcuda.so
    # update cache
    sudo ldconfig
    
  5. Lonas13 commented on Mar 31, 2023

    @Lonas13

    I think the correct line to add to the .bashrc file is:

    export LD_LIBRARY_PATH=/usr/lib/wsl/lib:$LD_LIBRARY_PATH

    THANK YOU, THIS FINALLY FIXED IT FOR ME!!!

  6. Abdulla060 commented on May 7, 2023

    @Abdulla060

    I think the correct line to add to the .bashrc file is:

    export LD_LIBRARY_PATH=/usr/lib/wsl/lib:$LD_LIBRARY_PATH

    I think this reply should be in the official docs

  7. fengyuexingzi commented on May 31, 2023

    @fengyuexingzi

    THANK YOU, THIS FINALLY FIXED IT FOR ME!!!

  8. ptn77 commented on Jun 3, 2023

    @ptn77

    Thank you very much! Cuda worked in VSCode running on windows, but in my WSL, I had to create the symbolic link in order for it to work on my Ubuntu WSL.

  9. microsoft-github-policy-service commented on Jun 6, 2024

    @microsoft-github-policy-service
    Contributor

    This issue has been automatically closed since it has not had any activity for the past year. If you're still experiencing this issue please re-file this as a new issue or feature request.

    Thank you!

  10. eric-gitta-moore commented on May 3, 2025

    @eric-gitta-moore

    I've found a way to get the content of lib cache and the libcuda.so and libcuda.so.1.1 is not there, even though it is in the wsl folder.

    $ ldconfig -p| grep libcuda
            libcudart.so.11.0 (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcudart.so.11.0
            libcudart.so (libc6,x86-64) => /usr/local/cuda/targets/x86_64-linux/lib/libcudart.so
            libcuda.so.1 (libc6,x86-64) => /usr/lib/wsl/lib/libcuda.so.1
    $ ls -la /usr/lib/wsl/lib/libcuda*
    -r-xr-xr-x 1 root root 141464 Jun 24 05:05 /usr/lib/wsl/lib/libcuda.so
    -r-xr-xr-x 1 root root 141464 Jun 24 05:05 /usr/lib/wsl/lib/libcuda.so.1
    -r-xr-xr-x 1 root root 141464 Jun 24 05:05 /usr/lib/wsl/lib/libcuda.so.1.1
    

    And here is how this works if you specify LD_LIBRARY_PATH

    $ LD_LIBRARY_PATH=/usr/lib/wsl/lib/ python
    Python 3.9.12 (main, Apr  5 2022, 06:56:58)
    [GCC 7.5.0] :: Anaconda, Inc. on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> from ctypes import cdll
    >>> cdll.LoadLibrary('libcuda.so.1')
    <CDLL 'libcuda.so.1', handle 55f038a352a0 at 0x7ff7bcdc68e0>
    >>> cdll.LoadLibrary('libcuda.so')
    <CDLL 'libcuda.so', handle 55f038a46510 at 0x7ff7bcd2eac0>
    >>> cdll.LoadLibrary('libcuda.so.1.1')
    <CDLL 'libcuda.so.1.1', handle 55f038a470a0 at 0x7ff7bcd2dc70>
    

    What I'm curious about is why only libcuda.so.1 is linked while libcuda.so is ignored. What's the magic behind this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions