Skip to main content

boost.python: os.environ and LD_LIBRARY_PATH

This article is again about programming language, so if you are not interested in this area, see you next time.

My first python project is a test suite. I like Martin Fowler's articles, so, naturally, my first program is a test. My project is usually started with test, documentation, and interface design, yet incremental.

This is about python's os.environ and LD_LIBRARY_PATH. This is a follow up of the boost.python's story. My development environment is Linux, but later I have a plan to go to Windows also.

In python interpreter, we can change the environment variables through os.environ. For example, if you want to change the LD_LIBRARY_PATH:

  os.environ['LD_LIBRARY_PATH'] = '/some/directory/lib'

I thought this is all for LD_LIBRARY_PATH, but I hit a wall. My department policy doesn't allow to have administrator right and Internet access to the developers. Therefore, if I want to use something not in my computer, for example, boost or python, I first need to ask to get the source code, and need to think how to install the software locally. Under this environment, the setting of shared library path is important.

My project has the following directory structure.

    test_x +--+ pymodule +--+ mypython_binding_module.so
           +--+ testbase +--+ test_run.py
           +--+ boost144 +--+ lib +--+ ...
                                  +--+ libboost_python.so

The problem here is mypython_binding_module.so depends on libboost_python.so. LD_LIBRARY_PATH is an environment variable.  Since environment variable is program global and has implicit effect, I usually try to avoid to use them. But this time I need it and I try to set this in the python interpreter only.

Python interpreter can recognize mypython_binding_module.so as a python module, so I can set sys.path to import this module. But, mypython_binding_module.so depends on libboost_python.so, this can not be done by sys.path. dlopen() does this task. Therefore, I need to set LD_LIBRARY_PATH to load libboost_python.so.

 I set LD_LIBRARY_PATH by os.environ, however, this does not work. I first suspect, python's os module only tells the current environment variables, but actually did not change them. However, manual said python changed them.

The problem was at the dlopen(). dlopen()'s manual said, this only looks up the LD_LIBRARY_PATH when the program starts. The process changes the LD_LIBRARY_PATH, it doesn't affect the dlopen()'s behavior. This is understandable behavior considering the security.

Once I understood this, I changed the LD_LIBRARY_PATH through os.environ, then I created a child process. In this child process, LD_LIBRARY_PATH's change is effective as expected.

Here, os.environ can change the environment variables, but the function takes it or not is depends on the function. This is a simple thing, but, I don't realize for two days.

Yet, the problem of setting library path seems a common problem. I looked into the documentation (e.g., site module, etc.), I might miss something since I am only a beginner of python.

Comments

Popular posts from this blog

Why A^{T}A is invertible? (2) Linear Algebra

Why A^{T}A has the inverse Let me explain why A^{T}A has the inverse, if the columns of A are independent. First, if a matrix is n by n, and all the columns are independent, then this is a square full rank matrix. Therefore, there is the inverse. So, the problem is when A is a m by n, rectangle matrix.  Strang's explanation is based on null space. Null space and column space are the fundamental of the linear algebra. This explanation is simple and clear. However, when I was a University student, I did not recall the explanation of the null space in my linear algebra class. Maybe I was careless. I regret that... Explanation based on null space This explanation is based on Strang's book. Column space and null space are the main characters. Let's start with this explanation. Assume  x  where x is in the null space of A .  The matrices ( A^{T} A ) and A share the null space as the following: This means, if x is in the null space of A , x is also in the null spa

Gauss's quote for positive, negative, and imaginary number

Recently I watched the following great videos about imaginary numbers by Welch Labs. https://youtu.be/T647CGsuOVU?list=PLiaHhY2iBX9g6KIvZ_703G3KJXapKkNaF I like this article about naming of math by Kalid Azad. https://betterexplained.com/articles/learning-tip-idea-name/ Both articles mentioned about Gauss, who suggested to use other names of positive, negative, and imaginary numbers. Gauss wrote these names are wrong and that is one of the reason people didn't get why negative times negative is positive, or, pure positive imaginary times pure positive imaginary is negative real number. I made a few videos about explaining why -1 * -1 = +1, too. Explanation: why -1 * -1 = +1 by pattern https://youtu.be/uD7JRdAzKP8 Explanation: why -1 * -1 = +1 by climbing a mountain https://youtu.be/uD7JRdAzKP8 But actually Gauss's insight is much powerful. The original is in the Gauß, Werke, Bd. 2, S. 178 . Hätte man +1, -1, √-1) nicht positiv, negative, imaginäre (oder gar um

Why parallelogram area is |ad-bc|?

Here is my question. The area of parallelogram is the difference of these two rectangles (red rectangle - blue rectangle). This is not intuitive for me. If you also think it is not so intuitive, you might interested in my slides. I try to explain this for hight school students. Slides:  A bit intuitive (for me) explanation of area of parallelogram  (to my site, external link) .