Running Two glibc Versions on the Same Machine (Safely) - A Practical Tutorial

If you’ve ever tried to run a newer ML binary on an older HPC cluster and hit something like:

…you’ve met the most “invisible” dependency in Linux: glibc (the GNU C Library).

On many clusters, the system glibc is intentionally old and tightly coupled to the OS. Upgrading it globally is risky and can break core tools. The good news: you can use a newer glibc side-by-side with the system one, without changing the system and without breaking your shell.

This post explains the trick: invoke your program with a different dynamic loader.

Why ldd --version lies (kind of)

Even if your IT team installs a new glibc under something like /opt/.../glibc/2.38, running:

ldd --version

will still show the system version (e.g., 2.28). That’s because:

So, don’t use ldd --version as your indicator of whether a new glibc is available.

The dynamic loader controls which glibc you use

When you run a Linux executable, the kernel usually hands it off to the dynamic loader:

That loader then finds and links:

So if you run your program through a different loader, you can make it use a different glibc for that process only.

That’s the “two glibc versions on one machine” trick.

Where glibc 2.38 is on our cluster (example: Rice CS Addis/Praia)

Installed paths:

Verify the loader version:

/opt/rice/libs/glibc/2.38/lib/ld-linux-x86-64.so.2 --version

If it prints 2.38, you’re good.

The safe way: run only the target program under glibc 2.38

1) Set a convenience variable

export GLIBC238=/opt/rice/libs/glibc/2.38

2) Run a program under the 2.38 loader

$GLIBC238/lib/ld-linux-x86-64.so.2 \
  --library-path $GLIBC238/lib:$GLIBC238/lib64 \
  /path/to/your_program [args...]

✅ This affects only that program (and its child processes).
✅ Your shell remains on the system glibc, so vi, gcc, etc. don’t randomly explode.

System Python (/usr/bin/python) is often tightly coupled to system libraries and may not behave well. The most reliable approach is a conda/mamba environment.

Activate your env, then:

export GLIBC238=/opt/rice/libs/glibc/2.38

$GLIBC238/lib/ld-linux-x86-64.so.2 \
  --library-path $GLIBC238/lib:$GLIBC238/lib64:$CONDA_PREFIX/lib \
  $(which python3) -c "import sys; print(sys.version)"

If that prints your Python version, you’ve successfully run Python under glibc 2.38.

To run a script:

$GLIBC238/lib/ld-linux-x86-64.so.2 \
  --library-path $GLIBC238/lib:$GLIBC238/lib64:$CONDA_PREFIX/lib \
  $(which python3) your_script.py

“I tried vi and it crashed” — should I worry?

Usually, no.

Two common gotchas:

  1. The loader does not search your PATH for executables.
    So ... ld-linux ... vi may not do what you think; use /usr/bin/vi.

  2. Even with the correct path, many system tools may break under a non-system glibc because they depend on other OS libraries built against the system runtime.

This is expected and is exactly why the per-program approach is recommended.

Debugging: confirm which libc.so.6 your program is actually using

If you want to prove it’s pulling the 2.38 libc:

LD_DEBUG=libs \
$GLIBC238/lib/ld-linux-x86-64.so.2 \
  --library-path $GLIBC238/lib:$GLIBC238/lib64:$CONDA_PREFIX/lib \
  /path/to/your_program 2>&1 | grep libc.so.6

You should see it referencing something under /opt/rice/libs/glibc/2.38/....

Troubleshooting checklist

1) “GLIBC_2.xx not found” persists

You might not be running the program under the loader. Make sure you’re using the ld-linux... invocation.

2) Missing conda libs (super common)

If you’re running conda Python, you almost always need:

3) libstdc++.so.6 / GLIBCXX_... not found

That’s a C++ runtime issue (not glibc). You may need to include the appropriate libstdc++ path (often inside your conda env) in --library-path as well.

4) Don’t export global LD_LIBRARY_PATH to glibc 2.38

This is how you break your session. Prefer the per-command --library-path.

Make it ergonomic: a tiny wrapper script

Create a helper script glibc238-run:

cat > glibc238-run <<'EOF'
#!/usr/bin/env bash
GLIBC238=/opt/rice/libs/glibc/2.38
exec $GLIBC238/lib/ld-linux-x86-64.so.2 \
  --library-path $GLIBC238/lib:$GLIBC238/lib64:${CONDA_PREFIX:+$CONDA_PREFIX/lib} \
  "$@"
EOF

chmod +x glibc238-run

Usage:

./glibc238-run $(which python3) -c "import sys; print(sys.version)"
./glibc238-run /path/to/your_program --args

Why this works (one-paragraph intuition)

Linux executables don’t “load glibc directly.” The dynamic loader loads glibc and everything else. If you run your program via a different loader and point it at a different library directory, you effectively swap the runtime only for that process, leaving the OS untouched. That’s how “two glibc versions on the same machine” is possible.

TL;DR

Use glibc 2.38 like this:

export GLIBC238=/opt/rice/libs/glibc/2.38
$GLIBC238/lib/ld-linux-x86-64.so.2 \
  --library-path $GLIBC238/lib:$GLIBC238/lib64:$CONDA_PREFIX/lib \
  $(which python3) -c "import sys; print(sys.version)"

If you hit a weird error, the fix is usually: add the right directories to --library-path, not “upgrade the system.”