Rambles around computer science

Library path strangenesses

It's about time I began sharing the hard-learnt arcane development knowledge I've managed to pick up. One of the most annoying “features” I've had to reverse-engineer is in the behaviour of Linux's dynamic linker, ld-linux. I have a lot of directories in my LD_LIBRARY_PATH environment variable.

srk31@font:~/scratch/kde$ echo 
/home/srk31/scratch/opt/lib:/home/srk31/scratch/kde/lib:/usr/lib/mysql:/usr/lib/opensync/:/usr/lib/o
pensync/formats/:/usr/lib/opensync/plugins/:/usr/lib/qt-3.3/lib:/usr/lib/wine/:/home/srk31/opt/lib:/
lib:/usr/lib 

They all get searched as you'd expect. Each directory is expanded to include searches for various architecture-specific variant subdirectories like tls, sse2, nosegneg etc.

srk31@font:~/scratch/kde$ LD_DEBUG=libs ldd bin/konsole 2>&1 | head
      5666:     find library=libtinfo.so.5 [0]; searching
      5666:      search path=/home/srk31/scratch/opt/lib/tls/i686/sse2/nosegneg:/home/srk31/scratch/
opt/lib/tls/i686/sse2:/home/srk31/scratch/opt/lib/tls/i686/nosegneg:/home/srk31/scratch/opt/lib/tls/
i686:/home/srk31/scratch/opt/lib/tls/sse2/nosegneg:/home/srk31/scratch/opt/lib/tls/sse2:/home/srk31/
scratch/opt/lib/tls/nosegneg:/home/srk31/scratch/opt/lib/tls:/home/srk31/scratch/opt/lib/i686/sse2/n

(snipped some -- view source for more)

osegneg:/home/srk31/opt/lib/i686/sse2:/home/srk31/opt/lib/i686/nosegneg:/home/srk31/opt/lib/i686:/ho
me/srk31/opt/lib/sse2/nosegneg:/home/srk31/opt/lib/sse2:/home/srk31/opt/lib/nosegneg:/home/srk31/opt
/lib            (LD_LIBRARY_PATH)
      5666:       trying file=/home/srk31/scratch/opt/lib/tls/i686/sse2/nosegneg/libtinfo.so.5
      5666:       trying file=/home/srk31/scratch/opt/lib/tls/i686/sse2/libtinfo.so.5
      5666:       trying file=/home/srk31/scratch/opt/lib/tls/i686/nosegneg/libtinfo.so.5

Notice that in my LD_LIBRARY_PATH, /usr/lib and lib are right at the end. Once I tried tweaking the ordering so that some paths came after these -- they were “backup” libraries I'd compiled myself in case the machine I was using didn't have them, but I wanted to pick up any local ones in preference. Unfortunately, this doesn't work. ld-linux will ignore all directories after the first directory it encounters that is part of the “system library path”.

srk31@font:~/scratch/kde$ LD_LIBRARY_PATH=/usr/lib:${LD_LIBRARY_PATH} LD_DEBUG=libs ldd bin/konsole 2>&1 | head
      5687:     find library=libtinfo.so.5 [0]; searching
      5687:      search path=/usr/lib/tls/i686/sse2/nosegneg:/usr/lib/tls/i686/sse2:/usr/lib/tls/i68
6/nosegneg:/usr/lib/tls/i686:/usr/lib/tls/sse2/nosegneg:/usr/lib/tls/sse2:/usr/lib/tls/nosegneg:/usr
/lib/tls:/usr/lib/i686/sse2/nosegneg:/usr/lib/i686/sse2:/usr/lib/i686/nosegneg:/usr/lib/i686:/usr/li
b/sse2/nosegneg:/usr/lib/sse2:/usr/lib/nosegneg:/usr/lib                (system search path)
      5687:       trying file=/usr/lib/tls/i686/sse2/nosegneg/libtinfo.so.5
      5687:       trying file=/usr/lib/tls/i686/sse2/libtinfo.so.5
      5687:       trying file=/usr/lib/tls/i686/nosegneg/libtinfo.so.5
      5687:       trying file=/usr/lib/tls/i686/libtinfo.so.5
      5687:       trying file=/usr/lib/tls/sse2/nosegneg/libtinfo.so.5
      5687:       trying file=/usr/lib/tls/sse2/libtinfo.so.5
      5687:       trying file=/usr/lib/tls/nosegneg/libtinfo.so.5
      5687:       trying file=/usr/lib/tls/libtinfo.so.5

I have no idea why it does this, but would guess it's intended behaviour for some reason. It's annoying though.

There's something about CS

It's not always a good thing though. We seem to have less teaching budget than other sciences, which might explain why practical classes are too few (though this appears to be improving slowly). In the research world, here's another peculiar way in which we lose out: unlike many other scientific disciplines, computer scientists are expected to typeset their own publications. As you probably know, this is not fun when you're camera-readying something for an exacting publisher (who may well not have bothered to provide a sufficiently complete LaTeX style file for their typesetting rules). Why do we have this responsibility, when in other disciplines (so I'm led to believe) the publishers take on much more responsibility for typesetting? Presumably because historically, computer scientists were almost universally also computer technicians of some sort or another, in the same way that the Computing Service here at the University of Cambridge used to be the same organisation as the Computer Laboratory. It's no longer true, of course, so a bit annoying that publishing anything means hours of LaTeX-twiddling pain.

There's a similarly bizarre phenomenon when doing any sort of practical computer science research: you are effectively your own technician. You're required to have a huge range of practical skills which you will not have learnt from being taught, nor even from practical classes. I'm thinking mainly about programming and Unix skills here. Everyone coming into my research group seems to have them, or at least the beginnings of them. I have them too, so I'm not complaining. But I imagine practically nobody learnt them by taking a degree, or even by doing a job. You could probably reach a reasonable level by combination of introductory programming courses and summer work, but it seems like a stretch. Instead, I'd bet that everyone coming into our group learnt them in their spare time by being a bit nerdy.

What other science displays this phenomenon? It can't be a good one, since it creates a big barrier to entry for people who would otherwise be good computer scientists, but aren't quite nerdy enough to have spent hundreds of hours of their life hacking around. This is perhaps blameable for the poor showing of women in CS research. Clearly, any practical CS researcher has to learn this stuff at some point, but given the high variance in nerdiness, will the non-nerds ever get enough experience to reach the required level? Summer jobs aren't long enough to teach programming -- I guess that any summer job which involves programming will expect you to have the skills already. So, clearly the universities have a big responsibility. But then, practical experience is a different animal from principled understanding. We clearly need some sort of scheme which ensures that incoming non-nerdy (but interested and bright) students can acquire the practical skills they need to do practical CS research, most likely by a combination of degree courses, summer work and summer/winter schools. I don't really have a solution, except “more of that”. Maybe systems research departments should have more encouragement or obligation to take undergraduate interns, and give them coaching (by paying PhD students, most likely)?