Kernel compilation and a pesky mouse
This post attempts to illustrate how bug-reporting can lead to a solution and the adventures therein
Let me first describe the problem at hand. I have not so old optical Logitech mouse (purchased around 3-4 years back) and purchased quite expensively, paid around Rs. 500/- or so () at that time. Since around 3-4 months ago started noticing that specifically the GNU/Linux desktop was lagging.
I tried the same mouse with a Windows XP machine for confirmation and although there was the arrow/pointer going astray the issue was barely noticeable, somewhere somehow the Windows machine/software was keeping it quiet.
As Debian Squeeze was very much in unstable or development stage didn’t think it was a mouse/mice problem but thought perhaps it might be a bug. I was able to do my work with a work-around from the 80’s, going to VT1,VT2 and using all command line stuff. Then around 2 months back or so finally put it up as a bug . Bug 607242 . First it didn’t make the Debian-X mailing list and me not being aggressive did not push it. Then around a week later when still nothing had happened went around and saw who was the owner/maintainer of xserver-xorg-input-all by looking at the file/package changelogs and sent the developer a private mail asking about what/if any attention has been paid to the bug or its a user error.
This is when it was found that it was indeed some kind of oversight or a filter issue at Debian. The Debian developer/maintainer (sorry if I mix the two terms, I still haven’t got to understand how to know who is a debian developer or/and a Debian Maintainer although do know both have overlapping yet distinct roles and responsibilities) was helpful and he pinged back on the bug-list making it active once again. He also chatted with other developer/s and experts on my behalf and asked me to try couple of things to get at the bottom of things which I did. To put it simply on X the desktop simply froze, I was not able to get the pointer moved anywhere at all.
Then around a month later during the course of investigation, enter Mr. Jim Hill. I had not known Mr. Jim Hill before this but the man is/was persistent and he was able to dig out things as can be seen in later messages.
One thing lead to another and on 7th January Jim put up his first version of the mouse driver modified file on the bug.
Then on the 10th of January got an e-mail from Jim urging me to try out the patch. As I hadn’t done anything like that for years and was also not really sure of the code as I didn’t know Jim hence went around asking people to take a look at the patch in question to couple of mailing lists as a precaution. One of them being the local GNU/Linux list . Thankfully Rahul Sundaram (of Redhat fame) and who eats and sleeps GNU/Linux responded kindly which egged me to communicate with Jim about how to get this patch working on my system. I know Mr. Rahul informally and do trust his judgement so went ahead.
This is the gist of the things that Jim wanted me to do :-
[… save the email containing the patch to some file …]
$ sudo apt-get install build-essential
$ sudo apt-get build-dep linux-2.6
$ mkdir linux && cd linux && git clone git://github.com/mirrors/linux-2.6.git
[… this is kinda big, ~450MB …]
$ cd linux-2.6
$ git checkout -b mykernel v2.6.37
$ make menuconfig
[… hit tab and enter, to accept defaults for new options …]
$ patch -p1 <“file with the patch email, patch will ignore the cruft”
$ make deb-pkg
[ … this’ll take a while … ]
$ sudo dpkg -i ../*.deb
Basically going the git way and getting the kernel git image, checking out the 2.6.37 kernel (naming it mykernel), patching the kernel and installing the damn thing.
Now this was a big commitment and while it seems and sounds simple for a guy who had done kernel compiles say 5 times or something in the last 5 years was not really sure if I could pull it off. The reason being simply lack of know-how and no idea what I was doing.
The kernel compilations I had done in the past were pretty complicated in the past and I hadn’t really known that nowadays with git the whole process had become so simple.
Anyways over 2 days downloaded the whole tree, checked out the kernel, put the modified psmouse. One of the things I had to modify was my sources.list as hadn’t put any deb-src lines in my sources.list (hadn’t needed it) .
Then chatted up with Jim in early hours on IRC and went the whole way in. Booted into 2.6.37 and still the proposed fix didn’t do any better. One of things it did though was, I was able to get some better logging behavior than before
$ sudo tee /sys/module/psmouse/parameters/filter <<<55
During the course of conversation back and forth also encountered ‘Harvard’s Law of Animal Behavior’ which simply says you cannot predict when things will happen. Sadly there is no wikipedia article on that subject but that’s a subject for another time. Put simply, what that states is you can never be sure of what you are looking for.
For instance it took me another couple of days of playing around with the mouse and my desktop before I got a group of bad messages in my syslog which I could send to Jim for analysis.
Then just a week before, got another mail from Jim as he had taken a break and looked again at the patch and found he could do better.
Note:- One thing to note is that the kernel compilation with the modified mouse took around 4 hours to compile, now no idea why it took/takes so much but that it did/does. Also a kernel compilation took/takes around 4-6 GB of data space for compilation which while in the beginning didn’t look/think about it later on started to bite me.
drivers/input/mouse/bcm5974.ko: final close failed: No space left on device
make: * [drivers/input/mouse/bcm5974.ko] Error 1
make: * [modules] Error 2
make: * [deb-pkg] Error 2
make: * [deb-pkg] Error 2
Jim’s instructions :-
from the kernel git toplevel, if you still have the build lying around the just-this-module remake will go very quickly:
$ git checkout v2.6.37 drivers/input/mouse/psmouse-base.c drivers/input/mouse/psmouse.h
$ git checkout v2.6.37 # <– for safety, be sure there are no other changes
$ patch -p1
$ make drivers/input/mouse/psmouse.ko
$ sudo rmmod psmouse && sudo insmod !$
This was also new to me and it took me almost 3-4 days as had taken out the git kernel and had never played with insmod and rmmod. Simply stating it is a way/method to remove and insert modules in a running system.
The way I understand it, it is something like drivers but unlike in Windows where drivers are permanent, that once you install them they are fixed in sessions and unless you install/upgrade or remove them, they are there in GNU/Linux you can temporarily install modules to check out how a new driver is functions and if it works out fine for a session then do it permanently.
Finally a big +1 for GNU/Linux from Windows . Apart from having newer shinier kernels one more thing I can share/evangelize about GNU/Linux 🙂
Note: The psmouse compilation took about 40 minutes and this is not even an aged machine. Although what was interesting is to see a variety of use-cases and scenarios for which the mouse driver compiles. For e.g. Wireless, multimedia mouses, ps/2, power,net as well as multiple mouses. I was amazed just to see the kind of complexity that the humble mouse is capable of during compilation.
Well, for a lot of people it would be termed as masochism, for me it was fun and a bit of learning. The one thing that both of us i.e. me and Jim (and perhaps Jim more) had to learn is to take it a notch lower/backwards.
The idea of the whole exercise is/was two fold :-
a. Get this patch or patches done upstream so just not me but other people can also benefit from it.
b. Learn how people do actual kernel work (this was the effect really not the goal though).
So if I as a non-technical person could get it done (with help of course) the guys who are supposed to have learnt all this, the Computer Science chaps they should have been wizards by this time. Attaching Jim Hill’s psmouse patch as well.
An Update :- While I was busy writing the post didn’t realize that I had failed to give a comparison of the mouse’s performance before and after the patch had been applied. It made a marked difference in performance. I now have a usable desktop. Remembering back I *think* I had been hit with the same issue in Ubuntu about a year but as I had other hardware issues/problems (failing hard disks, motherboard issues etc.) as well I could not distinguish between the two. As this is/was brand new hardware, bought components about 5 months back was able to zero into the faulty component/issue.
Now as far as Windows to Debian is concerned it would not be a fair comparison hence wouldn’t know how to state that scenario.
In fact, the way I see it, it is only through such kind of improvements that GNU/Linux can fight with Windows on the desktop. In fact if I hadn’t known about dmesg and bug-tracking and everything I would have been as clueless as the next guy as to what’s wrong and would have lumped GNU/Linux distributions as something that doesn’t work.
Now whether Jim’s work gets accepted upstream or not is another cup of tea altogether as they have their own standards and metrics (I guess) as to what is acceptable or not in the kernel.