Better package management

Hi all,

Today’s post would be talking about making a case for delta-debs and LZMA compression in  .deb packages making for smaller and more efficient use of machines as well as bandwidth.Also an introduction to a game I’ve been playing for almost the whole of last week “My Tribe”. I hope to carry out a whole review of the game next week when I’ve played the whole game.

The only thing that really has been pissing off is the updates and the bandwidth required to serve them. Take this e.g.

$sudo aptitude safe-upgrade
Reading package lists...
Building dependency tree...
Reading state information...
Reading extended state information...
Initializing package states...
Writing extended state information...
Resolving dependencies...
Resolving dependencies...
The following NEW packages will be installed:
libgnome-desktop-2-11{a} linux-headers-2.6.28-3{a}
linux-headers-2.6.28-3-generic{a} linux-image-2.6.28-3-generic{a}
linux-restricted-modules-2.6.28-3-generic{a}
The following packages will be REMOVED:
linux-headers-2.6.28-2{u} linux-headers-2.6.28-2-generic{u}
The following packages will be upgraded:
capplets-data compiz compiz-core compiz-fusion-plugins-main compiz-gnome
compiz-plugins compiz-wrapper cups cups-bsd cups-client cups-common
cupsys cupsys-client cupsys-common evince evince-dbg firefox firefox-3.0
firefox-3.0-branding firefox-3.0-gnome-support firefox-gnome-support
frozen-bubble frozen-bubble-data gnome-about gnome-applets
gnome-applets-data gnome-applets-dbg gnome-control-center
gnome-desktop-data gnome-panel gnome-panel-data gnome-panel-dbg
gnome-screensaver gnome-settings-daemon gnome-utils guile-1.8-libs
icedtea6-plugin jockey-common jockey-gtk klogd libakonadiprivate1
libcups2 libcupsimage2 libcupsys2 libdecoration0 libdeskbar-tracker
libeel2-2 libeel2-data libgdict-1.0-6 libgnome-window-settings1
libgtkhtml-editor-common libgtkhtml-editor0 libgtkhtml3.14-19
libgtkhtml3.14-dbg libmono-addins-gui0.2-cil libmono-addins0.2-cil
libnautilus-extension1 libpanel-applet2-0 libperl5.10 libtracker-gtk0
libtrackerclient0 linux linux-generic linux-headers-generic linux-image
linux-image-generic linux-restricted-modules
linux-restricted-modules-common linux-restricted-modules-generic nautilus
nautilus-data nautilus-dbg openjdk-6-jre openjdk-6-jre-headless
openjdk-6-jre-lib perl perl-base perl-modules python-gnome2-desktop
sysklogd tracker tracker-search-tool update-manager update-manager-core
xulrunner-1.9 xulrunner-1.9-gnome-support
86 packages upgraded, 5 newly installed, 2 to remove and 0 not upgraded.
Need to get 138MB of archives. After unpacking 94.0MB will be used.
Do you want to continue? [Y/n/?]

I have used the command-line version to show the difference. The actual diff. between the updates would be closer to 94 MB but to do so the package-manager (aptitude in this case) wants to download 138 MB of updates (this is on Jaunty or 9.04) and this is really lame. There are couple of solutions if both implemented it would do this sort of package upgrade at perhaps 60% or even less bandwidth then so far.

The first part of the solution is using delta-updates (more popularly known as delta-debs) . This has quite a bit of upside if implemented.

less mirrors bandwidth usages with a possible effects such as addition in number of mirrors increasing :-

Mirrors take quite a bit of hit whenever new releases are happening and they are not able to handle it as full packages are downloaded. This severely hampers serving more customers at the same time.

Possibility for people with slow Internet connections or time and/or bandwidth based :- If you look at most of the broadband packages given by telcos here they are mostly cost or bandwidth based. For e.g. look at the packages offered by BSNL , one of the larger ISP’s which has almost 50% of market share as per TRAI (the Indian regulator) figures . One can find the tarrif listed at BSNL site . As can be seen most of the packages are for time/cost/data based rather than unlimited. I’m sure this would be the case with other developing countries as well.

A change/push towards delta style of packaging would make it easier for having quicker updates as well as backports from package managers viewpoint as well as they should only need to make a debdiff and a .dsc file.

Let’s take a look at what other people are doing. SUSE (an .rpm based distribution) has been playing with delta-updates for almost 4 years now while Fedora is going to be doing this in F11 by name of Presto (another link).

Its also not as if this is being proposed for the first time. The developers are looking at it but dunno when it will get implemented. See the apt-sync page, APTPackage Deltas page as well as Debian’s take (slightly older) on the same. I just wish this gets implemented as soon as possible. All class of users (specifically business users would be much more positively affected if this gets done) . Somebody also gave some of the links given here in brainstorm as well.

The other part of the scenario is having LZMA or gzip+zsync or bzip2 compression while making packages rather than .gzip which is currently the norm. That would save another few precious bytes. Please see the ubuntu wiki page on the same as well as on in blog world.

The other thing I’ve been doing for almost a week and a half now is playing a game called My Tribe released by grubbygames . I won’t say much other than I’ve been enjoying the game for quite sometime now. Would give a review of the game next week (when I’m all played out) . That’s all from me for the moment.