If you’ve used Linux for more than ten minutes, you’ve almost certainly come across the nightmare that is package management. You know what I mean – dependancy hell has become legendary and it’s no exagguration to say that one of the most offputting aspects of Linux for a new user is the lack of InstallShield type 3 click installs. This article looks at how we ended up in the quagmire of RPM and dependancy hell, and then moves on to talk about a possible solution in the form of autopackage. It takes a high level overview of how autopackage works and what it’s capable of. If you want more technical details, check out the website. Finally, this article assumes only that you’re interested, not that you have any Linux experience.
First off, I’m not going to assume you’ve used Linux before, so I’ll start with a few definitions:
- RPM: The Redhat package manager. Despite the name, it’s
used by many Linux distributions and is probably the most popular
package manager statistically speaking today. Unfortunately,
RPM has many faults, perhaps the most important being that it was
never designed for internet based distribution of software for Linux
in general. What this means is that it’s optimized for transferring
large amounts of software to an OS from a CD or Redhat update
site. Because of the lack of flexibility in the RPM file format,
various distros have tweaked it and so forked the format – if you
want to install an RPM you’d better be sure it’s designed for your
distro. Only the very simplest of packages can be realistically
distributed in this way by third parties. To make matters worse
RedHat often gratuitously break backwards compatability. - Dependancy hell: Due to the non commercial nature of
Linux, packages often have lots of dependancies. These are other
packages that must be installed before that package can be
installed, we say one depends on the other. All operating
systems have dependancies, but Linux software has far more than
most. Dependancy hell arises when you want to install a package, but
your system doesn’t have the required dependancies present. Raw RPM
cannot fix this itself – you must manually track down and install
those packages. Just pray that there are no sub-dependancies as
well! - APT: Advanced Packaging Tools, originally built for
Debian and later ported to RPM, apt can automatically resolve
dependancies. Installing software with apt is easy, even easier than
on Windows – you just type in the name of the software you want, for
instance withapt-get install gimp. Unfortunately, apt
requires centralised repositories to be maintained for every distro
(and often, every subversion), and as such often the software you
want is simply not available in the repositories, or it is present,
but its dependancies are not. Although Debian has many thousands of
packages in its repositories, the manpower required to keep them all
up to date means the latest versions of the packages are often only
available in Debian some time after they have been
released. emerge is a similar idea, but tied to Gentoo. - RPM Hell: another type of unpleasantness that is caused
by the type of package managers in use today on Linux. If the
software you want to install isn’t available as a package for your
distro then all is not lost, you can just compile it from the source
(which unless you use a ports based distro a la Gentoo has its own
set of problems). Unfortunately, you just entered the twilight zone:
RPM (and also the debian system) assume that you use them to install
everything on your computer. Because often packages are not
available of course this doesn’t happen, but then RPM is not aware
of the presence of those packages and will refuse to install other
packages that depend on them without manual overrides. Hacks around
this problem exist (see checkinstall) but they only partially solve
the problem
Having witnessed several of my friends try Linux only to give up in
disgust after spending several hours attempting to do something as
trivial as upgrade Mozilla, I resolved to do something about the
problem. So, what issues do we face, and how to other operating
systems fix these issues?
The first one is that RPM doesn’t posseses enough flexibility to
deal with the myriad little differences between distributions. An RPM
is basically made up of a list of files and a set of pre and post
install scripts. The list of files is always relative to root, so if
your distro does not use the exact same filing system layout as the
one it was built for, the RPM will not install correctly. Another
example of distro differences would be the “install-info” script,
which is responsible for integrating TeXinfo manuals into the system:
Debian has its own version which is incompatible with the Free
Software Foundation version used in most other forms of Linux.
Another issue, perhaps one of the biggest, is that there are no
standard mechanisms for naming packages. Because of that, an RPM
designed for one system will know dependancies by different names to
that of another. The difference between package “python-dev” and
“python-devel” may be small, but it’s enough to break
compatability.
Finally, you have differences in the compiled binaries
themselves. A KDE application compiled for Redhat 7.3 will not install
on Redhat 8 as they use different versions of gcc, and the binaries
produced by these compilers are not compatible. RPM has no way of
dealing with this other than to force people to have 2 different
packages for the 2 different versions.
A bit of a mess isn’t it? If you’re a Windows or Mac user you’re
probably feeling a bit smug right now, but actually there’s no reason
to be. Windows and MacOS both have ways of dealing with these issues,
and neither of them are particularly elegant. On Windows, we are all
familiar with the name InstallShield. The standard mechanism for
installing software onto Windows has for many years been to create an
installer executable that the user runs and then extracts the program
from itself and performs any needed setup. The installer checks
dependancies (usually different versions of MS dlls) and often
contains copies of them just in case they aren’t present. Finally it
integrates the program with the environment (often too well!) by
creating menu entries, app associations and so on.
Why does this system suck? Well, from the point of view of the
user, it doesn’t, the experience is simple, fast and effective. Behind
the scenes however, things are rather ugly. For starters, modern
installers are complex beasts – InstallShield is actually a complete
scripting language and all the code necessary to interpret these
scripts, create the user front ends etc are shipped with the Windows
app in every single case. This is pretty wasteful, and the need to
ship all the DLLs an app might require just in case they are not
present on the system also adds a lot of overhead.
Installer executables bring with them their own set of
problems. The user interface is often inconsistent, and they are extremely
hard to automate. I once spent a week at a company in which all I did
was travel to every computer and put in the IE6 upgrade CD for each
one. There was almost no good way of transparently deploying
applications. Although products do exist that automate this on
Windows, they usually work by watching what the installer does then
“rebuilding” the installation from scratch which can lead to all kinds
of nasty issues. The current system was never really designed as such,
in the absence of any solution provided by Microsoft companies sprung
up to fill the gap. Windows is now moving to an appfolders style
system whereby dependancies can be sideloaded in an attempt to avoid
dependancies also.
Does the Mac fare any better? Unfortunately not. MacOS X has (in theory) totally eschewed installers in favour of App Folders, which are specially marked directories, the idea being that you simply drag and drop the app into the Applications directory. To uninstall, just drag it to the wastebasket. This is good UI, but bad everything else. By eliminating (un)install logic, it becomes extremely hard to check for dependancies, to provide system components, to clean up configuration changes, other application metadata and so on. Although technically apps can have dependancies via bundles, the lack of OS side install logic means that in practice it’s not possible to install these bundles into the OS if required – the user must do it for the app. As such, apps can only rely on funtionality that was known to ship with the operating system. That suits Apples goal of selling more copies of OS X, but is rather limiting for the user. By trying to hide the problems dependancies pose all together, it creates bigger problems further down the line. As such, some Mac apps ship with install apps anyway which rather defeats the point. Note that my beef with appfolders are more to do with the way they have been implemented in NeXTStep – you can use autopackage to install apps into an appfolders style arrangement (for instance i test with
/apps/packagename) and one day there may well be a VFS plugin that lets you view installed packages as folders/directories. I think it is highly unlikely that you’ll ever be able to just drag app directories off a CD onto the system however.
So does Linux have the right idea? Yes, it does. Operating systems
are just that – systems, systems that are made up of multiple
components. In the absence of centralised control and the use of
parallel development, properly managing dependancies and system
differences is a must. The problem is that we don’t do it very
well. So how can we improve?
I think the answer lies in autopackage, a project I started work on
over 6 months ago in response to these issues. It uses a variety of
ideas and approaches to solve the problems posed by Linux package
management.
- Distribution neutral: unlike many package managers, autopackage
was not developed by a distribution project. As such, it is
completely distribution neutral both in technical design and
political stance. You can build an autopackage once, and it will (in
theory 🙂 install on any distro, assuming it’s not totally off this
planet. - Designed for flexibility: In a similar vein to
InstallShield a .package file is script based rather than table
based. The install scripts, backed up by a large number of provided
library funtions, are easily capable of dealing with the multitude
of differences between distributions. - Net based: RPM was designed to manage everything a distro comes
with, from the kernel to the Solitaire games. In contrast,
autopackage is designed for net based distribution of software. It’s
not designed to install new kernels, or to be used exclusively by a
distro (maybe one day it would be capable of this too, but it’s not
a design priority). As a part of the project we will be constructing
the autopackage network, a network of resolution servers that
work like DNS to convert root names (see below) into URLs for where
the packages can be downloaded. By turning our back on the idea of
apt-style massively centralised repositories, we hope to allow the
network to scale. If a programmer creates a piece of software, they
don’t have to wait for somebody from the network to create a package
and upload it for them, they can create it themselves and then
register their own node of the network from which they can plug in
any packages they may create - Global names: from the start, autopackage was designed to allow
packages to have multiple names. All packages must have a root name,
to which other names can map to. A root name looks something like
this: “@gnome.org/gedit/2.2”. As you can see, rather than creating a
new naming authority, it leverages off the DNS system. As most
projects these days have a website, they can easily be assigned a
root name. Other, more human friendly names include short names,
which are rpm style (for instance “gimp”, “gnome2” or “xchat-text”),
and display names, which describe in the users native language
briefly what the application is, ie “Evolution Mail
Client”. - Database independant dependancy management: the approach
autopackage takes to dependancies is similar to the configure
scripts we all use, in that it checks the system directly for the
presence of the dependancy, ie for libraries it will check the
systems linker cache. I won’t go into the exact details of this
system, you can find out more information from the website. - Front end independance: right now we have only a terminal front
end, and very pretty it is too. As we work towards 0.3, we’ll be
developing a graphical front end based on GTK. We’ll also be adding
a front end that records your choices and can then play them back,
allowing seamless automation of installs. By insulating packages
from the details of how they interact with the user, packages are
simpler to build and we can get much better integration with the
users host environment. The best is chosen automatically, so if you
install the package from the command line it’ll use the terminal
front end, and if you install it from Nautilus or Konqueror (visual
file managers) it’ll use the graphical front end. - Automatic and easy: installing a .package file is simply a
matter of running it. If you’ve never used autopackage before, the
needed files will be downloaded and setup for you. Packages have
minimal bloat because of this, so far it runs to about 11kb.
So is this just fantasy, a pipe dream? Not at all thankfully, a few
days ago we released 0.2 which can build, install, verify, query and
uninstall packages in a distro-neutral fashion. We’re a long way from
having a complete solution to the problem, but let’s hope that in a
few years the days of RPM hell will be long forgotten.
About the Author:
I’m Mike Hearn, an 18 year old from England who has been using Linux for about a year. By day I work for what was Ministry of Defence research, and by night I hack on free software (when i’m not out with friends ;). I also help run the theoretic.com website and am a chronic daydreamer 🙂
I hope AutoPackage will become atruly great project with wide spread supprot and nice guis
These guys are doing an awesome job!
There just has to be something better than RPM, I hope you guys really manage to move things in the right direction.
Looking at it superficially, the autopackage scripting component does sound a lot like the rpm SPEC file though…
There just has to be something better than RPM…
There is. Debian packages.
Looking at it superficially, the autopackage scripting component does sound a lot like the rpm SPEC file though…
Yeah, autopackage does have some similarities to RPM (i never said everything about RPM sucks, we borrow/steal ideas from all over the place). Unlike RPM though it’s not table based, an RPM has preinstall, files, postinstall – we just have an install script. We also provide a large library of functions that can be used to easily install files and perform installation tasks – that’s really what autopackage is all out, the function library. RPM doesn’t really provide anything similar and where it does they take the form of macro expansion.
This was a very interesting article Mike.
It sounds a bit like Slackware’s way of installing packages (at least as it has been described to me. I haven’t used Slackware before).
I wish all who are working on the project luck. It sounds like a nice solution to many of the problems we see today.
… people working for Debian, Gentoo, Mandrake, RedHat, United Linux and the LSB could get together and design the NGPM (Next Generation Package Manager) that would be build on the strength of the existing package managers, allow for binary as well as source based distros (or both) and then be officially adopted by ALL these distros.
One has the right to dream…
/Fred
So it’s 2 AM where I’m at right now, and I just finished my first forray in to installing GCC by RPM. This being my first try at a dependancy nightmare situation, it took me about an hour and a half just to download and install all the packages GCC needed. Anyone remember the joying of typing “rpm -i <file.rpm>” and, for the first time ever, after hours of rpmfind.net and google.com, watching it do nothing but spit you back out on to the sweet, merciful command-line? I’m feelin’ goooood right now
.
Still, though, g++ doesn’t know what this “fstream” thing is, let alone “iostream.” what a fool I was to think hours of toiling would bear fruit! Bring on the packaging alternatives! This is a part of OSS that could use some competition!
I quite like the URPMI used on Mandrake, not only allows very easy package installation but parallel installation across the network, something Linux is a little bit behind on. A good GUI based around this would be good.
I think decentralised package mangeement using a p2p network style is a very good Idea.
Another One is to make aLL linux station that are willing to to become a node.
Another One is to be able to be able to produce every package you’ve installed from your installed station (useful for being a node)
(gpg signature is needed for distribution)
A Gui for producing package is essential.
An import Tool (from deb, rpm to autopackage)
An integrated bug report tool for each package that send the report to the good person (improving bug reports improve softwares)
Data to be installed must be downloaded separatly from the package scripts.
You download script, it checks what it needs, then download it.
(not like urpmi that get a huge list, then use it for requesting sofware needed. the package should be able to resolve thos)
A Security graduation notation for Package (installing a new kernel is more dangerous than a new text editor…)
Very good initiative !!
Hi there,
the article really sound great, there are very nice ideas behind it, espacially I love the idea with DNS-like servers.
Now some questions:
1. If this system available only for binary distribution or for source distribution, or for both?
2. What about different plattforms? I mean can this system afford packages which suits to your system?
3. What about mirrors? If a programmer writes a great programm and every “DNS”-entry shows on his repositry, his server will be very soon out of bandwith.
4. What about uninstalling the apps?
I wish you good luck and I really hope that a lot of distris will integrate autopacking as soon as it is stable.
basically it boils down to either being lazy or knowing some stuff. if youre lazy then use debian or gentoo, which have good package management systems and will resolve the “dependancy hell” for you.
if youre not lazy, and have half a brain, then you can compile everything yourself. you’ll find that most packages use the simple autoconf/make system which allows me to install new packages in no time at all, and i have complete and total control over their installation.
does autopackage allow installation into a home directory for use on systems where i am not sysadmin? i dont think so.
linux without compilation is like a kitchen without the fire. mmm… salad…
If I had broadband, I’d try gentoo. If that failed to impress, debian. Buying a CD every few months is easier than downloading stuff over 56K.
You learn to live with the fact that everything on your system is months old until you wipe your system to install a newer RPM version…
(Yes, I’ve tried upgrading, it doesn’t work)
I think decentralised package mangeement using a p2p network style is a very good Idea.
Another One is to make aLL linux station that are willing to to become a node.
We’re not going to be building a P2P network anytime soon, imho they are overrated. Bandwidth usage is mostly fixed by the international mirror network – for instance the debian apt repositories are mirrored throughout the world. This doesn’t cost anything, ISPs and network operators run mirrors themselves to cut down the bandwidth usage on outgoing pipes.
A Gui for producing package is essential.
An import Tool (from deb, rpm to autopackage)
An integrated bug report tool for each package that send the report to the good person (improving bug reports improve softwares)
Data to be installed must be downloaded separatly from the package scripts.
You download script, it checks what it needs, then download it.
Wow, lots of ideas there. GUIs for package production would be nice, but it’s not a priority – developers already deal with text files all the time so getting them to produce a specfile (or soon a specdir) isn’t so bad. If you want to contribute one though….
Integrated bug reporting is something we’ve considered, packages have a metadata entry for bug reports (ie bugbuddy integration etc). It’d be nice for bugzilla to get a web service interface so we can do it automatically.
GPG integration is something we’ve thought about. Not sure how to do it though – in a distributed network who signs the code, and how do you get the keys? ie if the website is compromised then anything on it is compromised too.
Cascaded downloads is how it works currently yes.
1. If this system available only for binary distribution or for source distribution, or for both?
At the moment it’s targetted at binary packages. It’s pretty flexible though, you could have the compile/make/make install sequence in the script instead of installLib, installDesktop etc, it’s just not something we’ve tried yet.
2. What about different plattforms? I mean can this system afford packages which suits to your system?
Not sure what you mean here. AP is for Linux only (other forms of unix are all centrally controlled so they don’t have version fragmentation to the same extent). It will be able to deal with multiple CPU architectures in the future when we implement package merging.
3. What about mirrors? If a programmer writes a great programm and every “DNS”-entry shows on his repositry, his server will be very soon out of bandwith.
The network is made up of namespace resolution servers (probably actually just cgi scripts). So for instance, if I attempt to resolve “gimp” it’ll be turned by a root server into “@gimp.org/gimp”, resolving that will result in a redirect to a resolution server on say “http://gimp.org/packages.net“ which resolves it to “@gimp.org/gimp/1.4” and then the server is called again to resolve it into “http://mirror.ac.uk/whatever“. The design of the network is still forming in my head, I think stuff like geographic location will be passed along with the queries so the resolution nodes can select the best mirrors (assuming there are any).
4. What about uninstalling the apps?
Works as you’d expect. Run “package uninstall foobar” to see this.
if youre not lazy, and have half a brain, then you can compile everything yourself.
I have at least half a brain, but I am lazy. Compiling stuff from the source is a pain in the ass for advanced users, experience doesn’t make it any less of a pain in the ass, and for newbies it’s a usability killer. Even with emerge/ports it still takes forever.
does autopackage allow installation into a home directory for use on systems where i am not sysadmin? i dont think so.
It does. In fact this is usually how I test it. When you install the foobar package you will be asked for a prefix, enter for instance /home/mike/tmp/foobar and hit enter, that directory will be created and the files installed there. Note that files will be placed and changed outside that prefix for instance for desktop menus and the linker cache, but this is currently unavoidable.
Non-root installs are something we have been designing into the framework, and we have a good idea of how we’re going to do it, but right now we have other things to do first. In particular it would involve creation of bootstrap scripts to do stuff like configure the linker, manpath etc. This won’t get done for a while unless somebody steps up to do it, as most desktop users have root access to their machine. It’s most useful on networks/remote servers, but on corporate LANs autopackage installation would probably be disabled anyway – combined with missing gcc this would be a pretty effective way of preventing software installation completely.
The idea is to have a p2p that can give you the nearest station holding the package, so network usage will be saved, and prevent mirror downtime.
The second idea coming from the first is to have a “Master” station in this p2p network. Every update on this station will be applied to all “slaves” (with an optionnal delay, or a “master” user action). Could facilitates intranet but ALSO USERS THAT want THE SAME CONFIG AS this PARTICULAR person. SO IF you wanna upgrade to something, you can also choose to upgrade to a Whole CONFIG.
That would be great…
All the important stuff, e.g. jEdit builds fine with CVS & Ant
http://jakarta.apache.org/ant/
For everything else, Java WebStart handles upgrades beautifully. http://java.sun.com/products/javawebstart/
So much for your platform-dependant binary hell!
I have at least half a brain, but I am lazy. Compiling stuff from the source is a pain in the ass for advanced users.
but at least you have control over the build process and can patch the system to make small changes without much effort. yes, you have to wait for gratification, so newbies can’t “have this cool app now now now”, but it does mean you know your system, can update intelligently and manage your security and make sure that everything works
which is surely the most important thing.
We’re not going to be building a P2P network anytime soon, imho they are overrated. Bandwidth usage is mostly fixed by the international mirror network
if i can make a suggestion: a p2p-like system could be implemented to automagically make new packages available to other people. say i compile a package for ppc-altivec which wasnt previously available, then i can make that available over p2p, or better, upload it to a mirror.
trust networks get complicated, but the software could offer a choice: install it unsigned, grab the source and compile or install into chroot jail (or similar).
One of the advantages Windows has is the built in installer API that the ubiqutous InstallShield sits on. This provides a common and consistent installation interface that Linux currently lacks. I hope the major distributions get behind this because the present incompatbilities and turf wars are a major pain to the end user experience.
Autopackage looks very good, but IMHO a potentially better package management system is uPM ( http://u-os.org/upm.html ). uPM is similar to Gentoo’s Portage, but it is (according to its Web site, anyway) more powerful while being easier to use and develop for. It is the primary package management system in the uOS distribution ( http://u-os.org/ ).
Hi,
have you took a look at fink (http://sourceforge.net/projects/fink/)? Some ideas are already implemented there, like installing files from different sources. For me it is one of the best apps for MacOSX
Anton
I once spent a week at a company in which all I did was travel to every computer and put in the IE6 upgrade CD for each one. There was almost no good way of transparently deploying applications
I’m assuming this ‘company’ didn’t have a managed desktop and therefore had no idea what software was running on its machines? Or were they just not LAN connected? Creating MSI files is great if you put the testing in, then deploy them in the login scripts. Almost all of the time upgrades go without a hitch – as long as enough effort has been put into creating the packages properly.
No offence, but you’re 18. I doubt you’ve had the experience of deploying apps properly in a corporate environment using a *managed* windows desktop.
Aside from that litle gripe, the article is great and I look forward to something like this to take off. One question though – what is wrong with including potentially redundant files in an installer especially as we move into the broadband age where downloading an extra couple of megs doesnt really raise an eyebrow?
Autopackage looks very good, but IMHO a potentially better package management system is uPM
Yeah, uPM is interesting. Again though it’s tied to a particular distro. I think they have somewhat separate aims, though I’m going to be talking to those guys soon to see if any of our goals overlap. autopacakage is not meant to manage a distro, it’s meant to allow easy software installation from 3rd parties.
Fink: it’s just a MacOS version of apt debian style, an although neat does suffer the same issues. Also I have no interest in MacOS….
I’m assuming this ‘company’ didn’t have a managed desktop and therefore had no idea what software was running on its machines?
They did, but because the hardware was not all the same (it had been bought over a period of years) not all the machines were powerful enough to run the latest version of Windows, so some machines were on 98, some on NT, some on 2000 etc. I’m not sure what you mean by a managed desktop, it was a fairly typical Windows/Netware setup from what I remember.
Creating MSI files is great if you put the testing in, then deploy them in the login scripts. Almost all of the time upgrades go without a hitch – as long as enough effort has been put into creating the packages properly.
I did suggest putting the updater into the login script, but I was cheap labor so it was easier to get me to do it
I have also seen similar problems at other companies that had more advanced software rollout – for instance my fathers machine was once inoperable for about an hour because an installation didn’t go right, bombed half way through and then ITS had to take control of the desktop with VNC (only 1 desktop on windows) to fix it, so he was left twiddling his thumbs for a bit. He also had to hang around while updates and new packages were downloaded to his machine, even if he never was going to use them etc…. yes a very well run Windows network can get around these issues but imho they are best addressed at the packaging level. Also note that only very big companies can really use these kind of corporate deployment scenarios – I’ve worked for several companies both large and small and never seen them in use (i have only read about them really).
Aside from that litle gripe, the article is great and I look forward to something like this to take off. One question though – what is wrong with including potentially redundant files in an installer especially as we move into the broadband age where downloading an extra couple of megs doesnt really raise an eyebrow?
For windows apps you probably only need ship say 4 or 5 DLLs at worst, normally different versions of say the richedit control, MFC, ATL etc. For Linux the problem is multiplied by 1000x because apps have so many more dependancies. Shipping GNOME 2 with every app isn’t practical for instance, you need a proper dependancy resolution system in place.
So, how does autopackage provide a solution to
the problem that you have to make one package file
for each distribution (or special cases for each distribution in one package file).
Maybe I just haven’t payed attention…
Every time somebody says windows installers are easier, i can only ask them: how easy is it to say to your system, check for all the software i’ve installed if a security patch is available and if it is, install it.
This is less than three clicks on debian.
Being a current “victim” of “dependency hell”, I can relate. Being a MAC Head, I recently got involved with FINK. Another potential alternative:
http://www.opendarwin.org/projects/darwinports/faq.php
Until Linux can solve this issue (Before it deteriorates beyond repair) it will remain relegated to the “geek elite” to work with on the desktop. The “masses” have enough problem with Windows et. al. Once a “unified” (meaning across ALL Linux Disributions) is found, then the concept of “free choice” will ultimately result in the demise of Linux on the desktop.
The only problem I ever had with Gentoo was installing it!
Best part is you can totally mess up with portage packages and just emerge world and all is well.
What packaging problems does Gentoo have (besides requiring a repository)?
Requiring a repository is NOT a problem. If you want to install everything from the internet fine. Who’s gonna pay for it? Its better to download what you need and just put it on a CDR. Then you can install it again later without tying up bandwidth.
If RPM was extended with a few more $directory macros then you could easily avoid “rpm hell”. No package manager currently uses all info available in RPMS. All you’d need to do is have something like apt-get that’s capable of pulling individual files out of RPMS and putting them in the proper $directory. Voila, no RPM hell. Mandrake’s URPMI is excellent. Not perfect but damn good.
If you can’t build packages, and you can’t compile source then just wait for the RPM for your distro. Simple. Otherwise there’s the –force and –with-nodeps options.
>[i]There just has to be something better than RPM…
There is. Debian packages.
[i]
And, there’s Gentoo Portage. It’s the easiest thing I’ve used since FreeBSD’s ports tree.
My one beef with most package systems? They don’t tell you where the installed executable is. Most of the time, it’s something I’ve used before and it’s not an issue. But, sometimes, I’ll kick off several package installs and walk away for a while. Then I go to try a package that I’ve not used before, and the executable is not intuitive. BSD’s ports tree included a readme file in each packages directory for any such confusion.
I can go to the packages website and get a read me, but a good pkg management system wouldn’t require this.
So, how does autopackage provide a solution to
the problem that you have to make one package file
for each distribution (or special cases for each distribution in one package file).
This article was meant as a high level overview of linux packaging and our approach rather than in depth technical details, but seeing as you asked….
It has a lot of predefined functions that deal with the differences between the distros for you. For instance installDesktop() will take a .desktop file and figure out the correct place to put it, how to register it, and will work out distro oddities – ie on Redhat 8 we have to alter the gnome1.4 entry to stop it being included twice. This function installs a menu entry in every desktop it can find.
installInfo() can deal with the differences between the FSF install-info script and Debians own version. Remappers are “plugins” that contain distro specific information.
ako: Windows installers have a lot of faults, but they are decentralised and easy to use. My views on apt/emerge are well documented both in this article and on the site.
Steven Beckwith: Once a “unified” (meaning across ALL Linux Disributions) is found, then the concept of “free choice” will ultimately result in the demise of Linux on the desktop.
That makes no sense? I don’t understand what you mean by this.
MackBolan: The only problem I ever had with Gentoo was installing it! Best part is you can totally mess up with portage packages and just emerge world and all is well.
What packaging problems does Gentoo have (besides requiring a repository)?
Er, the fact that it’s source based? The biggest problem with building from source after you’ve removed dependancy resolution is that it takes ages even on fast hardware. Windows users are not accustomed to waiting for an hour while their computer builds their web browser (with 100% cpu usage). Also, closed source products don’t fit in well in this approach (yes, i know it can be done).
Requiring a repository is NOT a problem.
Requiring a Debian/Gentoo style centralised repository is a problem because it doesn’t scale. Even a project the size of Debian has big problems with this.
If RPM was extended with a few more $directory macros then you could easily avoid “rpm hell”.
No, because the RPM file format would still be fractured, would still be full of distro-isms, would still use an easily corrupted binary database, would still be unaware of packages installed via other means (source installs, commercial installs etc). RPM has lots of problems beyond just dependancies.
If you can’t build packages, and you can’t compile source then just wait for the RPM for your distro.
Often it’s not a case of “just waiting”, because there never will be an RPM for your distro. This is why we need a decentralised system where you only need to package stuff once.
In a distributed environment, how do you make sure someone doesn’t slip in some unwanted code (eg. a bogus dependency)? How do you ensure the install script is authentic?
Also, what do you do if a server is all of the sudden swamped by burst of demand? What action will autopackage take?
Location: Anywhere usa
Date: Anytime
Distro: SuSE 8.0
Machine: Toshiba Tecra 720c Japanese
Wm: windowmaker
Is it due to the fact that I have the cds? Is is due to the fact that I play along and accept SuSE’s ‘my way or the highway’ yast rules? Is it because I never download 3rd party crap off the net? Is it because I don’t recompile anything?
Maybe I’m a linux zealot or god forbid a linux apologist. Maybe I am just a simple user that accepts the defaults for the OS and the packages? Maybe I am just being my old difficult self?
But I never have dependency problems. To say that dependency problems are ‘legendary’ in linux makes me want to laugh. I insert my cd#1. Start yast. Search through the inane interface for my desired package. Perhaps insert whatever cd that yast tells me to. Install. Quit yast. Use program.
As with openbsd, many users make life difficult by insisting on being as 3L1T3 as possible and recompiling everything they can get their hands on or changing settings everywhere they can just to eke out that last picosecond of performance.
As my boss used to say….
Stay off the moors and you don’t get eaten by the beasties.
Venture off the beaten path, and you should accept your punishment with grace and quit whining.
If someone can provide an example of a default install of SuSE where you have later installed something off the cds that caused your system to blow up or break the yast package manager, step forward and I will gladly eat my crow. Please give real examples. For as anybody that has worked in a software company will tell you.
———
If YOU can’t get the problem to repeat itself ,every time , then the problem was with YOU and not with the software.
————-
If the problem is repeatable, file a bug report and help the cause. This is a great way to learn how to QA software problems by the way.
As I don’t have the time to test out package managers on the multitudes of other linux distros I will have to accept the experience of others but as for me I can honestly say that I have not had these problems with SuSE.
It just works, resolves all dependency and you can choose between compiling yourself or downloading a binary packet.
Please, please merge this nice project with uPM !
There are very good ideas in autopackage & uPM.
This is potentially a Very Bad Thing ™ because you are downloading and installing packages from users you don’t trust. Maybe if you combine this with a web of trust, then this would work. For a less P2P-centric administration, but P2P-centric distribution, check out: http://www.linuxjournal.com/article.php?sid=4847 (apt-get over freenet)
sounds like a very good idea. any sort of time schedule on when a fully working version will be available? have any distros said they’d like to include it?
i like and hate apt-get at the same time. i like that it will install everything i need, but if i remove a package it always seems to uninstall something i dont want to be by default.
Will this form of package management hold a database on your local machine letting it know what has been installed and what dependencies there are? so that if i uninstall xyz package that depends on abc and fgh, and other packages depend on abc but no others on fgh, it will also uninstall package fgh?
RPM stands for RPM Package Manager. See rpm.org to verify.
RPM has *not* been forked for various distros. Rather, various distros build their own RPMs to satisfy their own requirements for glibc, shared libraries, naming conventions, etc. Some remain more compatible with each other than others. Source RPMs are often easy to recompile for the other platform.
RPMs can be built to be relocatable, i.e. the files they install are not required to be build based on the / directory. The fact that relocatable RPMs are not often used in distros is because they would be a liability there.
This fellow’s new project may well be progress, and there are certainly many unpleasantries with RPM, but I don’t get the impression he’s all that well versed with it.
This idea is sure better that we have currently as package management but it doesn’t fix the root of all the problems. The biggest problem is the shared object (.so files under Linux), each version has it’s bugs and quirks, so if you build your program on libsomething 2.1, it might not even be binary compatible with libsomething 2.1.1. So, if you install a binary that require the 2.1.1 version, you need both version of the library on the computer, what a waste of space! Until someone find a way to have a better management of shared object directly built into the OS, a perfect package management is only a dream.
You can have a workaround with this problem pretty easely. I guess that would work with most package management systems, I know for sure it works for RPM. If you release your source package as a developper, create a RPM but for *all* distribution at once. When you link your final binary, use the -static switch, this will create a huge binary that will contain your program and also all the shared objects needed will be contained in the binary. In this way, a rpm -i yourpackage.rpm will *always* work! No pain for the end user at all! Sure it have a lot of disavantages like having a huge binary and that binary is of course still tied to an architecture.
I think distro maker (at least commercial one) should build more package for their distro. Since it will be impossible to build everything, they should go with a simple system. They list all package on their web site that they can build (this list could be built with a script that roam around sourceforge and freshmeat) and end users could put money on the RPM they want. When for example there is 100$ or more on a single program, then someone at the company build the package et put it online. Of course, they keep the money.
i use gentoo and i have no problems with it and portage.its the best os in my opinion.i have a slow pc but this is not bad because compiling is fun in gentoo and you learn a lot.another distro “Arch Linux”has already a clone of portage called pacman which works as portage but handling binaries ,not sources
What’s the pb with dependancies? urpmi takes care of all dependancies. Under Mandrake, programs are much easier to install than under windows. urpmi myprogram. And that’s all 🙂
sounds like a very good idea. any sort of time schedule on when a fully working version will be available? have any distros said they’d like to include it?
No, and no (but I haven’t spoken to any about it either). I originally thought it’d take about a year of coding, that’s probably an underestimate. Obviously the more people that help the faster it will develop.
i like and hate apt-get at the same time. i like that it will install everything i need, but if i remove a package it always seems to uninstall something i dont want to be by default.
I like apt too, but normally the repositories (even on debian) don’t have the specific package I want.
Will this form of package management hold a database on your local machine letting it know what has been installed and what dependencies there are? so that if i uninstall xyz package that depends on abc and fgh, and other packages depend on abc but no others on fgh, it will also uninstall package fgh?
Yes, it does this right now. I hope that at some point we’ll have RPM integration too, so if you install a .package file the RPM database is updated so you can install other RPMs that may depend on things installed from autopackage. That’s a “one day” feature though.
Harry Frequent: RPM stands for RPM Package Manager. See rpm.org to verify.
Yes, very amusing. That’s what it stands for now, it originally stood for Redhat Package manager, and now they say it stands for RPM because other distros use it. It’s like “Wine is not an emulator”, everybody knows what it really stands for.
RPM has *not* been forked for various distros.
RPM itself frequently breaks backwards compatability between versions, and distros tend to all use different versions, so … well, it’s a moot point, I have no problem with Redhat breaking RPM backward compatability as it’s their tool, and was never really designed to be reused outside of their distro anyway. Better they have the flexibility they need to make a good distro, and something else deals with 3rd party packaging.
RPMs can be built to be relocatable, i.e. the files they install are not required to be build based on the / directory.
Yes, I know, but no RPMs are in practice. This is partially because it’s not the “culture”, but also because most Linux packages by default are not relocateable because they rely on autoconf macros at compile time to let them locate their data files. By contrast, one of our rules is that an autopackage must be relocateable by linking with libprefixdb. This makes it more effort to build a .package the first time because often the source must be patched, but the patches are not at all invasive and we hope that maintainers would accept them upstream.
This fellow’s new project may well be progress, and there are certainly many unpleasantries with RPM, but I don’t get the impression he’s all that well versed with it.
It’s true I’m no RPM expert, I know the general design and of some of it’s quirks. RPM4 is huge and I don’t know it inside out, but we’ll need to learn it inside out later to provide RPM integration, so if you’d be willing to offer some advice/help with this that’d be great
This idea is sure better that we have currently as package management but it doesn’t fix the root of all the problems…….
Actually that isn’t true in 99% of cases, libraries should not break binary compatability between minor versions and rarely do actually, if a library does then it’s a bug and should be reported to the maintainers. Now it’s true that sometimes a library contains a bug that’ll be fixed in say version 2.1.1 (this is why we have version numbers) and so an app will depend on 2.1.1 rather than 2.1.0, but unless binary compatability was broken which shouldn’t happen, you in theory won’t need 2.1.0 on your system.
As I don’t have the time to test out package managers on the multitudes of other linux distros I will have to accept the experience of others but as for me I can honestly say that I have not had these problems with SuSE.
I use SuSE 8 too, and it’s good, but for instance usually the first thing I do when I install a new Linux distro is upgrade Mozilla because they release at such a rate that SuSE don’t keep up. Sometimes I uninstall from YaST first, oops, now YaST whines every time I start about how Galeon and a few other things have broken, when really they are fine because I used the net installer. RPM hell is really very easy to get into, if you’ve never experienced it then you are fortunate indeed.
To those who talk about ports/emerge/apt – these systems would be great, but they don’t scale.
If you can compile your own software from source, it’s only a little bit harder to make a proper package.
Take Mozilla, for example. A new version is released and you want to upgrade, but RedHat has not built a package yet. What you do is download the Mozilla source, get RedHat’s Mozilla SRPM file, replace the old mozilla tarball with the new one. Update the version numbers and rebuild. It will probably work, unless RedHat’s patches conflict with the new code. If you have patch conflicts, either fix the patches or take them out.
I’ve never done this with debs, but I don’t imagine it’s much harder.
I’m not all that clear about what exactly Mike doesn’t like about the fact that you can install a full professional software suite (e.g. Microsoft Office) on OSX by just dragging the folder off the CD. This approach really works quite well. It is true that the first time you run Office, some libs will be dumped into the system, but these are MS specific. Deleting the folder you dragged and dropped deletes almost all of the app code, with the exception of a few (fairly easy to find) libs which won’t really be used. Personally, I’m a huge fan of this approach. I do not like installs. Let me say that again: I DO NOT LIKE INSTALLS. The whole idea of having to ‘install’ something to make it run irritates me. It should just work.
That said, I don’t think that Mike gives enough credulity to the Windows installer system, Darwin:
I’m not sure what you mean by a managed desktop, it was a fairly typical Windows/Netware setup from what I remember.
You mentioned that you had lots of different types of hardware you were installing on, some of it quite old, running antiquated versions of Windows. If you are going to compare your new installer technology, you should be comparing it to the newest Windows installer technology. As of Win2K networks, install on demand exists. That is, I double click a file with an extention unknown to my local system (e.g. .doc), Windows takes over, asks the domain app server what it should install to handle .doc files, installs Word, and opens the doc… without asking a single question! This is very slick, and solves another major problem, how are you going to get all this available software inside your corporate firewall? Most corporate entities do not want to go crawl the Internet to install mission critical LOB applications. This is especially true when they could be redirected to who knows what malicious server for the install.
For windows apps you probably only need ship say 4 or 5 DLLs at worst, normally different versions of say the richedit control, MFC, ATL etc
Hmm… this was a problem a long time ago, pre Win2K, and was one of the main contributors to Windows instability. Windows 2K and up keeps the dlls that Microsoft ships, due to the questionable quality of some of the 3rd party libraries that were provided. Windows keeps a list of system libraries, backup copies, and watches to see when they are changed. It allows the change, so that the install will not fail, then deletes the changed dll and replaces the one shipped by Microsoft.
Personally, I believe you should be able to copy down a .tgz’d folder, tar -zxvf download.tgz, cd download, ./download to run your app. That should be it. Down with installs of all types!
>> “Take Mozilla, for example. A new version is released and you want to upgrade, but RedHat has not built a package yet. What you do is download the Mozilla source, get RedHat’s Mozilla SRPM file, replace the old mozilla tarball with the new one. Update the version numbers and rebuild. It will probably work, unless RedHat’s patches conflict with the new code. If you have patch conflicts, either fix the patches or take them out.”
and
>> “Personally, I believe you should be able to copy down a .tgz’d folder, tar -zxvf download.tgz, cd download, ./download to run your app. That should be it. Down with installs of all types!”
This is easy for you and me, but it’s not for the average user who would like to try or even use Linux fulltime.
And even though I have no problems with the usual tar -xvzf blah, cd etc, etc, etc. I do think Autopackage will be able to save me a LOT of time (and remember, it’s about BINARY packages, not source packages)
I think Autopackage will rock!
(heck, it already does)
I’m not all that clear about what exactly Mike doesn’t like about the fact that you can install a full professional software suite (e.g. Microsoft Office) on OSX by just dragging the folder off the CD.
I do like it, but the way OS X does it leaves a lot to be desired as it strips installation logic entirely. You said it yourself, lots of stuff gets left lying around when you uninstall, and how do you deal with dependancies. MacOS X gets around this problem by the apps not having dependancies – that wouldn’t work for Linux.
I do not like installs. Let me say that again: I DO NOT LIKE INSTALLS. The whole idea of having to ‘install’ something to make it run irritates me. It should just work.
If you can think of a way that you can “just run” something without taking us back to DOS era technologies then please do tell us. Let’s pretend you wish to install Fontilus, a Nautilus plugin that lets you view and easily install fonts by dragging into a virtual folder (how ironic). This involves file installation, shared library registration, CORBA server file drops and more. You can’t do that if all you do is drag folders around. What you can do is provide a VFS plugin like for instance packages:/// which allows you to drag a .package into it and as it “copies” it’s actually installing, and dragging it out uninstalls it. UI wise that’d get you the effect you want.
One of the rules for making a .package is that all human interactions must have sensible defaults. If the logic to choose those defaults is complex, then so be it. If it varies according to taste, choose the one that will please the most people. This is so those who want it can have “headless” installs, ie with no interaction RPM style. You are not allowed to ask a question that could not be automatically accepted as a default, because when designing the system my Mac-mad friend kept going on about how great drag and drop applications were. Well, they’re ok, I don’t see what the big fuss is about myself, but clearly a lot of people like it.
As of Win2K networks, install on demand exists….. This is very slick, and solves another major problem, how are you going to get all this available software inside your corporate firewall?
Actually FYI this click-to-install feature of Office is Office specific, according to Jeremy White of CodeWeavers (who let you run Office under Linux) these APIs were undocumented, which is why I guess you don’t see this used more often. I’ve given quite a bit of thought to corporate installs because I believe that the first segment of the desktop market Linux will displace Windows in is the corporate desktop, hence the fact that you will be able to disable .package installation by local users entirely (not done yet, but easy to add).
Controlling software through firewalls is trivial, in the /etc/autopackage config file you will be able to specify the network resolution server to use. By default this will be autopackage.net but you can change it to a controlled server which will only resolve names you allow (ie block access to autopackage.net). A few days ago I realised we could have MS Office style “install on demand” for corp networks which would be great – you can see all the apps you might want on the menu, and clicking them passes control to an installer script. The vFolders system makes this very easy.
I’m still not decided how to do security yet, probably by having root locked down but with an ssh keypair setup so the software control server can simply ssh into a box, scp the .package file onto it and install it from there. A “playback” front end can answer any questions you may want to customize during setup, so automated installs become very easy.
Windows keeps a list of system libraries, backup copies, and watches to see when they are changed. It allows the change, so that the install will not fail, then deletes the changed dll and replaces the one shipped by Microsoft
Yes, a very ugly solution IMHO. You need to keep two copies of every DLL around, it’s not even implemented at the FS layer. Whoever desinged windows file protection was on crack imho, or the internals of windows are more bizarre than I imagined.
GPG integration is something we’ve thought about. Not sure how to do it though – in a distributed network who signs the code, and how do you get the keys? ie if the website is compromised then anything on it is compromised too.
I think this fact alone is enough for me to predict that this will never be used by any of the major distributions.
I think the project sounds nice enough, but being an independant project, i doubt they will be backed by the distributions, and if they arent backed by them, then whats the point? (unless you want to make your own distro)
As a side note, it might very well be that RPM isnt the greatest format, but i would dare claim that most of the problems with dependencies are due to bad packagers. I have often seen packages that were linked to a minor version of a library, which i dont think there is really any excuse for. I can understand if it would demand a library newer than x.y.z, but yet still link to x.y.
“This idea is sure better that we have currently as package management but it doesn’t fix the root of all the
problems. The biggest problem is the shared object (.so files under Linux), each version has it’s bugs and
quirks, so if you build your program on libsomething 2.1, it might not even be binary compatible with
libsomething 2.1.1. So, if you install a binary that require the 2.1.1 version, you need both version of the
library on the computer, what a waste of space! Until someone find a way to have a better management of
shared object directly built into the OS, a perfect package management
is only a dream.”
Not being a Linux user currently, I am puzzled by this. Are you saying
that a program written to use version 2.1 of a shared library may not
work with version 2.2 ?
Surely the jump table will be the same for all versions? New calls
would be added on the end of the table but would not affect any
program that doesn’t know about them.
Is there something wrong with the Linux shared library format?
No, the library format is just fine, the problem is that the libraries are not always backwards compatible.
The problem is with the developers, not the library formats.
I do think however that you can usually consider libraries to be backwards compatible within their minor versions.
That is, if you write a program for lib.so.2.1, then it should work with lib.so.2.2 too, but will probably not work with lib.so.3.0
And to reply to the post that you replied to: I dont think statically linking things are the right way to do it. It brings heaps of new problems. Just think of the implications if a bug is fixed in an oft used library. ALL packages that uses that library would have to be recompiled. You solve one problem, but introduce a new one that is even worse.
I think the whole dependency hell thing has become much less as a problem, and i think it will keep being so as the major libraries and software packages matures, and hence dont have to break binary compatibility often.
Actually FYI this click-to-install feature of Office is Office specific, according to Jeremy White of CodeWeavers (who let you run Office under Linux) these APIs were undocumented
Actually, this is a Windows feature. It was originally written and developed by Office, but then Windows integrated it. It works on solutions of all types, including Adobe Acrobat, and our Internal bug tracking software.
So, security is a huge deal. What legal guarantees can be made to corporations who use this package management system to install software? The ‘block the malicious site’ solution doesn’t work because by the time you to discover that a site is malicious, you have probably opened yourself up to it’s nature. The most important aspects of security are not the encryption of communication so much as knowing who you can trust. Is this package management system going to be accountable as the source of verified secure software? If so, what legal plans are in the mix which will make this guarantee? Who in the Open Source community is going to be legally obliaged to ensure that a secure binary version of their software is transmitted to these secure download locations?
All it really takes is one rogue app making it’s way into the system to turn off major Corporate sponsorship a’la Ford or Bank of America.
Yes impressive. I’m just a newbie, but this sounds like a good solution to some problems. You seem to have good answers also for the good questions others have set for you. The result of careful thinking it appears. I also have to tell you your careful, well-thoughout, polite responses to questions are very impressive for an 18 year old. (I noticed that not everyone posting for instance, maintaned this same level of respect). Paul once admonished Timothy not to let others despise his youth. May you also continue to grow in character as well as widsom, and may you come to know more of the God they served. (Bad form to say that last bit, I know, but it is my sincere desire nevertheless).
It seems that this project is going to be something nice… am looking forward to see how it fares once it is stable in the future.
I give credit to these guys, no matter how imperfect the AUTOPACKAGE project may seem to other people. AT LEAST THESE GUYS ARE DOING SOMETHING rather than WHINING and CRITICIZING emerging projects out there…yet remaining STAGNANT, like most a.k.a linux hackers, a.k.a linux experts, a.k.a linux power users do.
If you don’t like it don’t use it =)
I think this fact alone is enough for me to predict that this will never be used by any of the major distributions.
Actually I’m not betting on distros shipping with autopackage at all, although it’d be convenient if they did it’s not at all necessary – you’re right, if we needed that it would kill the project dead.
Some of you might be wondering why the packages are self-extracting given the added bloat that adds – well it’s so if you run a .package it can check for the autopackage support code and install it if not present – this doesn’t work yet, if you run a package without using the setup script first you’ll get a nice (albiet slightly dated) message and an offer to download the software, but pressing Y (or enter) just says “Please write me!
” or something similar. It wouldn’t be hard to add this code, like most things it’s just time.
Actually, this is a Windows feature. It was originally written and developed by Office, but then Windows integrated it. It works on solutions of all types, including Adobe Acrobat, and our Internal bug tracking software.
Thanks, I didn’t know that, the interview with Jeremy was pretty old.
So, security is a huge deal. What legal guarantees can be made to corporations who use this package management system to install software?
None. You don’t get any guarantees with InstallShield or Appfolders, as it’s basically impossible to guarantee in a decentralised environment. Maybe we can do funky things with signing/auditing authorities and GPG or something, I haven’t given it a great deal of consideration yet, but it’d be an added value service if this happened.
All it really takes is one rogue app making it’s way into the system to turn off major Corporate sponsorship a’la Ford or Bank of America.
Companies like that should be using the specialist corp deployment systems so they’d have done test installs first.
I think the whole dependency hell thing has become much less as a problem, and i think it will keep being so as the major libraries and software packages matures, and hence dont have to break binary compatibility often.
Right – for instance GNOME2 guarantees binary compatability throughout the 2.x cycle. ABI freezes are pretty new to Linux, but expect to see more in future.
Would a better solution be to have a dependency resolution system system (sort of like apt), with a predefined source list (that references the internals of a package itself) included in a package with all of the requisite dependencies that one could ever need, and driven by a script. i.e. you click on the package file, it extracts itself into a temporary directory and runs a script/small program that starts the install process.
That program searches the local system to see what dependencies or required libraries are already there, or queries the local package manager database to see if the dependencies are there. (probably better to look for lib’s you need than to rely on the local package manager’s versioning quirks. Although you could tell it what a specific distro’s versioning would require in the package configs). Then start the gui/terminal installer to ask where to put it, extra features etc. Then install the package and any missing dependencies.
If every distro shipped this app, then you wouldn’t need to put it in the package, and you could have it update one database of which files go with which package, which dependencies are required for which package etc. If not, then you could have it build a package for the local system to update it’s database.
ok, i just upgraded my OpenBSD system to OpenBSD 3.2 and installed a few new packages (php and mysql) and WOW what a dream!!!! why cant linux (all distros not just debian) have soemthing like this. I type pkg_add and the path to the ftp server where the package is at, and it downloads all necessary packages still needed, and tells me what its downloading and installing. Also someone mentioned about not knowing sometimes where the binaries are installed. well with pkg_add it tells me at the end wher the binaries are installed in short informative notices.. …VERY nice…
<blockquote>And to reply to the post that you replied to: I dont think statically linking things are the right way to do it. It brings heaps of new problems. Just think of the implications if a bug is fixed in an oft used library. ALL packages that uses that library would have to be recompiled. You solve one problem, but introduce a new one that is even worse.</blockquote>
This solution is only used when software is not distributed by the creator of a specific distribution. If a developper release a new version of his/her software, this person could create a statically linked binary in a RPM. Of course, there is no point to do this for software in a specific distribution, you add a lot of bloat and you kill the purpose of having shared onjects in the first place. I know bochs http://bochs.sourceforge.net does static linking and their RPM works like a charm, I would like that all developpers would do this, it would save the end user a lot of trouble and the developpers a lot of “this doesn’t build from source” questions from people (most end users) that should never do compiling their own software in the first place.
With this simple solution, on a properly configured system, the end user could download the RPM, click on it to install (in Konqueror or Nautilus for example), enter the root password and voila, everything works, it’s even more simple than Windows and it doesn’t play around with your other files like Windows installers like to do.
Congratulations, you just described exactly what autopackage does, every step of the way. Except we don’t update the RPM database yet.
I didn’t put lots of technical details into the article on purpose, it’s meant for general OS/Linux enthusiasts/users rather than programmers. There will be more technical articles sometime next year. Until then as you seem to have the right idea, come help us out :p
Static linking: the solution isn’t to try and pretend dependancies don’t exist, the solution is to make management of them not suck. Nuff said in my opinion.
I wish I knew how to code, if I did I would help you guys.
Thanks for taking the time to answer all my questions Mike, it’s a pleasure to have content contributors like you on OSNews.
1-How about reducing dependency on the user having an internet connection. Be careful of *assumptions*.
2-Downloading only what’s absolutely necessary.
Sometimes the only difference between package A & B is a single file. Better Meta info server side can help. Question at this point is how granular the process should be?
3-Ability to do a rollback. Bad package in the system? Rollback to the previous version. Unlimited undo basically.
4-Better tools for packagers, enforcing a certain order while making the packagers life easier. Discipline is the key to success.
5-Ability for the user to break in and out of the process as needed. Remember humans are still the most capable in this loop.
6-Security and trust has been mentioned. Maybe multilevel preference towards sites. In other words in the case of a tie one can prefer one site over another.
Well with SuSE, .rpm files are easy enough to install, but the dependicy issue is a pain. This might make it where more people will switch to linux. Lets just hope it comes in a .rpm already compiled. Look out Microsoft, linux is getting easier to use.
For the love of god, take a look at bsd ports/packages. The packaging problem has been solved for years.
That the author of this article has never used OS X to any consequence to form a judgment about the installation and removal of software.
He paints the problem that users have to worry about only being able to use functionality provided by the system if they want the easy use of the Application Package/Bundles structure.
The difference between this and with Linux is that OS X has a STANDARD set of system frameworks for everything from common data structures, to high level network APIs, to Audio and MIDI API’s, to a complete Appkit, etc. So while it’s true that the user must rely on what is built into the system, almost EVERYTHING that ANY app developer is going to use is provided by or extended from the functionality of the OS.
In the cases where an additional framework does need to be installed, like say if I’m using an app which was built on the OmniNetwork framework, a pretty popular network API, then that framework can just be included in the Application Bundle and linked to from in that structure.
The only time an installer needs to be used is when Authentication is required because the app needs to install something in an administrator owned location, like the /Library/Application Support directory, etc. Which would seem like it does defeat the purpose of the normal easy installation of OS X software at first is not as bad as he makes it out to be.
The reason for this is that there is a CONSISTENT and LOGICAL filesystem layout scheme. All system level frameworks, fonts, resources, etc. that are to umbrella all users are stored in the /Library directory and the appropriate sub-folder. Installing things here will require authentication, and thus likely an authenticating installer program, like the standard “Installer” program that essentially acts as Apple’s package management software.
All user level frameworks, fonts, resources, etc. that are specific to a single user are stored in the /Users/name/Library directory and the appropriate sub-folder. Installing things into these locations doesn’t require any authentication because those locations are owned by you, not the system. So if anything needs to be added to those locations the app can generate the files upon first run, or more accurately copy themselves from inside the Application bundle to the proper location in the user library.
Then for the actual core system there is /System/Library. 3rd party developers are not supposed to install things to that area of the system except for device driver kernel extensions. So that area of the system is updated and maintained by Apple and their software upgrades. That makes it easy for Apple to ensure that standard set of complete functionality that I talked about earlier.
Honestly I think a lot of Linux’s problem with package management could be solved by virtue of there being a standardized set of libraries that constitute the “Desktop Linux Base” or something which includes a very robust set of API’s to build apps with. That way you wouldn’t get developers who constantly use different versions of the same essential library to accomplish the development of their software. If that were done then you’d really only have to worry about a fringe set of developers that can’t build software based on what is available with a standard system and those developers could simply bundle their tailored libraries with their software. If developers actually had a complete system development API that they could rely on exisiting in every linux distro I suspect they’d use it, or at least they should. However in the lack of such an environment we get the mess that exists today, one dev uses libxml to handle something, and another uses some other xml library to handle what he wants, that forces me to have two libs of duplicate functionality on my system if I want to use both pieces of software. That’s just kinda silly.
Open the following link in a new window so you can refer to it during the next blurb:
http://barney.gonzaga.edu/~naschbac/chimera.png
Here’s how software installation works for most user level OS X software.
I clicked on the link in the browser, it downloaded a .tgz file containing the disk image file. The .tgz automatically gets unzipped after it’s done downloading. The Chimera.dmg file is the disk image, it automounts the Chimera disk/volume after the .tgz is finished unzipping. If I wanted to keep the disk image around and wanted to mount it later I’d just double click on it.
Anyway I can run the application “Navigator” right from the disk image, or to install it on my HD I can just copy it to wherever I like. I could drag it to my iPod and run it from there, and then if I didn’t want that I could drag it from my iPod to some other place or burn it to a CD and run it from wherever. If I want to get rid of the whole program I just drag it to the trash and throw it away. If I want to clean out every last trace of the program I just go to ~/Library/Preferences/ and toss it’s prefs file, and for one last check to be sure I go to ~/Library/Application Support/ and see if it installed anything there for itself like cache directories, etc. However even if all I ever did was just delete the program no other software on the system would ever be effected by the left over prefs file, or anything in the Application Support directory.
It’s a very simple metaphor for the user because they are a part of what is going on all the time, yet it is still mind-numbingly simple. So there’s never a feeling of, “Oh geez I wonder what they installed, and how do I get rid of it when/if the uninstaller doesn’t work all the way.” It’s a very spatial system, you feel in control of everything. The feeling was more satisfying prior to OS X because the only thing that mattered was that you had the “System Folder” at the root level of the Volume, but OS X is still more spatially understandable than both Linux/GNOME (or any other desktop) and Windows. You have three regions and only two you have to worry about. /Library and ~/Library. All the sub directories are clearly named and easy to understand, and typically the user won’t have to access the /Library area unless by intention to do something, but a new user won’t have to poke around in there.
http://barney.gonzaga.edu/~naschbac/local.png
That’s something one would think that the people trying to get Linux on the desktop, and to be a pervasive desktop solution, would be striving very hard to achieve. Give the user the feeling of control and connection, and make the system so simple that it’s easy for them to understand the layout and the mechanisms. If a user has used a platform for about a year and still feels like the computer does “voodoo magic” behind the scenes (as my mother puts it when talking about her Win98 PC at work) when installing and removing software, etc. then that system has been designed poorly from a perspective of end-user usability and intuition.
I’m not involved with this discussion because I want to try and say that MacOS X is better, and so you all might as well use it. I’m involved with this discussion because I really do want Linux/GNOME/KDE to be better and I think it has the potential to be if the community can really think about these problems and fix them even if it means gutting today’s solutions to implement all new better ones. There’s no sense in trying to patchwork together a solution from an already flawed way of doing things when there is all the time and resources available in the Open Source movement to build something right. I think the Linux community really just had two problems, first a “not invented here” attitude toward the study of how other OS’s tackle problems that Linux faces, and a lack of clear direction on behalf of development efforts. There are a lot of steps made sideways before any real steps are taken forward because of these two problems.
Static linking is NOT, I repeat: NOT the solution.
There are only cons about that and the worst of them is duplication of code and the waste of memory (no, not everyone has 1 Gig of memory and the ones who do have that amount of memory need it for more important stuff).
I’m not interested in the bsd/linux religious war, but bsd ports does mop the floor with the various linux offerings. Folks may want to take a look at:
An introduction to FreeBSD ports:
http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/ports.htm…
Darwin ports system (which appears to have support for multiple operating systems):
http://www.opendarwin.org/projects/darwinports/
and dirty details here:
http://darwinports.gene-hacker.net/docs/guide/
I only moved over to *nix from windows a year ago, and I find ports easier to use than windows installers. It’s _slick_. You literally type “portinstall [name of program]” and it fetches source for the program (and any dependancies you don’t already have), builds them, and installs them in the right order. You can add a -P to that command, and it will instead pull down pre-built binaries from a mirror. (avoiding the source/compile part).
It keeps track of what packages are on your system. You can do a “portupgrade *” and, for of every installed package on your system, it’ll check for a new version and if one exists, fetch/build/install it. (again, it’ll use pre-built packages if you do a -P)
And of course you can “pkg_deinstall [name of program]” to get rid of software you don’t want anymore.
Thanks..
That the author of this article has never used OS X to any consequence to form a judgment about the installation and removal of software.
au contraire, I have used it plenty.
The difference between this and with Linux is that OS X has a STANDARD set of system frameworks for everything from common data structures, to high level network APIs, to Audio and MIDI API’s, to a complete Appkit, etc. So while it’s true that the user must rely on what is built into the system, almost EVERYTHING that ANY app developer is going to use is provided by or extended from the functionality of the OS.
Untrue. OS X is brand new, and Apple have already broken compatability several times – witness the fact that QuickTime has separate installers for 10.2 and 10.1 – 10.0 doesn’t even appear to have an upgrade available for it.
The idea that no application developer will ever want anything that isn’t in base 10.2 is imho fallacy – Windows is equally centrally controlled and look how many times Microsoft has updates for that. DirectX for instance was added on first as an extra component that was later integrated.
In the cases where an additional framework does need to be installed, like say if I’m using an app which was built on the OmniNetwork framework, a pretty popular network API, then that framework can just be included in the Application Bundle and linked to from in that structure.
I know that at the bytecode level static linking isn’t used here, but it might as well be. The problems inherant with this approach to dependancy bundling are well documented elsewhere on the net and even this thread (impossible to do global updates, inefficient). MacOS can get away with this because it’s apps are largely monolithic and there is no competition driving things forward – Linux is if anything characterised by a free market of code, so statically linking everything is not even an option for us.
The only time an installer needs to be used is when Authentication is required …..
You said it yourself, appfolders are too simple even for the massively centralised MacOS.
The reason for this is that there is a CONSISTENT and LOGICAL filesystem layout scheme.
Linux has the FHS which although far from perfect is a similar idea. Note the use of more cryptic names had advantages for internationalization.
Honestly I think a lot of Linux’s problem with package management could be solved by virtue of there being a standardized set of libraries that constitute the “Desktop Linux Base” or something which includes a very robust set of API’s to build apps with
My idea of a robust set of APIs is the Gnome set – others prefer KDE, others prefer raw X and still others want to experiment with bleeding edge frameworks. Total standardisation of Linux is not going to happen any day soon, although interfaces are being standardised: see the LSB and freedesktop.org
It’s a very spatial system, you feel in control of everything
This is totally subjective – users should never need to feel that they should know what files are where, so there should be no need for appfolders. As far as they are concerned, they install it, they use it, they uninstall it. If they need to know file locations there is a bug in the system somewhere – only admins/coders should ever need that info.
That’s something one would think that the people trying to get Linux on the desktop, and to be a pervasive desktop solution, would be striving very hard to achieve. Give the user the feeling of control and connection, and make the system so simple that it’s easy for them to understand the layout and the mechanisms
As I have already said (and said in the article), the OS X system doesn’t even work all the time for OS X, let alone Linux which has totally different design goals. I have no problem with making systems appear simple to the user, but sacrificing functionality because it actually is simple is IMHO a bad idea. Modern cars are enormously complex, they perform many calculations per second, yet I have never heard people complaining about how their cars perform complex voodoo behind the scenes. That’s because they never need care. If we do our jobs right, it’ll be the same for computers.
There’s no sense in trying to patchwork together a solution from an already flawed way of doing things when there is all the time and resources available in the Open Source movement to build something right
On the contrary, the present system is nearly there – see how many people are devotees of apt/ports/emerge? That’s because they work. The problem is as I’ve said before that they don’t scale well due to their centralisation. Making a system that’s as easy to use as apt, but decentralised is a big challenge but if we succeed I believe we’ll be left with the best of all systems – the use of use of MacOS with the power of Linux.
I think the Linux community really just had two problems, first a “not invented here” attitude toward the study of how other OS’s tackle problems that Linux faces, and a lack of clear direction on behalf of development efforts. There are a lot of steps made sideways before any real steps are taken forward because of these two problems.
I respectfully disagree. I did quite a bit of research before starting this project, I looked at all other packaging systems I could try. This included OS X which I have experimented with and used first hand. I looked at it from all angles and concluded that:
– Windows installation sucked because it was designed ad hoc by 3rd party companies with no real thinkthrough of all the problems (ms are partially rectifying that now with .net and MSI)
– OS X sucked because it sacrifies virtually all flexibility for an extremely minor and trivial increase in usability that can be easily replicated with VFS plugins. It’s barely workable on MacOS and not workable at all on Linux.
– Linux sucked because the current systems were nearly there except not quite: there was no way of bringing the benefits of apt/ports/emerge to everybody at once in an efficient and decentralised manner.
As OS X and Windows are other peoples problems, but I could do something about Linux, I tried to figure out how to blend the best features of all of them.
I use SuSE 8.1, which I think is awesome. But I love to install new software off of the web.
In order to ensure a useful system unless of course you enjoy spending needless hours recompiling and making sure you have set the correct paths and flags etc like Mat things are a problem. It seems stupid that there needs to be a special RPM for every version of every distro.
I tried Ximian red carpet the other day. I installed Evolution 2. In order to install it a package that is needed for Mr. Project needs to be uninstalled. If I try and re-install Mr. Project SuSE indicates that it is already installed. Annoying.
It is for this reason that I have separate Linux partitions to play. Just to try new stuff, compile etc, just generally mess things up.
Something needs to be done to make things easier.
I personally don’t see a problem with having each app live in its own directory with all the needed packages installed in this directory. Hard drives are big enough that duplication is not an issue. if an app needs to know where another app is, just have the app prompt the user for the location of the app or it could even scan the drive and suggest if necessary potential locations.
Are there problems with permissions and access to devices, config files, etc doing it this way?
Good article and a great project. Best of luck.
Suggest you use a spell checker next time.
independant = independent
dependancy = dependency
“And to reply to the post that you replied to: I dont think statically linking things are the right way to do it. It
brings heaps of new problems. Just think of the implications if a bug is fixed in an oft used library. ALL
packages that uses that library would have to be recompiled. You solve one problem, but introduce a new
one that is even worse. ”
Surely all the programs would now work correctly instead of having a
bug? Why would they have to be recompiled.
Showing my ignorance, I’m not sure what you mean by “static linking”.
A shared library is a collection of subroutines. In general, I would
expect to call it with a subroutine number (which is indexed into a
jump table) and some arguments.
The library would be loaded into RAM on first being called, and freed
when no program is using it. Most libraries are around 30k in size
(looking in the shared libraries directory here) with only a few over
100k, so I doubt if they will fill up a Gig of RAM unless you are
running a huge variety of software.
>> “Most libraries are around 30k in size”
You shouldn’t look at the size of the library at your HD.
You should look at the size in memory..
If a library allocates 2 MB, and every instance does it, it takes a lot more than those 30k times the number of instances and they _will_ eventually fill up your Gig.
Well, I don’t know what you think about it.. but I know many better things to do with my memory.
“If a library allocates 2 MB, and every instance does it, it takes a lot more than those 30k times the number of
instances and they _will_ eventually fill up your Gig. ”
That would be an abnormally extravagant bit of code.
The library itself is only loaded once, so you are actually thinking
of the memory that might be used (for data manipulation presumably) by
one of its components, which might indeed be called by several
programs at once.
I guess the most likely type of routine to need so much memory would
be a video codec. How many copies of that would you be running at
once?
In any case, if you do want to perform that task several times
simultaneously, it will make no difference whether the routine is in a
library or not – the total data space needed is the same.
The point of shared libraries is not to save on memory but to save on
coding effort. Somebody writes a useful collection of routines, and
everybody can use them. The same routines would work just the same if
each author had to write them himself. The amount of memory saved is
small.
I still fail to see why a programme could work with version 2.2 of a
library and stop working with version 2.3 or 3.0, unless there is some
very dodgy coding going on. Of course it may not work with an earlier
version, which could be missing a new routine, but it should always
work with any later version.
Clear, complete and informative.
When you have finish with the sdk, the ee engine, a little jre to test client, put jdbc jar files to the right directory, try to find a intelligent classpath file ( approx 2 hundred lines… ), you are ready to install jakarta aka tomcat / ant, don’t make any error on classpath…, well and now ready for some jswat, be carefull with the classpath…, some junit, back to jakarta / regexp to also test swing, don’t forget the classpath, and, phew, ready for some xml, xerces, xalan, a little xindice, back to jdbc jar file…, cocoon to link them, phew, be carefull with the classpath ( now 4 hundred lines, and, obviously, no error… ), maybe are you ready now for some corba orb ? Not to bore you too much, i avoid versioning problems… But maybe a little word about chmod and chown to secure all this, just before looking at the proxy and firewall, of course if you’re a fan of corba, do not hesitate to link all this to orbit and xinetd.
But that’s apart, installing Java in child simplicity : it’s rock ;-)))
>> “The library itself is only loaded once”
We were talking about statically linked libraries, not shared libraries.
This was answer to a post praising the “simplicity” of Java installation via jedit plugin install, but this post has now disappeared.
Sorry.
and I am lost. I read the readme and got stuck on the first line.
I wrote $ cd libprefixdb in an open Konsole and I was informed me it was an unknown command.
Please be even more detailed for newbies like me, even include the opening a Console and where it should go etc.
Or maybe you can provide an RPM for it
(Actually I am serious)
Why doesnt Linus (I think thats the original makers name)make a packager
That way No one person will try to take advantage of the monitary aspects of Linux & everyone will SURELY follow his lead wouldnt You think?
Just a newbie suggestion
P.s. If ANYONE would have a clue how this would be possible…. Dont You think it would be him?
A program might stop working with newer version of a library, because the developers changed the functions they called. Be it adding the function signature (return type, and argument list), or renaming it, or removing it completly, or making it behave differently (which is the worth kind really). You might wonder why anyone would ever want to do this, but they quite often do if they realize that their old way of doing things sucked, or they simply needed to change things to make it scale better, etc. Usually the major number is increased when backwards compatibily breaks, and you can then choose to simply have more than one version of the library installed.
As for static vs dynamic linking.
With dynamic linking you load and call functions in a .so file. This way all programs are using the same copy of the library, and often it only has to be loaded into memory once, even though serveral programs use it.
If you link staticallly, you take the library, or at least the parts that are used, and add it to the binary. This way it can run on systems that dont have the library, at the expence of it being bigger and requiring more memory (as it cant share the library with other programs) This is also why it would have to be recompiled (actually relinked) if a bug is fixed in the library. This is why all statically linked programs will have to be recompiled when the library changes. (as they have their own copy that they would otherwise just keep using happily)
I switched to Debian because of the current package dilemma with RPM. It’s been three months now and I highly recommend anyone who just cannot take RPM anymore to give Debian a shot. There are certainly tradeoffs with a more difficult installation and less GUI’s but you learn more at the same time. In the three months I have not had to fight with a package yet or hunt down dependicies. If you make it far enough give “aptitude” a try instead of “dselect”. It’s far less intimidating and more forgiving. I also use FreeBSD and believe the ports and packages that it offers are also better than RPM. However, there are far more broken ports in FreeBSD than Debian, at least for the packages I use. VMWare and any of the programs that use Linux emulation in FreeBSD are a crap shot. Sometimes they work, sometimes they do not. In the end I use FreeBSD for its simplicity and as a firewall. IPFilter is so much nicer than IPTables, just my opinion after using both for three years. I can get a FreeBSD firewall running in 1 hour but it still takes days to get IPTables running correctly, its overly complex.
(1) Security is going to be a major concern with this. I need to know that the package has not been modified and/or rebuilt since it entered the node/repository (n/r). I also need to know that it REALLY CAME from the respective n/r, and the place it came from before that.
(2) A node system would actually be very good for this, especially if the nodes were regularly synched with the locations (if not the actual) packages on the network. This way Joe SysAdmin with his 2 gigs he wants to donate can host a few packages and tell you where to get the ones he doesn’t have.
(3) A way of notifying the autopackage’s sysadmins of security flaws in their currently installed packages would be a nice feature.
(4) A package database on each system is not really a bad idea as long as it’s not a single point of failure… what I mean is that the system should be able to reproduce it’s package database without too many issues (well I can dream can’t I?).
(5) System Restore… great idea, horribly implemented. It doesn’t mean anything to me that all of Program A’s dependancies are solved and installed on my system correctly if the program still doesn’t work due to either the package system breaking or anything else going wrong. (note: Murphy’s Law). I think you have an idea of what I’m trying to say here…
You’ve got yourself in a whole mess of trouble by trying to create this, although it’s a very, very noble goal
By the way, I’m not trying to get you to create this giant beast of a packaging system, I’m just trying to get you to see the possible points of failure of such a system and try to dull the effects should something horrible happen (a package get compromised and distributed, etc).
Lastly I just want to say a big THANK YOU. Without the constant innovations and donations of time, effort, and resources from people such as yourself, nothing would ever happen in the Linux community. Once again, thank you.
“A program might stop working with newer version of a library, because the developers changed the
functions they called. Be it adding the function signature (return type, and argument list), or renaming it, or
removing it completly, or making it behave differently (which is the worth kind really). You might wonder why
anyone would ever want to do this, but they quite often do if they realize that their old way of doing things
sucked, or they simply needed to change things to make it scale
better, etc.”
In that case, the name of the library should be changed. If there are
really programming practices like that going on, no wonder there are
problems.
Anyway, it has become clearer to this non Linux user. The “statically
linked” libraries are programmers’ aids, chunks of open source code
that can be incorporated into any program at the time of writing.
What I am used to are shared libraries that exist as separate files on
disk, typically in a libraries directory. There would only ever be one
copy of each in memory, and people cannot make arbitrary changes to
the list of arguments or the returned values, although new variables
can be added to the end of the list.
Actually, any program that is fully scriptable also acts like a
library, in that you can access its routines from within another
program. But a normal shared library can’t be used as a standalone
program.
Can I just quickly say, as a cheeky comment, the author, despite what he says, has only had EXTREMELY limited use of OS X, on my own machine ( i’m the mac-head friend 😉 ) =P
LOL. Just had to get my 2 cents in there!
I’m on your side Nathan ! hehe
Can I just quickly say, as a cheeky comment, the author, despite what he says, has only had EXTREMELY limited use of OS X, on my own machine ( i’m the mac-head friend 😉 ) =P
Hi Paul! I was wondering when you’d turn up here :p Well, my comments about OS X stand, it’s not like you need 4 months experience to see how it works and I used appfolder systems on RISC OS before that too….. and anyway, as I’ve said, it’s not workable on Linux at all so it’s kind of a moot point.
I wrote $ cd libprefixdb in an open Konsole and I was informed me it was an unknown command.
The $ at the start indicated it was a shell command, not something to be literally typed. It is kind of pseudo-code as well because libprefixdb is not the name of the directory exactly, you’re supposed to use tab completion to get libprefixdb-0.2 or whatever the version number is.
For now I suggest you wait a while – the screenshots basically sum up what can be done at present. Hopefully in a few versions time we’ll have autoinstall done and you will just be able to download and run a .package file to get everything you need installed and setup – we are targetting this at newbies such as yourself as well as power users, but not all the userfriendlyness stuff has been written yet.
Or maybe you can provide an RPM for it
(Actually I am serious)
Yeah, maybe. At the moment we intend to display messages and also perhaps ask a couple of questions during the setup stage – also the components that will be downloaded and installed will differ for instance depending on whether you want the GTK2 front end or the KDE/Qt front end (or both) etc.
Why doesnt Linus (I think thats the original makers name)make a packager
Linus is a kernel engineer – as you may have noticed he has his hands full just making Linux the kernel a succesful project. He never ever gets involved in other stuff – Linux the OS really does not have any leader, and that’s a good thing btw. It is so large these days that it operates more like an economy with the best solutions winning out over the worst (or existing side by side as with the various user interface projects around).
(1) Security is going to be a major concern with this. I need to know that the package has not been modified and/or rebuilt since it entered the node/repository (n/r). I also need to know that it REALLY CAME from the respective n/r, and the place it came from before that.
Indeed, this has been mentioned before. Not sure how to treat this – there is no such security mechanism for Windows or MacOS installers and I’ve never heard of a rogue installer blowing up systems. Yeah, I’m aware of the trojaned code that was going around some time ago (was it bitchx?) – this is more an issue of mirror security imho.
(4) A package database on each system is not really a bad idea as long as it’s not a single point of failure… what I mean is that the system should be able to reproduce it’s package database without too many issues (well I can dream can’t I?).
The “database” is something of a confusing issue with autopackage, especially as I originally said things like “we don’t use a database”. Obviously this is not true, we record information about how to uninstall packages and the relationships between them in a database (in /var/packages). What I meant was that we don’t use a database for dependancy checking, ie we don’t assume there is a database of already installed software.
The database takes the form of files, directories and symlinks, and as such can be easily backed up. There isn’t any obvious way I can think of to rebuild a database from the system.
I don’t experience any problems installing software using debian. Using the Synaptic frontend to apt-get you just click on the package you want installed – click install and it handles the rest for you. It’s wonderful.
http://www.debian.org
download a copy
http://www.linuxiso.org/distro.php?distro=4
screenshot of synaptic
http://www.dive.nl/images/articles/2002061711475754_1.jpg
I’m still not 100% sure you got the difference between static and dynamic linking, or maybe i just didnt understand your wording, which is quite likely as im hung over 🙂
Your description of shared libraries is correct, they are indeed files (usually in /usr/lib, but in many other places as well), and they are only loaded once.
It could be argued that these should always be backwards compatible when you upgrade them, but it is not like this. One reason is that you can just keep the old version installed as well as the new version. Another reason is that if all new versions were backwards compatible, they would end up being huge and unmaintable, and possibly support a lot of things that really were bugs, but that people use. It might also contain a lot of work arounds, as they had to redesign most of the internals to support new features. I think this is done correctly currently, and i dont think they should change the name, just the version number, and you can then keep the old library for compatibility.
As for static linking, thats like taking one of the shared objects, and stuffing it into the executeable, so each executeable would have their own version of the shared library, which would both make the file size, and memory consumption goes up. Most commercial software is static linked so it will run on all linux systems, without the system having to have the libraries needed to run it. So while linking statically seems to get rid of the dependency problem, it really creates a much worse maintenance problem regarding security updates, and they make the hardware requirements go up.
“It could be argued that these should always be backwards compatible when you upgrade them, but it is not
like this. One reason is that you can just keep the old version installed as well as the new version. Another
reason is that if all new versions were backwards compatible, they would end up being huge and
unmaintable, and possibly support a lot of things that really were bugs, but that people use. It might also
contain a lot of work arounds, as they had to redesign most of the internals to support new features. I think
this is done correctly currently, and i dont think they should change the name, just the version number, and
you can then keep the old library for compatibility. ”
That sounds like a recipe for chaos to me.
You would have two (or more) files with the same name in one
directory. The program would have to specify the exact version of the
library that it was calling, instead of the earliest acceptable. You
would end up having to keep on disk every version of the library ever
released, in case some program might ask for it.
If the library is well coded and tested in the first place, I see no
reason why new versions should end up huge. Surely it is up to the
author or maintainer to make sure this doesn’t happen?
The kind of mess you describe is the sort of thing I associate with
Windows.
I’ve had multiple ideas for ways to implement package level + repository security.
(1) A PGP/RSA ID inspired security measure embedded into the packages.
One idea I’ve had works like this: the package author generates a private key which is embedded into the package. Upon package installation, a public key is used for extraction(which obviously in some way must match up with the author’s private key that is embedded into the package). Since only that private key will work with that public key, this might help to insure package level security. I need to brush up on my Advanced Cyptography book, but a key exchange system sounds very ‘do-able’. The package itself also might contain a private key from the repository itself.
(2) What if the pseudo-pkgdb (package database) could be regenerated from the filesystem (and possibly with the aid of a package node/mirror)?
Say I’ve got a directory called /etc/autopkg/cache/ which holds the pkgdb. Say this gets deleted! (Oh no!) No problem. The package system after being passed a flag, starts scanning the system for packages. So that the system knows about the constantly on-going ordeal of handling package dependancies, it connects to one of the package distrobution SuperNodes (which in my opinion is still one of the best and easiest ways to get the autopkg network to scale with a certain level of redundancy) and scans a ruleset among other things, of what the installed packages may look like on the filesystem, as well as knowing what packages to look for from here. The installation ruleset could also be synched as often as my earlier proposed package mirror list that would reside on each node and/or supernode.
This is a lot of stuff to deal with… some of it might work, some of it might not. Hell at least I’m trying to help give you ideas here
In fact,most package managers in use today would be more than satisfactory, if the platform they’re deployed in is managed carefully, from a development point of view. If your platform is changing every two to three months, if changes in the libc library are likely to screw up your 1-year old applications, and so does the everchangeing kernel, then there is no package manager that will help.
One very much screwed platform, from this point of view, is Linux. This issue has to be addressed before Linux makes it big in corporations.
In fact,most package managers in use today would be more than satisfactory, if the platform they’re deployed in is managed carefully, from a development point of view.
Indeed, but management implies central control, which is not something that’s normally associated with Linux.
If your platform is changing every two to three months, if changes in the libc library are likely to screw up your 1-year old applications, and so does the everchangeing kernel, then there is no package manager that will help.
That’s an exagguration. The desktop platforms change in a binary incompatable way every 2/3 years, and this is because the Linux desktop is still rapidly maturing. Server apps for instance don’t suffer this volatility. glibc practically never breaks backwards compatability (although modern apps often require a modern glibc). Applications don’t directly interface to the kernel, that issue only affects drivers.
Congratulations to Mike Hearn on a great initiative.
I would like to reiterate Josh Reynold’s comments re security. Having worked in a large financial services institution, I cannot stress enough that security must be built-in from the beginning.
Even the simple starting point that a user can select a trusted source(s) of the new files that will be installed on their machine. There have been enough recent examples of hacked sites having downloadable programs replaced with trojaned versions to highlight the importance of this.
Including support for digitally signed files would be a good start in this direction. For example, I will only accept files that have been signed by person/company abc, xyz, etc.
I got few questions:-
1. Can autopackage be used to install whole distro from scratch?
2. Listing of location and installed files?
3. Upgrade an RPM package using autopackage (to autopackage format)?
4. Integrity/security of the package?
5. Can a downloaded *.tar.gz be used to install?
I think autopackage is great.
i do not think that rpm is so bad! actually, it rocks… of course, you need to know what you are doing (quite uncommon for many drag and drop windows programmers, now wondering what unix and linux are all about… 🙂 lol
the article is irrelevant… e.g. “What this means is that it’s optimized for transferring large amounts of software to an OS from a CD or Redhat update site.”
so the article is totally atechnical (and also smells like microsoft rummors…) what rpm dependency hell?!? are you stupid?
1. Can autopackage be used to install whole distro from scratch?
No, although a theoretical possibility autopackage is too slow to presently make this a reality. It’s designed to work in conjunction with RPM.
2. Listing of location and installed files?
You can list files installed by a given package, yes. We don’t have a “what owns this” feature (yet).
3. Upgrade an RPM package using autopackage (to autopackage format)?
No, you’d have to uninstall the RPM then install the .package
4. Integrity/security of the package?
This is a pretty common question. We may offer some limited integrity checking, for instance on networks being able to lock down machines so they can only install packages from trusted sources, however fully blown trust networks etc are currently beyond the scope of the project. Maybe post 1.0
5. Can a downloaded *.tar.gz be used to install?
If it contains an autopackage specdir then you can run the makeinstaller script to produce a .package file, but it cannot build a package from a raw tarball. Really, there should be no need for this – it’s designed to interoperate with other forms of software installation so if you want to/have to install from source, that’s fine, it won’t affect autopackage.
so the article is totally atechnical (and also smells like microsoft rummors…) what rpm dependency hell?!? are you stupid?
You are deluded. If you have never spent half an hour just resolving dependancies to satisfy an RPM you want to install, then you are indeed lucky. Go talk to pretty much any other Linux user and you will see the truth of what I say. Microsoft rumours? Read the article, I criticise the Windows way of doing things.
A shame that the otherwise nice Debian packages are such a royal pain in the dark hole to actually make. I currently struggle to make a .deb of a simple litte game, and it takes hours and hours to get it to build. Doesn’t seem to integrate with automake too well, requires dirs to be named in some weird fashion, then there’s a few million files that aren’t really documented and so on. It can be done, I’ve done it before, but it’s not simple.
If I didn’t otherwise like Debian I’d say it sucks really badly. Making RPM:s is *much* easier, especially with all nice tools to help the process.
and yes it is indeed the apt/rpm combo that is doing this.
Have a look at these pages:
http://apt4rpm.sourceforge.net
http://apt4rpm.sourceforge.net/web-apt-rep.html
The last link shows that apt is not build around a
centralized repository. It is just a p2p network that you
have proposed by the AP project.
SuSE users can find more information about apt at:
http://linux01.gwdg.de/apt4rpm
With apt it is even possible to upgrade an rpm distribution
from 8.0 to 8.1, without distributor support. Were is the dependency hell, if this possible?
http://linux01.gwdg.de/apt4rpm/linux01.upgrade-80to81.html
In case one installs from source, use checkinstall and
apt in combination with rpm are again satisfied. No
dependency problems.
The problem with different locations for the same files
at different distributions is solved by the automake/
autoconf/configure combo.
Apt is frontend independent, it is library based. Today
there is a command line interface and a wings and a gtk GUI.
The apt/rpm combination can be used to fully automate
system maintenance, which is nice if you have to maintain
thousands of systems:
http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/apt4rpm/apt4rpm/clie…
Apt/rpm itself are distribution neutral, check out
http://www.distrowatch.com/summary.php to see how many
distro’s support it.
RPM is net oriented as well. SuSE provide e.g. patch rpms,
these rpms provide only the patched files. A package like
kdebase3 can sometimes shrink 50% (8MB) by using a patch
rpm.
APT repositories can be gpg signed (it’s just supported),
making the trusted p2p network possible today.
It’s a pity that the autopkg energy is not put in e.g.
making the signup p2p network work for apt.
anyone whois ever ?worked? on windose for 10 seconds gnows what i mean.
all those payper liesense agreedmeNTs, the bsod, etc……
why would anybuddy do all that?
and yes it is indeed the apt/rpm combo that is doing this.
I went into the problems of apt in the article. In particular, it’s
a) tied to packaging systems such as RPM and DPKG that are distro-specific, there is no good way as far as I can tell to make RPMs sufficiently flexible to be distro neutral, so you have to have separate apt networks for every distro.
b) dependant on a database of all previously installed software. Things like the RPM database can quickly get out of date and being binary, easily corrupted. A corrupted RPM database is practically the only thing that can utterly take down a Linux box and require a reinstall
Checkinstall is a nice hack, but what about stuff like garnome that automates build systems?
In case one installs from source, use checkinstall and
apt in combination with rpm are again satisfied. No
dependency problems.
The problem with different locations for the same files
at different distributions is solved by the automake/
autoconf/configure combo.
IMHO that isn’t a solution at all. Source tarballs cannot be automatic dependancy resolution and take forever to compile (i believe it’s quite reasonable for people to not want to compile software). Checkinstall is a nice program, but it doesn’t work when you use port systems such as garnome – resolving one problem invalidates the resolution for the other.
Apt/rpm itself are distribution neutral, check out
http://www.distrowatch.com/summary.php to see how many
distro’s support it.
You are confusing two things – apt4rpm can be used on many distros yes (I use it here at work) but for each distro a new repository must be built and maintained. Of all distros with apt, only debian really gets close to having enough packages for apt to be a whole solution, and they have problems with inertia and timelyness. Even RedHat, the most popular of them all, only has a tiny fraction of all available software available via apt.
My aim is to build something like apt, but with a distro neutral packaging format and a DNS style resolver system. This means that you can have one network for all distros, and you don’t need to manually add sources. I won’t be reusing apt, because it relies upon a central database of all installed software to predetermine and calculate a dependancy tree, whereas we use cascading dependancy checks instead.
1st I can only praise you Mike, for such a good idea, very commandable goal.
2nd one of the post mentionned upgrade, Today I use autorpm
to check for new rpm and download them. It may be out of context or the autopackage task but have you thought/made provision of automatic download (and possibly install) of new packages as in autorpm?
Dom
While I am no fan of RH or RPM, I feel that some of the dependancy problems can be overcome in RPM if the standard naming convention is faithfully followed; a standard naming convention does exist, for packages. I am not sure if such a standard exists for library/file names.
My biggest problem with RPM is you cannot choose where to install unless you change the ‘/’ dir, which is not really an option. Backing up the database is also not trivial ( akin to backing up a registry) and is useless anyway if the filesystem changes. I should be able to backup say the DB, go to a different (vanilla) system/distro and say restore the DB and it should download files/packages and restore it to the same state !
Also, if you install s/w from tar.xx files or compile source and install, common on unix/linux, RPM is not very useful ( unless you build your own RPM).
O.S. developers MUST learn to standardize on libraries – everyone shouldn’t create a branch of glibc, gtk, freetype, QT etc !
While I am no fan of RH or RPM, I feel that some of the dependancy problems can be overcome in RPM if the standard naming convention is faithfully followed; a standard naming convention does exist, for packages. I am not sure if such a standard exists for library/file names.
Yes, but the distros don’t follow that naming scheme and because they all have large installed user bases it would be hard for them to change (there has been no indication of them doing so of their own accord either). We hope that by leveraging off of DNS we can avoid naming conflicts and namespace fractures.
My biggest problem with RPM is you cannot choose where to install unless you change the ‘/’ dir, which is not really an option
Indeed, this is an option for autopackage (though it normally requires patching of the program).
Backing up the database is also not trivial ( akin to backing up a registry) and is useless anyway if the filesystem changes
It also gets frequently corrupted registry-style too. The equivalent in our system is a set of directories, files and symlinks which can be backed up (incrementally) using tarballs. You can use the standard filing system commands to alter it at will, useful for testing as you might imagine!
Today I use autorpm
to check for new rpm and download them. It may be out of context or the autopackage task but have you thought/made provision of automatic download (and possibly install) of new packages as in autorpm?
Yes, this is a feature we want to have. We’ve also toyed with the idea of having indexing servers which you upload your database to, and they monitor new releases and send email when updates are available (but this might not be necessary).
I’ve recently converted to Gentoo because I like the way it manages software, but even so it has its limitations and isn’t for everyone.
I really hope autopackage takes off, because if it does it will become one of the most significant milestones in Linux development.
Just checking out your replies to my comments every few day or so, and I’m actually suprised and encouraged that you read (or at least) skim through alot of the comments posted here.
Got a few questions for you however.
(1) Do you really think that autopackage will be used? I mean, even if you made the greatest package manager on the planet, Gentoo is not giving up emerge, debian is not giving up apt, and RedHat/Mandrake/etc. are not going to give up RPM anytime soon. This being the case, where do you plan on having autopackage implemented, and how well will it integrate into an already installed system.
(2) What are your current project goals for version 1.0, and what can we look forward to in the future (post 1.0)?
(3) What current security checks have you considered and/or implemented?
Your reply would be greatly appreciated.
Just checking out your replies to my comments every few day or so, and I’m actually suprised and encouraged that you read (or at least) skim through alot of the comments posted here.
I love feedback
Having said that, this story has now fallen off the front page and I probably won’t be checking back again. If you or anybody else has more questions, please email me.
(1) Do you really think that autopackage will be used? I mean, even if you made the greatest package manager on the planet, Gentoo is not giving up emerge, debian is not giving up apt, and RedHat/Mandrake/etc. are not going to give up RPM anytime soon. This being the case, where do you plan on having autopackage implemented, and how well will it integrate into an already installed system
We don’t need distros to integrate what we make, although it would be nice – .package files are self extracting and can setup autopackage themselves if it isn’t present. Indeed, it’s not meant to replace RPM at all, it’d be too slow to deal with all the software for a whole OS. It’s meant to compliment RPM and this includes integration (ie manipulating the rpm database when an autopackage is installed).
(2) What are your current project goals for version 1.0, and what can we look forward to in the future (post 1.0)?
See the schedule on the site for a brief overview. More details can be found in the TODO. Post 1.0 probably extra nice to have features that aren’t actually required to solve the problem but might be useful for corporates/specialist needs etc. I will probably wind down my personal code writing after 1.0 and concentrate on evangelising the network and getting people to build .package files. Some time after that I’ll probably pass control of the project and the network to other people – Linux has more problems to solve than just this!
(3) What current security checks have you considered and/or implemented?
Well, I have answered this question before in above replies, it’s a pretty common query. If you can’t find the answer you’re looking for here then please join the dev list and ask there.
thanks -mike
Sounds fine, but there are already at least two Linux projects called Autopackage, both for Slackware, strangely enough. They do completely different things, but the confusion is unnecessary.
Some alternate names to try:
–Super Mega Power Package Deluxe Pro Awards Edition
–SmartParcel
–Zurp
–PackageMan