uPM - The micro Package Manager

Acknowlegements

Christoph Lameter, September 27, 2002

Why UPM?

uPM is an attempt to react to that challenge with a new way of packaging. The technology as always will also impact the way things are done.

1. Extreme reduction of the amount of information maintained on each package. Ideally procedures should be available that allow an automated retrieval from an upstream site with fully automated incorporation into the distribution. One of the greatest problems with maintaining package using dpkg or rpm is the need to keep the installation scripts current. Installation scripts of multiple packages mostly only do a limited number of tasks in a variety of ways. If changes are needed then all those scripts have to be changed. This is an errorprone process and releases are very difficult because there is always something broken in the packages. We need to do a radical change on that front. The amount of scripting for each package needs to be reduced by a factor of 10 to 100. Similar functionality in scripts of multiple packages needs to be centralized.
2. Separation of package integration issues that are distribution specific from development activities which need to be done by the open source project the source code is obtained from. Debian for example is frequently carrying what amounts to forks of upstream packages (X, Kernel etc). Other distributions encounter the same phenomenon. Often patches are not submitted and not integrated to open source projects. The package manager needs to support patch patch writing that is directly submissable to open source projects. On the other hand distribution specific installation does not need to be done with patches. In my opinion it would be best to form real upstream forks and not use the packaging to provide enhance functionality. Having an upstream fork eases access by others to the improvements made and puts pressure on the orignal upstream authors to integrate changes. It is the best way to avoid having to maintain a fork over a prolonged time period using packaging tools.
3. Standardization of build procedures in open source projects. There are already established customs on how software is to be build. Automake is one example of an attempt to formalize the build process. The package manager needs to provide a clean interface so that ideally no patching of sources complying to standardized build procedures is necessary for package integration. Patches that have to be done are to be of a generally nature which will warrant their inclusion in standard sources and allow the package maintainer to use these patches unmodified as obtained from the other sites.
4. Simplification of package management. Package managers should not work in a mysterious ways but be transparent and simple to operate. The metadata that the package generates needs to be easy to handle and change.
5. Comprehensive Management of all files on a System. Current Linux system already need tens of thousands of files for a well build out system. It is to be expected that future complex Linux systems will contain hundreds of thousands of files. The packagemanager must provide facilities to monitor file usage, audit problems and be able to insure the integrity of software installed. The system administrator cannot be expected to know the role of each individual file.
6. Upgrades must be fully automated. Prompting for user input on an upgrade problem is not very helpful these days since the complexity of Linux systems has increased. The user or administrator will not know what to do about a configuration issue. The package manager must provide intelligent schemes for unattended upgrades and refuse to install a new package if an upgrade cannot be performed without user intervention. If a user decision is forced, as reflected in the implementation of configuration file management in package managers today, then the user will make a decision which might be bad since insufficient knowledge about the issue is available. The current situation is that RPM simply moves the configuration file out of the way hoping for future user intervention. Gentoo does the same and provides a tool that asks lots of questions on how to treat these configuraton files. Debian prompts when the config file was changed from the default. All these approaches will not work if a system is being run by a non-unix person. There are a variety of approaches to work around these issues in the package managers (debconf, taking the management of configuration files out of the hands of the package managers etc). The problem though is an deficiency in package management and needs to be solved in the manager itself.
7. Flexible Build procedures. It should be possible to automatically build packages from source automatically customizing the build procedure to the needs on a special system and the desired overall configuration of software on the system. Debian has the custom of offering a variety of packages based on a common open source upstream package to address that. But if the build process can be customized then the package variety is unneccsary because custom versions can be build as necessary. This will reduce the number of packages needed. Build procedures need to sense the environment and then do an automated efficient build for the machine. Essential build characteristics need to be tracked by the package management system. Gentoo is allowing the specification of build options but does not effectively track that information which results often in a very fragile system.

uPM - Micro Package Manager

uPM is a package manager that attempts to address these issues by offering an integration of a builder, a classic package manager and an advanced management of package relations (like known from apt).

• Follows the Unix philosophy of keeping things simple. Reduce overhead typically associated with packaging software to a minimum possible. Very easy and fast packaging is possible. I was able to do more than 200 packages in the first month while developing the whole thing.
• Clear and concise metainformation in a single file with a very intuitive format allowing the total control over the build and installation process. Have a look at the almost 200 upm package descriptions already available here.
• Machine processable package description, The metadata is constructed in such a way as to easily processable by programs that can verify or enhance the package description as needed. The machine processable nature of uPM metainformation is essential for the maintenance of a potential huge archive of software package descriptions.
• Support for multiple stages of package building. uPM recognizes three different stages: First the RECIPE where just the highest level of metainformation exists. This basically contains locations of sources, patches, the procedure to build and references to centrally stored procedures on package maintenance. The RECIPE is usually written by a package maintainer and then enhanced and checked by various tools. Then the BUILD which results in an installable package. It adds more metainformation and makes the package specific to an architecture (such as Intel, Sparc, Alpha..). A package in build state is comparable to a .deb or .rpm package. Typically UPM stores these packages in a .tgz file with integrated metainformation. The metainformation can be also be stored somewhere else with a reference to the .tgz file so that the package manager can make an informed decision before choosing to download or unpack that .tgz file for installation. Finally the INSTALLED stage in which the package has been deployed on a system. Metainformation has been added again to complete information about the state of the package on the system. In the installed state all files of the package are tracked.
• Control of files overwriting. uPM does not allow random file overwriting (gentoo, dpkg, rpm all have attempted to solve this issue but have given up for various technical reasons) but tightly controls files being installed by packages. Blacklists and package options can be used to control that feature if file overwrites are needed in special situations (See for example the manpages package). Two overwrite strategies can be selected if a package needs to be forced into the system. uPM will track these events and can also detect that an outside source has corrupted a file. Tripwire clone build in.
• Secure build procedure. uPM uses fakeroot for all build operations. No real root access is needed or given during build operation. Builds essential run as a regular user that cannot impede the operation of the system as a whole. Packages can be build from a recipe by a simple Unix command.
• Use of Triggers, events, blacklists and other meta tools instead of maintainer scripts that repeat the same code over and over. A package does not need to provide any script for example if a library is added or a font to X11. The fact that a new file is deposited in certain directories causes an action that will update metainformation as necessary. The action can be defined by the package providing that service and does not need to be implemented by the packages using that service. A trigger is associated with the directory containing libraries by the package providing library services. If a new library is installed the trigger executes the necessary scripts in order to update the necessary metafiles. Instead of having a call to ldconfig in lots of postinst scripts there will be only one trigger. Similar measure can be taken for other situations. This vastly increases the reliability of the system, reduces the complexity and error proneness of installation scripts and takes a lot of hassles off the package maintainer. uPM preserves the ability to script the installation process through events. Events are much more flexible than the classic maintainer/ebuild script since they allow reference to centrally stored information and can react in a custom way depending on the configuration of the system. See an example of a package description for GLIBC.
• Meta-Packages allow the accumulation of package sets. A single uPM command can install complete subsystems. Meta-packages can also be used for building. They store common bits of the procedures needed to build packages. See for example the c-build or shell-build packages.
• Inheritance of build instructions from libraries and metapackages Libraries and metapackages can generate instructions for packages that have build dependencies on them. A whole class of recipies is simplified by such an approach. The kdelibs package for example generates special instructions so that other KDE packages just need to have a very small recipe.
• Advanced Configuration File Management. No prompting for changed configuration files anymore. UPM keeps a copy of the original configuration file around and allows the display of a diff to see what changes have been done. Configuration states of a package can be saved and retrieved. UPM can restore a package to its default configuration. On upgrades a diff is generated to the earlier config file. That diff is then applied to the new configfile. If that diff would fail then the upgrade aborts.
• Flexible Package Attributes so that any desired information about a package (such as source configuration options) can be tracked and used for dependency or build time configuration. Attributes can be used to construct complex configuration and build instructions for the source while allowing for accurate tracking of that information. Attribute construction can be partially centralized and partially package based so that the customizability of the individual package is preserved while reducing the amount of work needed to customize. B Records track the state of all attributes actually used during a build and can be used to track all elements that have influenced the build of a package.
• Blacklists control the installation of files to various areas of the file system. This is a way that files provided by multiple packages are handled. Files can simply vanish into a black hole or acted upon. The package manager can declare section of the file system tree to be off-limits for packages. Packages can lay claim sections of the file system so that proper cleanup can be done on removal.
• Clean removes. uPM tracks directories created by packages. A package that is removed vanishes completely and does not leave junk floating around the file system. The same is true for an upgrade of a package.
• Easy user environment settings control. Settings such a MANPATH, custom shell environment settings and so on can easily be set directly in the package description. V Records are used for that purpose. New hierachies can be generated with ease.
• Dependencies activated based on feature configuration. The build system allows to determine the configuration of the source code dynamically. That implies that the dependencies of a package can also change dynamically. uPM fully supports conditional dependencies.
• Tracking of packages used to build other packages. Build dependency cause an automatic inclusion of the version number of the depended on package, so that information about tools used to build packages is readily available for dependency checks and other queries. A simple command can be given to rebuild all the items that have been generated with some build tool that has a known problem or to insure that all software has been build using the newest tools.
• Simple realization using C and Shell scripts alone. No complex dependencies on other system components. uPM can be deployed in minimal environments and does not break when other components (like perl(dpkg) or python(gentoo)) are in "transitionary stages" or have been compromised in some way.
• Recovery tools and system audit capabilities All significant activities are logged in /var/log/upm.log. The history of changes to packages is logged separately so that a history of the files of a package can be produced.
• Extensability uPM provides hooks so that add-ons can increase functionality which might implement arbitrary package file formats, manage available software and build packages from source.
• No Base System requirement Most package managers cannot work without extensive functionality around them. uPM allows the building of small installation directories trees. For example an initrd environment of a few hundred k can be easily managed by uPM. A new root tree can be initialized from a single package. uPM can be run external to the root tree that it is supposed to manipulate. The initrd environment does not need to include uPM in it to be manageable. uPM has minimal overhead for meta information stored about packages so that it is feasable to use it in situations with very limited capacities.
• Scalability While uPM will happily cope with minimal resources it is still able to scale very well. uPM uses hashed algorithms for all search operations which allows very fast operation even with hundreds of thousands of files.
• Self-determination of Metadata If the build package provided to uPM does not provide dependency and other metainformation then uPM will by itself generate dependency information from:
  1. The library dependencies of binaries.
  2. The script interpreter used by scripts.
  3. The directories owned by other packages that files are installed in.
  Packages will be rejected that have unresolved dependencies. If metadata is provided in the directory tree then dependencies are checked and validated against self-generated dependency information. The approach here is the same as the one originally taken by alien the linux package converter I wrote in 1996 and debmake written in the same time frame.
• Logging of actions uPM logs all significant events so that it is possible to find out what has happened to the system should something go wrong. Daily checkups verify that all files are still in the installed condition and logs the presence of new files. No Tripwire is necessary anymore. uPM stores comprehensive information about each item installed.
• Priority Links support multiple packages providing similar functionality and allow the election of one of the packages to provide the standard binary.
• Direct control over /etc/init.d scripts. uPM directly interfaces with the init.d subsystem. Configuraton options to manage startup links are specified with uPM configuration comamnds so that no scripts are needed for initialization.
• Multiple patch support. uPM can apply any number of patches to a sourcecode archive. It can restore the sourcecode in any arbitrary patch or processing stage with a simple command.
• Easy source management. uPM can bulid a sourcecode directory at a requested stage of the build process with a simple command. Another command can generate a usable diff to be integrated into the recipe after changes have been made to the source archive.

UPM Sources available

• Installation CD. ISO image for booting a Telemetry/Linux system. This is a LiveCD. The complete system is available by booting from CD without installation. Autodetects hardware and network. Boots Linux in graphics mode and comes up with a webbrowser. DHCP used to discover network configuration. MP3 player working. Installation routine available but not tested yet. Two videos included but the video player is not working yet.

  The image can also be used to bootstrap a system via a chrooted environment (if you already have a linux distro installed). Loop mount the iso image and then copy the files out of the image. BEWARE: The files on the CD are compressed using ZISOFS. You either need to have ZISOFS support compiled into the kernel or use the mkzftree tool (zisofs-tools) to decompress the file.

  mount -oloop /path/to/image/sputnik-0.25.iso /path/to/tree
  mkzftree -u /path/to/tree /my/chroot
  umount /path/to/image/sputnik-0.25.iso
  mount -t proc proc /my/chroot/proc
  mount -t devfs dev /my/chroot/dev
  chroot /my/chroot
  

  now play arond with the upm environment

  umount /my/chroot/dev
  umount /my/chroot/proc
  

  UPM Sources

• UPM-0.29 Implement reporting and syncattr functionality. Sources moved to ibiblio.
• UPM-0.25 Path fixes. KDE builds completely. Add archive maintenance functionality.
• UPM-0.24 Lots of bug fixes. /etc/suid.conf. Beginning builds of mozilla and kde
• UPM-0.22 Hashes for package lookup. Lots of bug fixes.
• UPM-0.21 Add virtual handling in recipes and builds. Lots of bug fixes.
• UPM-0.20 X, xchat etc work. Higher level package management realized.

A list of available package description is at ftp.ibiblio.org/pub/linux/distributions/telemetry/sputnik/recipe.

Packaging Software

Starting a package description

W Sed - stream editor
S http://queen.rett.polimi.it/~paolob/seders/ssed/sed-3.58.tar.gz

The package description can now be used as a parameter to uPM. For starters lets simply build the package and not immediately install it. That would work using:

upm build sed-3.58

Making the package description version independent

W Sed - stream editor
S http://queen.rett.polimi.it/~paolob/seders/ssed/{P}-{V}.tar.gz

With this we have used a string expansion. uPM has a quite sophisticated mechanism for managing a database of strings and string snippets that can be used to extensively customize the build process.

Why did this work?

1. Sed uses the standard autoconf/configure/make mechanism for package building and uPM will default to an autoconf build mechanism.
2. We have relied on defaults for dependencies and other important aspects of building.

./configure --prefix=/usr --host={HOST}

make

make prefix={B}/usr install

Setting the LICENSE attribute

W Sed - stream editor
S http://queen.rett.polimi.it/~paolob/seders/ssed/{PV}.tar.gz
A LICENSE GPL-2

Setting the build type

W Sed - stream editor
S http://queen.rett.polimi.it/~paolob/seders/ssed/{PV}.tar.gz
H http://www.gnu.org/software/sed/{P}.html
A LICENSE GPL-2
R BUILD c-build

Conditional Dependencies

Typically this is done with a conditional dependency and conditional configure string generation. The R BUILD settings take a string parameter for the package name. If that string parameter is empty then no dependency will be generated. Instead of specifying the package name we can simply specify that the contents of the global NLS attribute should be used as the dependency. If NLS is not set then no dependency will be generated. If NLS is set then a dependency on the package that provides build/runtime support for national language support will be included. The augmented package description:

W Sed - stream editor
S http://queen.rett.polimi.it/~paolob/seders/ssed/{PV}.tar.gz
H http://www.gnu.org/software/sed/{P}.html
A LICENSE GPL-2
R BUILD c-build
R BUILD {NLS}
A CONFOPT {NLS?:"--disable-nls"}

Verification of the software build

W Sed - stream editor
S http://queen.rett.polimi.it/~paolob/seders/ssed/{PV}.tar.gz
H http://www.gnu.org/software/sed/{P}.html
A LICENSE GPL-2
R BUILD c-build
R BUILD {NLS}
A CONFOPT {NLS?:"--disable-nls"}
A CHECK make check
A DOC COPYING NEWS README* THANKS TODO AUTHORS BUGS

Moving files around after the install

W Sed - stream editor
S http://queen.rett.polimi.it/~paolob/seders/ssed/{PV}.tar.gz
H http://www.gnu.org/software/sed/{P}.html
A LICENSE GPL-2
R BUILD c-build
R BUILD {NLS}
A CONFOPT {NLS?:"--disable-nls"}
A CHECK make check
A DOC COPYING NEWS README* THANKS TODO AUTHORS BUGS
E POSTINST mv usr/bin/sed bin/sed-{V}
Y bin/sed sed-{V} 50

The Y Record is used to allow one name for multiple binaries. An Y record specifies the common name and then the special name and a priority. uPM will pick the one with the lowest priority from the available sed. A more powerful sed could be installed and have a lower priority. Then the /bin/sed link would point to the new sed. If that more powerful sed is later removed (naturally it had some strange behavior) then the /bin/sed link will be repointed to the old sed. This insures that a sed version is always available.

Conclusion

UPM Sample Package Description

GLIBC Package Description

W The Linux C library GLIBC
H http://www.gnu.org/software/libc/libc.html
M christoph@lameter.com
S ftp://sources.redhat.com/pub/glibc/releases/{PV}.tar.bz2
P ftp://sources.redhat.com/pub/glibc/releases/{P}-linuxthreads-{V}.tar.bz2
P {P}-manpages-{V}.tar.bz2@man
P {P}-2.2.4-string2.h.diff
P {PV}-threadsig.diff
P {P}-2.2.2-test-lfs-timeout.patch@io 
P {PV}-dns-network-overflow.diff
P {PV}-sunrpc-overflow.diff
P {P}-divdi3.diff
P {PV}-gcc311.patch
R BUILD linux-headers
R BUILD gcc
R BUILD make
R BUILD binutils
R BUILD texinfo
R BUILD gawk
R BUILD sed
R BUILD {NLS}
R DEPEND base
G R BUILD gcc
G R DEPEND glibc
A O 2
A DESTSPEC install_root={B} {NLS?"localedata/install-locales"}
A CONFOPT --with-gd=no --without-cvs --enable-add-ons=linuxthreads --disable-profile 
#A CHECK make check
E POSTINST install -m644 {S}/nscd/nscd.conf etc
E POSTINST rm -rf etc/ld.so.cache etc/localtime etc/rpc usr/libexec
V PATH LDPATH /lib:/usr/lib:/usr/local/lib
T lib ldconfig #
T usr/lib ldconfig #
T usr/local/lib ldconfig #
A DOC BUGS CONFORMANCE COPYING* ChangeLog* FAQ INSTALL INTERFACE NAMESPACE NEWS NOTES PROJECTS README* 
A MAN man/*.3thr
V ENV LANGUAGE C
V ENV LC_ALL C
V ENV TZ UTC

The second line is a H Record which is used to give a hint where more information about the package can be found.

Third line is a M mail address where bug reports and patches might be submitted to.

Fourth line is the Source Specification. This is the location of the source code of the package.

The next lines are all Patch Specifications. These are patches to be applied to the source package. Patches can be either tarballs which are untarred into the sourcecode directory or real patch files. The first two patches here are tarballs. The second one has a patch fix. This is the @ and the man name. This tarball should not be untarred into the main sourcecode archive but into a subdirectory thereof.

The patch lines also use Macro-Expansion features of uPM. Any text surrounded by { } are variable substitutions and expansions to be done by uPM. {P} {V} are short hand references to the packagename and the version. Use of these macros allows the reuse of the same package description for future releases. {P} and {V} are extracted from the filename. Simply renaming the package description often allows the installation of newer releases. The glibc patches are a mess though. Usually this list is much leaner and easier to maintain. {PV} is a default attribute defined in upm.conf.

Note also the @io at the test-lfs-timeout patch. That patch has to be applied to a subdirectory indicated by @.

The R Records are used to specify relationships to other package. First there are two BUILD time dependency. A build time dependency makes sure that this package is installed at build time and it tracks the versions of the dependent package in the GLIBC package. A B record will be generated indicating the version used to build GLIBC.

Then some runtime dependencies follow. When the package is installed a verification is done that these packages are installed. The last depend line is a conditional dependency. It checks if the NLS attribute is defined. If so then the gettext package is also needed at runtime.

Then some strage G R records appear. These generate dependencies for packages having a build dependency on GLIBC. If a package uses GLIBC for building then it also needs glibc for a runtime dependency (G R DEPEND). If a package used GLIBC for building then it will also need GCC during build time (G R BUILD).

The A Records are used to override attribute settings in upm.conf. They only make sense when viewed in the context of the attributes define there which control the overall build procedure. Attribute records in package descriptions are used to override elements of the build process. Here the O attribute is set to 2 which forces the optimization during build to -O2. The DESTSPEC sets the way how the install target is specified during "make ... install". There is a CHECK of the installation possible. The CHECK definition sets up the build engine to run a command and verify that there is no error before building the binary package.

The CONFOPT settings are the options set when invoking ./configure. The example here shows a conditional generation of these options depending on global settings in upm.conf about how the software on a machine should be configured.

The Events for POSTINST perform changes in the binary tree after the INSTALL action has been run. Here another file is extracted from the source archive into the binary tree and then some of the directories are removed.

Triggers are set for common library directories. If other packages deploy new libraries the ldconfig script will be run without these packages needing any special scripts.

The MAN and DOC attributes result in appropriate actions to deposit and manage both documentation and man pages. The V records cause the setting of environmental variables or search paths. Here a searchpath is set for the typical locations of libraries.

Here is the attribute section of upm.conf (no this is not a current version but only a subset of an old one!) to make some more sense out of these macro expansions:

[attributes]
# These are lists of default attributes to be used for
# building programs. 
# Attributes defined here can be override by attribute
# specifications in the package or by package specific
# overrides later....

# CONFIGURATION STAGE
# The configuration stage. If this package does not
# support autoconf override this setting using
# A CONFIGURE
# CONFOPT can be set with additional options
CONFIGURE=./configure --prefix=/usr {CONFOPT} --host={HOST}

# BUILD STAGE
# This works with most packages
# Override BUILD if additional targets are needed
BUILD=make

# VERIFICATION STAGE
# The check stage is typically unused. Define this to override and actually
# perform a regression test or so.
CHECK=

# INSTALLATION STAGE
# Installation. Typically the specification of the destination
# varies. Define DESTSPEC attribute to fill that one out if the
# DESTDIR default will not work.
INSTALL=make {DESTSPEC} install

# Override this to specify your way of setting the binary
# destination to install into.
DESTSPEC=DESTDIR={B}

CFLAGS=-mcpu={CPU} -O{O} -march={ARCH} {MMX?"-mmmx"}

# Shorthand often needed
PV={P}-{V}

# Most important is the control of the gcc compilation options
# these are environment variables that are passed to 
# the configure build and install stage`
HOST=i586-pc-linux-gnu
O=2
CPU=i686
ARCH=i486
MMX=Y

# Languages to support. If a language is disabled
# and a package requires it then it will be impossible
# to install the package. Notably C++ is used for 
# a large selection of packages.
CXX=Y
#JAVA=Y
#FORTRAN=Y
#OBJC=Y

# Package features
# Support for other languages
#NLS

BUILD Stage

The package description is equipped with additional information via B records and filling out of missing MD5 values. It looks something like this after the build stage:

W The Linux C library GLIBC
T lib ldconfig
T usr/lib ldconfig
T usr/local/lib ldconfig
H http://www.gnu.org/software/libc/libc.html
M christoph@lameter.com
S ftp://sources.redhat.com/pub/glibc/releases/{PV}.tar.bz2 5be613d02b934d8e305dd2f93062fa6c
P ftp://sources.redhat.com/pub/glibc/releases/{P}-linuxthreads-{V}.tar.bz2 33b9ae01d51263867d338adfba105278
P {P}-manpages-{V}.tar.bz2@man b712a49b5113fccb4c8b0ada2a30d390
P {P}-2.2.4-string2.h.diff 135f8145885a2f4f9876fe973f33ddf6
P {PV}-threadsig.diff 993732f56fdecf36f672198112fc5d5c
P {P}-2.2.2-test-lfs-timeout.patch@io 52cfc7627fc62dfb26d8d163aac361f6
P {PV}-dns-network-overflow.diff b12e1a7de85cd82bcb341863643c9ffd
P {PV}-sunrpc-overflow.diff d28a4b6eb36a891a259806430a612de5
P {P}-divdi3.diff a3587cb4338001848f4383529d244461
P {PV}-gcc311.patch 7481046043f42ecf4ac0a218abc896fc
R BUILD linux-headers
R BUILD gcc
R BUILD make
R BUILD binutils
R BUILD texinfo
R BUILD gawk
R BUILD sed
R BUILD {NLS}
R DEPEND base
G R BUILD gcc
G R DEPEND glibc
V PATH LDPATH /lib:/usr/lib:/usr/local/lib
V ENV LANGUAGE C
V ENV LC_ALL C
V ENV TZ UTC
A O 2
A DESTSPEC install_root={B} {NLS?"localedata/install-locales"}
A CONFOPT --with-gd=no --without-cvs --enable-add-ons=linuxthreads --disable-profile 
A DOC BUGS CONFORMANCE COPYING* ChangeLog* FAQ INSTALL INTERFACE NAMESPACE NEWS NOTES PROJECTS README* 
A MAN man/*.3thr
B PACKAGE glibc
B VERSION 2.2.5
B UPM_VERSION 0.9
B PV glibc-2.2.5
B BUILD_linux-headers 2.4.19
B BUILD_gcc 3.2
B BUILD_make 3.79.1
B BUILD_binutils 2.13.90.0.4
B BUILD_texinfo 4.2
B BUILD_gawk 3.1.1
B BUILD_sed 3.58
B CPU i686
B O 2
B ARCH i486
B CFLAGS -mcpu=i686 -O2 -march=i486  
B GODIR
B CONFOPT --with-gd=no --without-cvs --enable-add-ons=linuxthreads --disable-profile 
B HOST i486-pc-linux-gnu
B CONFIGURE ./configure --prefix=/usr --with-gd=no --without-cvs --enable-add-ons=linuxthreads --disable-profile  --host=i486-pc-linux-gnu
B BUILD make
B CHECK
B DESTSPEC install_root=/var/tmp/upm-binary/glibc-2.2.5 
B INSTALL make install_root=/var/tmp/upm-binary/glibc-2.2.5  install
B MAN man/*.3thr
B MAN_HANDLER /usr/upm/scripts/man_handler /var/tmp/upm-binary/glibc-2.2.5 man/*.3thr
B DOC BUGS CONFORMANCE COPYING* ChangeLog* FAQ INSTALL INTERFACE NAMESPACE NEWS NOTES PROJECTS README* 
B DOC_HANDLER /usr/upm/scripts/doc_handler /var/tmp/upm-binary/glibc-2.2.5 glibc BUGS CONFORMANCE COPYING* ChangeLog* FAQ INSTALL INTERFACE NAMESPACE NEWS NOTES PROJECTS README* 
B INFO_HANDLER
B FINAL_HANDLER /usr/upm/scripts/final
E POSTINST install -m644 {S}/nscd/nscd.conf etc
E POSTINST rm -rf etc/ld.so.cache etc/localtime usr/bin/lddlibc4 usr/libexec

The installation adds more information about files. Just a few excertps:

B DESTSPEC install_root=/var/tmp/upm-binary/glibc-2.2.5
B INSTALL make install_root=/var/tmp/upm-binary/glibc-2.2.5 install
D usr/lib/gconv 755 0 0
D usr/include/sys 755 0 0
D usr/include/bits 755 0 0

...

L usr/lib/libnss_compat.so ../../lib/libnss_compat.so.2
L usr/lib/libutil.so ../../lib/libutil.so.1
F etc/rpc 644 0 0 829918f13dcb14fdeab47541ff658cfa 1615
F etc/ld.so.cache 644 0 0 18aa7b9db00e80a90c825bdaa9473e5e 5630
F lib/libc-2.2.5.so 755 0 0 046d279de1888ee35f81bc306b6927cc 1355938

...

F usr/info/libc.info-9 644 0 0 e5b2e9de70017dd31c98532e3671f4a8 49111
F usr/info/dir 644 0 0 feae9e11b3fd7bf07613d124b56de336 64950
F usr/libexec/pt_chown 4755 0 0 fe81d227c68162eb57cbf2b129a76065 8620
F usr/upm/in/glibc-2.2.5 644 0 0 ddc3b478ccbdea7184af9e84e1aadc31 1832

Release names for UPM distribution

Package Description File Format

Reference of all Record Types Supported

R	Stages	Syntax	Purpose
A	RBI	A variable [value]	Set attributes to parameterize builds
B	BI	B variable [value]	Record values of attributes during builds.
C	B	C arch location	Shows location where binary tarballs of the package are available
D	I	D directoryname uid gid permissions	Information about a directory
E	RB	E event shell-code	Modifications on certain types of events
F	I	F filename uid gid permisson md5sum length	Information about installed file
G	R	G upm-command	Allows the insertion of commands into package descriptions depending on this one
H	RBI	H urls	URLs to find out more information about the package
I	I	I variable [value]	Record important values during installation.
L	I	L filename link-target	Information about an installed symlink
M	RBI	M email-addresses	Email addresses to contact regarding package maintenance
N	I	N filename uid gid permission devinfo	Information about special node installed
O	RBI	O option	Sets special processing options for a package
P	R	P url/filename[@patchfix]i [md5sum]	Apply patch during build process
Q	RBI	Q actionletter init.d-script update-rc.d-opts	Set up startup for scripts
R	RBI	R relationship package [condition]	Define a relationship to another package
S	RBI	S url/filename [md5sum]	Where to get the sourcecode from
T	RBI	T directory action	Actions to execute on modifications of files in directories
U	RBI	U username uid gid [Name]	Specify users necessary for a package
V	RBI	V type envvari [value]	Manager environment and paths
W	RBI	W text	Description of the package
X	RBI	X blacklist-type file/dir	Define Blacklist
Y	RBI	Y symlink file priority	Realize prioritylinks
Z	I	I file package	Information about a file that used to belong to a package

The Stages mentioned refer to R=Recipe, B=Build, I=Installed. If a letter is provide then the record type can be present in that stage. Filenames are always specified without a leading / since the package can be build in any directory and a relative filenames makes it easier to work with.

Package description files are used to store metadata about a package. This includes information on how to retrieve/build the package and what files are used by the package. Package description go through various phases until they are ready to be used:

Recipe

In this stage the package description is all there is. It is an instruction as to where to get the files and how to build a binary from the source code. Ideally the instructions are platform independent as well as version independent. The version number is derived from the recipe name so that the recipe can be simply renamed for newer releases. If no changes to the build process have been done then the recipe can simply be used for a newer release. Attributes are used to specify various flexible characteristics of the build environment. Recipies usually exist in /var/upm/recipe and its subdirectories. A single files contains all the build instructions. Examples can be found here.

Build

In the build stage B records have been added and the variable nature of the attributes has become fixed by commitments to a certain style of build. The binary package is bound to a CPU architecture and the features are fixed now. A binary package might be stored as a .tgz tarball somewhere or just be a temporary stage in a directory generated by uPM. The metadata of BUILD package can be stored separately to allow the package manager to search for fitting binaries. See examples here The metadata is usually stored on system in /var/upm/build. The binary tarballs are stord in /var/upm/binary/ if the upm is configured to save binary tarballs.

Installed

In the installed stage the package description contains records of files, directories and symlinks installed and exact records on the metainformation of all packages so that the current state of the package can be verified. The package descriptions of installed packages are to be found in /usr/upm/installed.

Package description files are simple lines of text with fields separated by blanks. The first parameter is a letter indicating the type of record. The last parameter might be a string containing blanks. No quoting is necessary.

String Expansion

Some records can contain references to variables implemented by String Expansion which might refer to the attribute definition in /etc/upm.conf and the package description.. String expansion is done when content is contained in { }. The following expansions are supported:

exp ? e1 : e2

Conditional. If exp is true/defined then e1 is used as a value otherwise e2

e1 & e2

Boolean logic & and | are supported.

e2 = e2

Comparisions. All typical forms of comparision in C are supported.

e1 + e2

String concatenation.

VARIABLE

Reference to a variable. Looked up in B records A records and attribute section of upm.conf

"string"

String literal

123.xx.44

String literal

( e )

Subexpression

! e

NOT expression

1. B The target directory where the install should place the binaries
2. S The source directory where the build process takes place
3. V The version number of the package (Shorthand for VERSION)
4. P The name of the package (Shorthand for PACKAGE)
5. UPM_VERSION The version of UPM runnig

Filesystem Entities Definition records

These lines are used to track the files owned by a package. They only exist during the installed phase of the package. Upon installation these lines are derived from the contents of the source tree.

Directories

D <FN> <PERM> <OWNER> <GROUP>

Files

F <FN> <PERM> <OWNER> <GROUP> <MD5SUM> <SIZE>

Links

L <FN> <TARGET>

Nodes (special devices, pipes and such)

N <FN> <PERM> <OWNER> <GROUP> <DEV>

Entities commandered by other packages

Z <FN> <Package>

Z records will be generated when another package has replaced files of the package. They are a memory of file that have been there and a record of who the villian was.

Header Records

Description of the Package

W <descriptive text>

This contains the text displayed when an overview of packages is requested.

Options

O <option>

Is used to set up special behavior of the packaging system. Supported are

OVERWRITE

Allows other packages to overwrite files in this package. Files will vanish as other files provide them. This is used for busybox to gradually relinguish its commands to the "real" thing. A package with the OVERWRITE package will not install files that are already provided by other packages.

BASE

A base package owns all directories it includes. No Husk will be deployed during the installation phase.

REALROOT

Package cannot be build unless the building process is run as root. This means that the package cannot be build by a regular user using fakeroot.

INSANE

The package does strange things like putting files into the / directory. Switches off all sanity checks and lets the package overwrite files of other packages at will.

NOBINARY

A binary package is not generated or stored for this package. Setting NOBINARY is useful for packages that have a fast build or for packages with License restrictions. If a package has a license that prohibits redistribution of binaries then this option can make sure that redistribution cannot happen.

Source specification

S <url> <md5sum>

Retrieve and unpack source code from the indicated location. Contents retrieved are checked against the (optional) md5sum. All the classic tar compressed formats are supported. If multiple lines of source specifications are given then these are understood to be alternate sources for the same information. The S records will be tried until one is successful. If a simple file is specified then the file will be fetches from the configured source location in upm.conf. By default this is http://ftp.ibiblio.org/pub/linux/distributions/telemetry/sputnik/sources. If the first character of the URL is a @ then the mirror list in upm.conf will be consulted and all mirror sites will be tried to obtain the sourcecode.

Patch Specifications

P <patch-url> <md5sum>

The patch specifications works in the same way as the Source specification. No multiple sources are supported though. Patches will be applied in the sequence as the P records are encountered in the package description. Patch-Fitting allows hinting to UPM how the patch has to be applied. Without the fitting specification UPM will try to do patch-magic by trying various way of applying the patch until it fits. Patch fitting is done by appending @ to the end of the patch. If a number follows then that is the -p parmeter value usually given to patch. If some text follows then that is the directory to CD to before doing a patch -p0. Tarballs can also be specified. They will simply be unpackaged into the main source directory or into the location specified by the @ hint.

Binary specification

C <arch< <binary-url> <md5sum>

C lines specifiy where binaries with default configuration are available. The arch specifies the basic architecture. Binaries are not as flexible as recipes. The binary will not install if there is a basic incompatibility of attribute settings.

Homepage specification

H <homepage-url>

Relationship specification

R <relationship> <package> <condition>

How does a package relate to other packages. Relationships allowed are DEPEND, CONFLICT, REPLACES, PROVIDE and BUILD. In the case of PROVIDE the condition is the version of the virtual package and not a condition. REPLACE means that the package referred to is completely removed on installation of this package. A BUILD reference causes version information from the depended on package (plus all the G records of that package) to be imported. G records can be used to generate necessary dependencies for packages using libraries or to generate types of build environments.

Event specifications

E <Event> line of shell command

Shell commands are run at specific events. All shell commands undergo string expansion first. It is possible to have these shell commands customized by the attributes of the package or of the system. It is strongly suggested NOT to use these event scripts but rather to rely on the attributes controlling the build engine, triggers, Blacklists and system wide scriptsi as much as possible. Package descriptions containing *MERGE events might be considered unsafe. All other events can be executed in a fakeroot or user environment during package build. Do not write maintainer scripts that are package specific If you need some functionality then please generalize it and put it into some kind of utility package. The event handlers can then invoke those scripts. Maintainer scripts tend to rot since no one bothers with them. Tools that can be useful for multiple packages will usually be maintained well.

Possible events are (shorthand notations in parentheses):

POSTINST (>I)

Execute after the package has been installed into the temporary binary tree (NOT into the / tree). Executed as the root user in a fakeroot environment. The current directory is the binary directory. This is typically used to clean up the binary tree, move files around the target tree and provide other finishing touches. Note that the base husk (directory and links from the base package) is removed automatically before the POSTINST is run.

PREINST (<I)

Execute before the package files are moved into the temporary binary tree (NOT into the / tree). Executed as the root user in a fake root environment. Current directory is the binary directory. This is typically used to setup directories that the installation stage expects. Note that the husk (directories and symlinks) of the base package is always provided in temporary binary trees. It is only necessary to create directories that are unique to the package.

PERM (<P>)

Run as fake root after all permissions and ownerships have been normalized and after all other scripts have been run on the binary tree. The native permissions are typically overwritten when the final script is run. Allows customization of permissions when needed.

PRECONFIGURE (<C)

Run as build user. Directory is sourcedirectory. Before configure.

POSTCONFIGURE (>C)

Run as build user. Directory is sourcedirectory. After configure.

PREBUILD (<B)

Run as build user. Directory is sourcedirectory. Before build.

POSTBUILD (>B)

Run as build user. Directory is sourcedirectory. After build.

PREMERGE (<M)

Executed before the package files are moved into the / tree. {B} is the directory of the binary tree to be merged in. This can be used to simulate Debian preinst scripts. Simply invoke a script from the {B} tree... maybe {B}/var/lib/dpkg/info/mypackage.preinst ?? I would recommend not to use such a script since this script has to run as real root.

POSTMERGE (>M)

Execute after all the package files have been moved into the / tree as real root. Current directory is the / directory.

PRERM (<R)

Run as root before the package is removed.

POSTRM (>R)

Run as root after the package is removed.

>file text

Write the text to the file specified

>>file text

Append the text to the file specified

@file sedcommand

Execute a sed command on the file. like cp file file.new;sed -e 'sedcommand' file.new >file;rm file.new

command add list of files

command remove list of files

T lib ldconfig #
T usr/lib ldconfig #

# UPM Configuration File
# /etc/upm.conf
#
# Christoph Lameter, , August 21, 2002
#

# Mirror to use to retrieve new package information
mirror=http://tbox.lameter.com/mirror/

# RSYNC server to synchronize availablity lists
sync=tbox.lameter.com::tbox/sputnik

# Default patch and source location (used if mirror fails)
#patch_loc=http://tbox.lameter.com/sputnik/patches
#source_loc=http://tbox.lameter.com/sputnik/sources

# For some packages sources and binaries are available.
# Binaries are usually only build for important big packages
# The following settings indicate to UPM if binaries should
# be used if available
# binaries=always|never|ifavail
binaries=ifavail

# The binaries generated during the build phase can be automatically
# saved if a path is specified here.
save_binaries=/var/upm/binary

# Logging control
logfile=/var/log/upm.log
loglevel=9
viewlevel=7
# File retrieval
retrieve_command=wget -nc --passive-ftp -c %s

[mirror]
# List of mirrors to be used using S @/path P @/path
# Resorting this so that the nearest mirror comes first improves fetch performance.

tbox=http://tbox.lameter.com http://telemetrybox.org/tbox
debian=ftp://ftp.debian.org/debian
kernel=ftp://mirrors.kernel.org/pub/linux
sourceforge=http://unc.dl.sourceforge.net/sourceforge http://belnet.dl.sourceforge.net/sourceforge ftp://telia.dl.sourceforge.net/pub/sourceforge
gentoo=http://www.ibiblio.org/pub/linux/distributions/gentoo/distfiles
gnome=ftp://ftp.gnome.org/pub/gnome ftp://archive.progeny.com/GNOME ftp://ftp.sunet.se/pub/X11/GNOME ftp://ftp.no.gnome.org/pub/GNOME ftp://ftp.gnome.org/pub/gnome/2.0.0
kde=http://ibiblio.org/pub/packages/desktops/kde/ ftp://ftp.kde.org/pub/kde/ ftp://download.us.kde.org/pub/kde/ ftp://ftp.gtlib.cc.gatech.edu/pub/kde ftp://download.uk.kde.org/pub/kde/ ftp://download.au.kde.org/pub/kde/ ftp://download.at.kde.org/pub/kde/ ftp://download.tw.kde.org/pub/kde/
gnu=http://www.ibiblio.org/pub/gnu ftp://mirrors.kernel.org/gnu ftp://aeneas.mit.edu/pub/gnu

[env]
#
# Environment settings for the build process.
# These are expanded using values from the attribute
# sections of the package and those defined here.

# Note that these are NOT attributes to be tracked
# or to be used in regular attribute expansion
# If you want to do that then define such attribute
# in the attributes section
CFLAGS={CFLAGS}
CXXFLAGS={CFLAGS}
CC=gcc

[attributes]
# These are lists of default attributes to be used for
# building programs. 
# Attributes defined here can be override by attribute
# specifications in the package or by package specific
# overrides later....

# The following attributes are always predefined by upm
# P The packagename (same as PACKAGE)
# V The version number (same as VERSION)
# S Source directory (Only available during source building and install )
# B Binary directory (Only available during installation phases )
# UPM_VERSION Version of UPM processing

# Specification what to do in each of the build stages
# The shell expansions here are deferred until the time
# just before the corresponding build stage is run

# These are preset for autoconf sources.

# CONFIGURATION STAGE
# The configuration stage. If this package does not
# support autoconf override this setting using
# A CONFIGURE
# CONFOPT can be set with additional options
CONFIGURE={GODIR}./configure --prefix=/usr {CONFOPT} --host={HOST}

# BUILD STAGE
# This works with most packages
# Override BUILD if additional targets are needed
BUILD={GODIR}make

# VERIFICATION STAGE
# The check stage is typically unused. Define this to override and actually
# perform a regression test or so.
CHECK=

# INSTALLATION STAGE
# Installation. Typically the specification of the destination
# varies. Define DESTSPEC attribute to fill that one out if the
# prefix default will not work.
INSTALL={GODIR}make {DESTSPEC} install

# POST Installation management
# Installation of manpage via the A MAN attribute
MAN_HANDLER={MAN?"/usr/upm/scripts/man_handler {B} {MAN}"}
# Installation of documentation via the DOC attribute
DOC_HANDLER={DOC?"/usr/upm/scripts/doc_handler {B} {P} {DOC}"}
# Installation of documentation via the INFO attribute
INFO_HANDLER={INFO?"/usr/upm/scripts/info_handler {B} {INFO}"}
# Final handler run unconditionally
FINAL_HANDLER=/usr/upm/scripts/final

# Override this to specify your way of setting the binary
# destination to install into in "make install"
DESTSPEC=prefix={B}/usr

#ALTDIR can be specified if packages have their configure elsewhere.

GODIR={ALTDIR?"cd {ALTDIR};"}

CFLAGS=-mcpu={CPU} -O{O} -march={ARCH} {MMX?"-mmmx"} {NOEXCEPTIONS?"-fno-exceptions"} {OMITFRAMEPOINTER?"-fomit-frame-pointer"}

# Shorthand often needed
PV={P}-{V}

# Most important is the control of the gcc compilation options
HOST={ARCH}-pc-linux-gnu
O=2
CPU=i686
ARCH=i486
#MMX=Y
#NOEXCEPTIONS=Y
#OMITFRAMEPOINTER=Y

# Languages to support. If a language is disabled
# and a package requires it then it will be impossible
# to install the package. Notably C++ is used for 
# a large selection of packages and should
# not be disabled unless you want 
# a very minimal system.

CXX=Y
#JAVA=Y
#FORTRAN=Y
#OBJC=Y

# Package features
# Support for other languages
# Package Feature Attributes should contain the package
# they require or a virtual package name.
#NLS=gettext

BERKDB=db
GDBM=gdbm
READLINE=readline
GPM=gpm
#GNOME=gnome-libs
GTK=gtk+1.2
X=xfree
#PERL=perl
#PYTHON=python
RUBY=ruby
SSL=openssl
#TCPD=tcpd
PAM=pam
#IPV6=xx
JPEG=jpeg
PNG=libpng
TIFF=tiff
ORGVORBIS=libvorbis

# SImple Flags
# Debugging support
#DEBUG=y
FBCON=y

# Package specific overrides
[rocksndiamonds]
# I want this package to be compiled with maximum optimization
O=6