Installation CD. ISO image for booting
a uOS system. This is a LiveCD. The complete system is available by booting from CD without installation.
Auto detects hardware and network. Boots Linux in graphics mode and comes up with a web browser. DHCP used to discover network
configuration. MP3 player working. Installation routine
available but not tested yet. Two videos included but the video player is not working yet.
The image can also be used to bootstrap a system via a chrooted environment (if you already have a Linux distro
installed). Loop mount the ISO image and then copy the files out of the image. BEWARE: The files on the CD
are compressed using ZISOFS. You either need to have ZISOFS support compiled into the kernel or use the mkzftree
tool (zisofs-tools) to decompress the file.
mount -oloop /path/to/image/sputnik-0.25.iso /path/to/tree
mkzftree -u /path/to/tree /my/chroot
umount /path/to/image/sputnik-0.25.iso
mount -t proc proc /my/chroot/proc
mount -t devfs dev /my/chroot/dev
chroot /my/chroot
now play arond with the upm environment
umount /my/chroot/dev
umount /my/chroot/proc
UPM Sources
Sources can now be found here.
I am going to stop mentioning individual releases here now since I was always too late in updating the list. Lets just
be content with a small change log.
- upm-0.4x
- Implementation of HOLDs, references to attributes of other packages, various bug fixes.
- upm-0.34
- Implement dependency checks on package removal. Various fixes. Clean up user/root permissions and directory
handling. Implement debugging of hash stastistics and rework hashing algorithms.
- upm-0.33
- Archive access via "upm update" was broken.
- upm-0.29
- Implement reporting and syncattr functionality. Sources moved to ibiblio.
- upm-0.25
- Path fixes. KDE builds completely. Add archive maintenance functionality.
- upm-0.24
- Lots of bug fixes. /etc/suid.conf. Beginning builds of mozilla and KDE
- upm-0.22
- Hashes for package lookup. Lots of bug fixes.
- upm-0.21
- Add virtual handling in recipes and builds. Lots of bug fixes.
- upm-0.20
- X, xchat etc work. Higher level package management realized.
A list of available package descriptions is at lameter.com/upm/devel/uos/sputnik/recipe.
Christoph Lameter, August 8, 2002.
Packaging Software
This is a short tutorial on how to start writing a package description. Package descriptions control all aspects
of binary tarball generation and the deployment of software. A package description is an ASCII files with lines called Records. A record begins with a capital letter and a blank. It is followed by parameters that are separated by blanks.
Starting a package description
In general it is a bad idea to work as the root. For the following exercise use a regular user id if at all
possible. Run your favorite editor and then type a W and a blank on the first line followed by a one line description of what this package
is all about. Then type a S and a blank on the next line followed by the URL where the source code for the software
you are trying to package can be obtained. This could look for example this way if one would try to package sed:
W Sed - stream editor
S http://queen.rett.polimi.it/~paolob/seders/ssed/sed-3.58.tar.gz
Save this file named packagename-version. So this one should be called sed-3.58. With that you have created
a fully functional package description with a W Record describing what the package is and a S Record to specify
the source location.
The package description can now be used as a parameter to uPM.
For starters lets simply build the package and not immediately install it.
That would work using:
upm build sed-3.58
upm should now do something and produce sed-3.58_i486.tgz. Read the documentation on upm to find out
what else can be done with that package description.
Making the package description version independent
One problem with the package description is that the version number is hard coded into the package description.
If 3.59 has been released then one would need to edit the package description which is bad.
It would be great to have package descriptions that are independent of version numbers.
We already have the package name and the package version number in the filename for the package. uPM can
extract the information from the filename and make that information available as {V} for VERSION and {P} for PACKAGE.
The revised version independent package description is now:
W Sed - stream editor
S http://queen.rett.polimi.it/~paolob/seders/ssed/{P}-{V}.tar.gz
Now the package description is version independent. The package description can be simply renamed to use new version.
The maintenance scripts for the uOS distribution check periodically for updates and will rename the
package description and attempt to build if new releases become available.
With this we have used a string expansion. uPM has a quite sophisticated mechanism for
managing a database of strings and string snippets that can be used to extensively customize the build process.
Why did this work?
It is kind of amazing that one was able to integrate a package by just writing two lines of a package description.
The reason that this worked so well is that:
- Sed uses the standard autoconf/configure/make mechanism for package building and uPM will default to an
autoconf build mechanism.
- We have relied on defaults for dependencies and other important aspects of building.
In the following text the package description will be made more complete. The default build mechanism can
be seen (and modified if so desired) in the settings of upm.conf.
The default action for configure is:
./configure --prefix=/usr --host={HOST}
The default build action is
make
And the default installation action is:
make prefix={B}/usr install
These settings are exactly fitting for a lot of packages that use the standard autoconf build mechanisms.
The above examples include some additional string expansions. {HOST} is
the name of the current build environment typically set in upm.conf and {B} is the temporary directory for installation.
Note that uPM never allows direct installation into the system files. Building is done in a user
environment and a special fakeroot daemon is used to simulate a root environment for permission setting
and package generation. The building process is completely secured. Nothing can happen to the system since the
build procedure never uses real root access. The build process will fail if an installation routine
tries to install files into unauthorized areas. Feel free to experiment with package descriptions. As long as
no installation is done no harm can come from it.
Setting the LICENSE attribute
It is useful to track the license that a package has. This is typically set using the LICENSE attribute.
Attributes are package characteristics that are defined in the package description and in upm.conf.
They can be used to attach arbitrary information to a package. The license for sed is GPL-2 and the attributes
are set using an A Record. The package description with the LICENSE attribute:
W Sed - stream editor
S http://queen.rett.polimi.it/~paolob/seders/ssed/{PV}.tar.gz
A LICENSE GPL-2
We have also changed the version reference from {P}-{V} to {PV}.
PV is frequently used and its defined in /etc/upm.conf as {P}-{V}.
Setting the build type
The package does not have an explicit specification of a build type yet.
This means it build without any build information. No tracking of the tools used to build sed is done.
The tracking of such tools might be useful. If a bug has been found in the compiler we would like to
be able to rebuild all affected packages. Or we might want to make sure that everything is build using
the newest tools. Build types are used to set default build environments. R Records control package
relationships. Sed is written in C and there is a c-build environment to be found under the meta section
of the recipes that sets up such an environment. Sed with setting of the build environment:
W Sed - stream editor
S http://queen.rett.polimi.it/~paolob/seders/ssed/{PV}.tar.gz
H http://www.gnu.org/software/sed/{P}.html
A LICENSE GPL-2
R BUILD c-build
The R Record takes three parameters. The first one is the type of relationship. Here is it is a build
relationship to the c-build meta package. The c-build meta package will set other build dependencies and runtime dependencies
that are used by all C applications. With the c-build dependency also the runtime dependency on GLIBC is established.
Incidentally a H Record slipped in. H records are used to contain information about the homepage of packages.
The H records are used by documentation tools to show where more information about a package can be found.
The URLs of H records are also used to realize keyword searches for packages.
Conditional Dependencies
Reading the documentation we find that sed does support NLS (national language support) and includes lots of
files that display messages in Hebrew, Greek, Spanish etc etc. But our system is only used in the United States so
we really do not want to carry that baggage around. On the other hand people in other countries might need those
languages. For that purpose a global package build directive can be set in /etc/upm.conf. if a NLS attribute
with the content of the package providing the NLS support (typically gettext) is defined then sed should
be build with NLS support. Otherwise NLS support should be skipped and all those foreign languages should not
land on our system.
Typically this is done with a conditional dependency and conditional configure string generation.
The R BUILD settings take a string parameter for the package name. If that string parameter
is empty then no dependency will be generated. Instead of specifying the package name we can simply specify
that the contents of the global NLS attribute should be used as the dependency. If NLS is not set then no dependency
will be generated. If NLS is set then a dependency on the package that provides build/runtime support for
national language support will be included. The augmented package description:
W Sed - stream editor
S http://queen.rett.polimi.it/~paolob/seders/ssed/{PV}.tar.gz
H http://www.gnu.org/software/sed/{P}.html
A LICENSE GPL-2
R BUILD c-build
R BUILD {NLS}
A CONFOPT {NLS?:"--disable-nls"}
Also included is the necessary customization for the parameters to the ./configure script. The default
definition for CONFIGURE expands CONFOPT after --prefix=/usr.
The default is that CONFOPT is empty. By assigning something to it we can change the configuration of
the package to be build. Here is another string expansion. This time it is a ternary operation like known from C.
This means that if NLS is not set then set CONFOPT to --disable-nls.
Verification of the software build
Some programs like sed here support the checking of the binary for proper functioning. By default uPM
does not generate any action for checkups but an attribute CHECK can be set to the command of the
test to generate it. The resulting package description is then:
W Sed - stream editor
S http://queen.rett.polimi.it/~paolob/seders/ssed/{PV}.tar.gz
H http://www.gnu.org/software/sed/{P}.html
A LICENSE GPL-2
R BUILD c-build
R BUILD {NLS}
A CONFOPT {NLS?:"--disable-nls"}
A CHECK make check
A DOC COPYING NEWS README* THANKS TODO AUTHORS BUGS
And we also set the DOC attribute to the files we would like to put in /usr/doc/sed from the source archive.
uPM also supports MAN for man pages and INFO for info files. These are only to be used if
the package does not install documentation on its own using the make install target.
Moving files around after the install
In some situations we might want to change what install has done. This is possible using event scripts.
One such event is the POSTINST. It is happening after the installation (into the temporary tree).
A shell command can be given that is running with the current directory set to the root of the temporary tree.
We will be using that to allow for the possibility of alternate implementations of sed using the Y records.
W Sed - stream editor
S http://queen.rett.polimi.it/~paolob/seders/ssed/{PV}.tar.gz
H http://www.gnu.org/software/sed/{P}.html
A LICENSE GPL-2
R BUILD c-build
R BUILD {NLS}
A CONFOPT {NLS?:"--disable-nls"}
A CHECK make check
A DOC COPYING NEWS README* THANKS TODO AUTHORS BUGS
E POSTINST mv usr/bin/sed bin/sed-{V}
Y bin/sed sed-{V} 50
Sed is important for us and it should even be there when /usr has not been mounted yet.
That is why we move sed to the /bin directory. Note that no leading / are used here. A leading / would cause
the script to fail since it would attempt to access the real /. Sed is also renamed to sed-3.58 to make
the name unique in case we have multiple Seds on the machine.
The Y Record is used to allow one name for multiple binaries. An Y record specifies the common name and then the
special name and a priority. uPM will pick the one with the lowest priority from the available sed. A more powerful
sed could be installed and have a lower priority. Then the /bin/sed link would point to the new sed.
If that more powerful sed is later removed (naturally it had some strange behavior) then the /bin/sed link
will be re-pointed to the old sed. This insures that a sed version is always available.
Conclusion
There are many other lines for other special effects that can be used with uPM. This was just a short
introduction covering the development of a package description. Please read the rest of this HTML file for
more details. Study the recipes on the uOS site for
examples.
UPM Sample Package Description
GLIBC Package Description
Here is an example of a more extensive package description for GLIBC. This is also used to illustrate
the various phases of supplementing information in the package description by the uPM system.
The first listing is of the RECIPE state. This is the pre-build stage. The Recipe can contain
MD5Sums for all sources and patches. Those have not been included.
W The GNU C library GLIBC
H http://www.gnu.org/software/libc/libc.html
M christoph@lameter.com
S ftp://sources.redhat.com/pub/glibc/releases/{PV}.tar.bz2
P ftp://sources.redhat.com/pub/glibc/releases/{P}-linuxthreads-{V}.tar.bz2
P {P}-manpages-2.2.5.tar.bz2@man
P {P}-2.2.4-string2.h.diff
P {P}-2.2.2-test-lfs-timeout.patch@io
P {P}-2.2.5-dns-network-overflow.diff
P {P}-2.2.5-sunrpc-overflow.diff
# glibc mucks around with the library path breaking fakeroot
O REALROOT
R BUILD linux-headers
R BUILD gcc
R BUILD make
R BUILD binutils
R BUILD texinfo
R BUILD gawk
R BUILD sed
R BUILD {NLS}
R DEPEND base
G R BUILD gcc
G R DEPEND glibc Vi>=2.3
# O2 seems to be good for fragile glibc....
A O 2
# New glibc requires a framepointer for errno handling.
A OMITFRAMEPOINTER
A DESTSPEC install_root={B} {NLS?"localedata/install-locales"}
A CONFOPT --with-gd=no --without-cvs --enable-add-ons=linuxthreads --disable-profile
#A CHECK make check
E POSTINST install -m644 {S}/nscd/nscd.conf etc
E POSTINST rm -rf etc/ld.so.cache etc/localtime etc/rpc usr/libexec
V PATH LDPATH /lib:/usr/lib:/usr/local/lib
T lib ldconfig #
T usr/lib ldconfig #
T usr/local/lib ldconfig #
A DOC BUGS CONFORMANCE COPYING* ChangeLog* FAQ INSTALL INTERFACE NAMESPACE NEWS NOTES PROJECTS README*
A MAN man/*.3thr
V ENV LANGUAGE C
V ENV LC_ALL C
V ENV TZ UTC
X usr/share/locale MINE
# Loader must be executable
E PERM chmod a+x lib/ld-*.so
The first line is a W Record. The line sets the Title of the package as displayed
using various tools.
The second line is a H Record which is used to give a hint where more information about
the package can be found.
Third line is a M mail address where bug reports and patches might be submitted to.
Fourth line is the Source Specification. This is the location of the source code of
the package.
The next lines are all Patch Specifications. These are patches to be applied to the
source package. Patches can be either tarballs which are untarred into the source code directory
or real patch files. The first two patches here are tarballs. The second one has a patch fix.
This is the @ and the man name. This tarball should not be untarred into the main source code archive
but into a subdirectory thereof.
The patch lines also use Macro-Expansion features of uPM. Any text surrounded by
{ } are variable substitutions and expansions to be done by uPM. {P} {V} are short hand references
to the package name and the version.
Use of these macros allows the reuse of the same package description for future releases. {P} and {V} are
extracted from the filename. Simply renaming the package description often allows the installation of newer releases.
The GLIBC patches are a mess though. Usually this list is much leaner and easier to maintain.
{PV} is a default attribute defined in upm.conf.
Note also the @io at the test-lfs-timeout patch. That patch has to be applied to a subdirectory
indicated by @.
The R Records are used to specify relationships to other package.
First there are two BUILD time dependency. A build time dependency makes sure that
this package is installed at build time and it tracks the versions of the dependent package
in the GLIBC package. A B record will be generated indicating the version used to build
GLIBC.
Then some runtime dependencies follow. When the package is installed a verification is done that
these packages are installed. The last depend line is a conditional dependency.
It checks if the NLS attribute is defined. If so then the gettext package is also needed
at runtime.
Then some strange G R records appear. These
generate dependencies for packages having a build dependency on GLIBC. If a package uses
GLIBC for building then it also needs GLIBC for a runtime dependency (G R DEPEND). If a package
used GLIBC for building then it will also need GCC during build time (G R BUILD).
The A Records are used to override attribute settings in upm.conf. They only make sense
when viewed in the context of the attributes define there which control the overall build
procedure. Attribute records in package descriptions are used to override elements of
the build process. Here the O attribute is set to 2 which forces the optimization during
build to -O2. The DESTSPEC sets the way how the install target is specified during "make ... install".
There is a CHECK of the installation possible. The CHECK definition sets up the build engine
to run a command and verify that there is no error before building the binary package.
The CONFOPT settings are the options set when invoking ./configure. The example here
shows a conditional generation of these options depending on global settings in upm.conf about
how the software on a machine should be configured.
The Events for POSTINST perform changes in the binary tree after the INSTALL action has been
run. Here another file is extracted from the source archive into the binary tree and then some of
the directories are removed.
Triggers are set for common library directories. If other packages deploy new libraries the
ldconfig script will be run without these packages needing any special scripts.
The MAN and DOC attributes result in appropriate actions to deposit and manage both
documentation and man pages. The V records cause the setting of environmental variables or
search paths. Here a search path is set for the typical locations of libraries.
Here is the attribute section of upm.conf (no this is not a current version but only a subset of an old one!)
to make some more sense out of these macro expansions:
[attributes]
# These are lists of default attributes to be used for
# building programs.
# Attributes defined here can be override by attribute
# specifications in the package or by package specific
# overrides later....
# CONFIGURATION STAGE
# The configuration stage. If this package does not
# support autoconf override this setting using
# A CONFIGURE
# CONFOPT can be set with additional options
CONFIGURE=./configure --prefix=/usr {CONFOPT} --host={HOST}
# BUILD STAGE
# This works with most packages
# Override BUILD if additional targets are needed
BUILD=make
# VERIFICATION STAGE
# The check stage is typically unused. Define this to override and actually
# perform a regression test or so.
CHECK=
# INSTALLATION STAGE
# Installation. Typically the specification of the destination
# varies. Define DESTSPEC attribute to fill that one out if the
# DESTDIR default will not work.
INSTALL=make {DESTSPEC} install
# Override this to specify your way of setting the binary
# destination to install into.
DESTSPEC=DESTDIR={B}
CFLAGS=-mcpu={CPU} -O{O} -march={ARCH} {MMX?"-mmmx"}
# Shorthand often needed
PV={P}-{V}
# Most important is the control of the gcc compilation options
# these are environment variables that are passed to
# the configure build and install stage`
HOST=i586-pc-linux-gnu
O=2
CPU=i686
ARCH=i486
MMX=Y
# Languages to support. If a language is disabled
# and a package requires it then it will be impossible
# to install the package. Notably C++ is used for
# a large selection of packages.
CXX=Y
#JAVA=Y
#FORTRAN=Y
#OBJC=Y
# Package features
# Support for other languages
#NLS
BUILD Stage
The package description is equipped with additional information via B records
and filling out of missing MD5 values. It looks something like this
after the build stage:
W The GNU C library GLIBC
T lib ldconfig
T usr/lib ldconfig
T usr/local/lib ldconfig
H http://www.gnu.org/software/libc/libc.html
M christoph@lameter.com
S ftp://sources.redhat.com/pub/glibc/releases/{PV}.tar.bz2 5be613d02b934d8e305dd2f93062fa6c
P ftp://sources.redhat.com/pub/glibc/releases/{P}-linuxthreads-{V}.tar.bz2 33b9ae01d51263867d338adfba105278
P {P}-manpages-{V}.tar.bz2@man b712a49b5113fccb4c8b0ada2a30d390
P {P}-2.2.4-string2.h.diff 135f8145885a2f4f9876fe973f33ddf6
P {PV}-threadsig.diff 993732f56fdecf36f672198112fc5d5c
P {P}-2.2.2-test-lfs-timeout.patch@io 52cfc7627fc62dfb26d8d163aac361f6
P {PV}-dns-network-overflow.diff b12e1a7de85cd82bcb341863643c9ffd
P {PV}-sunrpc-overflow.diff d28a4b6eb36a891a259806430a612de5
P {P}-divdi3.diff a3587cb4338001848f4383529d244461
P {PV}-gcc311.patch 7481046043f42ecf4ac0a218abc896fc
R BUILD linux-headers
R BUILD gcc
R BUILD make
R BUILD binutils
R BUILD texinfo
R BUILD gawk
R BUILD sed
R BUILD {NLS}
R DEPEND base
G R BUILD gcc
G R DEPEND glibc
V PATH LDPATH /lib:/usr/lib:/usr/local/lib
V ENV LANGUAGE C
V ENV LC_ALL C
V ENV TZ UTC
A O 2
A DESTSPEC install_root={B} {NLS?"localedata/install-locales"}
A CONFOPT --with-gd=no --without-cvs --enable-add-ons=linuxthreads --disable-profile
A DOC BUGS CONFORMANCE COPYING* ChangeLog* FAQ INSTALL INTERFACE NAMESPACE NEWS NOTES PROJECTS README*
A MAN man/*.3thr
B PACKAGE glibc
B VERSION 2.2.5
B UPM_VERSION 0.9
B PV glibc-2.2.5
B BUILD_linux-headers 2.4.19
B BUILD_gcc 3.2
B BUILD_make 3.79.1
B BUILD_binutils 2.13.90.0.4
B BUILD_texinfo 4.2
B BUILD_gawk 3.1.1
B BUILD_sed 3.58
B CPU i686
B O 2
B ARCH i486
B CFLAGS -mcpu=i686 -O2 -march=i486
B GODIR
B CONFOPT --with-gd=no --without-cvs --enable-add-ons=linuxthreads --disable-profile
B HOST i486-pc-linux-gnu
B CONFIGURE ./configure --prefix=/usr --with-gd=no --without-cvs --enable-add-ons=linuxthreads --disable-profile --host=i486-pc-linux-gnu
B BUILD make
B CHECK
B DESTSPEC install_root=/var/tmp/upm-binary/glibc-2.2.5
B INSTALL make install_root=/var/tmp/upm-binary/glibc-2.2.5 install
B MAN man/*.3thr
B MAN_HANDLER /usr/upm/scripts/man_handler /var/tmp/upm-binary/glibc-2.2.5 man/*.3thr
B DOC BUGS CONFORMANCE COPYING* ChangeLog* FAQ INSTALL INTERFACE NAMESPACE NEWS NOTES PROJECTS README*
B DOC_HANDLER /usr/upm/scripts/doc_handler /var/tmp/upm-binary/glibc-2.2.5 glibc BUGS CONFORMANCE COPYING* ChangeLog* FAQ INSTALL INTERFACE NAMESPACE NEWS NOTES PROJECTS README*
B INFO_HANDLER
B FINAL_HANDLER /usr/upm/scripts/final
E POSTINST install -m644 {S}/nscd/nscd.conf etc
E POSTINST rm -rf etc/ld.so.cache etc/localtime usr/bin/lddlibc4 usr/libexec
Note the addition of B records to track all the attribute
information referenced.
The installation adds more information about files. Just a few excertps:
B DESTSPEC install_root=/var/tmp/upm-binary/glibc-2.2.5
B INSTALL make install_root=/var/tmp/upm-binary/glibc-2.2.5 install
D usr/lib/gconv 755 0 0
D usr/include/sys 755 0 0
D usr/include/bits 755 0 0
...
L usr/lib/libnss_compat.so ../../lib/libnss_compat.so.2
L usr/lib/libutil.so ../../lib/libutil.so.1
F etc/rpc 644 0 0 829918f13dcb14fdeab47541ff658cfa 1615
F etc/ld.so.cache 644 0 0 18aa7b9db00e80a90c825bdaa9473e5e 5630
F lib/libc-2.2.5.so 755 0 0 046d279de1888ee35f81bc306b6927cc 1355938
...
F usr/info/libc.info-9 644 0 0 e5b2e9de70017dd31c98532e3671f4a8 49111
F usr/info/dir 644 0 0 feae9e11b3fd7bf07613d124b56de336 64950
F usr/libexec/pt_chown 4755 0 0 fe81d227c68162eb57cbf2b129a76065 8620
F usr/upm/in/glibc-2.2.5 644 0 0 ddc3b478ccbdea7184af9e84e1aadc31 1832
Well enough for now...
Release names for UPM distribution
| Name | Description | Date of release |
|---|
| Start | Beginning of development | August 7, 2002 |
| Sputnik | Initial release to establish baseline functionality.
- Fully functional package manager
- Ability to bootstrap the system
- Ability to install from CD media
- Core functionality set of 300 packages
- Basic X functionality
- Hardware autodetection
- Basic archive functionality
| Oct 2002 |
| Explorer | Release to get full functionality. Goals are:
- Build core development team (10 max)
- A large set of software (>1000 packages)
- Full support for KDE and Gnome desktops and all major browsers
- Enhanced Documentation
- Fully debugged package manager
- Library of metapackages and build-types supported that standardize packaging
- Basic packaging philosophy documented
- Functionality of the Debian based TelemetryBox releases available.
- Enhanced Hardware support
- Ability to import .deb , .rpm and .ebuilds
| Estimated March 2003 |
| Vanguard | Release to be able to compete with other distributions. Goals:
Large development team (50 max)
- Comprehensive coverage with package descriptions (~ 10000 packages)
- Very good documentation and a well-maintained website
- Support for multiple arches and established porting procedures
- Installation via the Network using a floppy disk.
- Customized installs for Desktop use, Server deployment etc.
| Estimated at the end of 2003 |
Reference of all Record Types Supported
This is an overview over all record types supported in all stages. See information following the
table for more explanations. The list covers all the elements used for the specification of
recipies as well as the metainformation structures used by UPM to keep track of packages and their content.
The Stages mentioned refer to R=Recipe, B=Build, I=Installed. If a letter is provide then
the record type can be present in that stage. Filenames are always specified without a leading / since
the package can be build in any directory and a relative filenames makes it easier to work with.
Package description files are used to store meta data about a package. This includes
information on how to retrieve/build the package and what files are used
by the package. Package description go through various phases until they are ready to be used:
- Recipe
- In this stage the package description is all there is. It is an instruction as to where
to get the files and how to build a binary from the source code. Ideally the instructions
are platform independent as well as version independent. The version number is derived
from the recipe name so that the recipe can be simply renamed for newer releases. If no
changes to the build process have been done then the recipe can simply be used
for a newer release.
Attributes are used to specify various flexible characteristics of the build environment.
Recipes usually exist in /var/upm/recipe and its subdirectories. A single files contains
all the build instructions. Examples can be found here.
- Build
- In the build stage B records have been added and the variable nature of the attributes
has become fixed by commitments to a certain style of build. The binary package
is bound to a CPU architecture and the features are fixed now. A binary
package might be stored as a .tgz tarball somewhere or just be a temporary stage
in a directory generated by uPM. The meta data of BUILD package can be stored separately
to allow the package manager to search for fitting binaries. See examples here
The metadata is usually stored on system in /var/upm/build. The bi. The bi. The bi. The bi in /var/upm/binary/ if the upm is configured to save binary tarballs.
- Installed
- In the installed stage the package description contains records of files, directories
and symlinks installed and exact records on the meta information of all packages so
that the current state of the package can be verified. The package descriptions of
installed packages are to be found in /usr/upm/installed.
Package description files are simple lines of text with fields separated by blanks.
The first parameter is a letter indicating the type of record.
The last parameter might be a string containing blanks.
No quoting is necessary.
String Expansion
Some records can contain references to variables implemented by
String Expansion which might refer to the attribute definition in
/etc/upm.conf and the package description..
String expansion is done when content is contained in { }.
The following expansions are supported:
- exp ? e1 : e2
- Conditional. If exp is true/defined then e1 is used as a value otherwise e2
- exp : pattern
- Pattern matching. If pattern does not match return "" or false
- e1 & e2
- Boolean logic & and | are supported.
- e2 = e2
- Comparison's. All typical forms of comparison in C are supported.
- e1 + e2
- String concatenation.
- VARIABLE
- Reference to a variable. Looked up in B records A records and attribute section of upm.conf
- "string"
- String literal
- 123.xx.44
- String literal
- ( e )
- Subexpression
- ! e
- NOT expression
The following fixed variables are available during
the build process:
- B The target directory where the install should place the binaries
- S The source directory where the build process takes place
- V The upstream version number of the package (VERSION without -dx)
- D The delta version of the package or nothing if no delta (-dx part of VERSION).
- P The name of the package (Shorthand for PACKAGE)
- UPM_VERSION The version of UPM running
File system Entities Definition records
These lines are used to track the files owned by a package. They only exist during the installed phase
of the package.
Upon installation these lines are derived from the contents of the source tree.
- Directories
- D <FN> <PERM> <OWNER> <GROUP>
- Files
- F <FN> <PERM> <OWNER> <GROUP> <MD5SUM> <SIZE>
- Links
- L <FN> <TARGET>
- Nodes (special devices, pipes and such)
- N <FN> <PERM> <OWNER> <GROUP> <DEV>
- Entities commandeered by other packages
- Z <FN> <Package>
Z records will be generated when another package has replaced files of the
package. They are a memory of file that have been there and a record of
who the villain was.
- Description of the Package
- W <descriptive text>
This contains the text displayed when an overview of packages is requested.
- Options
- O <option>
Is used to set up special behavior of the packaging system. Supported are
- OVERWRITE
-
Allows other packages to overwrite files in this package.
Files will vanish as other files provide them. This is used for busybox to gradually
relinquish its commands to the "real" thing.
A package with the OVERWRITE package will not install files that are already provided
by other packages.
- BASE
-
A base package owns all directories it includes. No Husk will be
deployed during the installation phase.
- REALROOT
-
Package cannot be build unless the building process is run as root. This means
that the package cannot be build by a regular user using fakeroot.
- INSANE
-
The package does strange things like putting files into the / directory.
Switches off all sanity checks and lets the package overwrite files of other
packages at will.
- NOBINARY
-
A binary package is not generated or stored for this package. Setting NOBINARY is useful
for packages that have a fast build or for packages with License restrictions. If a package
has a license that prohibits redistribution of binaries then this option can make sure
that redistribution cannot happen.
- HOLD
-
The package is on hold. uPM will not automatically upgrade this package.
- NOAUTODEP
-
Do not add dependencies generated from binaries and scripts during the build process.
This insures that the manual configuration of dependencies is possible.
Source specification
S <url> <md5sum>
Retrieve and unpack source code from the indicated location.
Contents retrieved are checked against the (optional) md5sum.
All the classic tar compressed formats are supported. If multiple lines of source
specifications are given then these are understood to be alternate sources for the same information.
The S records will be tried until one is successful. If a simple file is specified then the file
will be fetches from the configured source location in upm.conf. By default this is http://lameter.com/upm/devel/uos/sputnik/sources. If the first character of the URL is a @ then the mirror list in upm.conf will be consulted and
all mirror sites will be tried to obtain the source code.
Patch Specifications
P <patch-url> <md5sum>
The patch specifications works in the same way as the Source specification. No
multiple sources are supported though. Patches will be applied in the sequence as the P records are
encountered in the package description.
Patch-Fitting allows hinting to UPM how the patch has to be applied.
Without the fitting specification UPM will try to do patch-magic by trying various way of
applying the patch until it fits. Patch fitting is done by appending @ to the end of the
patch. If a number follows then that is the -p parameter value usually given to patch.
If some text follows then that is the directory to CD to before doing a patch -p0.
Tarballs can also be specified. They will simply be unpackaged into the main source directory or
into the location specified by the @ hint.
Binary specification
C <arch< <binary-url> <md5sum>
C lines specify where binaries with default configuration are available. The arch specifies
the basic architecture. Binaries are not as flexible as recipes. The binary will not
install if there is a basic incompatibility of attribute settings.
Homepage specification
H <homepage-url>
Relationship specification
R <relationship> <package> <condition>
How does a package relate to other packages. Relationships allowed are DEPEND, CONFLICT, REPLACES, PROVIDE and BUILD.
In the case of PROVIDE the condition is the version of the virtual package and not a condition.
REPLACE means that the package referred to is completely removed on installation of this package.
A BUILD reference causes version information from the depended on package (plus all the G records
of that package) to be imported. G records can be used to generate necessary dependencies for packages using
libraries or to generate types of build environments.
Event specifications
E <Event> line of shell command
Shell commands are run at specific events. All shell commands undergo string expansion first. It is
possible to have these shell commands customized by the attributes of the package or of the system.
It is strongly suggested NOT to use these event scripts but rather to rely on the attributes controlling the
build engine, triggers, Blacklists and system wide scripts as much as possible.
Package descriptions containing *MERGE events might be considered unsafe. All other events can
be executed in a fakeroot or user environment during package build.
Do not write maintainer scripts that are package specific
If you need some functionality then please generalize it and put it into some kind of utility package.
The event handlers can then invoke those scripts. Maintainer scripts tend to rot since no one bothers with them.
Tools that can be useful for multiple packages will usually be maintained well.
Possible events are (shorthand notations in parentheses):
- POSTINST (>I)
- Execute after the package has been installed into the temporary binary tree (NOT into the / tree).
Executed as the root user in a fakeroot environment.
The current directory is the binary directory.
This is typically used to clean up the binary tree, move files around the target tree and provide other finishing touches.
Note that the base husk (directory and links from the base package) is removed
automatically before the POSTINST is run.
- PREINST (<I)
- Execute before the package files are moved into the temporary binary tree (NOT into the / tree).
Executed as the root user in a fake root environment. Current directory is the binary directory. This is
typically used to setup directories that the installation stage expects. Note that the husk (directories
and symlinks) of the base package is always provided in temporary binary trees. It is only necessary
to create directories that are unique to the package.
- PERM (<P>)
Run as fake root after all permissions and ownerships have been normalized and after all
other scripts have been run on the binary tree. The native permissions are typically overwritten
when the final script is run. Allows
customization of permissions when needed.
PRECONFIGURE (<C)Run as build user. Directory is source directory. Before configure.
POSTCONFIGURE (>C)Run as build user. Directory is source directory. After configure.
PREBUILD (<B)Run as build user. Directory is source directory. Before build.
POSTBUILD (>B)Run as build user. Directory is source directory. After build.
PREMERGE (<M)Executed before the package files are moved into the / tree.
{B} is the directory of the binary tree to be merged in. This can be used to simulate Debian
preinst scripts. Simply invoke a script from the {B} tree... maybe {B}/var/lib/dpkg/info/mypackage.preinst ??
I would recommend not to use such a script since this script has to run as real root.
POSTMERGE (>M)Execute after all the package files have been moved into the / tree as real root.
Current directory is the / directory.
PRERM (<R)Run as root before the package is removed.
POSTRM (>R)Run as root after the package is removed.
There are a couple of shorthand notations for frequently used actions. These are:
- >file text
- Write the text to the file specified
- >>file text
- Append the text to the file specified
- @file sed-command
- Execute a sed command on the file. like cp file file.new;sed -e 'sedcommand' file.new >file;rm file.new
Attribute specification
A <Attribute> <contents>
Attributes allow to specify various information that might be used by dependency conditions and to control and track
the build process. An important attribute is VERSION which is the version of the package installed.
VERSION is generated from the version provided at install time. It might contain a delta which is a suffix -d followed
by the number of the delta.
Build Attributes
B <Attribute> <contents>
Build records are generated during the building of binaries from source.
They track the values attributes had when the binary was build.
Build records are also used to track the versions of all packages mentioned in
build depends. This enables a forced rebuild of all packages that have been generated
with a broken version. If dependency checks are being done and references to attributes
of packages are made then the build attributes are the significant values..
Installation Attributes
I <Attribute> <contents>
Installation attributes track what happened during the merging of the package into
the real tree. Currently that info includes only version of packages replaced.
Environment Variables
V <type> <variable> <value>
Environment variables allow the control of paths such as MANPATH, PATH and the paths used for
ld.so as well as default settings in /etc/profile. The V records are processed in order of
priority. The packages with the most depth of dependencies are processed first and the packages
depending on them later. Therefore packages depending on others can override settings as
needed.
Types of variables:
- PATH
- Multiple specifications of the variable result in concatenation of the
strings specified by :.
- ENV
- Simply set environment variables. Multiple occurrences cause an override.
- SET
- Multiple occurrences are not allowed. Fail if multiple settings exist.
The following files are generated from the V settings:
- /usr/upm/env/environment
- Raw list of the variables and their effective settings
- /usr/upm/env/profile.env
- File with a suitable format to be sourced by bash and others.
- /usr/upm/env/csh.evn
- File with a suitable format to be sourced by csh and others.
- /usr/upom/env/ld.so.conf
- Configuration file for ld.so suitable to generate a list of all directories
to be processed by ld.so
Priority Links
Y <FN> <TARGET> <Priority>
Priority Links are used to manage multiple programs providing the same functionality.
For example there are multiple ways to provide a vi editor.
Packages can define a priority and the system will point the link to the lowest priority link.
The user can display priorities and fix these pointers to a favorite implementation.
Blacklist
X <File> ABORT/WARN/IGNORE
Blacklisting is used to prevent packages from installing files.
Typically there are files that are directories of information such as /usr/info/dir.
Many packages provide these files. uPM enforces that no packages should overwrite other packages
files and therefore those installs will fail.
Blacklisting the file will make it possible for uPM to handle the situation intelligently.
ABORT means that packages providing that packages providing that file will not install.
WARN means allow the installation but warn about the situation.
IGNORE means to simply silently ignore the garbage.
Blacklisting can also be used for Directories. Some major directories
can be taken offline. The base package does that extensively to prohibit
accidents.
Trigger
T <Dir> <command<
Execute the script when a package writes new contents to the directory or removes files from the directory.
Invocations are
command add list of files
if new files are installed and as
command remove list of files
when files are removed from the directory.
A simple example for the use of a trigger is the invocation of ldconfig:
T lib ldconfig #
T usr/lib ldconfig #
will insure that ldconfig is run whenever there is a new file in those directories. The # results
in the parameters being ignored and not passed to ldconfig.
Start Stop Service support
Q <mode> <scriptname> <update-rc.d params>
Allows the control of services provided by the package.
The script name is a referring to a script which must be provided in the
/etc/init.d directory of the generated binary tree of the package.
Upon initial installation the service is installed. Service is started after installation of the
package is complete. Upon upgrade the service is stopped. Script removed and
then reinstalled and started when the new files have been installed.
The mode specifies what exactly to do and is a combination of characters that
enable specific actions of BUE. B (Begin) means run the service upon installation
of the package. U means take service down upon upgrade and start again after
upgrade. E (End) means take service down on package removal.
User/group modifications (/etc/passwd,/etc/group)
U <username> <uid> <gid;< <long name>
U records allow the adding of new user and group ids as needed by programs.
U records will fail if the user-id already exists. If the group id does not
exist then an entry in /etc/group is also generated with the same user name.
Inheritance from BUILD dependencies
G <line to put into package depending on this one>
G records allow the setting of additional records in package descriptions that depend
on another. This allows the setup of build types and by a simple R BUILD line a package
can use generic build instructions. This is very useful if many packages have always
the same settings for building.
Frequently Asked Questions
Why was UPM not written in C++?
- Unclear Language Design
- At this point I just put a couple of days into trying to switch UPM from C to C++.
I am teaching classes in C++ so I am familiar with the concepts of C++ and I have done stuff
before in C++.
I was again surprised by the wasting-time-on-what-in-the-world-is-going-on-with-the-compiler
factor that crept in and began to consume more and more of my time
as I used more and more advanced OO concepts of C++.
C++ seems to be always trying to do something in the background
that one has to imagine to properly construct a program.
Such issues are plainly unacceptable for programming and in particular for large programs.
The model in which programming takes places must
be comprehensible and self-consistent. That is simply not the case with C++.
One might argue that all high-level languages do that. But that is not the case.
Programming in Ruby, Python, Perl or PHP is possible without having the constant
feeling that someone in the back is trying to interfere with what you are trying to accomplish.
- Object Orientedness
- C++ has some elements attempting to support polymorphism.
In the implementation of UPM I had to use that polymorphism for the first time to
the fullest extend with multiple inheritance structures.
I am used to clean and easy implementations of polymorphism from Ruby.
C++ was a rude awakening.
Seems that C++ does some tricks with pointers to classes that one needs to be aware of
in order to write correct programs.
A pointer to a pointer of a derived class that I needed to convert to a pointer to a pointer of base
class was impossible to do whereas a simple pointer conversion would work. Surprise...
I was able to override C++ by using void** instead but then why not use C?
- Constructors
- have severe limitations in C++.
I had to rewrite my APIs several times because of the inability of
constructors inherited to a derived class to call a method
in that derived class through a virtual function.
A constructor therefore cannot call a helper function to analyze say a string passed to it into
the individual properties of the object.
- Strings
- C++ requires the use of a mixture of C and C++ in order to interface with the operating system.
There is no standard API even for managing file like renaming or deleting files.
This requires frequent conversions between the C and C ++ representations of strings.
The function provided for that purpose is c_str() which provides
a C string "as long as the C++ string was not changed".
A very vague notion and rather risky in view of the evil C++ compiler
in the background doing things that you have to mind read.
- "Dynamic" allocation of derived types
- Does not seem to be implemented. An array of types cannot be build and used for allocation.
That handicap makes it impossible to manage a set of objects
with a set of derived types from a common base types.
The generation of that object has to explicitly specify the derived type.
This means that a case statement is needed
to allocate each derived type individually.
This results in the inability to easily expand the program by adding new
derived types.
Fixes have to be made all over the program if a new type is needed.
Generic frameworks are difficult to construct.
- Excessive overloading
- The standard C++ libraries do excessive overloading.
As a result the compiler cannot really give too much help if
it cannot properly "match" a function.
Additional in-depth study of the library implementation and the way overloading is being used
is necessary to fix these things.
The C++ libraries follow the C++ pattern of doing something sophisticated
that one has to fully understand in order to be able to use it.
One gets the feeling that the C++ libraries are full of booby traps for the uninitiated.
Surprise after surprise.
Count on wasting lots of time on C++ language research and the finer points
of C++s way of implementing objects.
- Library dependencies
- The use of C++ would have meant additional use of libraries. libstdc++ is mandatory.
In order to use C++ strings for file management I could have used commonc++ for example.
The STL functions might have required additional libraries too.
C allows the writing of a program by just using the standard c library of the system.
Conclusion
C++ is a hybrid of object oriented concepts and C that went awry.
Developers are wasting lots of their time to acquire language and tool
know how instead of producing code.
It is probably better to use C (more transparent) for low level tasks and
use fully object capable languages such as Ruby for high level components.
Why is package meta data stored in /usr?"
The /usr partition contains the majority of all the applications residing on the system. It is natural
to store meta data on the same medium than the data that the meta data is about. The meta data in /usr/upm
only changes when files are removed and added to the system. That is when also access to
/usr must be read-write. If /usr has been mounted readonly then the package meta data is also
protected. UPM uses a special directory /usr/upm/froot to stage a newly installed package.
Presumably the staging area is on the same volume as the final resting place of the data. This means
that UPM can use "mv" to move files into the final locations rather than a costly copy operation.
It also insures that space problems become evident during the staging phase and not during
the move of files into the final locations.
Bootstrapping a system
There is a "bootstrap" script included in the upm tarball. That script generates a directory with a system
that is self-building. This means that it only includes components necessary for the building of the core
components. The chroot command can be used to enter that build environment to build things. A ready made
build-base system is available from the Download Section. It take about a day to build a
system from scratch. Note that the build system requires a host Linux system with all the build tools installed (can be
any Linux distribution) to bootstrap.
You need a recent gcc compiler to build upm. uPM is following C99 which is only supported by gcc 3.X versions. The bootstrap
will not work on systems with gcc 2.95.x (Debian 3.0 (woody) comes with gcc 2.95.x by default!).
Most of the build tools in that build environment also should be of a more recent
vintage.
Step by Step instructions for Bootstrap
The bootstrap has been fixed in upm-0.77. Please contact me if there are
still problems.
- Download, build and install upm on the host system if it is not a uOS system.
- Run the bootstrap script to build the key components using the foreign environment.
- copy /etc/resolv.conf to the etc directory of the chroot environment.
- Enter the chroot by using
chroot bootstrapdir
- Customize the settings in /etc/upm.conf as desired.
- Rebuild all components in new environment using
upm -d syncbuild
Syncbuild will insure that all components are build with the newest software installed.
- Build out the system to be a full uOS system. Use the following command:
upm -d install system
This will compile and chug along until a complete system is setup. The system can now be made
bootable by configuring grub to load a kernel from /boot/vmlinuz. The chroot environment can be
used to debug the new distribution. Software can be added and/or removed as necessary using
the UPM commands.
Porting uOS to a New architecture
WARNING: No one has tried this yet. This is how it SHOULD THEORETICALLY work.
Basically the same steps as before have to be followed if you already have another Linux distribution
running on the hardware but there is no uOS version for that hardware.
- Download upm sources and customize the build process for the new arch as needed. Build and install upm
in the foreign environment.
- Customize /etc/upm.conf. In particular change the ARCH and the other compiler settings to fit
the new architecture.
- Run the bootstrap script to generate a chrooted UPM environment.
- Copy the custom upm.conf to the build environment.
- copy /etc/resolv.conf to the etc directory of the chroot environment
- Customize settings in /etc/upm.conf in the build environment as needed.
- Rebuild all components using the new architecture by:
upm -d syncbuild
- Follow the instructions for bootstrap. There might be issues that need to be addressed in the build
process or in package selections. Add conditionals to the package descriptions as needed using the {ARCH='xxx'? ... }
string expansions. Submit the changes to the package descriptions when you have a working system.
Files and directories used by UPM
UPN uses a variety of files in /etc, /var and /usr.
| Directory | Purpose |
|---|
| /usr/upm | Contains all the information that is only changing when packages are installed
and removed from the system |
| /usr/upm/installed | State information about all currently installed packages and files. This is read on
UPM startup and used on most operations of UPM. |
| /usr/upm/in | The state of a package before it entered the system. This is a file included
in the original tarball. It is used to check that a tarball has been generated for the UPM packaging system. |
| /usr/upm/env | The environment settings (search paths, shell environment, ldconfig paths etc) as configured
by the installed packages. Some files from /etc are linked to /usr/upm/env. |
| /usr/upm/etc | All files in /etc in their originally installed state. |
| /usr/upm/scripts | Scripts that are executed by UPM for various purposes. |
| /usr/upm/froot | Temporary staging directory used to unpack tarballs during package installation. This directory
might exist after upm was aborted. It will be erased by the next install action. The directory might be safely removed
when UPM is not running. |
| /usr/upm/staging-xxx | Temporary staging directories created when UPM realizes after unpacking of a tarball
that additional software needs to be installed first. The directories might not be erased if UPM terminated abnormally.
The directories can be safely removed if UPM is not running. |
| /etc/ld.so.conf | points to /usr/upm/env/ld.so.conf. The file in env is generated from V records of package descriptions |
| /etc/profile | Typically sources /usr/upm/env/profile.env to get default environment variable settings set by
V records. |
| /etc/upm.conf | Master configuration file for UPM |
| /var/upm | Information by UPM that is subject to frequent change without necessarily affecting packages installed |
| /var/upm/binary | Binaries build on the system and saved here. Typically the setting of save-binaries in
/etc/upm.conf causes these files to be written |
| /var/upm/build | Package descriptions of binaries available anywhere. Usually the contents are mirrored
off a uOS master site. The setting of save-binaries writes new package descriptions for binaries to
this directory if enabled. |
| /var/upm/cache | All files retrieved from the INTERNET are staged in this directory. The contents
should be erased once in awhile. If a file is in here then the files will not be downloaded from the network. |
| /var/upm/descr | A directory with descriptions for each individual package. This directory is also
mirrored off a master site. It is used for keyword searches. |
| /var/upm/recipe | Recipes on how to build all the packages on the system. UPM will get instructions
from here on how to build software. Usually the contents are mirrored off a master site. |
| /var/upm/extra | The directory might exist if there is any information that does not fit into the
other directories |
UPM Commands
The upm command is used to manage software on a uOS system. The basic syntax is:
upm options command list-of-packages
Summary of commands:
| Command | Parameter | Purpose |
|---|
| install | package-descr / tarball | Install a package from a variety of sources |
| remove | package-name | Remove a package from the system |
| list | pattern | List installed packages |
| search | pattern | Search for a term in all packages and descriptions |
| report | type | Show suid files, broken links and verify the integrity of all files. |
| build | package-name | Generate binary tarball from sourcecode |
| update | - | Synchronize list of recipes, builds and descriptions |
| upgrade | - | Upgrade packages where new versions are available |
| check | package-name | Check actual state of files against meta data |
| repair | package-name | Synchronize meta data to actual files |
| info | package-name | Print information about package files |
| config-save | package-name | Generate -config.tgz tarball with config files |
| config-load | package-name | Load -config.tgz tarball with config files |
| config-default | package-name | Reset configuration to state at install time |
| config-diff | package-name | Build diff of actual config files against original config files |
| extract | package-name | Extract package as a binary package from the installed files
on the system |
| findfile | pattern/file | Find UPM meta data about file |
| track | directory | Check part of the file systems for conformity with meta data. Finds
extra files and lost files |
| directory | directory | Print directory contents with a list of package ownership for each
file |
| cleanup | - | Clean out temporary files |
| syncbuild | build-dependencies-to-ignore | Make sure that all packages are build with installed tools |
| syncattr | attributes-to-ignore | Make sure that the way all packages were build reflects the current attribute
settings in /etc/upm.conf |
| deprebuild | packagenamnes | Rebuild all packages depending on the packages listed |
| source | package [level] | Retrieve source and go up to in the build stage |
| diff | package [level] | Build diff against the source build up to |
| hold | packages | Put a list of packages on hold. Lists all holds if no parameter given. |
| unhold | packages | Release a list of packages. Release all holds if no parameter given. |
| rcdupdate | initscript rc.d-spec | Reconfigure SYSV Init configuraton for an init script |
| stop | initscript | Start service if it is supposed to be active in the current initlevel |
| start | initscript | Stop service if it is supposed to be active in the current initlevel |
Build Levels
| Level | What was done |
|---|
| 0 | Source and Patches retrieved into cache |
| 1 | Source unpacked |
| 2 | First patch applied |
| 3 | Second patch applied |
| n | n-1 patches applied |
| 45 | All patches applied |
| 49 | PRECONFIGURE events run |
| 50 | CONFIGURE action performed |
| 51 | POSTCONFIGURE events run |
| 59 | PREBUILD events run |
| 60 | BUILD action performed |
| 61 | POSTBUILD events run |
| 70 | CHECK action performed |
| 78 | Husk deployed |
| 79 | PREINST events run |
| 80 | INSTALL action run |
| 81 | MAN_HANDLER run |
| 82 | DOC_HANDLER run |
| 83 | INFO_HANDLER run |
| 84 | POSTINST events run |
| 85 | Husk removed |
| 86 | FINAL run |
| 90 | PERM events run |
| 99 | Complete build performed |
Summary of options
| Long Option | Short | Purpose |
|---|
| --version | -v | Print UPM version |
| --help | -h | Print short description of options and commands |
| --keepreal | -k | When doing repair work keep the settings of the files and adjust meta data |
| --deps | -d | Automatically handle dependencies. UPM will only display what packages
would be installed/handled without this option. |
| --fetchonly | -f | Only fetch the necessary files for the operation. Do not perform the operation itself. |
| -a attribute-value | Override attribute setting for building |
| --root dir | -r dir | Install to a directory rather than /. This disables dependency checking and marks packages in a special way so that they can later be rebuild using upm syncbuild |
| --verbose | -v | Increase verbosity of processing. The option can be given multiple times to increase
verbosity even more. |
| --quiet | -q | Decrease verbosity. The option can be given multiple times to decrease verbosity even more. |
| --generate | -g | Generate package description on the fly |
| --no-depends | -D | Generate warnings instead of errors on dependency problems |
| --no-protect | -P | Overwrite files of other packages installed. |
| --nice | -n | Omit files already installed by other packages. |
| --log | -L | Log build output for each build instead of displaying it on the console |
| --fakeroot | -F | Do not build using fakeroot but use real root (only possible as root) |
| --keepdirs | -A | Do not remove temporary build and binary directories |
| --match | -m | Do pattern matching instead of looking for exact match during find file or package |
| --thorough | -t | Be more thorough during track command. Check perms and md5sums. |
| --source | -s | Build and install from source, Ignore upm.conf settings. |
| --binary | -b | Install from a binary only. Ignore upm.conf settings. |
| --config type | -c | Do not abort in case of an unupgradable config file but do the specified action |
Possible Actions for --config
| type | Action |
|---|
| overwrite | Simply overwrite the old configuration file with the new one. |
| keep | Keep the old configuration file. Discard the new one. |
| rename | Rename the old configuration file to name.upm.old and install the new one |
Configuration File Management
Files in the /etc directory are managed in a special way.
In order to allow the restoration of a package to the original condition
all original unmodified /etc files are stored in an alternate location (/usr/upm/etc).
uPM can display the modifications made to the original (upm config-diff) as well as restore
the original configuration of each package (upm config-default).
If new configuration files are provided during an upgrade then uPM
will build a diff from the original old configuration file to the new configuration file
and apply that diff to the user modified configuration file.
In most cases this will accurately upgrade the configuration file without the need
for manual intervention.
If this process would fail then uPM will refuse to install the package.
The situation can be cleared by first saving the configuration files using upm config-save.
Then restoring the configuration files using upm config-default and then performing
the upgrade. Then the old configuration may be restored using upm config-load or the user might
choose another course of action to integrate the changes.
The default way of treating config files can be overridden with the -c option.
Reference of Attributes used by UPM code
(Note that more attributes are defined in /etc/upm.conf. These are attributes
that are directly used by UPM code.)
| Attribute | Purpose |
|---|
| UPM_VERSION | The version of UPM running |
| PACKAGE | The name of the package |
| VERSION | The version of the package |
| P | Shorthand for PACKAGE |
| D | Delta of the version or empty |
| V | Upstream version of the package (VERSION without delta) |
| S | Build directory with Makefile configure script etc |
| B | Temporary target directory |
| CFLAGS | Value of CFLAGS during configure and make. Set up in [env] section of upm.conf |
| LDFLAGS | Value of LDFLAGS during configure and make. Set up in [env] section of upm.conf |
| CONFIGURE | Shell code to execute to configure the package. Not executed if empty. Current directory sourcedir |
| BUILD | Shell code to execute to build the package. Not executed if empty. Current directory sourcedir |
| CHECK | Shell code to execute for testing. Not executed if empty. Current directory sourcedir |
| INSTALL | Shell code to perform the installation into the temporary {B} directory. Must be set. Current directory is sourcedir |
| MAN_HANDLER | Shell code to check for man pages. Current directory targetdir. |
| INFO_HANDLER | Shell code to check and install info pages. Current directory targetdir. |
| DOC_HANDLER | Shell code to check and install documentation pages. Current directory targetdir. |
| FINAL_HANDLER | Shell code to perform final adjustments in the archive. Current directory targetdir. |
| ARCH | Architecture. Used for architecture suffixes. |
| BUILDDIR | The build directory relative to the source directory. If not set uPM will look for
typical files indicating buildable software in the main directory of the source archive and in any directory below it.
If BUILDDIR is set then this search will not be done but the setting will be accepted as the build directory. This
build directory will be accessible via {S}. Note that this is also the directory relative to which patches are applied.
Using BUILDDIR is a mixed blessing because it can complicate access to other elements of the sourcecode archive that
are outside of the build directory. In a complicated situation with multiple build directories it is better to avoid
using BUILDDIR. Instead use cd shell commands on events and actions.
|
UPM Configuration file
This is the standard upm.conf for reference:
# UPM Configuration File
# /etc/upm.conf
#
# Christoph Lameter, , August 21, 2002
#
# Mirror to use to retrieve new package information
mirror=http://lameter.com/upm/devel/uos/mirror
# RSYNC server to synchronize availablity lists
sync=lameter.com::linux_distros/telemetry/sputnik
# Default patch and source location (used if mirror fails)
#patch_loc=http://lameter.com/upm/devel/uos/sputnik/patches
#source_loc=http://lameter.com/upm/devel/uos/sputnik/sources
# For some packages sources and binaries are available.
# Binaries are usually available from the mirror for important big packages.
# But binaries are also locally generated.
# The following settings indicate to UPM how binaries should be used
#
# binaries=always|never|ifavail|iflocal
binaries=iflocal
# The binaries generated during the build phase can be automatically
# saved if a path is specified here.
save_binaries=/var/upm/binary
# UserIDs for access to metadata and building
# uPM will build as root if these are commented out.
upmuser=upm
builduser=build
[mirror]
# List of mirrors to be used using S @/path P @/path
# Resorting this so that the nearest mirror comes first improves fetch performance.
tbox=http://lameter.com/telemetry http://tbox.lameter.com http://telemetrybox.org/tbox
debian=ftp://ftp.debian.org/debian
kernel=ftp://mirrors.kernel.org/pub/linux
sourceforge=http://unc.dl.sourceforge.net/sourceforge http://belnet.dl.sourceforge.net/sourceforge ftp://telia.dl.sourceforge.net/pub/sourceforge
gentoo=http://www.lameter.com/pub/linux/distributions/gentoo/distfiles
gnome=ftp://ftp.gnome.org/pub/gnome ftp://archive.progeny.com/GNOME ftp://ftp.sunet.se/pub/X11/GNOME ftp://ftp.no.gnome.org/pub/GNOME ftp://ftp.gnome.org/pub/gnome/2.0.0
kde=http://lameter.com/pub/packages/desktops/kde/ ftp://ftp.kde.org/pub/kde/ ftp://download.us.kde.org/pub/kde/ ftp://ftp.gtlib.cc.gatech.edu/pub/kde ftp://download.uk.kde.org/pub/kde/ ftp://download.au.kde.org/pub/kde/ ftp://download.at.kde.org/pub/kde/ ftp://download.tw.kde.org/pub/kde/
gnu=http://lameter.com/pub/gnu ftp://mirrors.kernel.org/gnu ftp://aeneas.mit.edu/pub/gnu
[env]
#
# Environment settings for the build process.
# These are expanded using values from the attribute
# sections of the package and those defined here.
# Note that these are NOT attributes to be tracked
# or to be used in regular attribute expansion
# If you want to do that then define such attribute
# in the attributes section
CFLAGS={CFLAGS}
CXXFLAGS={CFLAGS}
CC=gcc
[attributes]
# These are lists of default attributes to be used for
# building programs.
# Attributes defined here can be overridden by attribute
# specifications in the package or by package specific
# overrides later or by attributes specified on the command
# line of upm.
# The following attributes are always predefined by upm
# P The packagename (same as PACKAGE)
# V The upstream version number
# VERSION the full version number including delta
# D The delta
# S Source directory (Only available during source building and install )
# B Binary directory (Only available during installation phases )
# UPM_VERSION Version of UPM processing the current recipe
# Specification what to do in each of the build stages
# The shell expansions here are deferred until the time
# just before the corresponding build stage is run
# These are preset for autoconf sources.
# CONFIGURATION STAGE
# The configuration stage. If this package does not
# support autoconf override this setting using
# A CONFIGURE
# CONFOPT can be set with additional options
CONFIGURE=./configure --prefix=/usr {CONFOPT} --host={HOST}
# BUILD STAGE
# This works with most packages
# Override BUILD if additional targets are needed
BUILD=make
# VERIFICATION STAGE
# The check stage is typically unused. Define this to override and actually
# perform a regression test or so.
CHECK=
# INSTALLATION STAGE
# Installation. Typically the specification of the destination
# varies. Define DESTSPEC attribute to fill that one out if the
# prefix default will not work.
INSTALL=make {DESTSPEC} install
# POST Installation management
# Installation of manpages via the A MAN attribute
MAN_HANDLER={MAN?"man_handler {B} {MAN}"}
# Installation of documentation via the DOC attribute
DOC_HANDLER={DOC?"doc_handler {B} {P} {DOC}"}
# Installation of documentation via the INFO attribute
INFO_HANDLER={INFO?"info_handler {B} {INFO}"}
# Final handler run unconditionally
FINAL_HANDLER=final_handler
# Override this to specify your way of setting the binary
# destination to install into in "make install"
DESTSPEC=prefix={B}/usr
# The default format for the CFLAGS passed to programs.
# Other archs than Intel need to have this setup differently.
CFLAGS=-mcpu={CPU} -O{O} -march={ARCH} {MMX?"-mmmx"} {NOEXCEPTIONS?"-fno-exceptions"} {OMITFRAMEPOINTER?"-fomit-frame-pointer"}
# Frequently used shorthands
PV={P}-{V}
# Someone screwed up regexp processing in glib 2.3 \w does not match numbers anymore... sigh.
Vm={V:"[0-9A-Za-z]+"}
Vmm={V:"[0-9A-Za-z]+\.[0-9A-Za-z]+"}
# Can the platform support X86 assembly? Might have to be extended for future compatible archs.
# Typically this is used to generate specially optimized code. If this is not set C code is substituted.
X86={(ARCH:"i.86")|(ARCH:"k6.*")|(ARCH:"pentium.")|ARCH="athlon"|ARCH="duron"}
# CPAN is organized by sections. They are separated by a - in the packagename.
PERLSECT={(P:".+-.+")?(P:"(.+)-.+"):P}
# Common references to upstream sites
GET_GNOME=@gnome/sources/{P}/{Vmm}/{PV}.tar.bz2
GET_KDE=@kde/stable/{V}/src/{PV}.tar.bz2
GET_SF=@sourceforge/{P}/{PV}.tar.gz
GET_DEBIAN=@debian/pool/main/{P:"(.)"}/{P}/{P}_{V}.orig.tar.gz
GET_GNU=@gnu/{P}/{PV}.tar.gz
GET_DDIFF=@debian/pool/main/{P:"(.)"}/{P}/{P}_{V}-{DD}.diff.gz
GET_CPAN=http://cpan.org/modules/by-module/{PERLSECT}/{PV}.tar.gz
README_CPAN=http://cpan.org/modules/by-module/{PERLSECT}/{PV}.readme
# Control of GCC compilation options
HOST={ARCH}-pc-linux-gnu
# Optimization. O=2 is typically safe.
O=2
# For best performance set the following to the CPU type you actually have.
# The CPU setting optimizes the scheduling of the instructions for the specific CPU
# The ARCH setting makes the compiler use a specific instruction set.
CPU=i686
ARCH=i486
#MMX=Y
#NOEXCEPTIONS=Y
#OMITFRAMEPOINTER=Y
# Languages to support. If a language is disabled
# and a package requires it then it will be impossible
# to install the package. Notably C++ is used for
# a large selection of packages and should
# not be disabled unless you want
# a very minimal system.
CXX=Y
#JAVA=Y
#FORTRAN=Y
#OBJC=Y
# Interpreted languages. Uncommenting will cause all packages that can
# support that language be build with that language support
#PERL=perl
#PYTHON=python
RUBY=ruby
# National Language Support. This will include all other languages.
#NLS=gettext
# Databases
BERKDB=db
GDBM=gdbm
# MYSQL=mysql
# POSTGRES=postgresql
READLINE=readline
GPM=gpm
GTK1=gtk+1.2
GTK=gtk+
X=xfree
SSL=openssl
PAM=pam
JPEG=jpeg
PNG=libpng
TIFF=tiff
OGGVORBIS=libvorbis
# ACL=
# ALSA=
# ARTS=arts
# CRYPT=
# CUPS=cups
# CURL=curl
# DIRECTFB=directfb
# ENCODE=lame
# ESD=esound
# FIREBIRD=
# FLASH=
# FREETDS=
# GD=gd
# GGI=libggi
# GNOME=gnome-libs
# GPHOTO=
# GTK=gtk+
# GTK1=gtk+1.2
# LCMS=lcms
# LDAP=openldap
# LIBWWW=libwww
# MIKMOD=mikmod
# MOTIF=openMotif
# MTA_LDAP=openldap
# MTA_MYSQL=mysql
# MTA_POSTGRESQL=postgresql
# NAS=nas
# ODBC=odbc
# OPENGL=opengl
# PDA=
# PDFLIB=pdflib
# QT=qt
# SAMBA=samba
# SCANNER=
# SDL=SDL
# SLANG=slang
# SNMP=snmp
# SOCKS=socks
# SVGA=svga
# TCLTK=tcltk
# TCPD=tcp-wrappers
# TETEX=tetex
# XML=libxml2
# XML2=libxml2
# Simple Flags
# Debugging support
#DEBUG=y
FBCON=y
#DOCUMENTATION=y
#MTA_TLS=y
#IPV6=y
# Package specific overrides
[rocksndiamonds]
# I want this package to be compiled with maximum optimization
O=6
Comparison of different package managers
Comparison of various package managers
| Issue | uPM | RPM | DPKG(Debian) | Portage(Gentoo) |
|---|
| Meta data | All machine processable | spec file with partially machine processable parts. | control file machine processable. Extensive use of scripts otherwise | ebuilds are scripts and recently Gentoo has attempted to extract information from scripts which is rather difficult. Attempts to structure the scripts are visible |
| Overwriting files of other packages |
Off by default. Controllable via Blacklists, Package options and command line parameters |
Do not know |
Overwrites files by default for stable releases. Off in the unstable release (!). Controllable via command line option. |
Design allows overwriting files (source: Daniel Robbins) |
| Build procedure | Building and installation as a regular user (fakeroot environment) | Typically as root | fakeroot builds are default | Uses sandbox which filters file accesses(!) while build is being done as root (insecure) |
| Integration | Triggers, Blacklists and Meta data. Scripts as a fall back | Simple script scheme | Sophisticated scripting scheme that has proven itself over years | Simple script scheme |
| Binary packages | Yes. Full support | Yes | Yes. Advanced package management | Limited support. Impossible to generate a package without installing it. |
| Customizable build procedure | Yes | Source config fixed | Source config fixed by maintainer | Flexible build procedure |
| Tracking of build characteristics | Yes | N/A | N/A | Some support through "USE" flags. Fragile because changing of build characteristics cannot be tracked. Allows rebuild of all system components to fix that. |
| System wide package configuration | Attributes allow detailed configuration | No | No | USE flags allow simple on/off configuration |
| Language base | C+Shell | C+Shell | C+Perl+Shell | Python based +Shell+Some C. |
| Rebuild packages affected by global config change | Yes | No | No | No |
| Rebuild packages to reflect new build tools | Yes | No | No | No |
| Rebuild all buildable packages skipping broken builds | Yes | No | No | No |
| User can build package | Yes | Yes | Yes | No |
| Configuration file Management | Attempts to patch config file. Can show diff to default config. Can restore to default config. Strategies
for resolution selectable on the command line. |
Renames config file |
Updates if config file was not changed. Prompts for conflicts. |
Renames config file. Newer version check CVS id for changes (J) and update unchanged config files |
| Package removal | Removes everything associated with package | Macros allow the removal of everything | Typically done by script | PostRM script can remove leftover files (usually not done). |
| Package upgrades | Removes all traces of old files | Removes all traces of old files | Removes all traces of old files | Leaves rests of old files not replaced by the upgrade. Recent version remove leftovers(J). Experienced some breakage due to libraries vanishing. That might be fixed now. |
| Alternatives (multiple files with the same name) | Yes | Yes in Mandrake | Yes | No |
| SYSV init scripts | Integrated / Script generator | Maintainer scripts | Maintainer scripts | Uses its own scheme of dependencies between init.d scripts. Not compatible with SYSV init scripts. |
| Generalization of build procedures | BUILD/library dependencies rewrite RECIPE | N/A | N/A | Eclass (shell scripts overriding shell functions in ebuilds). |
| Multiple patches | Yes | Source archive | one patch (diff) | Yes |
| Guestimate of lines of text to write/generate to get a source package integrated | 10 (current average is 16 lines/package) | ~200 | ~500 | ~100 |
| Analysis of library dependencies to insure accuracy of runtime dependencies | Yes | In Mandrake | Yes | No |
| Complexity | Simple single C binary plus a few shell scripts | Simple single C binary + scripts |
Difficult to comprehend suite of C, Shell and perl scripts that has been added to for almost 10 years |
Difficult to comprehend mixture of Python scripts, Shell scripts and some C.
Lots of scripts that do not add significant functionality (doman,doins,doexe,insinto and so on). |
| Number of available packages | 350 | Most widely used distribution. But various flavors with incompatible .rpm packages | 9000 | 2500 (Maik Schreiber) |
| Homepage | http://lameter.com/telemetry | http://www.redhat.com | http://www.debian.org | http://gentoo.org |
Please email me should there be any inaccuracies in the comparison above or if you
could contribute more helpful information.
Building Debian Packages with uPM
uPM can use debian sources in a variety of ways to build packages. The uPM framework can be used to simulate a Debian
packaging environment using debian/rules.
In order to have that environment available a package needs to have a build dependency on deb-build-compat.
Note that the use of debian/rules is often unnecessary. Debian packages often build fine with the standard
R BUILD c-build environment. debian/rules is only necessary if things in addition to the standard
upstream install action are needed.
Retrieving Debian Sources
The following lines will retrieve debian sourcecode and the debian diff if the source archive is named the same
way as a Debian package and if the version number is the same. The Debian delta must be set using the DD attribute::
A DD <debian-delta>
S {GET_DEBIAN}
P {GET_DDIFF}
For details on how these attributes work have a look at /etc/upm.conf.
Dependencies
A way to process the Debian metadata in the debian/control file is not yet available. Until we have that working it is necessary
to declare at least BUILD time dependencies using R BUILD lines.
Building the software
If the package contains only a single binary .deb target then a R BUILD deb-single-compat will set up
a complete build environment and install the package as necessary using debian/rules.
Packages with multiple .deb targets need individual treatment. The cron package is an example of that.
Basically a R BUILD deb-build-compat is established. The package provides a set of scripts that
simulate typical scripts invoked from debian/rules. uPM will then be instructed to invoke debian/rules
to generate the target tree.
Limitations
- The script emulation is not complete. Some scripts might not have been provided and debian/rules might fail.
- Maintainer scripts are not invoked by default. Typically the trigger mechanisms of upm take care of all
necessary actions. If you need to execute the scripts put them into /usr/share/packagename/postinst etc and then
execute them from the corresponding E POSTMERGE etc events.
- The software infrastructure that uPM provides is not complete yet. It would be great if you could help to bring more debian infrastructure into the distribution.
TODO LIST
- Find and fix bugs in uPM, the bootup and the installation process.
- Fix bugs and enhance recipes that exist.
- Contribute more Recipes for more software. See the available recipes for examples on how to do this.
- Pick a name for the distribution. Telemetry is just a name that was available but might not be a suitable
name for a Linux distribution. The most favored name right now is uOS (Micro OS).
- Generate a logo and some design to produce nice looking web content.
- Create a web portal for uPM packaging with a mailing list for each recipe that exists.
- Create a tool that allows online editing of recipes.
- We need a volunteer to do the administration of the web and the mailing lists.
- We are looking for people responsible for subsystems (like networking, X etc)
- If you have any other gifts and talents that you might want to contribute ask us.
- Port to other architectures. We only have intel hardware right now. uPM has never been ported to another
architecture so surprises are to be expected when that will be attempted.
- Enhance the ability to directly build source code from Debian, RedHat and others.
- Write tools that process recipes and allow global changes to metadata.
Contact Christoph if you would like to help.
Contributing
I surely wish that uPM would be part of a larger distribution but I have to realize after an experience with another
distribution that this might not be realistic. Maybe this could one day be a real Linux distribution. It might
have to prove itself before becoming part of something larger.
If you want to help out you can find me on IRC (OPN err freenode network. See http://freenode.org). Talk to o-o. There is a channel for matters related to uPM and uOS called #uOS.
The best contribution right now is to just simply download the image and find out all that is wrong with it. Give us
feedback (use the tbox-talk list at sourceforge for that purpose) and if you can fix things then send patches.
Look at the Todo List for things that we know need help with.
SYSV Init scripts with UPM
The way uPM handles init scripts is significantly different from other what distributions do. Scripts are typically used that
run on install to generate links in /etc/rcX.d. uPM generates the links instead a result of Q Records in
recipe during build time. This means that no script needs to be run at installation time, nor when the package is to be removed. RC links are treated like any other link in a binary tarball. Symlinks are managed as configuraton files since they are in the /etc
directory.
Automatic start and stop of services
uPM starts and terminates services on install/removal/upgrade if the Q record specifies that this should be done. uPM also
verifies that the service is supposed to be active in the currently active runlevel during the installation. If symlinks
for the runlevel do not exist or if the runlevel cannot be determined (chroot environment) then no startup or termination will
be performed.
Customizing SYSV configuration for a init-script
Since uPM handles the /etc/rcX.d directory it is not advisable for the system administrator to change symlinks after
install or run rc.d-update to change symlinks. uPM will change them back on the next upgrade. uPM has a special
command upm rcdupdate that can be used to change the SYSV symlinks for an init.d script.
Definition of Runlevels
uPMs definition of runlevels is different. It is recognized that todays machines are typically connected to the Internet.
One major characteristic that needs to be controlled is the connectivity to the Internet. It must be possible to
switch machines between different service levels and react to threads or vulnerabilities. With that in mind uPM defines
the following runlevels:
- 0 HALT
- 1 Single User. Network might be used as a client only.
- 2 Safe Mode. Multi-user mode. All network services down. Console access only.
- 3 Diagnostic Mode. All network services down except for remote secure system administration (SSH, VNC etc)
- 4 Infrastructure Mode. Network Infrastructure Services up (DHCP, Routing, DNS etc)
- 5 Full Service Mode. All Internet Services up (httpd, rsync, ftp etc)
- 6 Reboot
The default runlevel to boot into is 3. In order to run a server or a router ./etc/inttab must be modified.
uPM will not automatically offer services to the outside world. If apache is installed in runlevel 3 then it will not be automatically started!