Tracking Architecture: all Packages per suite

Current Status (20091124)

Example: whenever a new source package for emacs23 gets uploaded the installation of the metapackage emacs_*_all.deb breaks on most architectures until the needed Architecture: any packages like emacs23 gets build by the buildd. That happens because dak removes all arch: all packages but the newest one.

The new code in dominate.py and generate_filelist.py has been activated for the dinstall cron job and replaces the old code in make_suite_file_list.py. Dominate is actually removing source and binary packages from suites and generate-filelist is generating index files for apt-ftparchive.

Suites marked as untouchable (lenny-r0, etch-r0, etch-m68k, oldstable, stable) are skipped by dominate. The cleanup process deletes records from the tables src_associations and bin_associations for all other suites.

Unfortunately some parts of cruft-report (the detection of obsolete sources) is broken now and has to be disabled temporarily. A replacement is in development.

New Algorithm

dominate

The new algorith allows multiple versions of source packages per suite as long as they are referenced by some valid arch: any package and it keeps all versions of arch: all packages that references any valid source package. We need to keep packages that build only all packages. Some principals for packages in the same suite:

From all packages with the same name and the same 'any' architecture we want to keep the newest package (1) only. Older ones will be removed. The process is called 'newer_any'.
We want to keep all source packages (2) that are referenced by all type (1) binary packages.
From all source packages with the same name we always want to keep the newest one (3) to support source only uploads.
We remove all source packages that are not of type (2) or (3). The process is called 'old_and_unreferenced'.
We remove all arch all packages that are not referenced by existing source packages but to be safe we always keep the newest version similar to the source removal case. The process is called 'old_and_unreferenced', too.

We ignore components in the algorithm and we have the following part disabled because it is too dangerous:

An arch all package obsoletes all arch: any packages with the same name and a lower version number. The obsolete arch any packages will be removed. This process is called 'newer_all'. This can lead to uninstallable packages (example: the introduction of a transitional arch all package).

obsolete source detection in cruft-report

TODO

dak dominate

GRANT SELECT permissions for new database views to ftpmaster.

dak rm

Extend dak rm to understand -a any which means every architecture except 'all'.

dak rm does not have on option for specifying the version(s) to remove. Something like -V VERSION could be added where VERSION is a comma separated list of versions and defaults to all versions in SUITE.

BTS

As soon as dak rm has -V it should be possible to specify a version number in bug reports. The current subject is 'RM: $PACKAGE/$SUITE [$ARCHLIST] -- REASON' where '/$SUITE' is optional and '$SUITE' defaults to 'unstable'. The new format could be 'RM: $PACKAGE/$SUITE [$ARCHLIST] $VERSION -- REASON' where the comma separated '$VERSION' is optional and defaults to all versions in '$SUITE'.

reportbug

Reportbug should support the new subject.

dak cruft-report

There are several new (and some old) corner cases not handled by cruft-report yet.