Daemon News Ezine BSD News BSD Mall BSD Support Forum BSD Advocacy BSD Updates

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Proposal for binup design document


I really think binup could be useful and fill a gap for FreeBSD.
Therefor to try to get things moving and to get some kind of structure
on binup I have begun make a kind of design document containing some
thought / ideas about binup. I think it is important to discuss these
issues (especially the HTTP proposal) so we know what to make binup do
and how it should be done.

I have a attached a text only version of the document, but it has been
written in HTML and can be found at
http://simon.nitro.dk/binup/binup-2003.html .

I plan to start looking at the server database backend today and
adding/changing the binup design document along the way.

Simon L. Nielsen
                            Binup 2003 proposal


     * [1]Introduction
     * [2]General
     * [3]HTTP
     * [4]Main use
     * [5]Package abstraction
     * [6]Package the base system
     * [7]Client/server communication
     * [8]Tasks

   Updated : $Date: 2002/12/27 21:17:46 $

binup 2003 introduction

   This is a proposal by Simon L. Nielsen <[9]simon@xxxxxxxx> for a
   design document for the FreeBSD Binary Updater (binup). The
   document contains some ideas for changes to the current binup
   code/design and the beginning to a general design document about

   The document is very much work in progress... and a bit of a mess
   at the moment.

   I have called this document binup 2003 to differentiate it from
   the current "old" binup code. The [10]"old" binup document should
   be read before this document since the main design and goals
   haven't changed.

   Please note that this document is meant as an early draft and most
   likely contains flaws, bad ideas and things that have not been
   thought fully through. There is also most likely a lot of spelling
   and grammatical errors - please ignore them for now. It contains
   comments/ideas on the subjects I think needs to be addressed
   before real coding on binup 2003 can begin.

   I would very much like comments on the document and the ideas. I
   would prefer that comments are posted to the [11]binup
   mailing-list for discussion but private comments directly to me
   are also very welcome.

   I plan to keep an updated version of this document on


   I think it would be a good idea to change the binup system to use
   HTTP instead of doing a new TCP level protocol. Mainly because I
   think this could save a lot of coding. See the [13]HTTP section
   for more.

   As suggested on the binup mailing-list some time ago I think binup
   should be made independent of the packages actually transfered by
   binup. This would make it possible to use distribution tar-balls,
   .tgz's, libh, binary patches or any other packaging format without
   any major change to binup. See the [14]package abstraction section
   for more.

   There have been talk about making the binup project do a package
   API (/var/db/pkg API) and do the breaking up the base system into
   packages. While I think these are good ideas that would certainly
   benefit binup I don't think we should wait for this to be finished
   since that may take a long time and it should be possible to do a
   good binary updated without these tasks completed.


   The current binup code uses a special protocol on top of TCP. This
   makes the protocol more flexible but also require a lot of code to
   get working. Instead binup could be implemented on top of HTTP.
   This would make some things a lot simpler since a lot of existing
   programs/code can be used both server and client side. The use of
   HTTP will simplify implementation since a lot of functionality
   will come "for free" e.g. file transfer, SSL and connection
   management. Standard server/clients also gives the advantage that
   less low level C code has to be written and therefore minimizing
   the risk for buffer overflows and other common C bugs.

   Using HTTP it would also be quite simple for the binup server to
   point a client at a normal FreeBSD mirror for the actual
   downloading of updated packages (or perhaps the client can pick
   it's preferred mirror site). A checksum/crypto signature could
   still be send from the binup server and therefore increasing
   security a bit for trojan-ed files. This might not be a big
   increase - but it would help a bit.

   The main drawback of using HTTP is the lack of state between
   requests but I don't see this as a major drawback since most the
   needed state code can be kept client side instead.

   Advantages of using HTTP :
     * Less reinventing the wheel
     * Will be using standard Internet protocol (big advantage if
       client is behind a firewall)
     * Simpler prototyping (a lot of the code can be initially
       implemented in a scripting language)
     * Less low level C code

   Disadvantages of using HTTP :
     * No state
     * Will mean scraping most of the current binup code

  Main binup uses

   This section contains some thoughts on where/how binup could be
   useful. I think this is important to keep in mind to determine
   exactly whan binup is supposed to do.

   The main uses for binup as I see it is (in more or less
   prioritized order) :
    1. Security updates (e.g. 4.7p1 -> 4.7p2)
    2. Port updates
    3. FreeBSD upgrades (e.g. 4.6 -> 4.7)
    4. Fresh FreeBSD installs with profiles

   I see two main types of operation for a binup server - private and
   public servers.

   A private server can for example be used for a company to have a
   simple way to deploy updates to a server park. In this scenario
   updated will most likely have to be tested before being deployed
   widespread in the administrators must therefor have ability to add
   a package to be updated on a per package basis (package in a
   general sense both OS upgrade and port upgrade).

   The public server could be used as a kind of replacement for
   buildworld/installworld for people who just want to do a network
   upgrade. This could also be very useful for people who want to
   keep up to date with security fixes without having to recompile
   everything for a "simple" small patch.

   When clients do an update the binup client should allow this to be
   done either automatically or in an interactive mode.

  Package abstraction

   I belive it would be useful for binup only to have a minimal
   "knowledge" of the packages being transfered. That way binup is
   not being bound to a specific packing format which might change
   once libh is fully implemented or the base system is packaged in
   some special format. This would also alow update packages to be
   available in multiple packaging formats which might be useful in
   some cases.

   It would of course make the tripwire functionality harder to
   implement, but it could still be done.

   This could mean that binup could be used to transfer all kind of
   odd package formats even .rpm, .deb packages if that should be

   binup mainly need three pieces of information for each package as
   far as I can see :
    1. Package name - must be unique
    2. Package format - so the client knows how to install the
    3. Package version - to find new versions

   The real tricky one is package version since third party programs
   use all kinds of odd version schemes.

  Client/server communication

   I think that when a client wants to upgrade the client should get
   a list of the most recent packages for a profile the client
   specifies and then the client decides which packages to download
   and install. This way partial upgrades should be handles
   automatically and the privacy of the client is preserved since the
   server does not know which packages the client does actually have

   The client -> server can be done by a simple HTTP GET or POST with
   the needed variables.

   For the server -> client data there must be some kind of
   formatting/encoding so the client can parse it in a simple way.
   The main formatting/encoding styles I see are comma separated or

   XML is very general and easy to extend later but it has a few
   major drawbacks. The main drawback is the requirement of an XML
   parser client side and I think the binup client should depend on
   as few external components as possible to minimize problems when
   blowing things away under the client and when doing from scratch
   installs. Another drawback for XML is the extra overhead both from
   larger network transfers and from the parsing.

   Comma separated would require special parsing code to be written
   and would most likely break if new fields are added to the lines.
   The advantage is that it requires no external library to parse and
   it is rather fast to parse.

   I lean mostly to comma separated format since it does not require
   a XML library on the client...

  binup tasks

   The tasks to get a complete binup system as I see it (not all may
   be required to get first version of binup working)
     * Find out what binup 2003 should really do
     * Specify client/server protocol
     * binup server
          + Create database back-end
          + Implement server in some scripting language
          + Implement server in C (perhaps apache module ?)
     * binup client
     * package up base system
          + Package lib API (API for /var/db/pkg)
          + Find out how to do versioning (Problem with source

    Implementation notes

   Database back-end should be implemented in server so it can be
   replaced later e.g. with dbm files if needed.
$Id: binup-2003.html,v 1.6 2002/12/27 21:17:46 simon Exp $


   1. #intro
   2. #general
   3. #http
   4. #main-use
   5. #package-abstraction
   6. #package-base
   7. #client-server-comm
   8. #tasks
   9. mailto:simon@xxxxxxxx
  10. http://www.freebsd.org/projects/updater.html
  11. mailto:freebsd-binup@xxxxxxxxxxx
  12. http://simon.nitro.dk/binup/binup-2003.html
  13. #http
  14. #package-abstraction

Attachment: pgpAmsZtWdKuE.pgp
Description: PGP signature