Re: [hfdb] Scope (Was Re: Grand Unified Hardware Database)

[Zenaan Harkness]

On Mon, 2004-07-19 at 19:56, David Zeuthen wrote:
> Hi,

G'day.

...
> > I'm guessing our little RDBMS/XML debate is really just a proxy for a more
> > basic one: What is the scope of the project?  What, exactly, are we trying
> > to track, and for whom?  I have my ideas, some of which are outlined
> > above, and quite a few others have been tossed around just prior to our ML
> > coming online.  
> >
> > Maybe a concrete example would help.  I suggest this:  Show me a mock
> > tabular report you'd like to see, something you imagine would benefit
> > someone we're trying to help, and tell me who that person is.  Just a half
> > dozen lines should do.  Or point me to something online and tell me what's
> > missing or overmuch in it.  Maybe if we can converge on what we're doing,
> > we'll agree on how.  
> 
> I think, at least from the HAL projects point of view, we want a lot
> of data, indulge some rambling :-)

Rambling happily indulged :)

> One thing that we're working on right now are the ''soft properties'' of
> hardware; e.g. today we can merge a HAL Free Device Information (.fdi file)
> like this one
> 
>  <?xml version="1.0" encoding="ISO-8859-1"?> <!-- -*- SGML -*- --> 
>  
>  <deviceinfo version="0.2">
>    <device>
>      <match key="info.bus" string="usb">
>        <match key="usb.vendor_id" int="0x066f">
>          <match key="usb.product_id" int="0x8000">
>            <merge key="info.category" 
> type="string">portable_audio_player</merge>
>            <merge key="info.capabilities" 
> type="string">portable_audio_player</merge>
>            <merge key="portable_audio_player.output" type="string">mp3 
> wma</merge>
>            <merge key="portable_audio_player.input" type="string">wav</merge>
>            <merge key="portable_audio_player.localdb" 
> type="bool">false</merge>
>          </match>
>        </match>
>      </match>
>    </device>
>  </deviceinfo>
> 
> which says that the USB device with VendorID 0x66f and ProductID 0x8000 
> is in fact not just a USB storage device, but it's actually a MP3 player.
> 
> Now, this is important as the OS sees it the device there's no chance,
> apart from unreliable heuristics, that the OS can identify it as a MP3
> player capable of playback of MP3 and WMA and recording of WAV. With
> this, today, we can actually display a nice icon on the desktop like
> this one
> 
>  http://freedesktop.org/~david/gnome-vfs-take3-1.png
> 
> instead of a dull grey icon :-). We're still working out to define the
> names of the various properties and integrating this into the desktop
> and applications. There's a few other devices that needs ''soft
> properties'', for instance cameras.

This is well within our scope as I see it - this is simply a mapping
from various device ids (sub-ids, etc) to device types. Exactly the sort
of information that a device database ought to contain I would have
thought.

> All this is important if we want Free Software OS's to meet, and exceed, 
> the level of hardware handling that is present in the commercial operating
> systems available today such as Mac OS X and Windows XP. It's important

Precisely!

> to note that such properties are completely OS independent; it's also
> important to note that while this give some impression of ''Making hardware
> just work'' it really requires operational and working driver software
> on the system to which the device is attached.

Lots of agreement here.

> In order to solve the second part of the puzzle, e.g. how do we somehow
> intelligently find out how to obtain driver software, I think this can
> also be partially solved through this proposed database.
> 
> Now, one problem of free software OS'es is that while there is driver
> software available it's, for one reason or another, not available in
> the installed OS. We've probably all tried this; we have to Google for
> the driver and most of the time there is one or two projects on sf.net
> where we can download the tarball and install it :-). This is not my
> idea of a usable operating system.

<jest>
Oh come on now! A little make, perhaps a grep/ find on the filesystem to
see what was installed, and even some modprobe-ing never hurt anyone :)
</>

> My proposal for this would be have a OS-dependent name of the driver
> in the database.

I seem to remember Intel starting some OS-independant driver project,
but that it never flew well (perhaps due to kernel developer's rather
disliking binary drivers). Anyway, if there's a free driver for free
software, we can classify it however it needs to be classified,
obviously. I agree that most if not all drivers will be OS-specific.

>  E.g. so from the database I might be able to generate
> this HAL .fdi file
> 
>  <?xml version="1.0" encoding="ISO-8859-1"?> <!-- -*- SGML -*- --> 
>  
>  <deviceinfo version="0.2">
>    <device>
>      <match key="info.bus" string="pci">
>        <match key="pci.vendor_id" int="0x109e">
>          <match key="pci.product_id" int="0x0369">
>            <match key="pci.subsystem_vendor_id" int="0x1002">
>              <match key="pci.subsystem_product_id" int="0x0003">
>                <merge key="linux.driver" type="string">bttv</merge>
>                <merge key="freebsd.driver" type="string">bktr</merge>
>              </match>
>            </match>
>          </match>
>        </match>
>      </match>
>    </device>
>  </deviceinfo>

Exactly.

> and from the OS will be able to know that it needs this driver. Granted,
> this example covers a device that is probably available in most Free OS
> kernels, but think of other devices (the above is only an example).

Well, the kernel maintainers have expressed an explicit intention to
remove all traces of PCI ID database/ file from the kernel sources, in
the 2.7 series kernel. Which means that generating such (external-to-
kernel) descrioptor files is going to become standard practice. Even for
devices supported directly in current mainstream kernels.

> So, basically, I think that it would be perfect if the database is
> designed in a way so all this data is stored in a convenient extensible
> way. It seems to me that a RDBMS with normalised tables is the best 
> option, because if a new device type emerges, say a portable video player
> we could just introduce a new table PortableVideoPlayer and link that
> to the Device table.

James, please correct/ enhance whatever I say here as needed: I suspect
that the main difference between RDBMS and XML storage, is simply one of
convenience and automation - an RDBMS has mechanisms (such as foreign
keys checking, constraints checking, and surely others) to help ensure
that the data going in is valid (and when we discover new "rules" for
the data, we can easily add those rules to the DB). Also, XML can be
readily generated from an SQL DB.

The downside is that it means we need to be actually running an RDBMS
somewhere - and we are hosting on savannah so that is an admin overhead
that we will be "imposing" on their admins at some point. Plain XML
storage doesn't have that overhead.

However I think the pros outweight the cons.

In terms of data entry, on the surface an XML file is human-readable and
therefore "easy" to manually enter data. On the other hand, to do this
right (proper normalization, various integrity constraints), you do need
the full relational semantic power. It is possible that there are tools
I am unaware of that provide this sort of thing on top of XML files - I
am of course willing to be enlightened.

At least in the early days, I think that simplifying data entry
integrity (constraints validation), is the most important thing for us
to achieve, wrt storage formats. Down the line, I imagine that additions
will slow down drastically (to 5 to 20 additions per week), at which
point, the various normalized fields/ data structure will be well
established, and so the difference between an RDBMS and XML storage
format will be able to be easily hidden from the data entry process. I
_suspect_, that it will be easiest to achieve what we want in the early
days by use of an RDBMS.

> Though I've spoke of HAL above, the database should definitely be indepedent
> of HAL, but I'd really like to see it support all the features that HAL
> can use :-)

We will most happily oblige your request.

> I hope I wasn't rambling too much :-)

Get the whip out! 50 lashes for such laxity! And another 50 lest you
forget for next time!

Thank you very much for contributing - every little bit clearly helps.

regards
zen