Re: [hfdb] Scope (Was Re: Grand Unified Hardware Database)

[Zenaan Harkness]

On Mon, 2004-07-26 at 09:03, James K. Lowden wrote:
> One caveat: Comment fields are notoriously abuseable.  It's always
> tempting to stuff data into a comment e.g., "has 2 USB 2.0 ports".  The
> problem with that sort of thing is that it's not (readily) searchable.  It
> will be important that people loading the database understand that
> comments (if we provide for them) are "metadata of last resort".  

This is a very good point.

> > > So, looking at my keyboard, I'd say:
> > > 
> > > Data that can be extracted:
> > > ===========================
> > > The USB product id is 0x058f, the USB product id is 0x9410 and the USB
> > > device revision is 0x0101. This uniquely identify my device. 
> 
> Hold on.  What's the difference between a USB product id and a USB product
> id?  

I suspect id and sub-id, or perhaps vendor + product. David?

> > > Data that is useful for software:
> > > =================================
> > > 
> > > My laptop got a nonstandard key on the keyboard is a button with a
> > > 'power off' symbol and the keycode is <some_kay_code_value> (I haven't
> > > looked it up yet). So, we might be interested in storing this and one
> > > day export it to HAL in this fashion
> 
> This is a perfect example of a DeviceFeature and a DriverFeature, and it
> raises an interesting point.   The intersection of these features has to
> be described:

"Is useful to describe" :)

> 1. Device features ignored by the driver.
> 2. Device features supported by the driver.
> 3. Device features required by the driver.
> 4. Device features not required by the driver.
> 
> Your nonstandard key is probably supported by the standard keyboard
> driver, but the poweroff button is inoperable (1).  Perhaps there is or
> will be a driver that supports it (2).  That driver might not work with
> keyboards lacking it (3).  But, perhaps that support will find its way
> into the standard driver, which will continue to support keyboards without
> it (4).  
> 
> (#3 and #4 are really subtypes of #2.)

> > Of course, with all the discussions of XML with RMS, we might have to
> > look into XML import filter or something. I hope to do enough reading on
> > it this week.
> > 
> > Which actually leads me to think, perhaps Richard didn't realise that we
> > already have a pretty good start on an SQL database... hmmm.
> 
> I think RMS's real concern is with complexity and resources, meaning,
> respectively, the work and us.  I'm still of the opinion that he's
> oversimplifying the work, but we agree that what gets done is strictly a
> function of who's willing to do what.  Absent rows, the model is moot.  

:)

Damn. And here I thought the model was almost the job finished!

> Likewise, he who's doing the work gets to decide what form it takes. 
> Unless someone actually working on the data wants to use XML, then it's
> not going to happen.  AFAIK, no actual worker has actually objected to an
> RDBMS, so as far as I'm concerned, that's the only technology we need to
> address.  

Well, there's the small matter of what external formats are in use (PCI
IDs is a free-form text file using whitespace indentation, and the
current printer db AIUI is in XML) - which means import filters as
mentioned elsewhere and export filters too (eg. if we import printer
data, there may happen to be a person or two already depending on the
existing format :)

Again, I don't think these are hard problems.

> > > We have a table for each physical bus type that contains the
> > > properties we can extract from the hardware. 
> > > 
> > >  USBDevice
> > >  IEEE1394Device
> ...
> > James anticipated this in the schema he put together, and created a
> > DeviceConnectors table
> 
> Right.  A device isn't a bus or a connector.  It *has* those things.  Else
> how would one store a printer with a parallel and USB port?  

Well, if they really were devices, we could have a device-device mapping
table. In fact, some printers from memory have "smart card" readers and
cards as "part of them", so this might not be an entirely theoretical
example. But let's wait till we have actual data.

> > > with category 2 data. Each table got FOREIGN keys into the tables from
> > > a. The PRIMARY key is a combination of said FOREIGN keys and other
> > > useful identifying data (such as IEEE1284 ID for printers).
> > 
> > Perhaps this is one of the reasons that IDs instead of (primary fields)
> > are used as primary key - it minimizes the duplication of data.
> 
> At some point, all devices are the same.  When you want to put a Device's
> key as a foreign key in another table -- and that table doesn't care what
> kind of a device it is -- then you need some uniform way to key the
> devices.  The natural key for different kinds of devices differs; it makes
> no sense, obviously, to try to use USB IDs for PCI cards.  So, every
> device gets an ID assigned to it (by the database, btw).  This ID is
> unique across devices: if you have a DeviceID, you can find its type
> because there's no overlap between IDs assigned, say, to printers and
> keyboards.  Neither can any inference about the device be made by
> examining its ID, because it's completely arbitrary, just a number.  
> 
> The natural key's uniqueness ("catagory 1 data", sometimes) should also be
> enforced where applicable.  

Thanks - this is useful to understand.

I know there is at least one ADSL modem on the planet that combines the
following (although it's a single device, not multiple): adsl modem,
ethernet switch, wireless access point, usb interface.

Do you have an example of where a table would want to link (foreign key)
to a completely arbitrary device?

> > > Many values here can be NULL; for Keyboard we probably want an
> > > elements for each keycode for multimedia keys but for my example above
> > > only one of them will contain useful data.
> > 
> > I think that proper normalization gets rid of most NULL values. You
> > simply have these "types" tables for various devices. 
> 
> Normalization gurus like Chris Date are of one mind on NULL: correct
> logical design eliminates them, but feasible physical designs frequently
> require them.  
> 
> A nullable column is just an optional relationship embedded in the table. 
> To eliminate it, one need only place the column in a separate table having
> the same key, and populate that table with only the non-null values. 
> Extant technology won't make you happy when you do that, however,

Why? I'd like to understand this one, if it's easy enough to describe.

>  so we
> use nulls where they "make sense" i.e., where they could obviously be
> removed to another table, but aren't, for reasons of convenience.  
> 
> > > I'm not a big database design myself,

Why David, I can see your normalization and key constraints within your
relational structure... are you so sure you're only a "small" database
design :)

> > If we are to enter data in XML files, we need to just do so, submitting
> > the (XML tagged) data to this list, and then we discuss what fields go
> > where, and start to create some types lists, that we can use for fields,
> > as well as XML tag names. It should be pretty straightforward I imagine.
> 
> I'm on record that XML is harder to generate than flat ASCII files, and

It can certainly be tedious to enter manually.

> that the data model has to be vetted irrespective of whether it's in XML
> or in an RDBMS.

I'm aware of this one, and I agree of course.

I think the issue we have is more one of "let's throw some data in an
email at the list" and then we can discuss it, plug it in, whatever. It
is not necessary to require XML markup for discussion.

However for storage, version control, etc, it should be easy to generate
XML from the db. WRT Till's linuxprinting database, it will make most
sense to auto-import the data, assuming it is already in a reasonably
standard format. As mentioned above though, we must maintain continuity
for existing users of that data by being able to re-generate the
format(s) they are using. So that's kind of like a precondition to
incorporating an external set of data I guess - we must be able to
readily export back to the currently used formats.

For new devices, we can export to whatever format(s) is/ are useful.

> > I feel that my main limitation is my lack of XML experience
> > right now, but I can see the benefits of simple file-based data entry.
> > I'll go do some reading this week.
> 
> I really do think the XML/RDBMS debate is complexity in technology
> clothing.  I would like to see us discussing data -- which will define the
> complexity -- and let that speak for itself.  

I think you're right. XML schema, SQL schema, the data issues will be
the same - either we will address them properly, or we'll address them
properly :)

cheers
zen