Re: GeoIP database redistribution?

[ng0]

Efraim Flashner <address@hidden> writes:

> On Tue, Jan 24, 2017 at 06:06:21AM +0000, Pjotr Prins wrote:
>> On Mon, Jan 23, 2017 at 07:17:12PM +0100, Marius Bakke wrote:
>> > ng0 <address@hidden> writes:
>> > 
>> > > I want to slowly package OONI (https://ooni.torproject.org/).
>> > > One of its dependencies, txtorcon, requires python-geoip which
>> > > depends on geoip-c-api. I've got both covered, but both of them
>> > > want (either to download or to be present) for tests (a/the)
>> > > legacy database file of maxmind.
>> > >
>> > > Question 1: Can we distribute the database in a source? I can't
>> > > access the homepage of maxmind for cloudflare reasons.
>> > 
>> > The database is distributed freely under cc-by-sa4.0:
>> > 
>> > https://dev.maxmind.com/geoip/legacy/geolite/#License
>> > 
>> > So packaging it should be fine. :)
>> 
>> This actually raises the issue of packaging large data files (we are
>> getting into TB's). Could there be a way Guix fetches external
>> datasets as part of the distribution? I think that if it is not
>> executable code and SHA values/pfff values match it would be safe to
>> do.
>> 
>
> The other thing is that guix downloads the files into ram and then
> writes them into the store, which can be a problem if the source is
> larger than the available memory.
>
> -- 
> Efraim Flashner   <address@hidden>   אפרים פלשנר
> GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
> Confidentiality cannot be guaranteed on emails sent or received unencrypted

TB? Oh. Due to cloudflare I had no clue how big this is. In this
case, for the current state of the way we distribute packages is
not itself using a distributed network (if just keeping the space
on hydra is the problem?), I would suggest that I package the
updater I mentioned and mention it in the description for other
MaxMind software.

This database is also an optional dependency for tor and not just
ooni if I understand it correctly, so even if it's not 100%
accurate (their commercial solution claims to be more accurate)
it would be nice to find a way to include it.
But I think size is really an issue, for example Gentoo dropped
the database from their source distribution network and points
users to the updater I mentioned (that's how I learned about the
updater).
-- 
♥Ⓐ  ng0 -- https://www.inventati.org/patternsinthechaos/