Re: [igraph] Saving Graphs

[Marco]

Hey,

that's a good idea.
I'll see if I can, really dumbly, dump a Graph object somewhere inside
the SQLite file using gml or pickle or whatever else.

Thanks!

marco

--
restituiremo il riso ai salici



On Thu, Dec 3, 2009 at 5:15 PM, Matthew Walker
<address@hidden> wrote:
> Marco,
>
> If you are concerned about the performance of re-reading the data, then yes,
> I agree, a compressed XML file might not be your best solution.
>
> Have you looked at SQLite [1]?  It is a library that allows you to read and
> write information to/from a database file using SQL.  This would have no
> problem with such quantities of data, and the read access time would be very
> fast.  I do not know the best way to store your data in such a database, but
> at worst you could store the XML representation.
>
> There exist Python libraries that interface with SQLite.
>
> [1] http://www.sqlite.org/
>
> Good luck!
>
> Matthew
>
> Marco wrote:
>
> Hi Matthew,
>
> thank you for your input!
>
> I forgot to mention a little problem that makes me wonder if this
> approach (which I thought of) is practicable.
>
> The files (of the graphs) I write (in the number of thousands, as I
> said, and totaling various gigabytes for each "species") have then to
> be reread and analyzed several times (by another piece of code).
>
> This makes me wonder if the overhead of compressing/decompressing so
> much data each time is something I want to deal with: I'd say, having tried
> just once, that the amount of time needed to do such
> compression/decompression is longer than the actual time for the
> analysis.
>
> Being this true "always", would  it be worth the effort?
>
> Thank you again!
>
> marco
>
> --
> restituiremo il riso ai salici
>
>
>
> On Thu, Dec 3, 2009 at 4:10 PM, Matthew Walker
> <address@hidden> wrote:
>
>
> Hi Marco,
>
> I am no igraph nor python expert, but maybe you could try writing your
> graphs into one large XML file.  You can easily compress such a file once
> you've finished adding graphs.  I would even suspect that python would have
> the ability to compress the file on the fly.
>
> I think GraphML (an XML format) would be ideal for this.  You could write
> out a graph to a temporary file (using igraph_write_graph_graphml(), or its
> python equivalent) and then merge the temporary file into a main multi-graph
> file (also in GraphML).  Just how to best do this, I am not sure.  I imagine
> python already has strong abilities to manipulate XML files.  If that's not
> for you, then there is a useful command-line utility called xmlstar [1] that
> could help you do such a merge.
>
> I note that igraph already has the ability to select which graph to read
> given such a multi-graph file (see the documentation for the "index"
> parameter for igraph_read_graph_graphml()).
>
> I hope that helps,
>
> [1] http://xmlstar.sourceforge.net/
>
> Cheers,
>
> Matthew
>
> Marco wrote:
>
>
> Hi all!
>
> In the simulations I write I have to save very often the graph, with
> all the attributes.
> If I save it using gml or something like this, I end up with many
> (say: thousands) of very small files, which are not very practical to
> carry around and tend to full the filesystem very quickly.
>
> I am looking for a way to save graphs (even not single graphs, but
> rather more graphs in the same file) which is a little more clever.
>
> So far I have been looking at pickling the whole thing, but I find it
> rather unsatisfactory.
> I have heard of hdf (or something like this). Looks to me that the
> overhead to use it is rather big.
>
> Someone has highlights, remarks, advice?
>
> Thank you in advance,
>
> marco
>
> --
> restituiremo il riso ai salici
>
>
> _______________________________________________
> igraph-help mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/igraph-help
>
>
>
>
> _______________________________________________
> igraph-help mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/igraph-help
>
>
>
> _______________________________________________
> igraph-help mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/igraph-help
>
>
>
> _______________________________________________
> igraph-help mailing list
> address@hidden
> http://lists.nongnu.org/mailman/listinfo/igraph-help
>
>