|
From: | Tarmo Toikkanen |
Subject: | Re: [Fle3-dev] Import performance |
Date: | Fri Jun 14 04:18:01 2002 |
I'm in the process of importing a fle3 system to a newer version. The dump is not huge (6 MB), and it has already been at it for some 20 minutes. The machine I'm doing this on is a p3-933MHz. Now consider we have a fle3 system that has a thousand user, and a million messages. Importing this will take days on even the fastest computer in the world. Does anyone have insight on why ImportExport manages to spend such ridiculous amounts of time importing stuff?
Importing data to a new Fle3 is done one object at a time. And creating each note takes time, as we currently create over a dozen DTML objects for each Note object.
I agree that this is much too slow, but I'm not sure there's much we can do about it, except perhaps restructure the entire knowledge building so that each note doesn't need its own dtml files. This would definitely be a change for the better, but it might prove difficult...
I tried this spike once at the beginning of the year, and ran into problems... If you replace the dtml files with single copies up in the acquisition tree, everything works as long as you have different names for all dtml files in the tree (notes, contexts, courses). This means that index_html cannot be handled like this. And if we use a python method as a mask, we run into weird problems with dtml files calling other dtml files. When the chain gets longer than 2 or 3 calls, the context object gets dropped in certain cases and things break down.
But apparently we should try to do this again. Of course, if anyone has any suggestions...
-- Tarmo Toikkanen http://www.iki.fi/tarmo/ The Fle3 Team http://fle3.uiah.fi
[Prev in Thread] | Current Thread | [Next in Thread] |