[Freecats-Dev] Server architecture - issues

[Henri Chorand]

Hi all,

I have begun finding anwsers to the issues I raised in my previous 
e-mail. Once again, I find Python's documentation impressive.

In fact, it seems what we want to implement on the TM server is... a 
socket. You will find a very nice article from Gordon McMillan at:
http://www.amk.ca/python/howto/sockets/

One of the few possible conclusions is that in fact, we don't need any 
HTTP server.

I reckon one should not restrict himself to what a given programming 
language has to offer, but considering my less than expert knowledge of 
Web & client-server programming, at this stage, I feel very relieved 
when I see that a programming language, chosen for its "abstract" 
qualities and after reading lots of comparative reviews, HAS (rather) 
simple answers.

For instance, the way a Python socket deals with requests waiting in an 
input queue with two lines of code seems a good news in itself.

Here is my request to all subscribers who happen to have programming 
experience, could you read the above document and provide feedback?


I also found a nice introduction to regular expressions and sorting, 
which should prove most helpful, at:
http://www.amk.ca/python/howto/.

Lastly, list management in Python seems powerful and simple.


Once I have received your feedback (hopefully), we'll begin coding our 
server module (some time in March). I'll probably try to store the TM 
database directly on the server's file system.

I have a few last performance-related questions for programming experts. 
When I learned programming, there were still a lot of memory size 
constraints. Now, the frontier between RAM and disk storage tends to 
become more subtle, as any modern & powerful OS will:
- load frequently read/written files in cache memory if it has enough 
available
- use swap partition to write data in RAM if it needs more RAM than 
available.

It's easy to understand that, the more internal data & index files can 
be found in RAM instead of hard disks, the faster our server will work.

So, at this stage, I'm tempted to define a number of sequential files in 
which to store index data, and to let the OS optimize filesystem access.

I'm also conscious that I should reduce the number of files used, and 
that all "persistent" data (server's data and indexes) are therefore to 
be stored on disk. So I believe I should, as a rule, treat data handled 
as files and let the OS do what it can to optimize processing speed.

My question is:
Considering the OS will put everything it can in RAM anyway, are there 
general rules that I should follow, concerning work with disk files 
and/or data objects in RAM?

I'm also conscious that, in order to "harden" server operation, one 
should try to take care of unexpected power-down/hard reset and 
automatic recovery. At this stage, I'm tempted to leave this for later on.

Please let me know your thoughts.

Also, please let me know who would be available to help coding our TM 
server.


Regards,

Henri