[GNUnet-SVN] r21095 - gnunet-java

[gnunet]

Author: grothoff
Date: 2012-04-23 09:03:35 +0200 (Mon, 23 Apr 2012)
New Revision: 21095

Added:
   gnunet-java/ISSUES
Log:
-merge

Copied: gnunet-java/ISSUES (from rev 21039, gnunet-java/ISSUES)
===================================================================
--- gnunet-java/ISSUES                          (rev 0)
+++ gnunet-java/ISSUES  2012-04-23 07:03:35 UTC (rev 21095)
@@ -0,0 +1,521 @@
+* what about licensing of libraries in general?
+  => GPLv3+ or compatible (d.h. GPLv2+, LGPLv2+, BSD, Apache License 2.0+)
+     http://www.gnu.org/licenses/quick-guide-gplv3.html
+  
+* java has no native unix domain sockets (=>use JNI library, licensing issues?)
+
+* package structure, inner classes vs. packages
+  
+
+* naming conventions: stick with java standards? YES.
+* ugly cross-reference to TransmitReadyNotify.java in 
org.gnunet.util.client.Client => util/TransmitReadyNotify interface
+* Client/Connection class hierarchy => avoid deep hierarchies!
+* how to handle disconnect/destroy (as java cannot directly dispose of 
objects) => finalizers
+* why is configuration passed around so often? Because we may have multiple!
+* wrt the scheduler: can you select on anything else than a Connection? pipes, 
sockets (tcp/udp), files, timeout
+
+
+* logging / assertions (use guava?) => slf4j
+* unit testing (JUnit vs NGUnit) => using JUnit4
+
+* Configuration.java:
+  * what about spaces in filenames? quotes in config values?
+  * formal spec of syntax?
+
+* error handling: what to do with io errors in some cases?
+
+* process priority
+
+* signals:
+  http://www.ibm.com/developerworks/java/library/i-signalhandling/
+
+* can multiple tasks wait for the same channel? (problem with nio api)
+
+* exception hierarchy in Construct
+
+
+
+
+* why do we even need transmit_ready_notify? can't we just queue the message?
+* do we need to be able to send stuff that is not a message?
+
+* why do we have tasks *and* callbacks?
+
+
+(* RelativeTime/AbsoluteTime: are wrappers are really necessary? do we need to 
distinguish
+  relative time from absolute time statically? most java APIs just use long)
+
+* should MessageHandler be a runabout? => yes
+
+
+
+
+(* idea: gnunet-java uses a lot of reflection, should we use apt (annotation 
processing too
+  to do some basic static checks for runabouts and maybe construct 
annotations?)
+  
+ 
+ 
+ 
+== Meeting: Thu Dec 15 ==
+ 
+ * big problem: no unix domain socket implementation supports select
+ * possible solutions:
+  * write our own implementation
+   * hard/impossible to integrate with the java api for sockets/selectors
+   * => use one thread that selects on all UDSs and writes to a pipe
+  * don't use unix domain sockets
+  * use threads to receive from UDSs
+  * write java daemon to multiplex UDSs over one TCP socket
+  
+  
+ * IDE: switching to IntelliJ community edition (available under Apache 2 
license)
+  
+  
+ * project structure
+  * where to put annotation processors?
+  * should work with ant and eclipse
+  * dependencies: ann.proc <-> annotation classes
+  
+
+
+====
+
+* (undocumented) issues with the Runabout: visit methods cannot access private 
properties/classes that are
+  accessible from the scope of the visit method
+  java.lang.IllegalAccessError: tried to access class 
org.gnunet.construct.Construct$ParserGenerator from class 
org.grothoff.GenCode00000003
+  when Construct.ParserGenerator is declared private
+
+
+* Runabout.java line 373: string comparison with "==" instead of .equals
+ * Does this rely on string interning?
+
+
+* buffer sizes / buffer allocation / buffer management  in general
+
+* Should Construct switch to ByteBuffer?
+ * otherwise we may get performance trouble with direct buffers?
+
+* why is there no receive_cancel in client.c (but in connection.c)
+
+* what about the timeout in all the transmit calls: what is the timeout 
relative to? call of the function?
+  the beginning of trying to select() to send?
+
+
+
+(* notifyTransmitReady: how often do we know the exact size of the message we 
want to send, even if the message
+  has not been constructed yet?)
+
+* when do we issue a transmit but cancel it (except in shutdown)
+
+
+=====
+
+
+* how do we handle RelativeTime(0) timeouts?
+
+ * header redundant -> introduce NetworkMessage with header automatically 
prepended?
+ * static distinction between Messages with and without header
+
+ * support for minimum message size in construct?
+
+* what about invalidating handles (e.g. TransmitHandle) after completion of 
the request?
+
+
+ * buffering scheme / when is "enough space" in a buffer?
+  * where to put messages in a buffer? at the beginning? at (capacity - size)?
+
+
+
+* setAccessible in the runabout?
+
+* what about autoRetry in the middle of sending a message?
+
+
+=====
+
+Fixes:
+ * MessageId generation now works with nested classes
+ * Construct can now handle message inheritance correctly
+ * loading of default configuration (room left for improving path search)
+ * frameOffset now handled correctly in ByteFillParser
+
+Changes:
+ * @construct.String can now also be of null / 0-length if the maybe-parameter 
is set to true
+ * @construct.Nested can be marked as optional: the target object is set to 
null when there is no data left in the message
+   * (does this make optional string length obsolete?)
+ * Parser.parse now takes a (Message frameObj) parameter, making it possible 
to refer to the size field
+   from a nested object
+
+
+Questions:
+ * problem with annotation-based parameter parsing: anonymous classes are 
private (fields not accessible),
+   parameter fields have to be public, too => just make them public / don't 
use anonymous classes   
+   => currently solved with setAccessible GOOD, KEEP
+ * what about unions in messages (example: ResolverRequest can have String, 
socket address or nothing as payload)
+  * trivial solution: just use a byte array in this case
+  * @Maybe annotation (for resolver response message)
+    @Maybe @ZeroTerminatedString String str allows str to be null if there is 
no space left
+  * @Union(tag="someField", keys={1, 2}, types={MessageTypeOne.class, 
MessageTypeTwo.class})
+  * more dynamic variants possible
+  * could be used to implement MessageHeader:
+   * class GnunetMessage {
+       @Nested
+       MessageHeader header;
+       @DynUnion(tag="header.type", map=MessageLoader.class)
+       Message body;
+     }
+   * MessageSink would add the header to the body, filling out the type and 
size fields
+   * Message declarations would look like this:
+     @GnunetMessage(type=321)
+     class NSEUpdateMessage extends Message {
+       @Uint32 reserved; @Nested RelativeTimeMsg ts; @Double double est; 
@Double double dev;
+     }
+
+
+
+================================================
+
+* AF_* etc. seem to be system specific
+* inconsistent message formats
+
+================================================
+
+* Should Connection connect in the Constructor or in a seperate connect method?
+  =>constructor
+
+* choice of coverage tool:
+ * cobertura sucks (bad documentation, non-existing error messages (e.g. when 
not passing the coverage data file to the jvm)
+  * results are poor / just doesn't seem to work correctly
+    =>seems fine now
+  * integration into the project structure was frustrating
+    =>but it is done
+ * EMMA: isn't maintained anymore, can't do branch coverage
+ * JaCoCo is very new / mostly undocumented / mostly an (still unstable) API
+  * uses agents 
(http://docs.oracle.com/javase/6/docs/api/java/lang/instrument/package-summary.html)
+    instead of compile-time instrumentation
+ * CodeCover (http://codecover.org)
+  * looks promising
+    => keep in mind if real trouble arises
+
+* bash scripts (for collecting source/class files, building, testing, 
coverage, annotation-processing etc.)
+  are non-portable
+  => sure.
+ * different compiler options (e.g. -g is needed for coverage)
+  => sure; add 2nd script for "build-jar" without "-g" and third script 
"build-jar-dbg" for jar + -g
+ * use some java-based scripting language? (e.g. clojure)
+  => no
+
+* how do we collect test cases
+ * test suites
+    => in the long-term, yes, test suites "fast run", plus "expensive tests" 
separately
+ * custom annotation processor (collects all classes with @Test-annotated 
methods)
+    => maybe in the long term, not urgent ;-)
+ * by filename convention
+    => also, always
+    
+
+
+
+================================================
+
+ * what are use cases for the scheduler's priorities?
+ * should shutdown really be a reason, not a separate callback-method in Task?
+  * some tasks re-schedule themselves, it's very easy to forget checking the 
shutdown flag in the context
+ * how should a disconnect from the service be handeled?
+  * probably there should be a callback for this, too
+  * service clients should try to re-connect
+ * should we introduce a general interface "Cancelable" for requests?
+ * what is the persistence in statistics? (esp. in the watch message)
+ * general api question: should callbacks have their corresponding request 
handler passed?
+
+
+
+ ====================================================
+
+ * how much stuff related to gnunet (and not just gnunet-java) should the 
tutorial cover?
+   
+
+ * regarding IzPack:
+  * are there really any advantages over just shipping a zip-file?
+  * no way to extend the $PATH, local ~/bin/ directory does not exist on e.g. 
ubuntu
+
+ * regarding jar-files:
+  * unforfunately recursive jar files are not supported by java
+  * => the jar file always has to ship with the lib-directory
+  * an alternative: copying all class files of gnunet-java's dependencies to 
the build directory
+
+ * developing extensions to gnunet-java:
+  * provide a way to generate a skeleton for an extension with has the 
gnunet-java.jar in its classpath
+  * provide scripts to generate msgtypes etc.
+
+ * alternative for dependency management: use some kind of repository-based 
dependency management,
+   e.g. maven, gradle, builr
+
+
+ * where should stuff like PeerIdentity and HashCode/GnunetHash go?
+
+ * should there be a more elegant / convenient way to handle Enums/EnumSets?
+  * is the use of enums desireable anyway? (java enums are not very flexible / 
you have to implement
+    a construtor everytime you want to use custom values)
+  * a simple way to do this would be to allow enums for @Integer-annotated 
fields
+   * this would only allow us to access the ordinal of the enum value
+   * otherwise every enum would have to implement an interface like IntEnum 
(enums in java cannot use extends)
+
+
+ * what is the purpose of the GNUNET_BLOCK_evaluate function?
+  * does is just check the validity of blocks / block requests?
+ * who is responsible for processing a query for a specific block type?
+ * what is a HELLO, what is a transport?
+
+
+proposal:
+
+change
+
address@hidden(SomeMessage.MSG_ID)
+public class SomeMessage extends Message {
+  public static final int MSG_ID = 12345;
+
+  @Nested
+  MessageHeader header;
+  @Uint16
+  int someValue
+  @Nested
+  SomeOtherMessage msg;
+}
+
+to
+
address@hidden(12345)
+public class SomeMessage extends GnunetMessageBody {
+    @Uint16
+    int someValue
+    @Nested
+    SomeOtherMessage msg;
+}
+
+Advantages:
+ * the user doesn't have to take care of the message header (the message 
header is redundant anyways)
+  * no need to define the MSG_ID constant in every Message
+ * no extra code for MessageId, simpler annotation processing, simpler 
MsgMap.txt
+ * with the current implementation it is not trivial to insert the message 
header automatically,
+   Union/UnionCase already implements this.
+
+
+public class GnunetMessage implements Message {
+    MessageHeader header;
+    @Union(tag = "header.msgType")
+    GnunetMessageBody body;
+
+}
+
+public interface GnunetMessageBody extends MessageUnion { }
+
+
+
+
+------------------------------------------------------------------------
+
+
+
+* DHT:
+ * gnunet-service-dht -h produces segfault
+   -> not anymore
+ * block type "any" does not work on request (gnunet-dht-get always changes 
any to test)
+   -> yes, any indicates default here ;-)
+ * why is there no ack from the dht that a put request has been received?
+ * how unique is the UID? globally? per connection?
+  * by looking at the sources: (client,uid) identifies a request unambiguously
+ * what is a "dealy do"? ;) (see dht.h)
+ * why does the ClientGetStopMessage include the Hash? Isn't the 64-bit UID 
sufficient?
+
+* GetOpt parser: currently no way of determining whether an option has been 
set or is just on its default
+
+
+* Core:
+ * is the array of MessageHandlers really a good API? shouldn't we use the 
runabout
+   and specify a set of GnunetMessage.Bodys that we are interested in?
+ * SendMessageRequest.queue_size?
+ * SendMessage: comment on peer field ambiguous
+ * why does SendMessageReady contain the peer?
+ * advantages / disadvantages of queueing messages (in client and core)
+
+* Server/Service API
+ *
+
+* should I check in stuff like the installer binary etc.?
+
+
+========================================================================================
+
+
+* message dispatching with runabout: what can go wrong with inheritance and 
how do we solve it
+ * an idea: inspect what messages a specific runabout can handle, record their 
IDs,
+   match IDs on message receive
+
+
+
+* are we in a hurry? when should gnunet-java be usable to students? --- 23.4.
+ * according to the website the project begins very soon
+* currently my progress with the tutorial is not that far, as the java APIs 
are not
+  yet completely implemented / stable.
+* needed: a place to host the javadocs (so the tutorial can link to it)  -> 
address@hidden
+
+* what is the best way to configure the classpath for gnunet-java-ext?
+ * a bash script that exports environment variables for e.g. the 
gnunet-java.jar?
+* once a MessageMap.txt has been generated for a new extension, how do we tell 
gnunet-java that/where it
+  exists?
+* what about allocation of message IDs for extesions / what about clashes?
+
+* core_api.c, notify_transmit_ready:
+  * pr = GNUNET_CONTAINER_multihashmap_get (handle->peers, 
&target->hashPubKey);
+  * the above line crashes when target==NULL, but comment indicates that NULL 
is valid for target
+   => bad comment, fixed
+
+* ...NOTIFY_PRE_CONNECT.. message type not used => DCE
+
+
+core:
+* we seem to get all incoming messages if the handlers array is empty (even if 
we don't request anything)
+  => correct
+* being notified about outbound messages does not seem to work at all
+  => bug
+ * comment in core_connect talks about timout, but there is no timeout 
parameter.
+ * GNUNET_MESSAGE_TYPE_CORE_NOTIFY_STATUS_CHANGE not used
+  * also GNUNET_CORE_OPTION_SEND_STATUS_CHANGE
+ * inconsistency in structs with trailing ATS information 
(NotifyTrafficMessage vs ConnectNotifyMessage)
+  * why zero-termination *and* size field? => legacy, we want NO zero 
termination anymore => file Mantis reports
+  * what does zero-termination really mean? is it only one byte? a whole ats 
entry? (double-zero)
+ * what happens when we can't connect to core / lose connection? in the c api 
implementation, this is never handeled?
+   -> auto-retry
+  * why is there even the init callback if init can't fail? => get PID
+
+* are there really any ambiguities when receiving messages with a runabout?
+
+
+
+* how does the Hash ascii-encoding work?
+* => solved, ported C-code, encountered some problems because of the lack of 
unsigned in java
+  (had to use long instead of int in java code because of sign) => fun
+* first implementation got some characters wrong, had to use
+  byte b = data[(int) rpos++];
+  int s = b >= 0 ? b : (256 + b);
+  to compensate for the signed-ness of byte
+* is the current implementation correct now? (it has the same output as 
peerinfo)
+  => most likely.  Compare with gnunet-rsa!
+
address@hidden
+UInt32Message[] foo;
+
+vs.
+
address@hidden
+int[] foo;
+
+vs.
+
+// Fill would then work with number annotations and @Nested (default)
address@hidden @UInt32
+int[] foo;
+
+YES, like this (ideally):
+
address@hidden @
+int foo_length;
+
address@hidden @ArrayLength(length="foo_length")
+int[] foo;
+
address@hidden @Fill
+int[] bar;
+
+
+
+* server:
+ * what is a MessageStreamTokenizer and why does it belong to the server?
+  * my guess: for fragmented messages inside other messages?
+  * GNUNET_SERVER_connect_socket: what is it for?
+  * how to disable corking in java? is corking the same as TCP_NODELAY?
+
+
+* discuss: the size parameter in the various notifyTransmitReady methods
+ * annoying to use from the java perspective
+ * proposal: keep for c api compatibility, introduce additional version that
+   gets a message object directly, and calls a continuation to notify about 
success/failure
+
+
+
+* server:
+ * what is client_keep?
+ * when handling signals through the pipe, how is the service / server 
notified?
+
+
+---------------------------------------------------------------------------------------
+
+* regarding message maps of extensions: isn't it a better solution to just
+  search the classpath (with 
getResources("org/gnunet/construct/MessageMap.txt")), or am I missing
+  something?
+  * this approach works, just implemented. one can now add message maps by 
adding them to classpath
+
+* questions on https://gnunet.org/bugs/view.php?id=2277
+
+
+
+* if bufferbloat is bad, why do we keep reimplementing action queues / buffers 
and in almost every
+  service api, just not on the lowest level (client)? (c.f. the ControlMessage 
queue)
+
+* Construct: due to some changes in construct (esp. the addition of unions and 
optional nested messages)
+  parser caching does not work correctly anymore, since parsers have now 
context, and not just a
+  target class.
+
+
+* DHT:
+ * monitor: is my observation correct that MonitorMessage is used to report a 
change back to the
+   client *and* to initiate a request(with some fields left at 0)? what are 
the different types of monitor requests?
+ * monitor_stop is wrong / doesn't do anything to cancel the monitor request
+  * it's also used nowhere in the entire code
+
+* Scheduler (had to be touched to implement the server)
+ * where do we need prerequisite tasks?
+  * according to the source, nowhere (except two times for memory deallocation)
+ * is there ever a case where a task should depend on events on a set of FDs?
+  * the can be emulated by multiple tasks
+ * do two tasks ever wait for the same event on the same FD?
+  * wouldn't this indicate a bug most of the time?
+ * currently the above to things are implemented, but complicate/slow down the 
implementation
+
+* slight design problem with PeerIdentity: data should be private final, but 
is public for Construct
+ * gnunet-java doesn't really care about access modifiers anyway, see runabout 
;-)
+
+* I'm still not entirely convinced that core connect needs no timout
+ * what about command line tools?
+ * they'd need an extra timeout task
+
+
+* core
+ * why is priority/timeout in SendMessageRequest and SendMessage?
+ * there is no comment indicating that SendMessage has the message to send as 
trailing data
+ * is 0 the highest or lowest priority? in what range are its values typically?
+ * when can we cancel a transmit?
+ * testing/test_testing_large_topology.c calls CORE_notify_transmit_ready with 
wrong number of arguments
+  (found by greping)
+ * PeerStatusNotify message unused
+
+
+* java reflection APIs have the worst error messages (and checked exceptions),
+  Construct.ReflectUtil now provides better error messages / exceptions
+
+* is it legal to write multiple messages in the callback to 
Core.notifyTransmitReady?
+ * currently this is illegal in gnunet-java
+
+* --with-sqlite=PFX       base of SQLite installation
+ * misleading/confusing
+
+
+* server:
+ * is there an elegent way to reuse code from Client.java in 
Server.ClientHandle?
+ * GNUNET_SERVER_receive_done (struct GNUNET_SERVER_Client *client, int 
success)
+  * comment about success wrong
+ * why is there the client_disconnect function, and receive_done(client, 
false)?
+  * convenience/performance
+ * comment on client_persist
\ No newline at end of file