Re: [GNUnet-developers] Proposal: Make GNUnet Great Again?

[Christian Grothoff]

Hi Martin,

Let's go over the subsystems in your list specifically:

- Secushare + social + psyc + psycstore + multicast
  IMO these should go together.  Nothing but
  Secushare uses the other 4 yet, and they are all
  similarly experimental.  As even the SecuShare team
  seems unsure about the code base's viability, splitting
  this off into its own Git makes some sense to me.

- voting + secretsharing: voting is already in gnunet-java
  and separate because the Java build system is completely
  different; secretsharing is overall pretty generic
  and dependency-wise unproblematic. I really don't see
  a good reason for pulling it out of core. If we did,
  rps, scalarproduct are in the same category (basically
  cryptographic building blocks that are not yet in real
  use). But for me, part of the GNUnet approach is that we
  need to raise the abstraction of the networking API and
  provide better cryptographic building blocks at the heart
  of our infrastructure.

- Conversation + speaker + microphone / fs (+datastore)
  How do you propose to build gnunet-gtk with this being
  moved out of core?
  Do you really want anyone working on this to first
  build gnunet-core, then gnunet-conversation, then
  gnunet-gtk-core, and then gnunet-conversation-gtk?
  If we merged gnunet-gtk-core into gnunet-core, that
  would at least be semi-sane, but as proposed, this seems
  just insane.

- pt (not 100% sure what it does maybe it can stay in code
  depending on how niche the API is)
  Well, PT (protocol translation, IPv4/IPv6) works with
  DNS, VPN and EXIT to translate DNS replies (think DNS6to4).
  As GNS needs VPN for some record types, it is not so trivial
  to unbundle VPN/EXIT/PT from core.
  This is a good example why it is not easy to unbundle.

- Gtk UI (already in separate repo)
  If we atomize gnunet.git as you propose, either the gnunet-gtk
  configure.ac must increase drastically in terms of complexity
  to test which of the gnunet-atoms is available (welcome to 500
  character configure invocations) or we atomize this as well,
  which then means you can write an novel for the
  installation-from-source handbook on the dependency graph.

- rest
  This confuses me the most of your proposal. We have json/,
  which requires libjansson, not just rest. And there are DB
  adapters (pq/my/sq) which also can (or maybe ought to) handle
  JSON data.
  More importantly, IMO the various rest-plugins probably
  belong with their respective subsystems, and
  some of those are in core, like for IDENTITY, the DHT or GNS.
  So if REST isn't in core, then we get build hierarchies like this:

  gnunet-core (with GNS)
    gnunet-gtk
    gnunet-rest
      gnunet-dht-rest
      gnunet-identity-rest
    gnunet-conversation
      gnunet-conversation-rest
      gnunet-conversation-gtk
      gnunet-fs-rest
      gnunet-fs-gtk

  Is this really what you believe will attract new developers!?
  Dozens of repositories with crazy dependency chains? I also do
  not quite see whatever you want to do with the GitLab CI that
  could not properly handle the bigger repo we have today
  (especially given that Buildbot is today fine with it)


- reclaim (not in that architecture figure because its outdated)
  --- some of it is already separate, right? Still, I don't quite
  see why you want to split it off. The result would just be that
  when I change stuff in GNS and break reclaim, I couldn't even see
  I broke reclaim and you would get to pick up my mess. Ok, you
  sometimes get to do that anyway, but I predict this would increase
  the frequency ;-).
  So here, I would rather see libgabe being dragged *into* our
  build (src/gabe/) than reclaim being pushed out.  After all,
  ABE might be useful for other things in the future (there is
  a feature proposal for Taler related to ABE, for example).


So from this list, the one thing that today for me would make sense to
move to another repo is the Secushare family. In terms of sanity when
building from source (and that *is* what developers do) I'd rather
integrate gabe than split off reclaim.

Note again that this has nothing to do with the binary package
structure, this is just for sanity when building from source. The
proposed structure that might end up with 50+ interdependent Gits would
drive *me* bananas, and I can't see it being helpful for downstream either.

But most importantly, I simply don't share your aversion to *optional*
dependencies (libgabe, libgstreamer, libpq, libmysql, libjansson). None
of these really justifies a change in our *Git* structure to me.


For me, any Git structure imposes two main constraints: (1) access
control (who has commit rights) --- and there we have so far never even
considered giving some devs access to some repos but not others. (2) is
that the resulting TGZs usually are 1:1 per source Git. Here, we DO have
some people building from source (downstream, students, developers, me),
and each additional TGZ/repo they need to download, configure and build
takes time without being inherently helpful for anything.

Sure, if libgabe was NOT an optional dependency and I had to install it
first, that would be a pain if I don't care about reclaim, but as long
as it is optional, I really don't see any pain for users to have reclaim
in the main tree. OTOH, it will take longer to install from source if
reclaim is a separate repo -- and it'll be easier to install reclaim
from source if libgabe is integrated.


But central to the entire argument seems to be that for CI, I don't at
all see the benefit of more repos either. It just means that the CI
configuration becomes also more complex, as I now have to teach the CI
the repo and dependency structure.  Sure, we get in theory a
combinatorial explosion of feature on/off states, but in practice
building once with Postgres and once without Postgres doesn't mean we
have to also try this with all _other_ possible on/off switches as the
features are usually sufficiently orthogonal.


Anyway, I should get back to actual coding and cleaning, maybe we should
resume this discussion at the GNUnet Hacker Meeting? At least I suspect
that might be more productive and I don't see any urgency here.


Happy hacking!

Christian


On 2/9/19 11:16 AM, Schanzenbach, Martin wrote:
> Well I proposed that we look at https://stage.gnunet.org/en/architecture.html 
> (which is not up to date currently).
> IMO you can quickly identify which components are basically just applications 
> that do not provide fundamental functionality (ie. what I referred to as 
> "gnunet core"-- Note: this has nothing to do with the GNUnet core service).
> 
> Example:
> 
> - Secushare + social
> - Conversation + speaker + microphone
> - psyc + psycstore
> - voting + secretsharing
> - fs (+datastore)
> - pt (not 100% sure what it does maybe it can stay in code depending on how 
> niche the API is)
> - reclaim (not in that architecture figure because its outdated)
> - rest
> - Gtk UI (already in separate repo)
> 
> I would argue that each of the above _could_ go into a separate repo for 2 
> major reasons:
> 
> 1. In the overwhelming case, they do not provide core functions to other 
> services/apps (they do not have "incoming" arrows in the figure)
> 2. Often, they pull in unreasonable dependencies not required by any other 
> component (rest: jansson, reclaim: libgabe, conversation: gstreamer etc)
> 
> As soon as a component does not fit into any of the above categories, it 
> could be merged into the "core" repo again.
> 
> BR
> 
>> On 8. Feb 2019, at 15:53, t3sserakt <address@hidden> wrote:
>>
>> Hey *,
>>
>> I also think it is better to have several repos. I can not tell how to split 
>> up the gnunet.git repo, but we should not merge gnunet-gtk.git into 
>> gnunet.git.
>>
>> cheers
>>
>> t3sserakt
>>
>> On 08.02.19 15:00, Schanzenbach, Martin wrote:
>>> Yes, I do not think this is a good idea at all and is contrary to the 
>>> initial motivation of this thread.
>>>
>>> We already agree the from a user perspective, the packages (.deb/.rpm et 
>>> al) should ideally be split into
>>> the respective services/applications and, of course, also Gtk+. For sane 
>>> dependency resolution at least.
>>>
>>> But it is also reasonable to separate things at source level as I already 
>>> gave various reasons, to which I have not heard a counterargument yet 
>>> except:
>>> Usability (???).
>>> You cannot argue with usability because USERS DO NOT INSTALL FROM THE GIT 
>>> REPO THEY INSTALL PACKAGES.
>>> And even the packages should be separate as you already agreed!
>>>
>>> A monolith _will_ bite us when it comes to testing and CI.
>>> Working on a single, huge codebase with a variety of build switches is a 
>>> pain for testing, development and deployment.
>>> Not to mention it is difficult to ascertain and ensure for an application 
>>> what components are built.
>>> Example: Do you really want to test everthing of the core gnunet functions 
>>> if a Gtk widget changes?
>>> Because that will inevitably happen.
>>> It will be really difficult to setup a CI/automated testing that correctly 
>>> separates this.
>>> It will be possible, maybe, but then we have a test process that is equally 
>>> difficult as our build process.
>>>
>>>
>>>
>>>> On 8. Feb 2019, at 14:39, Christian Grothoff <address@hidden>
>>>>  wrote:
>>>>
>>>> On 2/7/19 3:21 PM, Hartmut Goebel wrote:
>>>>
>>>>> Am 02.02.19 um 16:09 schrieb Christian Grothoff:
>>>>>
>>>>>> And I wonder if it wouldn't make sense to have the gnunet.git
>>>>>> configure.ac test for Gtk+ and *if* libgtk is detected, _then_ build Gtk
>>>>>> GUIs that are _included_ in gnunet.git, instead of requiring the user to
>>>>>> download and configure yet another TGZ.
>>>>>>
>>>>> *If* the gui is merged into the main repo, I suggest adding
>>>>> configure-options like `--without-gui`(which AFAIK is a autotools
>>>>> standard thing) to avoid building the gui even if libgtk is detected.
>>>>> This might happen if e.g. one is developing on her/his desktop.
>>>>>
>>>> Sure, that makes sense. Any opinions from the silent masses on merging
>>>> gnunet-gtk.git into gnunet.git and merging the source TGZs?
>>>>
>>>>
>>>> _______________________________________________
>>>> GNUnet-developers mailing list
>>>>
>>>> address@hidden
>>>> https://lists.gnu.org/mailman/listinfo/gnunet-developers
>>>
>>>
>>> _______________________________________________
>>> GNUnet-developers mailing list
>>>
>>> address@hidden
>>> https://lists.gnu.org/mailman/listinfo/gnunet-developers
> 
> 
> _______________________________________________
> GNUnet-developers mailing list
> address@hidden
> https://lists.gnu.org/mailman/listinfo/gnunet-developers
>

signature.asc
Description: OpenPGP digital signature