Re: [Savannah-hackers-public] bzr access problems

[Tim Cross]

On Tue, Oct 18, 2011 at 10:56 AM, Martin Pool <address@hidden> wrote:
> On 18 October 2011 10:34, Karl Berry <address@hidden> wrote:
>>    Also, I remind you that there's evidently another (or the same)
>>    problem on Savannah that drops the connection after 1 hour.
>>
>> Due to runaway processes killing the system, we instituted that
>> hardwired timeout some time ago:
>>
>> exec timeout 60m /usr/bin/bzr "$@"
>>
>> I just increased it to 120m.
>>
>> I know of no feasible way to determine whether the connection is
>> actually active.  I can imagine various infeasible ways, but I think
>> they would all cause more trouble than they would save.  I hope it is
>> merely a matter of rerunning the command that got killed off at the
>> magic moment, so not losing significant amounts of work.
>
> bzr 2.5b2 has a builtin timeout to drop idle connections
> <http://pad.lv/824797>.  I don't know if that would fix the "go crazy"
> case.  If you see that, please file a bug with a traceback and/or
> strace.
>
> m
>

My issue is resolved (sort of).

It turns out the problem was indeed specific to the bzr protocol and
my work network.

Our network firewall use deep packet inspection (DPI) to identify the
protocl and then decide whether to let it through or not. It seems
that somewhere between the 4/10/2011 and 10/10/2011, something
changed. There were no changes in our firewall's DPI signatures over
this period (there was a change on the 11/10/2011), but this is after
I started experiencing the issue. Initially, the problem showed up
with v2.3.4, but also existed with version 2.4.1.

Anyway, what we have done is capture the protocol signature in a dump
file which is being sent off to the vendor, who should update their
signatures to recognize the bzr protocol. This should fix the issue
for me and possibly others who may have hardware from the same vendor.

For reference, one of the symptoms to watch out for with this sort of
problem is that initially it looks like you have established the
remote connection and data begins to flow. THen you get a message
along the lines that the connection has been shutdown/terminated by
the remote server. This can give the impression it is the remote
server which is killing the connection when in fact, it is the local
firewall.

The firewall initially allows the connection and starts to perform
analysis of the data packets. Depending on the configuration, if it is
unable to classify the packets and/or identify the protocol, it will
drop the connection. Since this is done after the establishment of the
remote connection and after data has begun to flow, it gives the
impression the issue is a remote one when in fact it is a local config
issue. Much harder to diagnose than the old port blocking approach to
firewalls. At least with the old style, either you could connect or
you couldn't and where the problem originated was fairly clear. The
other advantage was that it kept a network technician employed
maintaining complex ACLs for various hosts and ports - now its all
supposed to be auto-magical!

Tim



-- 
Tim Cross