gluster-devel
[Top][All Lists]
Advanced
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Gluster-devel] Mounting with file servers is failing very frequentl

From: Joseph Job
Subject: Re: [Gluster-devel] Mounting with file servers is failing very frequently (in every day)
Date: Wed, 31 Oct 2007 20:14:07 +0530 
Hi Raghavendra,
I have commented the transport-timeout, and remounted it again. After working for some time the mounting failed again...see the logs in the client side...any help in this regard would be highly appreciated...
2007-10-31 10:26:20 W [client-protocol.c:210:call_bail] client_213: activating b ail-out. pending frames = 1. last sent = 2007-10-31 10:22:58. last received = 20 07-10-31 10:23:22 transport-timeout = 108 2007-10-31 10:26:20 C [client-protocol.c:218:call_bail] client_213: bailing tran sport 2007-10-31 10:26:20 D [tcp.c:131:cont_hand] tcp: forcing poll/read/write to brea k on blocked socket (if any) 


Regarding the type of files, all files with average of 2 to 4 mb each.

But there are around  2 to 3 lakhs in every folder...around 20 folders...

The servers are connected together with network bonding.

2Gig connection.

Thanks,
Joseph




At 05:07 PM 10/30/2007, you wrote:


Hi Joseph,
transport-timeout in clients is very less in your configuration (4 seconds). Use a higher timeout or just comment out the option to use the default one, which should be sufficient. 

regards,
On Oct 30, 2007 3:16 PM, Joseph Job <<mailto:address@hidden>address@hidden> wrote: 
My Setup details..

Operating System : Trustix Secure Linux release 3.0.5 (Mirch Masala)
Kernel Version : 2.6.19.7-1

Gluster version :
glusterfs-server-1.3.6-1
glusterfs-client-1.3.6-1
glusterfs-common-1.3.6-1
glusterfs-devel-1.3.6-1

Working Mode.

Two web servers are there and both web servers are using Gluster file
server in TCP network for file access.

see my server side configuration...



Server 1

## Define the stroage
volume fs3-storage
 type storage/posix                   # POSIX FS translator
 option directory /storage            # Export this directory
end-volume

volume iothreads                      #iothreads can give performance a boost
  type performance/io-threads
  option thread-count 16
  subvolumes fs3-storage
end-volume

## Add network serving capability to above brick.
volume server
 type protocol/server
 option transport-type tcp/server     # For TCP/IP transport
 option listen-port 6996              # Default is 6996
 option client-volume-filename /var/log/glusterfs/client.vol
 subvolumes iothreads
 option auth.ip.iothreads.allow * # Allow access to "brick" volume
end-volume

Server 2

## Define the stroage
volume fs4-storage
 type storage/posix                   # POSIX FS translator
 option directory /storage            # Export this directory
end-volume

volume iothreads                      #iothreads can give performance a boost
  type performance/io-threads
  option thread-count 16
  subvolumes fs4-storage
end-volume

## Add network serving capability to above brick.
volume server
 type protocol/server
 option transport-type tcp/server     # For TCP/IP transport
 option listen-port 6996              # Default is 6996
 option client-volume-filename /var/log/glusterfs/client.vol
 subvolumes iothreads
 option auth.ip.iothreads.allow * # Allow access to "brick" volume
end-volume


Client side configuration...

Client 1

### Add client feature and attach to remote subvolume
volume client_214
 type protocol/client
 option transport-type tcp/client     # for TCP/IP transport
option remote-host <http://10.10.0.214>10.10.0.214 # IP address of the remote brick 
 option remote-port 6996              # default server port is 6996
 option remote-subvolume iothreads        # name of the remote volume
 option transport-timeout 4
end-volume

### Add client feature and attach to remote subvolume
volume client_213
 type protocol/client
 option transport-type tcp/client     # for TCP/IP transport
option remote-host <http://10.10.0.213>10.10.0.213 # IP address of the remote brick 
 option remote-port 6996              # default server port is 6996
 option remote-subvolume iothreads        # name of the remote volume
 option transport-timeout 4
end-volume

volume afrbricks
 type cluster/afr
 subvolumes  client_214  client_213
 option replicate *:2
 option self-heal on
end-volume

volume iothreads    #iothreads can give performance a boost
  type performance/io-threads
  option thread-count 8
  subvolumes afrbricks
end-volume
##########################

Client 2

Client 1

### Add client feature and attach to remote subvolume
volume client_214
 type protocol/client
 option transport-type tcp/client     # for TCP/IP transport
option remote-host <http://10.10.0.214>10.10.0.214 # IP address of the remote brick 
 option remote-port 6996              # default server port is 6996
 option remote-subvolume iothreads        # name of the remote volume
 option transport-timeout 4
end-volume

### Add client feature and attach to remote subvolume
volume client_213
 type protocol/client
 option transport-type tcp/client     # for TCP/IP transport
option remote-host <http://10.10.0.213>10.10.0.213 # IP address of the remote brick 
 option remote-port 6996              # default server port is 6996
 option remote-subvolume iothreads        # name of the remote volume
 option transport-timeout 4
end-volume

volume afrbricks
 type cluster/afr
 subvolumes  client_214  client_213
 option replicate *:2
 option self-heal on
end-volume

volume iothreads    #iothreads can give performance a boost
  type performance/io-threads
  option thread-count 8
  subvolumes afrbricks
end-volume
##########################


I am mounting the server to client with glusterfs -f
/etc/glusterfs/glusterfs- client.vol /storage/

I could able to mount, files are getting replicated to both file
servers. But the problem, the mounting is braking very frequently...

I am getting the error in glusterfs.log

2007-10-30 00:46:10 C [ tcp.c:81:tcp_disconnect] client_213:
connection disconnected
2007-10-30 00:46:18 C [client-protocol.c:218:call_bail] client_214:
bailing transport
2007-10-30 00:46:18 C [client-protocol.c:218:call_bail] client_213:
bailing transport
2007-10-30 00:46:18 C [tcp.c:81:tcp_disconnect] client_214:
connection disconnected
2007-10-30 00:46:18 C [tcp.c:81:tcp_disconnect] client_213:
connection disconnected

But the physical connection is still there...I can ping from client
to servers...
The server is using gigabit networking bonding.

I am using kernel with Fuse supported

address@hidden ~# lsmod
Module                  Size  Used by
fuse                   39444  2
ipv6                  221344  24
tg3                   105860  0
bonding                79224  0
jfs                   163564  2
usbhid                 35936  0
ohci_hcd               18564  0
usbcore               112772  3 usbhid,ohci_hcd
parport_pc             21956  0
parport                20032  1 parport_pc
shpchp                 32416  0
serverworks             8840  0 [permanent]
cciss                  54020  8
dm_mod                 49432  0
sd_mod                 17024  0
piix                    9604  0 [permanent]
ide_disk               14336  0
ide_generic             2048  0 [permanent]
ide_core              106444  4 serverworks,piix,ide_disk,ide_generic


Also see the glusterfsd.log in server side

2007-10-30 00:03:43 E [server-protocol.c:197:generic_reply] server:
transport_writev failed
2007-10-30 00:03:43 E [tcp.c:118:tcp_except] server: shutdown () -
error: Transport endpoint is not connected
2007-10-30 00:03:43 C [tcp.c:81:tcp_disconnect] server: connection disconnected 
2007-10-30 00:03:43 E [protocol.c:253:gf_block_unserialize_transport]
server: EOF from peer ( <http://10.10.0.203:1018>10.10.0.203:1018)
2007-10-30 00:03:43 C [tcp.c:81:tcp_disconnect] server: connection disconnected 
2007-10-30 00:03:43 E [server-protocol.c:197:generic_reply] server:
transport_writev failed
2007-10-30 00:03:43 C [tcp.c:81:tcp_disconnect] server: connection disconnected 
2007-10-30 00:03:43 E [tcp.c:118:tcp_except] server: shutdown () -
error: Transport endpoint is not connected
2007-10-30 00:03:43 C [tcp.c:81:tcp_disconnect] server: connection disconnected 
2007-10-30 00:03:43 E [server-protocol.c:197:generic_reply] server:
transport_writev failed
2007-10-30 00:03:43 E [tcp.c:118:tcp_except] server: shutdown () -
error: Transport endpoint is not connected
2007-10-30 00:03:43 C [tcp.c:81:tcp_disconnect] server: connection disconnected 2007-10-30 00:03:43 C [tcp.c:81:tcp_disconnect] server: connection disconnected 
2007-10-30 00:03:43 E [ server-protocol.c:197:generic_reply] server:
transport_writev failed
2007-10-30 00:03:43 E [tcp.c:118:tcp_except] server: shutdown () -
error: Transport endpoint is not connected
2007-10-30 00:03:43 C [tcp.c:81:tcp_disconnect] server: connection disconnected 2007-10-30 00:03:43 C [tcp.c:81:tcp_disconnect] server: connection disconnected 2007-10-30 00:03:43 C [tcp.c:81:tcp_disconnect] server: connection disconnected 
2007-10-30 00:03:43 E [server-protocol.c:197:generic_reply] server:
transport_writev failed
2007-10-30 00:03:43 C [tcp.c:81:tcp_disconnect] server: connection disconnected 
2007-10-30 00:03:43 E [tcp.c:118:tcp_except] server: shutdown () -
error: Transport endpoint is not connected
2007-10-30 00:48:52 E [server-protocol.c:197:generic_reply] server:
transport_writev failed
2007-10-30 00:48:52 E [tcp.c:118:tcp_except] server: shutdown () -
error: Transport endpoint is not connected
2007-10-30 00:48:52 C [ tcp.c:81:tcp_disconnect] server: connection disconnected 
2007-10-30 00:48:52 E [server-protocol.c:197:generic_reply] server:
transport_writev failed
2007-10-30 00:48:52 C [tcp.c:81:tcp_disconnect] server: connection disconnected 
2007-10-30 00:48:52 E [tcp.c:118:tcp_except] server: shutdown () -
error: Transport endpoint is not connected
2007-10-30 00:48:52 C [tcp.c:81:tcp_disconnect] server: connection disconnected 
2007-10-30 00:48:52 E [ server-protocol.c:197:generic_reply] server:
transport_writev failed
2007-10-30 00:48:52 E [tcp.c:118:tcp_except] server: shutdown () -
error: Transport endpoint is not connected
2007-10-30 00:48:52 C [tcp.c:81:tcp_disconnect] server: connection disconnected 2007-10-30 00:48:52 C [tcp.c:81:tcp_disconnect] server: connection disconnected 2007-10-30 00:48:52 C [tcp.c:81:tcp_disconnect] server: connection disconnected 




JOSEPH JOB
Spectrum Softtech Solutions(P)Ltd.
MahaKavi G Road,
Karikkamuri Cross Road
Kochi-682011
0484-4082000
<mailto:address@hidden>address@hidden
Visit at <http://www.spectrum.net.in>www.spectrum.net.in


_______________________________________________
Gluster-devel mailing list
<mailto:address@hidden>address@hidden
http://lists.nongnu.org/mailman/listinfo/gluster-devel




--
Raghavendra G

A centipede was happy quite, until a toad in fun,
Said, "Prey, which leg comes after which?",
This raised his doubts to such a pitch,
He fell flat into the ditch,
Not knowing how to run.
-Anonymous


JOSEPH JOB
Spectrum Softtech Solutions(P)Ltd.
MahaKavi G Road,
Karikkamuri Cross Road
Kochi-682011
0484-4082000
address@hidden
Visit at www.spectrum.net.in
reply via email to
[Prev in Thread] Current Thread [Next in Thread]