Re: blocked jobs

[Todd Denniston]

Jeevesh Kaul wrote, On 04/21/2008 03:29 PM:
> we have a situation where if we run ps on the state we see blocked jobs that
> are high in number around 8 ( vmstat )

what options are you passing to ps?
what options are you passing to vmstat?
what makes you think _cvs_, instead of something else, may be causing this 
high blocked number?
do you have any cvs jobs that are not being blocked?
when you run top, what is are the 4 items at the top of the list and how much 
cpu are they pulling?

what does the output of `vmstat 5 5` look like?

how many cvs processes?
how many different users own those processes?
Are _all_ of those users physically at their terminals right now? (i.e., has 
someone started a commit or other operation that locked the repo, but either 
left it hanging or somehow killed the controlling process?

> The cvs server is run on a linux box RH AS release 4.
> using Nagios to monitor server load we dont find any underlying problems
> with NFS or memory or disk, yet the cvs app is slow in response.

What do you mean slow in response?

Does the same operation take nearly the same time to do if only ONE user is 
accessing the server machine?

> how should we go about debugging what puts the cvs app into the sleep state.
> There are probably high cvs reads happening and there is nothing obvious
> that leaps up.
> cvs server version used 1.11.17-9.

What is the cvs connection method? (:ext:, :ext: with ssh, pserver, NFS)?

Are all of your clients (developers) using the same connection method?

Are you sure that something else on the server is not slowing things down? 
i.e., did the admin make the mistake of 1) leaving the RH install booting in 
runlevel 5 instead of 3 and 2) logged in and then lock the screen running the 
3D Gears screen saver, or even just stay logged in and let one of the gnome 
applets go crazy? (I have seen both on THE SAME machine, it is a real drag 
even with quad processors)

Is the repository on a local disk or NFS mounted?

What file system is the repository on? any non-default options used in the 
creation or mounting of that file system?

How much memory?

How much VM?

How much disk space in /tmp?

Any IO errors showing up in /var/log/messages or dmesg

How large are the four largest files in the repository and are any of them in 
the same directory?

Are many of your developers working on branches instead of the trunk?

does /usr/share/cvs*/contrib/ check_cvs [2] or validate_repo [1] indicate any 
problems?

i.e., not nearly enough info to make an educated guess.

[1] 
http://cvs.savannah.nongnu.org/viewvc/*checkout*/ccvs/contrib/validate_repo.pl?root=cvs&content-type=text%2Fplain
[2] 
http://cvs.savannah.nongnu.org/viewvc/ccvs/contrib/check_cvs.in?revision=1.17&root=cvs&view=markup

--
Todd Denniston
Crane Division, Naval Surface Warfare Center (NSWC Crane)
Harnessing the Power of Technology for the Warfighter