Re: [monit] Quad Core Monitoring

[monit ml]

Actually, you are probably better off if mysql hogs 1 core rather than 1/4 of 
each of the four cores because then you might have a more optimal situation 
for L2/L1 cache not being flushed when other processes are running on the same 
core.  That is what you want, mysql sitting on a core, never getting context 
switched out so that the data in the high speed caches are never invalidated 
and no overhead of context switching - let all the other stuff run and context 
switch on some other cores, those processes probably are not doing much and 
they don't need a huge, constant stream of data in the on-chip caches.

Ideally what you want is a process scheduler that allows you to give mysql a 
big fat time slice that never gets changed by the scheduling algo - since your 
machine is dedicated to mysql, you don't want "fair" scheduling, you want 
unfair scheduling - mysql gets to hog some cores and doesn't get its time 
slice constantly modified by some process scheduler that thinks it knows 
better. Then if you can affinity your mysql process to N-1 cores, you will 
always have at least one core that can do other work, when needed. task cpu 
(nowadays core) migrations and context switches are the enemy, they add 
unnecessary overhead and blow the wealth of data that you have built up in 
your L3/L2/L1 uber fast caches.

http://www.linuxjournal.com/article/6799

Regarding monit's behavior:
monit properly reports total cpu usage, threads hop around onto different 
cores at the whim of the kernel scheduler and there is no benefit to 
differentiating between 100% of 1 core vs 25% of 4 cores each because the 
scheduler could decide to move threads to different cores at any time.

Have you tried htop? It will show you in more detail what is running where,
and you might be able to see the individual threads of mysql move around onto 
different cores. I would get dstat and watch the context switch counts, not 
which core mysql is running on.

address@hidden wrote:
> Thanks, but I don't think that would work.  If all four processors were 
> above 25%, it would give a false positive.  Am I correct in thinking 
> that monit doesn't actually have the capability to monitor more than one 
> cpu?  Maybe this should be a wishlist item?
>
>
> On Tue, Apr 29, 2008 at 12:34 PM, Martin Pala <address@hidden 
> <mailto:address@hidden>> wrote:
>
>     Monit is reporting the CPU usage as total (corresponding to first
>     example of top).
>
>     If the application will get 100% of one core on 4-core system, it's
>     CPU usage should be reported by monit is 25% (since 4 * 25% = 100%).
>
>     You can thus use rule like this:
>
>      if cpu usage > 25% then restart
>
>     Martin
>
>
>
>
>     address@hidden <mailto:address@hidden> wrote:
>
>         We have an issue were mysqld hogs up 100% of one of our cpu
>         cores from time to time.  How can I configure monit to restart
>         mysqld when this happens.
>         For an example of what I am talking about, top normally shows
>         only one cpu but if you press '1' it will show all the cpus.
>         (normal top)
>         top - 18:13:11 up 10 days, 21:59,  1 user,  load average: 1.56,
>         1.81, 2.08
>         Tasks: 186 total,   5 running, 181 sleeping,   0 stopped,   0 zombie
>         Cpu(s): 31.6% us,  7.6% sy,  0.0% ni, 60.8% id,  0.0% wa,  0.0%
>         hi,  0.0% si
>
>         (top after pressing '1')
>         top - 18:13:54 up 10 days, 22:00,  1 user,  load average: 1.73,
>         1.81, 2.07
>         Tasks: 185 total,   3 running, 182 sleeping,   0 stopped,   0 zombie
>         Cpu0  : 29.4% us,  5.9% sy,  0.0% ni, 63.9% id,  0.7% wa,  0.0%
>         hi,  0.0% si
>         Cpu1  : 27.9% us,  5.6% sy,  0.0% ni, 65.5% id,  1.0% wa,  0.0%
>         hi,  0.0% si
>         Cpu2  : 23.1% us,  5.7% sy,  0.0% ni, 70.6% id,  0.7% wa,  0.0%
>         hi,  0.0% si
>         Cpu3  : 25.3% us,  6.2% sy,  0.0% ni, 67.5% id,  1.0% wa,  0.0%
>         hi,  0.0% si
>
>         (sample when mysqld is hogging cpu time)
>         top - 18:13:54 up 10 days, 22:00,  1 user,  load average: 1.73,
>         1.81, 2.07
>         Tasks: 185 total,   3 running, 182 sleeping,   0 stopped,   0 zombie
>         Cpu0  : 100 % us,  5.9% sy,  0.0% ni, 63.9% id,  0.7% wa,  0.0%
>         hi,  0.0% si
>         Cpu1  : 7.9% us,  5.6% sy,  0.0% ni, 65.5% id,  1.0% wa,  0.0%
>         hi,  0.0% si
>         Cpu2  : 3.1% us,  5.7% sy,  0.0% ni, 70.6% id,  0.7% wa,  0.0%
>         hi,  0.0% si
>         Cpu3  : 5.3% us,  6.2% sy,  0.0% ni, 67.5% id,  1.0% wa,  0.0%
>         hi,  0.0% si
>
>         So, how do I get monit to restart mysqld when just one cpu is
>         pegged at 100%?  It's not always Cpu0, it could be any of them.
>
>         Thanks,