Re: Monit did not restart mysql

On Jun 4, 2013, at 10:46 AM, frwa onto <address@hidden> wrote:

Dear Martin,
Why I cant use service mysqld stop and start for the start+stop command? Am I using the correct one? Thank you.

Regards,
Frwa.

On Tue, Jun 4, 2013 at 4:38 PM, Martin Pala <address@hidden> wrote:

No. If you'll change the start+stop commands inside monit to "monit <action> <service>" then monit will enter endless loop (given by the configuration) and recursively call itself. The start/stop commands have to be able to start/stop the service.

Regards,
Martin

On Jun 4, 2013, at 10:35 AM, frwa onto <address@hidden> wrote:

Dear Martin,
Thank you for you input yes I was always stopping it via service mysqld stop which is wrong here then. How about these ? Should I also change them to monit start mysql and monit stop mysql?

c timeout 30 second(s)
Stop program = '/etc/init.d/mysqld stop' timeout 30 second(s)

Regards,
Frwa.

On Tue, Jun 4, 2013 at 2:01 AM, Martin Pala <address@hidden> wrote:

If the service is monitored by monit, the stop/start should be done via monit. If you'll stop mysql via upstart directly ("stop mysqld") wihtout letting monit know about the stop, then monit will try to restart the process (if the pidfile will be removed or the given PID doesn't exist). The correct way to stop monitored process is either: "monit stop mysqld" or "monit unmonitor mysql" (which disabled the monitoring) - then you can do any manual action without triggering monit automatic recovery.

The "1 times within 1 cycle" is just verbose expansion of then testing rule and means "perform action on first error" - and as the "cycle" word hints, it's based on the monit cycle length (configured with "set daemon X"). If you need more "solid" error before the action (such as 8 failures within 10 cycles), then you can use the extended options.

See monit manual for more details: http://www.mmonit.com/monit/documentation/monit.html

Regards,

Martin

On Jun 3, 2013, at 6:31 AM, frwa onto <address@hidden> wrote:

Dear Martin,
Sorry kind of new to using the features. I did monit -v so I found there is nt log file enabled. If you notice I have now changed it to 127.0.0.1. So what will be on the look for this file right /var/run/mysqld/mysqld.pid ? Say when I stop mysql will this file will be gone. So then what will monit do run the stop is it? Then run the start? Can I ask for run and stop to be service mysqld stop and service mysqld start? Kind of confuse with these statement "with timeout 5 seconds and retry 0 time(s) 1 times within 1 cycle(s) then restart else if succeeded 1 times within 1 cycle(s) then alert" What is the 1 times within 1 cycle is it within the 1 minute of polling? Thank you.

monit -v
monit: Debug: Adding host allow 'localhost'
monit: Debug: Skipping redundant host 'localhost'
monit: Debug: Skipping redundant host 'localhost'

monit: Debug: Skipping redundant host 'localhost'
monit: Debug: Skipping redundant host 'localhost'
monit: Debug: Skipping redundant host 'localhost'
monit: Debug: Adding credentials for user 'admin'.

monit: Adding PAM group 'monit'.
monit: Adding PAM group 'users'.
Runtime constants:
Control file = /etc/monit.conf
Log file = (not defined)

Pid file = /var/run/monit.pid
Id file = /var/monit/id
Debug = True
Log = False
Use syslog = False
Is Daemon = True

Use process engine = True
Poll time = 60 seconds with start delay 0 seconds
Expect buffer = 256 bytes
Mail server(s) = smtp.gmail.com:587(ssl) with timeout 30 seconds

Mail from = (not defined)
Mail subject = (not defined)
Mail message = (not defined)
Start monit httpd = True
httpd bind address = localhost
httpd portnumber = 2812
httpd signature = True
Use ssl encryption = False
httpd auth. style = Basic Authentication and Host/Net allow list
Alert mail to = address@hidden

Alert on = All events
Alert mail to = address@hidden
Alert on = All events

The service list contains the following entries:

Process Name = mysqld
Pid file = /var/run/mysqld/mysqld.pid
Monitoring mode = active
Start program = '/etc/init.d/mysqld start' timeout 30 second(s)

Stop program = '/etc/init.d/mysqld stop' timeout 30 second(s)
Existence = if does not exist 1 times within 1 cycle(s) then restart else if succeeded 1 times within 1 cycle(s) then alert

Pid = if changed 1 times within 1 cycle(s) then alert
Ppid = if changed 1 times within 1 cycle(s) then alert
Port = if failed [127.0.0.1:3306 [MYSQL via TCP] with timeout 5 seconds and retry 0 time(s) 1 times within 1 cycle(s) then restart else if succeeded 1 times within 1 cycle(s) then alert

Timeout = If restarted 5 times within 5 cycle(s) then unmonitor

System Name = localhost.localdomain
Monitoring mode = active

-------------------------------------------------------------------------------

monit daemon with PID 8592 awakened

On Sun, Jun 2, 2013 at 4:43 PM, Martin Pala <address@hidden> wrote:

Hi,

please check monit logs and "monit status" for mysqld. You can run monit in debug mode using "-v" option to see more details.

From the description i'm not sure exactly what happened - it seems that mysql was running and you tried to kill/stop it to see whether monit will recover it? As you have two tests, the mysql error should be detected when the process stops (i.e. the PID in the /var/run/mysqld/mysqld.pid file is not present in the process table) and/or if the connection fails.

As Sven correctly pointed out the 127.0.1.1 is wrong, so the connection test probably always failed. Because you have restart action on connection failure, monit will try to restart it - and since there is also the timeout statement, it'll give up after 5 attempts and will disable the service monitoring.

My guess is, that the misconfigured connection test (127.0.1.1 instead of 127.0.0.1) caused monit to restart mysql five times (as the test failed after the restart too), which led to service unmonitoring => monit most probably doesn't watch mysql anymore and if you killed it, it won't restart it.

The solution if this state is to fix the configuration (127.0.1.1 -> 127.0.0.1) and enable mysql monitoring again: "monit monitor mysqld"

Regards,

Martin

On Jun 2, 2013, at 10:19 AM, frwa onto <address@hidden> wrote:

Hi,
I have tried both 127.0.1.1 and 127.0.0.1 but yet its not working what else could be done?

On Sun, Jun 2, 2013 at 12:58 AM, frwa onto <address@hidden> wrote:

Sorry I got mixed up what different will 127.0.1.1 make indeed? Thank you .

On Sun, Jun 2, 2013 at 12:55 AM, frwa onto <address@hidden> wrote:

What ROFL ?

On Sun, Jun 2, 2013 at 12:44 AM, sven falempin <address@hidden> wrote:

ROFL ROFL

WELCOME TO FAIL
ProTip stop using your brain use CTRL+F or /

On Sat, Jun 1, 2013 at 11:03 AM, frwa onto <address@hidden> wrote:

Hi Sven,
I already have it as 127.0.0.1 see below
check process mysqld with pidfile /var/run/mysqld/mysqld.pid

start program = "/etc/init.d/mysqld start"
stop program = "/etc/init.d/mysqld stop"

if failed host 127.0.1.1 port 3306 protocol mysql then restart
if 5 restarts within 5 cycles then timeout

On Sat, Jun 1, 2013 at 8:25 PM, sven falempin <address@hidden> wrote:

127.0.1.1

xd

On Sat, Jun 1, 2013 at 2:58 AM, frwa onto <address@hidden> wrote:

Hi All,
We have this in our monit.conf. Earlier it was working smoothly when mysql goes down. Recently I just tried to stop mysqld and the monit did not restart it. But in the top I can see monit is running and I am using upstart so I just kill and it respawn again. Below is part of the config file.

check process mysqld with pidfile /var/run/mysqld/mysqld.pid
start program = "/etc/init.d/mysqld start"
stop program = "/etc/init.d/mysqld stop"

if failed host 127.0.1.1 port 3306 protocol mysql then restart
if 5 restarts within 5 cycles then timeout

Regards,
Frwa.

--
To unsubscribe:
https://lists.nongnu.org/mailman/listinfo/monit-general

--
---------------------------------------------------------------------------------------------------------------------
() ascii ribbon campaign - against html e-mail
/\

--
To unsubscribe:
https://lists.nongnu.org/mailman/listinfo/monit-general

--
To unsubscribe:
https://lists.nongnu.org/mailman/listinfo/monit-general

--
---------------------------------------------------------------------------------------------------------------------
() ascii ribbon campaign - against html e-mail
/\

--
To unsubscribe:
https://lists.nongnu.org/mailman/listinfo/monit-general

--
To unsubscribe:
https://lists.nongnu.org/mailman/listinfo/monit-general

--
To unsubscribe:
https://lists.nongnu.org/mailman/listinfo/monit-general

--
To unsubscribe:
https://lists.nongnu.org/mailman/listinfo/monit-general

--
To unsubscribe:
https://lists.nongnu.org/mailman/listinfo/monit-general

--
To unsubscribe:
https://lists.nongnu.org/mailman/listinfo/monit-general

--
To unsubscribe:
https://lists.nongnu.org/mailman/listinfo/monit-general

--
To unsubscribe:
https://lists.nongnu.org/mailman/listinfo/monit-general

From:	Martin Pala
Subject:	Re: Monit did not restart mysql
Date:	Tue, 4 Jun 2013 10:50:44 +0200