Re: Monit did not restart mysql

[Martin Pala]

The manual action event can be filtered out, for example:

        set alert address@hidden but not on { action }

See monit manual for more details: 
http://www.mmonit.com/monit/documentation/monit.html#alert_messages

The logfile settings - syslog or monit controlled file are equal, it just 
depends on your preference.


On Jun 4, 2013, at 3:29 PM, frwa onto <address@hidden> wrote:

> Dear Martin,
>                  Thank you for the start+stop command. The issue now when I 
> do this monit stop mysqld and monit start mysqld I get email for both events. 
> Why when I monit stop mysqld it does not show up message as how I do like 
> this service mysqld stop? Secondly once I monit stop mysqld it did not 
> restart? I have just uncomment this line set logfile syslog facility 
> log_daemon. Then I did was initctl stop and then initctl start mysql. Next 
> when I did this monit -v I can see now Log file = syslog. Is this correctly 
> done is better to specific my own log file say /usr/local/monit.log. Thank 
> you.
> 
> Regards,
> Frwa.
> 
> 
> 
> On Tue, Jun 4, 2013 at 4:50 PM, Martin Pala <address@hidden> wrote:
> 1.) these are OK:
> 
>       Stop program = '/etc/init.d/mysqld stop'
> 
>       Stop program = '/sbin/stop mysqld'
> 
> 
> 2.) this is NOT OK:
> 
>       Stop program = '/usr/bin/monit stop mysqld'
> 
> 
> Regards,
> Martin
> 
> 
> 
> On Jun 4, 2013, at 10:46 AM, frwa onto <address@hidden> wrote:
> 
>> Dear Martin,
>>                   Why I cant use service mysqld stop and start for the 
>> start+stop command? Am I using the correct one? Thank you.
>> 
>> Regards,
>> Frwa.
>> 
>> 
>> On Tue, Jun 4, 2013 at 4:38 PM, Martin Pala <address@hidden> wrote:
>> No. If you'll change the start+stop commands inside monit to "monit <action> 
>> <service>" then monit will enter endless loop (given by the configuration) 
>> and recursively call itself. The start/stop commands have to be able to 
>> start/stop the service.
>> 
>> Regards,
>> Martin
>> 
>> 
>> On Jun 4, 2013, at 10:35 AM, frwa onto <address@hidden> wrote:
>> 
>>> Dear Martin,
>>>                   Thank you for you input yes I was always stopping it via 
>>> service mysqld stop which is wrong here then. How about these ? Should I 
>>> also change them to monit start mysql and monit stop mysql?
>>> c timeout 30 second(s)
>>>  Stop program         = '/etc/init.d/mysqld stop' timeout 30 second(s)
>>> 
>>> 
>>> Regards,
>>> Frwa.
>>> 
>>> 
>>> On Tue, Jun 4, 2013 at 2:01 AM, Martin Pala <address@hidden> wrote:
>>> If the service is monitored by monit, the stop/start should be done via 
>>> monit. If you'll stop mysql via upstart directly ("stop mysqld") wihtout 
>>> letting monit know about the stop, then monit will try to restart the 
>>> process (if the pidfile will be removed  or the given PID doesn't exist). 
>>> The correct way to stop monitored process is either: "monit stop mysqld" or 
>>> "monit unmonitor mysql" (which disabled the monitoring) - then you can do 
>>> any manual action without triggering monit automatic recovery.
>>> 
>>> The "1 times within 1 cycle" is just verbose expansion of then testing rule 
>>> and means "perform action on first error" - and as the "cycle" word hints, 
>>> it's based on the monit cycle length (configured with "set daemon X"). If 
>>> you need more "solid" error  before the action (such as 8 failures within 
>>> 10 cycles), then you can use the extended options.
>>> 
>>> See monit manual for more details: 
>>> http://www.mmonit.com/monit/documentation/monit.html
>>> 
>>> Regards,
>>> Martin
>>> 
>>> 
>>> 
>>> On Jun 3, 2013, at 6:31 AM, frwa onto <address@hidden> wrote:
>>> 
>>>> Dear Martin,
>>>>                  Sorry kind of new to using the features. I did monit -v 
>>>> so I found there is nt log file enabled. If you notice I have now changed 
>>>> it to 127.0.0.1. So what will be on the look for this file right 
>>>> /var/run/mysqld/mysqld.pid ? Say when I stop mysql will this file will be 
>>>> gone. So then what will monit do run the stop is it? Then run the start? 
>>>> Can I ask for run and stop to be service mysqld stop and service mysqld 
>>>> start? Kind of confuse with these statement "with timeout 5 seconds and 
>>>> retry 0 time(s) 1 times within 1 cycle(s) then restart else if succeeded 1 
>>>> times within 1 cycle(s) then alert" What is the 1 times within 1 cycle is 
>>>> it within the 1 minute of polling? Thank you.
>>>> 
>>>> monit -v
>>>> monit: Debug: Adding host allow 'localhost'
>>>> monit: Debug: Skipping redundant host 'localhost'
>>>> monit: Debug: Skipping redundant host 'localhost'
>>>> monit: Debug: Skipping redundant host 'localhost'
>>>> monit: Debug: Skipping redundant host 'localhost'
>>>> monit: Debug: Skipping redundant host 'localhost'
>>>> monit: Debug: Adding credentials for user 'admin'.
>>>> monit: Adding PAM group 'monit'.
>>>> monit: Adding PAM group 'users'.
>>>> Runtime constants:
>>>>  Control file       = /etc/monit.conf
>>>>  Log file           = (not defined)
>>>>  Pid file           = /var/run/monit.pid
>>>>  Id file            = /var/monit/id
>>>>  Debug              = True
>>>>  Log                = False
>>>>  Use syslog         = False
>>>>  Is Daemon          = True
>>>>  Use process engine = True
>>>>  Poll time          = 60 seconds with start delay 0 seconds
>>>>  Expect buffer      = 256 bytes
>>>>  Mail server(s)     = smtp.gmail.com:587(ssl) with timeout 30 seconds
>>>>  Mail from          = (not defined)
>>>>  Mail subject       = (not defined)
>>>>  Mail message       = (not defined)
>>>>  Start monit httpd  = True
>>>>  httpd bind address = localhost
>>>>  httpd portnumber   = 2812
>>>>  httpd signature    = True
>>>>  Use ssl encryption = False
>>>>  httpd auth. style  = Basic Authentication and Host/Net allow list
>>>>  Alert mail to      = address@hidden
>>>>    Alert on         = All events
>>>>  Alert mail to      = address@hidden
>>>>    Alert on         = All events
>>>> 
>>>> The service list contains the following entries:
>>>> 
>>>> Process Name          = mysqld
>>>>  Pid file             = /var/run/mysqld/mysqld.pid
>>>>  Monitoring mode      = active
>>>>  Start program        = '/etc/init.d/mysqld start' timeout 30 second(s)
>>>>  Stop program         = '/etc/init.d/mysqld stop' timeout 30 second(s)
>>>>  Existence            = if does not exist 1 times within 1 cycle(s) then 
>>>> restart else if succeeded 1 times within 1 cycle(s) then alert
>>>>  Pid                  = if changed 1 times within 1 cycle(s) then alert
>>>>  Ppid                 = if changed 1 times within 1 cycle(s) then alert
>>>>  Port                 = if failed [127.0.0.1:3306 [MYSQL via TCP] with 
>>>> timeout 5 seconds and retry 0 time(s) 1 times within 1 cycle(s) then 
>>>> restart else if succeeded 1 times within 1 cycle(s) then alert
>>>>  Timeout              = If restarted 5 times within 5 cycle(s) then 
>>>> unmonitor
>>>> 
>>>> System Name           = localhost.localdomain
>>>>  Monitoring mode      = active
>>>> 
>>>> -------------------------------------------------------------------------------
>>>> monit daemon with PID 8592 awakened
>>>> 
>>>> 
>>>> 
>>>> On Sun, Jun 2, 2013 at 4:43 PM, Martin Pala <address@hidden> wrote:
>>>> Hi,
>>>> 
>>>> please check monit logs and "monit status" for mysqld. You can run monit 
>>>> in debug mode using "-v" option to see more details.
>>>> 
>>>> From the description i'm not sure exactly what happened - it seems that 
>>>> mysql was running and you tried to kill/stop it to see whether monit will 
>>>> recover it? As you have two tests, the mysql error should be detected when 
>>>> the process stops (i.e. the PID in the /var/run/mysqld/mysqld.pid file is 
>>>> not present in the process table) and/or if the connection fails.
>>>> 
>>>> As Sven correctly pointed out the 127.0.1.1 is wrong, so the connection 
>>>> test probably always failed. Because you have restart action on connection 
>>>> failure, monit will try to restart it - and since there is also the 
>>>> timeout statement, it'll give up after 5 attempts and will disable the 
>>>> service monitoring.
>>>> 
>>>> My guess is, that the misconfigured connection test (127.0.1.1 instead of 
>>>> 127.0.0.1) caused monit to restart mysql five times (as the test failed 
>>>> after the restart too), which led to service unmonitoring => monit most 
>>>> probably doesn't watch mysql anymore and if you killed it, it won't 
>>>> restart it.
>>>> 
>>>> The solution if this state is to fix the configuration (127.0.1.1 -> 
>>>> 127.0.0.1) and enable mysql monitoring again: "monit monitor mysqld"
>>>> 
>>>> 
>>>> Regards,
>>>> Martin
>>>> 
>>>> 
>>>> 
>>>> On Jun 2, 2013, at 10:19 AM, frwa onto <address@hidden> wrote:
>>>> 
>>>>> Hi,
>>>>>     I have tried both 127.0.1.1 and 127.0.0.1 but yet its not working 
>>>>> what else could be done?
>>>>> 
>>>>> 
>>>>> On Sun, Jun 2, 2013 at 12:58 AM, frwa onto <address@hidden> wrote:
>>>>> Sorry I got mixed up what different will 127.0.1.1 make indeed? Thank you 
>>>>> .
>>>>> 
>>>>> 
>>>>> On Sun, Jun 2, 2013 at 12:55 AM, frwa onto <address@hidden> wrote:
>>>>> What ROFL ?
>>>>> 
>>>>> 
>>>>> On Sun, Jun 2, 2013 at 12:44 AM, sven falempin <address@hidden> wrote:
>>>>> 
>>>>> ROFL ROFL
>>>>> 
>>>>> WELCOME TO FAIL
>>>>> ProTip stop using your brain use CTRL+F or /
>>>>> 
>>>>> 
>>>>> 
>>>>> On Sat, Jun 1, 2013 at 11:03 AM, frwa onto <address@hidden> wrote:
>>>>> Hi Sven,
>>>>>             I already have it as 127.0.0.1 see below
>>>>> check process mysqld with pidfile /var/run/mysqld/mysqld.pid
>>>>>    start program = "/etc/init.d/mysqld start"
>>>>>    stop program = "/etc/init.d/mysqld stop"
>>>>>    if failed host 127.0.1.1 port 3306 protocol mysql then restart
>>>>>    if 5 restarts within 5 cycles then timeout
>>>>> 
>>>>> 
>>>>> On Sat, Jun 1, 2013 at 8:25 PM, sven falempin <address@hidden> wrote:
>>>>> 127.0.1.1
>>>>> 
>>>>> xd
>>>>> 
>>>>> 
>>>>> On Sat, Jun 1, 2013 at 2:58 AM, frwa onto <address@hidden> wrote:
>>>>> Hi All,
>>>>>         We have this in our monit.conf. Earlier it was working smoothly 
>>>>> when mysql goes down. Recently I just tried to stop mysqld and the monit 
>>>>> did not restart it. But in the top I can see monit is running and I am 
>>>>> using upstart so I just kill and it respawn again. Below is part of the 
>>>>> config file.
>>>>> 
>>>>> 
>>>>> check process mysqld with pidfile /var/run/mysqld/mysqld.pid
>>>>>    start program = "/etc/init.d/mysqld start"
>>>>>    stop program = "/etc/init.d/mysqld stop"
>>>>>    if failed host 127.0.1.1 port 3306 protocol mysql then restart
>>>>>    if 5 restarts within 5 cycles then timeout
>>>>> 
>>>>> 
>>>>> Regards,
>>>>> Frwa.
>>>>> 
>>>>> --
>>>>> To unsubscribe:
>>>>> https://lists.nongnu.org/mailman/listinfo/monit-general
>>>>> 
>>>>> 
>>>>> 
>>>>> -- 
>>>>> ---------------------------------------------------------------------------------------------------------------------
>>>>> () ascii ribbon campaign - against html e-mail 
>>>>> /\ 
>>>>> 
>>>>> --
>>>>> To unsubscribe:
>>>>> https://lists.nongnu.org/mailman/listinfo/monit-general
>>>>> 
>>>>> 
>>>>> --
>>>>> To unsubscribe:
>>>>> https://lists.nongnu.org/mailman/listinfo/monit-general
>>>>> 
>>>>> 
>>>>> 
>>>>> -- 
>>>>> ---------------------------------------------------------------------------------------------------------------------
>>>>> () ascii ribbon campaign - against html e-mail 
>>>>> /\ 
>>>>> 
>>>>> --
>>>>> To unsubscribe:
>>>>> https://lists.nongnu.org/mailman/listinfo/monit-general
>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> To unsubscribe:
>>>>> https://lists.nongnu.org/mailman/listinfo/monit-general
>>>> 
>>>> 
>>>> --
>>>> To unsubscribe:
>>>> https://lists.nongnu.org/mailman/listinfo/monit-general
>>>> 
>>>> --
>>>> To unsubscribe:
>>>> https://lists.nongnu.org/mailman/listinfo/monit-general
>>> 
>>> 
>>> --
>>> To unsubscribe:
>>> https://lists.nongnu.org/mailman/listinfo/monit-general
>>> 
>>> --
>>> To unsubscribe:
>>> https://lists.nongnu.org/mailman/listinfo/monit-general
>> 
>> 
>> --
>> To unsubscribe:
>> https://lists.nongnu.org/mailman/listinfo/monit-general
>> 
>> --
>> To unsubscribe:
>> https://lists.nongnu.org/mailman/listinfo/monit-general
> 
> 
> --
> To unsubscribe:
> https://lists.nongnu.org/mailman/listinfo/monit-general
> 
> --
> To unsubscribe:
> https://lists.nongnu.org/mailman/listinfo/monit-general