[00:00:04] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [00:02:53] New patchset: Pyoungmeister; "making number of jobrunner procs configurable and setting role class to spin up 12 procs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23627 [00:03:45] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/23627 [00:05:10] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [00:05:19] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23627 [00:08:20] New patchset: Dzahn; "move colors for projects into CSS, move names for get-methods to config" [operations/debs/wikistats] (master) - https://gerrit.wikimedia.org/r/23629 [00:09:23] New patchset: Dzahn; "move colors for projects into CSS, move names for get-methods to config" [operations/debs/wikistats] (master) - https://gerrit.wikimedia.org/r/23629 [00:09:48] Change merged: Dzahn; [operations/debs/wikistats] (master) - https://gerrit.wikimedia.org/r/23629 [00:09:52] binasher: can you purge the mobile varnish cache? [00:10:07] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [00:10:39] or LeslieCarr ^ [00:10:51] oops, binasher is on it [00:15:04] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [00:17:10] RECOVERY - Apache HTTP on srv281 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.023 second response time [00:17:46] RECOVERY - Apache HTTP on mw2 is OK: HTTP OK HTTP/1.1 200 OK - 454 bytes in 0.006 seconds [00:19:04] New patchset: Robmoen; "Config to enable footerCleanup module in Vector extension." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/23632 [00:19:16] PROBLEM - udp2log log age for locke on locke is CRITICAL: CRITICAL: log files /a/squid/fundraising/logs/bannerImpressions-sampled100.log, have not been written in a critical amount of time. For most logs, this is 4 hours. For slow logs, this is 4 days. [00:20:10] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [00:25:08] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [00:27:11] New patchset: awjrichards; "Disable WLM banner" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/23634 [00:29:25] Change merged: MaxSem; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/23634 [00:30:04] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [00:32:28] RECOVERY - Apache HTTP on mw3 is OK: HTTP OK HTTP/1.1 200 OK - 454 bytes in 0.002 seconds [00:33:04] PROBLEM - Apache HTTP on mw2 is CRITICAL: Connection refused [00:35:10] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [00:36:33] New patchset: Pyoungmeister; "adding ganglia aggregator data sources for pmtpa jobrunner group" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23636 [00:37:26] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/23636 [00:37:38] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23636 [00:40:26] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [00:44:04] New patchset: Andrew Bogott; "Update wiki instance-status pages automatically." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23155 [00:44:20] PROBLEM - Apache HTTP on mw3 is CRITICAL: Connection refused [00:44:57] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/23155 [00:45:32] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [00:45:39] New patchset: awjrichards; "Better configuration for MF banner; disabled everywhere by default except testwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/23637 [00:45:59] RECOVERY - udp2log log age for locke on locke is OK: OK: all log files active [00:47:32] New patchset: awjrichards; "Better configuration for MF banner; disabled everywhere by default except testwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/23637 [00:48:23] Change merged: MaxSem; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/23637 [00:48:32] RECOVERY - NTP on mw2 is OK: NTP OK: Offset -0.008253216743 secs [00:50:29] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [00:54:32] PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours [00:55:05] New patchset: Ryan Lane; "Remove whitespace from grains output" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23641 [00:55:26] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [00:55:58] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/23641 [00:56:52] New patchset: Ryan Lane; "Add instanceproject to labs grains" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23642 [00:57:43] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23641 [00:57:44] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/23642 [00:57:50] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23642 [00:58:44] RECOVERY - NTP on mw3 is OK: NTP OK: Offset -0.009525299072 secs [01:00:32] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [01:05:29] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [01:10:08] RECOVERY - Apache HTTP on mw4 is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 0.016 seconds [01:10:17] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [01:10:17] RECOVERY - Apache HTTP on mw11 is OK: HTTP OK HTTP/1.1 200 OK - 454 bytes in 0.005 seconds [01:15:32] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [01:20:29] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [01:20:38] RECOVERY - Apache HTTP on mw5 is OK: HTTP OK HTTP/1.1 200 OK - 454 bytes in 0.010 seconds [01:20:38] PROBLEM - Apache HTTP on mw4 is CRITICAL: Connection refused [01:23:47] PROBLEM - Apache HTTP on mw11 is CRITICAL: Connection refused [01:23:56] RECOVERY - Apache HTTP on mw12 is OK: HTTP OK HTTP/1.1 200 OK - 454 bytes in 0.007 seconds [01:25:26] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [01:30:32] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [01:31:44] RECOVERY - Apache HTTP on mw6 is OK: HTTP OK HTTP/1.1 200 OK - 454 bytes in 0.004 seconds [01:32:38] PROBLEM - Apache HTTP on mw5 is CRITICAL: Connection refused [01:34:08] RECOVERY - Apache HTTP on mw13 is OK: HTTP OK HTTP/1.1 200 OK - 454 bytes in 0.005 seconds [01:34:26] PROBLEM - Apache HTTP on mw12 is CRITICAL: Connection refused [01:35:29] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [01:36:32] RECOVERY - NTP on mw4 is OK: NTP OK: Offset -0.009260416031 secs [01:37:53] RECOVERY - NTP on mw11 is OK: NTP OK: Offset -0.01080667973 secs [01:40:27] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [01:42:18] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 297 seconds [01:42:36] PROBLEM - MySQL Slave Delay on storage3 is CRITICAL: CRIT replication delay 314 seconds [01:43:48] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 9 seconds [01:44:24] PROBLEM - Puppet freshness on ms-be7 is CRITICAL: Puppet has not run in the last 10 hours [01:44:42] PROBLEM - Apache HTTP on mw6 is CRITICAL: Connection refused [01:45:18] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [01:45:27] RECOVERY - NTP on mw13 is OK: NTP OK: Offset -0.0930249691 secs [01:45:27] RECOVERY - Apache HTTP on mw7 is OK: HTTP OK HTTP/1.1 200 OK - 454 bytes in 0.008 seconds [01:45:36] RECOVERY - MySQL Slave Delay on storage3 is OK: OK replication delay 1 seconds [01:46:12] RECOVERY - Apache HTTP on mw14 is OK: HTTP OK HTTP/1.1 200 OK - 454 bytes in 0.010 seconds [01:46:12] PROBLEM - Apache HTTP on mw13 is CRITICAL: Connection refused [01:46:57] RECOVERY - NTP on mw5 is OK: NTP OK: Offset -0.01239728928 secs [01:49:57] RECOVERY - NTP on mw12 is OK: NTP OK: Offset -0.007373571396 secs [01:50:24] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [01:55:21] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [01:56:42] PROBLEM - Apache HTTP on mw14 is CRITICAL: Connection refused [01:56:51] RECOVERY - Apache HTTP on mw8 is OK: HTTP OK HTTP/1.1 200 OK - 454 bytes in 0.003 seconds [01:57:09] RECOVERY - Apache HTTP on mw15 is OK: HTTP OK HTTP/1.1 200 OK - 454 bytes in 0.003 seconds [01:57:27] PROBLEM - Apache HTTP on mw7 is CRITICAL: Connection refused [01:58:57] RECOVERY - NTP on mw6 is OK: NTP OK: Offset -0.009036898613 secs [02:00:18] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [02:05:24] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [02:07:39] PROBLEM - Apache HTTP on mw15 is CRITICAL: Connection refused [02:07:48] RECOVERY - Apache HTTP on mw9 is OK: HTTP OK HTTP/1.1 200 OK - 454 bytes in 0.003 seconds [02:07:57] RECOVERY - Apache HTTP on mw16 is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 0.007 seconds [02:08:06] PROBLEM - Apache HTTP on mw8 is CRITICAL: Connection refused [02:10:21] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [02:11:42] RECOVERY - NTP on mw7 is OK: NTP OK: Offset -0.00726556778 secs [02:12:27] RECOVERY - NTP on mw14 is OK: NTP OK: Offset -0.01538312435 secs [02:15:18] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [02:18:09] RECOVERY - Apache HTTP on mw10 is OK: HTTP OK HTTP/1.1 200 OK - 454 bytes in 0.003 seconds [02:18:18] PROBLEM - Apache HTTP on mw9 is CRITICAL: Connection refused [02:18:27] PROBLEM - Apache HTTP on mw16 is CRITICAL: Connection refused [02:20:15] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [02:22:03] RECOVERY - NTP on mw8 is OK: NTP OK: Offset -0.01163136959 secs [02:23:06] RECOVERY - NTP on mw15 is OK: NTP OK: Offset -0.01224040985 secs [02:25:21] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [02:30:09] PROBLEM - Apache HTTP on mw10 is CRITICAL: Connection refused [02:30:18] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [02:33:36] PROBLEM - udp2log log age for locke on locke is CRITICAL: CRITICAL: log files /a/squid/fundraising/logs/bannerImpressions-sampled100.log, have not been written in a critical amount of time. For most logs, this is 4 hours. For slow logs, this is 4 days. [02:33:45] RECOVERY - NTP on mw16 is OK: NTP OK: Offset -0.01024210453 secs [02:34:12] RECOVERY - NTP on mw9 is OK: NTP OK: Offset -0.007680058479 secs [02:35:24] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [02:40:22] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [02:44:24] RECOVERY - NTP on mw10 is OK: NTP OK: Offset -0.009763360023 secs [02:45:18] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [02:50:24] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [02:55:21] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [03:00:18] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [03:03:27] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [03:05:15] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [03:10:21] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [03:15:18] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [03:15:36] RECOVERY - udp2log log age for locke on locke is OK: OK: all log files active [03:20:24] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [03:25:21] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [03:30:18] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [03:35:24] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [03:40:39] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [03:45:27] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [03:50:33] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [03:50:52] PROBLEM - Host srv281 is DOWN: PING CRITICAL - Packet loss = 100% [03:55:39] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [04:00:27] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [04:05:33] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [04:10:39] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [04:15:36] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [04:20:33] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [04:25:39] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [04:28:39] PROBLEM - Puppet freshness on magnesium is CRITICAL: Puppet has not run in the last 10 hours [04:28:39] PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours [04:30:36] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [04:34:50] PROBLEM - udp2log log age for locke on locke is CRITICAL: CRITICAL: log files /a/squid/fundraising/logs/bannerImpressions-sampled100.log, have not been written in a critical amount of time. For most logs, this is 4 hours. For slow logs, this is 4 days. [04:35:08] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [04:40:14] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [04:45:11] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [04:50:09] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [04:55:14] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [05:00:11] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [05:05:08] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [05:10:14] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [05:15:11] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [05:15:38] RECOVERY - udp2log log age for locke on locke is OK: OK: all log files active [05:20:08] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [05:25:14] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [05:30:09] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [05:32:15] PROBLEM - Puppet freshness on manganese is CRITICAL: Puppet has not run in the last 10 hours [05:35:15] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [05:40:12] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [05:45:09] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [05:45:18] PROBLEM - Puppet freshness on ms-be6 is CRITICAL: Puppet has not run in the last 10 hours [05:50:15] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [05:55:12] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [06:00:09] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [06:05:15] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [06:10:12] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [06:15:09] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [06:20:15] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [06:25:12] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [06:30:19] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [06:33:10] PROBLEM - udp2log log age for locke on locke is CRITICAL: CRITICAL: log files /a/squid/fundraising/logs/bannerImpressions-sampled100.log, have not been written in a critical amount of time. For most logs, this is 4 hours. For slow logs, this is 4 days. [06:35:34] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [06:40:31] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [06:45:28] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [06:50:34] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [06:55:12] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [07:00:18] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [07:05:15] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [07:10:12] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [07:15:18] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [07:20:15] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [07:25:22] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [07:30:19] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [07:35:25] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [07:40:22] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [07:45:19] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [07:50:25] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [07:55:13] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [07:58:14] New patchset: Dereckson; "(this commit message is intended let blank, to let create another branch without bug reference)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/23652 [07:59:10] New patchset: Dereckson; "(bug 40186) New logo for es.wiktionary" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/23652 [08:00:19] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [08:05:16] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [08:10:22] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [08:15:19] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [08:20:25] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [08:25:24] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [08:30:21] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [08:35:18] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [08:36:25] ouch [08:40:24] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [08:45:21] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [08:50:18] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [08:55:24] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [08:59:27] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [08:59:27] PROBLEM - Puppet freshness on singer is CRITICAL: Puppet has not run in the last 10 hours [08:59:27] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [08:59:27] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [08:59:27] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [08:59:27] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [09:00:22] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [09:05:18] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [09:10:24] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [09:15:21] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [09:20:11] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [09:25:08] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [09:30:05] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [09:35:11] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [09:38:11] PROBLEM - Puppet freshness on ms-be10 is CRITICAL: Puppet has not run in the last 10 hours [09:40:08] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [09:45:05] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [09:50:11] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [09:55:08] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [10:00:05] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [10:05:11] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [10:10:08] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [10:15:05] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [10:20:18] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [10:25:24] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [10:30:21] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [10:35:18] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [10:40:24] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [10:45:21] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [10:50:18] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [10:55:24] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [10:55:24] PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours [11:00:21] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [11:05:18] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [11:10:24] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [11:15:12] RECOVERY - udp2log log age for locke on locke is OK: OK: all log files active [11:15:21] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [11:20:34] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [11:25:30] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [11:30:27] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [11:35:33] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [11:40:30] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [11:45:18] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [11:45:36] PROBLEM - Puppet freshness on ms-be7 is CRITICAL: Puppet has not run in the last 10 hours [12:00:27] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [12:05:15] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [12:10:31] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [12:15:27] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [12:20:15] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [12:25:29] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [12:30:26] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [12:35:32] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [12:40:29] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [12:45:26] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [12:50:32] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [12:55:29] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [13:00:26] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [13:04:02] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [13:05:32] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [13:10:41] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [13:15:58] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [13:20:32] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [13:40:18] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [13:45:24] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [13:49:34] New review: Dereckson; "shellpolicy -> shell" [operations/mediawiki-config] (master) C: 0; - https://gerrit.wikimedia.org/r/23490 [13:50:22] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [13:55:18] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [14:00:24] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [14:05:21] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [14:10:18] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [14:15:24] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [14:20:21] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [14:25:18] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [14:30:10] PROBLEM - Puppet freshness on magnesium is CRITICAL: Puppet has not run in the last 10 hours [14:30:10] PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours [14:30:10] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [14:35:16] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [14:37:18] New patchset: Dereckson; "Removes transcoding stub" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/17365 [14:40:13] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [14:45:10] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [14:50:16] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [14:55:13] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [15:00:10] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [15:05:16] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [15:10:13] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [15:15:10] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [15:20:16] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [15:25:13] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [15:30:10] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [15:33:19] PROBLEM - Puppet freshness on manganese is CRITICAL: Puppet has not run in the last 10 hours [15:35:07] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [15:40:13] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [15:45:10] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [15:46:13] PROBLEM - Puppet freshness on ms-be6 is CRITICAL: Puppet has not run in the last 10 hours [15:50:07] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [15:55:13] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [16:00:10] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [16:02:18] New review: Nikerabbit; "I don't think this addresses the real problem of having easy way to remove sensitive data from the i..." [operations/debs/lucene-search-2] (master) C: 0; - https://gerrit.wikimedia.org/r/23583 [16:05:07] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [16:10:13] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [16:15:10] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [16:20:08] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [16:25:13] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [16:30:10] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [16:35:07] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [16:40:17] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [16:45:14] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [16:47:14] New review: Diederik; "@NikeRabbit: the goal is not to clean sensitive information from the index, and I am not sure wheth..." [operations/debs/lucene-search-2] (master) C: 0; - https://gerrit.wikimedia.org/r/23583 [16:48:55] mutante: http://wikitech.wikimedia.org/view/MobileFrontend#Flushing_the_cache [16:50:11] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [16:50:39] mutante, http://wikitech.wikimedia.org/view/Varnish#One-off_purges [16:51:02] mutante, MaxSem: for mobile varnish, use this: http://wikitech.wikimedia.org/view/MobileFrontend#Flushing_the_cache [16:54:16] New patchset: awjrichards; "Turn on WLM banner everywhere" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/23672 [16:55:17] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [16:57:13] awjr: alright, just tell me when [16:57:42] mutante: groovy thanks [16:57:46] Change merged: MaxSem; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/23672 [17:00:14] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [17:04:23] mutante: ok, go ahead [17:05:11] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [17:05:38] !log flushing mobile varnish cache [17:05:47] Logged the message, Master [17:06:20] awjr: done [17:07:12] mutante: awesome thanks :) looks like it workeds [17:07:24] yw [17:10:17] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [17:14:38] exit [17:15:14] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [17:18:59] PROBLEM - udp2log log age for locke on locke is CRITICAL: CRITICAL: log files /a/squid/fundraising/logs/bannerImpressions-sampled100.log, have not been written in a critical amount of time. For most logs, this is 4 hours. For slow logs, this is 4 days. [17:20:11] PROBLEM - check_mysql on payments1001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) [17:25:17] RECOVERY - check_mysql on payments1001 is OK: Uptime: 1454735 Threads: 4 Questions: 16515 Slow queries: 15 Opens: 226 Flush tables: 4 Open tables: 30 Queries per second avg: 0.011 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [17:32:46] New review: Hashar; "Good for me." [operations/mediawiki-config] (master); V: 1 C: 1; - https://gerrit.wikimedia.org/r/23652 [17:38:07] New review: Hashar; "I find the proposed logo to be very horrible. What about using the "piece" icon already used on seve..." [operations/mediawiki-config] (master); V: 0 C: -1; - https://gerrit.wikimedia.org/r/23610 [17:43:49] New patchset: Hashar; "migrate test favicon from upload to bits" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/23675 [17:45:32] RECOVERY - udp2log log age for locke on locke is OK: OK: all log files active [18:00:22] New review: Hashar; "Note that I27a795f6 migrates the favicon to bits.wm.org. So the favicon4.ico in this change will not..." [operations/mediawiki-config] (master); V: 0 C: 0; - https://gerrit.wikimedia.org/r/16935 [18:39:01] New patchset: DamianZaremba; "Removing value output so status info is useful + misc whitespace fix" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23681 [18:39:57] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/23681 [18:47:51] !log authdns-update for wikimedia.biz [18:48:00] Logged the message, RobH [18:53:15] New review: Hashar; "I guess I now need to find some op to merge the change :)" [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/22673 [19:00:27] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [19:00:27] PROBLEM - Puppet freshness on singer is CRITICAL: Puppet has not run in the last 10 hours [19:00:27] PROBLEM - Puppet freshness on virt1001 is CRITICAL: Puppet has not run in the last 10 hours [19:00:27] PROBLEM - Puppet freshness on virt1002 is CRITICAL: Puppet has not run in the last 10 hours [19:00:27] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [19:00:28] PROBLEM - Puppet freshness on virt1003 is CRITICAL: Puppet has not run in the last 10 hours [19:39:27] PROBLEM - Puppet freshness on ms-be10 is CRITICAL: Puppet has not run in the last 10 hours [20:44:08] New patchset: Dzahn; "add Chris Steipp to admins.pp and class mortals" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23718 [20:45:13] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/23718 [20:47:49] New review: RobH; "looks good, matches RT request, merging" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/23718 [20:47:52] Change merged: RobH; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23718 [20:48:07] New patchset: Dzahn; "change UID for spage to 608, 607 was already used for krinkle" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23748 [20:49:23] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/23748 [20:50:44] New review: Dzahn; "useraccount is not in any class or on bastion host yet, fix this before it is in use" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/23748 [20:50:45] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23748 [20:56:24] PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours [21:02:23] !log fixing UID for spage on stat1 (/etc/passwd and files), was used twice [21:02:32] Logged the message, Master [21:04:21] PROBLEM - udp2log log age for locke on locke is CRITICAL: CRITICAL: log files /a/squid/fundraising/logs/bannerImpressions-sampled100.log, have not been written in a critical amount of time. For most logs, this is 4 hours. For slow logs, this is 4 days. [21:26:12] New patchset: Demon; "Initial commit of the new deployment system" [operations/deployment] (master) - https://gerrit.wikimedia.org/r/8732 [21:27:06] New review: Demon; "PS5 rebases to account for .gitreview already existing. No other changes." [operations/deployment] (master); V: 0 C: 0; - https://gerrit.wikimedia.org/r/8732 [21:29:23] New patchset: Dzahn; "sort useraccounts alphabetically within classes" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23761 [21:30:17] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/23761 [21:31:26] New patchset: Dzahn; "sort useraccounts alphabetically within classes" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23761 [21:32:19] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/23761 [21:35:49] New patchset: Ottomata; "Adding bi-filter to list of slow logs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23762 [21:36:42] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/23762 [21:46:52] PROBLEM - Puppet freshness on ms-be7 is CRITICAL: Puppet has not run in the last 10 hours [22:01:48] Change merged: Jgreen; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23762 [22:02:06] New patchset: Asher; "file_per_table for new es clusters" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23765 [22:02:59] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/23765 [22:04:21] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23765 [22:10:45] New patchset: Asher; "excluding es clusters from heartbeat monitoring for now" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23767 [22:11:38] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/23767 [22:11:55] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23767 [22:22:16] RECOVERY - mysqld processes on es6 is OK: PROCS OK: 1 process with command name mysqld [22:23:37] PROBLEM - mysqld processes on es5 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld [22:38:46] RECOVERY - mysqld processes on es7 is OK: PROCS OK: 1 process with command name mysqld [22:41:28] RECOVERY - mysqld processes on es1005 is OK: PROCS OK: 1 process with command name mysqld [22:43:34] RECOVERY - mysqld processes on es1006 is OK: PROCS OK: 1 process with command name mysqld [22:47:29] RECOVERY - mysqld processes on es1007 is OK: PROCS OK: 1 process with command name mysqld [22:53:43] New patchset: Dzahn; "add redirect for wikimedia.biz -> wikimedia.org" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/23768 [22:55:49] New review: Dzahn; "per RT-3371" [operations/apache-config] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/23768 [22:55:49] Change merged: Dzahn; [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/23768 [22:58:59] RECOVERY - mysqld processes on es5 is OK: PROCS OK: 1 process with command name mysqld [23:04:17] !log apache-graceful-all, pushing out redirect for wikimedia.biz [23:04:26] Logged the message, Master [23:04:59] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [23:15:29] RECOVERY - udp2log log age for locke on locke is OK: OK: all log files active [23:20:51] New patchset: Hashar; "mw logger needs a default directory" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/23770 [23:21:43] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/23770 [23:23:49] ahha [23:23:56] you guys keep refactoring stuff in puppet :-D [23:24:04] misc::scripts no more exist duh [23:32:49] misc::deployment::scripts does the job :-]