[00:02:59] PROBLEM - Puppet freshness on cp4001 is CRITICAL: No successful Puppet run in the last 10 hours [00:09:41] AaronSchulz: sent an email about drinks [00:09:42] :D [00:10:00] laziest way to do it. [00:22:59] PROBLEM - Puppet freshness on cp4014 is CRITICAL: No successful Puppet run in the last 10 hours [00:23:59] PROBLEM - Puppet freshness on cp4019 is CRITICAL: No successful Puppet run in the last 10 hours [00:24:59] PROBLEM - Puppet freshness on cp4005 is CRITICAL: No successful Puppet run in the last 10 hours [00:24:59] PROBLEM - Puppet freshness on cp4015 is CRITICAL: No successful Puppet run in the last 10 hours [00:26:59] PROBLEM - Puppet freshness on cp4017 is CRITICAL: No successful Puppet run in the last 10 hours [00:26:59] PROBLEM - Puppet freshness on lvs4001 is CRITICAL: No successful Puppet run in the last 10 hours [02:09:45] !log LocalisationUpdate completed (1.22wmf20) at Sat Oct 12 02:09:44 UTC 2013 [02:10:04] Logged the message, Master [02:13:59] PROBLEM - Puppet freshness on neon is CRITICAL: No successful Puppet run in the last 10 hours [02:15:59] PROBLEM - Puppet freshness on cp4002 is CRITICAL: No successful Puppet run in the last 10 hours [02:15:59] PROBLEM - Puppet freshness on bast4001 is CRITICAL: No successful Puppet run in the last 10 hours [02:15:59] PROBLEM - Puppet freshness on cp4003 is CRITICAL: No successful Puppet run in the last 10 hours [02:15:59] PROBLEM - Puppet freshness on cp4007 is CRITICAL: No successful Puppet run in the last 10 hours [02:15:59] PROBLEM - Puppet freshness on cp4008 is CRITICAL: No successful Puppet run in the last 10 hours [02:15:59] PROBLEM - Puppet freshness on cp4004 is CRITICAL: No successful Puppet run in the last 10 hours [02:15:59] PROBLEM - Puppet freshness on cp4006 is CRITICAL: No successful Puppet run in the last 10 hours [02:16:00] PROBLEM - Puppet freshness on cp4009 is CRITICAL: No successful Puppet run in the last 10 hours [02:16:00] PROBLEM - Puppet freshness on cp4011 is CRITICAL: No successful Puppet run in the last 10 hours [02:16:01] PROBLEM - Puppet freshness on cp4012 is CRITICAL: No successful Puppet run in the last 10 hours [02:16:01] PROBLEM - Puppet freshness on cp4010 is CRITICAL: No successful Puppet run in the last 10 hours [02:16:02] PROBLEM - Puppet freshness on cp4013 is CRITICAL: No successful Puppet run in the last 10 hours [02:16:02] PROBLEM - Puppet freshness on cp4016 is CRITICAL: No successful Puppet run in the last 10 hours [02:16:03] PROBLEM - Puppet freshness on cp4018 is CRITICAL: No successful Puppet run in the last 10 hours [02:16:03] PROBLEM - Puppet freshness on cp4020 is CRITICAL: No successful Puppet run in the last 10 hours [02:16:04] PROBLEM - Puppet freshness on lvs4002 is CRITICAL: No successful Puppet run in the last 10 hours [02:16:04] PROBLEM - Puppet freshness on lvs4003 is CRITICAL: No successful Puppet run in the last 10 hours [02:16:05] PROBLEM - Puppet freshness on lvs4004 is CRITICAL: No successful Puppet run in the last 10 hours [02:17:30] !log LocalisationUpdate completed (1.22wmf21) at Sat Oct 12 02:17:30 UTC 2013 [02:17:43] Logged the message, Master [02:29:58] !log LocalisationUpdate ResourceLoader cache refresh completed at Sat Oct 12 02:29:58 UTC 2013 [02:30:10] Logged the message, Master [03:31:09] PROBLEM - Host wikidata-lb.esams.wikimedia.org_ipv6 is DOWN: PING CRITICAL - Packet loss = 100% [03:31:49] RECOVERY - Host wikidata-lb.esams.wikimedia.org_ipv6 is UP: PING OK - Packet loss = 0%, RTA = 87.22 ms [10:03:59] PROBLEM - Puppet freshness on cp4001 is CRITICAL: No successful Puppet run in the last 10 hours [10:23:59] PROBLEM - Puppet freshness on cp4014 is CRITICAL: No successful Puppet run in the last 10 hours [10:24:59] PROBLEM - Puppet freshness on cp4019 is CRITICAL: No successful Puppet run in the last 10 hours [10:25:59] PROBLEM - Puppet freshness on cp4005 is CRITICAL: No successful Puppet run in the last 10 hours [10:25:59] PROBLEM - Puppet freshness on cp4015 is CRITICAL: No successful Puppet run in the last 10 hours [10:27:59] PROBLEM - Puppet freshness on cp4017 is CRITICAL: No successful Puppet run in the last 10 hours [10:27:59] PROBLEM - Puppet freshness on lvs4001 is CRITICAL: No successful Puppet run in the last 10 hours [12:14:59] PROBLEM - Puppet freshness on neon is CRITICAL: No successful Puppet run in the last 10 hours [12:16:59] PROBLEM - Puppet freshness on bast4001 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:59] PROBLEM - Puppet freshness on cp4002 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:59] PROBLEM - Puppet freshness on cp4003 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:59] PROBLEM - Puppet freshness on cp4006 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:59] PROBLEM - Puppet freshness on cp4004 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:59] PROBLEM - Puppet freshness on cp4007 is CRITICAL: No successful Puppet run in the last 10 hours [12:16:59] PROBLEM - Puppet freshness on cp4008 is CRITICAL: No successful Puppet run in the last 10 hours [12:17:00] PROBLEM - Puppet freshness on cp4010 is CRITICAL: No successful Puppet run in the last 10 hours [12:17:00] PROBLEM - Puppet freshness on cp4009 is CRITICAL: No successful Puppet run in the last 10 hours [12:17:01] PROBLEM - Puppet freshness on cp4011 is CRITICAL: No successful Puppet run in the last 10 hours [12:17:01] PROBLEM - Puppet freshness on cp4012 is CRITICAL: No successful Puppet run in the last 10 hours [12:17:02] PROBLEM - Puppet freshness on cp4013 is CRITICAL: No successful Puppet run in the last 10 hours [12:17:02] PROBLEM - Puppet freshness on cp4016 is CRITICAL: No successful Puppet run in the last 10 hours [12:17:03] PROBLEM - Puppet freshness on cp4018 is CRITICAL: No successful Puppet run in the last 10 hours [12:17:03] PROBLEM - Puppet freshness on cp4020 is CRITICAL: No successful Puppet run in the last 10 hours [12:17:04] PROBLEM - Puppet freshness on lvs4002 is CRITICAL: No successful Puppet run in the last 10 hours [12:17:04] PROBLEM - Puppet freshness on lvs4003 is CRITICAL: No successful Puppet run in the last 10 hours [12:17:05] PROBLEM - Puppet freshness on lvs4004 is CRITICAL: No successful Puppet run in the last 10 hours [14:00:26] (03PS1) 10coren: Tool Labs: tweaks to the lighttpd startup script [operations/puppet] - 10https://gerrit.wikimedia.org/r/89402 [14:01:30] (03CR) 10coren: [C: 032] Tool Labs: tweaks to the lighttpd startup script [operations/puppet] - 10https://gerrit.wikimedia.org/r/89402 (owner: 10coren) [15:11:08] who is in charge of setting up maillists? [15:14:57] (03CR) 10Andrew Bogott: [C: 032] Rearrange mysql module my.cnf defaults. [operations/puppet] - 10https://gerrit.wikimedia.org/r/89346 (owner: 10Andrew Bogott) [15:19:07] (03PS1) 10Andrew Bogott: Change role::labs-mysql-server to use the mysql module. [operations/puppet] - 10https://gerrit.wikimedia.org/r/89404 [15:51:10] Danny_B: err, mostly Thehelpfulone. depends [15:52:05] and doesn't matter, because you have to ask on bugzilla and wait anyway [15:52:05] thanks [16:22:58] Danny_B, if I missed a list, feel free to poke me [16:33:39] Thehelpfulone: added a few tix to mailman master [16:34:04] ah, thanks [17:31:20] (03PS1) 10coren: Tool Labs: completely rejigger the lighttpd setup [operations/puppet] - 10https://gerrit.wikimedia.org/r/89416 [17:33:22] (03CR) 10coren: [C: 032] Tool Labs: completely rejigger the lighttpd setup [operations/puppet] - 10https://gerrit.wikimedia.org/r/89416 (owner: 10coren) [17:42:00] (03PS1) 10coren: Tool Labs: lighttpd: move to final paths [operations/puppet] - 10https://gerrit.wikimedia.org/r/89417 [17:43:00] (03CR) 10coren: [C: 032] Tool Labs: lighttpd: move to final paths [operations/puppet] - 10https://gerrit.wikimedia.org/r/89417 (owner: 10coren) [17:48:09] PROBLEM - MySQL Slave Delay on db1026 is CRITICAL: CRIT replication delay 323 seconds [17:48:09] PROBLEM - MySQL Replication Heartbeat on db1026 is CRITICAL: CRIT replication delay 325 seconds [18:02:19] (03PS1) 10coren: Tool Labs: Tweaks to portgranter [operations/puppet] - 10https://gerrit.wikimedia.org/r/89420 [18:03:15] (03CR) 10coren: [C: 032] Tool Labs: Tweaks to portgranter [operations/puppet] - 10https://gerrit.wikimedia.org/r/89420 (owner: 10coren) [18:09:09] (03PS1) 10Yuvipanda: Toollabs: Add sbt to dev environments [operations/puppet] - 10https://gerrit.wikimedia.org/r/89421 [18:09:25] Coren|Away: ^ trivial chenge [18:10:05] (03CR) 10coren: [C: 032] Toollabs: Add sbt to dev environments [operations/puppet] - 10https://gerrit.wikimedia.org/r/89421 (owner: 10Yuvipanda) [18:10:12] ty Coren|Away [18:10:22] (03PS2) 10coren: Toollabs: Add sbt to dev environments [operations/puppet] - 10https://gerrit.wikimedia.org/r/89421 (owner: 10Yuvipanda) [18:11:36] (03CR) 10coren: [C: 032] Toollabs: Add sbt to dev environments [operations/puppet] - 10https://gerrit.wikimedia.org/r/89421 (owner: 10Yuvipanda) [18:14:09] RECOVERY - MySQL Slave Delay on db1026 is OK: OK replication delay 25 seconds [18:14:09] RECOVERY - MySQL Replication Heartbeat on db1026 is OK: OK replication delay 13 seconds [19:21:29] PROBLEM - Disk space on labsdb1003 is CRITICAL: DISK CRITICAL - free space: /a 125011 MB (3% inode=99%): [20:04:59] PROBLEM - Puppet freshness on cp4001 is CRITICAL: No successful Puppet run in the last 10 hours [20:24:59] PROBLEM - Puppet freshness on cp4014 is CRITICAL: No successful Puppet run in the last 10 hours [20:25:59] PROBLEM - Puppet freshness on cp4019 is CRITICAL: No successful Puppet run in the last 10 hours [20:27:03] PROBLEM - Puppet freshness on cp4005 is CRITICAL: No successful Puppet run in the last 10 hours [20:27:03] PROBLEM - Puppet freshness on cp4015 is CRITICAL: No successful Puppet run in the last 10 hours [20:28:59] PROBLEM - Puppet freshness on cp4017 is CRITICAL: No successful Puppet run in the last 10 hours [20:28:59] PROBLEM - Puppet freshness on lvs4001 is CRITICAL: No successful Puppet run in the last 10 hours [20:29:47] (03PS1) 10coren: Tool Labs: minor bugfix to portgrabber [operations/puppet] - 10https://gerrit.wikimedia.org/r/89479 [20:37:23] (03CR) 10coren: [C: 032] Tool Labs: minor bugfix to portgrabber [operations/puppet] - 10https://gerrit.wikimedia.org/r/89479 (owner: 10coren) [21:10:23] (03PS1) 10Reedy: sysop -> 'crat for ptwiki fr settings [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89482 [21:10:38] (03CR) 10Reedy: [C: 032] sysop -> 'crat for ptwiki fr settings [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89482 (owner: 10Reedy) [21:10:49] (03Merged) 10jenkins-bot: sysop -> 'crat for ptwiki fr settings [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89482 (owner: 10Reedy) [21:11:30] !log reedy synchronized wmf-config/flaggedrevs.php [21:11:50] Logged the message, Master [21:13:58] Coren|Away: did you see https://bugzilla.wikimedia.org/show_bug.cgi?id=49350 ? [21:15:34] Oh, you did. [22:15:59] PROBLEM - Puppet freshness on neon is CRITICAL: No successful Puppet run in the last 10 hours [22:17:59] PROBLEM - Puppet freshness on bast4001 is CRITICAL: No successful Puppet run in the last 10 hours [22:17:59] PROBLEM - Puppet freshness on cp4002 is CRITICAL: No successful Puppet run in the last 10 hours [22:17:59] PROBLEM - Puppet freshness on cp4003 is CRITICAL: No successful Puppet run in the last 10 hours [22:17:59] PROBLEM - Puppet freshness on cp4004 is CRITICAL: No successful Puppet run in the last 10 hours [22:17:59] PROBLEM - Puppet freshness on cp4006 is CRITICAL: No successful Puppet run in the last 10 hours [22:17:59] PROBLEM - Puppet freshness on cp4007 is CRITICAL: No successful Puppet run in the last 10 hours [22:17:59] PROBLEM - Puppet freshness on cp4008 is CRITICAL: No successful Puppet run in the last 10 hours [22:18:00] PROBLEM - Puppet freshness on cp4009 is CRITICAL: No successful Puppet run in the last 10 hours [22:18:00] PROBLEM - Puppet freshness on cp4011 is CRITICAL: No successful Puppet run in the last 10 hours [22:18:01] PROBLEM - Puppet freshness on cp4010 is CRITICAL: No successful Puppet run in the last 10 hours [22:18:01] PROBLEM - Puppet freshness on cp4012 is CRITICAL: No successful Puppet run in the last 10 hours [22:18:02] PROBLEM - Puppet freshness on cp4013 is CRITICAL: No successful Puppet run in the last 10 hours [22:18:02] PROBLEM - Puppet freshness on cp4016 is CRITICAL: No successful Puppet run in the last 10 hours [22:18:03] PROBLEM - Puppet freshness on cp4018 is CRITICAL: No successful Puppet run in the last 10 hours [22:18:03] PROBLEM - Puppet freshness on cp4020 is CRITICAL: No successful Puppet run in the last 10 hours [22:18:04] PROBLEM - Puppet freshness on lvs4002 is CRITICAL: No successful Puppet run in the last 10 hours [22:18:04] PROBLEM - Puppet freshness on lvs4003 is CRITICAL: No successful Puppet run in the last 10 hours [22:18:05] PROBLEM - Puppet freshness on lvs4004 is CRITICAL: No successful Puppet run in the last 10 hours [23:21:11] (03Abandoned) 10Ori.livneh: Add 'static-current' w/symlinks to /usr/local/apache/common-local/php [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89024 (owner: 10Ori.livneh) [23:42:03] (03PS1) 10Ryan Lane: Remove redundant sartoris definition [operations/puppet] - 10https://gerrit.wikimedia.org/r/89490 [23:42:59] PROBLEM - SSH on lvs4001 is CRITICAL: Server answer: [23:43:59] RECOVERY - SSH on lvs4001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [23:52:44] (03CR) 10Ryan Lane: [C: 032] Remove redundant sartoris definition [operations/puppet] - 10https://gerrit.wikimedia.org/r/89490 (owner: 10Ryan Lane)