[01:25:59] PROBLEM - SSH on lvs4001 is CRITICAL: Server answer: [01:27:59] RECOVERY - SSH on lvs4001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [02:07:59] PROBLEM - Puppet freshness on cp4001 is CRITICAL: No successful Puppet run in the last 10 hours [02:08:37] !log LocalisationUpdate completed (1.22wmf20) at Mon Oct 14 02:08:37 UTC 2013 [02:08:56] Logged the message, Master [02:15:08] !log LocalisationUpdate completed (1.22wmf21) at Mon Oct 14 02:15:08 UTC 2013 [02:15:22] Logged the message, Master [02:25:25] !log LocalisationUpdate ResourceLoader cache refresh completed at Mon Oct 14 02:25:24 UTC 2013 [02:25:38] Logged the message, Master [02:27:59] PROBLEM - Puppet freshness on cp4014 is CRITICAL: No successful Puppet run in the last 10 hours [02:28:59] PROBLEM - Puppet freshness on cp4019 is CRITICAL: No successful Puppet run in the last 10 hours [02:29:59] PROBLEM - Puppet freshness on cp4005 is CRITICAL: No successful Puppet run in the last 10 hours [02:29:59] PROBLEM - Puppet freshness on cp4015 is CRITICAL: No successful Puppet run in the last 10 hours [02:31:59] PROBLEM - Puppet freshness on cp4017 is CRITICAL: No successful Puppet run in the last 10 hours [02:31:59] PROBLEM - Puppet freshness on lvs4001 is CRITICAL: No successful Puppet run in the last 10 hours [02:38:59] PROBLEM - Puppet freshness on es10 is CRITICAL: No successful Puppet run in the last 10 hours [02:42:59] PROBLEM - Puppet freshness on ms-be1006 is CRITICAL: No successful Puppet run in the last 10 hours [02:42:59] PROBLEM - Puppet freshness on ms-fe3 is CRITICAL: No successful Puppet run in the last 10 hours [02:42:59] PROBLEM - Puppet freshness on mw43 is CRITICAL: No successful Puppet run in the last 10 hours [02:43:59] PROBLEM - Puppet freshness on db1031 is CRITICAL: No successful Puppet run in the last 10 hours [02:46:59] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [02:46:59] PROBLEM - Puppet freshness on searchidx1001 is CRITICAL: No successful Puppet run in the last 10 hours [02:48:59] PROBLEM - Puppet freshness on amslvs4 is CRITICAL: No successful Puppet run in the last 10 hours [02:51:59] PROBLEM - MySQL Replication Heartbeat on db46 is CRITICAL: CRIT replication delay 312 seconds [02:51:59] PROBLEM - MySQL Replication Heartbeat on db47 is CRITICAL: CRIT replication delay 312 seconds [02:52:39] PROBLEM - MySQL Replication Heartbeat on db50 is CRITICAL: CRIT replication delay 355 seconds [02:52:59] PROBLEM - Puppet freshness on virt2 is CRITICAL: No successful Puppet run in the last 10 hours [02:53:59] PROBLEM - Puppet freshness on search27 is CRITICAL: No successful Puppet run in the last 10 hours [02:54:39] RECOVERY - MySQL Replication Heartbeat on db50 is OK: OK replication delay 0 seconds [02:54:59] PROBLEM - Puppet freshness on analytics1018 is CRITICAL: No successful Puppet run in the last 10 hours [02:54:59] RECOVERY - MySQL Replication Heartbeat on db46 is OK: OK replication delay 0 seconds [02:54:59] RECOVERY - MySQL Replication Heartbeat on db47 is OK: OK replication delay 0 seconds [02:55:59] PROBLEM - Puppet freshness on virt8 is CRITICAL: No successful Puppet run in the last 10 hours [03:00:59] PROBLEM - Puppet freshness on mw1109 is CRITICAL: No successful Puppet run in the last 10 hours [03:01:59] PROBLEM - Puppet freshness on mw1067 is CRITICAL: No successful Puppet run in the last 10 hours [03:01:59] PROBLEM - Puppet freshness on pc1001 is CRITICAL: No successful Puppet run in the last 10 hours [03:02:59] PROBLEM - Puppet freshness on amslvs3 is CRITICAL: No successful Puppet run in the last 10 hours [03:02:59] PROBLEM - Puppet freshness on mw31 is CRITICAL: No successful Puppet run in the last 10 hours [03:05:59] PROBLEM - Puppet freshness on cp1061 is CRITICAL: No successful Puppet run in the last 10 hours [03:05:59] PROBLEM - Puppet freshness on lvs3 is CRITICAL: No successful Puppet run in the last 10 hours [03:10:59] PROBLEM - Puppet freshness on srv255 is CRITICAL: No successful Puppet run in the last 10 hours [03:23:59] PROBLEM - Puppet freshness on cp3006 is CRITICAL: No successful Puppet run in the last 10 hours [03:23:59] PROBLEM - Puppet freshness on mw1 is CRITICAL: No successful Puppet run in the last 10 hours [03:23:59] PROBLEM - Puppet freshness on mw59 is CRITICAL: No successful Puppet run in the last 10 hours [03:23:59] PROBLEM - Puppet freshness on search1016 is CRITICAL: No successful Puppet run in the last 10 hours [03:26:59] PROBLEM - Puppet freshness on ms-be7 is CRITICAL: No successful Puppet run in the last 10 hours [03:27:59] PROBLEM - Puppet freshness on db1039 is CRITICAL: No successful Puppet run in the last 10 hours [03:29:59] PROBLEM - Puppet freshness on mw1050 is CRITICAL: No successful Puppet run in the last 10 hours [03:29:59] PROBLEM - Puppet freshness on virt7 is CRITICAL: No successful Puppet run in the last 10 hours [03:31:19] (03PS1) 10Springle: depool db1027 before promotion [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89613 [03:31:54] (03CR) 10Springle: [C: 032] depool db1027 before promotion [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89613 (owner: 10Springle) [03:33:34] !log springle synchronized wmf-config/db-eqiad.php 'depool db1027 before promotion' [03:33:46] Logged the message, Master [03:33:59] PROBLEM - Puppet freshness on es1010 is CRITICAL: No successful Puppet run in the last 10 hours [03:37:59] PROBLEM - Puppet freshness on lvs4 is CRITICAL: No successful Puppet run in the last 10 hours [03:40:39] RECOVERY - Puppet freshness on ms-fe3 is OK: puppet ran at Mon Oct 14 03:40:31 UTC 2013 [03:49:59] PROBLEM - Puppet freshness on analytics1008 is CRITICAL: No successful Puppet run in the last 10 hours [03:50:59] PROBLEM - Puppet freshness on search1023 is CRITICAL: No successful Puppet run in the last 10 hours [03:59:36] Anyone around? Had a few reports of possible mail issues. [04:00:06] I can send email within WMF/Gmail, but my external test didn't get through. [04:01:54] * springle is around but knows nothing of mail... [04:02:54] ksnider: works for me: http://www.mytrashmail.com/MyTrashMail_message.aspx?email=wmf-email-test&id=138789521 [04:03:51] Sorry, I should have specified *inbound* mail. [04:06:59] PROBLEM - Puppet freshness on mw1002 is CRITICAL: No successful Puppet run in the last 10 hours [04:06:59] PROBLEM - Puppet freshness on mw1126 is CRITICAL: No successful Puppet run in the last 10 hours [04:09:59] PROBLEM - Puppet freshness on mw58 is CRITICAL: No successful Puppet run in the last 10 hours [04:10:59] PROBLEM - Puppet freshness on srv273 is CRITICAL: No successful Puppet run in the last 10 hours [04:14:59] PROBLEM - Puppet freshness on mw1204 is CRITICAL: No successful Puppet run in the last 10 hours [04:18:57] (03PS1) 10Springle: depool db1006 to investigate connection errors. promote db1027 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89615 [04:18:59] PROBLEM - Puppet freshness on neon is CRITICAL: No successful Puppet run in the last 10 hours [04:20:00] (03CR) 10Springle: [C: 032] depool db1006 to investigate connection errors. promote db1027 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89615 (owner: 10Springle) [04:20:39] RECOVERY - Puppet freshness on cp3006 is OK: puppet ran at Mon Oct 14 04:20:33 UTC 2013 [04:20:59] PROBLEM - Puppet freshness on bast4001 is CRITICAL: No successful Puppet run in the last 10 hours [04:20:59] PROBLEM - Puppet freshness on cp4002 is CRITICAL: No successful Puppet run in the last 10 hours [04:20:59] PROBLEM - Puppet freshness on cp4003 is CRITICAL: No successful Puppet run in the last 10 hours [04:20:59] PROBLEM - Puppet freshness on cp4004 is CRITICAL: No successful Puppet run in the last 10 hours [04:20:59] PROBLEM - Puppet freshness on cp4006 is CRITICAL: No successful Puppet run in the last 10 hours [04:20:59] PROBLEM - Puppet freshness on cp4007 is CRITICAL: No successful Puppet run in the last 10 hours [04:20:59] PROBLEM - Puppet freshness on cp4008 is CRITICAL: No successful Puppet run in the last 10 hours [04:21:00] PROBLEM - Puppet freshness on cp4009 is CRITICAL: No successful Puppet run in the last 10 hours [04:21:00] PROBLEM - Puppet freshness on cp4010 is CRITICAL: No successful Puppet run in the last 10 hours [04:21:01] PROBLEM - Puppet freshness on cp4011 is CRITICAL: No successful Puppet run in the last 10 hours [04:21:01] PROBLEM - Puppet freshness on cp4012 is CRITICAL: No successful Puppet run in the last 10 hours [04:21:02] PROBLEM - Puppet freshness on cp4013 is CRITICAL: No successful Puppet run in the last 10 hours [04:21:02] PROBLEM - Puppet freshness on cp4016 is CRITICAL: No successful Puppet run in the last 10 hours [04:21:03] PROBLEM - Puppet freshness on cp4018 is CRITICAL: No successful Puppet run in the last 10 hours [04:21:03] PROBLEM - Puppet freshness on cp4020 is CRITICAL: No successful Puppet run in the last 10 hours [04:21:04] PROBLEM - Puppet freshness on lvs4002 is CRITICAL: No successful Puppet run in the last 10 hours [04:21:04] PROBLEM - Puppet freshness on lvs4003 is CRITICAL: No successful Puppet run in the last 10 hours [04:21:05] PROBLEM - Puppet freshness on lvs4004 is CRITICAL: No successful Puppet run in the last 10 hours [04:21:05] PROBLEM - Puppet freshness on sq64 is CRITICAL: No successful Puppet run in the last 10 hours [04:21:59] PROBLEM - Puppet freshness on mw1134 is CRITICAL: No successful Puppet run in the last 10 hours [04:22:59] PROBLEM - Puppet freshness on analytics1017 is CRITICAL: No successful Puppet run in the last 10 hours [04:24:33] !log springle synchronized wmf-config/db-eqiad.php 'depool db1006 to investigate connection errors. promote db1027' [04:24:47] Logged the message, Master [04:26:59] PROBLEM - Puppet freshness on mw1177 is CRITICAL: No successful Puppet run in the last 10 hours [04:27:19] RECOVERY - Disk space on stafford is OK: DISK OK [04:28:51] (03PS1) 10Springle: db1027 is s6 master [operations/puppet] - 10https://gerrit.wikimedia.org/r/89616 [04:28:59] PROBLEM - MySQL Replication Heartbeat on db46 is CRITICAL: CRIT replication delay 311 seconds [04:28:59] PROBLEM - MySQL Replication Heartbeat on db47 is CRITICAL: CRIT replication delay 311 seconds [04:28:59] RECOVERY - Puppet freshness on virt7 is OK: puppet ran at Mon Oct 14 04:28:55 UTC 2013 [04:29:09] RECOVERY - Puppet freshness on mw1050 is OK: puppet ran at Mon Oct 14 04:29:00 UTC 2013 [04:29:09] PROBLEM - MySQL Replication Heartbeat on db1022 is CRITICAL: CRIT replication delay 324 seconds [04:29:09] PROBLEM - MySQL Replication Heartbeat on db1040 is CRITICAL: CRIT replication delay 324 seconds [04:29:29] PROBLEM - MySQL Replication Heartbeat on db1015 is CRITICAL: CRIT replication delay 344 seconds [04:29:39] PROBLEM - MySQL Replication Heartbeat on db50 is CRITICAL: CRIT replication delay 354 seconds [04:29:59] PROBLEM - Puppet freshness on mw1149 is CRITICAL: No successful Puppet run in the last 10 hours [04:32:49] RECOVERY - Puppet freshness on es1010 is OK: puppet ran at Mon Oct 14 04:32:46 UTC 2013 [04:32:57] (03PS1) 10Springle: db1027 promoted [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89618 [04:33:51] (03CR) 10Springle: [C: 032] db1027 promoted [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89618 (owner: 10Springle) [04:33:59] PROBLEM - Puppet freshness on mw22 is CRITICAL: No successful Puppet run in the last 10 hours [04:33:59] PROBLEM - Puppet freshness on search15 is CRITICAL: No successful Puppet run in the last 10 hours [04:34:45] !log springle synchronized wmf-config/db-pmtpa.php 'update s6 master' [04:34:56] Logged the message, Master [04:35:49] RECOVERY - Puppet freshness on search15 is OK: puppet ran at Mon Oct 14 04:35:47 UTC 2013 [04:35:59] RECOVERY - Puppet freshness on mw22 is OK: puppet ran at Mon Oct 14 04:35:52 UTC 2013 [04:37:29] RECOVERY - Puppet freshness on lvs4 is OK: puppet ran at Mon Oct 14 04:37:22 UTC 2013 [04:38:39] RECOVERY - Puppet freshness on es10 is OK: puppet ran at Mon Oct 14 04:38:32 UTC 2013 [04:42:49] RECOVERY - Puppet freshness on mw43 is OK: puppet ran at Mon Oct 14 04:42:40 UTC 2013 [04:42:49] RECOVERY - Puppet freshness on db1031 is OK: puppet ran at Mon Oct 14 04:42:40 UTC 2013 [04:44:59] RECOVERY - Puppet freshness on lvs1005 is OK: puppet ran at Mon Oct 14 04:44:50 UTC 2013 [04:45:59] PROBLEM - Puppet freshness on mw1166 is CRITICAL: No successful Puppet run in the last 10 hours [04:46:29] RECOVERY - Puppet freshness on searchidx1001 is OK: puppet ran at Mon Oct 14 04:46:26 UTC 2013 [04:47:59] RECOVERY - Puppet freshness on amslvs4 is OK: puppet ran at Mon Oct 14 04:47:51 UTC 2013 [04:49:49] RECOVERY - Puppet freshness on analytics1008 is OK: puppet ran at Mon Oct 14 04:49:42 UTC 2013 [04:49:59] RECOVERY - Puppet freshness on search1023 is OK: puppet ran at Mon Oct 14 04:49:52 UTC 2013 [04:52:49] RECOVERY - Puppet freshness on virt2 is OK: puppet ran at Mon Oct 14 04:52:47 UTC 2013 [04:53:19] RECOVERY - Puppet freshness on search27 is OK: puppet ran at Mon Oct 14 04:53:12 UTC 2013 [04:53:59] RECOVERY - Puppet freshness on analytics1018 is OK: puppet ran at Mon Oct 14 04:53:58 UTC 2013 [04:55:49] RECOVERY - Puppet freshness on virt8 is OK: puppet ran at Mon Oct 14 04:55:48 UTC 2013 [04:55:59] PROBLEM - Puppet freshness on db1056 is CRITICAL: No successful Puppet run in the last 10 hours [05:00:09] RECOVERY - Puppet freshness on mw1109 is OK: puppet ran at Mon Oct 14 05:00:04 UTC 2013 [05:00:49] RECOVERY - Puppet freshness on pc1001 is OK: puppet ran at Mon Oct 14 05:00:45 UTC 2013 [05:00:59] RECOVERY - Puppet freshness on mw1067 is OK: puppet ran at Mon Oct 14 05:00:55 UTC 2013 [05:02:09] RECOVERY - Puppet freshness on mw31 is OK: puppet ran at Mon Oct 14 05:02:02 UTC 2013 [05:02:49] RECOVERY - Puppet freshness on amslvs3 is OK: puppet ran at Mon Oct 14 05:02:48 UTC 2013 [05:05:29] RECOVERY - Puppet freshness on cp1061 is OK: puppet ran at Mon Oct 14 05:05:24 UTC 2013 [05:06:09] RECOVERY - Puppet freshness on mw1126 is OK: puppet ran at Mon Oct 14 05:05:59 UTC 2013 [05:07:39] PROBLEM - MySQL Replication Heartbeat on db1027 is CRITICAL: CRIT replication delay 2633 seconds [05:09:29] PROBLEM - MySQL Replication Heartbeat on db43 is CRITICAL: CRIT replication delay 2739 seconds [05:09:59] RECOVERY - Puppet freshness on mw58 is OK: puppet ran at Mon Oct 14 05:09:54 UTC 2013 [05:10:49] RECOVERY - Puppet freshness on srv255 is OK: puppet ran at Mon Oct 14 05:10:40 UTC 2013 [05:10:49] RECOVERY - Puppet freshness on srv273 is OK: puppet ran at Mon Oct 14 05:10:40 UTC 2013 [05:15:33] (03PS1) 10Springle: repool db1007 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89622 [05:16:32] (03CR) 10Springle: [C: 032] repool db1007 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89622 (owner: 10Springle) [05:17:40] !log springle synchronized wmf-config/db-eqiad.php 'repool db1007' [05:17:57] Logged the message, Master [05:18:07] !log springle synchronized wmf-config/db-pmtpa.php 'update s6 master' [05:20:29] RECOVERY - Puppet freshness on sq64 is OK: puppet ran at Mon Oct 14 05:20:23 UTC 2013 [05:20:49] RECOVERY - Puppet freshness on mw1134 is OK: puppet ran at Mon Oct 14 05:20:48 UTC 2013 [05:20:59] PROBLEM - Puppet freshness on ms-fe1003 is CRITICAL: No successful Puppet run in the last 10 hours [05:22:19] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:22:29] RECOVERY - Puppet freshness on analytics1017 is OK: puppet ran at Mon Oct 14 05:22:23 UTC 2013 [05:22:59] RECOVERY - Puppet freshness on mw1 is OK: puppet ran at Mon Oct 14 05:22:49 UTC 2013 [05:22:59] RECOVERY - Puppet freshness on mw59 is OK: puppet ran at Mon Oct 14 05:22:49 UTC 2013 [05:23:09] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 1 logical drive(s), 4 physical drive(s) [05:23:19] RECOVERY - Puppet freshness on search1016 is OK: puppet ran at Mon Oct 14 05:23:09 UTC 2013 [05:25:59] RECOVERY - Puppet freshness on mw1177 is OK: puppet ran at Mon Oct 14 05:25:49 UTC 2013 [05:27:29] RECOVERY - Puppet freshness on mw1149 is OK: puppet ran at Mon Oct 14 05:27:24 UTC 2013 [05:28:09] RECOVERY - Puppet freshness on ms-be7 is OK: puppet ran at Mon Oct 14 05:28:00 UTC 2013 [05:30:59] PROBLEM - Puppet freshness on mw1163 is CRITICAL: No successful Puppet run in the last 10 hours [05:38:59] PROBLEM - Puppet freshness on mw81 is CRITICAL: No successful Puppet run in the last 10 hours [05:47:41] (03PS1) 10Springle: db1027 new s6 master [operations/dns] - 10https://gerrit.wikimedia.org/r/89623 [06:16:27] (03CR) 10Springle: [C: 032] db1027 new s6 master [operations/dns] - 10https://gerrit.wikimedia.org/r/89623 (owner: 10Springle) [06:19:29] RECOVERY - MySQL Replication Heartbeat on db1015 is OK: OK replication delay -1 seconds [06:19:59] RECOVERY - MySQL Replication Heartbeat on db46 is OK: OK replication delay 0 seconds [06:19:59] RECOVERY - MySQL Replication Heartbeat on db47 is OK: OK replication delay 0 seconds [06:20:09] RECOVERY - MySQL Replication Heartbeat on db1022 is OK: OK replication delay -1 seconds [06:20:09] RECOVERY - MySQL Replication Heartbeat on db1040 is OK: OK replication delay -1 seconds [06:20:29] RECOVERY - MySQL Replication Heartbeat on db43 is OK: OK replication delay -1 seconds [06:20:39] RECOVERY - MySQL Replication Heartbeat on db50 is OK: OK replication delay 0 seconds [06:25:59] PROBLEM - Puppet freshness on mw78 is CRITICAL: No successful Puppet run in the last 10 hours [06:37:59] PROBLEM - Puppet freshness on mw49 is CRITICAL: No successful Puppet run in the last 10 hours [06:44:59] PROBLEM - Puppet freshness on mw1046 is CRITICAL: No successful Puppet run in the last 10 hours [06:45:36] !log resized /var/lib/puppet on stafford, it ran out of room last night (but recovered after cron job reports cleanup) [06:45:52] Logged the message, Master [06:54:59] PROBLEM - Puppet freshness on mw92 is CRITICAL: No successful Puppet run in the last 10 hours [07:43:59] PROBLEM - Puppet freshness on mw1048 is CRITICAL: No successful Puppet run in the last 10 hours [08:02:59] PROBLEM - Puppet freshness on yvon is CRITICAL: No successful Puppet run in the last 10 hours [08:39:49] RECOVERY - Puppet freshness on yvon is OK: puppet ran at Mon Oct 14 08:39:42 UTC 2013 [08:42:19] RECOVERY - Puppet freshness on mw92 is OK: puppet ran at Mon Oct 14 08:42:13 UTC 2013 [08:44:29] RECOVERY - Puppet freshness on mw81 is OK: puppet ran at Mon Oct 14 08:44:24 UTC 2013 [08:46:59] RECOVERY - Puppet freshness on mw78 is OK: puppet ran at Mon Oct 14 08:46:50 UTC 2013 [08:48:59] RECOVERY - Puppet freshness on mw49 is OK: puppet ran at Mon Oct 14 08:48:50 UTC 2013 [08:51:09] RECOVERY - Puppet freshness on mw1204 is OK: puppet ran at Mon Oct 14 08:51:01 UTC 2013 [08:53:29] RECOVERY - Puppet freshness on mw1166 is OK: puppet ran at Mon Oct 14 08:53:27 UTC 2013 [08:54:59] PROBLEM - Puppet freshness on ms-be1002 is CRITICAL: No successful Puppet run in the last 10 hours [08:55:29] RECOVERY - Puppet freshness on mw1163 is OK: puppet ran at Mon Oct 14 08:55:25 UTC 2013 [08:57:39] RECOVERY - Puppet freshness on mw1048 is OK: puppet ran at Mon Oct 14 08:57:32 UTC 2013 [08:59:39] RECOVERY - Puppet freshness on mw1046 is OK: puppet ran at Mon Oct 14 08:59:29 UTC 2013 [09:01:39] RECOVERY - Puppet freshness on mw1002 is OK: puppet ran at Mon Oct 14 09:01:35 UTC 2013 [09:04:39] RECOVERY - Puppet freshness on ms-fe1003 is OK: puppet ran at Mon Oct 14 09:04:35 UTC 2013 [09:06:29] RECOVERY - Puppet freshness on ms-be1006 is OK: puppet ran at Mon Oct 14 09:06:27 UTC 2013 [09:06:57] (03PS1) 10TTO: Add throttling exception for Philippines editing workshop [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89642 [09:08:29] RECOVERY - Puppet freshness on db1039 is OK: puppet ran at Mon Oct 14 09:08:22 UTC 2013 [09:09:59] RECOVERY - Puppet freshness on db1056 is OK: puppet ran at Mon Oct 14 09:09:58 UTC 2013 [09:14:19] RECOVERY - Puppet freshness on ms-be1002 is OK: puppet ran at Mon Oct 14 09:14:14 UTC 2013 [09:16:09] RECOVERY - Puppet freshness on lvs3 is OK: puppet ran at Mon Oct 14 09:16:05 UTC 2013 [09:16:59] PROBLEM - Puppet freshness on lvs6 is CRITICAL: No successful Puppet run in the last 10 hours [09:21:19] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [09:22:49] RECOVERY - Puppet freshness on lvs6 is OK: puppet ran at Mon Oct 14 09:22:39 UTC 2013 [09:23:19] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 1 logical drive(s), 4 physical drive(s) [10:29:50] (03PS1) 10ArielGlenn: remove puppet reports older than 36 hours (was 2 days) [operations/puppet] - 10https://gerrit.wikimedia.org/r/89690 [10:30:55] (03CR) 10ArielGlenn: [C: 032] remove puppet reports older than 36 hours (was 2 days) [operations/puppet] - 10https://gerrit.wikimedia.org/r/89690 (owner: 10ArielGlenn) [11:25:50] (03PS1) 10ArielGlenn: rotate passenger logs on puppetmaster [operations/puppet] - 10https://gerrit.wikimedia.org/r/89696 [11:28:04] (03CR) 10ArielGlenn: [C: 032] rotate passenger logs on puppetmaster [operations/puppet] - 10https://gerrit.wikimedia.org/r/89696 (owner: 10ArielGlenn) [11:35:10] (03PS1) 10Lcarr: adding dickson's ipv6 address and renaming it's rdns to .freenode.net. [operations/dns] - 10https://gerrit.wikimedia.org/r/89697 [11:42:38] (03CR) 10Lcarr: [C: 032] adding dickson's ipv6 address and renaming it's rdns to .freenode.net. [operations/dns] - 10https://gerrit.wikimedia.org/r/89697 (owner: 10Lcarr) [12:07:40] Reedy: here? [12:07:50] NO [12:08:59] PROBLEM - Puppet freshness on cp4001 is CRITICAL: No successful Puppet run in the last 10 hours [12:23:07] YuviPanda: are you the only one working hard? [12:28:59] PROBLEM - Puppet freshness on cp4014 is CRITICAL: No successful Puppet run in the last 10 hours [12:29:59] PROBLEM - Puppet freshness on cp4019 is CRITICAL: No successful Puppet run in the last 10 hours [12:30:59] PROBLEM - Puppet freshness on cp4005 is CRITICAL: No successful Puppet run in the last 10 hours [12:30:59] PROBLEM - Puppet freshness on cp4015 is CRITICAL: No successful Puppet run in the last 10 hours [12:31:58] yurik_: i'm not working :P [12:32:46] working for WM is never really working ... its more like an extended vacation where you get to play with your favorite toys [12:32:59] PROBLEM - Puppet freshness on cp4017 is CRITICAL: No successful Puppet run in the last 10 hours [12:32:59] PROBLEM - Puppet freshness on lvs4001 is CRITICAL: No successful Puppet run in the last 10 hours [13:14:32] (03CR) 10Nemo bis: [C: 031] "(2 comments)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86415 (owner: 10TTO) [14:00:45] yurik_: heh, true [14:00:47] yurik_: not playing with my toys either, because hand pain [14:05:09] * yurik_ bans YuviPanda for bad imagery [14:05:36] yurik_: hah! :P [14:19:00] andrewbogott: Is WMF open today? [14:19:17] what does that mean? [14:19:25] Elsie: It's a holiday in the US, so attendance (including mine) will be spotty. [14:19:28] It's a U.S. federal holiday. [14:19:34] Though it's not fully observed. [14:19:41] what does "WMF" being open means? [14:19:51] Whether the office is open. :-) [14:19:58] The office is always open [14:19:59] PROBLEM - Puppet freshness on neon is CRITICAL: No successful Puppet run in the last 10 hours [14:20:16] US employees tend to observe it, European employees less so [14:20:19] There's usually a spike in activity Monday through Friday on Bugzilla and IRC. [14:20:19] or australian etc. [14:21:02] Actually, this is something I don't know: paravoid, how are holidays handled for out-of-US reqs? Does being a 'contractor' mean that everything is handled hourly? [14:21:34] Or do you get a quota of holidays to apply to the calendar as appropriate in your locale? [14:21:59] PROBLEM - Puppet freshness on bast4001 is CRITICAL: No successful Puppet run in the last 10 hours [14:21:59] PROBLEM - Puppet freshness on cp4002 is CRITICAL: No successful Puppet run in the last 10 hours [14:21:59] PROBLEM - Puppet freshness on cp4003 is CRITICAL: No successful Puppet run in the last 10 hours [14:21:59] PROBLEM - Puppet freshness on cp4004 is CRITICAL: No successful Puppet run in the last 10 hours [14:21:59] PROBLEM - Puppet freshness on cp4006 is CRITICAL: No successful Puppet run in the last 10 hours [14:21:59] PROBLEM - Puppet freshness on cp4007 is CRITICAL: No successful Puppet run in the last 10 hours [14:21:59] PROBLEM - Puppet freshness on cp4008 is CRITICAL: No successful Puppet run in the last 10 hours [14:22:00] PROBLEM - Puppet freshness on cp4009 is CRITICAL: No successful Puppet run in the last 10 hours [14:22:00] PROBLEM - Puppet freshness on cp4010 is CRITICAL: No successful Puppet run in the last 10 hours [14:22:01] PROBLEM - Puppet freshness on cp4012 is CRITICAL: No successful Puppet run in the last 10 hours [14:22:01] PROBLEM - Puppet freshness on cp4011 is CRITICAL: No successful Puppet run in the last 10 hours [14:22:02] PROBLEM - Puppet freshness on cp4013 is CRITICAL: No successful Puppet run in the last 10 hours [14:22:02] PROBLEM - Puppet freshness on cp4016 is CRITICAL: No successful Puppet run in the last 10 hours [14:22:03] PROBLEM - Puppet freshness on cp4018 is CRITICAL: No successful Puppet run in the last 10 hours [14:22:03] PROBLEM - Puppet freshness on cp4020 is CRITICAL: No successful Puppet run in the last 10 hours [14:22:04] PROBLEM - Puppet freshness on lvs4002 is CRITICAL: No successful Puppet run in the last 10 hours [14:22:04] PROBLEM - Puppet freshness on lvs4003 is CRITICAL: No successful Puppet run in the last 10 hours [14:22:05] PROBLEM - Puppet freshness on lvs4004 is CRITICAL: No successful Puppet run in the last 10 hours [14:22:27] grrrrr... puppet freshness checks will be my next victim after multiple puppetmasters [14:23:09] Even after switching to hourly updates we're still getting those? [14:23:36] the latter [14:23:51] personally, I work on most US and Greek holidays :P [14:24:00] worst of both worlds :) [14:24:12] and weekends [14:24:16] (not because anyone tells me to, to be fair) [14:24:22] nah [14:24:32] maybe read mails, but not that much otherwise [14:24:59] Have a day off [14:25:07] Then still do your weeks hours in the next 6 days [14:26:31] i think that's something with snmp not getting in [14:27:18] i thought daniel fixed that [14:30:26] oh [14:30:35] Heh, actually, the staff HR page has a list of holidays and says 'The office will be closed on these days.' So Elsie isn't totally off :) [14:30:42] the problem is because neon hasn't been able ot run puppet ot update the iptables [14:32:51] andrewbogott: Columbus Day is usually traded with Veterans Day or the day after Thanksgiving. [14:33:49] Elsie: Weirdly WMF observes all three. Although I believe today is technically "Indigenous People's Day" [14:34:01] :D [14:34:02] And/or 'holiday we are all politely refraining from saying the name of' [14:37:54] I wonder if there's a presidential proclamation today. [14:39:10] http://www.whitehouse.gov/the-press-office/2012/10/05/presidential-proclamation-columbus-day-2012 # This is the one from last year. [14:41:35] andrewbogott: akosiaris tricked me in deploying the rspec Jenkins job for operations/puppet [14:41:42] andrewbogott: so it has been running since thursday :-] [14:41:58] but only for whitelisted users right? [14:45:37] yes [14:45:50] I was looking this morning at puppetlabs-rspec [14:45:56] to make it output a Junit format [14:46:06] seems I will have to patch their code a bit to make it possible [14:50:27] hashar, do you know about labs instance 'deployment-eventlogging' [14:50:40] andrewbogott: got setup by ori / maxsem for eventlogging [14:50:54] that is an UDP receiver to which json data are sent [14:51:10] It has mysql installed; any idea if that is actually used by the logger? [14:51:18] the json being validated by a json schema, it is put on a zero mqueue bus and ends up in some database backend [14:51:33] Um… context: I want to mess with the puppet manifests for labs mysql, looking for a 'canary' system to detect breakage. [14:51:33] the db backend is most probably mysql, might have mongoDB as well [14:51:40] ahh [14:51:55] Are you familiar enough with the system to watch while I break it? [14:52:03] not at all :/ [14:52:16] OK, I'll wait for ori or max [14:53:55] ottomata, working today? [14:54:04] half, ja [14:54:09] right now yup [14:54:13] whaaats happening? [14:54:25] Are wikimetrics or wikistats-history-01 yours? [14:54:34] no, dan's [14:54:47] but I might know some things about wikimetrics [14:55:03] Hm… do I know Dan? [14:55:08] milimetric [14:55:11] hi andrewbogott [14:55:21] are you breaking them by testing the proxy? [14:55:35] nope, just cleaning up mysql puppet manifests. [14:55:37] ah [14:55:37] ok [14:56:30] * YuviPanda goes back to attempting to not work [15:06:30] (03Abandoned) 10Hashar: Enable SUL2 in beta [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/81148 (owner: 10CSteipp) [15:10:11] (03PS2) 10Andrew Bogott: Change the labs rt role to use the mysql module. [operations/puppet] - 10https://gerrit.wikimedia.org/r/89539 [15:10:12] (03PS3) 10Andrew Bogott: Change role::labs-mysql-server to use the mysql module. [operations/puppet] - 10https://gerrit.wikimedia.org/r/89404 [15:14:49] (03PS1) 10ArielGlenn: clear out old puppet reports twice a day (gives more breathing room) [operations/puppet] - 10https://gerrit.wikimedia.org/r/89719 [15:16:39] (03CR) 10ArielGlenn: [C: 032] clear out old puppet reports twice a day (gives more breathing room) [operations/puppet] - 10https://gerrit.wikimedia.org/r/89719 (owner: 10ArielGlenn) [15:35:18] (03PS1) 10ArielGlenn: one more text -> text-varnish (wikibookslb eqiad) [operations/puppet] - 10https://gerrit.wikimedia.org/r/89722 [15:35:59] (03CR) 10ArielGlenn: [C: 032] one more text -> text-varnish (wikibookslb eqiad) [operations/puppet] - 10https://gerrit.wikimedia.org/r/89722 (owner: 10ArielGlenn) [15:43:59] RECOVERY - Puppet freshness on neon is OK: puppet ran at Mon Oct 14 15:43:53 UTC 2013 [15:46:39] RECOVERY - Puppet freshness on cp4014 is OK: puppet ran at Mon Oct 14 15:46:35 UTC 2013 [15:47:29] RECOVERY - Puppet freshness on cp4019 is OK: puppet ran at Mon Oct 14 15:47:20 UTC 2013 [15:48:39] RECOVERY - Puppet freshness on cp4005 is OK: puppet ran at Mon Oct 14 15:48:30 UTC 2013 [15:50:50] I am off, be out tomorrow [15:54:45] RECOVERY - Puppet freshness on cp4009 is OK: puppet ran at Mon Oct 14 15:54:37 UTC 2013 [15:55:42] (03PS3) 10Andrew Bogott: Change the labs rt role to use the mysql module. [operations/puppet] - 10https://gerrit.wikimedia.org/r/89539 [15:56:35] RECOVERY - Puppet freshness on cp4016 is OK: puppet ran at Mon Oct 14 15:56:32 UTC 2013 [15:56:35] RECOVERY - Puppet freshness on cp4010 is OK: puppet ran at Mon Oct 14 15:56:32 UTC 2013 [15:56:49] (03CR) 10Andrew Bogott: [C: 032] Change the labs rt role to use the mysql module. [operations/puppet] - 10https://gerrit.wikimedia.org/r/89539 (owner: 10Andrew Bogott) [15:57:31] (03Abandoned) 10Andrew Bogott: Change role::labs-mysql-server to use the mysql module. [operations/puppet] - 10https://gerrit.wikimedia.org/r/89404 (owner: 10Andrew Bogott) [15:58:25] RECOVERY - Puppet freshness on cp4013 is OK: puppet ran at Mon Oct 14 15:58:22 UTC 2013 [16:02:55] RECOVERY - Puppet freshness on cp4006 is OK: puppet ran at Mon Oct 14 16:02:50 UTC 2013 [16:05:51] RECOVERY - Puppet freshness on cp4008 is OK: puppet ran at Mon Oct 14 16:05:46 UTC 2013 [16:06:01] RECOVERY - Puppet freshness on cp4003 is OK: puppet ran at Mon Oct 14 16:05:51 UTC 2013 [16:06:21] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [16:06:41] RECOVERY - Puppet freshness on cp4004 is OK: puppet ran at Mon Oct 14 16:06:37 UTC 2013 [16:07:31] RECOVERY - Puppet freshness on cp4001 is OK: puppet ran at Mon Oct 14 16:07:22 UTC 2013 [16:08:31] RECOVERY - Puppet freshness on lvs4003 is OK: puppet ran at Mon Oct 14 16:08:22 UTC 2013 [16:11:15] !log Setup LVS service IP range aggregates on the core routers [16:11:29] Logged the message, Master [16:14:23] RECOVERY - Puppet freshness on cp4002 is OK: puppet ran at Mon Oct 14 16:14:18 UTC 2013 [16:17:33] RECOVERY - Puppet freshness on cp4015 is OK: puppet ran at Mon Oct 14 16:17:31 UTC 2013 [16:18:33] RECOVERY - Puppet freshness on lvs4001 is OK: puppet ran at Mon Oct 14 16:18:22 UTC 2013 [16:18:53] RECOVERY - Puppet freshness on cp4017 is OK: puppet ran at Mon Oct 14 16:18:43 UTC 2013 [16:23:03] RECOVERY - Puppet freshness on lvs4002 is OK: puppet ran at Mon Oct 14 16:22:54 UTC 2013 [16:26:23] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [16:27:23] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 1 logical drive(s), 4 physical drive(s) [16:28:43] RECOVERY - Puppet freshness on cp4018 is OK: puppet ran at Mon Oct 14 16:28:37 UTC 2013 [16:31:53] RECOVERY - Puppet freshness on bast4001 is OK: puppet ran at Mon Oct 14 16:31:43 UTC 2013 [16:33:43] RECOVERY - Puppet freshness on cp4020 is OK: puppet ran at Mon Oct 14 16:33:39 UTC 2013 [16:34:23] PROBLEM - SSH on cp1059 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:34:33] RECOVERY - Puppet freshness on cp4012 is OK: puppet ran at Mon Oct 14 16:34:29 UTC 2013 [16:35:14] RECOVERY - SSH on cp1059 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [16:35:33] RECOVERY - Puppet freshness on lvs4004 is OK: puppet ran at Mon Oct 14 16:35:24 UTC 2013 [16:36:23] RECOVERY - Puppet freshness on cp4011 is OK: puppet ran at Mon Oct 14 16:36:20 UTC 2013 [16:39:50] What's the story with apparmor? A fraction of our mysql installs use it, but it isn't mentioned anywhere else in puppet. [16:40:33] RECOVERY - Puppet freshness on cp4007 is OK: puppet ran at Mon Oct 14 16:40:25 UTC 2013 [16:45:42] andrewbogott: Any particular type of mysql servers? [16:46:06] Reedy: It's only referenced on labs as best I can tell. [16:46:14] A relic of times past, perhaps/ [16:47:26] (03PS1) 10Mark Bergsma: Add local SSL terminators to ulsfo bits caches [operations/puppet] - 10https://gerrit.wikimedia.org/r/89726 [16:49:11] (03CR) 10Mark Bergsma: [C: 032] Add local SSL terminators to ulsfo bits caches [operations/puppet] - 10https://gerrit.wikimedia.org/r/89726 (owner: 10Mark Bergsma) [16:52:18] Reedy: Y'think I should just ensure=>absent the current apparmor profile and forget about it? [16:59:05] we would like to do more with apparmor [16:59:18] mysql ships with its own apparmor profile in the ubuntu/debian deb i believe [16:59:26] I think we adapted that to our needs [17:05:47] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [17:08:07] PROBLEM - HTTPS on cp4001 is CRITICAL: Connection refused [17:37:27] RECOVERY - Puppet freshness on williams is OK: puppet ran at Mon Oct 14 17:37:20 UTC 2013 [17:37:47] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [17:59:06] (03PS1) 10Mark Bergsma: Fix private key installation [operations/puppet] - 10https://gerrit.wikimedia.org/r/89732 [18:00:16] (03CR) 10Mark Bergsma: [C: 032] Fix private key installation [operations/puppet] - 10https://gerrit.wikimedia.org/r/89732 (owner: 10Mark Bergsma) [18:04:07] RECOVERY - HTTPS on cp4001 is OK: OK - Certificate will expire on 01/20/2016 12:00. [18:07:48] PROBLEM - HTTPS on cp4003 is CRITICAL: Connection refused [18:07:58] PROBLEM - HTTPS on cp4004 is CRITICAL: Connection refused [18:08:28] PROBLEM - HTTPS on cp4002 is CRITICAL: Connection refused [18:20:19] anyone else see gerrit throwing "error: RPC failed; result=22, HTTP code = 405 fatal: The remote end hung up unexpectedly" when you try to use http(s) repo urls? [18:21:17] pgehres: had that happen intermittently to me. usually works when you try again [18:21:19] (I just clone from github these days if i want https) [18:21:51] YuviPanda: oh, right, totally forgot about github. That'll work. Thanks [18:22:14] yw [18:37:28] RECOVERY - Puppet freshness on williams is OK: puppet ran at Mon Oct 14 18:37:18 UTC 2013 [18:37:48] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [19:37:25] RECOVERY - Puppet freshness on williams is OK: puppet ran at Mon Oct 14 19:37:19 UTC 2013 [19:37:45] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [20:37:36] RECOVERY - Puppet freshness on williams is OK: puppet ran at Mon Oct 14 20:37:20 UTC 2013 [20:37:52] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [21:06:12] RECOVERY - Disk space on labsdb1003 is OK: DISK OK [21:37:02] (03CR) 10TTO: "(1 comment)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86415 (owner: 10TTO) [21:37:22] RECOVERY - Puppet freshness on williams is OK: puppet ran at Mon Oct 14 21:37:19 UTC 2013 [21:37:42] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [21:38:06] (03PS3) 10TTO: Miscellaneous cleanup of InitialiseSettings [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86415 [21:39:07] (03CR) 10jenkins-bot: [V: 04-1] Miscellaneous cleanup of InitialiseSettings [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86415 (owner: 10TTO) [21:40:37] (03CR) 10TTO: [C: 04-1] "Will fix this when I have more time :(" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/86415 (owner: 10TTO) [21:54:37] (03CR) 10Hashar: "Feel free to merge whenever you want. /srv/slave-scripts is not reference in Jenkins jobs yet so this change has no impact." [operations/puppet] - 10https://gerrit.wikimedia.org/r/87058 (owner: 10Hashar) [22:16:20] (03PS1) 10Ori.livneh: Declare 'Math' debug log group [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89750 [22:17:40] paravoid: if you sync that, touch fluorine:/a/mw-log/math.log & chown it to udp2log.udp2log, we'll get some logging [22:17:51] no deploy rights? [22:18:09] you don't have deploy rights? [22:18:09] I can sync it, sure. You'd still have to touch it, though. [22:18:14] oh [22:18:23] * ori-l syncs [22:18:29] (03CR) 10Ori.livneh: [C: 032] Declare 'Math' debug log group [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89750 (owner: 10Ori.livneh) [22:18:31] ori-l: is this in regards to git-deploy? [22:18:40] (03Merged) 10jenkins-bot: Declare 'Math' debug log group [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89750 (owner: 10Ori.livneh) [22:18:43] Ryan_Lane: no, logs on fluorine [22:18:49] oh. ok [22:20:24] ori-l: touch/chown is done, are you syncing or should I? [22:20:46] I'm already on it. thanks for the touch/chown [22:21:07] !log olivneh synchronized wmf-config/InitialiseSettings.php 'Enable Math debug log group' [22:21:21] Logged the message, Master [22:21:31] hm, that's quite a bit of logging [22:22:13] heh. let's keep it enabled, but i'll patch the extension itself to do less "everything is normal" logging [22:23:05] $verbosity--; [22:23:22] why is rendering so much? [22:23:42] it's been a while since I looked at this extension, but aren't things supposed to be cached pretty extensively? [22:28:37] I see a few "TeX output" then \n then nothing [22:28:50] hm maybe I should move this back to -tech [22:37:18] RECOVERY - Puppet freshness on williams is OK: puppet ran at Mon Oct 14 22:37:16 UTC 2013 [22:37:48] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [22:39:48] (03PS1) 10Andrew Bogott: Add a tiny bit of app-armor management to mysql module. [operations/puppet] - 10https://gerrit.wikimedia.org/r/89754 [22:39:50] ori-l: As I recall, there's a file system cache for renderings. [22:39:59] It never/rarely gets purged, maybe. [22:40:13] Though math has changed a lot lately. [22:41:23] (03CR) 10Andrew Bogott: [C: 032] Add a tiny bit of app-armor management to mysql module. [operations/puppet] - 10https://gerrit.wikimedia.org/r/89754 (owner: 10Andrew Bogott) [22:41:51] that is correct [22:41:56] "file system" as in swift [22:42:46] Ryan_Lane: https://wiki.mozilla.org/Security/Server_Side_TLS [22:42:48] it's really good [22:43:29] nothing new for us [23:03:35] (03PS1) 10Andrew Bogott: Dumb variable rename; attempting to mollify puppet lint. [operations/puppet] - 10https://gerrit.wikimedia.org/r/89756 [23:04:11] (03CR) 10jenkins-bot: [V: 04-1] Dumb variable rename; attempting to mollify puppet lint. [operations/puppet] - 10https://gerrit.wikimedia.org/r/89756 (owner: 10Andrew Bogott) [23:04:39] puppet lint remains unmoved [23:06:37] (03PS2) 10Andrew Bogott: Dumb variable rename; attempting to mollify puppet lint. [operations/puppet] - 10https://gerrit.wikimedia.org/r/89756 [23:08:31] (03CR) 10Andrew Bogott: [C: 032] Dumb variable rename; attempting to mollify puppet lint. [operations/puppet] - 10https://gerrit.wikimedia.org/r/89756 (owner: 10Andrew Bogott) [23:10:33] (03PS1) 10Ori.livneh: Turn off UDP logging for Math debug log group [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89757 [23:11:51] paravoid: this reverts the earlier commit. I amended my patch to the Math extension to add texvc error logging to a 'texvc' channel, but that would need to be merged and backported to be useful, so for now I'm just turning 'Math' off and not adding anything to replace it. [23:12:12] (03CR) 10Ori.livneh: [C: 032] Turn off UDP logging for Math debug log group [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89757 (owner: 10Ori.livneh) [23:12:26] (03Merged) 10jenkins-bot: Turn off UDP logging for Math debug log group [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89757 (owner: 10Ori.livneh) [23:14:22] !log olivneh synchronized wmf-config/InitialiseSettings.php 'Disable Math debug log group (found to be not pertinent & too verbose)' [23:14:33] Logged the message, Master [23:28:21] paravoid: yeah, this doc is good [23:37:24] RECOVERY - Puppet freshness on williams is OK: puppet ran at Mon Oct 14 23:37:17 UTC 2013 [23:37:44] PROBLEM - Puppet freshness on williams is CRITICAL: No successful Puppet run in the last 10 hours [23:40:28] !log tstarling synchronized php-1.22wmf20/extensions/Math/MathTexvc.php [23:40:42] Logged the message, Master [23:41:05] !log pushed out temporary checkout of extension/Math 679ce7fe2fe, will update submodule soon [23:41:16] Logged the message, Master [23:41:59] !log tstarling synchronized php-1.22wmf20/extensions/Math [23:42:14] Logged the message, Master [23:45:27] (03PS1) 10Tim Starling: Log texvc errors [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89759 [23:45:49] (03CR) 10Tim Starling: [C: 032] Log texvc errors [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89759 (owner: 10Tim Starling) [23:45:57] (03Merged) 10jenkins-bot: Log texvc errors [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/89759 (owner: 10Tim Starling) [23:46:32] !log tstarling synchronized wmf-config/InitialiseSettings.php [23:46:45] Logged the message, Master