[00:00:12] PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: Puppet last ran 30526 seconds ago, expected 28800 [00:00:33] (03PS1) 10Dzahn: Tampa cleanup - delete dynamicproxy::pmtpa role [puppet] - 10https://gerrit.wikimedia.org/r/168729 [00:05:16] PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: Puppet last ran 30825 seconds ago, expected 28800 [00:05:28] (03PS1) 10Dzahn: ntp, lucene - remove pmtpa remnants [puppet] - 10https://gerrit.wikimedia.org/r/168730 [00:06:08] (03CR) 10jenkins-bot: [V: 04-1] ntp, lucene - remove pmtpa remnants [puppet] - 10https://gerrit.wikimedia.org/r/168730 (owner: 10Dzahn) [00:07:15] (03PS2) 10Dzahn: NTP, lucene - remove pmtpa remnants [puppet] - 10https://gerrit.wikimedia.org/r/168730 [00:10:14] PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: Puppet last ran 31126 seconds ago, expected 28800 [00:15:12] PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: Puppet last ran 31426 seconds ago, expected 28800 [00:20:11] PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: Puppet last ran 31725 seconds ago, expected 28800 [00:25:16] PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: Puppet last ran 32026 seconds ago, expected 28800 [00:25:16] (03CR) 10Manybubbles: [C: 031] NTP, lucene - remove pmtpa remnants [puppet] - 10https://gerrit.wikimedia.org/r/168730 (owner: 10Dzahn) [00:30:11] PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: Puppet last ran 32326 seconds ago, expected 28800 [00:35:11] PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: Puppet last ran 32626 seconds ago, expected 28800 [00:40:13] PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: Puppet last ran 32926 seconds ago, expected 28800 [00:45:12] PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: Puppet last ran 33226 seconds ago, expected 28800 [00:50:11] PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: Puppet last ran 33526 seconds ago, expected 28800 [00:52:02] PROBLEM - MySQL Slave Delay on db1016 is CRITICAL: CRIT replication delay 332 seconds [00:52:40] PROBLEM - MySQL Replication Heartbeat on db1016 is CRITICAL: CRIT replication delay 361 seconds [00:53:31] RECOVERY - MySQL Replication Heartbeat on db1016 is OK: OK replication delay -0 seconds [00:54:28] RECOVERY - MySQL Slave Delay on db1016 is OK: OK replication delay 0 seconds [00:55:18] PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: Puppet last ran 33826 seconds ago, expected 28800 [00:56:23] (03PS1) 10Dzahn: RT - add mail alias file for phab redirect [puppet] - 10https://gerrit.wikimedia.org/r/168733 [00:57:50] (03CR) 10Dzahn: [C: 04-2] "needs discussion with chase" [puppet] - 10https://gerrit.wikimedia.org/r/168733 (owner: 10Dzahn) [01:00:08] PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: Puppet last ran 34126 seconds ago, expected 28800 [01:05:21] PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: Puppet last ran 34426 seconds ago, expected 28800 [01:07:49] PROBLEM - CI: Puppet failure events on labmon1001 is CRITICAL: CRITICAL: integration.integration-dev.puppetagent.failed_events.value (30.00%) [01:10:11] PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: Puppet last ran 34726 seconds ago, expected 28800 [01:15:10] PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: Puppet last ran 35025 seconds ago, expected 28800 [01:20:11] PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: Puppet last ran 35325 seconds ago, expected 28800 [01:25:14] PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: Puppet last ran 35625 seconds ago, expected 28800 [01:30:17] PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: Puppet last ran 35926 seconds ago, expected 28800 [01:32:20] PROBLEM - HHVM rendering on mw1114 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki exception - 50550 bytes in 0.095 second response time [01:32:53] PROBLEM - Apache HTTP on mw1114 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 MediaWiki exception - 50550 bytes in 0.096 second response time [01:34:17] RECOVERY - CI: Puppet failure events on labmon1001 is OK: OK: All targets OK [01:35:14] PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: Puppet last ran 36226 seconds ago, expected 28800 [01:40:14] PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: Puppet last ran 36525 seconds ago, expected 28800 [01:45:14] PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: Puppet last ran 36826 seconds ago, expected 28800 [01:49:54] PROBLEM - puppet last run on mw1114 is CRITICAL: CRITICAL: Puppet last ran 291803 seconds ago, expected 14400 [01:50:14] PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: Puppet last ran 37126 seconds ago, expected 28800 [01:51:10] RECOVERY - puppet last run on mw1114 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [01:55:10] PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: Puppet last ran 37426 seconds ago, expected 28800 [02:00:10] PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: Puppet last ran 37726 seconds ago, expected 28800 [02:05:16] PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: Puppet last ran 38026 seconds ago, expected 28800 [02:10:16] PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: Puppet last ran 38326 seconds ago, expected 28800 [02:15:12] RECOVERY - HHVM rendering on mw1114 is OK: HTTP OK: HTTP/1.1 200 OK - 67317 bytes in 0.297 second response time [02:15:18] PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: Puppet last ran 38626 seconds ago, expected 28800 [02:15:58] RECOVERY - Apache HTTP on mw1114 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 440 bytes in 0.204 second response time [02:18:15] !log LocalisationUpdate completed (1.25wmf4) at 2014-10-25 02:18:14+00:00 [02:18:23] Logged the message, Master [02:20:18] PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: Puppet last ran 38926 seconds ago, expected 28800 [02:25:09] PROBLEM - check_puppetrun on payments1004 is CRITICAL: CRITICAL: Puppet last ran 39226 seconds ago, expected 28800 [02:29:30] !log LocalisationUpdate completed (1.25wmf5) at 2014-10-25 02:29:29+00:00 [02:29:37] Logged the message, Master [02:30:12] RECOVERY - check_puppetrun on payments1004 is OK: OK: Puppet is currently enabled, last run 136 seconds ago with 0 failures [02:48:41] PROBLEM - puppet last run on osmium is CRITICAL: CRITICAL: puppet fail [02:58:51] PROBLEM - Disk space on ocg1001 is CRITICAL: DISK CRITICAL - free space: / 350 MB (3% inode=72%): [03:46:48] !log LocalisationUpdate ResourceLoader cache refresh completed at Sat Oct 25 03:46:48 UTC 2014 (duration 46m 47s) [03:46:55] Logged the message, Master [05:39:38] (03PS7) 1020after4: preamble script to read client address from HTTP_X_FORWARDED_FOR [puppet] - 10https://gerrit.wikimedia.org/r/168509 [06:25:43] RECOVERY - Disk space on ocg1001 is OK: DISK OK [06:27:43] PROBLEM - puppet last run on amssq55 is CRITICAL: CRITICAL: puppet fail [06:28:34] PROBLEM - puppet last run on amslvs1 is CRITICAL: CRITICAL: puppet fail [06:28:43] PROBLEM - puppet last run on db1023 is CRITICAL: CRITICAL: puppet fail [06:28:53] PROBLEM - puppet last run on lvs2001 is CRITICAL: CRITICAL: puppet fail [06:29:13] PROBLEM - puppet last run on amssq60 is CRITICAL: CRITICAL: puppet fail [06:29:25] PROBLEM - puppet last run on cp3016 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:34] PROBLEM - puppet last run on db2018 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:42] PROBLEM - puppet last run on mw1042 is CRITICAL: CRITICAL: Puppet has 2 failures [06:29:43] PROBLEM - puppet last run on lvs2004 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:53] PROBLEM - puppet last run on search1018 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:53] PROBLEM - puppet last run on db1046 is CRITICAL: CRITICAL: Puppet has 3 failures [06:30:04] PROBLEM - puppet last run on labsdb1003 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:04] PROBLEM - puppet last run on mw1061 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:05] PROBLEM - puppet last run on db1015 is CRITICAL: CRITICAL: Puppet has 2 failures [06:30:05] PROBLEM - puppet last run on virt1006 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:05] PROBLEM - puppet last run on mw1052 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:12] PROBLEM - puppet last run on cp1061 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:12] PROBLEM - puppet last run on db1002 is CRITICAL: CRITICAL: Puppet has 2 failures [06:30:13] PROBLEM - puppet last run on mw1118 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:32] PROBLEM - puppet last run on mw1120 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:33] PROBLEM - puppet last run on cp3014 is CRITICAL: CRITICAL: Puppet has 2 failures [06:30:33] PROBLEM - puppet last run on mw1144 is CRITICAL: CRITICAL: Puppet has 2 failures [06:30:42] PROBLEM - puppet last run on cp4008 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:42] PROBLEM - puppet last run on mw1092 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:43] PROBLEM - puppet last run on mw1170 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:43] PROBLEM - puppet last run on mw1211 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:53] PROBLEM - puppet last run on mw1119 is CRITICAL: CRITICAL: Puppet has 3 failures [06:31:13] PROBLEM - puppet last run on cp4004 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:24] PROBLEM - puppet last run on mw1025 is CRITICAL: CRITICAL: Puppet has 1 failures [06:36:43] PROBLEM - puppet last run on db1006 is CRITICAL: CRITICAL: Puppet has 2 failures [06:44:55] RECOVERY - puppet last run on db1015 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [06:45:03] RECOVERY - puppet last run on db1002 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [06:45:13] RECOVERY - puppet last run on mw1120 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [06:45:33] RECOVERY - puppet last run on cp3016 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [06:45:34] RECOVERY - puppet last run on db2018 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [06:45:43] RECOVERY - puppet last run on mw1170 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [06:45:43] RECOVERY - puppet last run on lvs2004 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [06:45:44] RECOVERY - puppet last run on db1046 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [06:45:52] RECOVERY - puppet last run on labsdb1003 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [06:45:53] RECOVERY - puppet last run on mw1119 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [06:45:53] RECOVERY - puppet last run on mw1061 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [06:45:53] RECOVERY - puppet last run on virt1006 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [06:46:02] RECOVERY - puppet last run on mw1052 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [06:46:12] RECOVERY - puppet last run on cp1061 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [06:46:13] RECOVERY - puppet last run on mw1118 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [06:46:22] RECOVERY - puppet last run on mw1025 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [06:46:32] RECOVERY - puppet last run on mw1144 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [06:46:33] RECOVERY - puppet last run on mw1092 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [06:46:33] RECOVERY - puppet last run on cp3014 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [06:46:34] RECOVERY - puppet last run on cp4008 is OK: OK: Puppet is currently enabled, last run 60 seconds ago with 0 failures [06:46:42] RECOVERY - puppet last run on mw1042 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [06:46:43] RECOVERY - puppet last run on mw1211 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [06:46:52] RECOVERY - puppet last run on db1023 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [06:46:57] RECOVERY - puppet last run on search1018 is OK: OK: Puppet is currently enabled, last run 63 seconds ago with 0 failures [06:47:02] RECOVERY - puppet last run on amssq55 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [06:47:03] RECOVERY - puppet last run on lvs2001 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [06:47:13] RECOVERY - puppet last run on cp4004 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [06:47:24] RECOVERY - puppet last run on amssq60 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [06:47:53] RECOVERY - puppet last run on amslvs1 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [06:54:33] RECOVERY - puppet last run on db1006 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [07:00:14] (03PS2) 10Qgil: RT - add mail alias file for phab redirect [puppet] - 10https://gerrit.wikimedia.org/r/168733 (owner: 10Dzahn) [09:56:58] (03CR) 10John F. Lewis: [C: 031] NTP, lucene - remove pmtpa remnants [puppet] - 10https://gerrit.wikimedia.org/r/168730 (owner: 10Dzahn) [09:59:53] (03CR) 10John F. Lewis: [C: 031] Tampa cleanup - delete dynamicproxy::pmtpa role [puppet] - 10https://gerrit.wikimedia.org/r/168729 (owner: 10Dzahn) [10:07:26] (03CR) 10John F. Lewis: [C: 031] rancid - move to module [puppet] - 10https://gerrit.wikimedia.org/r/168698 (owner: 10Dzahn) [10:08:16] (03CR) 10John F. Lewis: [C: 031] role/deployment - remove pmtpa [puppet] - 10https://gerrit.wikimedia.org/r/167888 (owner: 10Dzahn) [14:43:52] !log all active labs instances now running salt minion 2014.1.11 except for: logstash-* (have their own master), fabapi (pingable, can't ssh on), upload-wizard (running oneiric, not setting up a repo for that!). instances shutoff or w/ nova error were left untouched [14:43:59] Logged the message, Master [15:15:48] Nemo_bis: about? [15:53:32] Reedy: yes? [15:55:24] (03PS3) 10ArielGlenn: script to clean up salt keys of deleted labs instances [puppet] - 10https://gerrit.wikimedia.org/r/168601 [15:59:32] (03CR) 10ArielGlenn: "turned out I broke the salt name canonicaliztion after testing (making a 'trivial' change), so this patchset unbreaks it." [puppet] - 10https://gerrit.wikimedia.org/r/168601 (owner: 10ArielGlenn) [16:05:00] (03PS1) 10Glaisher: Set wgCategoryCollation to 'uca-fr' on frwikibooks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/168748 (https://bugzilla.wikimedia.org/72513) [16:31:52] ah, https://github.com/EFForg/https-everywhere/issues/686#issuecomment-60485955 I guess [16:46:01] https://github.com/EFForg/https-everywhere/issues/686#issuecomment-60485955 [17:39:22] (03PS1) 10Glaisher: Add 'centralnoticeadmin' group to wgGroupsAddToSelf for metawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/168752 [18:07:18] (03CR) 10Hoo man: "Can you point out where this changed?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/168752 (owner: 10Glaisher) [18:20:14] (03CR) 10Vogone: [C: 04-1] "If it was unintended, just remove the effects (i.e. make the user rights available to all admins again)." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/168752 (owner: 10Glaisher) [19:08:05] (03PS1) 10KartikMistry: Update Debian package to upstream r57669 [debs/contenttranslation/apertium-apy] - 10https://gerrit.wikimedia.org/r/168760 [19:13:32] (03CR) 10John F. Lewis: [C: 031] "+1 for the actual idea of the patch. Seems like there are no kafka instances in labs from what I can find and a ssh results in the usual '" [puppet] - 10https://gerrit.wikimedia.org/r/168727 (owner: 10Dzahn) [19:46:11] (03PS1) 10Vogone: Create new user group for WMDE staff at wikidatawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/168771 (https://bugzilla.wikimedia.org/72459) [19:48:05] (03CR) 10Vogone: "Also -1 on the actual change itself, 'wgGroupsRemoveFromSelf' is missing." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/168752 (owner: 10Glaisher) [19:50:59] mh.... I think the CN thing might be my fault actually [19:51:41] causing more chaos are we hoo? preventing poor admins editing CN :p [19:54:38] ... :P [19:55:26] so, what did you do to break this? [19:56:43] https://github.com/wikimedia/mediawiki-extensions-CentralNotice/commit/e76aeb085abf70937227ad7eec914c5ecdde6f3b [19:56:45] probably [19:57:02] yep [19:57:07] trying to figure why that broke [19:57:34] oh, I see [19:57:36] that's stupid [19:57:52] https://github.com/wikimedia/operations-mediawiki-config/commit/71d8efc724e3ee7749a2eef5a94b1dd1e0d46c3e [19:57:54] that one? [19:58:06] no [19:58:12] I just forgot to declare global [19:58:26] ohh [19:58:31] PROBLEM - DPKG on mw1028 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [19:58:36] PROBLEM - DPKG on mw1024 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [19:58:36] PROBLEM - DPKG on mw1026 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [19:58:49] and hoo can deploy stuff to the servers. Shall we run and hide now? :D [19:58:51] PROBLEM - DPKG on mw1023 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [19:58:53] PROBLEM - DPKG on mw1018 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [19:58:53] PROBLEM - DPKG on mw1022 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [19:59:03] PROBLEM - DPKG on mw1019 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [19:59:04] PROBLEM - DPKG on mw1025 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [19:59:11] PROBLEM - DPKG on mw1027 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [19:59:11] PROBLEM - DPKG on mw1021 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [19:59:21] PROBLEM - DPKG on mw1053 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [19:59:22] PROBLEM - DPKG on mw1029 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [19:59:31] JohnLewis: Definitely! [19:59:42] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0] [19:59:44] More seriously... is this blocking anything now? [20:00:03] RECOVERY - DPKG on mw1023 is OK: All packages OK [20:00:03] RECOVERY - DPKG on mw1018 is OK: All packages OK [20:00:03] RECOVERY - DPKG on mw1022 is OK: All packages OK [20:00:11] RECOVERY - DPKG on mw1025 is OK: All packages OK [20:00:11] RECOVERY - DPKG on mw1019 is OK: All packages OK [20:00:11] RECOVERY - DPKG on mw1027 is OK: All packages OK [20:00:11] RECOVERY - DPKG on mw1021 is OK: All packages OK [20:00:12] hoo: well... it blocks sysops who want to edit CN. [20:00:21] RECOVERY - DPKG on mw1053 is OK: All packages OK [20:00:22] RECOVERY - DPKG on mw1029 is OK: All packages OK [20:00:22] PROBLEM - puppet last run on mw1018 is CRITICAL: CRITICAL: Puppet has 1 failures [20:00:26] * hoo rewords [20:00:34] hoo: well, PiRSquared17 was about to clone a CN banner for SE2015 (he prepares the pages already … but he also said it wasn't too urgent (obviously)) [20:00:39] is it urgently screwed or can this be deployed on Monday? [20:00:44] RECOVERY - DPKG on mw1026 is OK: All packages OK [20:00:44] RECOVERY - DPKG on mw1028 is OK: All packages OK [20:00:45] RECOVERY - DPKG on mw1024 is OK: All packages OK [20:00:47] hoo: Monday :D [20:01:01] Relief :) [20:01:08] not something would classify for a Wikimedia-weekend deployment. [20:01:18] *that would [20:01:42] Depends on what it blocks... but that sounds fairly low prio [20:02:20] yup, CN editors could always asks bureaucrats to be added to the CN admin group (which can still edit CN) [20:02:32] hoo: and if you have a spare moment - mind looking at https://gerrit.wikimedia.org/r/#/c/168702/ :p [20:03:15] JohnLewis: Maybe in a bit [20:03:47] hey - it'll end up like most patches likely. 'what is the status' being commented in 2015 [20:03:51] ;) [20:04:33] (03CR) 10Ori.livneh: "what is the status?" [puppet] - 10https://gerrit.wikimedia.org/r/168702 (https://bugzilla.wikimedia.org/48786) (owner: 10John F. Lewis) [20:04:39] (03CR) 10Hoo man: [C: 04-2] "https://gerrit.wikimedia.org/r/168774 should be done instead" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/168752 (owner: 10Glaisher) [20:04:49] -2, zomg [20:05:03] ori: <3 [20:05:13] Vogone: Yeah, I missed the abandon button [20:05:16] :D [20:05:33] :D [20:07:24] JohnLewis: where is that variable defined? [20:07:28] Anyone wants to SWAT-partner https://gerrit.wikimedia.org/r/168777 ? [20:07:43] I'm not really going to be around on Monday [20:07:49] at least not during the early SWAT [20:08:04] ori: https://gerrit.wikimedia.org/r/#/c/168701/ (mentioned in the commit) [20:09:03] JohnLewis: why not use $wmgAddWikiNotify for that purpose? have it be either an e-mail address or 'false' to disable notifications. [20:09:27] eh true [20:10:22] ori: You merged it on wmf deploy but not master [20:10:31] (03PS3) 10John F. Lewis: Remove hardcoding from notifyNewProjects [puppet] - 10https://gerrit.wikimedia.org/r/168702 [20:13:02] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0] [20:13:19] (03PS2) 10John F. Lewis: Add $wmgAddWikiNotifyEmail for use by notifyNewProjects [mediawiki-config] - 10https://gerrit.wikimedia.org/r/168701 [20:14:07] ori: Will you push it now-ish? [20:14:36] (03CR) 10Ori.livneh: "Could you update the comment to say something like, "@var string|bool: E-mail address to send notifications to, or false to disable notifi" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/168701 (owner: 10John F. Lewis) [20:14:43] hoo: sure-ish [20:14:50] ok with me [20:16:37] lazy so c&p :p [20:16:53] RECOVERY - puppet last run on mw1018 is OK: OK: Puppet is currently enabled, last run 1 seconds ago with 0 failures [20:16:57] (03PS3) 10John F. Lewis: Add $wmgAddWikiNotifyEmail for use by notifyNewProjects [mediawiki-config] - 10https://gerrit.wikimedia.org/r/168701 [20:17:58] hoo@fluorine:/a/mw-log$ ls /a/mw-log/ | wc -l [20:17:58] 349 [20:18:01] ugh [20:18:36] ugh: command not found [20:20:53] ... [20:21:12] Most of that is pure PHP 5.3 goodness [20:41:10] !log updated logstash-* labs instances to salt minion 2014.1.11 (thanks for the ping apergos) [20:41:18] Logged the message, Master [21:30:10] !log ori Synchronized php-1.25wmf4/extensions/CentralNotice/CentralNotice.hooks.php: Iee2072ac7: Make sure we declare globals before using them (duration: 00m 06s) [21:30:15] Logged the message, Master [21:30:16] !log ori Synchronized php-1.25wmf5/extensions/CentralNotice/CentralNotice.hooks.php: Iee2072ac7: Make sure we declare globals before using them (duration: 00m 06s) [21:30:21] Logged the message, Master [21:30:22] ^ hoo [21:30:37] looks ok, /me gone for real. [21:37:05] ok :) [21:37:35] Looks good on meta [22:32:06] !log scheduling downtime for all ulsfo -lb- & cr1/2-ulsfo [22:32:12] Logged the message, Master [22:49:44] !log upgrading JunOS on cr1-ulsfo [22:49:51] Logged the message, Master [22:58:11] PROBLEM - Host cr1-ulsfo is DOWN: CRITICAL - Network Unreachable (198.35.26.192) [23:07:31] RECOVERY - Host cr1-ulsfo is UP: PING OK - Packet loss = 0%, RTA = 79.50 ms