[00:04:48] RECOVERY - puppet last run on cp3003 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [00:08:57] PROBLEM - puppet last run on amslvs1 is CRITICAL: CRITICAL: Epic puppet fail [00:27:57] RECOVERY - puppet last run on amslvs1 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [01:21:49] PROBLEM - Puppet freshness on es7 is CRITICAL: Last successful Puppet run was Sun 10 Aug 2014 01:18:58 UTC [01:23:49] PROBLEM - Puppet freshness on es7 is CRITICAL: Last successful Puppet run was Sun 10 Aug 2014 01:18:58 UTC [01:25:49] PROBLEM - Puppet freshness on es7 is CRITICAL: Last successful Puppet run was Sun 10 Aug 2014 01:18:58 UTC [01:27:49] PROBLEM - Puppet freshness on es7 is CRITICAL: Last successful Puppet run was Sun 10 Aug 2014 01:18:58 UTC [01:29:49] PROBLEM - Puppet freshness on es7 is CRITICAL: Last successful Puppet run was Sun 10 Aug 2014 01:18:58 UTC [01:31:49] PROBLEM - Puppet freshness on es7 is CRITICAL: Last successful Puppet run was Sun 10 Aug 2014 01:18:58 UTC [01:33:49] PROBLEM - Puppet freshness on es7 is CRITICAL: Last successful Puppet run was Sun 10 Aug 2014 01:18:58 UTC [01:35:49] PROBLEM - Puppet freshness on es7 is CRITICAL: Last successful Puppet run was Sun 10 Aug 2014 01:18:58 UTC [01:37:49] PROBLEM - Puppet freshness on es7 is CRITICAL: Last successful Puppet run was Sun 10 Aug 2014 01:18:58 UTC [01:38:59] RECOVERY - Puppet freshness on es7 is OK: puppet ran at Sun Aug 10 01:38:51 UTC 2014 [01:41:44] PROBLEM - Puppet freshness on lvs4001 is CRITICAL: Last successful Puppet run was Sun 10 Aug 2014 01:38:41 UTC [01:43:44] PROBLEM - Puppet freshness on lvs4001 is CRITICAL: Last successful Puppet run was Sun 10 Aug 2014 01:38:41 UTC [01:45:44] PROBLEM - Puppet freshness on lvs4001 is CRITICAL: Last successful Puppet run was Sun 10 Aug 2014 01:38:41 UTC [01:47:44] PROBLEM - Puppet freshness on lvs4001 is CRITICAL: Last successful Puppet run was Sun 10 Aug 2014 01:38:41 UTC [01:49:44] PROBLEM - Puppet freshness on lvs4001 is CRITICAL: Last successful Puppet run was Sun 10 Aug 2014 01:38:41 UTC [01:51:44] PROBLEM - Puppet freshness on lvs4001 is CRITICAL: Last successful Puppet run was Sun 10 Aug 2014 01:38:41 UTC [01:53:44] PROBLEM - Puppet freshness on lvs4001 is CRITICAL: Last successful Puppet run was Sun 10 Aug 2014 01:38:41 UTC [01:55:44] PROBLEM - Puppet freshness on lvs4001 is CRITICAL: Last successful Puppet run was Sun 10 Aug 2014 01:38:41 UTC [01:57:44] PROBLEM - Puppet freshness on lvs4001 is CRITICAL: Last successful Puppet run was Sun 10 Aug 2014 01:38:41 UTC [01:58:54] RECOVERY - Puppet freshness on lvs4001 is OK: puppet ran at Sun Aug 10 01:58:43 UTC 2014 [02:00:44] PROBLEM - Puppet freshness on lvs4001 is CRITICAL: Last successful Puppet run was Sun 10 Aug 2014 01:58:43 UTC [02:18:48] RECOVERY - Puppet freshness on lvs4001 is OK: puppet ran at Sun Aug 10 02:18:36 UTC 2014 [02:20:56] !log LocalisationUpdate completed (1.24wmf15) at 2014-08-10 02:19:53+00:00 [02:21:07] Logged the message, Master [02:34:08] PROBLEM - Puppet freshness on db1006 is CRITICAL: Last successful Puppet run was Sun 10 Aug 2014 00:33:36 UTC [02:34:18] !log LocalisationUpdate completed (1.24wmf16) at 2014-08-10 02:33:15+00:00 [02:34:23] Logged the message, Master [03:15:58] !log LocalisationUpdate ResourceLoader cache refresh completed at Sun Aug 10 03:14:52 UTC 2014 (duration 14m 51s) [03:16:03] Logged the message, Master [03:33:18] RECOVERY - Puppet freshness on db1006 is OK: puppet ran at Sun Aug 10 03:33:08 UTC 2014 [06:28:50] PROBLEM - puppet last run on cp1061 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:19] PROBLEM - puppet last run on amssq35 is CRITICAL: CRITICAL: Puppet has 2 failures [06:29:29] PROBLEM - puppet last run on mw1100 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:48] PROBLEM - puppet last run on mw1060 is CRITICAL: CRITICAL: Puppet has 2 failures [06:29:49] PROBLEM - puppet last run on mw1164 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:49] PROBLEM - puppet last run on mw1069 is CRITICAL: CRITICAL: Puppet has 2 failures [06:29:49] PROBLEM - puppet last run on mw1068 is CRITICAL: CRITICAL: Puppet has 2 failures [06:30:08] PROBLEM - puppet last run on mw1088 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:08] PROBLEM - puppet last run on gallium is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:18] PROBLEM - puppet last run on mw1046 is CRITICAL: CRITICAL: Puppet has 1 failures [06:44:58] RECOVERY - puppet last run on mw1069 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [06:45:08] RECOVERY - puppet last run on mw1088 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [06:45:48] RECOVERY - puppet last run on mw1060 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [06:45:58] RECOVERY - puppet last run on mw1164 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [06:45:58] RECOVERY - puppet last run on mw1068 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [06:46:19] RECOVERY - puppet last run on mw1046 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [06:46:28] RECOVERY - puppet last run on amssq35 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [06:46:38] RECOVERY - puppet last run on mw1100 is OK: OK: Puppet is currently enabled, last run 58 seconds ago with 0 failures [06:46:59] RECOVERY - puppet last run on cp1061 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [06:48:08] PROBLEM - puppet last run on mw1053 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [06:48:08] RECOVERY - puppet last run on gallium is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [06:49:58] PROBLEM - nutcracker process on mw1053 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [06:49:58] PROBLEM - puppet disabled on mw1053 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [06:49:59] PROBLEM - Disk space on mw1053 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [06:49:59] PROBLEM - DPKG on mw1053 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [06:50:08] PROBLEM - nutcracker port on mw1053 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [06:50:09] PROBLEM - SSH on mw1053 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:50:18] PROBLEM - RAID on mw1053 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [06:50:18] PROBLEM - check configured eth on mw1053 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [06:50:19] PROBLEM - check if dhclient is running on mw1053 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [06:53:09] RECOVERY - SSH on mw1053 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2 (protocol 2.0) [06:53:09] RECOVERY - check configured eth on mw1053 is OK: NRPE: Unable to read output [06:53:09] RECOVERY - check if dhclient is running on mw1053 is OK: PROCS OK: 0 processes with command name dhclient [06:53:18] RECOVERY - RAID on mw1053 is OK: OK: no RAID installed [06:53:48] RECOVERY - puppet disabled on mw1053 is OK: OK [06:53:48] RECOVERY - nutcracker process on mw1053 is OK: PROCS OK: 1 process with UID = 110 (nutcracker), command name nutcracker [06:53:49] RECOVERY - Disk space on mw1053 is OK: DISK OK [06:53:49] RECOVERY - DPKG on mw1053 is OK: All packages OK [06:53:58] RECOVERY - nutcracker port on mw1053 is OK: TCP OK - 0.000 second response time on port 11212 [07:21:08] PROBLEM - Puppet freshness on db1007 is CRITICAL: Last successful Puppet run was Sun 10 Aug 2014 05:20:27 UTC [08:20:18] RECOVERY - Puppet freshness on db1007 is OK: puppet ran at Sun Aug 10 08:20:10 UTC 2014 [08:22:28] PROBLEM - puppet last run on ssl3003 is CRITICAL: CRITICAL: Epic puppet fail [08:35:29] PROBLEM - Number of mediawiki jobs queued on tungsten is CRITICAL: CRITICAL: Anomaly detected: 29 data above and 0 below the confidence bounds [08:35:29] PROBLEM - Number of mediawiki jobs running on tungsten is CRITICAL: CRITICAL: Anomaly detected: 29 data above and 0 below the confidence bounds [08:42:28] RECOVERY - puppet last run on ssl3003 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [08:55:18] (03CR) 10JanZerebecki: [C: 031] planet - use ssl_ciphersuite [operations/puppet] - 10https://gerrit.wikimedia.org/r/153215 (owner: 10Dzahn) [10:24:08] (03PS1) 10Ori.livneh: Varnish: add 'HHVM' backend [operations/puppet] - 10https://gerrit.wikimedia.org/r/153288 [10:24:10] (03PS1) 10Ori.livneh: Varnish: route requests with magic cookie to HHVM backend [operations/puppet] - 10https://gerrit.wikimedia.org/r/153289 [10:24:58] (03CR) 10jenkins-bot: [V: 04-1] Varnish: route requests with magic cookie to HHVM backend [operations/puppet] - 10https://gerrit.wikimedia.org/r/153289 (owner: 10Ori.livneh) [10:25:08] whatevs, jenkins [10:25:17] haters gonna hate [10:28:26] (03PS2) 10Ori.livneh: Varnish: route requests with magic cookie to HHVM backend [operations/puppet] - 10https://gerrit.wikimedia.org/r/153289 [10:32:46] (03PS3) 10Ori.livneh: Varnish: route requests with magic cookie to HHVM backend [operations/puppet] - 10https://gerrit.wikimedia.org/r/153289 [10:48:59] <_joe_> ori: all those patches have been done by mark [10:49:17] <_joe_> https://gerrit.wikimedia.org/r/#/c/152903/ [10:49:58] <_joe_> ori: on monday I will cherry-pick it on beta [10:50:41] (03CR) 10Giuseppe Lavagetto: [C: 04-1] "already in https://gerrit.wikimedia.org/r/#/c/152903/" [operations/puppet] - 10https://gerrit.wikimedia.org/r/153288 (owner: 10Ori.livneh) [10:51:24] (03CR) 10Dzahn: [C: 032] gerrit - retab apache template [operations/puppet] - 10https://gerrit.wikimedia.org/r/153216 (owner: 10Dzahn) [10:51:34] <_joe_> also, I see you want to set the cookie in varnish.... mmh [10:51:47] <_joe_> on no sorry, read wrong [10:54:00] (03CR) 10Giuseppe Lavagetto: [C: 04-1] "See https://gerrit.wikimedia.org/r/#/c/152903/ as well; a part of the logic here should get ported to that patchset though" [operations/puppet] - 10https://gerrit.wikimedia.org/r/153289 (owner: 10Ori.livneh) [10:54:32] (03CR) 10Dzahn: [C: 032] sudo.pp - minor linting fixes [operations/puppet] - 10https://gerrit.wikimedia.org/r/153223 (owner: 10Dzahn) [10:58:08] (03CR) 10Dzahn: [C: 032] "reduce number of lint warns/errors" [operations/puppet] - 10https://gerrit.wikimedia.org/r/153225 (owner: 10Dzahn) [11:02:43] (03CR) 10Dzahn: [C: 032] gerrit apache template - qualify vars [operations/puppet] - 10https://gerrit.wikimedia.org/r/153217 (owner: 10Dzahn) [11:05:34] (03CR) 10Dzahn: "also see Change-Id: I682cb816e08d886" [operations/puppet] - 10https://gerrit.wikimedia.org/r/153117 (owner: 10Dzahn) [11:06:49] (03CR) 10Dzahn: [C: 032] planet - use ssl_ciphersuite [operations/puppet] - 10https://gerrit.wikimedia.org/r/153215 (owner: 10Dzahn) [11:10:18] PROBLEM - puppet last run on cp4004 is CRITICAL: CRITICAL: Puppet has 1 failures [11:13:00] (03CR) 10Dzahn: [C: 032] "will be merged automatically with the dependent change" [operations/puppet] - 10https://gerrit.wikimedia.org/r/153034 (owner: 10Hoo man) [11:13:08] PROBLEM - Puppet freshness on db1011 is CRITICAL: Last successful Puppet run was Sun 10 Aug 2014 09:12:51 UTC [11:13:28] RECOVERY - Puppet freshness on db1011 is OK: puppet ran at Sun Aug 10 11:13:23 UTC 2014 [11:17:09] so, some originals seems to be disappearing. what would I need to have access to in order to troubleshoot this? eg. to query swift directly and check if the object is gone from there as well [11:17:19] https://bugzilla.wikimedia.org/show_bug.cgi?id=69362 [11:21:04] gi11es: still at wikimania? I'm in the garden room at the labs table, we can take a look [11:21:34] godog: yes, I'm in the middle of a roundtable hosted by my team. will you still be around there at 1? [11:22:28] gi11es: yep [11:22:39] godog: alright, I'll see you then, thanks! [11:22:55] no problem! it might be related to https://bugzilla.wikimedia.org/show_bug.cgi?id=69311 too [11:24:55] bd808: that was fast, thanks :)) r/153294 [11:25:22] godog: I merge all doc patches as soon as possible! [11:25:37] Also I have an irc ping on "vagrant" so I see them :) [11:27:54] (03CR) 10Filippo Giunchedi: [C: 031] "files on the host will need to be removed but LGTM otherwise" [operations/puppet] - 10https://gerrit.wikimedia.org/r/153117 (owner: 10Dzahn) [11:28:18] RECOVERY - puppet last run on cp4004 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [11:28:19] bd808: haha cool [11:29:48] PROBLEM - puppet last run on mw1025 is CRITICAL: CRITICAL: Puppet has 1 failures [11:32:16] (03CR) 10Dzahn: [C: 032] noc.wm - use apache::site and move to templates [operations/puppet] - 10https://gerrit.wikimedia.org/r/153214 (owner: 10Dzahn) [11:32:28] PROBLEM - puppet last run on stat1002 is CRITICAL: CRITICAL: Puppet has 35 failures [11:32:48] RECOVERY - puppet last run on mw1025 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [11:33:15] (03PS4) 10Yuvipanda: Tools: Add some i386 compat packages to exec nodes [operations/puppet] - 10https://gerrit.wikimedia.org/r/125241 [11:34:57] icinga-wm: liar [11:35:28] RECOVERY - puppet last run on stat1002 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [11:36:19] (03PS5) 10Yuvipanda: Tools: Add some i386 compat packages to exec nodes [operations/puppet] - 10https://gerrit.wikimedia.org/r/125241 [11:36:24] Coren_WM2014: ^ [11:40:09] PROBLEM - puppet last run on fenari is CRITICAL: CRITICAL: Puppet has 1 failures [11:41:09] RECOVERY - puppet last run on fenari is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [11:53:28] Coren_WM2014: https://gerrit.wikimedia.org/r/#/c/153053/ [11:57:32] !log tarin - stopping poolcounterd, gmond,.. (Tampa, should really not be in use) [11:57:37] Logged the message, Master [11:58:48] PROBLEM - Poolcounter connection on tarin is CRITICAL: Connection refused [11:59:18] PROBLEM - poolcounter on tarin is CRITICAL: PROCS CRITICAL: 0 processes with command name poolcounterd [12:00:10] ACKNOWLEDGEMENT - Poolcounter connection on tarin is CRITICAL: Connection refused daniel_zahn RT: 6265 , gerrit 152154 [12:00:10] ACKNOWLEDGEMENT - poolcounter on tarin is CRITICAL: PROCS CRITICAL: 0 processes with command name poolcounterd daniel_zahn RT: 6265 , gerrit 152154 [12:00:18] RECOVERY - poolcounter on tarin is OK: PROCS OK: 1 process with command name poolcounterd [12:08:15] !log re-enabling puppet and services on tarin [12:08:20] Logged the message, Master [12:09:07] (03CR) 10Dzahn: [C: 04-1] "still ganglia aggregator for misc. tampa cluster" [operations/puppet] - 10https://gerrit.wikimedia.org/r/152154 (owner: 10Dzahn) [12:10:46] (03CR) 10Dzahn: [C: 032] OTRS - use apache::site instead of file resource [operations/puppet] - 10https://gerrit.wikimedia.org/r/153213 (owner: 10Dzahn) [12:22:53] (03CR) 10Dzahn: [C: 032] contint: analytics packages are not on Trusty yet [operations/puppet] - 10https://gerrit.wikimedia.org/r/153209 (owner: 10Hashar) [12:24:10] (03CR) 10Dzahn: [C: 032] zuul: typo in role::zuul::merger description [operations/puppet] - 10https://gerrit.wikimedia.org/r/153208 (owner: 10Hashar) [12:28:38] PROBLEM - puppet last run on amslvs1 is CRITICAL: CRITICAL: Epic puppet fail [12:33:38] (03CR) 10Dzahn: "duh, sorry, can't do this for toollabs - just for physical nodes in site.pp" [operations/puppet] - 10https://gerrit.wikimedia.org/r/124001 (owner: 10Tim Landscheidt) [12:33:38] PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: There are 2 unmerged changes in puppet (dir /var/lib/git/operations/puppet). [12:40:14] (03CR) 10coren: [C: 032] Tools: Use apt::repository instead of file resources [operations/puppet] - 10https://gerrit.wikimedia.org/r/123882 (owner: 10Tim Landscheidt) [12:41:38] RECOVERY - Unmerged changes on repository puppet on strontium is OK: No changes to merge. [12:48:38] RECOVERY - puppet last run on amslvs1 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [12:57:52] (03PS1) 10Tim Starling: Add a new protection level called "superprotect" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/153302 [12:59:00] (03CR) 10Tim Starling: [C: 032] Add a new protection level called "superprotect" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/153302 (owner: 10Tim Starling) [12:59:07] (03Merged) 10jenkins-bot: Add a new protection level called "superprotect" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/153302 (owner: 10Tim Starling) [13:02:08] !log tstarling Synchronized wmf-config/CommonSettings.php: Idfa21125 (duration: 00m 06s) [13:02:14] Logged the message, Master [13:02:25] !log tstarling Synchronized wmf-config/InitialiseSettings.php: Idfa21125 (duration: 00m 05s) [13:02:30] Logged the message, Master [13:04:39] (03CR) 10Nemo bis: "Updated: https://meta.wikimedia.org/w/index.php?title=Wikipedia_power_structure&diff=9475131&oldid=7479635" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/153302 (owner: 10Tim Starling) [13:06:01] (03CR) 10Odder: "This should be reverted immediately." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/153302 (owner: 10Tim Starling) [13:08:44] odder: It's not even deployed yet ;) [13:09:50] And it shouldn't be. [13:10:30] (03CR) 10Alex Monk: "It's kind of pointless because sysops can just add javascript to all sorts of pages that could abuse sysadmin/staff accounts to carry out" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/153302 (owner: 10Tim Starling) [13:11:07] It is [13:11:08] lolol [13:12:32] (03CR) 10Reedy: "Message(s) need adding to WikimediaMessages" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/153302 (owner: 10Tim Starling) [13:15:13] * MatmaRex_mobile gets popcorn [13:16:41] (03CR) 10Jarry1250: "You might as well make the topic line "Declare war on local site admins"" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/153302 (owner: 10Tim Starling) [13:25:36] valhallasw`mania: ££££££££££££££ [13:26:49] give me ALL the quids [13:29:42] (03CR) 10Eloquence: "Details: http://lists.wikimedia.org/pipermail/wikitech-l/2014-August/078089.html" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/153302 (owner: 10Tim Starling) [13:33:47] (03CR) 10Legoktm: Add a new protection level called "superprotect" (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/153302 (owner: 10Tim Starling) [13:34:48] PROBLEM - puppet last run on mw1201 is CRITICAL: CRITICAL: Puppet has 1 failures [13:38:17] (03CR) 10Tim Starling: Add a new protection level called "superprotect" (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/153302 (owner: 10Tim Starling) [13:41:33] (03PS1) 10Legoktm: Generalize comment for "templateeditor" since it's on other wikis now [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/153306 [13:42:13] (03CR) 10Legoktm: Add a new protection level called "superprotect" (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/153302 (owner: 10Tim Starling) [13:53:19] Reedy: can you log that the change was deployed? [13:53:48] RECOVERY - puppet last run on mw1201 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [13:53:50] !log Grant staff "superprotect" right per Robla/Erik request [13:53:56] Logged the message, Master [13:54:04] Staff* [13:54:08] PROBLEM - Puppet freshness on db1011 is CRITICAL: Last successful Puppet run was Sun 10 Aug 2014 11:52:59 UTC [13:54:16] thanks again, Reedy [13:54:19] * Reedy kicks legoktm [13:56:09] Reedy: No kicking [13:56:14] FRIENDLY SPACES POLICY [13:56:19] (03CR) 10Odder: "FYI: The patch is now deployed on the production cluster, and users in the global staff user group have the 'superprotect' user right." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/153302 (owner: 10Tim Starling) [13:56:21] well i'm pretty sure the friendly spaces policy covers that [13:58:52] I think the policy only covers "harrasment" ? [14:00:30] (03CR) 10Tim Starling: "I have not reviewed the situation on de.wp and have no opinion as to whether this is a good idea. I was specifically asked to implement th" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/153302 (owner: 10Tim Starling) [14:00:45] pretty sure any violence at a friendly spaces event would automatically include a harassment factor [14:01:03] it's not like they are hosting kink festivals or anything Lcawte [14:01:19] Depends on how you define harrasment :P [14:02:50] Eloquence: I guess we should get the relevant messages for the group/right added to WikimediaMessages, backported and deployed [14:03:37] Would that hand it off to TWN immediately? If the primary audience is German, you'll want some Geran translatiosn running pretty quick [14:03:49] reedy: *German translations [14:10:41] Reedy, sounds good. [14:15:10] (03CR) 10Nemo bis: "Tim, de.wp? Nobody mentioned it, do you know more? :-)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/153302 (owner: 10Tim Starling) [14:16:02] Jarry1250: not automatically [14:16:08] (03CR) 10Dzahn: [C: 031] "lgtm" [operations/puppet] - 10https://gerrit.wikimedia.org/r/124001 (owner: 10Tim Landscheidt) [14:16:38] Nemo_bis: Yeah. We've been talking IRL about it shipping with German translations rather than waiting for TWN import/export [14:16:49] pointles [14:17:03] just ask raymond to sync [14:19:33] (03CR) 10Chad: "I'm so glad I'm not a member of that godforsaken staff group. It's far too powerful considering what most people need/use it for." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/153302 (owner: 10Tim Starling) [14:22:07] (03CR) 10Multichill: "https://de.wikipedia.org/w/index.php?title=MediaWiki%3ACommon.js&diff=132942446&oldid=131891714" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/153302 (owner: 10Tim Starling) [14:25:03] (03CR) 10Chad: "Per Reedy, WikimediaMessages still needs updating, sync'ing, and i18n rebuilt if we're going to start using this. (03CR) 10Odder: "It /has/ just been used to protect MediaWiki:Common.js on the German Wikipedia. Leaves a very shitty edit summary with the message key vis" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/153302 (owner: 10Tim Starling) [14:26:00] (03CR) 10Alex Monk: "We already did:" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/153302 (owner: 10Tim Starling) [14:29:18] Oh my. [14:29:45] Tim-away: You still tempering with the cluster? [14:29:49] or anyone? [14:29:52] tempering? [14:29:56] you mean tampering? [14:29:58] doing stuffs [14:30:00] whatever [14:30:03] I'm not doing anything [14:30:06] * hoo wants to push two smallish backport [14:30:08] ok [14:30:43] * hoo better heads for a power source first [14:31:57] I'm downloading my wiki e-mail now. [14:32:14] hoo: where are you? [14:33:27] (03CR) 10Chad: "Well since things were done in the wrong order we've got an ugly log entry. Someone plan on fixing that?" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/153302 (owner: 10Tim Starling) [14:33:48] RECOVERY - Puppet freshness on db1011 is OK: puppet ran at Sun Aug 10 14:33:39 UTC 2014 [14:34:04] Message proposals are in https://gerrit.wikimedia.org/r/153309 [14:34:14] Could do with de additions [14:34:32] legoktm: WMDE thingy in chapters village [14:34:39] waiting for someone there, so can't really leave [14:34:41] where is chapters village? [14:34:47] legoktm: ground floor [14:34:59] okay, I'll probably just stay here. [14:35:08] the wmf15 backport can just be forcemerged [14:35:09] PROBLEM - puppet last run on mw1210 is CRITICAL: CRITICAL: Puppet has 1 failures [14:35:17] I know [14:35:25] also had that trouble with the last set of backports [14:36:15] (03CR) 10Ori.livneh: "Ah, cool. I completely missed Mark's patches. Yes, by all means lets use his." [operations/puppet] - 10https://gerrit.wikimedia.org/r/153288 (owner: 10Ori.livneh) [14:36:17] Eloquence: Want to do some german translations? [14:38:41] 'Übergesichert'? [14:38:58] (03PS1) 10Steinsplitter: Revert "Add a new protection level called "superprotect"" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/153313 [14:39:16] g1 [14:39:25] I have no idea what is going on [14:39:34] ori: lols [14:39:37] lols are going on [14:39:41] lols or lulz? [14:40:00] * ori stays out of it. [14:40:09] HHVM! [14:40:47] HAT [14:40:49] (03CR) 10Chad: "It's not actually reverted, a revert was just proposed. Gerrit's wording here is misleading." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/153302 (owner: 10Tim Starling) [14:41:37] what would actually happen if thst was merged? [14:41:56] wouldn't de.wp's common.js stay protected forever? [14:42:12] Yeah, you'd need to unprotect it first. [14:42:22] yes... [14:42:24] pfft. [14:42:26] Or go fishing in the database [14:42:32] hilarious [14:43:44] (03CR) 10Ori.livneh: "@Giuseppe: thanks for the review. Mind if I leave this one for you to +2 / deploy? (At your convenience, of course.)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/153128 (owner: 10Ori.livneh) [14:46:57] ori: this is on our table right now https://www.dropbox.com/s/8a73a4v9jcocoal/2014-08-10%2015.41.37.jpg [14:47:02] (03CR) 10Odder: "Would need to unprotect de:MediaWiki:Common.js before this is merged." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/153313 (owner: 10Steinsplitter) [14:47:11] Eloquence: Can you do the german translations? [14:47:24] https://gerrit.wikimedia.org/r/#/c/153309/5/i18n/wikimedia/en.json,unified [14:50:51] http://www.pokewiki.de/index.php?title=Superschutz [14:51:35] mutante: I guess we can use that as translation [14:51:37] seems legit [14:52:09] RECOVERY - puppet last run on mw1210 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [14:54:19] (03CR) 10MZMcBride: [C: 031] Revert "Add a new protection level called "superprotect"" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/153313 (owner: 10Steinsplitter) [14:54:43] (03Abandoned) 10Hoo man: Revert "Add a new protection level called "superprotect"" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/153313 (owner: 10Steinsplitter) [14:54:50] seriously people -.- [14:55:17] gerrit war [14:55:37] ... [14:55:42] it's photo time btw [14:55:56] (03PS1) 10Odder: Revert "Add a new protection level called "superprotect"" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/153318 [14:56:22] :S [14:56:28] odder: seriously, please don't spam our inboxes [14:56:37] for heavens sake, yes [14:56:37] (03Abandoned) 10Reedy: Revert "Add a new protection level called "superprotect"" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/153318 (owner: 10Odder) [14:56:46] Awww, no comment [14:56:52] omg, next we're gonna add superprotect on gerrit itself? [14:57:06] This is Erik's fault. [14:57:15] If you want to place blame, this falls squarely on him. [14:57:17] (03PS1) 10Odder: Revert "Add a new protection level called "superprotect"" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/153319 [14:57:24] odder: seriously stop. spam someone else [14:57:34] legoktm: You're not listed as a reviewer. [14:57:39] I'm not sure why you'd be getting "spam." [14:57:41] It adds comments to the other one [14:57:49] Ah, well, stop abandoning, then. [14:57:49] You are not even CC'd to that change, Carmela [14:57:57] legoktm, I meant [14:58:00] :-) [14:58:09] He's getting hit by the auto-comment on the old change. [14:58:09] who messed with Wikidata in wmf15 [14:58:11] :P [14:58:15] * hoo will look at that later [14:58:16] rage [14:58:20] aude: You? [14:58:27] who, me? [14:58:35] what? [14:59:42] photo time! [15:00:56] !log hoo Synchronized php-1.24wmf15/extensions/CentralAuth/: (no message) (duration: 00m 25s) [15:01:02] Logged the message, Master [15:01:29] !log hoo Synchronized php-1.24wmf16/extensions/CentralAuth/: (no message) (duration: 00m 25s) [15:01:33] Logged the message, Master [15:04:14] Picture time :) [15:28:12] (03Abandoned) 10MaxSem: Revert "Add a new protection level called "superprotect"" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/153319 (owner: 10Odder) [15:28:38] (03PS1) 10Odder: Revert "Add a new protection level called "superprotect"" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/153321 [15:28:40] (03CR) 10Alex Monk: "Sam made I8c2c15f1 to fix the i18n, I approved it. I think we need translatewiki to configure their end then we can get German translation" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/153302 (owner: 10Tim Starling) [15:30:04] (03CR) 10Reedy: [C: 04-2] Revert "Add a new protection level called "superprotect"" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/153321 (owner: 10Odder) [15:30:43] (03CR) 10Jarry1250: "AFAIK you can simple add de.json and it will just work, or you can wait for Siebrand et al. to rush through the TWN bits and pieces." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/153302 (owner: 10Tim Starling) [15:31:12] Jarry1250: Right [15:31:16] We just need someone to translate :) [15:33:37] Reedy: https://gerrit.wikimedia.org/r/#/c/153322/ but fails.. [15:33:58] mutante: thanks! [15:34:01] needs a comma on line 205 [15:34:05] 205 205 » "interlanguage-link-sitename-dmoz": "DMOZ-Webverzeichnis" [15:36:34] PS2 [15:37:14] (03PS1) 10Perhelion: Revert "Add a new protection level called "superprotect"" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/153323 [15:37:26] qlawl [15:37:46] (03Abandoned) 10Reedy: Revert "Add a new protection level called "superprotect"" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/153323 (owner: 10Perhelion) [15:51:23] Did Jimmy just say block DaB? :P [15:52:22] Lcawte: Where? [15:52:25] Lcawte: Behave. :-) [15:59:09] hoo@tin:/a/common/php-1.24wmf15/extensions/Wikidata$ ls -l extensions/Wikibase/lib/config/ [15:59:09] total 8 [15:59:10] -rw-rw-r-- 1 reedy wikidev 0 Jul 29 18:28 WikibaseLib.default.ph [15:59:20] (yes, the last p is cut of) [15:59:26] Reedy: ^ any idea why that is? [16:00:49] july 29? [16:02:07] No idea [16:02:46] it's untracked and empty, and duplicate [16:02:47] !log reedy Started scap: Rebuild l10n cache for WikimediaMessages [16:02:53] Logged the message, Master [16:03:04] aude: yes, yes and yes [16:09:28] PROBLEM - puppet last run on mw1053 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [16:18:28] PROBLEM - puppet last run on mw1053 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [16:24:59] !log reedy Finished scap: Rebuild l10n cache for WikimediaMessages (duration: 22m 12s) [16:25:04] Logged the message, Master [16:51:18] PROBLEM - puppet last run on db69 is CRITICAL: CRITICAL: Epic puppet fail [17:10:18] RECOVERY - puppet last run on db69 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [17:10:32] (03PS7) 10Dzahn: turn icinga into module [operations/puppet] - 10https://gerrit.wikimedia.org/r/145472 [17:10:43] (03CR) 10jenkins-bot: [V: 04-1] turn icinga into module [operations/puppet] - 10https://gerrit.wikimedia.org/r/145472 (owner: 10Dzahn) [17:15:11] (03PS8) 10Dzahn: turn icinga into module [operations/puppet] - 10https://gerrit.wikimedia.org/r/145472 [17:18:44] great it breaks before and after the change, no diff :P [17:18:47] cya [17:19:27] (03CR) 10Dzahn: "'undef' from left operand of 'in' expression is not a string at /opt/wmf/software/compare-puppet-catalogs/external/puppet/manifests/networ" [operations/puppet] - 10https://gerrit.wikimedia.org/r/145472 (owner: 10Dzahn) [17:24:45] (03CR) 10Ricordisamoa: [C: 031] Revert "Add a new protection level called "superprotect"" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/153321 (owner: 10Odder) [17:39:37] (03CR) 10Giuseppe Lavagetto: "@dzahn: use the fqdn." [operations/puppet] - 10https://gerrit.wikimedia.org/r/145472 (owner: 10Dzahn) [17:51:29] PROBLEM - puppet last run on mw1053 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [18:27:17] (03CR) 10Tim Landscheidt: [C: 04-1] "Duplicate declaration: Package[ant] is already declared in file /etc/puppet/modules/androidsdk/manifests/dependencies.pp:17; cannot redecl" [operations/puppet] - 10https://gerrit.wikimedia.org/r/125241 (owner: 10Yuvipanda) [20:31:58] (03PS1) 10Withoutaname: Remove $wgExtDistArchiveAPI and $wgExtDistProxy [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/153347 [21:21:38] PROBLEM - check configured eth on amssq33 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [21:52:58] PROBLEM - graphite.wikimedia.org on tungsten is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:53:49] PROBLEM - Number of mediawiki jobs running on tungsten is CRITICAL: CRITICAL: Anomaly detected: 29 data above and 0 below the confidence bounds [21:53:49] PROBLEM - Number of mediawiki jobs queued on tungsten is CRITICAL: CRITICAL: Anomaly detected: 29 data above and 0 below the confidence bounds [21:54:42] icinga-wm: why the repitition? [21:54:48] RECOVERY - graphite.wikimedia.org on tungsten is OK: HTTP OK: HTTP/1.1 200 OK - 1607 bytes in 0.004 second response time [22:15:39] PROBLEM - puppet last run on amssq31 is CRITICAL: CRITICAL: Epic puppet fail [22:15:59] (03PS6) 10Tim Landscheidt: Tools: Add some i386 compat packages to exec nodes [operations/puppet] - 10https://gerrit.wikimedia.org/r/125241 (owner: 10Yuvipanda) [22:33:02] !log csteipp Started scap: Deploy Ibe28a69c9fbab00b81c53b1643df722a3f1fbf19 at Eriks request [22:33:09] Logged the message, Master [22:34:39] RECOVERY - puppet last run on amssq31 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [22:36:03] (03CR) 10Tim Landscheidt: "Tested on Toolsbeta (that is, the installation of the packages, not if Android SDK works afterwards)." [operations/puppet] - 10https://gerrit.wikimedia.org/r/125241 (owner: 10Yuvipanda) [22:37:59] PROBLEM - graphite.wikimedia.org on tungsten is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:42:29] PROBLEM - puppet last run on mw1053 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [22:51:48] RECOVERY - graphite.wikimedia.org on tungsten is OK: HTTP OK: HTTP/1.1 200 OK - 1607 bytes in 0.006 second response time [22:51:58] PROBLEM - Number of mediawiki jobs running on tungsten is CRITICAL: CRITICAL: Anomaly detected: 29 data above and 0 below the confidence bounds [22:51:58] PROBLEM - Number of mediawiki jobs queued on tungsten is CRITICAL: CRITICAL: Anomaly detected: 29 data above and 0 below the confidence bounds [22:52:29] PROBLEM - puppet last run on mw1053 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [22:59:16] !log csteipp Finished scap: Deploy Ibe28a69c9fbab00b81c53b1643df722a3f1fbf19 at Eriks request (duration: 26m 13s) [22:59:21] Logged the message, Master [22:59:34] Eloquence: ^ [23:01:38] Nasty stuff. [23:27:46] (03CR) 10Legoktm: [C: 031] "Can be deployed whenever." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/153347 (owner: 10Withoutaname) [23:36:16] !log ori Synchronized php-1.24wmf16/extensions/MassMessage: Update MassMessage for I840c98dca: Fix MassMessage::getMessengerUser() after Password API changes (duration: 00m 06s) [23:36:22] Logged the message, Master [23:36:33] legoktm: can you verify it? [23:36:56] uhh, not unless you create a new wiki ;) [23:38:10] the account already exists on every single WMF site [23:39:48] hah [23:40:02] i wonder if we could talk greg-g into creating a new wiki [23:40:09] * ori not serious [23:40:26] * greg-g shouldn't still be up [23:40:36] testwikidata2wiki [23:40:46] * ori submits patch [23:42:01] OK, I'll remain pingable for a bit if there are any issues, but am otherwise packing up [23:42:06] see you folks [23:43:36] öhm: Does mw.org works for you? [23:43:49] I get this error: PHP fatal error in /usr/local/apache/common-local/php-1.24wmf16/extensions/MassMessage/includes/MassMessage.php line 45: Call to undefined method User::getPassword() [23:44:07] yay. [23:44:12] legoktm: ^ [23:44:26] errrrr [23:44:26] if I don't come up with a fix in the next minute or two I'll revert. [23:44:29] waaaat [23:44:43] oh, that's the problem you're talking about, ok :) [23:45:04] oh crap [23:45:11] but it's only for logged in users, but that you know i think?! [23:45:15] see what happens when I stay up late? [23:45:22] :P [23:45:32] greg-g: sleep; i'll fix or revert in the next minute [23:45:36] :) [23:45:45] (I can't sleep yet) [23:46:19] ori: I forgot https://gerrit.wikimedia.org/r/#/c/151569/ wasn't deployed yet. Either that needs to be backported to wmf16 (preferred) or revert the MM change [23:46:46] no, I'm reverting [23:46:52] I can't reason about the safety of backporting it now, sorry [23:47:08] ok, that's reasonable. [23:47:10] sorry :/ [23:47:51] !log ori Synchronized php-1.24wmf16/extensions/MassMessage/includes: Revert MassMessage to 9884fbb50a (duration: 00m 06s) [23:47:56] Logged the message, Master [23:48:44] thx ori, legoktm [23:49:05] FlorianSW: is whatever you were seeing back to normal now? [23:49:15] yes [23:49:16] greg-g: yes :) [23:49:48] though, why is that function being called in a normal page load? [23:49:56] * legoktm finds stacktrace [23:50:19] I probably need to think about this a bit more. [23:50:43] let's try this again: [23:50:47] OK, I'll remain pingable for a bit if there are any issues, but am otherwise packing up [23:50:50] see you folks [23:51:02] later ori, thanks :) [23:51:06] hah [23:51:08] bye, and thanks [23:51:25] legoktm: https://github.com/wikimedia/mediawiki-extensions-MassMessage/blob/4f9f5bd864151b26eeb236e4a4692250d031afde/MassMessage.hooks.php#L164 [23:51:42] oh ughhhhh [23:51:44] legoktm: also [23:51:45] that makes sense [23:51:47] needs a test [23:51:50] evidently :) [23:52:01] ori: there is a test! but the rest runs against master [23:52:31] test* [23:52:38] but not on backported changes?! [23:54:25] guess not. [23:54:33] FlorianSW: https://bugzilla.wikimedia.org/show_bug.cgi?id=69381 btw [23:54:48] yeah, see this bug already :) [23:59:44] (03CR) 10Chad: [C: 031] Point wgSiteMatrixFile at full path (not /apache symlink) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/152758 (owner: 10Reedy)