[00:00:04] RoanKattouw, ^d, marktraceur, MaxSem: Dear anthropoid, the time has come. Please deploy Evening SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20141112T0000). [00:02:59] PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet). [00:03:23] (03PS1) 10Alexandros Kosiaris: Change postgresql dir to have version and cluster [puppet] - 10https://gerrit.wikimedia.org/r/172659 [00:10:28] RECOVERY - Unmerged changes on repository puppet on strontium is OK: No changes to merge. [00:44:09] PROBLEM - git.wikimedia.org on antimony is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:48:19] RECOVERY - git.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 57905 bytes in 0.214 second response time [00:52:11] PROBLEM - MySQL Replication Heartbeat on db1016 is CRITICAL: CRIT replication delay 327 seconds [00:52:12] PROBLEM - MySQL Slave Delay on db1016 is CRITICAL: CRIT replication delay 330 seconds [00:53:28] RECOVERY - MySQL Replication Heartbeat on db1016 is OK: OK replication delay -0 seconds [00:53:32] RECOVERY - MySQL Slave Delay on db1016 is OK: OK replication delay 0 seconds [01:12:47] (03PS1) 10Matanya: (bug 73197) allow admins to give patroller right on Hebrew Wiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/172663 [01:14:34] Reedy: i forget this ^ part earlier, if you don't nind, quite useless with out [01:49:13] apergos, DNS is down for mail.wikipedia.org: https://bugzilla.wikimedia.org/show_bug.cgi?id=73290 [01:50:54] PROBLEM - Disk space on logstash1002 is CRITICAL: DISK CRITICAL - free space: / 16707 MB (3% inode=99%): [01:52:47] superm401: https://github.com/wikimedia/operations-dns/commit/a1607cedac1454ad9cad4f1e2742655f6bb13e59 https://github.com/wikimedia/operations-dns/commit/3a7f472cb3e9bcd03f0492cfdd8c0a2156f448d3 [01:52:49] Presumably [01:54:45] Reedy, hmm, thanks. People have been working on fixing SSL on mail.wikipedia.org quite recently, on the bug I see also-ed. [02:09:15] !log LocalisationUpdate completed (1.25wmf6) at 2014-11-12 02:09:14+00:00 [02:09:21] Logged the message, Master [02:12:52] RECOVERY - Disk space on logstash1002 is OK: DISK OK [02:15:49] !log LocalisationUpdate completed (1.25wmf7) at 2014-11-12 02:15:49+00:00 [02:15:53] Logged the message, Master [02:20:00] greg-g: bd808|BUFFER ^^ That's promising [02:20:09] 16 minutes to do 2 l10n updates [03:35:17] !log LocalisationUpdate ResourceLoader cache refresh completed at Wed Nov 12 03:35:17 UTC 2014 (duration 35m 16s) [03:35:21] Logged the message, Master [03:54:13] PROBLEM - puppetmaster https on virt1000 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [04:56:23] PROBLEM - git.wikimedia.org on antimony is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:03:32] RECOVERY - git.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 57901 bytes in 0.112 second response time [05:12:33] PROBLEM - Disk space on vanadium is CRITICAL: DISK CRITICAL - free space: / 4276 MB (3% inode=94%): [05:21:52] PROBLEM - Disk space on vanadium is CRITICAL: DISK CRITICAL - free space: / 4227 MB (3% inode=94%): [06:29:04] PROBLEM - puppet last run on mw1144 is CRITICAL: CRITICAL: Puppet has 3 failures [06:29:04] RECOVERY - Disk space on vanadium is OK: DISK OK [06:29:15] PROBLEM - puppet last run on db2018 is CRITICAL: CRITICAL: Puppet has 1 failures [06:46:13] RECOVERY - puppet last run on db2018 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [06:46:54] RECOVERY - puppet last run on mw1144 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [06:49:23] RECOVERY - puppetmaster https on virt1000 is OK: HTTP OK: Status line output matched 400 - 335 bytes in 0.017 second response time [07:25:15] PROBLEM - git.wikimedia.org on antimony is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:29:35] sei pesante, antimony [07:42:55] RECOVERY - git.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 57901 bytes in 0.323 second response time [08:06:07] (03PS2) 10Giuseppe Lavagetto: HAT: remove addencoding directives that are harmful on apache 2.4 [puppet] - 10https://gerrit.wikimedia.org/r/172578 [08:06:20] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] HAT: remove addencoding directives that are harmful on apache 2.4 [puppet] - 10https://gerrit.wikimedia.org/r/172578 (owner: 10Giuseppe Lavagetto) [08:17:53] <_joe_> !log stress testing a group of HHVM servers in anticipation for the move to 20% of traffic [08:17:57] Logged the message, Master [08:29:45] (03PS2) 10Yuvipanda: diamond: Don't choke on puppet syntax error failures [puppet] - 10https://gerrit.wikimedia.org/r/172592 [08:29:52] anyone wanna +1 ^? [08:32:27] (03CR) 10Yuvipanda: [C: 032] diamond: Don't choke on puppet syntax error failures [puppet] - 10https://gerrit.wikimedia.org/r/172592 (owner: 10Yuvipanda) [08:40:15] PROBLEM - git.wikimedia.org on antimony is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:46:26] RECOVERY - git.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 57901 bytes in 6.325 second response time [08:50:05] (03PS7) 10Nemo bis: Make BounceHandler extension work on Meta-Wiki [puppet] - 10https://gerrit.wikimedia.org/r/168622 (owner: 1001tonythomas) [08:50:49] (03PS5) 10Nemo bis: Deploy BounceHandler extension to production [mediawiki-config] - 10https://gerrit.wikimedia.org/r/172322 (https://bugzilla.wikimedia.org/69019) (owner: 10Legoktm) [08:51:36] (03CR) 10Nemo bis: "This certainly needs something to be done as regards mail relay etc.; I *think* that's what I7badb2383a3e4d6d78e28bea6a97c5a51d5be64d is f" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/172322 (https://bugzilla.wikimedia.org/69019) (owner: 10Legoktm) [08:57:52] (03PS4) 10Jalexander: Add SecurePoll specific dblist and allow SecurePoll to use [mediawiki-config] - 10https://gerrit.wikimedia.org/r/172449 (https://bugzilla.wikimedia.org/73245) [09:02:16] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0] [09:03:56] (03CR) 10Nemo bis: [C: 04-1] "Not clear why a new list would be needed, please reply to Reedy. https://bugzilla.wikimedia.org/show_bug.cgi?id=73245#c4" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/172449 (https://bugzilla.wikimedia.org/73245) (owner: 10Jalexander) [09:14:26] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0] [09:27:24] (03PS1) 10ArielGlenn: re-add mail.wp.org, lost in I1489f35f9fd335da9b509cc21015177020732d87. [dns] - 10https://gerrit.wikimedia.org/r/172677 [09:29:22] (03CR) 10ArielGlenn: [C: 032] re-add mail.wp.org, lost in I1489f35f9fd335da9b509cc21015177020732d87. [dns] - 10https://gerrit.wikimedia.org/r/172677 (owner: 10ArielGlenn) [10:16:35] <_joe_> !log depooling mw1189 from the api pool for reimaging [10:16:37] Logged the message, Master [10:28:03] <_joe_> !log repooling mw1189 with a reduced hhvm thread count for testing (puppet disabled, as well) [10:28:06] Logged the message, Master [10:50:06] PROBLEM - puppet last run on bast4001 is CRITICAL: CRITICAL: puppet fail [11:01:56] (03CR) 10Alexandros Kosiaris: [C: 031] swift: report statsd data to localhost [puppet] - 10https://gerrit.wikimedia.org/r/172549 (owner: 10Filippo Giunchedi) [11:07:42] (03CR) 10TTO: [C: 04-1] "Per the bug, we don't need yet another dblist..." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/172449 (https://bugzilla.wikimedia.org/73245) (owner: 10Jalexander) [11:08:45] RECOVERY - puppet last run on bast4001 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [11:16:51] (03CR) 10Alexandros Kosiaris: [C: 032] Introduce role::ganglia classes [puppet] - 10https://gerrit.wikimedia.org/r/172553 (owner: 10Alexandros Kosiaris) [11:16:51] (03CR) 10Alexandros Kosiaris: [C: 032] Assign role::ganglia::web to uranium [puppet] - 10https://gerrit.wikimedia.org/r/172555 (owner: 10Alexandros Kosiaris) [11:21:06] PROBLEM - puppet last run on uranium is CRITICAL: CRITICAL: puppet fail [11:25:28] (03PS3) 10Filippo Giunchedi: carbon-c-relay: add debian packaging [debs/carbon-c-relay] - 10https://gerrit.wikimedia.org/r/172228 [11:27:49] (03PS4) 10Filippo Giunchedi: carbon-c-relay: add debian packaging [debs/carbon-c-relay] - 10https://gerrit.wikimedia.org/r/172228 [11:28:06] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] carbon-c-relay: add debian packaging [debs/carbon-c-relay] - 10https://gerrit.wikimedia.org/r/172228 (owner: 10Filippo Giunchedi) [11:28:45] akosiaris: thanks! [11:34:23] (03PS1) 10Alexandros Kosiaris: Don't include misc::monitoring::views on uranium [puppet] - 10https://gerrit.wikimedia.org/r/172689 [11:36:58] (03CR) 10Alexandros Kosiaris: [C: 032] Don't include misc::monitoring::views on uranium [puppet] - 10https://gerrit.wikimedia.org/r/172689 (owner: 10Alexandros Kosiaris) [11:38:59] <_joe_> misc::monitoring? [11:41:41] <_joe_> I can't see ganglia.wikimedia.org [11:41:47] <_joe_> is someone on it? [11:42:14] yes [11:42:15] me [11:42:26] <_joe_> ok [11:42:28] <_joe_> :) [11:42:46] (03PS1) 10Alexandros Kosiaris: Fixes for role::ganglia::web [puppet] - 10https://gerrit.wikimedia.org/r/172690 [11:42:58] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] Fixes for role::ganglia::web [puppet] - 10https://gerrit.wikimedia.org/r/172690 (owner: 10Alexandros Kosiaris) [11:44:26] RECOVERY - puppet last run on uranium is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [11:52:59] (03PS1) 10Alexandros Kosiaris: role::ganglia::web rrdcached fixes [puppet] - 10https://gerrit.wikimedia.org/r/172693 [11:57:58] (03CR) 10Alexandros Kosiaris: [C: 032] role::ganglia::web rrdcached fixes [puppet] - 10https://gerrit.wikimedia.org/r/172693 (owner: 10Alexandros Kosiaris) [12:02:35] PROBLEM - Disk space on uranium is CRITICAL: DISK CRITICAL - free space: / 326 MB (3% inode=46%): [12:04:33] akosiaris: ^ [12:05:32] yeah I know [12:08:45] PROBLEM - HTTP on uranium is CRITICAL: Connection refused [12:12:17] (03PS1) 10Alexandros Kosiaris: role::ganglia::web rrdcached socket support [puppet] - 10https://gerrit.wikimedia.org/r/172695 [12:15:04] (03CR) 10Alexandros Kosiaris: [C: 032] role::ganglia::web rrdcached socket support [puppet] - 10https://gerrit.wikimedia.org/r/172695 (owner: 10Alexandros Kosiaris) [12:17:00] RECOVERY - HTTP on uranium is OK: HTTP OK: HTTP/1.1 302 Found - 426 bytes in 0.008 second response time [12:18:35] _joe_: paravoid: ganglia fixed [12:18:42] sorry, I should have tested it more :-( [12:18:45] RECOVERY - Disk space on uranium is OK: DISK OK [12:19:00] it was meant to be way less problematic [12:19:15] <_joe_> akosiaris: well, I'd be worried if it was :P [12:19:16] at least now we got rrdcached support :-) [12:26:56] Preparing for firmware update. Please wait [12:27:05] atftpd is a PITA ... [12:29:23] (03CR) 10Faidon Liambotis: [C: 032] "LGTM." [puppet] - 10https://gerrit.wikimedia.org/r/165779 (owner: 10Ori.livneh) [12:34:25] PROBLEM - Disk space on uranium is CRITICAL: DISK CRITICAL - free space: / 337 MB (3% inode=59%): [12:39:25] RECOVERY - Disk space on uranium is OK: DISK OK [12:53:12] (03PS1) 10ArielGlenn: draft notifier to delete salt keys of a labs instance when it's deleted [puppet] - 10https://gerrit.wikimedia.org/r/172700 [13:15:41] (03PS1) 10QChris: Extract EventLoggings log directory into a variable [puppet] - 10https://gerrit.wikimedia.org/r/172705 [13:15:43] (03PS1) 10QChris: Move Eventlogging logs underneath /srv, which has more free space [puppet] - 10https://gerrit.wikimedia.org/r/172706 [13:15:45] (03PS1) 10QChris: Retain 90 days of EventLogging logs [puppet] - 10https://gerrit.wikimedia.org/r/172707 (https://bugzilla.wikimedia.org/69029) [13:18:08] (03PS2) 10Filippo Giunchedi: swift: report statsd data to localhost [puppet] - 10https://gerrit.wikimedia.org/r/172549 [13:18:25] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] swift: report statsd data to localhost [puppet] - 10https://gerrit.wikimedia.org/r/172549 (owner: 10Filippo Giunchedi) [13:19:45] (03CR) 10QChris: Move Eventlogging logs underneath /srv, which has more free space (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/172706 (owner: 10QChris) [13:32:25] !log rolling reload of swift on ms-fe1* to pick up statsd changes [13:32:32] Logged the message, Master [13:36:46] PROBLEM - Disk space on uranium is CRITICAL: DISK CRITICAL - free space: / 307 MB (3% inode=87%): [13:42:56] RECOVERY - Disk space on uranium is OK: DISK OK [13:46:57] (03CR) 10QChris: Link aggregator dataset into wikimetrics public webspace (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/172285 (https://bugzilla.wikimedia.org/72740) (owner: 10QChris) [13:49:35] mark: around? tony & reedy are ready to turn on VERP addresses [13:49:49] I am fwiw :) [13:49:54] ok [13:50:30] paravoid: I can deal with the mail puppet merge and watch logs to confirm that things are working as expected [13:50:39] which one? [13:50:45] https://gerrit.wikimedia.org/r/#/c/168622/7/manifests/role/mail.pp [13:51:08] give me a sec [13:51:11] k [13:55:00] !log rolling reload of swift on ms-be1* to pick up statsd changes [13:55:04] Logged the message, Master [13:55:55] (03CR) 10Faidon Liambotis: [C: 031] Make BounceHandler extension work on Meta-Wiki [puppet] - 10https://gerrit.wikimedia.org/r/168622 (owner: 1001tonythomas) [13:56:32] Jeff_Green: yay :) thanks paravoid ! [13:56:47] although I would start from mediawiki.org, as Aaron noted [13:57:08] but if you're feeling confident enough and Greg is okay with it, meta works for me as well [13:57:39] i think aaron is referring to the hostname for the HTTP request [13:57:59] or is there another comment I'm not seeing? [13:58:12] oh wait, aaron said metawiki [13:58:20] yeah :) [13:58:42] paravoid: there's been some back and forth on the best HTTP endpoint [13:58:51] right, I see that [14:01:34] Jeff_Green: we can have Reedy finish https://gerrit.wikimedia.org/r/#/c/172322/ ? [14:01:34] how come we're not using each user's homewiki? [14:02:21] paravoid: i think because we don't know it until we decode the verp address [14:03:22] paravoid: we have the extension installed on every wiki in prod - and the exim talking back only to meta wiki [14:03:45] and once a bounce from some other wiki arrives - the $wikiId in the prefix part can help us look in the right table https://github.com/wikimedia/mediawiki-extensions-BounceHandler/blob/master/includes/ApiBounceHandler.php#L36 [14:03:51] (03PS1) 10Alexandros Kosiaris: Allow specifying journal_dir for rrdcached [puppet] - 10https://gerrit.wikimedia.org/r/172715 [14:03:56] and get the user details! [14:04:25] do emails originate from a common domain, e.g. @wikimedia.org, though? [14:04:42] they do [14:05:06] same as before [14:05:10] what's an example address? I saw the $wikiId-base36()... on the bug report but I'm not sure if it was accurate [14:05:25] paravoid: wiki-testwiki-2-nevvrp-Ru3q8IfTlVGXczYk@wikimedia.org [14:05:49] that will make sure that the bounce get redirected to wiki with wikiId = testwiki [14:05:53] right [14:06:18] Jeff_Green: so why did you submit that SPF change for wikipedia.org then? [14:06:41] they changed the plan [14:06:54] when I submitted that, they were talking about sending out with per-wiki domains [14:07:02] right, because that's what I remembered [14:07:02] that's gone back and forth too [14:07:06] hence my confusion :) [14:07:10] yeah [14:07:25] I didn't object b/c I think a single domain is cleaner anyway [14:07:32] yup, agreed [14:07:48] i'll purge that commit [14:07:55] "abandon" :P [14:08:15] stab [14:08:19] hah [14:08:28] where is git stab when you need it [14:09:18] tonythomas: are any special provisions taken to ensure the localpart is going to be < 64 chars? [14:09:44] (03Abandoned) 10Jgreen: add a neutral SPF record for wikipedia.org and other domains of template [dns] - 10https://gerrit.wikimedia.org/r/171324 (owner: 10Jgreen) [14:10:16] paravoid: of course ! as per https://github.com/wikimedia/mediawiki-extensions-BounceHandler/blob/master/includes/VerpAddressGenerator.php#L67 : we have The generated hash is cut down to 12 ( 96 bits ) instead of the full 120 bits. [14:12:02] we made sure that VERP gets < 64 in this commit https://github.com/wikimedia/mediawiki-extensions-BounceHandler/commit/d325bb411370c7aca4f3d5e1fef2e2efc0a1fd50 [14:12:20] that's what I'd like to hear :) [14:12:55] :) thanks to the reviewers ! [14:14:03] ok [14:14:06] looks reasonable [14:14:45] (03CR) 10Anomie: "I note that Nemo's -1, and the discussion on the bug that is the basis for TTO's -1, seems to be based entirely on a faulty understanding " [mediawiki-config] - 10https://gerrit.wikimedia.org/r/172449 (https://bugzilla.wikimedia.org/73245) (owner: 10Jalexander) [14:14:48] great [14:15:01] we're ready to fire it up then? [14:15:07] yup, go ahead :) [14:15:11] thx [14:15:33] I didn't review the mediawiki extension of course [14:15:36] I don't think I need to :) [14:15:59] yep, there have been a lot of eyes on that [14:16:14] should we coordinate the config changes here or in #dev? [14:16:39] Reedy: hey ! you around ? [14:16:40] mediawiki-config? [14:16:56] yeah - this one https://gerrit.wikimedia.org/r/#/c/172322/ [14:16:58] on which wikis do you plan to roll it out today? [14:17:10] meta ! [14:18:06] tonythomas: which wiki's start using VERP addresses as of today's changes? [14:18:12] yes, that was the question :) [14:18:29] all, as far as I can see [14:18:31] let's not do that [14:18:34] Jeff_Green: we are doing this one meta right ? [14:19:54] (03CR) 10Faidon Liambotis: [C: 04-1] Deploy BounceHandler extension to production (033 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/172322 (https://bugzilla.wikimedia.org/69019) (owner: 10Legoktm) [14:20:23] tonythomas: hmm. I haven't looked at the wiki configuration since last week [14:20:43] let's look that over now [14:20:50] my understanding is that "meta" will be special, in the sense that its API will accept action=bouncehandler [14:21:09] but that the other wikis will also change their behavior, by sending VERP emails [14:21:11] paravoid: true - once we have our extension over there [14:21:22] those two rollouts should be independent, though [14:21:41] paravoid: true that - in that case we should have https://gerrit.wikimedia.org/r/#/c/172322/ rolling out only on meta [14:21:57] meta can start acception action=bouncehandler, but it shouldn't send VERP emails yet [14:22:07] PROBLEM - git.wikimedia.org on antimony is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:22:11] these two shouldn't be tied with each other [14:22:31] I thought the same of the exim config actually [14:22:58] i.e. it would be good to have a toggle to turn off HTTP processing and revert :blackhole: [14:23:14] err revert to :blackhole: for wikimail bounces [14:23:25] it should be rolled out per-wiki imho [14:23:34] paravoid: we should have VERP generation first - and later meta receiving action=bouncehandler, right ? [14:23:46] no, the other way around [14:23:47] as we did with beta [14:24:08] paravoid: we will have nothing arriving at action=bouncehandler though [14:24:13] <_joe_> !log load test on hhvm done [14:24:18] Logged the message, Master [14:24:27] if we are not generating VERP address in meta [14:24:47] (03PS1) 10Gilles: Set different ImageMetrics sampling factor for logged-in users [mediawiki-config] - 10https://gerrit.wikimedia.org/r/172720 [14:27:27] !log restarting elastic1016 to pick up new plugins.... half way done [14:27:29] Logged the message, Master [14:29:23] gi11es: You seem to have linked the wrong change for this morning's SWAT [14:29:39] anomie: the first one was done overnight by Reedy [14:29:42] it seems [14:30:12] gi11es: You linked to a Flow bugfix, nothing to do with FSFileBackend [14:30:22] ah, ok, let me fix that [14:31:57] PROBLEM - swift-object-auditor on ms-be2010 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-object-auditor [14:32:45] (03CR) 10Ottomata: "I am confued about /var/log/eventlogging/archive and /srv/eventlogging-logs. They look like they have the same files, but /srv/eventloggi" [puppet] - 10https://gerrit.wikimedia.org/r/172706 (owner: 10QChris) [14:32:52] ms-be2010 is me [14:33:12] (03PS1) 10Giuseppe Lavagetto: Have 20% of the anonymous traffic sent to HHVM [mediawiki-config] - 10https://gerrit.wikimedia.org/r/172724 [14:33:34] (03PS2) 10Ottomata: Extract EventLoggings log directory into a variable [puppet] - 10https://gerrit.wikimedia.org/r/172705 (owner: 10QChris) [14:34:17] RECOVERY - git.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 57929 bytes in 0.314 second response time [14:36:05] (03CR) 10Ottomata: "Ah I see. eventlogging-logs is some old artifact we can delete." [puppet] - 10https://gerrit.wikimedia.org/r/172706 (owner: 10QChris) [14:36:23] (03CR) 10Ottomata: [C: 032] Extract EventLoggings log directory into a variable [puppet] - 10https://gerrit.wikimedia.org/r/172705 (owner: 10QChris) [14:38:07] RECOVERY - swift-object-auditor on ms-be2010 is OK: PROCS OK: 3 processes with regex args ^/usr/bin/python /usr/bin/swift-object-auditor [14:39:09] (03CR) 10Ottomata: [C: 032] "All good points. I concede!" [puppet] - 10https://gerrit.wikimedia.org/r/172285 (https://bugzilla.wikimedia.org/72740) (owner: 10QChris) [14:39:27] PROBLEM - swift-object-replicator on ms-be2010 is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/bin/python /usr/bin/swift-object-replicator [14:40:49] Jeff_Green, tonythomas: so anyway; I'd strongly prefer it if this was done in an incremental rollout, not an on/off toggle [14:41:16] paravoid: agreed [14:51:57] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] Have 20% of the anonymous traffic sent to HHVM [mediawiki-config] - 10https://gerrit.wikimedia.org/r/172724 (owner: 10Giuseppe Lavagetto) [14:55:27] !log oblivian Synchronized wmf-config/CommonSettings.php: Open HHVM to 20% of anons (duration: 00m 06s) [14:55:31] Logged the message, Master [15:03:37] (03PS12) 10Ottomata: Initial commit of Cassandra puppet module [puppet/cassandra] - 10https://gerrit.wikimedia.org/r/166888 [15:09:56] PROBLEM - Disk space on uranium is CRITICAL: DISK CRITICAL - free space: / 306 MB (3% inode=87%): [15:16:37] <_joe_> akosiaris: are you looking into this? ^^ [15:20:14] yup [15:21:17] RECOVERY - Disk space on uranium is OK: DISK OK [15:21:55] (03CR) 10Alexandros Kosiaris: [C: 032] Allow specifying journal_dir for rrdcached [puppet] - 10https://gerrit.wikimedia.org/r/172715 (owner: 10Alexandros Kosiaris) [15:23:07] PROBLEM - puppet last run on uranium is CRITICAL: CRITICAL: puppet fail [15:26:07] (03PS1) 10Alexandros Kosiaris: role::ganglia::web fix invocation of rrdcached class [puppet] - 10https://gerrit.wikimedia.org/r/172747 [15:27:37] (03CR) 10Alexandros Kosiaris: [C: 032] role::ganglia::web fix invocation of rrdcached class [puppet] - 10https://gerrit.wikimedia.org/r/172747 (owner: 10Alexandros Kosiaris) [15:29:17] RECOVERY - puppet last run on uranium is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [15:29:36] _joe_: ^ this should fix the uranium disk issues permanently [15:29:46] https://gerrit.wikimedia.org/r/172715 [15:31:38] <_joe_> akosiaris: nice :) [15:32:19] I was not expecting the journal to grow at a rate of 1,1G per 20mins [15:32:41] not that I had run the numbers anyway... [15:37:11] <_joe_> akosiaris: eheh [15:37:25] <_joe_> 1 gb/20 mins is _a_lot_ [15:42:51] (03CR) 10QChris: "> I think we should symlink /var/log/eventlogging ->" [puppet] - 10https://gerrit.wikimedia.org/r/172706 (owner: 10QChris) [15:48:19] I suppose I may as well SWAT today [15:50:09] gi11es, hoo, anomie: Ping for SWAT in 10 minutes [15:50:11] * anomie is here [15:50:19] anomie: pong [15:50:25] * hoo around [15:53:13] (03CR) 10Ottomata: "Hm, good point, so these logs are a data output of eventlogging. Hm." [puppet] - 10https://gerrit.wikimedia.org/r/172706 (owner: 10QChris) [16:00:04] manybubbles, anomie, ^d, marktraceur, gi11es: Respected human, time to deploy Morning SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20141112T1600). Please do the needful. [16:00:09] * anomie starts SWAT [16:00:12] in meeting today! [16:00:19] thanks anomie! [16:00:19] hoo: I'll do yours first because it should be quick [16:00:33] (03PS2) 10Anomie: Add "featured portal" badge (Q17580674) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/172580 (https://bugzilla.wikimedia.org/73193) (owner: 10Hoo man) [16:00:41] ok [16:00:42] (03CR) 10Anomie: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/172580 (https://bugzilla.wikimedia.org/73193) (owner: 10Hoo man) [16:00:53] (03Merged) 10jenkins-bot: Add "featured portal" badge (Q17580674) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/172580 (https://bugzilla.wikimedia.org/73193) (owner: 10Hoo man) [16:01:13] !log anomie Synchronized wmf-config/Wikibase.php: SWAT: Add "featured portal" badge (Q17580674) [[gerrit:172729]] (duration: 00m 10s) [16:01:14] hoo: ^ Test please [16:01:16] Logged the message, Master [16:01:23] gi11es: You're next [16:01:29] on that [16:03:16] anomie: Looks fine, thanks [16:03:32] (03PS2) 10Anomie: Set different ImageMetrics sampling factor for logged-in users [mediawiki-config] - 10https://gerrit.wikimedia.org/r/172720 (owner: 10Gilles) [16:03:39] (03CR) 10Anomie: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/172720 (owner: 10Gilles) [16:03:47] (03Merged) 10jenkins-bot: Set different ImageMetrics sampling factor for logged-in users [mediawiki-config] - 10https://gerrit.wikimedia.org/r/172720 (owner: 10Gilles) [16:04:10] !log anomie Synchronized wmf-config: SWAT: Set different ImageMetrics sampling factor for logged-in users [[gerrit:172720]] (duration: 00m 12s) [16:04:11] gi11es: ^ Test please [16:04:11] Logged the message, Master [16:07:33] gi11es: Are you there? [16:08:17] PROBLEM - HHVM busy threads on mw1114 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [90.0] [16:11:10] hmm. Well, moving on then. [16:12:21] anomie: yeah I'm testing the change [16:14:28] RECOVERY - HHVM busy threads on mw1114 is OK: OK: Less than 1.00% above the threshold [60.0] [16:14:35] I see the value in eval.php on terbium but not in JS yet [16:14:46] (03CR) 10Nemo bis: Deploy BounceHandler extension to production (032 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/172322 (https://bugzilla.wikimedia.org/69019) (owner: 10Legoktm) [16:16:01] ah, nevermind... it's because the code that sends it to JS is only on master right now :) [16:16:07] anomie: change deployed fine [16:16:47] gi11es: ok [16:17:54] (03CR) 10Nuria: [C: 031] Link aggregator dataset into wikimetrics public webspace [puppet] - 10https://gerrit.wikimedia.org/r/172285 (https://bugzilla.wikimedia.org/72740) (owner: 10QChris) [16:19:36] PROBLEM - Disk space on db1017 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=74%): [16:21:53] yeah yeah [16:22:37] RECOVERY - Disk space on db1017 is OK: DISK OK [16:25:09] !log anomie Synchronized php-1.25wmf7/extensions/MultimediaViewer/: SWAT: Backport MediaViewer options menu layout fix [[gerrit:172737]] (duration: 00m 09s) [16:25:11] gi11es: ^ Test please [16:25:16] Logged the message, Master [16:25:33] anomie: testing [16:25:55] anomie: works fine, all clear [16:26:05] thanks! [16:26:11] anomie: Your turn [16:26:13] !log anomie Started scap: SWAT: SecurePoll fix for jump-text and title on create/edit [[gerrit:172718]] [16:26:16] Logged the message, Master [16:30:25] (03CR) 10Rush: [C: 04-1] "just -1'ing as we decided to wait on this for the moment, only to keep our changes to a minimum pre-migration." [puppet] - 10https://gerrit.wikimedia.org/r/168509 (owner: 1020after4) [16:40:20] paravoid: is lead another mail server? [16:40:34] (03PS2) 10Giuseppe Lavagetto: hiera: allow regex-based searches [puppet] - 10https://gerrit.wikimedia.org/r/172552 [16:40:45] legoktm: yes [16:40:54] and it's going to be replaced soonish as well [16:41:11] with yet another one [16:41:16] ok, what's it's IP? (or where should I look to find it?) [16:41:23] lead.wikimedia.org? :) [16:41:43] both of those mailservers have IPv6 as well [16:41:58] right now it doesn't matter I guess, since the exim config justs hits appservers.svc.eqiad.wmnet, which is ipv4 [16:42:28] but if e.g. switched it over to our frontends, then you'd see its IPv6 address instead (in XFF even!) [16:42:38] so I have to say, why are we doing IP-based authentication here in the first place? [16:43:19] what would you suggest instead? [16:45:48] token-based maybe? [16:45:55] using a header? [16:48:26] !log anomie Finished scap: SWAT: SecurePoll fix for jump-text and title on create/edit [[gerrit:172718]] (duration: 22m 13s) [16:48:29] anomie: ^ Test please [16:48:29] Logged the message, Master [16:49:26] anomie: d'oh, it didn't work. Forgot to actually pull the revision :( [16:50:00] paravoid: how would that work on the MW side? I'm not really following your idea :/ [16:51:03] !log anomie Synchronized php-1.25wmf7/extensions/SecurePoll/: SWAT: SecurePoll fix for jump-text and title on create/edit [[gerrit:172718]] (for real this time) (duration: 00m 09s) [16:51:06] Logged the message, Master [16:51:09] anomie: ^ Try that. i18n might not be there though. [16:51:31] anomie: ^ Yes, change is there except for i18n. :/ [16:51:44] legoktm: send a secret via curl... and verify it in the API [16:52:03] would be defined as a private var for MediaWiki and probably be in puppet private for exim [16:52:07] ah [16:52:30] yeah, that would be easy to implement [16:53:09] (03PS1) 10Ottomata: Add README.md with Usage info [puppet/kafkatee] - 10https://gerrit.wikimedia.org/r/172763 [16:53:35] anomie: I think we'll just live with the i18n being wrong until this afternoon's train deploy. [16:53:41] (03PS2) 10Ottomata: Add README.md with Usage info [puppet/kafkatee] - 10https://gerrit.wikimedia.org/r/172763 [16:53:43] * anomie is more-or-less done with SWAT [16:54:15] (03CR) 10Ottomata: [C: 032 V: 032] Add README.md with Usage info [puppet/kafkatee] - 10https://gerrit.wikimedia.org/r/172763 (owner: 10Ottomata) [17:00:04] manybubbles, ^d: Dear anthropoid, the time has come. Please deploy Search (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20141112T1700). [17:00:36] PROBLEM - puppet last run on cp3018 is CRITICAL: CRITICAL: Puppet has 1 failures [17:01:27] PROBLEM - puppet last run on amssq39 is CRITICAL: CRITICAL: Puppet has 1 failures [17:03:24] (03PS1) 10Chad: dewiki gets cirrus as default [mediawiki-config] - 10https://gerrit.wikimedia.org/r/172766 [17:05:11] (03CR) 10Manybubbles: [C: 031] dewiki gets cirrus as default [mediawiki-config] - 10https://gerrit.wikimedia.org/r/172766 (owner: 10Chad) [17:05:13] (03CR) 10Chad: [C: 032] dewiki gets cirrus as default [mediawiki-config] - 10https://gerrit.wikimedia.org/r/172766 (owner: 10Chad) [17:05:19] (03Merged) 10jenkins-bot: dewiki gets cirrus as default [mediawiki-config] - 10https://gerrit.wikimedia.org/r/172766 (owner: 10Chad) [17:05:41] <_joe_> manybubbles: \o/ [17:05:51] !log demon Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 04s) [17:05:53] Logged the message, Master [17:05:54] <_joe_> ^d: you as well [17:06:01] <_joe_> congrats [17:06:05] _joe_: wait a few hours to see if people complain! [17:06:15] <^d> Dieses Wiki verwendet eine neue Suchmaschine. (Mehr erfahren) [17:06:16] <^d> :D [17:06:44] yay [17:07:06] Suchmaschine? that doesn't sound pleasant [17:07:14] <_joe_> manybubbles: yeah, just being able to pull the lever is an accomplishment in itself [17:07:19] yeah [17:07:21] <_joe_> manybubbles: "search engine" [17:07:24] yeah [17:07:59] I had two years of German more than ten years ago. So it *looks* familiar and sometimes if I squint I can guess and be almost right. [17:08:41] <_joe_> I never actually studied german [17:09:28] <_joe_> but the words there were of quite common use [17:10:24] (03CR) 10Giuseppe Lavagetto: [C: 032] hiera: allow regex-based searches [puppet] - 10https://gerrit.wikimedia.org/r/172552 (owner: 10Giuseppe Lavagetto) [17:12:59] (03PS1) 10Giuseppe Lavagetto: hiera: use lookup_key, not key [puppet] - 10https://gerrit.wikimedia.org/r/172768 [17:13:21] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] hiera: use lookup_key, not key [puppet] - 10https://gerrit.wikimedia.org/r/172768 (owner: 10Giuseppe Lavagetto) [17:16:56] RECOVERY - puppet last run on amssq39 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [17:17:07] RECOVERY - puppet last run on cp3018 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [17:30:13] Is hashar the only maintainer for CI Jenkins? [17:31:14] I need to make a job non-voting, https://gerrit.wikimedia.org/r/#/c/172765/ [17:38:09] awight: you can add Timo as well [17:38:26] greg-g: thx! [17:38:27] they're both pretty responsive [17:38:38] filing a bug would also help :) [17:38:49] Krinkle|detached: hehe, when you are feeling less detached, I could use non-voting on this CI job: https://gerrit.wikimedia.org/r/#/c/172765/ [17:47:13] (03CR) 10GWicke: [C: 031] "LGTM." [puppet/cassandra] - 10https://gerrit.wikimedia.org/r/166888 (owner: 10Ottomata) [17:47:33] akosiaris: ping [17:47:56] gwicke: pong [17:48:19] hey; are you planning to review the restbase puppetization? [17:48:29] yup [17:48:35] cool [17:48:40] chasemp: has the day arrived where when I want a new procurement ticket I should just enter it directly in phab? Or is RT still the right place for that? [17:48:51] still RT [17:49:06] ok! [17:50:51] akosiaris: am happy to chat about it any time [17:51:13] gwicke: cool. Good to know, thanks! [17:53:26] Fatal log is full of "PHP Fatal error: Base lambda function for closure not found in /srv/mediawiki/php-1.25wmf7/extensions/Wikidata/extensions/Wikibase/lib/config/WikibaseLib.default.php on line 18" [17:53:38] question, for anyone.... [17:53:43] 1721 times in last hour [17:54:12] does bits.beta.wmflabs.org have its own varnish hosts? [17:54:56] <_joe_> bd808: from which servers? [17:55:10] _joe_: Trying to figure that out. [17:55:13] <_joe_> bd808: usually happens on API after a graceful, or scap [17:56:33] (03CR) 10GWicke: "This has been tested against a cassandra cluster set up with https://gerrit.wikimedia.org/r/#/c/166888/ in the services labs project. The " [puppet] - 10https://gerrit.wikimedia.org/r/167213 (owner: 10GWicke) [17:56:46] _joe_: Just from the top20 hosts I'd guess mw1195 mw1196 mw1205 mw1147 mw1141 [17:57:15] <_joe_> ok look how that matches http://ganglia.wikimedia.org/latest/?r=hour&cs=&ce=&m=cpu_report&s=by+name&c=API%2520application%2520servers%2520eqiad&tab=m&vn=&hide-hf=false [17:57:31] <_joe_> those servers need a reload/restart of apache [17:57:43] <_joe_> bd808: on it [17:57:56] Thanks _joe_ [17:58:46] <_joe_> !log gracefulling apache on problematic API hosts [17:58:54] Logged the message, Master [18:11:24] <_joe_> bd808: errors should've stopped [18:13:05] _joe_: Cool. Looks like the error rate looks much better in last 5 mins [18:14:12] (03PS1) 10Yuvipanda: cache: Don't setup ganglia monitoring on labs [puppet] - 10https://gerrit.wikimedia.org/r/172776 (https://bugzilla.wikimedia.org/73263) [18:14:24] ^ this fixes some betalabs puppet errors that've been erroring for a month... [18:14:26] can someone +1? [18:14:41] bblack: ^ (since it touches some varnish code) [18:16:41] <_joe_> sigh [18:16:55] _joe_: It's realm branching, and I'm not fully sure how to do this with hiera [18:17:16] <_joe_> YuviPanda: well you'd have to refactor a few things [18:17:18] yeah [18:17:21] <_joe_> for now, it is good [18:17:31] argh.. realm branching in varnish too ? [18:17:38] (03CR) 10Reedy: Deploy BounceHandler extension to production (033 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/172322 (https://bugzilla.wikimedia.org/69019) (owner: 10Legoktm) [18:17:52] I was more than happy to kill them in the ganglia_new module tbh... [18:18:07] (03CR) 10Reedy: [C: 031] "Yup" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/172449 (https://bugzilla.wikimedia.org/73245) (owner: 10Jalexander) [18:18:44] akosiaris: labs doesn't have ganglia, and the cache module includes them indiscriminately [18:19:02] _joe_: akosiaris another option is to include gmond in the labs base class, but then it's just running there uselessly, trying to send metrics that are never received... [18:19:35] <_joe_> YuviPanda: nah [18:19:40] I get the problem you are trying to solve... I don't really have a better solution ready, I just dislike all the labs branching [18:19:49] sigh, there's going to be a *lot* more branching... [18:19:49] <_joe_> YuviPanda: I can take a shot at that, but tomorrow please :) [18:19:57] lots of ganglia in the varnish code... [18:20:16] <_joe_> YuviPanda: so let's solve this once and for all in a beter way [18:21:02] What repo do integration/slave-scripts come from? [18:22:20] nvm, found it: integration-jenkins/bin/ [18:22:30] so puppet hasn't run in any of the cache nodes in betalabs for like a month now [18:22:31] damn, instance.pp has a dependency on an exec from inside the ganglia varnish class that's included in manifests/role/cache.pp.... [18:22:45] (03CR) 10BBlack: [C: 031] cache: Don't setup ganglia monitoring on labs [puppet] - 10https://gerrit.wikimedia.org/r/172776 (https://bugzilla.wikimedia.org/73263) (owner: 10Yuvipanda) [18:22:46] <_joe_> YuviPanda: lol [18:23:04] YuviPanda: not surprised [18:23:07] etherpad just hiccuped [18:23:09] I'm not going to realm branch inside the varnish module... [18:23:17] Back pretty fast [18:23:23] marktraceur: not surprised as well [18:23:27] <_joe_> YuviPanda: ping me tomorrow [18:23:30] not the best software out there [18:23:32] _joe_: ok, will do. [18:23:34] <_joe_> we can work on this toghether [18:23:51] hey btw, I am fighting with the ganglia stuff as well [18:24:03] _joe_: I'll note on the bug. I'm sure there's plenty of other similar things going on that we'll uncover as time goes by. [18:24:14] I 'll help too [18:24:16] * YuviPanda hopes ganglia in our infra dies in the next few years, replaced by graphite [18:24:20] akosiaris: :D will ping you too! [18:24:36] * akosiaris actually likes ganglia [18:24:41]