[00:00:01] probably a liaison [00:00:30] Yes, community liaison... Hence a lot more likely to be reading community discussions than developers I think [00:50:07] (03CR) 10TTO: "(2 comments)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76342 (owner: 10TTO) [00:51:11] (03CR) 10TTO: "(1 comment)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76342 (owner: 10TTO) [00:57:14] (03PS3) 10TTO: Clean up headers in CommonSettings and InitialiseSettings [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76342 [00:58:24] (03PS4) 10TTO: Clean up headers in CommonSettings and InitialiseSettings [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76342 [01:01:02] (03CR) 10Krinkle: "(1 comment)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76342 (owner: 10TTO) [01:01:57] PROBLEM - Puppet freshness on holmium is CRITICAL: No successful Puppet run in the last 10 hours [01:07:06] (03CR) 10TTO: "(1 comment)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76342 (owner: 10TTO) [01:22:18] !log delaying slave db45 [01:22:30] Logged the message, Master [02:10:34] !log LocalisationUpdate completed (1.22wmf11) at Mon Jul 29 02:10:33 UTC 2013 [02:10:45] Logged the message, Master [02:10:52] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [02:17:51] !log enabling slave db45 [02:18:01] Logged the message, Master [02:19:09] !log LocalisationUpdate completed (1.22wmf12) at Mon Jul 29 02:19:08 UTC 2013 [02:19:19] Logged the message, Master [02:33:00] !log LocalisationUpdate ResourceLoader cache refresh completed at Mon Jul 29 02:33:00 UTC 2013 [02:33:19] Logged the message, Master [02:49:02] PROBLEM - Puppet freshness on manutius is CRITICAL: No successful Puppet run in the last 10 hours [03:05:39] (03CR) 10MZMcBride: "(1 comment)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76342 (owner: 10TTO) [03:06:25] (03CR) 10MZMcBride: "(1 comment)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76342 (owner: 10TTO) [03:53:48] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:54:38] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.148 second response time [04:24:14] PROBLEM - Puppet freshness on sq41 is CRITICAL: No successful Puppet run in the last 10 hours [04:30:02] !log delaying slave db56 during OSC bug 49199 [04:30:11] Logged the message, Master [04:39:27] (03PS2) 10MZMcBride: Enable anonymous use of VisualEditor on es/fr/he/it/pl/ru/sv [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76199 (owner: 10Jforrester) [04:52:00] (03PS1) 10MZMcBride: Explicitly disabled VisualEditor by default on dewiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76468 [05:10:59] (03PS2) 10MZMcBride: Explicitly disable VisualEditor by default on dewiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76468 [05:12:02] (03CR) 10MZMcBride: "(1 comment)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/75543 (owner: 10Jforrester) [05:15:18] (03PS3) 10MZMcBride: Enable anonymous use of VisualEditor on es/fr/he/it/pl/ru/sv [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76199 (owner: 10Jforrester) [05:26:35] (03PS14) 10MZMcBride: Enable CAPTCHA for all edits of non-confirmed users on pt.wikipedia in order to reduce editing activity [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/69982 (owner: 10Alex Monk) [05:28:01] Oh, it did go through. [05:39:44] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [05:40:44] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [05:47:23] (03CR) 10Krinkle: [C: 04-1] "(1 comment)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76342 (owner: 10TTO) [05:50:01] (03CR) 10MZMcBride: "(1 comment)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76342 (owner: 10TTO) [06:10:50] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [06:11:40] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [06:18:10] PROBLEM - SSH on pdf3 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:19:10] RECOVERY - SSH on pdf3 is OK: SSH OK - OpenSSH_4.7p1 Debian-8ubuntu3 (protocol 2.0) [06:28:13] (03CR) 10Tim Starling: [C: 04-1] "I suggest changing the variable name in a separate commit. The discussion about the variable name is a distraction, and makes deployment m" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/69982 (owner: 10Alex Monk) [06:34:33] (03PS15) 10MZMcBride: Enable CAPTCHA for all edits of non-confirmed users on pt.wikipedia in order to reduce editing activity [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/69982 (owner: 10Alex Monk) [06:36:44] (03CR) 10MZMcBride: "A suggestion and marking a change -1 seem different to me." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/69982 (owner: 10Alex Monk) [07:20:07] (03CR) 10TTO: "(1 comment)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76342 (owner: 10TTO) [07:22:00] RECOVERY - check_job_queue on fenari is OK: JOBQUEUE OK - all job queues below 10,000 [07:25:10] PROBLEM - check_job_queue on fenari is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:30:40] PROBLEM - Puppetmaster HTTPS on virt0 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8140: HTTP/1.1 500 Internal Server Error [07:53:42] PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours [07:53:42] PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours [07:53:42] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours [07:53:42] PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours [07:53:42] PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours [07:53:43] PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours [08:12:38] moin [08:14:52] yo [09:50:09] RECOVERY - check_job_queue on hume is OK: JOBQUEUE OK - all job queues below 10,000 [09:53:19] PROBLEM - check_job_queue on hume is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [10:50:27] (03PS5) 10Mark Bergsma: Fix XFF handling on all Varnish clusters [operations/puppet] - 10https://gerrit.wikimedia.org/r/75860 [10:54:59] (03PS1) 10Aude: Add DataTypes extension [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76481 [10:56:12] (03CR) 10Aude: [C: 04-1] "depends on https://gerrit.wikimedia.org/r/#/c/76480/ and same thing done in wmf/1.22wmf11" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76481 (owner: 10Aude) [10:56:29] PROBLEM - Host mw31 is DOWN: PING CRITICAL - Packet loss = 100% [10:57:19] RECOVERY - Host mw31 is UP: PING OK - Packet loss = 0%, RTA = 26.79 ms [10:59:29] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:00:19] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [11:00:37] (03CR) 10Aude: "depends also on https://gerrit.wikimedia.org/r/#/c/76483/ for wmf/1.22wmf11" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76481 (owner: 10Aude) [11:01:59] PROBLEM - Puppet freshness on holmium is CRITICAL: No successful Puppet run in the last 10 hours [11:07:08] (03PS6) 10Mark Bergsma: Fix XFF handling on all Varnish clusters [operations/puppet] - 10https://gerrit.wikimedia.org/r/75860 [11:15:41] PROBLEM - Disk space on mw1158 is CRITICAL: DISK CRITICAL - free space: /tmp 659 MB (3% inode=99%): [11:18:22] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:19:21] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [11:28:22] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:29:22] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [11:38:28] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [11:39:08] PROBLEM - SSH on searchidx1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:39:30] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [11:39:58] RECOVERY - SSH on searchidx1001 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0) [11:43:19] hmm mark, does https://gerrit.wikimedia.org/r/#/c/75860/ just unset XFF for all non-Zero requests? [11:49:52] (03CR) 10MaxSem: "(1 comment)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/75860 (owner: 10Mark Bergsma) [11:58:53] MaxSem: the old code did too [11:59:04] XFF logic is terribly broken, especially so on mobile [11:59:16] i'm trying to make it slightly more sane, within the possibilities [11:59:48] yeah, but ppl will instantly notice this if you move them from squid [12:00:24] yes [12:00:40] ok, if mediawiki knows about the opera mini (and other) proxies, I don't need to strip them [12:01:40] and now that mobile introduced editing, people started noticing this stuff already. so far not with third-party hosts but with out internal networks not being treated as such, but they will bring up other proxies too [12:01:48] i kno [12:01:50] i know [12:01:57] if I had known that mobile editing would get enabled, I would have raised this [12:07:18] it's broken for ipv6 too [12:07:31] (03CR) 10Nemo bis: "Any reason not to hardcode the expiry?" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/69982 (owner: 10Alex Monk) [12:08:19] (03CR) 10MaxSem: [C: 032] Whitelist our IPv6 range [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76205 (owner: 10MaxSem) [12:08:29] speaking of which... [12:08:35] (03Merged) 10jenkins-bot: Whitelist our IPv6 range [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76205 (owner: 10MaxSem) [12:09:43] (03CR) 10Mark Bergsma: "(1 comment)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76205 (owner: 10MaxSem) [12:10:40] (03CR) 10MaxSem: "(1 comment)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76205 (owner: 10MaxSem) [12:11:07] (03PS1) 10Mark Bergsma: Revert "Whitelist our IPv6 range" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76491 [12:11:14] PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours [12:11:21] (03CR) 10Mark Bergsma: [C: 032] Revert "Whitelist our IPv6 range" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76491 (owner: 10Mark Bergsma) [12:12:56] we'll have to fix this in a better way [12:13:09] sure [12:13:16] so for mobile [12:13:22] i'm thinking of sending all traffic to eqiad again [12:13:25] that's the short term hack [12:13:40] (03CR) 10Hashar: "(2 comments)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/71932 (owner: 10Alex Monk) [12:15:40] there are currently only two caches in europe [12:15:43] we could list their individual ips [12:15:46] but it's ugly [12:15:59] especially since they're using dynamically chosen ips [12:16:20] can't you collect the host IP having a certain puppet class applied? [12:16:39] and deploy that into mediawiki config? [12:16:43] it's possible, but good luck [12:17:06] mmm, I could write a monitoring plugin:) [12:17:25] or we make trustedxff properly subnet aware [12:17:59] i mean $wgSquidServersNoPurge [12:18:08] and that variable needs a rename :P [12:18:54] please fill all the forms, and we might consider renaming it for 1.32 - b/c iz seruiz biznis:P [12:19:47] i guess if an entry has a /prefix it can become a subnet check [12:20:12] nah, it needs some coding for it [12:20:27] that's what I mean [12:20:27] so far it's just a in_array [12:20:34] i know, I looked at the code [12:20:38] I was just talking with krenair how $wgSquidServersNoPurge is an horrible name [12:21:07] $wgXFFWhitelist = array( '127.0.0.0/8', '10.0.0.0/8' ); would be nicer imho [12:21:34] (03PS4) 10Alex Monk: Partially revert "beta: $wgSquidServers is no more needed" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/71932 [12:21:37] how does it differ from TrustedXFF really? [12:21:44] hm [12:21:56] no idea [12:21:59] TrustedXFF is for third parties too [12:22:00] What IPs do labs instances show as to production servers? [12:22:20] currently labs doesn't do ipv6 yet I realized [12:22:25] but once it does, it'll be in that range [12:22:26] so we can't include our own IPs into TrustedXFF [12:22:46] Krenair: the varnish frontend talk to the varnish backend on the loopback 127.0.0.1 [12:22:52] can we borrow some of its code to parse our internal list? ;) [12:22:54] isn't that range only for knams? [12:22:59] ... I have no idea what you're talking about [12:23:04] Krenair: then it is a 10.x.x.x varnish cache address that is hitting the MediaWiki backends [12:23:09] MaxSem: what range? [12:23:22] the one I attempted to whitelist [12:23:23] If I connected to a production server from a labs instance, say by HTTP or something, what IP will the production server think I have? [12:23:25] no [12:23:30] uh [12:23:32] it's ALL our ipv6 [12:23:39] Krenair: I have no clue :( [12:23:41] and even if, esams has toolserver [12:23:45] although they recently moved out of that range [12:23:48] for this kind of reason ;) [12:23:58] If it's 10.* you can't trust the whole subnet [12:24:18] Krenair: some 208.80.15x ip [12:24:31] oh ok then [12:24:37] or even 10.x possibly [12:24:50] ... not okay then [12:24:54] we certainly shouldn't whitelist 10/8 [12:25:26] Are there any more untrusted servers on the network? other than labs [12:26:04] these days I consider lots of stuff untrusted, as more and more people access machines in various ways [12:28:51] anyway, how unstable these IPs are - can we just whitelist them individually while coding a more permanent solution? [12:30:09] (03CR) 10Hashar: [C: 04-1] "(2 comments)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/71932 (owner: 10Alex Monk) [12:30:19] Krenair: https://gerrit.wikimedia.org/r/#/c/71932/ needs some more love :-] [12:30:32] Krenair: wgSquidServersNoPurge is already defined in squids-labs.php [12:31:28] MaxSem: if I depool these esams servers until we have a more permanent solution it's fixed too [12:31:50] meh [12:31:54] i'll add these two ips for now [12:32:01] (03PS5) 10Alex Monk: Partially revert "beta: $wgSquidServers is no more needed" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/71932 [12:32:16] agha [12:32:48] Krenair: sorry [12:33:04] Krenair: in your commit message you mention a 10.x.x.x showing in http://en.wikipedia.beta.wmflabs.org/wiki/Special:Log/articlefeedbackv5 [12:33:09] sigh [12:33:19] Krenair: which got fixed when I have added the varnish cache text IP in $wgSquidServersNoPurge. [12:33:20] does mediawiki canonicalize those ipv6 ips before it does the in_array check? [12:33:59] Krenair: do you mind rewriting the commit summary ? [12:34:30] and is it case sensitive? [12:34:33] If that got fixed then what is this change for now? [12:34:50] ha it does canonicalize [12:38:03] except not for ipv6 [12:38:03] sigh [12:44:44] having fun are we [12:50:02] PROBLEM - Puppet freshness on manutius is CRITICAL: No successful Puppet run in the last 10 hours [12:51:03] (03PS1) 10Faidon: Switch our own recursors to new NS service IPs [operations/puppet] - 10https://gerrit.wikimedia.org/r/76492 [12:51:44] (03PS2) 10Faidon: Switch our own recursors to new NS service IPs [operations/puppet] - 10https://gerrit.wikimedia.org/r/76492 [12:52:16] (03CR) 10Faidon: [C: 032] Switch our own recursors to new NS service IPs [operations/puppet] - 10https://gerrit.wikimedia.org/r/76492 (owner: 10Faidon) [12:52:22] (03CR) 10Faidon: [V: 032] Switch our own recursors to new NS service IPs [operations/puppet] - 10https://gerrit.wikimedia.org/r/76492 (owner: 10Faidon) [12:54:12] RECOVERY - check_job_queue on hume is OK: JOBQUEUE OK - all job queues below 10,000 [12:56:22] PROBLEM - DPKG on sockpuppet is CRITICAL: DPKG CRITICAL dpkg reports broken packages [12:56:23] (03PS1) 10Petr Onderka: added XML output; additions necessary for XML output [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/76494 [12:57:22] PROBLEM - check_job_queue on hume is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [13:03:02] heya, anybody around know anything about maerlant.esams.wikimedia.org? [13:03:03] mark? [13:03:22] RECOVERY - DPKG on sockpuppet is OK: All packages OK [13:03:58] what about it? [13:11:10] Destination does not have e/eb/.ogv/.ogv.360p.webm, syncing [13:11:13] interesting [13:11:24] j^: around? [13:12:58] oh, well, qchris noticed that it is still serving occasional IPv6 requests [13:13:06] that are not tab separated, so they were messing with his analysis stuff [13:13:16] he asked me what it was, and if we could make its output format be like the rest of the logs [13:13:43] IPv6 isn't puppetized there anymore, it looks like maybe it is meant to be decommissioned or repurposed [13:15:07] ipv6proxy you mean [13:15:25] guess so? [13:15:26] yeah [13:15:49] ipv6/ssl proxy [13:15:49] all it has puppetized is 'include standard' [13:16:05] (03PS1) 10MaxSem: Enable mobile redirection for wikimediafoundation.org [operations/puppet] - 10https://gerrit.wikimedia.org/r/76499 [13:16:05] wait unless my repo is not up to date! [13:16:23] phew, nope it is [13:16:58] yeah no idea cat ./esams/wikimedialbsecure [13:16:58] {'host': 'maerlant.esams.wikimedia.org', 'weight': 30, 'enabled': True } [13:17:06] (03PS1) 10Hashar: mw-update-l10n: only copy when files differs [operations/puppet] - 10https://gerrit.wikimedia.org/r/76500 [13:17:11] wow [13:17:11] looks like ancient history :) [13:18:28] can we remove it? [13:19:33] looks like it [13:21:04] where are you cat-ting that file? [13:21:12] fenari /h/w/conf/pybal [13:21:42] doesn't look like being used anywhere though [13:21:52] that file? [13:21:54] just remnants I think [13:22:01] yeah, so the machine probably needs turned off or something [13:22:11] yes I think so [13:22:17] start with nginx :) [13:22:22] I could just stop nginx [13:22:22] yeah [13:22:43] !log stopping nginx on maerlant.esams.wikimedia.org (this machine should probably be fully decommissioned) [13:22:51] Logged the message, Master [13:23:07] (03CR) 10Anomie: [C: 031] "Looks ok. Haven't tested." [operations/puppet] - 10https://gerrit.wikimedia.org/r/76500 (owner: 10Hashar) [13:23:41] (03PS7) 10Mark Bergsma: Fix XFF handling on all Varnish clusters [operations/puppet] - 10https://gerrit.wikimedia.org/r/75860 [13:23:59] maelant is not ancient history [13:24:11] ah mark! [13:24:11] it's not 2 years old [13:24:20] so it is 'hisotry' [13:24:34] but nginx on it can be removed yes [13:24:34] just not 'ancient'? [13:24:34] yup [13:24:44] (03PS2) 10Petr Onderka: added XML output; additions necessary for XML output [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/76494 [13:24:46] ok cool, i just stopped nginx and ran update-rc.d remove [13:24:50] i wonder how many more XFF patchsets I'm gonna need [13:25:00] well, it's all relative isn't it :) [13:25:03] i'll uninstall nginx too [13:25:07] it's before my time, that's ancient for me! [13:25:23] for mark, ancient is probably 2003 or so :) [13:26:12] I could use a merge of some change to the MediaWiki l10n updater https://gerrit.wikimedia.org/r/#/c/76500/ [13:26:13] tested out on labs manually [13:26:37] hi please :-] [13:26:44] (03PS8) 10Mark Bergsma: Fix XFF handling on all Varnish clusters [operations/puppet] - 10https://gerrit.wikimedia.org/r/75860 [13:26:48] ancient? [13:26:53] yer all newbies [13:26:57] :) [13:27:03] everything we have would be ancient by that definition [13:27:17] what's wikimedialbsecure? [13:27:43] the lvs ip of https wikimedialb? [13:27:47] the pybal group that is, not the service ip [13:27:59] pybal group? [13:28:24] root@fenari:/home/w/conf/pybal/esams# ls wikimedialbsecure https [13:28:24] https wikimedialbsecure [13:28:28] root@fenari:/home/w/conf/pybal/esams# cat wikimedialbsecure [13:28:29] {'host': 'maerlant.esams.wikimedia.org', 'weight': 30, 'enabled': True } [13:28:39] (and of course https has ssl3001-3004) [13:29:03] no idea [13:29:06] probably ancient history [13:29:12] by your definition [13:29:34] I think "< mark> no idea" is a good definition too [13:29:34] :) [13:29:38] have a look at gerrit 75860 if you want [13:30:06] i'm getting old and forgetful and senile after all [13:30:06] looking [13:31:51] omg I'm confused already [13:32:03] my brain hurts [13:32:29] :) [13:32:32] » /* Ensure we only accept Forwarded-Proto headers from the SSL proxies */ 227 [13:32:34] 228 » » // Do nothing. It seems you can't do !~ with IP matches » if (client.ip !~ allow_xff) { [13:32:44] this ACL really needs to be renamed to something else then [13:32:45] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:32:49] ssl_proxies or something [13:32:57] true [13:33:07] or allow_xfp :) [13:33:24] (03PS9) 10Mark Bergsma: Fix XFF handling on all Varnish clusters [operations/puppet] - 10https://gerrit.wikimedia.org/r/75860 [13:33:35] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time [13:33:39] i'll rename it [13:34:48] oh and now we call replace_clientip twice [13:34:59] on the SSL + opera_mini case [13:35:06] (03CR) 10Aude: [C: 031] "not tested, but looks like this will solve the issue" [operations/puppet] - 10https://gerrit.wikimedia.org/r/76500 (owner: 10Hashar) [13:35:29] (03PS2) 10Aude: mw-update-l10n: only copy when files differs [operations/puppet] - 10https://gerrit.wikimedia.org/r/76500 (owner: 10Hashar) [13:36:23] !log removed spammy deprecated cron from gadolinium [13:36:24] (03CR) 10Hashar: [C: 031] mw-update-l10n: only copy when files differs [operations/puppet] - 10https://gerrit.wikimedia.org/r/76500 (owner: 10Hashar) [13:36:34] Logged the message, Master [13:36:47] paravoid: could you merge in https://gerrit.wikimedia.org/r/#/c/76500/ for me please ? :-] [13:37:11] (03CR) 10Aude: [C: 031] "restore +1 (fixed a typo in the commit message)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/76500 (owner: 10Hashar) [13:38:10] (03CR) 10Faidon: [C: 032] mw-update-l10n: only copy when files differs [operations/puppet] - 10https://gerrit.wikimedia.org/r/76500 (owner: 10Hashar) [13:38:21] mark: still no AF_INET6 though? [13:38:27] no [13:38:29] thank you! [13:38:40] kinda difficult to copy that into an ipv4 sockaddr_storage [13:38:53] i want that thing to be replaced by brandon's module [13:39:56] sockaddr_storage is protocol agnostic iirc [13:42:15] yes but is it always the right size? [13:42:36] can I assume I can copy in an ipv6 address in there when it was allocated for AF_INET? [13:42:45] i don't think so ;) [13:42:55] varnish has some fields like sockaddrlen as well [13:42:59] so it quickly becomes very very hairy [13:43:24] blergh [13:43:27] true that [13:49:04] * mark smiles [13:49:09] (03PS10) 10Mark Bergsma: Fix XFF handling on all Varnish clusters [operations/puppet] - 10https://gerrit.wikimedia.org/r/75860 [13:49:16] I got to use the penultimate word in there [13:49:28] that's ancient history [13:50:05] what is? [13:51:29] so I can haz the list of Amsterdam caches IPs? [13:51:30] :) [13:54:24] MaxSem: no, mediawiki doesn't canonicalize those ipv6 ips [13:55:07] because an ipv6 ip can take many forms as a string, simple in_array won't even work reliably for a single ip [13:56:45] that's the point of canonicalizing :P [13:57:13] yes [13:57:23] but unless I misread the code, it doesn't do that for that check [14:00:30] mark, I'll fix this [14:00:37] ok [14:01:15] and anyway, Ip not canonicalised is false negative while what we need to fear here is false positive [14:05:39] (03PS1) 10Hashar: prepare wmf-beta-autoupdate to be launched by Jenkins [operations/puppet] - 10https://gerrit.wikimedia.org/r/76504 [14:09:00] (03PS11) 10Mark Bergsma: Fix XFF handling on all Varnish clusters [operations/puppet] - 10https://gerrit.wikimedia.org/r/75860 [14:09:46] (03PS3) 10Petr Onderka: added XML output; additions necessary for XML output [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/76494 [14:12:56] (03PS12) 10Mark Bergsma: Fix XFF handling on all Varnish clusters [operations/puppet] - 10https://gerrit.wikimedia.org/r/75860 [14:15:28] (03CR) 10Hashar: "(5 comments)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/76504 (owner: 10Hashar) [14:19:28] RECOVERY - Puppetmaster HTTPS on virt0 is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.129 second response time [14:19:30] (03PS1) 10Mark Bergsma: Add cp3011 and cp3012 IPv6 IPs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76505 [14:19:50] maxsem: ^ [14:20:23] mark, thanks! [14:24:34] !log restarted apache on virt0, fixing (at least temporarily) bug 52217 [14:24:41] Logged the message, Master [14:25:08] PROBLEM - Puppet freshness on sq41 is CRITICAL: No successful Puppet run in the last 10 hours [14:37:07] (03PS1) 10Manybubbles: Give labs elasticsearch better heap. [operations/puppet] - 10https://gerrit.wikimedia.org/r/76509 [14:39:41] (03CR) 10Andrew Bogott: [C: 032] "Blindly merging as per Hashar's request :)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/76504 (owner: 10Hashar) [14:40:51] (03PS13) 10Mark Bergsma: Fix XFF handling on all Varnish clusters [operations/puppet] - 10https://gerrit.wikimedia.org/r/75860 [14:43:51] (03PS2) 10MaxSem: Add cp3011 and cp3012 IPv6 IPs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76505 (owner: 10Mark Bergsma) [14:45:39] (03CR) 10MaxSem: [C: 032] "Converted the IPs into the form we're receiving from Varnishes for now - I'll code up proper canonicalisation soon." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76505 (owner: 10Mark Bergsma) [14:45:46] (03Merged) 10jenkins-bot: Add cp3011 and cp3012 IPv6 IPs [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76505 (owner: 10Mark Bergsma) [14:46:50] (03PS14) 10Mark Bergsma: Fix XFF handling on all Varnish clusters [operations/puppet] - 10https://gerrit.wikimedia.org/r/75860 [14:48:59] !log maxsem synchronized wmf-config/squid.php 'Amsterdam varnish IPv6' [14:49:09] Logged the message, Master [14:50:58] (03CR) 10Manybubbles: [C: 031] "Now that we know the names of the machines this looks good to me though someone else should approve it." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/75507 (owner: 10Manybubbles) [14:55:43] (03PS15) 10Mark Bergsma: Fix XFF handling on all Varnish clusters [operations/puppet] - 10https://gerrit.wikimedia.org/r/75860 [15:07:39] (03CR) 10Hashar: [C: 04-1] "(2 comments)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/75507 (owner: 10Manybubbles) [15:09:29] PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [15:10:19] RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s) [15:10:38] (03CR) 10Faidon: [C: 032] Give labs elasticsearch better heap. [operations/puppet] - 10https://gerrit.wikimedia.org/r/76509 (owner: 10Manybubbles) [15:11:15] MaxSem: m.wikimediafoundation.org from my desktop looks very desktop-y [15:11:49] paravoid, some pages are explicitly displaying desktop skin [15:12:04] that's intended:) [15:12:07] or:( [15:12:12] home too? [15:12:30] http://m.wikimediafoundation.org/wiki/Home that is [15:12:49] yep. a hard-fought consensus to prevent crucial pages from being ever unreadable [15:13:02] ok [15:13:06] http://m.wikimediafoundation.org/wiki/Work_with_us is also very weird :) [15:13:07] http://m.wikimediafoundation.org/wiki/Our_projects is mobile [15:13:22] right [15:13:26] the question is [15:13:50] if we don't even mobilize (sic) /Home, is there much point for redirecting? [15:15:30] most pages are mobile [15:15:55] and eventually the rest should be redesigned to be mobile-compatible [15:16:22] damn this is one ugly regexp [15:17:13] hopefully, when we will fully switch to varnish we'll be able to split it to several simpler ones [15:17:19] yeah :) [15:17:26] meanwhile, there are tests [15:17:26] we were discussing this at the hackathon [15:17:33] (03CR) 10Faidon: [C: 032] Enable mobile redirection for wikimediafoundation.org [operations/puppet] - 10https://gerrit.wikimedia.org/r/76499 (owner: 10MaxSem) [15:21:31] (03CR) 10Manybubbles: [C: 04-1] "(2 comments)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/75507 (owner: 10Manybubbles) [15:22:09] (03PS3) 10Manybubbles: Enable CirrusSearch in beta. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/75507 [15:27:14] manybubbles, hey - please let me know when you will definitely choose ES over Solr so that I could start migrating myself, it's killing me:P [15:28:02] i think i'm going to deploy that latest patchset [15:28:05] MaxSem: wanna review? [15:28:27] sure [15:28:30] * MaxSem looks [15:29:21] (03CR) 10Nikerabbit: "Roan: was this forgotten? Should I do it on i18n deployment window tomororw?" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/72356 (owner: 10Amire80) [15:31:22] (03CR) 10Jforrester: "We'll do it now (deploying in 29 minutes' time)." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/72356 (owner: 10Amire80) [15:32:22] (03CR) 10Hashar: [C: 031] "Make sure to warn the Mobile and QA team whenever this is merged. They have Selenium browser test that test the searches user interface :]" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/75507 (owner: 10Manybubbles) [15:33:31] (03CR) 10MaxSem: [C: 031] Fix XFF handling on all Varnish clusters [operations/puppet] - 10https://gerrit.wikimedia.org/r/75860 (owner: 10Mark Bergsma) [15:35:13] * paravoid reviews too [15:35:20] 15 patchsets, poor mark [15:35:30] :) [15:37:12] (03CR) 10Hashar: [C: 04-1] "Need to rewrite the commit summary. Will probably do it myself tonight and then merge this change :-]" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/71932 (owner: 10Alex Monk) [15:37:36] and I am off till tonight *wave* [15:39:02] x-stripped-xff [15:39:08] getting uglier and uglier [15:40:05] (03CR) 10MZMcBride: "It adds considerable code complexity (relatively) and I didn't feel like it." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/69982 (owner: 10Alex Monk) [15:42:41] Is anybody from ops around who could take a look at https://bugzilla.wikimedia.org/show_bug.cgi?id=52148 ? [15:43:33] andre__, it's not an ops issue [15:43:47] ahaha, okay. Greg and I guessed it could be. Thanks. :) [15:44:40] It's a shell issue. [15:45:04] You need someone to check the master database tables, though they may be replicated to Labs. [15:47:36] (03PS4) 10Jforrester: Enable anonymous use of VisualEditor on es/fr/he/it/pl/ru/sv [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76199 [15:47:37] (03PS1) 10Jforrester: Switch VisualEditor back to alpha (opt-in) mode on dewiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76516 [15:49:45] (03CR) 10Demon: [C: 031] Fix in process runjobs in singlenode mediawiki. [operations/puppet] - 10https://gerrit.wikimedia.org/r/76196 (owner: 10Manybubbles) [15:50:41] (03CR) 10MZMcBride: "This change (poorly) duplicates . This change should be abandoned." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76516 (owner: 10Jforrester) [15:51:15] (03PS16) 10Mark Bergsma: Fix XFF handling on all Varnish clusters [operations/puppet] - 10https://gerrit.wikimedia.org/r/75860 [15:54:55] (03CR) 10MZMcBride: "(2 comments)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76199 (owner: 10Jforrester) [15:55:09] (03PS17) 10Mark Bergsma: Fix XFF handling on all Varnish clusters [operations/puppet] - 10https://gerrit.wikimedia.org/r/75860 [15:58:36] (03PS1) 10Andrew Bogott: Pass an array to function_versioncmp. [operations/puppet] - 10https://gerrit.wikimedia.org/r/76517 [16:00:04] (03CR) 10MZMcBride: "What's the status of this change?" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/69420 (owner: 10Dr0ptp4kt) [16:02:49] notpeter, https://gerrit.wikimedia.org/r/#/c/76517/ [16:04:02] (03CR) 10Diederik: "What's the reasoning to disable indexing for Zero?" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/69420 (owner: 10Dr0ptp4kt) [16:04:51] (03CR) 10Parent5446: [C: 031] Enable CAPTCHA for all edits of non-confirmed users on pt.wikipedia in order to reduce editing activity [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/69982 (owner: 10Alex Monk) [16:06:09] (03PS2) 10Catrope: Switch VisualEditor back to alpha (opt-in) mode on dewiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76516 (owner: 10Jforrester) [16:06:10] (03PS5) 10Catrope: Enable anonymous use of VisualEditor on es/fr/he/it/pl/ru/sv [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76199 (owner: 10Jforrester) [16:06:41] (03CR) 10Se4598: "(2 comments)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76199 (owner: 10Jforrester) [16:07:14] (03PS18) 10Mark Bergsma: Fix XFF handling on all Varnish clusters [operations/puppet] - 10https://gerrit.wikimedia.org/r/75860 [16:08:45] (03PS3) 10Jforrester: Switch VisualEditor back to alpha (opt-in) mode on dewiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76516 [16:11:15] (03CR) 10Mark Bergsma: [C: 032] Fix XFF handling on all Varnish clusters [operations/puppet] - 10https://gerrit.wikimedia.org/r/75860 (owner: 10Mark Bergsma) [16:33:21] !log catrope synchronized php-1.22wmf11/extensions/VisualEditor 'Update VE to master' [16:33:31] (03PS4) 10Petr Onderka: added XML output; additions necessary for XML output [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/76494 [16:33:33] Logged the message, Master [16:33:51] !log catrope synchronized php-1.22wmf12/extensions/VisualEditor 'Update VE to master' [16:34:02] Logged the message, Master [16:36:12] (03CR) 10Catrope: [C: 032] Remove unnecessary ULS IME selector from VE headings menu [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/72356 (owner: 10Amire80) [16:37:37] andrewbogott: that's the new syntax? [16:38:16] (03Merged) 10jenkins-bot: Remove unnecessary ULS IME selector from VE headings menu [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/72356 (owner: 10Amire80) [16:38:25] (03CR) 10Catrope: [C: 032] Enable anonymous use of VisualEditor on es/fr/he/it/pl/ru/sv [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76199 (owner: 10Jforrester) [16:38:25] (03Merged) 10jenkins-bot: Enable anonymous use of VisualEditor on es/fr/he/it/pl/ru/sv [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76199 (owner: 10Jforrester) [16:38:25] (03CR) 10Catrope: [C: 032] Switch VisualEditor back to alpha (opt-in) mode on dewiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76516 (owner: 10Jforrester) [16:38:25] (03Merged) 10jenkins-bot: Switch VisualEditor back to alpha (opt-in) mode on dewiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76516 (owner: 10Jforrester) [16:38:25] andrewbogott: if you've tested in labs, looks good to me [16:38:42] (I think it's worth testing, as breaking puppet on the cluster is very annoying to fix. trust me, i know :) ) [16:38:48] netpeter: As best I can tell, erb functions always took/take an array of args. That bug says "The fact that it works in 1.8.7 isn’t intentional (as evident by the warning)" [16:38:59] ah, ok [16:39:01] go for it! [16:39:02] I verified that nslcd.conf is generated the same on labs. [16:39:03] 'k [16:39:31] (03CR) 10Andrew Bogott: [C: 032] Pass an array to function_versioncmp. [operations/puppet] - 10https://gerrit.wikimedia.org/r/76517 (owner: 10Andrew Bogott) [16:40:57] (03PS1) 10Catrope: Revert "Remove unnecessary ULS IME selector from VE headings menu" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76531 [16:41:02] (03CR) 10Catrope: [C: 032 V: 032] Revert "Remove unnecessary ULS IME selector from VE headings menu" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76531 (owner: 10Catrope) [16:41:35] (03Abandoned) 10MZMcBride: Explicitly disable VisualEditor by default on dewiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76468 (owner: 10MZMcBride) [16:43:09] !log catrope synchronized wmf-config/InitialiseSettings.php 'Enable VE for anons on es/fr/he/it/pl/ru/svwiki; set dewiki back to opt-in mode' [16:43:20] Logged the message, Master [16:45:34] (03CR) 10Petr Onderka: [C: 032 V: 032] added XML output; additions necessary for XML output [operations/dumps/incremental] (gsoc) - 10https://gerrit.wikimedia.org/r/76494 (owner: 10Petr Onderka) [17:03:25] PROBLEM - Disk space on mw1159 is CRITICAL: DISK CRITICAL - free space: /tmp 702 MB (3% inode=99%): [17:06:29] PROBLEM - Disk space on mw1159 is CRITICAL: DISK CRITICAL - free space: /tmp 711 MB (3% inode=99%): [17:09:29] PROBLEM - Disk space on mw1159 is CRITICAL: DISK CRITICAL - free space: /tmp 710 MB (3% inode=99%): [17:12:29] PROBLEM - Disk space on mw1159 is CRITICAL: DISK CRITICAL - free space: /tmp 670 MB (3% inode=99%): [17:13:54] -rw-r--r-- 1 apache apache 400M Jul 27 09:16 vips-1-KW280W.v [17:13:55] -rw-r--r-- 1 apache apache 400M Jul 25 19:05 vips-1-NHHX0W.v [17:13:55] -rw-r--r-- 1 apache apache 400M Jul 28 02:54 vips-1-QPY30W.v [17:13:59] root@mw1159:/tmp# du -hs [17:13:59] 17G . [17:14:00] yay [17:14:29] PROBLEM - Disk space on mw1159 is CRITICAL: DISK CRITICAL - free space: /tmp 608 MB (3% inode=99%): [17:15:34] paravoid, https://bugzilla.wikimedia.org/show_bug.cgi?id=52203 [17:16:02] oh, I was about to file one [17:16:03] thanks! [17:16:29] PROBLEM - Disk space on mw1159 is CRITICAL: DISK CRITICAL - free space: /tmp 709 MB (3% inode=99%): [17:16:33] shouldn't that be critical? [17:17:23] as long as it's more than 24h between failures, prolly not:P [17:17:27] or I'm being too lenient.... [17:18:33] RoanKattouw: there are edits with the VE from IPs on dewiki. I think that should be impossible? https://de.wikipedia.org/w/index.php?title=Spezial:Letzte_%C3%84nderungen&tagfilter=visualeditor [17:20:36] Yeah that shouldn't be happening... [17:20:52] * RoanKattouw checks config [17:21:49] (03PS1) 10Ryan Lane: Extend dhcp time to lessen load on network node [operations/puppet] - 10https://gerrit.wikimedia.org/r/76537 [17:22:35] I think DisableForAnons should be explicit. [17:22:39] I said this already. [17:22:59] Though I guess the default is sane. [17:22:59] Elsie: DisableForAnons is ignored if Default=0 [17:22:59] No, it's the opposite. [17:23:03] God I fucking hate these settings. [17:23:06] Yeah they suck [17:23:21] Negative configuration variables were deprecated in like 2005. [17:23:28] Default is ignored if DisableForAnons=1 [17:23:32] Both statements are true, actually :) [17:23:42] The config looks good and I can't reproduce [17:24:00] It's possible that it's logged-in users losing their session and choosing to save anyway, I suppose [17:24:01] Is it possible the anons are getting cached HTML? [17:24:04] And it still works or something? [17:24:35] Or rather: will ?veaction=edit by an anon work? [17:24:39] VE was never enabled by default for anons on dewiki [17:24:44] But ?veaction=edit by an anon might work [17:24:49] I just tried it and it didn't work for me [17:24:56] And it should hit cache because of the query string [17:25:04] shouldn't [17:25:17] Yeah we all know what "should" means in computer science [17:25:27] Heh. [17:25:58] Hmm, anonymous VE edits have suddenly become more common for some reason https://de.wikipedia.org/w/index.php?title=Spezial:Letzte_%C3%84nderungen&hideliu=1&tagfilter=visualeditor [17:26:32] that seems [17:26:37] very implausible doesn't it [17:26:57] Yeah, I mean it must be related to my deploy somehow [17:27:16] But I don't know how, and I can't reproduce [17:27:19] what date was that? [17:27:24] or yo umean today's [17:27:27] Today's. [17:27:29] yes [17:27:35] There's one anon VE edit from July 25. [17:27:39] It could be bad tagging. [17:28:19] I doubt https://de.wikipedia.org/w/index.php?title=Inka_Bause&curid=115631&diff=121014432&oldid=119971005 is bad tagging [17:28:42] :-) [17:28:56] so a leading space -> must add nowii because otherwise it's considered 
 or something
[17:29:06] 	 *nowiki
[17:29:09] 	 Right.
[17:29:11] 	 Yeah :(
[17:29:15] 	 Reedy: heya, when do you get back home from Wikimania?
[17:29:17] 	 but why that would trigger anon edit...
[17:29:27] 	 We need to more intelligently trim things so that doesn't hpappen
[17:29:32] 	 eh yeah
[17:29:46] 	 So how are anons making VE edits...
[17:29:53] 	 64 million dollar question
[17:30:02] 	 That's a lot of money.
[17:30:04] 	 Raymond_: So there was a wave of these for 20 minutes. I have no idea why, but the wave stopped 15 mins ago. Ping me if it starts again
[17:30:11] 	 Raymond_: (See https://de.wikipedia.org/w/index.php?title=Spezial:Letzte_%C3%84nderungen&hideliu=1&tagfilter=visualeditor )
[17:30:20] * RoanKattouw  runs away to a meeting
[17:30:33] 	 RoanKattouw_away: thanks 
[17:30:51] 	 Raymond_: ping https://de.wikipedia.org/w/index.php?title=Thronfolge_%28Vereinigtes_K%C3%B6nigreich%29&curid=855946&diff=121015215&oldid=120984101
[17:30:56] 	 :)
[17:31:22] 	 (03CR) 10Nemo bis: [C: 04-1] "Code complexity doesn't sound like a valid reason not to add a one-line if clause saving us an additional bug, an additional patch and an " [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/69982 (owner: 10Alex Monk)
[17:31:35] 	 robh around?
[17:31:37] 	 rats, already
[17:31:38] 	 strange
[17:31:49] 	 RoanKattouw_away: https://de.wikipedia.org/w/index.php?title=Spezial:Letzte_%C3%84nderungen&limit=500&days=30&hideliu=1&tagfilter=visualeditor
[17:31:53] 	 Two new edits.
[17:31:54] 	 At :30.
[17:32:13] 	 One of them is vandalism.
[17:33:24] 	 eval.php might useful here.
[17:33:27] 	 might be *
[17:33:33] 	 Verbs overrated.
[17:34:26] <^demon>	 eval.php is the most useful maintenance script we have :)
[17:34:27] 	 I GOT ONE!!!
[17:34:41] 	 one page via random article with VE loaded
[17:34:53] 	 Which page?
[17:34:58] 	 http://de.wikipedia.org/wiki/Westdeutscher_Zeitschriftenverlag
[17:35:16] 	 https Tsssssss.
[17:35:28] 	 normally I do :)
[17:35:38] 	 but logged out and caching etc^^
[17:36:10] 	 apergos, RoanKattouw_away: So my guess is cache pollution/cross-pollination.
[17:36:41] 	 ?veaction=edit won't work while logged out, but it will have the VE init JS is stored in the page cache.
[17:36:56] 	 will have --> will if
[17:36:58] 	 I need to eat.
[17:37:13] 	 enjoy yer meal :-)
[17:37:33] 	 :-)
[17:38:08] 	 en has anon ve turned on?
[17:38:27] 	 en wp I mean
[17:38:31] 	 Yes.
[17:38:35] 	 https://noc.wikimedia.org/conf/InitialiseSettings.php.txt
[17:38:41] 	 'wmgVisualEditorDefault' => array(
[17:38:45] 	 that's a big deal
[17:38:56] 	 I guess I don't pretend to edit over there enough to notice when it went live
[17:39:17] 	 I saw the ginormous dicussion threads but they lasted long enough for me to lose track
[17:40:37] 	 I wonder how the VE could be cached or get a problem if the configuration effectivly doesn't changed
[17:40:46] 	 for dewiki
[17:41:28] 	 Logged-in user hits page --> page cached.
[17:41:36] 	 Anonymous user receives cached page.
[17:42:02] 	 It's just a theory, though.
[17:43:59] 	 could be, but also something must changed since it had not been a problem over the week. Can it have something to do with the switch back to opt-in, which uses a other preference-setting?
[17:44:26] 	 PROBLEM - Disk space on mw1159 is CRITICAL: DISK CRITICAL - free space: /tmp 700 MB (3% inode=99%):  
[17:45:28] 	 (03CR) 10Parent5446: "Is it really that hard for the operations team to just remember this?" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/69982 (owner: 10Alex Monk)
[17:47:10] 	 (03CR) 10MZMcBride: "Nemo doesn't like this change and has tried to erect hurdles to its deployment about a half-dozen times now. :-)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/69982 (owner: 10Alex Monk)
[17:47:58] 	 Nemo_bis: You're trying to use -1 as a weapon and it's going to backfire.
[17:49:27] 	 se4598: Grahhh, you have two Bugzilla accounts.
[17:49:35] 	 se4598: You can get those merged. :-)
[17:49:36] 	 PROBLEM - Host colby is DOWN: PING CRITICAL - Packet loss = 100%  
[17:49:39] 	 https://bugzilla.wikimedia.org/show_bug.cgi?id=52232
[17:50:19] 	 Elsie: do i want that? :)
[17:50:32] 	 i guess you do
[17:50:34] 	 I do!
[17:50:49] 	 I didn't know which account to CC.
[17:50:52] 	 (03CR) 10ArielGlenn: "Parnt5446: Not speaking for all members of the ops team, but as someone that has been following this bug closely, I can guarantee that if " [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/69982 (owner: 10Alex Monk)
[17:51:02] 	 se4598: open a bug to have them merged and it can be done
[17:51:36] 	 PROBLEM - Disk space on mw1156 is CRITICAL: DISK CRITICAL - free space: /tmp 460 MB (2% inode=99%):  
[17:51:42] 	 everything on the one gets per watcher to the other, so i don't miss something. but yes, i could merge
[17:51:58] 	 Ori used to have three accounts. It drove me crazy.
[17:52:01] 	 His got merged. :-)
[17:52:30] 	 se4598: It's fine to have more than one account, just add "(do not CC)" or similar to one. :-)
[17:53:34] 	 (03CR) 10Parent5446: [C: 04-1] "Kk, that's legitimate." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/69982 (owner: 10Alex Monk)
[17:53:56] 	 PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours  
[17:53:56] 	 PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours  
[17:53:56] 	 PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours  
[17:53:56] 	 PROBLEM - Puppet freshness on virt1 is CRITICAL: No successful Puppet run in the last 10 hours  
[17:53:56] 	 PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours  
[17:53:57] 	 PROBLEM - Puppet freshness on virt4 is CRITICAL: No successful Puppet run in the last 10 hours  
[17:55:32] 	 what virt3/4? 
[17:55:32] 	 bah
[17:56:03] 	 back to topic: could the changed VE preference (disable/enable) used (opt-out back to opt-in)  have something to with this?
[17:56:26] 	 PROBLEM - Disk space on mw1159 is CRITICAL: DISK CRITICAL - free space: /tmp 637 MB (3% inode=99%):  
[17:57:05] 	 se4598: if things weren't deployed with proper synchronizations, someone might have managed to slipin the short while when both settings were disabled, orboth enabled, or something similar
[17:57:15] 	 se4598: if it's not happening anymore, i wouldn't be too alarmed
[17:57:29] * MatmaRex  has no context
[17:57:51] 	 MatmaRex: it's happening right now, VisualEditor on dewiki for anons
[17:58:01] 	 on all articles?
[17:58:22] 	 no, seems some cache issue, but we dunno why
[17:58:33] 	 if you know which particular pages are affected, just purge them
[17:59:26] 	 PROBLEM - Disk space on mw1159 is CRITICAL: DISK CRITICAL - free space: /tmp 712 MB (3% inode=99%):  
[18:02:48] 	 (03PS1) 10ArielGlenn: add the svg cruft to the imagescaler cron job that cleans up /tmp [operations/puppet] - 10https://gerrit.wikimedia.org/r/76546 
[18:04:26] 	 (03CR) 10ArielGlenn: [C: 032] add the svg cruft to the imagescaler cron job that cleans up /tmp [operations/puppet] - 10https://gerrit.wikimedia.org/r/76546 (owner: 10ArielGlenn)
[18:06:09] 	 (03CR) 10Dr0ptp4kt: "Posted this to the mailing lists, but forgot to update Gerrit: the implementation date has been pushed back while Google's index further u" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/69420 (owner: 10Dr0ptp4kt)
[18:06:37] 	 Elsie: added notice to not CC, never know that bz has a autocomplete feature there till now :-)
[18:11:19] 	 !log restarting db56 slave threads
[18:11:29] 	 Logged the message, Master
[18:11:34] 	 (03PS1) 10ArielGlenn: add vips cruft to imagescaler cron cleanup of /tmp [operations/puppet] - 10https://gerrit.wikimedia.org/r/76549 
[18:12:19] 	 (03CR) 10ArielGlenn: [C: 032] add vips cruft to imagescaler cron cleanup of /tmp [operations/puppet] - 10https://gerrit.wikimedia.org/r/76549 (owner: 10ArielGlenn)
[18:17:37] 	 !log DNS update - added voyagewiki.org and .com
[18:17:47] 	 Logged the message, Master
[18:27:45] 	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[18:30:18] 	 (03PS1) 10ArielGlenn: actually get the option right for imagescaler tmp cleanup cron [operations/puppet] - 10https://gerrit.wikimedia.org/r/76551 
[18:30:36] 	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.136 second response time  
[18:30:55] 	 (03CR) 10ArielGlenn: [C: 032] actually get the option right for imagescaler tmp cleanup cron [operations/puppet] - 10https://gerrit.wikimedia.org/r/76551 (owner: 10ArielGlenn)
[18:35:25] 	 RECOVERY - Disk space on mw1159 is OK: DISK OK  
[18:39:59] 	 !log reedy synchronized php-1.22wmf12/extensions/Wikibase
[18:40:12] 	 Logged the message, Master
[18:40:53] 	 !log server pappas being taken offline to ship to EQIAD
[18:41:00] 	 RECOVERY - Disk space on mw1160 is OK: DISK OK  
[18:41:03] 	 Logged the message, Master
[18:45:50] 	 PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[18:46:30] 	 PROBLEM - Host pappas is DOWN: PING CRITICAL - Packet loss = 100%  
[18:46:50] 	 RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s)  
[18:47:01] 	 (03PS3) 10Ottomata: Puppetizing HA NameNode via Quorum Based JournalNode. [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/76018 
[18:48:17] 	 (03PS4) 10Ottomata: Puppetizing HA NameNode via Quorum Based JournalNode. [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/76018 
[18:49:12] 	 (03PS1) 10Catrope: Explicitly disable VE for anons on dewiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76552 
[18:50:15] 	 (03CR) 10Krinkle: [C: 031] Explicitly disable VE for anons on dewiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76552 (owner: 10Catrope)
[18:50:33] 	 (03CR) 10Catrope: [C: 032] Explicitly disable VE for anons on dewiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76552 (owner: 10Catrope)
[18:50:41] 	 (03Merged) 10jenkins-bot: Explicitly disable VE for anons on dewiki [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76552 (owner: 10Catrope)
[18:53:29] 	 !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: closed, wikimedia and special to 1.22wmf12
[18:53:30] 	 PROBLEM - Host thulium is DOWN: PING CRITICAL - Packet loss = 100%  
[18:53:40] 	 Logged the message, Master
[18:53:55] <^demon>	 !log restarted gitblit service
[18:54:05] 	 Logged the message, Master
[18:56:00] 	 RECOVERY - Disk space on mw1158 is OK: DISK OK  
[18:56:52] 	 !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews, wikivoyage and wikiversity
[18:57:01] 	 Logged the message, Master
[18:58:40] 	 RECOVERY - Host thulium is UP: PING OK - Packet loss = 0%, RTA = 2.75 ms  
[18:58:50] 	 PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[18:59:30] 	 !log set root password to ! on all labs instances.
[18:59:41] 	 Logged the message, Master
[18:59:48] 	 !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiquote and wikisource to 1.22wmf12
[18:59:50] 	 RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s)  
[18:59:59] 	 Logged the message, Master
[19:00:14] 	 !log delaying slave db43 during OSC bug 49199
[19:00:25] 	 Logged the message, Master
[19:00:31] 	 !log thulium kernel update, reboot, tweak CPU power settings
[19:00:40] 	 RECOVERY - Disk space on mw1156 is OK: DISK OK  
[19:00:41] 	 Logged the message, Master
[19:01:27] 	 (03CR) 10Aude: "see also https://gerrit.wikimedia.org/r/#/c/76485/ to add to make release branch" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76481 (owner: 10Aude)
[19:04:41] 	 !log catrope synchronized wmf-config/InitialiseSettings.php  'Actually disable VE for anons on dewiki, hopefully'
[19:04:52] 	 Logged the message, Master
[19:09:29] 	 (03PS2) 10Ryan Lane: Extend dhcp time to lessen load on network node [operations/puppet] - 10https://gerrit.wikimedia.org/r/76537 
[19:09:58] 	 andrewbogott: setting root password to "!" ?
[19:10:31] 	 AzaToth, I have it on good authority (i.e. Ryan_Lane) that ! means 'no password'.
[19:10:37] 	 ah
[19:10:49] 	 yeah, setting the hash to !
[19:10:54] 	 disables the password
[19:11:10] 	 yea, not hashing ! and set password to the result
[19:11:18] 	 I thought you meant that
[19:12:07] 	 (03CR) 10Ryan Lane: [C: 032] Extend dhcp time to lessen load on network node [operations/puppet] - 10https://gerrit.wikimedia.org/r/76537 (owner: 10Ryan Lane)
[19:12:14] * AzaToth  sees andrewbogott scrambling to see if he did that
[19:13:07] 	 !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionary and wikibooks to 1.22wmf12
[19:13:08] 	 Ryan_Lane: what's the current lease time?
[19:13:21] 	 Logged the message, Master
[19:13:30] 	 (03PS1) 10Reedy: Everything non 'pedia to 1.22wmf12 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76555 
[19:13:35] 	 something absurdly short
[19:13:38] 	 like minutes
[19:13:41] 	 ok
[19:13:41] 	 I'm pretty sure I don't actually allow root logins with password:!
[19:13:56] 	 (03CR) 10Reedy: [C: 032] Everything non 'pedia to 1.22wmf12 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76555 (owner: 10Reedy)
[19:14:06] 	 (03Merged) 10jenkins-bot: Everything non 'pedia to 1.22wmf12 [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76555 (owner: 10Reedy)
[19:14:31] 	 hey bblack, need help/discussion/anything from me wrt OSM? :)
[19:14:31] 	 Ryan_Lane: I  assume some dhcp requests each minute will bring your network to a halt
[19:14:47] 	 PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[19:14:56] 	 AzaToth: the network node needs to deal with the requests at more than just the dnsmasq level
[19:15:05] 	 so it keeps the nova-network daemon busy
[19:15:47] 	 RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s)  
[19:16:14] 	 what actually is nova?
[19:17:01] 	 Nova is the name of the software that manages the labs cloud.  It's the Openstack Compute later, specifically.
[19:17:07] 	 *layer
[19:18:16] 	 (03PS1) 10Ryan Lane: Enable tls1.1/1.2 for nginx [operations/puppet] - 10https://gerrit.wikimedia.org/r/76556 
[19:18:19] 	 (03CR) 10Ryan Lane: [C: 032] Enable tls1.1/1.2 for nginx [operations/puppet] - 10https://gerrit.wikimedia.org/r/76556 (owner: 10Ryan Lane)
[19:19:00] 	 AzaToth, so, for example, when you create an instance on wikitech, wikitech passes your request to Nova, and nova picks a server to run the VM on and then creates the VM (using a KVM virtual machine I believe) and then keeps tabs on it.
[19:19:55] 	 I see
[19:20:42] 	 (03PS5) 10Ottomata: Puppetizing HA NameNode via Quorum Based JournalNode. [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/76018 
[19:20:56] 	 hehe, I was looking now at https://wiki.openstack.org/wiki/HypervisorSupportMatrix andrewbogott, and I notice it's a red X for "Set Admin Pass" ツ
[19:21:19] 	 we don't use nova for that ;)
[19:22:28] 	 PROBLEM - HTTPS on ssl1003 is CRITICAL: Connection refused  
[19:22:51] 	 stupid nginx
[19:22:57] 	 PROBLEM - HTTPS on ssl1006 is CRITICAL: Connection refused  
[19:23:03] 	 I hate how a reload will cause it to fail on some nodes
[19:23:27] 	 RECOVERY - HTTPS on ssl1003 is OK: OK - Certificate will expire on 01/20/2016 12:00.  
[19:23:34] 	 Ryan_Lane: rage comment to the developers and go back to apache
[19:23:47] 	 PROBLEM - LVS HTTP IPv6 on wiktionary-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: Connection refused  
[19:23:57] 	 RECOVERY - HTTPS on ssl1006 is OK: OK - Certificate will expire on 01/20/2016 12:00.  
[19:24:47] 	 RECOVERY - LVS HTTP IPv6 on wiktionary-lb.eqiad.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 65569 bytes in 0.008 second response time  
[19:26:17] 	 PROBLEM - HTTPS on ssl3001 is CRITICAL: Connection refused  
[19:29:59] 	 I just lost sv.wikipedia.org
[19:30:08] 	 "Firefox can't establish a connection to the server at sv.wikipedia.org."
[19:30:27] 	 PROBLEM - LVS HTTP IPv6 on wikidata-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection refused  
[19:30:28] 	 PROBLEM - LVS HTTP IPv6 on wikivoyage-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection refused  
[19:30:28] 	 PROBLEM - LVS HTTP IPv6 on wikimedia-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection refused  
[19:30:28] 	 PROBLEM - LVS HTTPS IPv4 on wikivoyage-lb.esams.wikimedia.org is CRITICAL: Connection refused  
[19:30:37] 	 PROBLEM - LVS HTTP IPv6 on wikinews-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection refused  
[19:30:37] 	 PROBLEM - LVS HTTPS IPv4 on wikisource-lb.esams.wikimedia.org is CRITICAL: Connection refused  
[19:30:38] 	 PROBLEM - LVS HTTPS IPv4 on wikiquote-lb.esams.wikimedia.org is CRITICAL: Connection refused  
[19:30:38] 	 PROBLEM - LVS HTTPS IPv4 on wikibooks-lb.esams.wikimedia.org is CRITICAL: Connection refused  
[19:30:38] 	 PROBLEM - LVS HTTPS IPv4 on wikiversity-lb.esams.wikimedia.org is CRITICAL: Connection refused  
[19:30:38] 	 PROBLEM - LVS HTTPS IPv4 on wiktionary-lb.esams.wikimedia.org is CRITICAL: Connection refused  
[19:30:38] 	 PROBLEM - LVS HTTPS IPv4 on wikipedia-lb.esams.wikimedia.org is CRITICAL: Connection refused  
[19:30:39] 	 PROBLEM - LVS HTTPS IPv4 on wikinews-lb.esams.wikimedia.org is CRITICAL: Connection refused  
[19:30:39] 	 PROBLEM - LVS HTTPS IPv4 on bits.esams.wikimedia.org is CRITICAL: Connection refused  
[19:30:47] 	 first I just lost css, and then sv.w.o was a goner
[19:30:47] 	 PROBLEM - LVS HTTPS IPv4 on foundation-lb.esams.wikimedia.org is CRITICAL: Connection refused  
[19:30:47] 	 PROBLEM - LVS HTTPS IPv4 on mediawiki-lb.esams.wikimedia.org is CRITICAL: Connection refused  
[19:30:57] 	 RIP
[19:30:57] 	 PROBLEM - LVS HTTPS IPv4 on wikimedia-lb.esams.wikimedia.org is CRITICAL: Connection refused  
[19:31:01] 	 I'm in sweden btw
[19:31:17] 	 PROBLEM - LVS HTTPS IPv6 on mediawiki-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection refused  
[19:31:17] 	 PROBLEM - LVS HTTPS IPv6 on mobile-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection refused  
[19:31:21] 	 PROBLEM - LVS HTTPS IPv6 on wikisource-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection refused  
[19:31:21] 	 PROBLEM - LVS HTTP IPv6 on upload-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection refused  
[19:31:22] 	 oh?
[19:31:27] 	 PROBLEM - LVS HTTPS IPv6 on wikiquote-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection refused  
[19:31:27] 	 PROBLEM - HTTPS on ssl3003 is CRITICAL: Connection refused  
[19:31:27] 	 PROBLEM - LVS HTTPS IPv4 on mobile-lb.esams.wikimedia.org is CRITICAL: Connection refused  
[19:31:30] 	 PROBLEM - LVS HTTPS IPv6 on wikinews-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection refused  
[19:31:35] 	 possibly above could have been the culpit...
[19:31:40] 	 PROBLEM - LVS HTTPS IPv4 on wikidata-lb.esams.wikimedia.org is CRITICAL: Connection refused  
[19:31:40] 	 PROBLEM - LVS HTTP IPv6 on wikibooks-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection refused  
[19:31:40] 	 PROBLEM - LVS HTTPS IPv4 on upload.esams.wikimedia.org is CRITICAL: Connection refused  
[19:31:40] 	 PROBLEM - LVS HTTPS IPv6 on wikidata-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection refused  
[19:31:49] 	 -_-
[19:31:50] 	 PROBLEM - LVS HTTPS IPv6 on bits-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection refused  
[19:31:54] 	 PROBLEM - LVS HTTPS IPv6 on upload-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection refused  
[19:31:58] 	 PROBLEM - LVS HTTP IPv6 on wiktionary-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection refused  
[19:31:58] 	 PROBLEM - LVS HTTP IPv6 on wikiquote-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection refused  
[19:31:58] 	 PROBLEM - LVS HTTPS IPv6 on wikiversity-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection refused  
[19:31:58] 	 PROBLEM - LVS HTTPS IPv6 on wikibooks-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection refused  
[19:31:58] 	 PROBLEM - LVS HTTPS IPv6 on wikivoyage-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection refused  
[19:31:58] 	 PROBLEM - LVS HTTPS IPv6 on wikipedia-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection refused  
[19:31:59] 	 fucking nginx
[19:31:59] 	 works for me and i'm not proxying
[19:32:03] 	 PROBLEM - LVS HTTPS IPv6 on foundation-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection refused  
[19:32:03] 	 PROBLEM - LVS HTTP IPv6 on wikipedia-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection refused  
[19:32:06] 	 Ryan_Lane: do you have it or do you need help  ? 
[19:32:06] 	 PROBLEM - HTTPS on ssl3002 is CRITICAL: Connection refused  
[19:32:06] 	 PROBLEM - LVS HTTP IPv6 on mediawiki-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection refused  
[19:32:06] 	 PROBLEM - LVS HTTP IPv6 on foundation-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection refused  
[19:32:06] 	 PROBLEM - LVS HTTPS IPv6 on wikimedia-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection refused  
[19:32:06] 	 PROBLEM - LVS HTTP IPv6 on wikisource-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection refused  
[19:32:07] 	 PROBLEM - LVS HTTP IPv6 on wikiversity-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection refused  
[19:32:10] 	 I have it
[19:32:14] 	 cool
[19:32:16] 	 e.g. should be goign through esams
[19:32:16] 	 PROBLEM - LVS HTTPS IPv6 on wiktionary-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection refused  
[19:32:19] 	 wt
[19:32:25] 	 oh, it's only https that's not working
[19:32:29] 	 oh
[19:32:30] 	 weee
[19:32:31] * aude  tries
[19:32:36] 	 RECOVERY - LVS HTTPS IPv4 on wikidata-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 1189 bytes in 0.461 second response time  
[19:32:36] 	 RECOVERY - LVS HTTP IPv6 on wikivoyage-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 43451 bytes in 0.382 second response time  
[19:32:37] 	 RECOVERY - LVS HTTPS IPv4 on upload.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 574 bytes in 0.502 second response time  
[19:32:37] 	 RECOVERY - LVS HTTP IPv6 on wikimedia-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 100396 bytes in 0.566 second response time  
[19:32:37] 	 RECOVERY - LVS HTTPS IPv6 on wikidata-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 1189 bytes in 0.467 second response time  
[19:32:43] 	 so
[19:32:46] 	 RECOVERY - LVS HTTPS IPv6 on upload-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 574 bytes in 0.471 second response time  
[19:32:48] 	 AzaToth: you are right :(
[19:32:49] 	 now it works again
[19:32:50] 	 RECOVERY - LVS HTTPS IPv6 on bits-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 3893 bytes in 0.536 second response time  
[19:32:53] 	 RECOVERY - LVS HTTPS IPv4 on wikibooks-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 45717 bytes in 0.853 second response time  
[19:32:53] 	 RECOVERY - LVS HTTPS IPv4 on wiktionary-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 73023 bytes in 1.656 second response time  
[19:32:53] 	 RECOVERY - LVS HTTP IPv6 on wiktionary-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 66147 bytes in 0.451 second response time  
[19:32:53] 	 RECOVERY - LVS HTTPS IPv4 on wikipedia-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 66147 bytes in 0.797 second response time  
[19:32:53] 	 RECOVERY - LVS HTTP IPv6 on wikiquote-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 66147 bytes in 0.443 second response time  
[19:32:53] 	 :D
[19:32:54] 	 RECOVERY - LVS HTTPS IPv4 on foundation-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 39447 bytes in 0.647 second response time  
[19:32:54] 	 RECOVERY - LVS HTTPS IPv4 on wikiquote-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 58285 bytes in 0.671 second response time  
[19:32:55] 	 RECOVERY - LVS HTTPS IPv4 on wikisource-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 47639 bytes in 0.804 second response time  
[19:32:55] 	 RECOVERY - LVS HTTPS IPv4 on mediawiki-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 1028 bytes in 0.588 second response time  
[19:32:56] 	 RECOVERY - LVS HTTPS IPv6 on wikivoyage-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 43451 bytes in 0.629 second response time  
[19:32:56] 	 RECOVERY - LVS HTTPS IPv6 on wikiversity-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 66147 bytes in 0.738 second response time  
[19:32:57] 	 RECOVERY - LVS HTTPS IPv4 on wikinews-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 74912 bytes in 0.798 second response time  
[19:32:57] 	 RECOVERY - LVS HTTPS IPv6 on wikibooks-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 66147 bytes in 0.834 second response time  
[19:32:58] 	 RECOVERY - LVS HTTP IPv6 on wikinews-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 66147 bytes in 0.469 second response time  
[19:32:58] 	 RECOVERY - LVS HTTPS IPv6 on wikipedia-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 66147 bytes in 1.362 second response time  
[19:33:00] 	 RECOVERY - LVS HTTPS IPv6 on foundation-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 66147 bytes in 1.406 second response time  
[19:33:00] 	 RECOVERY - LVS HTTPS IPv4 on bits.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 3903 bytes in 1.561 second response time  
[19:33:00] 	 RECOVERY - LVS HTTP IPv6 on wikipedia-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 66147 bytes in 0.384 second response time  
[19:33:01] 	 that was quick
[19:33:02] 	 if you push a change to nginx.conf, puppet will do an nginx reload
[19:33:04] 	 RECOVERY - LVS HTTP IPv6 on mediawiki-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 66147 bytes in 0.360 second response time  
[19:33:04] 	 RECOVERY - HTTPS on ssl3002 is OK: OK - Certificate will expire on 01/20/2016 12:00.  
[19:33:04] 	 RECOVERY - LVS HTTP IPv6 on foundation-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 66147 bytes in 0.421 second response time  
[19:33:04] 	 RECOVERY - LVS HTTP IPv6 on wikisource-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 66147 bytes in 0.366 second response time  
[19:33:04] 	 RECOVERY - LVS HTTP IPv6 on wikiversity-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 66147 bytes in 0.488 second response time  
[19:33:05] 	 RECOVERY - LVS HTTPS IPv6 on wikimedia-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 100396 bytes in 0.918 second response time  
[19:33:05] 	 RECOVERY - LVS HTTPS IPv4 on wikiversity-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 53060 bytes in 0.955 second response time  
[19:33:06] 	 RECOVERY - LVS HTTPS IPv4 on wikimedia-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 100396 bytes in 0.823 second response time  
[19:33:15] 	 lovely
[19:33:16] 	 RECOVERY - LVS HTTPS IPv6 on wiktionary-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 66147 bytes in 0.713 second response time  
[19:33:16] 	 RECOVERY - HTTPS on ssl3001 is OK: OK - Certificate will expire on 01/20/2016 12:00.  
[19:33:17] 	 RECOVERY - LVS HTTP IPv6 on upload-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 573 bytes in 0.177 second response time  
[19:33:20] 	 RECOVERY - HTTPS on ssl3003 is OK: OK - Certificate will expire on 01/20/2016 12:00.  
[19:33:20] 	 RECOVERY - LVS HTTPS IPv6 on mobile-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 22079 bytes in 0.535 second response time  
[19:33:22] 	 Ryan_Lane: you were the suspect?
[19:33:23] 	 RECOVERY - LVS HTTPS IPv4 on mobile-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 22064 bytes in 0.556 second response time  
[19:33:25] 	 which has a high probability to make nginx hang
[19:33:27] 	 RECOVERY - LVS HTTPS IPv6 on wikiquote-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 66147 bytes in 0.701 second response time  
[19:33:27] 	 RECOVERY - LVS HTTPS IPv6 on mediawiki-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 66147 bytes in 0.708 second response time  
[19:33:27] 	 RECOVERY - LVS HTTPS IPv6 on wikisource-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 66147 bytes in 0.751 second response time  
[19:33:27] 	 RECOVERY - LVS HTTPS IPv6 on wikinews-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 66147 bytes in 0.734 second response time  
[19:33:36] 	 AzaToth: yeah, the TLS changes I pushed caused this
[19:33:37] 	 RECOVERY - LVS HTTP IPv6 on wikidata-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 1187 bytes in 0.177 second response time  
[19:33:38] 	 RECOVERY - LVS HTTP IPv6 on wikibooks-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 66147 bytes in 0.399 second response time  
[19:33:38] 	 RECOVERY - LVS HTTPS IPv4 on wikivoyage-lb.esams.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 43449 bytes in 0.643 second response time  
[19:33:50] 	 I used salt to fix the situation in eqiad/pmtpa
[19:33:52] * AzaToth  throws Ryan_Lane to the wolves
[19:33:58] 	 as long as it was brief, but still not great
[19:34:13] 	 I only noticed it becuase a link I used was https
[19:34:36] 	 good
[19:35:15] 	 we should change puppet's behavior to do a full restart, rather than doing a reload for nginx
[19:35:28] * Ryan_Lane  looks at that
[19:36:08] * binasher  shoots the wolves
[19:36:20] <^demon>	 binasher: From a helicopter?
[19:36:46] 	 maybe from an a10 warthog 
[19:37:01] <^demon>	 Even better.
[19:37:06] 	 Ryan_Lane: first do a syntax check though :)
[19:37:19] <^demon>	 binasher: I'm sure they'd use those in Alaska if they could.
[19:37:19] 	 ah. that's a good idea
[19:37:24] 	 in this case the syntax was fine
[19:37:29] 	 I had checked it before I pushed it in
[19:40:21] 	 !log delaying slave db45 during OSC bug 49199
[19:40:31] 	 Logged the message, Master
[19:41:57] 	 (03CR) 10Ottomata: "I'm still doing a lot of testing on this, so there will probably be some more patchsets coming. But, this is mostly complete and ready fo" [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/76018 (owner: 10Ottomata)
[19:44:00] 	 qchris: i don't understand http://gp.wmflabs.org/graphs/free_mobile_page_requests_as_percent_of_country as i don't see any country information
[19:46:39] 	 PROBLEM - HTTPS on amssq47 is CRITICAL: Connection refused  
[19:47:59] 	 !log reedy synchronized php-1.22wmf12/extensions/DataTypes
[19:48:09] 	 Logged the message, Master
[19:54:29] 	 (03CR) 10Andrew Bogott: [C: 032 V: 032] "Private is now reorganized on stafford, we need this corresponding change." [operations/puppet] - 10https://gerrit.wikimedia.org/r/76129 (owner: 10Andrew Bogott)
[19:54:35] 	 (03PS2) 10Andrew Bogott: Support modularized private repo. [operations/puppet] - 10https://gerrit.wikimedia.org/r/76129 
[19:55:57] 	 (03CR) 10Andrew Bogott: [C: 032] Support modularized private repo. [operations/puppet] - 10https://gerrit.wikimedia.org/r/76129 (owner: 10Andrew Bogott)
[20:02:03] 	 !log reedy synchronized php-1.22wmf12/extensions/CirrusSearch
[20:02:16] 	 Logged the message, Master
[20:04:29] 	 (03CR) 10Demon: [C: 032] Enable CirrusSearch in beta. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/75507 (owner: 10Manybubbles)
[20:05:09] 	 (03Merged) 10jenkins-bot: Enable CirrusSearch in beta. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/75507 (owner: 10Manybubbles)
[20:07:24] 	 !log demon synchronized wmf-config/CirrusSearch-common.php  'Syncing Cirrus config'
[20:07:35] 	 Logged the message, Master
[20:07:57] 	 ^demon: I'll see if I can run those scripts now
[20:08:06] 	 !log demon synchronized wmf-config/CirrusSearch-labs.php  'Syncing Cirrus config'
[20:08:17] 	 Logged the message, Master
[20:08:28] 	 or maybe you're still doing things
[20:08:46] <^demon>	 I'm just sync'ing the config on production so nobody comes along wondering what our changes are for :)
[20:08:48] 	 !log demon synchronized wmf-config/CommonSettings.php  'Syncing Cirrus config'
[20:08:50] 	 (03PS1) 10Reedy: Add new CirrusSearch config files [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76618 
[20:08:51] <^demon>	 Should be no-op in prod though.
[20:08:59] 	 Logged the message, Master
[20:09:58] 	 (03PS2) 10Reedy: Update wgCodeReviewRepoStatsCacheTime to 1 day [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76348 
[20:10:04] 	 (03CR) 10Reedy: [C: 032] Update wgCodeReviewRepoStatsCacheTime to 1 day [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76348 (owner: 10Reedy)
[20:10:09] <^demon>	 greg-g: Ok, the part of our beta deploy that touched prod is done.
[20:10:17] 	 (03Merged) 10jenkins-bot: Update wgCodeReviewRepoStatsCacheTime to 1 day [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76348 (owner: 10Reedy)
[20:10:26] <^demon>	 As planned, had zero actual impact on prod.
[20:10:48] 	 (03PS2) 10Reedy: Add new CirrusSearch config files [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76618 
[20:10:54] 	 (03CR) 10Reedy: [C: 032] Add new CirrusSearch config files [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76618 (owner: 10Reedy)
[20:11:06] 	 (03Merged) 10jenkins-bot: Add new CirrusSearch config files [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76618 (owner: 10Reedy)
[20:11:11] 	 ^demon: whew!
[20:11:39] <^demon>	 manybubbles: Ok, we should be set. Everything's merged and I'm done on prod.
[20:11:46] <^demon>	 Should be able to bang away at beta.
[20:12:01] 	 ^demon: did you run the maintenance scripts?
[20:12:06] 	 !log reedy synchronized wmf-config/
[20:12:12] <^demon>	 manybubbles: No, I didn't yet.
[20:12:17] 	 Logged the message, Master
[20:13:04] 	 so I can run them like this I think:sudo -u mwdeploy mwscript extensions/CirrusSearch/maintenance/updateSearchConfig.php --wiki enwiki
[20:13:23] <^demon>	 Sounds about right.
[20:13:25] 	 Might need to be -u apache
[20:13:35] 	 (is on the cluster)
[20:13:40] 	 well, almost
[20:13:41] <^demon>	 Hmm, labs can't find PoolCounter_Client.
[20:13:48] <^demon>	 Do we not have it enabled on labs?
[20:13:52] <^demon>	 hashar: ^
[20:14:29] 	 The config would suggest so
[20:14:44] <^demon>	 Config for Cirrus or general config for beta?
[20:15:15] 	 if ( $wmgUsePoolCounter ) {
[20:15:15] 	 	include( getRealmSpecificFilename( "$wmfConfigDir/PoolCounterSettings.php" ) );
[20:15:15] 	 }
[20:15:44] 	 Hmm, I wonder what that actually evaluates to on labs..
[20:15:50] 	 ^demon: I'm pretty sure I didn't check if it was enabled before writing the config.  just assumed...
[20:15:52] 	 As there's only -eqiad and -pmtpa configs
[20:16:06] <^demon>	 Yeah I saw that bit.
[20:16:35] 	 !log reedy synchronized php-1.22wmf11/extensions/DataTypes
[20:16:41] 	 http://en.wikipedia.beta.wmflabs.org/wiki/Special:Version says no
[20:16:46] 	 Logged the message, Master
[20:17:33] 	 > > echo getRealmSpecificFilename( "$wmfConfigDir/PoolCounterSettings.php" );
[20:17:34] 	 /data/project/apache/common-local/php-master/../wmf-config/PoolCounterSettings-pmtpa.php
[20:18:29] 	 I'll disable it in CirrusSearch then
[20:18:57] <^demon>	 Easy enough to do for now. I'll also file a bug, would be nice to have Poolcounter in beta.
[20:19:15] 	 Indeed
[20:19:25] 	 And IIRC it's puppetised, so should be simple enough
[20:20:42] <^demon>	 https://bugzilla.wikimedia.org/show_bug.cgi?id=36891 - hmm, Antoine + Tim said not needed.
[20:20:55] 	 (03PS1) 10Manybubbles: Stop cirrussearch from using poolcounter in beta. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76620 
[20:20:57] <^demon>	 Would be nice then to have a PoolCounterSettings-labs that just disables it then.
[20:21:03] 	 https://noc.wikimedia.org/conf/highlight.php?file=InitialiseSettings-labs.php
[20:21:10] 	 It's actually overriden to be disabled
[20:21:19] 	 $wgUsePoolCounterForCirrusSearch
[20:21:47] 	 Ah https://bugzilla.wikimedia.org/show_bug.cgi?id=36891
[20:22:00] 	 (03PS1) 10Andrew Bogott: Slight rearranging [operations/puppet] - 10https://gerrit.wikimedia.org/r/76621 
[20:22:07] 	 Who wants to go pretend Jackson died (again) on labs?
[20:22:17] 	 Reedy_: I thought about making one of those but everything about the pool counter config is so, well, configurable, that I'd end up making 7 globals to cover all the knobs people could twist
[20:22:36] 	 andrewbogott: wanna review/merge https://gerrit.wikimedia.org/r/#/c/76059/ ?
[20:22:37] 	 Reedy_: The mere idea of Jackson dying makes me scream in horror.
[20:22:46] 	 it fulfills your TODO, and i tested it.
[20:23:24] 	 um… yes, I'm in the midst of breaking things so need to get puppet working again and then I can review :)
[20:23:30] 	 isset( $wgPoolCounterConf['Cirrus'] )
[20:23:38] 	 meh
[20:23:47] 	 (03CR) 10Andrew Bogott: [C: 032] Slight rearranging [operations/puppet] - 10https://gerrit.wikimedia.org/r/76621 (owner: 10Andrew Bogott)
[20:24:17] <^demon>	 Reedy_: I'm a little more curious why disabling PoolCounter doesn't work for Cirrus. Works for the other settings :\
[20:24:19] 	 Reedy_: just a minor, unrelated thing but https://www.wikidata.org/wiki/Special:ListDatatypes?uselang=en has old cached stuff
[20:24:27] 	 https://www.wikidata.org/wiki/Special:ListDatatypes?uselang=de for example uses the correct message
[20:24:35] 	 and test.wikidata is fine
[20:24:53] 	 can we purge the special page somehow? or do you know why it's like that?
[20:25:17] 	 ^demon: it is because most stuff checks to see if pool counter is enabled before configuring it - I didn't check in the labs setting file.
[20:25:57] 	 aude: How/where is it cachec?
[20:26:01] 	 ^demon: if you configure pool counter beyond the default you pretty much always disable to the stub implementation
[20:26:01] 	 no idea
[20:26:12] 	 just the english shows geo-coordinate
[20:26:17] <^demon>	 manybubbles: Ah, then maybe we should do that then? Rather than just nuking the whole config block?
[20:26:27] <^demon>	 That way it'll work going forward as well.
[20:26:28] 	 http://www.wikidata.org/wiki/Special:ListDatatypes?uselang=qqx
[20:26:32] 	 looks fine
[20:26:34] 	 ^demon: picky picky
[20:26:40] 	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[20:26:55] <^demon>	 manybubbles: We're gonna have to do it later before we go to prod anyway ;-)
[20:26:56] 	 probably nobody looks at that special page (or it would've been reported a while ago)
[20:27:38] 	 i can't see wrong settings anywhere (or if they were wrong, wikidata would be very broken)
[20:27:39] 	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 3.378 second response time  
[20:28:02] 	 AllMessages doesn't seem to know about Wikibase-listdatatypes-geo-coordinate-head
[20:28:08] 	 (03PS2) 10Manybubbles: CirrusSearch will only use poolcounter if enabled. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76620 
[20:28:22] 	 it shouldn't
[20:28:29] 	 it should know about globe-coordinate
[20:28:56] 	 oh, nevermknd
[20:29:02] * aude  finds an admin to fix it :)
[20:29:07] 	 (03CR) 10Demon: [C: 032] CirrusSearch will only use poolcounter if enabled. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76620 (owner: 10Manybubbles)
[20:29:16] 	 (03Merged) 10jenkins-bot: CirrusSearch will only use poolcounter if enabled. [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76620 (owner: 10Manybubbles)
[20:29:16] 	 https://www.wikidata.org/wiki/MediaWiki:Wikibase-listdatatypes-globe-coordinate-head
[20:31:07] 	 ^demon: I'm building the search index.  slower than I'd like
[20:32:05] 	 ^demon: also, I'm only doing enwiki now.  I suppose I should do the others....
[20:32:10] <^demon>	 How slow is slow?
[20:32:56] 	 virtualised slow
[20:33:22] <^demon>	 So does that mean it's virtually slow or actually slow? ;-)
[20:33:25] 	 !log demon synchronized wmf-config/CirrusSearch-common.php  'More Cirrus goodies'
[20:33:36] 	 Logged the message, Master
[20:34:35] <^demon>	 Yay, labs no longer throwing fatals on searching.
[20:34:35] 	 ^demon:11/sec
[20:34:39] <^demon>	 Ouch.
[20:34:41] <^demon>	 That's.
[20:34:42] <^demon>	 Ouch.
[20:34:44] 	 painful!
[20:34:54] 	 I'm going to have to kill it and restart it in a screen session....
[20:35:34] <^demon>	 Where's the lag? DB? -es hosts?
[20:38:24] 	 There certainly isn't any load on the machine running the script.  We're probably seeing db lag, yeah.  Let me checkthat
[20:39:46] <^demon>	 Ouch. Is all of beta really running from just one host?
[20:39:51] <^demon>	 *beta dbs
[20:41:13] 	 so no lag on the db
[20:41:54] 	 and I'm pretty sure the command I ran (wmscript) runs the command locally
[20:42:33] 	 When the cache was full it moved _faster_ but something really isn't fast
[20:42:52] 	 ^demon: did you found out PoolCounter on labs ? 
[20:43:00] 	 ^demon: I am not even sure it is puppetized
[20:43:21] 	 ^demon: --profile-ing the script now
[20:44:07] <^demon>	 hashar: Yeah, we worked around it for now. I saw bug 36891 where you and Tim said you weren't going to bother having it in beta.
[20:44:30] <^demon>	 (I kinda disagree, but we can discuss that)
[20:44:46] <^demon>	 manybubbles: Sounds good. Need me on anything right now?
[20:46:05] 	 ^demon: I think it might be cool if we only ran cirrussearch on beta's enwiki-  mostly because I'm not quite ready to run it with other languages right yet
[20:46:17] 	 so it might be cool to change the config to that.
[20:46:28] 	 I didn't think of it until just now
[20:47:07] 	 (03CR) 10Aude: "all the requirements are there now" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76481 (owner: 10Aude)
[20:47:16] <^demon>	 manybubbles: I'll take care of it :)
[20:47:27] 	 my hero
[20:47:49] 	 ^demon: ah you found the pool counter wont fix.  Guess you could reopen it if you now have a use case
[20:48:25] <^demon>	 Well it's no more of a use-case than the other poolcounter usages. But if we plan to make beta a production clone it makes the most sense (at least to me) to set it up.
[20:48:43] 	 ^demon: I guess the discussion with Tim was how we did not care about a Michael Jackson effect on beta which would "never" happen.  But nowadays PoolCounter seems to be used for other backend stuff to, so it might be worth reopening the bug.
[20:49:56] 	 ^demon: fully agree.  Could have fixed it as LATER :]
[20:53:17] 	 (03PS1) 10Demon: CirrusSearch only for enwiki at the moment [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76623 
[20:54:29] <^demon>	 hashar: Mind having a gander at ^?
[20:57:48] 	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[20:58:38] 	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time  
[20:59:13] 	 ^demon: conf callll :D
[21:02:58] 	 PROBLEM - Puppet freshness on holmium is CRITICAL: No successful Puppet run in the last 10 hours  
[21:03:40] 	 (03CR) 10Hashar: "(1 comment)" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76623 (owner: 10Demon)
[21:04:05] 	 ^demon: I guess the beta wiki have broken search right now since they are all pointing to unconfigured Cirrus Search.
[21:04:15] 	 ^demon: probably good enough for now though so I voted CR+0 :)
[21:05:45] 	 (03CR) 10Demon: "They wouldn't? I figured by moving the break into the if block they'd fall back to default: and thus lucene." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76623 (owner: 10Demon)
[21:08:55] 	 ^demon:   91.53% 239.067228    294 - Parser::internalParse
[21:09:22] 	 (03CR) 10Hashar: [C: 031] "I havent spotted that sneaky break; :-] You are such a hacker! Good to me feel free to merge/deploy whenever you want." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76623 (owner: 10Demon)
[21:09:30] <^demon>	 manybubbles: Blarggghhhh. How much you wanna bet we're not hitting the pcache?
[21:09:34] <^demon>	 Can you pastebin the full profile?
[21:09:51] 	 ^demon: yeah!  we're hitting the cache but it is empty in beta
[21:09:59] 	 I am not even sure where the parser cache is on beta
[21:10:04] 	 might not even be enabled: /
[21:11:11] 	 ^demon: https://gist.github.com/nik9000/7f3b0661bbede303281e
[21:11:15] <^demon>	 hashar: And yeah, I don't like the hack either but this is a short-term thing.
[21:11:29] 	 ^demon: yup I understood that, that is good enough for now
[21:11:47] 	 (03CR) 10Demon: [C: 032] CirrusSearch only for enwiki at the moment [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76623 (owner: 10Demon)
[21:11:54] 	 !log upgrading packages on hooper (etherpad/racktables)
[21:12:05] 	 Logged the message, Master
[21:12:27] 	 ^demon: I spent so much time thinking about being nice to the db....  it is 3%
[21:12:37] 	 (03Merged) 10jenkins-bot: CirrusSearch only for enwiki at the moment [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76623 (owner: 10Demon)
[21:12:47] <^demon>	 manybubbles: Hey, I'll take it :)
[21:14:46] 	 !log demon synchronized wmf-config/CommonSettings.php  'Hack for Cirrus in labs'
[21:14:57] 	 Logged the message, Master
[21:16:43] 	 ^demon: looks to me like we're actually hitting the cpu pretty hard when we get that slow rate
[21:17:06] <^demon>	 Yeah, parsing is cpu intensive. That's where our bottleneck was before and why I did the pcache support.
[21:17:24] 	 So pcache - is that going to cache templates?
[21:17:27] 	 !log removing unused mysql-server from hooper, the only service uses db9
[21:17:38] 	 Logged the message, Master
[21:18:30] 	 we're not super consistent but we spike the cpu utilization about 3/4 of the time now.
[21:18:38] 	 ^demon: apparently the parser cache on beta is memcached backed and only has two 100MB (iirc) instances
[21:18:45] <^demon>	 manybubbles: It caches ParserOutput of things that are parsed.
[21:19:08] <^demon>	 hashar: 100MB ain't much
[21:19:11] 	 hashar:  I'm blowing through it
[21:19:25] 	 filling it up over and over and over again
[21:20:10] 	 well I don't even know how to find out the memcached statistics :/
[21:20:18] 	 potentially you could get more instances installed
[21:21:24] 	 ^demon: not good - https://gist.github.com/nik9000/9c49e60235117ef26531
[21:21:44] 	 afaik, we have 16 machines with 96G memory (at each DC?  at least thats how i read the configs).  The machines run memcache and redis
[21:21:53] <^demon>	 ebernhardson: We're talking labs.
[21:22:00] <^demon>	 Beta, more specifically.
[21:22:04] 	 ^demon: oh, well ignore me then :)
[21:22:47] <^demon>	 manybubbles: The backtrace? Or the 1/s?
[21:22:51] <^demon>	 Or both :p
[21:27:45] 	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[21:28:35] 	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time  
[21:31:45] 	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[21:32:35] 	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time  
[21:41:52] 	 se4598_2: Thanks!
[21:41:56] 	 se4598: You too!
[21:44:57] 	 Elsie: i'm out of context. Was that about the bz-name CC?
[21:50:03] 	 ^demon: feel free to add more memcached instances in beta. I am not sure how I got them setup though
[21:50:47] 	 role::memcached  applied to  deployment-apache{32,33}
[21:51:16] 	 se4598: Yes. :-)
[21:53:25] 	 ^demon: the backtrace - that stops us cold
[21:53:54] 	 1/s is bad but I'm used to it at this point.....  
[21:57:46] 	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds  
[21:58:35] 	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time  
[21:58:42] 	 (03PS6) 10Ottomata: Puppetizing HA NameNode via Quorum Based JournalNode. [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/76018 
[21:59:34] 	 (03PS7) 10Ottomata: Puppetizing HA NameNode via Quorum Based JournalNode. [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/76018 
[22:00:30] 	 (03PS8) 10Ottomata: Puppetizing HA NameNode via Quorum Based JournalNode. [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/76018 
[22:03:40] 	 (03PS9) 10Ottomata: Puppetizing HA NameNode via Quorum Based JournalNode. [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/76018 
[22:10:25] 	 (03CR) 10MZMcBride: "For the record, this changeset was eventually implemented a few hours later in Gerrit changeset 76516. Bug 52232 comment 6 provides a few " [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76468 (owner: 10MZMcBride)
[22:11:50] 	 gj
[22:11:52] 	 PROBLEM - Puppet freshness on erzurumi is CRITICAL: No successful Puppet run in the last 10 hours  
[22:14:28] 	 (03CR) 10MZMcBride: "For the record, this changeset caused bug 52232. It should not have been merged and deployed. As previously stated on this changeset, 	 RECOVERY - check_job_queue on fenari is OK: JOBQUEUE OK - all job queues below 10,000  
[22:17:39] 	 heh, that's been a while
[22:17:46] 	 yay zh wiki
[22:18:10] 	 !log maxsem synchronized php-1.22wmf12/extensions/MobileFrontend/javascripts/modules/editor/EditorApi.js
[22:18:22] 	 Logged the message, Master
[22:19:26] 	 woah, tiny jobqueue --- i almost now worry that something is broken
[22:20:02] 	 haha, yea, like the check itself, how does it even work:)
[22:20:02] 	 "what is this?! a job queue for ANTS?! it needs to be ... at least three times that big!"
[22:20:02] 	 it relied on nfs didnt it
[22:20:08] 	 !log maxsem synchronized php-1.22wmf11/extensions/MobileFrontend/javascripts/modules/editor/EditorApi.js
[22:20:18] 	 ori-l++
[22:20:20] 	 Logged the message, Master
[22:20:32] 	 PROBLEM - check_job_queue on fenari is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[22:20:42] 	 thats more like it:)
[22:22:00] 	 hah
[22:26:58] 	 (03CR) 10MZMcBride: "This changeset was necessary due to the patch sets on  and  be" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76552 (owner: 10Catrope)
[22:29:47] 	 (03CR) 10Lcarr: "(1 comment)" [operations/puppet/cdh4] - 10https://gerrit.wikimedia.org/r/76018 (owner: 10Ottomata)
[22:29:53] 	 !log creating operations/debs/etherpad-lite gerrit project
[22:30:05] 	 Logged the message, Master
[22:34:00] 	 !log moved private puppet manifests into modules
[22:34:09] 	 Logged the message, Master
[22:43:14] <^demon>	 hashar: Sooooo, turns out Timeline extension totally doesn't work on beta.
[22:43:21] <^demon>	 eg: http://en.wikipedia.beta.wmflabs.org/w/index.php?oldid=52225
[22:44:23] 	 timeline of Timeline extension not working on beta:
[22:44:28] 	 <---------------------------- not working -------------------------->
[22:44:55] 	 That's actually more attractive the Timeline extension's output.
[22:44:58] 	 timeline of one of my outstanding patches to timeline:
[22:44:58] 	 than the
[22:45:01] 	 I can't type today.
[22:45:11] * ^demon  thwacks ori-l with a stick
[22:45:15] 	 :P
[22:45:16] 	 submitted| ------------------------ |
[22:45:20] 	 ^ half a year ago ..... now
[22:45:44] <^demon>	 Nobody here is being helpful.
[22:45:46] * ^demon  pouts
[22:45:55] 	 Poutier.
[22:46:00] 	 you're in the wrong channel
[22:46:01] * ori-l  ducks
[22:46:06] 	 also, if we're talking about timeline, bug 4.
[22:46:07] 	 !bug 4
[22:46:08] 	 i kid, i kid
[22:46:08] 	 https://bugzilla.wikimedia.org/4
[22:46:26] <^demon>	 MatmaRex: I don't care about old bugs like that.
[22:46:28] 	 hey, it'll have it's nine-year anniersary soon.
[22:46:31] <^demon>	 I just want the damn thing to work :D
[22:46:34] 	 its*
[22:48:02] <^demon>	 OK SO SINCE NOBODY HAS ANY BETTER IDEAS IM JUST GONNA TURN TIMELINE OFF ON BETA KTHNX.
[22:48:08] <^demon>	 SILLY EXTENSION ANYWAY
[22:48:47] 	 ^demon: git push -f origin :master
[22:49:03] 	 (03PS1) 10Dzahn: add initial .gitreview file [operations/debs/etherpad-lite] - 10https://gerrit.wikimedia.org/r/76645 
[22:50:50] 	 PROBLEM - Puppet freshness on manutius is CRITICAL: No successful Puppet run in the last 10 hours  
[22:51:10] 	 (03CR) 10Dzahn: [C: 032] add initial .gitreview file [operations/debs/etherpad-lite] - 10https://gerrit.wikimedia.org/r/76645 (owner: 10Dzahn)
[22:51:16] 	 (03CR) 10Dzahn: [V: 032] add initial .gitreview file [operations/debs/etherpad-lite] - 10https://gerrit.wikimedia.org/r/76645 (owner: 10Dzahn)
[22:52:19] 	 (03PS16) 10Tim Starling: Enable CAPTCHA for all edits of non-confirmed users on pt.wikipedia in order to reduce editing activity [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/69982 (owner: 10Alex Monk)
[22:52:30] 	 (03CR) 10Tim Starling: [C: 032 V: 032] "I'll set a calendar reminder." [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/69982 (owner: 10Alex Monk)
[22:53:43] 	 ^demon: poor timeline
[22:53:59] 	 ^demon: No backend defined with the name `local-multiwrite`.
[22:54:16] <^demon>	 Yeahhhh
[22:54:23] 	 ^demon: seems to be a misconfiguration of $wgFileBackend in beta context, should be fixable by tweaking in mw-config 
[22:54:39] <^demon>	 Yeah, just not sure what we should set it to.
[22:54:41] 	 though that requires to find out proper parameters to pass to that evil $wg setting :-]
[22:55:00] 	 I guess it is used to store the timelines rendering ?
[22:55:19] 	 that might also lead to other issues, I am afraid local-multiwrite is used by other pieces
[22:55:30] 	 of code
[22:56:44] <^demon>	 CommonSettings has this bit:
[22:56:45] <^demon>	 $wgTimelineSettings->fileBackend = 'local-multiwrite';
[22:56:55] <^demon>	 So we can change it just for timeline. I just dunno what we want.
[22:57:24] 	 do we have a way to nicely dump out the filebackend conf  ?
[22:57:31] 	 like using xml / ini file format or whatever ?
[22:58:07] <^demon>	 eval.php on deployment-bastion?
[22:58:28] 	 the poor man swiss army knife :(
[22:58:56] 	 $ mwscript eval.php --wiki=enwiki
[22:58:56] 	 > return $wgFileBackends;
[22:58:57] 	 array(0) {}
[22:58:58] 	 pfff
[22:59:05] 	 (in beta)
[22:59:28] 	 ah we have $wgLocalFileRepo
[23:02:20] 	 ^demon: don't you like how the local-filebackend is actually a swift backend ? :D
[23:02:25] 	 !log tstarling synchronized wmf-config/InitialiseSettings.php  'enable captcha on ptwiki'
[23:02:36] 	 Logged the message, Master
[23:05:36] <^demon>	 hashar: It looks like we can just swap 'local-multiwrite' for 'local'
[23:05:37] <^demon>	 Lemme prepare a patch
[23:05:51] 	 ^demon: seems that if we unset  $wgTimelineSettings->fileBackend  timeline has code to fallback to a new FSFileBacken
[23:05:52] 	 d
[23:06:05] 	 that points to {$wgUploadDirectory}/timeline
[23:08:45] 	 (03PS1) 10Demon: Fix timeline settings for beta [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76651 
[23:10:39] 	 (03PS1) 10Hashar: beta: points timeline backend to local directory [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76652 
[23:10:44] 	 lol
[23:11:08] * hashar  looks for a patchsets fighting simulation
[23:11:34] 	 (03PS1) 10Mwalker: Enable wgNoticeUseLanguageConversion [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76653 
[23:11:51] 	 (03CR) 10Demon: [C: 032] Fix timeline settings for beta [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76651 (owner: 10Demon)
[23:12:25] 	 (03CR) 10Mwalker: [C: 032] Enable wgNoticeUseLanguageConversion [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76653 (owner: 10Mwalker)
[23:12:54] 	 (03CR) 10Hashar: "Antoine long explanation at https://gerrit.wikimedia.org/r/76652 :-D" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76651 (owner: 10Demon)
[23:13:17] <^demon>	 hashar: Haha, missed your change :)
[23:13:17] 	 (03Merged) 10jenkins-bot: Fix timeline settings for beta [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76651 (owner: 10Demon)
[23:13:33] <^demon>	 beta-code-update already in progress.
[23:13:39] 	 (03Abandoned) 10Hashar: beta: points timeline backend to local directory [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76652 (owner: 10Hashar)
[23:13:56] 	 ^demon: that job update mediawiki core + extensions
[23:14:03] 	 ^demon: mediawiki-config is updated by another job
[23:14:07] <^demon>	 Ah
[23:14:24] 	 https://integration.wikimedia.org/ci/job/beta-mediawiki-config-update/ :-)
[23:14:42] <^demon>	 Ah, just need to be patient?
[23:14:43] <^demon>	 :)
[23:15:08] 	 https://integration.wikimedia.org/zuul/
[23:15:13] 	 yup
[23:15:32] 	 PROBLEM - RAID on searchidx1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.  
[23:15:59] 	 ^demon: zuul reports back that the change got deployed on beta :-]
[23:16:14] 	 No backend defined with the name `local`.
[23:16:17] 	 that is creative
[23:16:32] 	 RECOVERY - RAID on searchidx1001 is OK: OK: State is Optimal, checked 4 logical device(s)  
[23:16:42] <^demon>	 Dangit.
[23:16:47] <^demon>	 Maybe we should've done it your way.
[23:16:57] 	 !log mwalker synchronized php-1.22wmf11/extensions/CentralNotice  'Updating CentralNotice to master on wmf11'
[23:16:59] 	 (03PS1) 10Dzahn: the existing etherpad-lite_1.0-wm2 package in a operations/debs repo for completeness [operations/debs/etherpad-lite] - 10https://gerrit.wikimedia.org/r/76654 
[23:17:07] 	 Logged the message, Master
[23:17:16] 	 ^demon: I can never remember in which order the conf files are loaded
[23:17:52] 	 ahh +$wgTimelineSettings->fileBackend = 'local';
[23:17:55] 	 yeah that does not work
[23:17:59] 	 let me restore my change :-D
[23:18:08] 	 (03PS1) 10Demon: Revert "Fix timeline settings for beta" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76656 
[23:18:16] 	 (03Restored) 10Hashar: beta: points timeline backend to local directory [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76652 (owner: 10Hashar)
[23:18:49] 	 (03CR) 10Hashar: [C: 032] "'local' is not a defined filebackend indeed :-D" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76656 (owner: 10Demon)
[23:18:57] 	 (03Merged) 10jenkins-bot: Revert "Fix timeline settings for beta" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76656 (owner: 10Demon)
[23:19:17] 	 ^demon: my version should probably be moved to CommonSettings-labs.php
[23:19:24] 	 ^demon:  I think y'all broke search suggestions for mobile on beta at least http://en.m.wikipedia.beta.wmflabs.org
[23:19:58] 	 !log mwalker synchronized php-1.22wmf12/extensions/CentralNotice  'Updating CentralNotice to master on wmf12'
[23:20:02] <^demon>	 chrismcmahon: We haven't finished indexing yet.
[23:20:08] 	 Logged the message, Master
[23:20:17] 	 ^demon: ah, thanks, that'd do it
[23:20:58] 	 so many, many ways to break beta labs still :)
[23:21:31] <^demon>	 We actually found a completely unrelated bug in the process of indexing, which is what hashar and I are trying to fix now.
[23:21:38] <^demon>	 Turns out the Timeline extension never worked in beta :)
[23:22:09] 	 ^demon:  I sort of saw that discussion go by.  I've never even heard of the Timeline extension :)
[23:22:13] 	 !log mwalker synchronized wmf-config/CommonSettings.php  'Enabling wgNoticeUseLanguageConversion for automatic zh conversion in CentralNotice'
[23:22:24] 	 Logged the message, Master
[23:24:19] 	 (03PS2) 10Hashar: beta: points timeline backend to local directory [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76652 
[23:24:24] <^demon>	 chrismcmahon: It's the extension that generates stuff like https://en.wikipedia.org/wiki/The_Five_(composers)#Timeline
[23:24:29] 	 (03PS3) 10Hashar: beta: points timeline backend to local directory [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76652 
[23:24:45] 	 ^demon: slightly enhanced my patch by moving my bits under filebackend-labs.php
[23:24:46] <^demon>	 chrismcmahon: Ancient extension half written in Perl :)
[23:25:02] 	 that is a tech debt
[23:25:08] 	 which half :)
[23:25:09] 	 we could do something much more powerful nowadays
[23:25:16] <^demon>	 chrismcmahon: The half that we'd like to get rid of ;-)
[23:25:55] 	 (03CR) 10Demon: [C: 032] beta: points timeline backend to local directory [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76652 (owner: 10Hashar)
[23:26:09] 	 (03Merged) 10jenkins-bot: beta: points timeline backend to local directory [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/76652 (owner: 10Hashar)
[23:26:13] 	 chrismcmahon: some examples at http://www.mediawiki.org/wiki/Extension:Timeline#Charts_examples :)
[23:26:56] 	 ^demon: that solved it
[23:27:04] 	 http://upload.beta.wmflabs.org/wikipedia/en/timeline/f0b64505a9ffbc4a62b65f9b6eb6f4f1.png
[23:27:10] 	 though it is lacking fonts hehe
[23:27:34] <^demon>	 We can deal with that later.
[23:27:38] 	 welcome in the world of the never ending "lets fix beta"
[23:27:43] <^demon>	 :)
[23:28:06] <^demon>	 manybubbles: Indexing should be possible again.
[23:28:09] 	 with that projects I have discovered a lot of our infrastructure
[23:28:36] 	 that made me realize how complicated our infrastructure is
[23:29:10] <^demon>	 hashar: And that parts of it are held together with duct tape and prayers ;-)
[23:29:20] 	 hashar:  I like to think we're fixing beta so that we can deploy to prod better in the future ^demon
[23:29:30] 	 ^demon: if the stack trace was blocking indexing, it is maybe something you want to handle gracefully, maybe skip the page and log it somewhere
[23:29:59] <^demon>	 Possibly. At the same time tho it was a pretty bad error that affected a lot more than just indexing.
[23:30:02] 	 chrismcmahon: rob asked me earlier my feeling about how much bugs are related to the beta infras and how much are related to actual code issue
[23:30:22] 	 I went saying that nowadays most bugs are in the software (or puppet :D )
[23:30:52] * hashar  digs for FreeSansWMF.ttf
[23:35:27] 	 ahhh
[23:35:40] 	 wmf-config/CommonSettings.php:putenv( "GDFONTPATH=/usr/local/apache/common/fonts" );
[23:35:42] 	 of course
[23:36:11] 	 now I am wondering whether I should put those files in the git repository :-D
[23:36:32] 	 125MB hmm no
[23:37:11] 	 (03PS2) 10Dzahn: the existing etherpad-lite_1.0-wm2 package in a operations/debs repo for completeness [operations/debs/etherpad-lite] - 10https://gerrit.wikimedia.org/r/76654 
[23:38:42] <^demon>	 lol
[23:39:16] 	 as a good lazy guy
[23:39:33] 	 I created a tar ball, transferred it to fenari and now wgetting it  on beta
[23:40:26] 	 100%[=================================================================>] 60,709,725  26.0M/s   in 2.2s    
[23:40:27] 	  :-D
[23:40:56] 	 narf, not going to install nodejs on my laptop to get npm, to be able to build 
[23:41:04] 	 do you guys have a nodejs build host or soemthing :p
[23:41:29] 	 brew install nodejs ? :D
[23:41:37] 	 i don't have any, sorry
[23:42:07] 	 hehe, gem install ?
[23:42:36] 	 ^demon: http://upload.beta.wmflabs.org/wikipedia/en/timeline/f0b64505a9ffbc4a62b65f9b6eb6f4f1.png?foobar :-]
[23:42:42] * hashar  flexes
[23:43:25] * ^demon  gives hashar the hacker barnstar
[23:44:26] 	 so that also mean we are probably syncing 125MB of font whenever we sync-dir
[23:44:33] 	 or scap or whatever
[23:44:40] 	 which is … not that nice
[23:44:56] 	 though rsync is probably smart enough to by pass them based on timestamp
[23:45:26] 	 I am wondering whether they should be packaged
[23:46:42] 	 chrismcmahon: did I show you the CI/beta dashboard at https://integration.wikimedia.org/dashboard/  ? That list the status of beta Jenkins jobs
[23:47:15] 	 chrismcmahon: even migrated this afternoon my old lame shell script loop to a jenkins job https://integration.wikimedia.org/ci/job/beta-code-update/  that should break whenever extension is not properly fetched
[23:47:21] 	 (03PS1) 10Dzahn: RT #5464 - apply etherpad-lite live hack fix by apergos [operations/debs/etherpad-lite] - 10https://gerrit.wikimedia.org/r/76661 
[23:47:35] 	 nice thanks hashar
[23:47:54] 	 chrismcmahon: I need to split that job in smaller part to make it obvious as to what is wrong.
[23:48:04] 	 would be for later I guess
[23:52:07] 	 (03CR) 10Dzahn: "http://apt.wikimedia.org/wikimedia/pool/main/e/etherpad-lite/" [operations/debs/etherpad-lite] - 10https://gerrit.wikimedia.org/r/76654 (owner: 10Dzahn)