[00:04:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 9.671 seconds [00:04:33] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:04:43] !log kaldari synchronized php-1.21wmf2/extensions/CentralNotice/modules/ext.centralNotice.bannerController/bannerController.js 'fixing bannerController' [00:04:48] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/30004 [00:04:57] Logged the message, Master [00:06:30] RECOVERY - SSH on sq82 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1 (protocol 2.0) [00:09:12] PROBLEM - Swift HTTP on ms-fe1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:09:32] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:12:48] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:13:27] Can someone take a look at mw34 and m45? They're causing a lot of "fclose()/fwrite() expects parameter 1 to be resource, boolean given" warnings [00:13:33] !log pgehres synchronized php-1.21wmf2/extensions/FundraiserLandingPage/ 'Updating FundraiserLandingPage' [00:13:48] Logged the message, Master [00:14:19] !log pgehres synchronized php-1.21wmf2/extensions/ContributionTracking/ 'Updating ContributionTracking' [00:14:34] Logged the message, Master [00:15:05] !log olivneh synchronized php-1.21wmf2/extensions/ArticleFeedbackv5/ArticleFeedbackv5.hooks.php [00:15:20] Logged the message, Master [00:15:39] PROBLEM - NTP on sq82 is CRITICAL: NTP CRITICAL: No response from NTP server [00:17:45] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:20:45] PROBLEM - Swift HTTP on ms-fe1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:25:43] PROBLEM - LVS HTTP IPv6 on wikivoyage-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: Connection refused [00:26:00] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:26:13] PROBLEM - LVS HTTP IPv6 on wikivoyage-lb.pmtpa.wikimedia.org_ipv6 is CRITICAL: Connection refused [00:26:29] PROBLEM - LVS HTTPS IPv6 on wikivoyage-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: Connection refused [00:26:59] PROBLEM - LVS HTTPS IPv6 on wikivoyage-lb.pmtpa.wikimedia.org_ipv6 is CRITICAL: Connection refused [00:29:01] RECOVERY - NTP on sq82 is OK: NTP OK: Offset -0.06016409397 secs [00:35:39] PROBLEM - Puppet freshness on db9 is CRITICAL: Puppet has not run in the last 10 hours [00:37:15] PROBLEM - Swift HTTP on ms-fe1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:37:33] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:38:18] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:38:36] PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours [00:45:30] PROBLEM - Swift HTTP on ms-fe1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:50:45] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:51:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 3.455 seconds [00:53:45] PROBLEM - Swift HTTP on ms-fe1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:59:01] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:02:04] PROBLEM - Swift HTTP on ms-fe1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:05:23] PROBLEM - Swift HTTP on ms-fe1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:07:15] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:10:16] PROBLEM - Swift HTTP on ms-fe1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:15:30] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:18:32] PROBLEM - Swift HTTP on ms-fe1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:21:39] !log kaldari synchronized php-1.21wmf2/extensions/CentralNotice/CentralNotice.db.php 'updating CentralNotice.db.php' [01:21:48] PROBLEM - Swift HTTP on ms-fe1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:21:48] Logged the message, Master [01:23:45] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:25:51] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:26:49] PROBLEM - Swift HTTP on ms-fe1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:27:05] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:30:06] PROBLEM - Swift HTTP on ms-fe1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:32:00] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:35:00] PROBLEM - Swift HTTP on ms-fe1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:37:48] !log kaldari synchronized php-1.21wmf2/extensions/CentralNotice 'updating CentralNotice' [01:38:02] Logged the message, Master [01:38:18] PROBLEM - Swift HTTP on ms-fe1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:40:33] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.039 seconds [01:42:12] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 297 seconds [01:43:15] PROBLEM - Swift HTTP on ms-fe1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:43:33] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:43:51] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 12 seconds [01:46:33] PROBLEM - Swift HTTP on ms-fe1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:48:33] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:51:51] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:54:48] PROBLEM - Swift HTTP on ms-fe1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:56:45] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:00:03] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 271 seconds [02:00:03] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:05:02] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:08:18] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:12:12] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:13:15] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:16:15] PROBLEM - Swift HTTP on ms-fe1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:16:33] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:21:30] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:24:30] PROBLEM - Swift HTTP on ms-fe1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:24:55] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:26:57] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.056 seconds [02:32:34] !log LocalisationUpdate completed (1.21wmf2) at Fri Oct 26 02:32:33 UTC 2012 [02:32:47] PROBLEM - Swift HTTP on ms-fe1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:32:48] Logged the message, Master [02:33:03] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:38:00] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:38:21] !log kaldari synchronized php-1.21wmf2/extensions/CentralNotice/CentralNotice.db.php 'updating CentralNotice.db.php' [02:38:31] Logged the message, Master [02:41:00] PROBLEM - Swift HTTP on ms-fe1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:46:15] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:49:18] PROBLEM - Swift HTTP on ms-fe1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:54:12] PROBLEM - Swift HTTP on ms-fe1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:54:30] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:54:57] RECOVERY - Puppet freshness on db9 is OK: puppet ran at Fri Oct 26 02:54:37 UTC 2012 [02:57:30] PROBLEM - Swift HTTP on ms-fe1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:57:48] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:05:45] PROBLEM - Swift HTTP on ms-fe1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:06:03] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:09:03] PROBLEM - Swift HTTP on ms-fe1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:11:00] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:14:00] PROBLEM - Swift HTTP on ms-fe1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:14:09] RECOVERY - Puppet freshness on cp1043 is OK: puppet ran at Fri Oct 26 03:13:57 UTC 2012 [03:14:18] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:19:15] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:22:15] PROBLEM - Swift HTTP on ms-fe1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:30:49] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:32:09] PROBLEM - Swift HTTP on ms-fe1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:38:45] PROBLEM - Swift HTTP on ms-fe1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:39:03] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:44:00] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:46:00] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 18 seconds [03:47:00] PROBLEM - Swift HTTP on ms-fe1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:47:18] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:50:28] New patchset: Krinkle; "Adjust wikidata.org apache config" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/30056 [03:50:58] New review: Krinkle; "* Removed legacy stuff for old Wikipedia domains." [operations/apache-config] (master) C: 1; - https://gerrit.wikimedia.org/r/30056 [03:52:15] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:55:15] PROBLEM - Swift HTTP on ms-fe1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:58:33] PROBLEM - Swift HTTP on ms-fe1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:03:33] PROBLEM - Swift HTTP on ms-fe1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:06:49] PROBLEM - Swift HTTP on ms-fe1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:08:45] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:11:46] PROBLEM - Swift HTTP on ms-fe1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:13:45] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:15:03] PROBLEM - Swift HTTP on ms-fe1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:22:07] PROBLEM - Swift HTTP on ms-fe1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:25:24] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:26:56] PROBLEM - LVS HTTP IPv6 on wikivoyage-lb.pmtpa.wikimedia.org_ipv6 is CRITICAL: Connection refused [04:27:01] PROBLEM - LVS HTTP IPv6 on wikivoyage-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: Connection refused [04:27:19] PROBLEM - LVS HTTPS IPv6 on wikivoyage-lb.pmtpa.wikimedia.org_ipv6 is CRITICAL: Connection refused [04:27:19] PROBLEM - LVS HTTPS IPv6 on wikivoyage-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: Connection refused [04:30:14] PROBLEM - Swift HTTP on ms-fe1002 is CRITICAL: Connection refused [04:39:43] New review: Aude; "@krinkle: thanks for removing the legacy stuff :) the config looks good to me." [operations/apache-config] (master) C: 1; - https://gerrit.wikimedia.org/r/30056 [05:07:58] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:22:12] PROBLEM - Swift HTTP on ms-fe1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:27:11] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:40:51] mornin [05:41:59] PROBLEM - Swift HTTP on ms-fe1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:43:01] !log powering off ms-fe1001 & ms-fe1002 [05:43:08] Logged the message, Master [05:44:50] PROBLEM - Puppet freshness on db42 is CRITICAL: Puppet has not run in the last 10 hours [05:44:50] PROBLEM - Puppet freshness on ms-be7 is CRITICAL: Puppet has not run in the last 10 hours [05:44:50] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [05:45:26] PROBLEM - Host ms-fe1001 is DOWN: PING CRITICAL - Packet loss = 100% [05:45:26] PROBLEM - Host ms-fe1002 is DOWN: PING CRITICAL - Packet loss = 100% [07:55:05] PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours [08:02:08] PROBLEM - Puppet freshness on sq76 is CRITICAL: Puppet has not run in the last 10 hours [08:04:28] New patchset: Raimond Spekking; "Remove leftover from wgEnableHtmlDiff experiment" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/30129 [08:26:44] morning [08:27:29] PROBLEM - LVS HTTP IPv6 on wikivoyage-lb.pmtpa.wikimedia.org_ipv6 is CRITICAL: Connection refused [08:27:38] PROBLEM - LVS HTTP IPv6 on wikivoyage-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: Connection refused [08:28:14] false alarm I guess? [08:28:43] PROBLEM - LVS HTTPS IPv6 on wikivoyage-lb.pmtpa.wikimedia.org_ipv6 is CRITICAL: Connection refused [08:28:50] PROBLEM - LVS HTTPS IPv6 on wikivoyage-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: Connection refused [08:30:28] yeah the checks were supposedly disabled but... [08:30:31] guess they got reenabled [08:31:35] apergos: did you fix that virtual resource? :-) [08:31:43] ah [08:31:43] (morning) [08:31:44] no :-D [08:32:43] I've been debating whether this is a sick day or not (in the literal sense, it is: I am sick, some sort of stomach flu... just hoping I'll feel better enough in a few hours to do actual work) [08:32:50] I only sat down here cause of the alerts [08:34:28] I wonder if he edited something manually to disable the checks [08:40:57] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [08:40:57] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [08:58:03] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [09:11:40] paravoid: thank to your upgrade of the formey server, we now have a nicer documentation for MediaWiki :-] http://svn.wikimedia.org/doc/ [09:11:55] the old doxygen 1.6 produced some very old / badly shaped HTML [09:12:12] cool! [09:13:40] one less RT ticket [09:13:43] one less bug :-] [09:14:15] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:18:24] RECOVERY - NTP on sodium is OK: NTP OK: Offset 0.07437431812 secs [09:19:03] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.059 seconds [09:23:26] * aude waves [09:23:41] hello aude [09:29:34] anyone got an idea, what the best way to debug why urls like http://upload.wikimedia.org/wikipedia/commons/thumb/9/94/Folgers.ogv/352px--Folgers.ogv.jpg give me a 404 would be? I am not able to reproduce this with my local setup (not sure how to setup 404 handler) i.e. is it possible to add something like print_r($params, true) to the error in thumbs.php:245 shortly to see what state it fails in [09:31:03] they the double dash? [09:31:25] paravoid: not sure it was always like that [09:31:36] its width-position-name [09:33:17] is anyone looking at the apache config for wikidata today? (e.g. before Reedy or Chad come online) [09:33:17] https://gerrit.wikimedia.org/r/#/c/30056/ [09:41:32] j^: so, you should take upload.wm.org out of the equation [09:41:39] it's http://commons.wikimedia.org/w/thumb_handler.php/9/94/Folgers.ogv/352px--Folgers.ogv.jpg [09:41:49] aude, isn't there some proposal about redirecting subdomains for languages to some place? [09:41:57] paravoid: but that would not get redirected to the imagescalers [09:42:09] aude: (for wikidata) [09:42:09] Hydriz: yes but doesn't need to be done for initial setup [09:42:19] aude: Oh good :) [09:42:27] just a simple setup with the extensions is a good step [09:42:40] paravoid: upload makes a request to the imagescalers with Host: commons.wikimedia.org [09:42:55] Hydriz: http://meta.wikimedia.org/wiki/Wikidata/Notes/URI_scheme [09:42:57] yes [09:43:10] you can try the same from fenari [09:43:10] aude: yep, that is what I am referring to :) [09:43:13] not sure they are most up-to-date but if you want to look and give feedback... [09:43:16] ok [09:43:17] rendering.svc.pmtpa.wmnet or a random imagescaler box [09:43:30] paravoid: can i login to fenari? [09:43:44] oh you can't? [09:44:01] so far not, no access to production [09:44:28] oh [09:44:45] kinda hard to debug issues with no production access I'd guess [09:45:02] sometimes [09:49:03] New review: Hydriz; "Looks okay, the language codes will be properly redirected (and configured) later on anyways." [operations/apache-config] (master) C: 1; - https://gerrit.wikimedia.org/r/30056 [09:49:24] grrr bad link [09:50:08] j^: I think you'll have better luck asking in a MW channel [09:50:20] or chat with Aaron [09:52:16] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:55:33] aude: I'm about, just got a visitor atm [09:55:41] hi Reedy [09:55:49] Need to poke ops to review and deploy the new apache config [09:55:56] * aude is around if anyone has questions but it looks good [09:56:15] i'll submit a patch for the mediawiki-config to exclude the property and query namespaces for now [09:56:35] paravoid: j^ : is that about requesting http://commons.wikimedia.org/w/thumb_handler.php/9/94/Folgers.ogv/352px--Folgers.ogv.jpg to rendering.svc.pmtpa.wmnet ? [09:56:35] ok :) [09:56:42] otherwise that one looks good also [09:56:42] I can run that using curl on fenari [09:59:02] j^: here the result http://dpaste.org/8Iriw/ :-} [10:03:22] Reedy: denny suggest to keep the namespaces, but the way the extension is setup, people can't create stuff in the property or query namespace yet [10:03:41] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.033 seconds [10:03:43] but the namespaces are reserved [10:13:15] New review: Aude; "The configuration looks good from our side." [operations/mediawiki-config] (master) C: 1; - https://gerrit.wikimedia.org/r/30068 [10:33:17] Hey. Could someone please review and merge https://gerrit.wikimedia.org/r/#/c/30056/ and hten deploy and graceful the apaches? Thanks! [10:34:17] hashar: thanks, i would need more debug output, will chat with Aaron later [10:36:49] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:37:24] New patchset: Aude; "add wikidata logo" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/30139 [10:37:34] Reedy: i hope that's helpful [10:37:49] * aude is getting help to fully protect the logo on commons [10:37:49] heh :) [10:37:56] I can protect it if needed [10:39:23] we got it [10:39:50] * aude is admin too but don't want to get in trouble for abusing admin powers [10:39:50] PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours [10:41:56] New patchset: Aude; "add wikidata logo" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/30139 [10:43:22] and then it will be wikidatawiki => '$stdlogo', [10:51:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.024 seconds [11:05:02] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/29886 [11:05:22] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/29807 [11:06:13] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/30129 [11:07:11] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/28699 [11:07:36] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/28044 [11:07:39] moaaar shell requests! [11:08:08] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/27212 [11:08:38] noo am I missing another train [11:09:09] Reedy: you can merge the logos IMHO, they're not really broken (just not perfect) and will be perfectioned this afternoon [11:09:19] Yeah [11:09:32] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/29890 [11:09:55] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/23985 [11:10:55] yay [11:11:30] And now my laptop is playing up [11:12:33] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/29775 [11:13:01] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/29611 [11:13:40] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/30092 [11:20:33] !log reedy synchronized wmf-config/ [11:20:45] Logged the message, Master [11:22:47] Reedy: you said something about needing ops help? [11:23:46] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:23:53] paravoid: https://gerrit.wikimedia.org/r/#/c/30056/ [11:36:59] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 8.511 seconds [12:11:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:24:42] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.048 seconds [12:27:49] PROBLEM - LVS HTTP IPv6 on wikivoyage-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: Connection refused [12:28:16] PROBLEM - LVS HTTP IPv6 on wikivoyage-lb.pmtpa.wikimedia.org_ipv6 is CRITICAL: Connection refused [12:29:55] PROBLEM - LVS HTTPS IPv6 on wikivoyage-lb.eqiad.wikimedia.org_ipv6 is CRITICAL: Connection refused [12:29:56] PROBLEM - LVS HTTPS IPv6 on wikivoyage-lb.pmtpa.wikimedia.org_ipv6 is CRITICAL: Connection refused [12:45:33] PROBLEM - Backend Squid HTTP on sq82 is CRITICAL: Connection refused [12:46:18] !log stopping squid3(!) on sq82, box was reprovisioned with precise but we don't support squid3/squid on precise yet [12:46:27] Logged the message, Master [12:47:18] oh oops [12:49:12] hehe [12:49:28] it had a shiny new error page! [12:53:27] How long hasn't unified login been working? A few people confirmed that it wasn't loggin in on other projects a few weeks ago when I asked, but seems to still be the case. [12:55:24] New patchset: jan; "Refactor the webserver classes" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/30147 [12:57:30] RD: username and project, please? [12:57:43] there are some known bugs with unification [12:57:44] Any and any. [12:57:46] :P [12:57:51] not possibl [12:57:52] e [12:58:12] Rjd0060/3 other users meta/en.wiki/etc [12:58:12] don't ever trust generic SUL complaints, there are too variables [12:58:18] hmm? [12:58:27] Me [12:58:29] And 3 other users. [12:58:36] On meta. And enwiki. And other projects. [12:58:42] logging in where and finding you logged out where else? [12:58:42] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:58:45] Correct. [12:58:59] Each project is making me log in. I am aware of cookie settings, etc. [12:59:04] Just started happening a while ago. [12:59:36] Logging out seems to log out of all projects. [12:59:47] Try for yourself and see. :-) [12:59:58] Maybe it'll work for you. [13:01:05] sorry, I don't understand what's the problem [13:01:20] New review: Aude; "depending on the database setup, I think the logo might need to go in the section just below, such as:" [operations/mediawiki-config] (master) C: 0; - https://gerrit.wikimedia.org/r/30139 [13:02:12] works for me [13:02:16] except that it tries to load a non existing icon from http://www.wikidata.org/wiki/Special:AutoLogin?token=3ded606ef57602aac112a4b0b988e227 [13:02:20] aude? [13:03:05] New patchset: Aude; "add wikidata logo" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/30139 [13:04:06] by a minute :p [13:05:34] New review: Aude; "I think there is sufficient consensus now to approve this change:" [operations/mediawiki-config] (master) C: 0; - https://gerrit.wikimedia.org/r/23927 [13:06:23] Change merged: Demon; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/30139 [13:06:47] Nemo_bis: what? [13:06:59] aude: you just fixed it :) [13:07:04] ok :) [13:07:14] <^demon> Syncing the wikidata logo. [13:07:14] !log demon synchronized wmf-config/InitialiseSettings.php 'Syncing I85721736' [13:07:17] yay! [13:07:29] Logged the message, Master [13:08:14] <^demon> Nemo_bis: The Special:Autologin stuff is dependent on the apache changes to be sync'd out. [13:13:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.035 seconds [13:24:15] Reedy or ^demon: when you get a chance, i think https://gerrit.wikimedia.org/r/#/c/30113/ should be backported and deployed [13:32:17] is anyone deploying anything right now? I need to sync a configuration change [13:36:59] Change merged: MaxSem; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/30070 [13:37:29] <^demon> MaxSem: I deployed something about 30m ago. Nobody else is doing anything right now afaik. [13:38:05] yeah, I already took silence as no^_^ [13:40:59] !log maxsem synchronized wmf-config/mobile.php 'https://gerrit.wikimedia.org/r/#/c/30070/' [13:41:11] Logged the message, Master [13:44:39] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:51:18] PROBLEM - check_apache2 on payments2 is CRITICAL: PROCS CRITICAL: 0 processes with command name apache2 [13:55:21] RECOVERY - check_apache2 on payments2 is OK: PROCS OK: 7 processes with command name apache2 [13:57:40] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 8.349 seconds [14:08:46] !log demon synchronized php-1.21wmf2/extensions/AbuseFilter/modules 'Updating AbuseFilter to f52f940e' [14:08:59] Logged the message, Master [14:12:54] PROBLEM - Host payments1004 is DOWN: PING CRITICAL - Packet loss = 100% [14:15:18] RECOVERY - Host payments1004 is UP: PING OK - Packet loss = 0%, RTA = 35.67 ms [14:26:04] mark: are you still around? Would you mind talking about the varnish CC parameter ordering ? https://gerrit.wikimedia.org/r/#/c/24797/ [14:28:18] !log demon synchronized php-1.21wmf2/extensions/AbuseFilter/modules/ext.abuseFilter.examine.js 'Fixing JS for AbuseFilter' [14:28:30] Logged the message, Master [14:28:42] !log demon synchronized php-1.21wmf2/extensions/AbuseFilter/modules/ext.abuseFilter.tools.js 'Fixing JS for AbuseFilter' [14:28:49] Logged the message, Master [14:29:00] works... finally... :) [14:31:21] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:35:42] PROBLEM - check_apache2 on payments4 is CRITICAL: PROCS CRITICAL: 0 processes with command name apache2 [14:40:12] RECOVERY - check_apache2 on payments4 is OK: PROCS OK: 8 processes with command name apache2 [14:46:03] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.028 seconds [14:48:52] <^demon> Anyone mind reviewing some apache config changes for me? https://gerrit.wikimedia.org/r/#/c/30056/ [15:04:21] +1 on the apache changes bug someone more knowledgable, please review https://gerrit.wikimedia.org/r/#/c/30056/ [15:18:36] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:20:15] PROBLEM - check_mysql on payments2 is CRITICAL: Slave IO: No Slave SQL: Yes Seconds Behind Master: (null) [15:20:15] PROBLEM - check_mysql on payments4 is CRITICAL: Slave IO: No Slave SQL: Yes Seconds Behind Master: (null) [15:20:15] PROBLEM - check_mysql on payments3 is CRITICAL: Slave IO: No Slave SQL: Yes Seconds Behind Master: (null) [15:24:27] PROBLEM - Puppet freshness on mw48 is CRITICAL: Puppet has not run in the last 10 hours [15:25:12] RECOVERY - check_mysql on payments2 is OK: Uptime: 5831 Threads: 2 Questions: 37610 Slow queries: 0 Opens: 33 Flush tables: 1 Open tables: 26 Queries per second avg: 6.450 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [15:25:13] RECOVERY - check_mysql on payments4 is OK: Uptime: 3068 Threads: 1 Questions: 15464 Slow queries: 0 Opens: 29 Flush tables: 1 Open tables: 22 Queries per second avg: 5.40 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [15:25:13] RECOVERY - check_mysql on payments3 is OK: Uptime: 65343 Threads: 1 Questions: 401846 Slow queries: 11 Opens: 277 Flush tables: 1 Open tables: 64 Queries per second avg: 6.149 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [15:31:48] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 9.814 seconds [15:45:34] PROBLEM - Puppet freshness on db42 is CRITICAL: Puppet has not run in the last 10 hours [15:45:34] PROBLEM - Puppet freshness on ms-be7 is CRITICAL: Puppet has not run in the last 10 hours [15:45:34] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [16:06:02] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:17:22] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 7.717 seconds [16:46:56] New patchset: Demon; "Disabling EventLogging on enwiki until it's fixed" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/30174 [16:49:04] Change merged: RobLa; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/30174 [16:49:09] LeslieCarr: are you going to be in the office today at all? [16:49:41] !log demon synchronized wmf-config/InitialiseSettings.php 'Disabling EventLogging on enwiki' [16:49:54] Logged the message, Master [16:50:11] ^demon: Did you just do that because of the editing issue? [16:50:38] <^demon> Yes. [16:51:08] ^demon: okay thanks for the clarification [16:51:55] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:53:58] LeslieCarr: I'd like to talk with you about the esams a bit if you have any spare cycles today [17:08:07] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.017 seconds [17:13:30] preilly - looks like leslie is sick today .... [17:13:43] LeslieCarr: I hope you feel better soon — get some rest [17:13:53] woosters: thanks for the update [17:21:50] New review: Dzahn; "yep, +1 for removing from wikipedia.conf" [operations/apache-config] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/30056 [17:21:50] Change merged: Dzahn; [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/30056 [17:32:16] !log sync-apache new wikidata config [17:32:42] Logged the message, Master [17:35:01] dzahn is doing a graceful restart of all apaches [17:35:26] !log dzahn gracefulled all apaches [17:35:40] Logged the message, Master [17:36:28] <^demon> Reedy: So, new apache config is out but we're still ending up at [[m:Missing wiki]] [17:36:32] Reedy: see above [17:36:42] http://de.wikidata.org * 302 Found http://meta.wikimedia.org/wiki/Missing_wiki [17:37:11] (using Jeff-test-script) [17:37:11] Yeah, was just looking also :/ [17:37:11] called "apache-fast-test" on fenari [17:37:27] www is also still showing the landing page... [17:37:42] <^demon> That should've disappeared since the stopgap stuff was removed. [17:37:56] and IIRC it's got no cache headers [17:38:58] New patchset: CSteipp; "Initial apache config for Wikivoyage" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/30182 [17:39:10] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:42:17] !log aaron synchronized php-1.21wmf2/includes/upload/UploadStash.php 'debug logging' [17:42:30] Logged the message, Master [17:44:20] yeah, Chris' config actually looks like it should work ..like the wikidata one .. [17:44:38] !log aaron synchronized php-1.21wmf2/includes/upload/UploadStash.php [17:44:51] Logged the message, Master [17:47:29] <^demon> mutante: So by that you mean not at all? [17:47:59] New patchset: Ryan Lane; "Add backups to labsconsole" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/30184 [17:48:09] hmm, yes. are we just caching that redirect to missing wikis ? [17:48:19] looking [17:50:12] !log aaron synchronized php-1.21wmf2/includes/upload/UploadStash.php 'done' [17:50:22] Logged the message, Master [17:50:57] <^demon> mutante: Could be. [17:54:11] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.034 seconds [17:54:33] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/30184 [17:55:58] PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours [17:57:46] ^demon: meeting.. bbiab [17:58:08] <^demon> mmk [17:58:38] !log aaron synchronized php-1.21wmf2/thumb.php 'improve path mismatch error message' [17:58:50] Logged the message, Master [18:02:10] !log aaron synchronized php-1.21wmf2/thumb.php 'done' [18:02:22] Logged the message, Master [18:02:52] PROBLEM - Puppet freshness on sq76 is CRITICAL: Puppet has not run in the last 10 hours [18:21:58] New patchset: Ryan Lane; "Mute the backup crons for labsconsole" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/30191 [18:25:10] Ryan_Lane: /dev/null 2>&1 is your friend [18:25:17] +1 [18:27:01] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:27:27] !log erzurumi dist-upgrade and reboot [18:27:42] Logged the message, Master [18:29:16] PROBLEM - Host erzurumi is DOWN: PING CRITICAL - Packet loss = 100% [18:29:48] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/30191 [18:30:15] Change merged: Ryan Lane; [operations/debs/adminbot] (master) - https://gerrit.wikimedia.org/r/27657 [18:30:55] RECOVERY - Host erzurumi is UP: PING OK - Packet loss = 0%, RTA = 0.21 ms [18:31:33] !log aluminium dist upgrade and reboot [18:31:45] Logged the message, Master [18:36:31] yay, wikidata.org is a missing wiki :) [18:36:38]