[00:02:38] !log reedy synchronized wmf-config/InitialiseSettings.php [00:02:44] Logged the message, Master [00:05:28] !log reedy synchronized wmf-config/InitialiseSettings.php [00:05:34] Logged the message, Master [00:19:24] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:31:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.402 seconds [00:36:05] is wikivoyage already connected to sul? [00:40:52] Yup [00:45:17] now just to add the icon on login pages [00:45:22] i bet there's a bug request for it already though [00:45:36] No [00:45:38] It's already there [00:46:03]

[00:46:24] I'd already pre-emptively prepared a changeset for it ;) [00:46:47] hey Reedy [00:46:53] ohai [00:47:02] everything okay? [00:47:38] Seems to be... :) [00:48:06] great [00:48:10] do you need anything? [00:48:58] Not unless you really really want to poke at lucene [00:49:48] fuck no [00:49:56] I was never here [00:49:57] :P [00:50:05] We'll wait 5 or 6 hours then [00:50:10] And see if Asher did some magic :D [00:59:15] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [01:07:21] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:08:33] !log reedy synchronized wmf-config/InitialiseSettings.php [01:08:40] Logged the message, Master [01:20:55] New patchset: Reedy; "Revert "Change RelatedSites for nlwikivoyage"" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32849 [01:21:08] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32849 [01:21:37] !log reedy synchronized wmf-config/InitialiseSettings.php [01:21:44] Logged the message, Master [01:22:12] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.031 seconds [01:30:43] Hi. I'm trying to change some CentralAuth wikisets to add wikidatawiki and I get [33418f51] 2012-11-11 01:29:56: Fatal exception of type MWException [01:30:57] (Internal error) [01:31:05] Reedy ^ [01:31:49] the numbers in brackets changes each time I try [01:32:06] now its [51a0338b] [01:32:25] bugzilla time [01:42:18] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 278 seconds [01:42:27] PROBLEM - MySQL Slave Delay on db78 is CRITICAL: CRIT replication delay 281 seconds [01:44:06] RECOVERY - MySQL Slave Delay on db78 is OK: OK replication delay 0 seconds [01:47:15] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 26 seconds [01:53:51] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:00:27] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 278 seconds [02:00:36] PROBLEM - MySQL Slave Delay on db78 is CRITICAL: CRIT replication delay 286 seconds [02:07:03] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 1.659 seconds [02:27:52] PROBLEM - Puppet freshness on analytics1011 is CRITICAL: Puppet has not run in the last 10 hours [02:27:52] PROBLEM - Puppet freshness on analytics1012 is CRITICAL: Puppet has not run in the last 10 hours [02:27:52] PROBLEM - Puppet freshness on analytics1013 is CRITICAL: Puppet has not run in the last 10 hours [02:27:52] PROBLEM - Puppet freshness on analytics1015 is CRITICAL: Puppet has not run in the last 10 hours [02:27:52] PROBLEM - Puppet freshness on analytics1018 is CRITICAL: Puppet has not run in the last 10 hours [02:27:53] PROBLEM - Puppet freshness on analytics1016 is CRITICAL: Puppet has not run in the last 10 hours [02:27:53] PROBLEM - Puppet freshness on analytics1014 is CRITICAL: Puppet has not run in the last 10 hours [02:27:54] PROBLEM - Puppet freshness on analytics1017 is CRITICAL: Puppet has not run in the last 10 hours [02:27:54] PROBLEM - Puppet freshness on analytics1019 is CRITICAL: Puppet has not run in the last 10 hours [02:27:55] PROBLEM - Puppet freshness on analytics1021 is CRITICAL: Puppet has not run in the last 10 hours [02:27:55] PROBLEM - Puppet freshness on analytics1020 is CRITICAL: Puppet has not run in the last 10 hours [02:27:56] PROBLEM - Puppet freshness on analytics1022 is CRITICAL: Puppet has not run in the last 10 hours [02:27:56] PROBLEM - Puppet freshness on db42 is CRITICAL: Puppet has not run in the last 10 hours [02:27:57] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [02:32:27] !log LocalisationUpdate completed (1.21wmf3) at Sun Nov 11 02:32:27 UTC 2012 [02:32:34] Logged the message, Master [03:44:07] RECOVERY - MySQL Slave Delay on db78 is OK: OK replication delay 0 seconds [03:50:07] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 17 seconds [04:38:42] PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours [04:48:00] PROBLEM - Host tmh1 is DOWN: PING CRITICAL - Packet loss = 100% [04:50:42] PROBLEM - Puppet freshness on cp1042 is CRITICAL: Puppet has not run in the last 10 hours [05:37:00] PROBLEM - LVS Lucene on search-pool4.svc.eqiad.wmnet is CRITICAL: Connection timed out [05:38:21] RECOVERY - LVS Lucene on search-pool4.svc.eqiad.wmnet is OK: TCP OK - 0.027 second response time on port 8123 [05:42:59] New patchset: Ori.livneh; "Enable event logging for mobile beta" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32864 [05:57:46] New patchset: awjrichards; "Add HTTP header X-WAP for mobilefrontend" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/32866 [06:00:53] New review: awjrichards; "Related change: https://gerrit.wikimedia.org/r/#/c/32867/" [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/32866 [06:09:24] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [06:09:24] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [06:09:24] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [06:16:11] paravoid: you around? [06:27:20] Reedy: Are you there? [06:59:18] New review: MaxSem; "This change depends on latest MobileFrontend, so it shouldn't be merged before the mobile deployment..." [operations/mediawiki-config] (master); V: 0 C: -2; - https://gerrit.wikimedia.org/r/32864 [07:17:53] is search-redirect.php in git somewhere? [07:25:56] PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours [07:29:42] Eloquence_: looks like wmf-mediawiki-config/search-redirect.php [07:30:12] (so operations/mediawiki-config.git iirc) [07:30:25] got it, thanks [07:30:57] * jeremyb has to in turn thank mlocate ;) [07:33:42] New patchset: Eloquence; "Add Wikivoyage family to search-redirect lookup." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32881 [09:15:17] Reedy, look in fatalmonitor [11:00:09] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [11:07:21] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:13:48] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.042 seconds [11:42:09] RECOVERY - SSH on ms1002 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [11:45:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:58:21] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 3.862 seconds [12:29:21] PROBLEM - Puppet freshness on analytics1012 is CRITICAL: Puppet has not run in the last 10 hours [12:29:21] PROBLEM - Puppet freshness on analytics1011 is CRITICAL: Puppet has not run in the last 10 hours [12:29:21] PROBLEM - Puppet freshness on analytics1014 is CRITICAL: Puppet has not run in the last 10 hours [12:29:21] PROBLEM - Puppet freshness on analytics1013 is CRITICAL: Puppet has not run in the last 10 hours [12:29:21] PROBLEM - Puppet freshness on analytics1015 is CRITICAL: Puppet has not run in the last 10 hours [12:29:22] PROBLEM - Puppet freshness on analytics1018 is CRITICAL: Puppet has not run in the last 10 hours [12:29:22] PROBLEM - Puppet freshness on analytics1020 is CRITICAL: Puppet has not run in the last 10 hours [12:29:23] PROBLEM - Puppet freshness on analytics1019 is CRITICAL: Puppet has not run in the last 10 hours [12:29:23] PROBLEM - Puppet freshness on analytics1017 is CRITICAL: Puppet has not run in the last 10 hours [12:29:24] PROBLEM - Puppet freshness on analytics1022 is CRITICAL: Puppet has not run in the last 10 hours [12:29:24] PROBLEM - Puppet freshness on analytics1021 is CRITICAL: Puppet has not run in the last 10 hours [12:29:25] PROBLEM - Puppet freshness on analytics1016 is CRITICAL: Puppet has not run in the last 10 hours [12:29:25] PROBLEM - Puppet freshness on db42 is CRITICAL: Puppet has not run in the last 10 hours [12:29:26] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [12:33:51] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:48:33] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.024 seconds [13:20:03] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:33:33] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 3.760 seconds [14:03:06] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32881 [14:09:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:15:57] !log reedy synchronized search-redirect.php [14:16:08] Logged the message, Master [14:17:09] Hmmm [14:17:19] tmh1 seems to be down [14:17:53] !log reedy synchronized php/cache/interwiki.cdb 'Updating interwiki cache' [14:17:59] Logged the message, Master [14:22:57] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 8.484 seconds [14:34:24] !log reedy synchronized php-1.21wmf3/extensions/WikimediaIncubator/ [14:34:30] Logged the message, Master [14:40:03] PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours [14:44:00] New patchset: MF-Warburg; "abusefilter-log-detail right from sysop to autoconfirmed" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32681 [14:45:40] New review: Nemo bis; "See comment above." [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/32681 [14:46:17] New review: MF-Warburg; "I don't understand why wikis which have already explicitly put this change into place would now, whe..." [operations/mediawiki-config] (master) C: 0; - https://gerrit.wikimedia.org/r/32681 [14:48:21] !log reedy synchronized php-1.21wmf3/extensions/CentralAuth [14:48:28] Logged the message, Master [14:52:12] PROBLEM - Puppet freshness on cp1042 is CRITICAL: Puppet has not run in the last 10 hours [14:52:30] New review: Nemo bis; "I said the contrary: wikis which didn't set it explicitly as it was the default will probably want t..." [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/32681 [14:55:48] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:09:00] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 7.854 seconds [15:09:08] New review: MF-Warburg; "Bug 42012." [operations/mediawiki-config] (master) C: 0; - https://gerrit.wikimedia.org/r/32681 [15:12:11] !log reedy synchronized php-1.21wmf3/extensions/TimedMediaHandler [15:12:18] Logged the message, Master [15:44:12] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:55:45] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 3.086 seconds [16:08:57] PROBLEM - Puppet freshness on copper is CRITICAL: Puppet has not run in the last 10 hours [16:08:57] PROBLEM - Puppet freshness on db1001 is CRITICAL: Puppet has not run in the last 10 hours [16:08:57] PROBLEM - Puppet freshness on cp1005 is CRITICAL: Puppet has not run in the last 10 hours [16:08:57] PROBLEM - Puppet freshness on mc8 is CRITICAL: Puppet has not run in the last 10 hours [16:08:57] PROBLEM - Puppet freshness on hooper is CRITICAL: Puppet has not run in the last 10 hours [16:08:58] PROBLEM - Puppet freshness on db1031 is CRITICAL: Puppet has not run in the last 10 hours [16:08:58] PROBLEM - Puppet freshness on search33 is CRITICAL: Puppet has not run in the last 10 hours [16:08:59] PROBLEM - Puppet freshness on srv251 is CRITICAL: Puppet has not run in the last 10 hours [16:08:59] PROBLEM - Puppet freshness on sodium is CRITICAL: Puppet has not run in the last 10 hours [16:09:00] PROBLEM - Puppet freshness on srv223 is CRITICAL: Puppet has not run in the last 10 hours [16:10:00] PROBLEM - Puppet freshness on search17 is CRITICAL: Puppet has not run in the last 10 hours [16:10:00] PROBLEM - Puppet freshness on mc3 is CRITICAL: Puppet has not run in the last 10 hours [16:10:00] PROBLEM - Puppet freshness on sq33 is CRITICAL: Puppet has not run in the last 10 hours [16:10:00] PROBLEM - Puppet freshness on srv272 is CRITICAL: Puppet has not run in the last 10 hours [16:10:00] PROBLEM - Puppet freshness on es5 is CRITICAL: Puppet has not run in the last 10 hours [16:10:54] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [16:10:54] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [16:10:54] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [16:18:52] New patchset: Eloquence; "Don't disable edits my unmerged Wikivoyage accounts." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32916 [16:19:28] New patchset: Eloquence; "Don't disable edits by unmerged Wikivoyage accounts." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32916 [16:30:24] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:40:00] PROBLEM - Puppet freshness on search1011 is CRITICAL: Puppet has not run in the last 10 hours [16:40:00] PROBLEM - Puppet freshness on cp1006 is CRITICAL: Puppet has not run in the last 10 hours [16:40:00] PROBLEM - Puppet freshness on analytics1009 is CRITICAL: Puppet has not run in the last 10 hours [16:40:00] PROBLEM - Puppet freshness on cp1025 is CRITICAL: Puppet has not run in the last 10 hours [16:40:00] PROBLEM - Puppet freshness on searchidx1001 is CRITICAL: Puppet has not run in the last 10 hours [16:40:01] PROBLEM - Puppet freshness on srv202 is CRITICAL: Puppet has not run in the last 10 hours [16:45:06] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.036 seconds [17:18:20] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:27:29] PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours [17:28:14] PROBLEM - Apache HTTP on mw49 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:28:41] PROBLEM - Apache HTTP on mw18 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:28:50] PROBLEM - Apache HTTP on mw52 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:28:59] PROBLEM - Apache HTTP on mw35 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:29:08] PROBLEM - Apache HTTP on mw26 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:29:17] PROBLEM - Apache HTTP on mw58 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:29:26] PROBLEM - Apache HTTP on mw54 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:29:35] PROBLEM - Apache HTTP on mw24 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:29:35] PROBLEM - Apache HTTP on mw42 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:29:35] PROBLEM - Apache HTTP on mw28 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:29:44] PROBLEM - Apache HTTP on mw44 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:29:53] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 8.610 seconds [17:30:02] PROBLEM - Apache HTTP on mw22 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:30:09] urgh [17:30:20] RECOVERY - Apache HTTP on mw18 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 1.964 second response time [17:30:29] RECOVERY - Apache HTTP on mw52 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.811 second response time [17:30:38] RECOVERY - Apache HTTP on mw35 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.619 second response time [17:30:38] RECOVERY - Apache HTTP on mw26 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.073 second response time [17:30:56] RECOVERY - Apache HTTP on mw58 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.056 second response time [17:30:56] RECOVERY - Apache HTTP on mw54 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 8.112 second response time [17:31:05] RECOVERY - Apache HTTP on mw24 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.066 second response time [17:31:15] RECOVERY - Apache HTTP on mw42 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.060 second response time [17:31:15] RECOVERY - Apache HTTP on mw28 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.074 second response time [17:31:23] RECOVERY - Apache HTTP on mw44 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 9.739 second response time [17:31:32] RECOVERY - Apache HTTP on mw49 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 7.406 second response time [17:31:32] RECOVERY - Apache HTTP on mw22 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.065 second response time [17:49:21] Change merged: Dzahn; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32916 [17:50:58] there seems to be serious trouble on dewikivoyage [17:51:05] securtiy relevant, pm please [17:52:57] anyone? [17:55:44] !log dzahn synchronized ./wmf-config/InitialiseSettings.php [17:55:52] Logged the message, Master [17:57:29] PROBLEM - Apache HTTP on mw25 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:58:59] RECOVERY - Apache HTTP on mw25 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.055 second response time [18:03:27] New patchset: MF-Warburg; "abusefilter-log-detail right from sysop to autoconfirmed" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32681 [18:04:31] hoo_, has someone PM'd you? If not, file a security bug [18:04:48] Krenair: yep [18:05:46] are security issues suppose to be bugs? I thought there was an email for that due to the impact of others seeing the issue if it's not well know? [18:06:02] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:06:10] Damianz: Those are private in BZ [18:06:13] Damianz, there a security product and group in BZ [18:06:18] Ah [18:06:21] there's* [18:11:03] we are already talking to hoo [18:15:00] bad for 2 users but not a general issue, people are on it [18:20:44] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.028 seconds [18:53:50] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:05:14] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 3.975 seconds [19:06:05] New patchset: Reedy; "Rename *.wikimedia.org.crt to star.wikimedia.org.crt like it is used in files/owa/owa-apache" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/32924 [19:06:51] New patchset: Reedy; "Rename *.wikimedia.org.crt to star.wikimedia.org.crt like it is used in files/owa/owa-apache" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/32924 [19:10:50] New review: Reedy; "Though, should this just be updated to use star.wikimedia.org.pem, and the current *.wikimedia.org.c..." [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/32924 [19:25:59] New patchset: Reedy; "Disable uploads on wikivoyage. Point to commons" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32925 [19:40:02] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:40:34] New patchset: Reedy; "If cli mode, show exceptions!" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32926 [19:41:51] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32926 [19:43:09] Change merged: Dzahn; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32925 [19:43:56] !log reedy synchronized wmf-config/CommonSettings.php [19:44:04] Logged the message, Master [19:47:56] !log reedy synchronized php-1.21wmf3/maintenance/refreshLinks.php [19:48:02] Logged the message, Master [19:49:27] !log demon synchronized wmf-config/InitialiseSettings.php [19:49:33] Logged the message, Master [19:49:45] !log reedy synchronized wmf-config/ [19:49:51] Logged the message, Master [19:50:09] <^demon> sync all the files ;-) [19:51:55] !log reedy synchronized php-1.21wmf3/maintenance/refreshLinks.php [19:52:02] Logged the message, Master [19:53:05] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.450 seconds [19:55:12] New review: Eloquence; "What is this based on? Some wikis use uploads for non-free content, including en. See:" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32925 [19:58:32] !log powercycling tmh1 [19:58:40] Logged the message, Master [19:59:06] New patchset: Reedy; "Revert "If cli mode, show exceptions!"" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32927 [19:59:18] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32927 [20:01:55] !log Running sync-common on tmh1 [20:02:01] Logged the message, Master [20:02:32] RECOVERY - Host tmh1 is UP: PING OK - Packet loss = 0%, RTA = 0.27 ms [20:27:53] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:42:53] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 6.576 seconds [20:53:08] hey... mutante [20:53:15] about the dewikivoyage stuff [20:56:42] DAS IST VERBOTEN [20:57:11] Reedy: huh? [21:01:11] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [21:08:18] ^demon: You're aware of the dewikivoyage mess... don't you? [21:09:04] <^demon> Yes, I was working on trying to fix it earlier. [21:09:31] 10:49 < ^demon> If Hoo logs in now, he *should* get a fresh local [21:09:31] account. If he doesn't, we'll need to muck about in CA some more. [21:09:46] <^demon> Please tell me it worked. [21:09:51] I did that, but my registration date still is 19:14, 10 November 2012 [21:10:18] and I still got 3 edit [21:10:18] s [21:10:51] 20121110191441 | 3 [21:11:46] that's it, yep [21:12:33] <^demon> Well, you've only got 3 edits. [21:12:37] <^demon> http://de.wikivoyage.org/wiki/Spezial:Beitr%C3%A4ge/Hoo_man [21:12:48] mysql:wikiadmin@db37 [centralauth]> select * from localuser where lu_name = 'Hoo man' and lu_wiki = 'dewikivoyage'; [21:12:48] +--------------+---------+-----------------------+--------------------+ [21:12:48] | lu_wiki | lu_name | lu_attached_timestamp | lu_attached_method | [21:12:48] +--------------+---------+-----------------------+--------------------+ [21:12:48] | dewikivoyage | Hoo man | 20121106210502 | login | [21:12:49] +--------------+---------+-----------------------+--------------------+ [21:12:49] yes, that's totally true [21:12:51] 1 row in set (0.00 sec) [21:13:23] Reedy: My idea was to totally delete my global account and remerge [21:13:31] alternative would be to wipe that line, probabl [21:13:31] y [21:13:33] Why bother? [21:13:59] Reedy: huh? [21:14:05] What's the benefit? [21:14:41] So... the accounts (Unger; Hoo man) are separated now [21:14:53] ?! [21:15:13] <^demon> Yes, they are. [21:16:20] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:22:15] !log reedy synchronized wmf-config/CommonSettings.php 'Disable mysql search for wikidata wiki...' [21:22:22] Logged the message, Master [21:26:43] !log reedy synchronized wmf-config/CommonSettings.php 'rv' [21:26:49] Logged the message, Master [21:29:32] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 5.908 seconds [21:44:13] !log deploying squid config, adding wikivoyage to the swift regexp [21:44:19] Logged the message, Master [21:47:14] New patchset: Reedy; "Revert "Disable uploads on wikivoyage. Point to commons"" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/33035 [21:48:12] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/33035 [21:51:20] New patchset: Faidon; "Add Wikivoyage to regexps in upload Varnish config" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/33036 [21:51:51] !log reedy synchronized wmf-config/InitialiseSettings.php [21:51:57] Logged the message, Master [21:52:20] gerrit/jenkins trouble? [21:53:08] triggered manually [21:53:38] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/33036 [21:56:44] New patchset: Reedy; "Cleanup Lucene config, removing old globals and unused code" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/33037 [22:04:27] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:15:51] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 4.406 seconds [22:30:51] PROBLEM - Puppet freshness on analytics1013 is CRITICAL: Puppet has not run in the last 10 hours [22:30:51] PROBLEM - Puppet freshness on analytics1012 is CRITICAL: Puppet has not run in the last 10 hours [22:30:51] PROBLEM - Puppet freshness on analytics1011 is CRITICAL: Puppet has not run in the last 10 hours [22:30:51] PROBLEM - Puppet freshness on analytics1014 is CRITICAL: Puppet has not run in the last 10 hours [22:30:51] PROBLEM - Puppet freshness on analytics1015 is CRITICAL: Puppet has not run in the last 10 hours [22:30:52] PROBLEM - Puppet freshness on analytics1016 is CRITICAL: Puppet has not run in the last 10 hours [22:30:52] PROBLEM - Puppet freshness on analytics1017 is CRITICAL: Puppet has not run in the last 10 hours [22:30:53] PROBLEM - Puppet freshness on analytics1018 is CRITICAL: Puppet has not run in the last 10 hours [22:30:53] PROBLEM - Puppet freshness on analytics1020 is CRITICAL: Puppet has not run in the last 10 hours [22:30:54] PROBLEM - Puppet freshness on analytics1022 is CRITICAL: Puppet has not run in the last 10 hours [22:30:54] PROBLEM - Puppet freshness on analytics1021 is CRITICAL: Puppet has not run in the last 10 hours [22:30:55] PROBLEM - Puppet freshness on analytics1019 is CRITICAL: Puppet has not run in the last 10 hours [22:30:55] PROBLEM - Puppet freshness on db42 is CRITICAL: Puppet has not run in the last 10 hours [22:30:56] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [22:41:45] Is https://gerrit.wikimedia.org/r/#/c/13429/ ever going to be approved? Last updated: September 14th by me. [22:43:35] Krenair: seems like a good request to take up with the rt on duty [22:43:50] mutante / paravoid ^ [22:44:44] there's no duty on the weekends [22:44:51] but since I'm the commiter on that... :) [22:44:59] yeah, I guess I should deploy it [22:45:00] oh, oops. sorry. [22:45:06] just going by the /topic. [22:50:48] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:54:41] do iiiit [22:56:06] "weekend! get a life!" [22:56:07] :P [22:56:34] lol [22:56:42] Probably needs scheduling and announcing [22:59:42] nobody has given it +2 [23:01:12] ..sounds like a good next step [23:01:24] and nobody who gave it +1 said that they were confident that it was correct [23:02:24] Tim-away: i don't think Krenair's request was for deployment; just some action on the patchset, which seems reasonable given that it's been languishing for this long. [23:02:41] at least that's how i took it [23:03:32] sure, but there's only about 10 minutes of work left to do on it, so it doesn't really need to be a "process" [23:03:40] I request that someone schedules it's deployment and announces that [23:03:44] just someone needs to spend 10 minutes on it [23:03:56] Because at the moment it's just sitting in Gerrit gathering dust [23:04:21] Krenair: Which change? [23:04:29] wasn't it announced at the time the new gateway was deployed? [23:04:31] Brooke: 14:41 Is https://gerrit.wikimedia.org/r/#/c/13429/ ever going to be approved? Last updated: September 14th by me. [23:04:45] Oh. [23:05:01] Tim-away, no idea, I probably wasn't around at the time [23:05:39] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.064 seconds [23:05:55] Krenair: There's a deployments schedule on wikitech.wikimedia.org. You just gotta get on there. [23:06:56] Brooke: needs a final review first [23:07:21] It looks like it has sign-offs from Daniel Z. and Roan K. [23:08:34]

and nobody who gave it +1 said that they were confident that it was correct [23:08:54] ... I was under the impression that giving +1 implies that you think it is correct [23:09:09] +1 means you can use a mouse [23:09:31] And saying you think it is correct means you can use a keyboard, what's your point? [23:09:31] mark gave it +1 when it was still broken, didn't he? [23:10:04] no, saying it is correct means you think it is correct [23:10:08] +1 doesn't mean anything [23:10:34] if you really believe that, you ought to say something on the list. if it's true, it's noise and probably pernicious. [23:11:23] in my experience, the more +1's something has, the more likely it is to be broken [23:12:06] because a high number of +1's means that it is big, a lot of people think it is good, but nobody has been brave enough to deploy it while the +1's accumulate [23:12:13] so I'm extra careful reviewing such a revision [23:12:15] that makes sense [23:12:42] I'd be most concerned about weird encoding + redirects. [23:12:55] secure.wikimedia.org/wikipedia/wiki/Foo%3Awhatever [23:13:09] That kind of stuff has a tendency to get trampled on or mangled with Apache redirects, I've found. [23:13:15] I take it there's no easy way to test this in labs first..? [23:14:29] hrm, we could. sec. [23:15:38] New patchset: Dereckson; "(bug 41962) Namespace configuration for bar.wikipedia" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/33043 [23:15:51] I don't think it's that big or that it needs a deployment window [23:15:55] but famous last words [23:16:07] &, :, and / are probably the encoded characters to poke at most closely. [23:16:28] it's secure.wm.org, we're still getting a considerable amount of hits but not that many [23:16:43] Are there any public stats on secure.wm.o usage? [23:16:50] don't think so [23:16:52] And Brooke, I'd be surprised if those caused any issues.. [23:16:57] it's currently being server out of one single box [23:17:08] proxying back directly to appservers [23:17:20] http://meta.wikipedia.org/w/index.php?title=AT%25T [23:17:28] New patchset: Dereckson; "(bug 41962) Namespace configuration for bar.wikipedia" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/33043 [23:17:35] Krenair: http://meta.wikipedia.org/w/index.php?title=AT%25T is the kind of bug I'm thinking of. [23:17:50] New review: Dereckson; "PS2: improving commit message" [operations/mediawiki-config] (master) C: 0; - https://gerrit.wikimedia.org/r/33043 [23:18:25] http://en2.wikipedia.org/w/index.php?title=AT%25T [23:18:34] fwiw, I wouldn't give anyone grief for doing it outside of a deployment window, and I'm usually the biggest stickler about that [23:18:34] I'm not sure anyone really cares about the double-encoding. [23:18:53] labs so soooooooooo slow sometimes [23:18:55] I mean, it's obviously broken, but it's clearly been broken forever. [23:19:01] ori-l: Someone should fix that. [23:19:28] * robla goes back to paying attention to the Gerrit 2.7 roadmap discsussion [23:19:52] robla: where's that happening? [23:19:54] 2.7? [23:20:03] Brooke, yeah I'm not sure this messes around with any encoding [23:20:19] gerrit user summit at Google this weekend. ^demon and I are at it [23:20:21] It's just regexes to redirect from one URL to another [23:20:33] Krenair: It might be fine. Just needs testing. [23:20:45] I just know that these high-level redirects often cause an issue in this area. [23:20:54] robla: cool, bring gerrit swag! [23:21:00] :) [23:21:03] if you wait i'll set up these redirects for kubo.wmflabs.org [23:21:04] Poorly designed coofee mugs, etc. [23:21:10] Damn typos. :-( [23:21:11] and then you can test [23:21:28] I imagine a Gerrit coffee mug would have no openings. [23:21:45] thankfully I didn't laugh *too* loud at that [23:21:53] <^demon> robla: We'll pick up some of the stickers of the new mascot :p [23:22:06] dibs [23:22:10] <^demon> Diffy: the code-review cuckoo. [23:32:28] i can see my keystrokes echo in slow-mo [23:38:27] Krenair: ok, https://piramido.wmflabs.org/ [23:38:29] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:38:32] has the same rewrite config [23:38:38] as your patchset proposes [23:38:56] Brooke: can you help Krenair test it? [23:39:06]