[00:05:01] RECOVERY - NTP on mw60 is OK: NTP OK: Offset -0.005389332771 secs [00:06:22] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [00:06:22] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [00:06:22] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [00:10:41] New patchset: Pyoungmeister; "re-adding mw60 to bits backends" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/32672 [00:14:10] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:14:14] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/32672 [00:27:14] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.731 seconds [00:45:46] Change abandoned: Pyoungmeister; "resubmitting" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/21019 [00:45:55] Change abandoned: Pyoungmeister; "resubmitting" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/28311 [00:55:13] !authdns update to set MX for wikivoyage-old to us [00:55:31] New patchset: Pyoungmeister; "new coredb module." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/32674 [00:55:33] !log authdns update to set MX for wikivoyage-old to us [00:55:39] Logged the message, Master [00:59:41] New patchset: Asher; "For https-only wikis, the redirect from http to https for bits assets should occur in varnish instead of apache, since the apache redirect and mediawiki doesn't vary by protocol. This can result in a redirect loop and assets not loading." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/32676 [01:02:19] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:02:48] New patchset: Pyoungmeister; "new coredb module." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/32674 [01:04:03] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/32674 [01:15:49] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 8.219 seconds [01:22:25] PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours [01:30:17] New patchset: Dzahn; "add redirect for wikivoyage.de to .org" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/32680 [01:30:53] New patchset: SPQRobin; "abusefilter-log-detail right from sysop to autoconfirmed" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32681 [01:38:38] New review: Dzahn; "the question is more on the timing than the content, but feel free to check that too of course:)" [operations/apache-config] (master); V: 0 C: 0; - https://gerrit.wikimedia.org/r/32680 [01:39:31] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 260 seconds [01:40:25] PROBLEM - MySQL Slave Delay on db78 is CRITICAL: CRIT replication delay 315 seconds [01:42:18] New review: Dzahn; "merge with DNS switch or we would be pointing back to old servers until then. not sure if that would..." [operations/apache-config] (master); V: 0 C: 0; - https://gerrit.wikimedia.org/r/32680 [01:50:28] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:51:58] RECOVERY - MySQL Slave Delay on db78 is OK: OK replication delay 5 seconds [02:01:43] PROBLEM - MySQL Slave Delay on db78 is CRITICAL: CRIT replication delay 279 seconds [02:03:40] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 9.939 seconds [02:06:04] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 1 seconds [02:15:58] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 304 seconds [02:30:06] !log LocalisationUpdate completed (1.21wmf3) at Sat Nov 10 02:30:06 UTC 2012 [02:30:14] Logged the message, Master [02:38:01] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:39:31] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.030 seconds [02:40:16] RECOVERY - Puppet freshness on ms-fe2 is OK: puppet ran at Sat Nov 10 02:39:40 UTC 2012 [02:43:43] RECOVERY - Puppet freshness on ms-fe1 is OK: puppet ran at Sat Nov 10 02:43:25 UTC 2012 [02:44:10] RECOVERY - Puppet freshness on ms-be3 is OK: puppet ran at Sat Nov 10 02:43:59 UTC 2012 [03:00:40] RECOVERY - Puppet freshness on ms-fe3 is OK: puppet ran at Sat Nov 10 03:00:14 UTC 2012 [04:00:24] PROBLEM - Host srv278 is DOWN: PING CRITICAL - Packet loss = 100% [04:00:51] RECOVERY - Host srv278 is UP: PING OK - Packet loss = 0%, RTA = 0.30 ms [04:04:27] PROBLEM - Apache HTTP on srv278 is CRITICAL: Connection refused [04:09:51] RECOVERY - MySQL Slave Delay on db78 is OK: OK replication delay 0 seconds [04:16:45] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 4 seconds [04:17:40] New review: Tim Landscheidt; "One more nitpick :-): Please don't mention anything specific to a patchset/the evolution of the chan..." [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/30593 [04:20:12] RECOVERY - Apache HTTP on srv278 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.043 second response time [04:57:06] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [05:18:15] PROBLEM - MySQL Replication Heartbeat on db1042 is CRITICAL: CRIT replication delay 199 seconds [05:18:33] PROBLEM - MySQL Slave Delay on db1042 is CRITICAL: CRIT replication delay 204 seconds [05:23:12] RECOVERY - MySQL Replication Heartbeat on db1042 is OK: OK replication delay 6 seconds [05:23:39] RECOVERY - MySQL Slave Delay on db1042 is OK: OK replication delay 10 seconds [05:44:23] PROBLEM - LVS Lucene on search-pool4.svc.eqiad.wmnet is CRITICAL: Connection timed out [05:46:20] RECOVERY - LVS Lucene on search-pool4.svc.eqiad.wmnet is OK: TCP OK - 9.020 second response time on port 8123 [05:52:20] PROBLEM - Lucene on search1016 is CRITICAL: Connection timed out [05:59:41] PROBLEM - LVS Lucene on search-pool4.svc.eqiad.wmnet is CRITICAL: Connection timed out [06:02:05] New review: Hydriz; "Looks good, provided that those wiki databases *really* exists on s2, else just use addWiki.php with..." [operations/mediawiki-config] (master) C: 1; - https://gerrit.wikimedia.org/r/29344 [06:02:14] RECOVERY - Lucene on search1016 is OK: TCP OK - 9.022 second response time on port 8123 [06:04:12] New review: Hydriz; "Not too sure about whether you like to submit another patch to include wmflabs.org as well?" [operations/mediawiki-config] (master) C: 1; - https://gerrit.wikimedia.org/r/32226 [06:11:05] RECOVERY - LVS Lucene on search-pool4.svc.eqiad.wmnet is OK: TCP OK - 3.021 second response time on port 8123 [06:26:22] PROBLEM - Puppet freshness on analytics1011 is CRITICAL: Puppet has not run in the last 10 hours [06:26:22] PROBLEM - Puppet freshness on analytics1013 is CRITICAL: Puppet has not run in the last 10 hours [06:26:22] PROBLEM - Puppet freshness on analytics1012 is CRITICAL: Puppet has not run in the last 10 hours [06:26:22] PROBLEM - Puppet freshness on analytics1015 is CRITICAL: Puppet has not run in the last 10 hours [06:26:23] PROBLEM - Puppet freshness on analytics1016 is CRITICAL: Puppet has not run in the last 10 hours [06:26:23] PROBLEM - Puppet freshness on analytics1017 is CRITICAL: Puppet has not run in the last 10 hours [06:26:23] PROBLEM - Puppet freshness on analytics1018 is CRITICAL: Puppet has not run in the last 10 hours [06:26:24] PROBLEM - Puppet freshness on analytics1022 is CRITICAL: Puppet has not run in the last 10 hours [06:26:24] PROBLEM - Puppet freshness on analytics1021 is CRITICAL: Puppet has not run in the last 10 hours [06:26:25] PROBLEM - Puppet freshness on analytics1020 is CRITICAL: Puppet has not run in the last 10 hours [06:26:25] PROBLEM - Puppet freshness on analytics1014 is CRITICAL: Puppet has not run in the last 10 hours [06:26:26] PROBLEM - Puppet freshness on analytics1019 is CRITICAL: Puppet has not run in the last 10 hours [06:26:26] PROBLEM - Puppet freshness on db42 is CRITICAL: Puppet has not run in the last 10 hours [06:26:27] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [08:08:35] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:10:05] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.060 seconds [08:36:47] PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours [08:47:21] PROBLEM - Puppet freshness on cp1042 is CRITICAL: Puppet has not run in the last 10 hours [09:54:18] New patchset: Eloquence; "Remove broken link, consolidate GitHub links" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32694 [10:06:21] New patchset: Eloquence; "noc: Remove broken link, consolidate GitHub links" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32694 [10:07:40] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [10:07:40] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [10:07:40] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [11:16:08] New patchset: Reedy; "Moar memory" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32701 [11:18:04] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32701 [11:18:37] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32255 [11:19:18] !log reedy synchronized wmf-config/InitialiseSettings.php [11:19:24] Logged the message, Master [11:23:41] PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours [11:41:41] PROBLEM - MySQL Replication Heartbeat on db1035 is CRITICAL: CRIT replication delay 181 seconds [11:43:20] RECOVERY - MySQL Replication Heartbeat on db1035 is OK: OK replication delay 0 seconds [12:33:09] is there a public copy of squid.conf somewhere? It doesn't seem to be in operations/puppet [12:35:15] Eloquence: Apparently not, I am not sure if it contains private information, but that wasn't the answer I got when I last asked for one. [12:37:08] New patchset: Reedy; "Revert "Enable lucene for wikidatawiki"" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32710 [12:37:46] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32710 [12:39:26] Eloquence: What are you looking for? [12:39:31] It might be in with the deb... [12:39:54] Reedy, just trying to better understand the vary settings of the text squids [12:39:54] New patchset: Reedy; "Add a reason for disabling of Lucene for wikidatawiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32711 [12:40:22] https://gerrit.wikimedia.org/r/gitweb?p=operations/debs/squid.git [12:40:24] Maybe.. [12:40:39] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32711 [12:41:06] Reedy, since I got you .. could I ask you to poke at the interwiki config for wikivoyage :p [12:41:07] !log reedy synchronized wmf-config/CommonSettings.php [12:41:13] Logged the message, Master [12:45:17] Eloquence: have you also decided something for the "shortened" interwiki? [12:45:32] [that's way less urgent] [12:46:01] Nemo_bis, I thought they were going to go with [[y:]] [12:46:20] I personally don't get the need for one-letter prefixes and would prefer something like [[voy:]] [12:46:29] Eloquence: I agree [12:46:33] heh, I thought this was already resolved [12:46:36] we already have wikt: [12:47:59] y: is just an arbitrarily chosen letter, might as well be wiktionarY [12:51:58] anyway, the relevant changes seem to be https://gerrit.wikimedia.org/r/#/c/31202/ and https://gerrit.wikimedia.org/r/#/c/28241/3 , but it looks like the change is either not deployed yet or the interwiki map not yet rebuilt, or both [12:52:26] the broken interlanguage links are the bigger issue right now [12:53:15] I rebuilt the iw map last night [12:53:16] I think [12:53:20] OH [12:53:23] lol, no I didn't [12:53:26] let me fix that [12:54:24] bloody symlinks [12:55:01] Reedy, let's change the prefix to [[voy:]] before we do it [12:55:03] !log reedy synchronized php-1.21wmf3/cache/interwiki.cdb [12:55:07] oh well [12:55:10] Logged the message, Master [12:55:19] If you want to update the page on meta, I can do it again [12:56:00] I don't think those short prefixes come from the interwiki map [12:56:01] on meta [12:56:04] see https://gerrit.wikimedia.org/r/#/c/31202/3/dumpInterwiki.php [12:56:14] :( [12:57:40] well, that appears to have fixed the interlanguage link issue at least, yay. [12:57:44] heh [12:58:03] and [[y:]] is now live [12:58:26] let's fix it before it goes viral :p [12:58:29] !log reedy synchronized php-1.21wmf3/includes/ [12:58:35] Logged the message, Master [12:58:54] apergos: ^^ eswiki dump should be ok now... [12:58:59] !log reedy synchronized php-1.21wmf3/includes/resourceloader/ResourceLoaderWikiModule.php [12:59:05] Logged the message, Master [13:03:42] PROBLEM - Memcached on virt0 is CRITICAL: Connection refused [13:10:09] RECOVERY - Memcached on virt0 is OK: TCP OK - 0.004 second response time on port 11000 [13:17:00] Reedy: https://www.mediawiki.org/wiki/Developer_access#User:Ruud_Koot [13:23:02] !log reedy synchronized php-1.21wmf3/includes/Revision.php [13:23:08] Logged the message, Master [13:27:11] !log Pointing php symlink to 1.21wmf3 [13:27:16] Logged the message, Master [13:31:26] !log Deleting 1.21wmf2 l10n cache [13:31:31] Logged the message, Master [13:32:33] New patchset: Reedy; "Point php to 1.21wmf3" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32721 [13:32:49] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32721 [14:05:05] !log reedy synchronized php/cache/interwiki.cdb 'Updating interwiki cache' [14:05:11] Logged the message, Master [14:58:06] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [15:35:33] !log reedy synchronized php-1.21wmf3/includes/Article.php [15:35:39] Logged the message, Master [15:44:06] PROBLEM - Host srv278 is DOWN: PING CRITICAL - Packet loss = 100% [15:44:29] !log reedy synchronized wmf-config/InitialiseSettings.php 'Disable MF for wikivoyage' [15:44:34] Logged the message, Master [15:46:21] RECOVERY - Host srv278 is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms [15:50:24] PROBLEM - Apache HTTP on srv278 is CRITICAL: Connection refused [15:53:47] New patchset: Reedy; "Disable MobileFrontend on wikidatawikis and wikivyoage due to lack of domain setup" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32734 [15:56:22] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32734 [15:57:00] !log reedy synchronized wmf-config/InitialiseSettings.php [15:57:06] Logged the message, Master [15:58:48] RECOVERY - Apache HTTP on srv278 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.052 second response time [16:26:44] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32226 [16:27:09] PROBLEM - Puppet freshness on analytics1011 is CRITICAL: Puppet has not run in the last 10 hours [16:27:09] PROBLEM - Puppet freshness on analytics1012 is CRITICAL: Puppet has not run in the last 10 hours [16:27:09] PROBLEM - Puppet freshness on analytics1015 is CRITICAL: Puppet has not run in the last 10 hours [16:27:09] PROBLEM - Puppet freshness on analytics1016 is CRITICAL: Puppet has not run in the last 10 hours [16:27:09] PROBLEM - Puppet freshness on analytics1013 is CRITICAL: Puppet has not run in the last 10 hours [16:27:10] PROBLEM - Puppet freshness on analytics1014 is CRITICAL: Puppet has not run in the last 10 hours [16:27:10] PROBLEM - Puppet freshness on analytics1018 is CRITICAL: Puppet has not run in the last 10 hours [16:27:11] PROBLEM - Puppet freshness on analytics1017 is CRITICAL: Puppet has not run in the last 10 hours [16:27:11] PROBLEM - Puppet freshness on analytics1019 is CRITICAL: Puppet has not run in the last 10 hours [16:27:12] PROBLEM - Puppet freshness on analytics1020 is CRITICAL: Puppet has not run in the last 10 hours [16:27:12] PROBLEM - Puppet freshness on analytics1022 is CRITICAL: Puppet has not run in the last 10 hours [16:27:13] PROBLEM - Puppet freshness on analytics1021 is CRITICAL: Puppet has not run in the last 10 hours [16:27:13] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [16:27:14] PROBLEM - Puppet freshness on db42 is CRITICAL: Puppet has not run in the last 10 hours [16:28:35] !log authdns-update for wikivoyage.org - project launch! wee [16:28:40] Logged the message, Master [16:29:18] !log reedy synchronized wmf-config/InitialiseSettings.php [16:29:24] Logged the message, Master [16:31:13] New review: Hydriz; "No, that change is unrelated to this, this concerns the prefix used by the Incubator, which we have ..." [operations/mediawiki-config] (master) C: 1; - https://gerrit.wikimedia.org/r/32258 [16:32:49] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32258 [16:35:02] New review: Dzahn; "we have a lift-off" [operations/apache-config] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/32680 [16:35:03] Change merged: Dzahn; [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/32680 [16:35:48] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/31998 [16:36:26] !log reedy synchronized images/sul/wikivoyage.png [16:36:32] Logged the message, Master [16:37:19] !log reedy synchronized wmf-config/ [16:37:25] Logged the message, Master [16:38:27] !log reedy synchronized wmf-config/ [16:38:33] Logged the message, Master [16:39:10] !log sync-apache and graceful to redirect wikivoyage.de [16:39:16] Logged the message, Master [16:40:27] dzahn is doing a graceful restart of all apaches [16:41:08] !log dzahn gracefulled all apaches [16:41:14] Logged the message, Master [16:59:33] New patchset: Reedy; "Add wikivoyage portal to extract2" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32740 [16:59:58] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32740 [17:00:54] !log reedy synchronized extract2.php [17:01:02] Logged the message, Master [17:01:20] New patchset: Reedy; "Kill wikidata docroot" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32741 [17:03:32] New review: Reedy; "Un-needed" [operations/mediawiki-config] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/32741 [17:03:32] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32741 [17:05:45] New patchset: Reedy; "Add config for www.wikivoyage.org portal" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/32742 [17:10:37] New patchset: Reedy; "Add www.wikivoyage.org docroot" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32743 [17:11:31] New patchset: Dzahn; "main.conf - turn wikivoyage ServerAlias into * again vs. single languages" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/32744 [17:11:39] New patchset: Reedy; "Add www.wikivoyage.org docroot" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32743 [17:12:30] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32743 [17:14:13] Change merged: Dzahn; [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/32744 [17:16:11] dzahn is doing a graceful restart of all apaches [17:16:48] !log dzahn gracefulled all apaches [17:16:54] Logged the message, Master [17:27:32] !log reedy synchronized images/sul/ [17:27:38] Logged the message, Master [17:29:11] New patchset: CSteipp; "Fix www -> en bug" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/32746 [17:30:25] Change merged: Dzahn; [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/32746 [17:35:01] Change merged: Dzahn; [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/32742 [17:36:04] dzahn is doing a graceful restart of all apaches [17:36:45] !log dzahn gracefulled all apaches [17:36:50] Logged the message, Master [17:40:31] !log reedy synchronized extract2.php [17:40:37] Logged the message, Master [17:41:46] New patchset: CSteipp; "Fix redirect to old-style /de and /it urls" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/32748 [17:42:47] New patchset: CSteipp; "Fix redirect to old-style /de and /it urls" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/32748 [17:45:13] Change merged: Dzahn; [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/32748 [17:48:49] dzahn is doing a graceful restart of all apaches [17:51:34] dzahn is doing a graceful restart of all apaches [17:52:11] !log dzahn gracefulled all apaches [17:52:17] Logged the message, Master [18:02:47] !log reedy synchronized php-1.21wmf3/cache/interwiki.cdb 'Updating interwiki cache' [18:02:53] Logged the message, Master [18:16:48] !log reedy synchronized php/cache/interwiki.cdb 'Updating interwiki cache' [18:16:54] Logged the message, Master [18:21:43] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:22:10] New patchset: Dzahn; "wikivoyage - disable "de" to investigate data consistency issues" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/32753 [18:23:13] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.041 seconds [18:23:39] New patchset: Dzahn; "wikivoyage - disable "de" to investigate data consistency issues" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/32753 [18:37:46] PROBLEM - Puppet freshness on db62 is CRITICAL: Puppet has not run in the last 10 hours [18:50:02] PROBLEM - Puppet freshness on cp1042 is CRITICAL: Puppet has not run in the last 10 hours [18:53:04] New patchset: Reedy; "Fix wikivoyage typo" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32759 [18:54:17] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32759 [18:55:15] !log reedy synchronized wmf-config/InitialiseSettings.php 'Fix wikivoyage sul image location' [18:55:21] Logged the message, Master [18:57:18] !log reedy synchronized php-1.21wmf3/extensions/TimedMediaHandler/ [18:57:23] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:57:24] Logged the message, Master [18:58:53] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.720 seconds [19:21:27] New review: Nemo bis; "Please create a bug on this first." [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/32681 [19:22:18] Vito_away: ^ [19:22:28] fyi [19:23:15] eh infatti [19:23:25] quant'è bello 'sto vizio di bypassare i setting delle wiki in allegria [19:23:44] be', è solo una proposta :) [19:24:11] la cosa piú facile è cambiarlo solo per i wiki a cui è stata attivato senza chiedere [19:28:08] New patchset: Alex Monk; "(bug 41960) Disable news: url protocol on dewikivoyage as it interferes with News namespace" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32763 [19:29:21] New review: Demon; "Is this really a bug? What happens if you tried to make an HTTP: namespace?" [operations/mediawiki-config] (master); V: 0 C: 0; - https://gerrit.wikimedia.org/r/32763 [19:31:40] ^demon, that bug was filed by Eloquence .... [19:31:58] <^demon> Ok, fair enough. [19:32:14] go to dewikivoyage and try creating a page with [[News:2012]] [19:32:16] That page exists [19:32:39] But it's rendered as [[1]], with the [1] being an external link to "news:2012" [19:32:44] Reedy: have you fixed the interwiki? [19:32:44] <^demon> Ah. [19:32:54] Erik asked to chenge it to voy: as soon as possible [19:32:58] Yes [19:33:02] We're discussing [[news]] [19:33:03] oh nice [19:33:09] in #wikimedia-wikivoyage [19:33:24] It's being parsed as if it's the news protocol [19:33:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:33:37] <^demon> Krenair: Interesting. I never knew there was a collision between $wgUrlProtocols and namespaces. [19:33:42] as in, news group [19:33:51] Eloquence: ^^ $wgUrlProtocols [19:34:47] ^demon, neither did I until I tested that bug :) [19:34:55] Reedy: you've not run dumpiInterwiki though? [19:35:01] Nemo_bis: I have [19:35:02] Multiple times [19:35:06] :< [19:35:16] doesn't work [19:35:39] i've done it 2 or 3 times at least today [19:35:58] New review: Reedy; "Yes it is." [operations/mediawiki-config] (master); V: 0 C: -2; - https://gerrit.wikimedia.org/r/32763 [19:36:05] ^demon: this protocol has been added in some recent release, I think it should be documented [19:36:08] It's almost likes it's being cached somewhere [19:36:19] (or the behaviour changed for some other reason) [19:37:10] <^demon> news:// was added ages ago. [19:37:22] <^demon> I don't think we've adjusted $wgUrlProtocols in a couple of releases. [19:37:31] how many ages? [19:37:44] the other bug I linked affected a wiki which updated from 1.18 to 1.20 or so [19:37:48] not from 1.13 [19:39:38] Reedy, what would be right then? [19:39:52] [19:32:36] so let's just disable news on dewv [19:39:57] Or rename the extension [19:40:21] *namespace [19:40:55] Change abandoned: Alex Monk; "(no reason)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32763 [19:43:35] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 8.526 seconds [19:45:05] New patchset: Reedy; "Disable news namespace on dewiki per Erik" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32767 [19:47:17] New review: Eloquence; "We shouldn't disable the namespace but the protocol handler til a better solution is found." [operations/mediawiki-config] (master) C: -1; - https://gerrit.wikimedia.org/r/32767 [19:54:06] New review: Eloquence; "I suggest we" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32763 [20:05:27] New patchset: Reedy; "Rename dewikivoyage news NS" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32772 [20:06:12] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32772 [20:06:46] !log reedy synchronized wmf-config/InitialiseSettings.php [20:06:52] Logged the message, Master [20:08:20] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [20:08:20] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [20:08:20] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [20:10:00] !log reedy synchronized php-1.21wmf3/cache/interwiki.cdb [20:10:07] Logged the message, Master [20:10:48] Damn it [20:10:53] Now y has completely disappeared :/ [20:10:57] !log reedy synchronized php-1.21wmf3/cache/interwiki.cdb [20:11:03] Logged the message, Master [20:11:17] lol [20:11:37] New patchset: Asher; "attempting to fix first-index-build for new wikis - "build-new" in this script is not it" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/32773 [20:12:23] !log attempting to build a lucene index for enwikivoyage [20:12:30] Logged the message, Master [20:13:46] > var_dump( Interwiki::fetch( 'voy' ) ); [20:13:46] object(Interwiki)#30 (6) { [20:13:46] ["mPrefix":protected]=> [20:13:46] string(3) "voy" [20:13:48] ["mURL":protected]=> [20:13:51] string(27) "//en.wikivoyage.org/wiki/$1" [20:13:57] Hey binasher [20:14:10] hey [20:18:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:21:17] Nemo_bis: FIXED [20:33:35] Change abandoned: Reedy; "(no reason)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32767 [20:34:06] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/32773 [20:34:17] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.034 seconds [20:35:09] Reedy: BRUTE FORCE? [20:36:35] !log indexing wikidatawiki [20:36:41] Logged the message, Master [20:36:48] binasher: ooh [20:37:23] it probably won't get picked up by the search nodes til tomorrow tho (if it works) [20:40:07] http://wikitech.wikimedia.org/view/Lucene#Adding_new_wikis [20:40:14] It looks like you can force it to index now... [20:41:11] that documentation is totally wrong [20:42:00] Ah [20:42:34] the "build-new" function of the script invokes the incremental indexer, which bombs if there isn't an index present. fail! [20:43:22] ... [20:43:59] private wikis don't use incremental indexing though, so i stole the bits for building, and it's working.. hopefully incrementals will still work after [20:51:21] !log indexing {de,fr,it,nl,ru,sv}wikivoyage [20:51:27] Logged the message, Master [21:03:53] binasher: Can we get away with leaving mysql search on overnight? [21:05:23] If not, I can revert it in an hour or so. Or you can revert my gerrit revert, and push it [21:06:41] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:06:50] will not having mwsearch interfere with incremental indexing? [21:07:00] Oh, errm [21:07:37] there are the searchupdatehooks [21:07:50] OAI is running anyway [21:08:00] Meh, it's getting late for europeans [21:08:08] I suspect we can get away with doing it overnight ;) [21:08:53] I've gotta go [21:08:59] I'll be back in 90 minutes or so I guess [21:09:02] i'm fine either way.. but lsearchd better push out wikidata in the morning! [21:09:19] ok i'm going to go too, ttyl [21:21:23] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.018 seconds [21:25:17] PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours [21:46:46] New patchset: Ori.livneh; "(Untested) Declare rules for syncing Git in rsync filter file" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/31146 [21:54:32] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:01:03] do we have an apache with the clock wrong? [22:02:04] !ops [22:02:34] There's some script to check, I think. Mark has run it previously. [22:02:52] Are you seeing edits with bad timestamps or something? [22:03:11] yep, edits done on 11 november http://es.wikipedia.org/wiki/Especial:CambiosRecientes [22:03:54] quite bad [22:05:50] should show NTP "criticals" in Nagios if it were, dont see any on srv's just on some brandnew analytics hosts [22:06:13] well, http://es.wikipedia.org/w/index.php?title=Quinto_Lutacio_C%C3%A1tulo&curid=3658877&diff=61272982&oldid=45834409 should not be possible [22:07:00] * Platonides laps itself [22:07:30] I had my timezone preferences forced in +2 [22:07:51] *slaps [22:09:28] heh [22:09:32] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 2.971 seconds [22:42:50] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:44:25] !log DNS - raise wikivoyage.org TTL back to regular 3600 (1h) [22:44:31] Logged the message, Master [22:45:26] and logging off, bye for now [22:57:41] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.037 seconds [23:25:00] !log reedy synchronized php-1.21wmf3/extensions/CreditsSource/ [23:25:06] Logged the message, Master [23:30:12] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:43:15] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 7.204 seconds [23:58:38] New patchset: Reedy; "Change RelatedSites for nlwikivoyage" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32843 [23:59:07] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/32843