[00:05:55] !log reedy synchronized php/cache/interwiki-pr.cdb 'Updating interwiki cache' [00:05:57] Logged the message, Master [00:07:37] !log reedy synchronized php/cache/interwiki-pr.cdb 'Updating interwiki cache' [00:07:38] Logged the message, Master [00:08:00] Dmcdevit: you could file a chrome bug [00:09:24] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:13:51] https://it.wikipedia.org/wiki/Speciale:UltimeModifiche <-- there's something wrong with interwikis [00:14:07] In spanish project there's the same problem :( [00:15:23] probably everywhere, in fact (same pb en wp-FR) [00:16:00] yep [00:16:10] I see an interwiki cache purge [00:16:12] mmm.... in es.wikipedia and es.wikinews :( [00:20:12] New patchset: Bhartshorne; "changed health check URL for swift's lvs to something small that I know exists" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2159 [00:20:33] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2159 [00:20:34] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2159 [00:21:00] nl-wiki has also problems with interwiki's [00:21:12] Hey. [00:21:25] all interwiki's are shown as red links at the bottom of the page instead of the sidebar [00:21:26] Looks like all interlanguage links are broken on Wikipedia. [00:21:51] same at commons and dewp ;) [00:21:58] also zhwp :-p [00:22:02] same on pl wiki [00:22:05] ...And fiwp and svwp. [00:22:24] so much like everywhere... [00:22:31] Reedy: noticed? [00:22:41] is there a def awake, alive and kicking? [00:22:50] Whaaaa [00:22:51] example: http://commons.wikimedia.org/wiki/Category:Pedersen_bicycles [00:23:45] isn't that awesome [00:24:03] es.wp and es.wn too :( [00:24:11] it'll be the same everywhere [00:24:12] Reedy: yes, you made that very pretty! :D [00:24:22] yes, interwikis are simply read as internal (red) links [00:24:34] Some work, some don't [00:24:36] How useful [00:25:01] mystery :D [00:25:12] <[MarkW]> i quite fancy it. Finally some red links. [00:25:32] a new fresh style [00:25:59] is someone investigating the broken interwiki? [00:26:03] Yes [00:26:06] thx [00:26:21] clever [00:26:30] thx, that was my next question [00:26:38] here is a screenshot of RC in es.wikipedia: http://img838.imageshack.us/img838/797/seleccin001p.png [00:26:46] :) [00:26:49] <- hazarding a guess that it broke with the interwiki map update [00:26:56] Yes [00:26:56] oh, very interesting, Superzerocool :D [00:26:57] It's broken [00:27:12] I will tell meta where is it getting a few whines [00:27:39] Saibo: yep... [00:28:18] I was just going to mention those [00:28:19] I wonder [00:28:39] who breakked it? [00:28:52] Me [00:28:53] pir^2: someone who hasn't fun now [00:29:14] reedy, thanks for the other fixes with protocol links [00:29:24] * KFP gathers a mob armed with pitchforks and torches. [00:29:53] KFP: that is ideal to chase Jimbo after :p [00:29:55] * sDrewth jopes that they are LED torches [00:29:57] Someone messed up up the interwiki thing [00:29:58] hopes [00:30:01] * Bsadowski1 blames Reedy [00:30:14] Bsadowski1: shh! - he needs to work [00:30:22] -up [00:30:31] I won't bother asking why we don't version control this [00:30:34] hmm, jopes = jokingly hopes [00:30:35] reedy synchronized php/cache/interwiki-pr.cdb 'Updating interwiki cache' [00:30:37] D: [00:31:25] Reedy: life would be no fun without surprises [00:31:34] sDrewth: you mean relative URL fixes? some month ago I checked all URLs - there are new URLs which are https capable? [00:31:35] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:32:30] redlink interwikis look weird [00:32:43] pir^2: Yep :P [00:33:10] what happened to interwikis? [00:33:22] !log reedy synchronized php/cache/interwiki-pr.cdb 'Updating interwiki cache' [00:33:23] Logged the message, Master [00:33:32] Fixed [00:33:34] malafaya: They're working on it, see ^ [00:33:35] malafaya: They got broken. [00:33:37] * pir^2 sees whether there's already a VPT discussion about it [00:33:43] ok :) [00:33:46] yep [00:33:50] malafaya: Apparently, fixed too. [00:33:50] works again [00:33:53] i was too quick :) [00:33:55] "Is it just me or has something gone really weird with interwiki links? Jenks24 (talk) 00:16, 1 February 2012 (UTC)" [00:33:57] works again =) [00:34:06] malafaya: topic ^^^ [00:34:19] okay, all back to work now! [00:36:24] Does anybody else viewing http://en.wikinews.org/wiki/Special:RecentChanges see redlinked interwikis near the top? [00:36:38] Force refresh [00:36:38] purge your page [00:36:39] B-R-S: They were broken, but now fixed. [00:36:40] should be fixed now [00:36:41] B-R-S: THey just fixed that. [00:36:47] * sDrewth giggles [00:36:53] Damn, you guys are too fast :p [00:37:06] O.O Lots of people paying attention to this channel, I see :) [00:37:08] It was actually fixed 4 minutes ago ;) [00:37:18] B-R-S gets dumped in the surf break [00:37:22] Reedy: thank you :-) [00:38:32] O_O [00:40:49] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 9.078 seconds [00:42:59] PROBLEM - Disk space on srv220 is CRITICAL: DISK CRITICAL - free space: / 260 MB (3% inode=60%): /var/lib/ureadahead/debugfs 260 MB (3% inode=60%): [00:54:10] PROBLEM - Disk space on srv221 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=60%): /var/lib/ureadahead/debugfs 0 MB (0% inode=60%): [00:54:11] New patchset: Bhartshorne; "adding the IP for ms-fe to the lvs node definition so it will listen on that IP" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2160 [00:54:41] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2160 [00:54:42] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2160 [01:05:19] RECOVERY - Disk space on srv220 is OK: DISK OK [01:05:19] RECOVERY - Disk space on srv221 is OK: DISK OK [01:07:29] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 0.024 seconds [01:34:39] RECOVERY - Disk space on ms-fe1 is OK: DISK OK [01:38:56] New patchset: Ottomata; "moving pipeline files into pipeline/ directory. This should be a standalone python pacakge. Launching and parallelizing individual pipelines might go in here later" [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2161 [01:38:58] New patchset: Ottomata; "Rearranging test and file structures, organizing things a bit" [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2162 [01:40:41] New review: Diederik; "Ok." [analytics/reportcard] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2161 [01:40:42] Change merged: Diederik; [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2161 [01:41:24] New review: Diederik; "Ok." [analytics/reportcard] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2162 [01:41:25] Change merged: Diederik; [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2162 [01:43:19] New patchset: Ottomata; "Adding reorganized files. I should have added these with the last commit." [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2163 [01:45:47] New review: Diederik; "Ok." [analytics/reportcard] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2163 [01:45:47] Change merged: Diederik; [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2163 [01:50:19] zzzz =_= [01:59:43] Hello. I need help. [02:00:24] Hello sweeks. What's up? [02:00:46] I was redirected here from the main help chat. [02:01:18] I have a local Wiki and I would like to implement the following behavior: Trigger a Windows Explorer window to open and browse to a local "\\path". [02:01:47] I have been searching for a method to implement this (i.e. javascript -> this is no longer allows for such behavior). [02:02:00] sweeks: Trigger from what? [02:02:00] The behavior is similar to clicking a html link and a web browse opening. [02:03:11] sweeks: Also, not all wiki users use Windows. There are Mac and Linux users also, as well as others. [02:03:22] Clicking on the text, a link. For example the action text "AAC Codec" would open a Windows Explorer window to some path, "\\networkshare\AAC" [02:03:45] I understand the interplatform concerns - we are considering Windows users only. [02:05:26] sweeks: Since this is about a local wiki, I'm afraid I'll have to redirect you again. [02:06:15] sweeks: #mediawiki deals with the software that runs Wikipedia and other Wikimedia projects. This channel is specific to WMF projects. [02:06:21] !log LocalisationUpdate completed (1.18) at Wed Feb 1 02:06:20 UTC 2012 [02:06:23] Logged the message, Master [02:07:22] PROBLEM - Disk space on ms-fe1 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [02:10:42] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 1068s [02:23:02] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:26:42] PROBLEM - MySQL replication status on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 2028s [02:38:02] RECOVERY - MySQL replication status on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 8s [02:44:22] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 30s [02:49:32] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:58:02] Trying to log in to https://lists.wikimedia.org/mailman/admindb/wikimedia-dc [02:58:06] It's taking forever. [02:59:02] PROBLEM - Disk space on srv224 is CRITICAL: DISK CRITICAL - free space: / 144 MB (2% inode=60%): /var/lib/ureadahead/debugfs 144 MB (2% inode=60%): [02:59:12] PROBLEM - Disk space on srv220 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=60%): /var/lib/ureadahead/debugfs 0 MB (0% inode=60%): [02:59:13] PROBLEM - Disk space on srv222 is CRITICAL: DISK CRITICAL - free space: / 199 MB (2% inode=60%): /var/lib/ureadahead/debugfs 199 MB (2% inode=60%): [02:59:43] PROBLEM - Disk space on srv219 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=60%): /var/lib/ureadahead/debugfs 0 MB (0% inode=60%): [03:00:32] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 9.717 seconds [03:01:56] Chrome is returning "Error 324 (net::ERR_EMPTY_RESPONSE): The server closed the connection without sending any data." [03:09:13] New patchset: Diederik; "Initial commit, feedback Catrope incorporated, feedback Tim Starling (3x) incorporated" [analytics/udp-filters] (master) - https://gerrit.wikimedia.org/r/2142 [03:10:12] RECOVERY - Disk space on srv224 is OK: DISK OK [03:10:22] RECOVERY - Disk space on srv220 is OK: DISK OK [03:10:22] RECOVERY - Disk space on srv222 is OK: DISK OK [03:11:12] RECOVERY - Disk space on srv219 is OK: DISK OK [03:14:15] New review: Tim Starling; "(no comment)" [analytics/udp-filters] (master); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2142 [03:18:42] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 8.261 seconds [03:56:32] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:17:32] PROBLEM - Disk space on srv223 is CRITICAL: DISK CRITICAL - free space: / 140 MB (1% inode=60%): /var/lib/ureadahead/debugfs 140 MB (1% inode=60%): [04:18:42] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 8.211 seconds [04:20:02] RECOVERY - Disk space on es1004 is OK: DISK OK [04:22:22] RECOVERY - MySQL disk space on es1004 is OK: DISK OK [04:28:42] RECOVERY - Disk space on srv223 is OK: DISK OK [04:44:12] PROBLEM - Puppet freshness on lvs1003 is CRITICAL: Puppet has not run in the last 10 hours [04:44:13] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Puppet has not run in the last 10 hours [04:48:02] PROBLEM - MySQL slave status on es1004 is CRITICAL: CRITICAL: Slave running: expected Yes, got No [04:51:02] PROBLEM - Disk space on srv221 is CRITICAL: DISK CRITICAL - free space: / 258 MB (3% inode=60%): /var/lib/ureadahead/debugfs 258 MB (3% inode=60%): [05:02:12] RECOVERY - Disk space on srv221 is OK: DISK OK [05:03:49] Hi need help on enabling RSS or atom feeds on Telugu wikipedia, I am a bureaucrat on that. [05:09:31] JeLuF ^ [05:26:37] Hi need help on enabling RSS or atom feeds on Telugu wikipedia, I am a bureaucrat on that. [05:46:46] Hi Is it possible for sysops to modify Localsettings.php [05:57:01] arjunaraoc: no a bug needs to be submitted into bugzilla requesting the featture be turned on [05:57:26] thx p858snake|l [05:57:29] but RSS/Atom should already be enabled cluster wise [05:57:49] unless you are talking about the new FeatureFeeds feature [05:58:59] I checked and found the atom feed. But the feed is not respecting my preferences. I filed a bug one year back https://bugzilla.wikimedia.org/show_bug.cgi?id=23804 but got no response [06:00:41] p858snake|l: do you have any advice [06:42:31] PROBLEM - Puppet freshness on knsq9 is CRITICAL: Puppet has not run in the last 10 hours [07:10:21] PROBLEM - Disk space on srv223 is CRITICAL: DISK CRITICAL - free space: / 198 MB (2% inode=60%): /var/lib/ureadahead/debugfs 198 MB (2% inode=60%): [07:21:41] RECOVERY - Disk space on srv223 is OK: DISK OK [07:32:51] PROBLEM - Disk space on srv220 is CRITICAL: DISK CRITICAL - free space: / 91 MB (1% inode=60%): /var/lib/ureadahead/debugfs 91 MB (1% inode=60%): [07:44:01] RECOVERY - Disk space on srv220 is OK: DISK OK [07:56:30] jeremyb: it was unrelated to ekrem itself, turned out to be a problem on a puppetmaster. adding monitoring for that [08:02:07] New patchset: Dzahn; "add https monitoring for puppetmaster on stafford (RT 2361)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2164 [08:18:29] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2164 [08:18:29] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2164 [08:29:01] PROBLEM - Disk space on srv219 is CRITICAL: DISK CRITICAL - free space: / 196 MB (2% inode=60%): /var/lib/ureadahead/debugfs 196 MB (2% inode=60%): [08:47:28] PROBLEM - Puppet freshness on mw8 is CRITICAL: Puppet has not run in the last 10 hours [08:52:58] RECOVERY - Disk space on srv219 is OK: DISK OK [08:56:26] New patchset: Dzahn; "adding the puppetmaster http(s) monitoring to sockpuppet" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2165 [08:56:54] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2165 [08:57:23] New review: Dzahn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2165 [08:57:23] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2165 [09:27:40] New review: Dzahn; "did somebody else work on this meanwhile? at least re: the duplicate checks it looks a lot better th..." [operations/puppet] (production); V: 0 C: 1; - https://gerrit.wikimedia.org/r/1726 [09:56:11] PROBLEM - Disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 404752 MB (3% inode=99%): [09:57:01] PROBLEM - MySQL disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 401042 MB (3% inode=99%): [10:13:41] RECOVERY - MySQL Slave Delay on db42 is OK: OK replication delay 0 seconds [10:21:01] RECOVERY - MySQL Replication Heartbeat on db42 is OK: OK replication delay 0 seconds [11:18:01] what's "ersch".wikimedia wiki for? [11:20:51] RECOVERY - MySQL slave status on es1004 is OK: OK: [12:27:32] New review: Mark Bergsma; "Daniel, can you please move this out of site.pp and put it where it belongs, i.e. puppetmaster.pp?" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2164 [12:38:54] New review: Dzahn; "sure, will do. makes more sense to put in service class" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2164 [12:42:12] PROBLEM - Host db1018 is DOWN: PING CRITICAL - Packet loss = 100% [12:44:31] New patchset: Dzahn; "move puppetmaster monitoring out of site.pp into puppetmaster.pp" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2166 [12:47:08] New patchset: Dzahn; "move puppetmaster monitoring out of site.pp into puppetmaster.pp" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2166 [12:47:24] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2166 [12:55:11] RECOVERY - Host ms-fe.pmtpa.wmnet is UP: PING OK - Packet loss = 0%, RTA = 0.24 ms [13:27:41] !log reedy synchronized wmf-config/CommonSettings.php 'Only use the protocol relative interwiki cdb' [13:27:43] Logged the message, Master [13:30:23] !log reedy synchronized wmf-config/CommonSettings.php 'Remove some of the wmgHTTPSExperiment related conditionals' [13:30:24] Logged the message, Master [13:35:32] !log reedy synchronized wmf-config/CommonSettings.php 'Tidy up cache epoch code' [13:35:34] Logged the message, Master [13:36:47] !log reedy synchronized wmf-config/InitialiseSettings.php 'Kill wmgHTTPSExperiment' [13:36:48] Logged the message, Master [13:41:51] !log reedy synchronized php/cache/interwiki.cdb 'Updating interwiki cache copying protocol relative over interwiki.cdb' [13:41:53] Logged the message, Master [13:44:34] !log reedy synchronized wmf-config/InitialiseSettings.php 'Point wgInterwikiCache at interwiki.cdb' [13:44:35] Logged the message, Master [14:01:41] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:25:01] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:30:38] !log shutting down "oldusability" linode instance [14:30:39] Logged the message, Master [14:45:36] !log running authdns-update to remove oldusability [14:45:38] Logged the message, Master [14:54:51] PROBLEM - Puppet freshness on lvs1003 is CRITICAL: Puppet has not run in the last 10 hours [14:54:51] PROBLEM - Puppet freshness on lvs1006 is CRITICAL: Puppet has not run in the last 10 hours [14:58:41] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 7.359 seconds [15:02:37] New patchset: Mark Bergsma; "Simplify lvs_realserver_ips configuration in role/cache.pp" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2167 [15:02:53] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2167 [15:03:12] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2167 [15:03:13] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2167 [15:07:01] New patchset: Mark Bergsma; "Test if puppet can reference hashes multiple levels deep when using parentheses" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2168 [15:07:17] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/2168 [15:08:14] New patchset: Mark Bergsma; "Test if puppet can reference hashes multiple levels deep when using parentheses" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2168 [15:08:31] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/2168 [15:12:00] New patchset: Mark Bergsma; "Test if puppet can reference hashes multiple levels deep when using parentheses" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2168 [15:13:37] New patchset: Mark Bergsma; "Test if puppet can reference hashes multiple levels deep when not using quotes" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2168 [15:13:53] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2168 [15:14:43] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2168 [15:14:44] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2168 [15:19:18] New patchset: Mark Bergsma; "Retab squid.pp" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2169 [15:19:35] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2169 [15:21:42] New patchset: Mark Bergsma; "Fix modes, simplify file definitions" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2170 [15:21:59] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2170 [15:23:41] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2169 [15:23:41] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2169 [15:23:59] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2170 [15:23:59] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2170 [15:32:51] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:45:00] !log reedy synchronized php/cache/interwiki.cdb 'Try a quietened dumpInterwiki script' [15:45:01] Logged the message, Master [15:48:52] New patchset: Mark Bergsma; "Remove usage of old wikimedia-task-squid package, do everything in Puppet" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2171 [15:49:08] New review: gerrit2; "Change did not pass lint check. You will need to send an amended patchset for this (see: https://lab..." [operations/puppet] (production); V: -1 - https://gerrit.wikimedia.org/r/2171 [15:49:39] New patchset: Mark Bergsma; "Remove usage of old wikimedia-task-squid package, do everything in Puppet" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2171 [15:49:55] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2171 [15:50:29] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2171 [15:50:30] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2171 [15:52:26] New patchset: Mark Bergsma; "xfsprogs is installed on all servers, in base.pp" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2172 [15:52:32] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2172 [15:52:38] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2172 [15:52:39] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2172 [15:55:21] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 7.097 seconds [15:55:44] New patchset: Mark Bergsma; "Fix mode on /aufs dir" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2173 [15:56:02] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2173 [15:56:08] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2173 [15:56:08] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2173 [16:05:26] https://oc.wikipedia.org/wiki/Ceret is whitepageofdeath for me... :-( [16:06:16] dmituzas_: ^ [16:08:11] Happy-melon: to sum that up, the problem is known... [16:08:41] well that's sort-of-ok then :-) [16:08:51] what is it? [16:09:52] Happy-melon: they got templates which are so big, that they break the parser [16:10:04] (very big switch statements which use up to 1GiB of RAM) [16:10:13] yeah that's what I was looking at [16:10:21] trying to work out what the massive switch actually does [16:10:40] Happy-melon: Forget it, domas is working on that [16:10:51] that will be resolved ;) [16:11:03] AFAICT (and given my... limited... grasp of Occitian...), it seems to be a map between postcodes and altitudes :) [16:11:10] It is [16:11:25] well, population and zip codes, AFAIK [16:11:55] I think there are several of them, you just stick the postcode/zipcode into the template and it digs out various other parameters from that [16:12:36] yes... they agreed to get rid of those templates, but that might take some time... [16:12:45] they're JS is buggy as well, as I just saw -.- [16:13:11] shouldn't be that hard, it's a pretty trivial substitution, they just need a bot for it [16:13:25] its simplicity is presumably why they tried to template it in the first place. [16:13:28] Yes, I think they already got on running [16:13:41] * one [16:14:41] so why do 36,000 single-string switch branches take up a GiB of RAM? Each PPNode must be a kB?! [16:15:06] definitely scope for an optimisation there, regardless of whether we limit it [16:15:25] Happy-melon: Well, yes, but the parser is very complicated [16:15:47] this is the preprocessor, which is at least well-defined and vaguely sane [16:15:50] I thought there was rewrite planed anyway... [16:16:22] that memory is assigned outsight of PHP itself, so it can't be limited (at least with the usual PHP build) [16:16:26] s/usual/default/ [16:16:41] you can't limit it, but you can make your PHP code use it more efficiently [16:17:02] I guess you can feel free to fix it :P [16:17:05] like not creating a massive PPNode object if it's not necessary [16:17:12] I thought Brion was assigned to that? No? [16:17:13] * Happy-melon conspicuously does not volunteer :p [16:18:11] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 0.019 seconds [16:37:45] !log reedy synchronized wmf-config/InitialiseSettings.php 'wikilove default on fawikis' [16:37:46] Logged the message, Master [16:47:44] New patchset: Hashar; "web holder for WiktionaryMobile nightly builds" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2174 [16:48:03] New patchset: Mark Bergsma; "Add removal support for system roles" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2175 [16:48:18] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2174 [16:53:51] PROBLEM - Puppet freshness on knsq9 is CRITICAL: Puppet has not run in the last 10 hours [16:54:20] !log reedy synchronized php/cache/interwiki.cdb 'Updating interwiki cache' [16:54:21] Logged the message, Master [16:57:45] !log reedy synchronized php/cache/interwiki.cdb 'Updating interwiki cache' [16:57:46] Logged the message, Master [16:59:54] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 1; - https://gerrit.wikimedia.org/r/2166 [17:33:51] PROBLEM - RAID on searchidx2 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [17:45:01] RECOVERY - RAID on searchidx2 is OK: OK: State is Optimal, checked 4 logical device(s) [17:45:17] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 1; - https://gerrit.wikimedia.org/r/2175 [18:10:12] PROBLEM - HTTP on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:43:51] RECOVERY - HTTP on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 7.265 seconds [18:47:51] New review: Diederik; "(no comment)" [analytics/udp-filters] (master); V: 1 C: 0; - https://gerrit.wikimedia.org/r/2142 [18:47:51] Change merged: Diederik; [analytics/udp-filters] (master) - https://gerrit.wikimedia.org/r/2142 [18:50:50] I need a reference to the value of $wgJobRunRate used for the Wikipedia servers -- is it documented somewhere? The only place I have found it was here: http://wikitech.wikimedia.org/view/Server_admin_log/Archive_8#July_3 [18:53:39] <^demon> tr|nn|: http://noc.wikimedia.org/conf/highlight.php?file=InitialiseSettings.php, ctrl+f for it. [18:54:31] ^demon: thx -- that's exactly what I'm looking for :) [18:56:01] <^demon> You're welcome. The job run rate is set to 0 because we don't run jobs on pageviews. [18:56:12] <^demon> We run them manually with the runJobs.php maintenance script. [18:56:51] <^demon> Well, "manually." The job runners pretty much sit there in a loop doing their thing. [18:57:41] PROBLEM - Puppet freshness on mw8 is CRITICAL: Puppet has not run in the last 10 hours [19:00:26] ^demon: yep, I just needed a document I could refer to in my writings (I have already been told about this setting some weeks ago, but dont quite remember who it was) [19:01:41] you dont happen to know where i can find any documentation about the cronjob running runJobs.php? [19:02:21] RECOVERY - RAID on db47 is OK: OK: State is Optimal, checked 2 logical device(s) [19:03:07] <^demon> tr|nn|: http://www.mediawiki.org/wiki/Manual:Job_queue should have some good info. [19:03:59] <^demon> Also, http://svn.wikimedia.org/viewvc/mediawiki/branches/wmf/1.18wmf1/maintenance/jobs-loop.sh?view=annotate is the script we run on the cluster. [19:05:18] nw why does the first link on https://blog.wikimedia.org/2012/01/31/getting-ready-for-when-the-freeze-is-done/ give me all sorts of errors? [19:05:28] ^demon: Great, the script is what I was looking for! [19:06:25] (either opening an absurd mingle.corp.wikimedia.orgprojects or NoScript complaining about XSS attacks) [19:07:03] <^demon> Nemo_bis: orgprojects -> org/projects [19:07:06] <^demon> Should work then [19:07:11] PROBLEM - Mobile WAP site on ekrem is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:07:18] ^demon, yes, but the HTML link is correct [19:07:25] I don't understand why it tries to open that [19:07:42] <^demon> Oh I see. Bizzare. [19:08:42] <^demon> Ahh, seems to auto-redirect http -> https. [19:08:53] <^demon> So the redirect's busted it seems [19:13:56] ^demon, so a bug in mingle's apache rewrite? [19:14:01] should I file a bug? [19:14:20] <^demon> Sounds like it. [19:29:31] RECOVERY - Mobile WAP site on ekrem is OK: HTTP OK HTTP/1.1 200 OK - 1642 bytes in 5.543 seconds [19:29:51] !log reedy ran sync-common-all [19:29:53] Logged the message, Master [19:34:08] I have red interwikis again... https://commons.wikimedia.org/wiki/Category:Pedersen_bicycles [19:34:21] me too on bs.wiki [19:34:33] http://bs.wikipedia.org/wiki/Posebno:NedavneIzmjene [19:34:35] same at dewp [19:34:43] Reedy: ↑ [19:34:58] http://en.wikipedia.org/wiki/de:Test [19:35:00] etc. broken [19:35:01] Yes, in all wikis [19:35:42] that is the last log entry: " 19:29 logmsgbot: reedy ran sync-common-all" [19:38:15] ah, that problem is known ok, just wanted to complain :P [19:38:21] New patchset: Diederik; "Regex support" [analytics/udp-filters] (master) - https://gerrit.wikimedia.org/r/2176 [19:38:38] .. I just do not know if someone is working on it [19:39:01] and all interwikis are on the top of the page... not on the left [19:39:14] Reedy ^ [19:39:20] WizardOfOz: exactly the same happened yesterday, yup [19:39:45] i was not around yesterday, my come back was today :D [19:39:54] so it should work :D [19:40:07] commons too has red interwikilinks [19:40:29] well.. not really yesterday.. to be precise shortly after 0:00 UTC today :D [19:40:48] So, what can we do about it, what happened [19:41:01] I don't know [19:41:06] is there a reset button somewhere? :D [19:41:07] I admit to eating dinner [19:41:58] Did something related to interwikis break? [19:42:10] Reedy: Did you rebuild the interwiki cache recently? [19:42:37] we guess so [19:43:06] <^demon> But don't know what he changed. A rebuild might just leave it broken. [19:43:09] there was [19:43:18] "Could someone run sync-apache then apache-graceful-all for me please? I've added be.wikimedia.org to /h/w/c/httpd/wikimedia.org" [19:43:21] shortly before the boom. [19:43:55] <^demon> That shouldn't explode iw links. [19:43:59] hmm, he ran sync-common-all [19:44:09] That would've pushed a broken iw cache if there was one [19:44:16] likely [19:44:19] He was asking about CDB readers earlier [19:44:24] So maybe he was messing around there [19:44:26] <^demon> Right, presumably he didn't know it was broken before he pushed. [19:44:48] <^demon> Are there any local changes to dumpInterwiki in /h/w/c/p? [19:44:54] i just hope that he isn´t a microsoft employee... [19:45:50] nope, no changes [19:45:54] Rebuilding it *should* work [19:46:15] <^demon> Hopefully. Worst case we'll end up right where we're at right now. [19:46:31] it was changed today [19:46:39] -rw-rw-r-- 1 reedy wikidev 9055 2012-02-01 15:40 dumpInterwiki.php [19:46:41] PROBLEM - Puppet freshness on spence is CRITICAL: Puppet has not run in the last 10 hours [19:46:48] Or maybe worse, having all local language links point to foreign warz [19:46:58] Oooh, addwiki.php [19:47:00] !log catrope synchronized php-1.18/cache/interwiki.cdb 'rebuilt interwiki cache' [19:47:01] Logged the message, Master [19:47:02] That'll have rebuilt it [19:47:06] Probably in a broken way [19:47:13] ouch [19:47:23] !log catrope synchronized php-1.18/cache/interwiki-pr.cdb 'rebuilt interwiki cache' [19:47:24] Logged the message, Master [19:47:26] Merge quietening part of r110487 [19:47:33] that was today's change to dumpiw [19:48:11] still nope? [19:48:27] I pushed a change out earlier with that in [19:48:35] -pr ? [19:48:43] ah you're back [19:48:46] protocol-relative [19:48:49] right [19:48:51] like 3 hours ago [19:48:56] -pr isn't used any more [19:49:01] I tidied up the config earlier [19:49:16] * pir^2 blames Ryan_Lane [19:49:32] Oh, OK [19:49:34] Damn [19:49:35] pretty unlikely, pir^2 [19:49:36] Crap [19:49:38] Fixing [19:49:39] eh? [19:49:56] Reedy: what's used now? [19:50:05]