[00:00:36] RECOVERY - Disk space on search1004 is OK: DISK OK [00:02:15] RECOVERY - Disk space on ms-be4 is OK: DISK OK [00:02:24] RECOVERY - DPKG on ms-be4 is OK: All packages OK [00:02:42] RECOVERY - RAID on ms-be4 is OK: OK: Active: 2, Working: 2, Failed: 0, Spare: 0 [00:03:28] New patchset: Ryan Lane; "Test" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2951 [00:03:36] RECOVERY - RAID on search1005 is OK: OK: Active: 4, Working: 4, Failed: 0, Spare: 0 [00:04:21] RECOVERY - DPKG on search1005 is OK: All packages OK [00:04:57] RECOVERY - Disk space on search1005 is OK: DISK OK [00:07:12] RECOVERY - NTP on search1003 is OK: NTP OK: Offset -0.008622169495 secs [00:07:39] RECOVERY - RAID on search1006 is OK: OK: Active: 4, Working: 4, Failed: 0, Spare: 0 [00:08:24] RECOVERY - Disk space on search1006 is OK: DISK OK [00:08:24] RECOVERY - DPKG on search1006 is OK: All packages OK [00:10:30] RECOVERY - DPKG on search1007 is OK: All packages OK [00:11:42] RECOVERY - RAID on search1007 is OK: OK: Active: 4, Working: 4, Failed: 0, Spare: 0 [00:11:42] RECOVERY - Disk space on search1007 is OK: DISK OK [00:11:42] RECOVERY - NTP on search1005 is OK: NTP OK: Offset -0.03319811821 secs [00:12:00] RECOVERY - NTP on search1004 is OK: NTP OK: Offset 0.006244897842 secs [00:12:54] RECOVERY - NTP on ms-be4 is OK: NTP OK: Offset -0.01974022388 secs [00:13:03] RECOVERY - RAID on search1009 is OK: OK: Active: 4, Working: 4, Failed: 0, Spare: 0 [00:13:30] RECOVERY - Disk space on search1009 is OK: DISK OK [00:13:48] RECOVERY - DPKG on search1009 is OK: All packages OK [00:17:51] RECOVERY - NTP on search1007 is OK: NTP OK: Offset 0.004569292068 secs [00:17:51] RECOVERY - NTP on search1006 is OK: NTP OK: Offset -0.0298012495 secs [00:18:27] New patchset: Ryan Lane; "Test" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2951 [00:19:39] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:23:24] RECOVERY - NTP on search1009 is OK: NTP OK: Offset 0.02111792564 secs [00:23:42] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 6.268 seconds [00:27:09] RECOVERY - RAID on search1010 is OK: OK: Active: 4, Working: 4, Failed: 0, Spare: 0 [00:27:18] RECOVERY - Disk space on search1010 is OK: DISK OK [00:27:27] RECOVERY - DPKG on search1010 is OK: All packages OK [00:29:42] RECOVERY - Disk space on search1011 is OK: DISK OK [00:29:51] RECOVERY - DPKG on search1011 is OK: All packages OK [00:30:54] RECOVERY - RAID on search1011 is OK: OK: Active: 4, Working: 4, Failed: 0, Spare: 0 [00:33:09] RECOVERY - RAID on search1012 is OK: OK: Active: 4, Working: 4, Failed: 0, Spare: 0 [00:33:18] RECOVERY - DPKG on search1012 is OK: All packages OK [00:33:45] RECOVERY - Disk space on search1012 is OK: DISK OK [00:33:45] RECOVERY - NTP on search1010 is OK: NTP OK: Offset -0.01545226574 secs [00:35:03] New patchset: Ryan Lane; "Test" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2951 [00:36:54] RECOVERY - RAID on search1013 is OK: OK: Active: 4, Working: 4, Failed: 0, Spare: 0 [00:37:12] RECOVERY - NTP on search1011 is OK: NTP OK: Offset 0.02468931675 secs [00:37:30] RECOVERY - DPKG on search1013 is OK: All packages OK [00:37:57] RECOVERY - Disk space on search1013 is OK: DISK OK [00:40:12] RECOVERY - NTP on search1012 is OK: NTP OK: Offset 0.02842032909 secs [00:41:33] RECOVERY - DPKG on search1015 is OK: All packages OK [00:41:33] RECOVERY - RAID on search1015 is OK: OK: Active: 4, Working: 4, Failed: 0, Spare: 0 [00:42:00] RECOVERY - Disk space on search1015 is OK: DISK OK [00:42:57] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2951 [00:43:57] RECOVERY - NTP on search1013 is OK: NTP OK: Offset 0.03063797951 secs [00:44:51] RECOVERY - Disk space on search1016 is OK: DISK OK [00:44:51] RECOVERY - RAID on search1016 is OK: OK: Active: 4, Working: 4, Failed: 0, Spare: 0 [00:45:17] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2952 [00:45:27] RECOVERY - DPKG on search1016 is OK: All packages OK [00:47:00] New review: Asher; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2952 [00:47:03] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2952 [00:50:03] New patchset: Ryan Lane; "Fixing lint check" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2953 [00:50:14] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2953 [00:50:42] RECOVERY - NTP on search1015 is OK: NTP OK: Offset -0.01510822773 secs [00:51:22] New patchset: Ryan Lane; "Fixing lint check" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2953 [00:51:33] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2953 [00:51:44] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2953 [00:51:47] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2953 [00:52:04] Change abandoned: Ryan Lane; "Was just a test change." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2951 [00:54:09] RECOVERY - NTP on search1016 is OK: NTP OK: Offset 0.03278541565 secs [00:59:24] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:03:27] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 7.268 seconds [01:05:36] New patchset: Ryan Lane; "Test" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2954 [01:05:47] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2954 [01:08:42] RECOVERY - SSH on ms-be5 is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [01:20:51] RECOVERY - Puppet freshness on ms-be5 is OK: puppet ran at Wed Mar 7 01:20:37 UTC 2012 [01:23:24] RECOVERY - DPKG on ms-be5 is OK: All packages OK [01:23:39] Ugh, why can't I ever find something in this channel. It's always full of stuff [01:23:56] need live search in my irc client [01:24:27] RECOVERY - Disk space on ms-be5 is OK: DISK OK [01:24:54] RECOVERY - RAID on ms-be5 is OK: OK: Active: 2, Working: 2, Failed: 0, Spare: 0 [01:31:48] RECOVERY - NTP on ms-be5 is OK: NTP OK: Offset 0.01478517056 secs [01:31:57] PROBLEM - Puppet freshness on cadmium is CRITICAL: Puppet has not run in the last 10 hours [01:39:09] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:45:18] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 0.089 seconds [01:58:49] !log aaron synchronized php-1.19/includes/filerepo/backend/FileBackend.php 'bumped max file size to 4GiB' [01:58:53] Logged the message, Master [02:04:49] PHP Warning: array_shift() expects parameter 1 to be array, string given in /usr/local/apache/common-local/php-1.19/includes/api/ApiBase.php on line 1272 [02:04:57] RoanKattouw: swear I've seen that before [02:06:00] Huh [02:06:01] Lemme see [02:06:36] Oh [02:06:45] That's probably a bad invocation of one of the error functions [02:06:55] That deserves patching on the callee's side rather than the caller's side, I'll do it [02:11:27] !log catrope synchronized php-1.19/includes/api/ApiBase.php 'r113212' [02:11:30] Logged the message, Master [02:11:38] AaronSchulz: Fixed ---^ , thanks for poking [02:18:01] !log LocalisationUpdate completed (1.19) at Wed Mar 7 02:18:01 UTC 2012 [02:18:04] Logged the message, Master [02:19:21] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:23:24] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 335 bytes in 8.340 seconds [03:06:00] PROBLEM - Puppet freshness on mw1010 is CRITICAL: Puppet has not run in the last 10 hours [03:15:22] PROBLEM - Puppet freshness on mw1110 is CRITICAL: Puppet has not run in the last 10 hours [03:15:22] PROBLEM - Puppet freshness on mw1020 is CRITICAL: Puppet has not run in the last 10 hours [04:42:46] Krinkle: define live search? does your client have /lastlog ? [04:44:09] I guess not, what is it? [04:45:05] /lastlog Krinkle 5 gives me http://dpaste.com/712829/ [04:46:41] the messages show as a new block inline (but not interspersed) with messages in the channel. the whole thing scrolls up with other messages as new messages come in. i can then remove that chunk of messages with /lastlog -clear [04:46:48] i hope that wasn't too confusing [04:47:01] PROBLEM - Puppet freshness on db1004 is CRITICAL: Puppet has not run in the last 10 hours [04:56:01] PROBLEM - Puppet freshness on owa3 is CRITICAL: Puppet has not run in the last 10 hours [05:04:26] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [05:04:26] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [05:11:30] jeremyb: Very well explained [05:11:36] jeremyb: Sounds very nice :) [05:12:31] Krinkle: yay ;) [05:12:54] I'm using Colloquy, it does have a GUI search bar ( http://i.imgur.com/9xd6f.png ) but its horrible [05:13:52] here's what it looks like: [05:14:07] in that shot I've searched for the word "synchronized" http://i.imgur.com/vggFX.png [05:14:13] yeah, i've used it. i can't remember how well it worked [05:14:16] all it does is put tiny red dashes in the scroll bar [05:14:49] i don't remember red dashes. maybe gray dashes [05:15:10] grey bars are behind stalked words [05:15:15] (see first shot) [05:15:21] anyway, the red dashes are unusable [05:15:55] they are above the scroll bar area, but behind the scroll bar 'dragger'. so right now I can't' see half the results [05:16:28] and when you click them, well you can't click them. it'll just initial a regular scroll action, which still leaves you about 30 lines to look through [05:21:25] heh [06:11:50] New patchset: Dzahn; "add missing locale pl_PL.UTF-8 to fix broken pl.planet" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2955 [06:12:01] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2955 [06:12:37] New review: Dzahn; "pl.planet needs this. RT 2416" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2955 [06:12:40] Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2955 [07:02:51] PROBLEM - Puppet freshness on search1020 is CRITICAL: Puppet has not run in the last 10 hours [07:27:36] RECOVERY - Disk space on ms1004 is OK: DISK OK [07:31:33] New patchset: Hashar; "allow hashar on formey host" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2821 [07:31:45] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2821 [07:39:46] New review: Dzahn; "your change seems fine, just makes me wonder about the sudo users that don't have an account then. (..." [operations/puppet] (production); V: 1 C: 1; - https://gerrit.wikimedia.org/r/2821 [07:42:45] PROBLEM - Puppet freshness on search1019 is CRITICAL: Puppet has not run in the last 10 hours [08:24:03] New review: Hashar; "I have no idea. I guess the accounts were added manually over time." [operations/puppet] (production) C: 0; - https://gerrit.wikimedia.org/r/2821 [08:40:53] PROBLEM - RAID on searchidx2 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [08:42:50] RECOVERY - RAID on searchidx2 is OK: OK: State is Optimal, checked 4 logical device(s) [08:47:11] PROBLEM - Puppet freshness on db1022 is CRITICAL: Puppet has not run in the last 10 hours [09:15:09] PROBLEM - RAID on searchidx2 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [09:17:06] RECOVERY - RAID on searchidx2 is OK: OK: State is Optimal, checked 4 logical device(s) [11:21:00] they report that moving page on zhwiki triggers HTTP 500 and I can confirm it [11:21:03] can someone have a look? [11:29:09] !log hashar synchronizing Wikimedia installation... : trigger a rebuild of l10n cache [11:29:19] Logged the message, Master [11:29:33] liangent: can you open a bug please ? :/ [11:29:51] try to verify if it can be reproduced [11:30:18] hashar: move https://zh.wikipedia.org/wiki/Special:%E7%A7%BB%E5%8A%A8%E9%A1%B5%E9%9D%A2/Android_Market to [[Google Play]] [11:30:33] can you do it? [11:31:59] I cant [11:32:07] and going to cook my lunch right now :) [11:32:20] hence why I asked to open a bug, thus someone can have a look [11:33:12] PROBLEM - Puppet freshness on cadmium is CRITICAL: Puppet has not run in the last 10 hours [11:37:13] sync done. [11:38:10] * Damianz eyes logmsgbot [13:01:59] New review: Mark Bergsma; "I think it would be better to make a list of Nagios/Icinga monitoring IPs in a class (icinga::config..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2934 [13:03:38] New patchset: Mark Bergsma; "There is one in network.pp" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2956 [13:03:42] hmmm search on Commons is broken, isn't it? [13:03:49] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2956 [13:07:33] PROBLEM - Puppet freshness on mw1010 is CRITICAL: Puppet has not run in the last 10 hours [13:10:58] works again [13:16:38] PROBLEM - Puppet freshness on mw1020 is CRITICAL: Puppet has not run in the last 10 hours [13:16:38] PROBLEM - Puppet freshness on mw1110 is CRITICAL: Puppet has not run in the last 10 hours [13:34:53] New patchset: Mark Bergsma; "Remove csw5-pmtpa from monitoring" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2957 [13:35:04] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2957 [13:35:57] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2957 [13:36:00] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2957 [13:43:55] New review: Mark Bergsma; "Please fix indentation." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2936 [13:47:05] New review: Mark Bergsma; "Do you see that FIXME above the code you just copy pasted? :)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2936 [13:51:52] New review: Mark Bergsma; "So why are these files owned by rainman if they're managed by Puppet? rainman can't really edit them..." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2947 [13:59:43] hi [14:00:22] Please tell me someone [14:00:22] https://bugzilla.wikimedia.org/show_bug.cgi?id=34412 [14:00:22] Why not be processed? [14:00:22] When did you or will be created? [14:03:22] You had better ask to Casey? [14:10:08] New patchset: Hashar; "allow hashar on formey host" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2821 [14:10:20] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2821 [14:11:49] New review: Demon; "Per IRC discussion--doesn't need an account created, just needed the default shell adjusted in LDAP ..." [operations/puppet] (production) C: 1; - https://gerrit.wikimedia.org/r/2821 [14:34:26] Change abandoned: Hashar; "This is now unneeded. Just had to be allowed in the LDAP directory :-]" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2821 [14:38:44] v_a_: I will ask [14:39:10] v_a_: I think contact is Philippe Beaudette [14:48:03] PROBLEM - Puppet freshness on db1004 is CRITICAL: Puppet has not run in the last 10 hours [14:57:03] PROBLEM - Puppet freshness on owa3 is CRITICAL: Puppet has not run in the last 10 hours [15:06:03] PROBLEM - Puppet freshness on owa1 is CRITICAL: Puppet has not run in the last 10 hours [15:06:03] PROBLEM - Puppet freshness on owa2 is CRITICAL: Puppet has not run in the last 10 hours [15:17:55] RECOVERY - Puppet freshness on search1019 is OK: puppet ran at Wed Mar 7 15:17:47 UTC 2012 [15:20:09] RECOVERY - Disk space on search1019 is OK: DISK OK [15:20:28] hashar:thanks [15:20:44] v_a_: we will see. [15:21:03] RECOVERY - RAID on search1019 is OK: OK: Active: 4, Working: 4, Failed: 0, Spare: 0 [15:21:30] RECOVERY - Puppet freshness on search1020 is OK: puppet ran at Wed Mar 7 15:21:23 UTC 2012 [15:21:48] RECOVERY - DPKG on search1019 is OK: All packages OK [15:22:15] RECOVERY - Lucene on search1002 is OK: TCP OK - 0.030 second response time on port 8123 [15:23:36] RECOVERY - Disk space on search1020 is OK: DISK OK [15:24:12] RECOVERY - RAID on search1020 is OK: OK: Active: 4, Working: 4, Failed: 0, Spare: 0 [15:25:24] RECOVERY - DPKG on search1020 is OK: All packages OK [15:26:36] RECOVERY - Lucene on search1003 is OK: TCP OK - 0.027 second response time on port 8123 [15:28:06] RECOVERY - Lucene on search1006 is OK: TCP OK - 0.027 second response time on port 8123 [15:28:33] RECOVERY - Lucene on search1004 is OK: TCP OK - 0.027 second response time on port 8123 [15:28:42] RECOVERY - Lucene on search1005 is OK: TCP OK - 0.027 second response time on port 8123 [15:28:51] RECOVERY - Lucene on search1007 is OK: TCP OK - 0.027 second response time on port 8123 [15:29:54] RECOVERY - Lucene on search1012 is OK: TCP OK - 0.026 second response time on port 8123 [15:31:19] RECOVERY - Lucene on search1010 is OK: TCP OK - 0.026 second response time on port 8123 [15:31:19] RECOVERY - Lucene on search1011 is OK: TCP OK - 0.027 second response time on port 8123 [15:31:19] RECOVERY - Lucene on search1013 is OK: TCP OK - 0.027 second response time on port 8123 [15:31:19] RECOVERY - Lucene on search1015 is OK: TCP OK - 0.027 second response time on port 8123 [15:31:19] RECOVERY - Lucene on search1016 is OK: TCP OK - 0.027 second response time on port 8123 [15:33:43] RECOVERY - NTP on search1019 is OK: NTP OK: Offset 0.03436875343 secs [15:35:40] RECOVERY - NTP on search1020 is OK: NTP OK: Offset 0.02554428577 secs [15:37:52] New patchset: Pyoungmeister; "that's a bit better" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2958 [15:38:03] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2958 [15:40:23] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2958 [15:40:23] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2958 [15:40:55] PROBLEM - Disk space on srv220 is CRITICAL: DISK CRITICAL - free space: / 284 MB (3% inode=62%): /var/lib/ureadahead/debugfs 284 MB (3% inode=62%): [15:40:55] PROBLEM - Disk space on srv223 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=61%): /var/lib/ureadahead/debugfs 0 MB (0% inode=61%): [15:40:55] PROBLEM - Disk space on srv219 is CRITICAL: DISK CRITICAL - free space: / 179 MB (2% inode=61%): /var/lib/ureadahead/debugfs 179 MB (2% inode=61%): [15:40:55] PROBLEM - Disk space on srv224 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=62%): /var/lib/ureadahead/debugfs 0 MB (0% inode=62%): [15:56:49] RECOVERY - Disk space on srv224 is OK: DISK OK [15:56:58] RECOVERY - Disk space on srv219 is OK: DISK OK [15:56:58] RECOVERY - Disk space on srv220 is OK: DISK OK [15:56:58] RECOVERY - Disk space on srv223 is OK: DISK OK [16:09:16] RECOVERY - Lucene on search1020 is OK: TCP OK - 0.027 second response time on port 8123 [16:09:16] RECOVERY - Lucene on search1019 is OK: TCP OK - 0.026 second response time on port 8123 [16:52:54] RECOVERY - RAID on db22 is OK: OK: 1 logical device(s) checked [17:21:59] MW 1.19 has been having more occurrence of bad page caches than the previous version [17:22:07] any clue when it's going to be fixed? [17:22:17] example: https://pt.wiktionary.org/w/index.php?title=Especial:P%C3%A1ginas_%C3%B3rf%C3%A3s&limit=500&offset=0 [17:22:24] most of these pages are not orphans [17:22:47] brrr: http://nl.wikipedia.org/w/index.php?title=Speciaal:GevraagdeSjablonen&limit=500&offset=0 [17:22:59] every time the special pages are generated, more of these appear [17:23:40] https://bugzilla.wikimedia.org/show_bug.cgi?id=31576 [17:23:44] Romaine: i guess i'm not alone here nor original in the subject :) [17:23:57] and related to: https://bugzilla.wikimedia.org/show_bug.cgi?id=31577 [17:24:08] this bug is occurring agains since 1.19 [17:24:17] yep, i already saw those but i see no evolution there [17:24:28] devolution ;) [17:24:50] i had submitted an issue some time ago which was considered duplicated of that one, but no solution yet [17:24:58] :( [17:25:45] it should be quite easy to catch since it happens on every special page generation [17:26:07] of course, the problem is in the page generation [17:26:16] not the special pages [17:26:24] the problem is on the articles itself [17:26:51] example: http://nl.wikipedia.org/w/index.php?title=NGC_1260&action=edit#footer [17:27:04] these magic words are considered to be templates [17:27:09] which of course do not exist [17:27:28] Roan had been busy with it previous, but now the problem is back since 1.19 [17:31:18] malafaya: posted http://www.mediawiki.org/wiki/User_talk:Catrope#Problems_with_magic_words_considered_as_.28non-existing.29_templates [17:59:21] https://nl.wikipedia.org/w/index.php?title=Wilkesboro&action=edit#footer [18:10:15] brrrrr [18:10:24] I hope someone can solve it this time [18:10:38] a bug since 1.18, continues with 1.19 [18:21:15] New patchset: Bhartshorne; "Add 500 tracking to the Swift proxy logtailer" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2960 [18:21:26] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2960 [18:22:03] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2960 [18:22:05] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2960 [18:30:33] New patchset: Bhartshorne; "Add 503 tracking to the Swift proxy logtailer" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2961 [18:30:44] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2961 [18:31:07] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2961 [18:31:10] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2961 [18:36:35] !log catrope synchronized php-1.19/extensions/CustomUserSignup/modules/AccountCreationUserBucket.js 'touch' [18:36:38] Logged the message, Master [18:36:46] !log catrope synchronized php-1.19/resources/startup.js 'touch' [18:36:49] Logged the message, Master [18:49:10] PROBLEM - Puppet freshness on db1022 is CRITICAL: Puppet has not run in the last 10 hours [19:00:01] New patchset: Bhartshorne; "counting all swift hits by status code" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2962 [19:00:12] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2962 [19:00:24] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2962 [19:00:27] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2962 [19:27:24] New patchset: Bhartshorne; "corrected hits by status code to per-second instead of per-measurement interval. added percentage of hits by status code." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2963 [19:27:35] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2963 [19:27:46] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2963 [19:27:49] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2963 [19:38:06] !log catrope synchronized php-1.19/includes/resourceloader/ 'r113278' [19:38:09] Logged the message, Master [19:41:21] New patchset: Asher; "it's wikipedia zero time!" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2964 [19:41:28] preilly: ^^^^ [19:41:32] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2964 [19:44:07] New patchset: Bhartshorne; "changed the name of the ganglia logtailer from being proxy specific to being http specific since it works for the backend stoarge nodes too. moved the puppet config from proxy-specific to swift::base to include it on all nodes." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2965 [19:44:19] New patchset: Bhartshorne; "changed the name of the ganglia logtailer from being proxy specific to being http specific since it works for the backend stoarge nodes too. moved the puppet config from proxy-specific to swift::base to include it on all nodes." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2966 [19:44:30] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2965 [19:44:30] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2966 [19:44:38] New patchset: Asher; "it's wikipedia zero time!" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2964 [19:44:49] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2964 [19:45:09] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2965 [19:45:12] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2965 [19:45:19] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2966 [19:45:23] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2966 [19:45:57] New review: Asher; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2964 [19:46:00] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2964 [20:20:44] !log catrope synchronized php-1.19/resources/startup.js 'touch' [20:20:47] Logged the message, Master [20:25:47] New patchset: Pyoungmeister; "required for catch-all term" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2967 [20:25:58] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2967 [20:26:27] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2967 [20:26:29] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2967 [20:31:31] Ryan_Lane: hmm, you cannot add a global user-right from a local user group; generally philippe asks a steward to perform the +staff change. [20:31:47] -_- [20:32:02] should I just do this in the database directly? [20:32:04] cough cough [20:32:06] it's silly I don't have these rights [20:32:18] need help, Ryan_Lane ? [20:32:40] I need to be able to modify global groups [20:32:56] Ask Brion ;) [20:33:00] heh [20:33:00] ssh root@ :D [20:34:15] well, I *can* just do it in the database, but it's a really bad idea for me to edit the database directly [20:34:58] Then don't. [20:35:00] :P [20:35:13] The mw database permissions wise is a bit weird. [20:35:18] It's a rights change, right? [20:35:45] when a staff member needs rights to do something, it's ridiculous for me to have to ask a community member and have someone's work blocked for hours waiting. [20:35:54] Philippe usually handles those by asking the stewards directly in their channel. [20:36:05] so there is a record, and a procedure. [20:36:06] it usually means I need to spend my time doing it for them [20:36:16] it's a waste of time and money [20:36:17] not really [20:36:25] Not really. We perform changes immediately when Philippe asked. [20:36:28] Yes, following policies is a pain. :P [20:36:29] *asks. [20:36:41] *why* is it a policy? [20:36:45] like within seconds, faster than you need to assign rights. [20:36:49] Why is it not? [20:36:56] Ryan_Lane: ask your boss, not us. [20:37:02] it is the way things have been done for a while. [20:37:10] before you joined, even., [20:37:15] well, it's apparently a community policy [20:37:22] otherwise you can run for next stward elections, I'll surely support you ^^ [20:37:23] Actually staff. [20:37:28] and one that makes little sense, seeing as that I have full access to the databases [20:37:39] and all of the systems [20:37:51] So you can trample on the existing community members whenever? [20:38:01] it's not a contest. [20:38:12] I'm not sure how adding someone to the staff group tramples members [20:38:24] or giving someone permission to do recursive exports [20:38:34] It tramples the existing established policy. [20:38:40] We've had this issue a lot on Meta. [20:38:46] tech staff assigning global rights. [20:38:48] OTOH stewards did the change immediately when we asked. So it didn't block a staff member's work for hours. [20:38:50] Stewards dont like it. [20:38:50] working via db is not the highest degree of transparency [20:39:02] Vito: which is why I don't do it [20:39:09] I'd much prefer to give these rights via wiki [20:39:12] you just threaten to do it. ;) [20:39:21] so it's useless to go on saying you can do it [20:39:25] no. I'm saying I have the ability to, not that I would [20:39:32] heh [20:39:46] I want to have my changes tracked [20:40:16] how can I go about changing this policy? [20:40:18] that is what the stewards and the community want to. [20:40:28] Talk to Philippe, write a page on Meta. [20:40:37] point stewards to comment. [20:40:41] That should be it. [20:40:47] is there a specific place for this? [20:41:05] depends, on the exact nature of what you propose. [20:41:15] afaik the best thing would be having staff rights managed by the staff itself [20:41:55] I'm going to propose that operations team members be given the ability to globally manage groups, with the expectation they'll only ever edit rights needed for doing technical work [20:42:06] err [20:42:23] members of the sysadmin global group, rather than operations team members [20:42:41] http://meta.wikimedia.org/wiki/Steward_requests/Global_permissions [20:42:42] I dont follow, BTW, Isn't guillom the tech communications person? [20:42:46] honestly my only concern is about their knowledge of groups and centralauth [20:42:47] seems like an appropriate place [20:42:53] Theo10011: so? [20:43:05] can you help direct this query guillom? [20:43:19] Vito: if they don't understand them properly, it's an issue already, seeing as that we add them to the configuration files [20:43:21] You are a steward too, you should know better than anyone else. [20:43:39] Ryan_Lane, I wouldn't put it there. It's a policy change you are proposing [20:43:47] If Ryan_Lane wants my help he can surely ask me, but it's not a requirement to have all communications go through me (fortunately) [20:43:48] BZ it :D [20:44:01] guillom: where should I ask to have the policy changed? [20:44:06] You know, Joan usually knows more about this stuff than most of us. [20:44:11] Joan rivers! [20:44:15] Ryan_Lane: well, but even you had just made a mistake with the centralauth [20:44:42] I'd tend to have some doubt about people which has no "user-side" background at all [20:44:47] Vito: in the past, adding myself to steward on meta allowed me to edit global groups [20:45:23] in the past there were no global groups I suppose [20:45:52] Ryan_Lane: honestly, I'm not sure; which is why I was trying to stay out of this :) Perhaps https://meta.wikimedia.org/wiki/Stewards%27_noticeboard ; if it's not the right place they'll tell you. [20:46:33] well, I'm going to propose global rights editing with a limitation on what we're allowed to change [20:46:43] anyway I already supported a similar idea [20:46:46] I don't want to add people to global sysop, for instance [20:46:52] just tech rights [20:46:57] the main problem is there's just a global group for staffers [20:47:08] there's also global sysadmin [20:47:10] I'll support a limited set of senior staff with wikimedia community experience to get a special user group to do this kind of things [20:47:13] which is a much more limited group [20:47:13] All staff, not really. [20:47:49] yeah. I don't think all staff need this right [20:48:04] well the staff right group was going to be divided a while ago, if I recall. Philippe and guillom were gonna work on it. [20:48:09] Don't know what happened. [20:48:11] Tho in the current situation, the users who I'd support getting the right already have it by asking stewards [20:48:15] It didn't happen. [20:48:20] aw [20:48:23] Ask Philippe to revive it. [20:48:28] Nah :P [20:48:48] I've seen some pretty heavy duty rights already being doled out to people with less than 10 edits. [20:49:03] * Vito sighs [20:49:21] heh. everything is always so drama filled :D [20:49:29] You have no idea. [20:49:35] It's been this way for a while. [20:49:55] ah: https://meta.wikimedia.org/wiki/Meta:Requests_and_proposals [20:50:49] and very likely: https://meta.wikimedia.org/wiki/Requests_for_comment [20:56:17] <^demon> {{support}} Ryan's ideas are always the best! ~~~~ [20:56:34] what's the idea [20:56:40] * petan hands a cookie to Ryan_Lane [20:56:47] <^demon> Don't know, but it's bound to be awesome :) [20:56:49] {support}}} --~~~~~ fits ore [20:56:51] *more [20:56:52] aha [20:56:59]