[00:03:28] New patchset: Ryan Lane; "Decommissioning mobile2" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1895 [00:04:06] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1895 [00:04:07] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1895 [00:05:56] Thehelpfulone: what timezone are you in? [00:06:24] Thehelpfulone: because I have spread the options over 11 hours on both week days and the weekend.. [00:06:50] i'm in the UK (GMT) [00:07:23] I should be able to come, but it was just something that occurred to me [00:07:26] Thehelpfulone: that's just one our from my CET... [00:17:30] !log stopping puppet on all virt nodes [00:17:31] Logged the message, Master [00:23:58] PROBLEM - Disk space on srv223 is CRITICAL: DISK CRITICAL - free space: / 191 MB (2% inode=60%): /var/lib/ureadahead/debugfs 191 MB (2% inode=60%): [00:26:55] New patchset: Ryan Lane; "Making virt0 the new controller. Moving all nova config to point to it." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1896 [00:27:10] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1896 [00:28:58] !log reedy synchronized closed.dblist 'Closing en_labswikimedia, de_labswikimedia, liquidthreads_labswikimedia' [00:29:00] Logged the message, Master [00:32:28] PROBLEM - Disk space on srv219 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=60%): /var/lib/ureadahead/debugfs 0 MB (0% inode=60%): [00:33:10] !log reedy synchronized wmf-config/InitialiseSettings.php [00:33:12] Logged the message, Master [00:33:16] !log That was only touch [00:33:18] Logged the message, Master [00:33:32] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1896 [00:33:33] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1896 [00:35:59] hi there, seeing as reddit is doing a blackout on the 18th for SOPA, is wikipedia doing the same? [00:42:46] I don't know what our official stance is on this [00:43:01] Sue Gardener did IRC office hours earlier, but I wasn't there [00:43:34] *Gardner [00:44:20] New patchset: Ryan Lane; "This dependency is needed" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1897 [00:44:35] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1897 [00:44:35] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1897 [00:44:43] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1897 [00:44:43] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1897 [00:48:54] ahh ok Reedy, what are office hours anyway? [00:49:16] http://meta.wikimedia.org/wiki/IRC_office_hours [00:49:50] ahh [00:56:19] RECOVERY - Disk space on srv219 is OK: DISK OK [00:57:49] RECOVERY - Disk space on srv223 is OK: DISK OK [01:02:10] !log reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33468 - Email notifications for eswikibooks' [01:02:12] Logged the message, Master [01:02:47] !log reedy synchronized closed.dblist 'Closing en_labswikimedia, de_labswikimedia, liquidthreads_labswikimedia (resync)' [01:02:49] Logged the message, Master [01:09:55] !log reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33556 - ArticleFeedback settings on Chinese wikipedia' [01:09:56] Logged the message, Master [01:16:14] Nikerabbit: is r101507 ok now? [01:39:57] !log reedy synchronized wmf-config/InitialiseSettings.php 'bug 33507 for aswiki' [01:39:59] Logged the message, Master [01:47:07] !log reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33469 - Enable rollback function for editor group kawiki' [01:47:09] Logged the message, Master [02:05:30] !log LocalisationUpdate completed (1.18) at Fri Jan 13 02:05:29 UTC 2012 [02:05:31] Logged the message, Master [02:16:42] gn8 folks [02:21:17] PROBLEM - Memcached on virt0 is CRITICAL: Connection refused [02:21:37] PROBLEM - Auth DNS on labsconsole.wikimedia.org is CRITICAL: CRITICAL - Plugin timed out while executing system call [02:56:40] !log switched mysql masters for labs to virt0 [02:56:42] Logged the message, Master [02:56:51] !log switched rabbitmq server in labs to virt0 [02:56:52] Logged the message, Master [02:57:17] !log switched active ldap server in labs to virt0, for nova itself. instances still need to be re-pointed [02:57:19] Logged the message, Master [02:58:05] !log dns server is up on virt0 [02:58:06] Logged the message, Master [03:01:16] RECOVERY - Auth DNS on labsconsole.wikimedia.org is OK: DNS OK: 0.134 seconds response time. www.wikipedia.wmflabs.org returns 208.80.153.197 [03:29:12] !log switching labsconsole.wikimedia.org address to point to virt0 [03:29:13] Logged the message, Master [04:17:00] RECOVERY - Disk space on es1004 is OK: DISK OK [04:20:13] RECOVERY - MySQL disk space on es1004 is OK: DISK OK [04:42:00] PROBLEM - MySQL slave status on es1004 is CRITICAL: CRITICAL: Slave running: expected Yes, got No [05:29:04] PROBLEM - Puppet freshness on db22 is CRITICAL: Puppet has not run in the last 10 hours [07:01:08] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [07:42:30] !log fixed memcached port in mediawiki configuration on labsconsole to fix slowness issue [07:42:33] Logged the message, Master [07:43:34] !log added a grant for mediawiki in the database to fix labsconsole mediawiki outage [07:43:36] Logged the message, Master [07:45:17] !log disassociated and reassociated some floating IP addresses, to fix NAT issues. Some NAT rules went missing. [07:45:19] Logged the message, Master [08:01:12] PROBLEM - Disk space on srv221 is CRITICAL: DISK CRITICAL - free space: / 193 MB (2% inode=60%): /var/lib/ureadahead/debugfs 193 MB (2% inode=60%): [08:14:04] PROBLEM - Disk space on srv222 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=60%): /var/lib/ureadahead/debugfs 0 MB (0% inode=60%): [08:32:12] PROBLEM - Disk space on srv223 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=60%): /var/lib/ureadahead/debugfs 0 MB (0% inode=60%): [08:38:43] PROBLEM - Disk space on srv220 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=60%): /var/lib/ureadahead/debugfs 0 MB (0% inode=60%): [08:42:02] RECOVERY - Disk space on srv223 is OK: DISK OK [08:47:42] RECOVERY - Disk space on srv221 is OK: DISK OK [08:49:42] RECOVERY - Disk space on srv222 is OK: DISK OK [08:58:22] RECOVERY - Disk space on srv220 is OK: DISK OK [09:15:42] RECOVERY - Memcached on virt0 is OK: TCP OK - 0.008 second response time on port 11000 [09:55:43] PROBLEM - MySQL disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 448656 MB (3% inode=99%): [10:01:33] PROBLEM - Disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 417067 MB (3% inode=99%): [10:24:34] RECOVERY - MySQL slave status on es1004 is OK: OK: [10:38:24] hi, could some one please look into this bug https://bugzilla.wikimedia.org/show_bug.cgi?id=33507 [10:39:08] because of the current fix all pages under Wikipedia namespace are missing [10:41:22] I think a space at the end of the localized version (ৱিকিপিডিয়া - Assamese name for Wikipedi) is creating the issue [12:14:47] PROBLEM - Puppet freshness on db1001 is CRITICAL: Puppet has not run in the last 10 hours [12:32:34] hi, could some one please look into this bug https://bugzilla.wikimedia.org/show_bug.cgi?id=33507 [12:32:52] because of the current fix all pages under Wikipedia namespace are missing [12:33:02] I think a space at the end of the localized version (ৱিকিপিডিয়া - Assamese name for Wikipedia) is creating the issue [12:57:09] RECOVERY - HTTPS on sodium is OK: OK - Certificate will expire on 08/22/2015 22:23. [13:03:49] RECOVERY - Host srv191 is UP: PING OK - Packet loss = 0%, RTA = 0.28 ms [13:07:38] New patchset: Mark Bergsma; "Add exim::roled class documentation" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1900 [13:07:54] New patchset: Mark Bergsma; "Rename relay_domains file" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1901 [13:08:08] New patchset: Mark Bergsma; "Add IPv6 service IP for lists.wikimedia.org on sodium" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1902 [13:08:22] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1900 [13:08:22] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1900 [13:08:23] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1902 [13:08:35] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1901 [13:08:36] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1902 [13:08:36] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1901 [13:12:16] New patchset: Mark Bergsma; "Enable v6 for outbound as well" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1903 [13:12:57] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1903 [13:12:57] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1903 [13:15:26] New patchset: Mark Bergsma; "Notify service exim4 on config changes" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1904 [13:15:40] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1904 [13:15:45] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1904 [13:15:46] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1904 [13:24:29] PROBLEM - DPKG on srv191 is CRITICAL: Connection refused by host [13:24:39] PROBLEM - Memcached on srv191 is CRITICAL: Connection refused [13:28:09] PROBLEM - Disk space on srv191 is CRITICAL: Connection refused by host [13:32:19] PROBLEM - RAID on srv191 is CRITICAL: Connection refused by host [13:32:39] PROBLEM - Apache HTTP on srv191 is CRITICAL: Connection refused [13:59:53] PROBLEM - Host sodium is DOWN: PING CRITICAL - Packet loss = 100% [14:06:43] RECOVERY - Puppet freshness on srv191 is OK: puppet ran at Fri Jan 13 14:06:32 UTC 2012 [14:09:43] RECOVERY - Apache HTTP on srv191 is OK: HTTP OK HTTP/1.1 200 OK - 453 bytes in 0.012 seconds [14:14:04] PROBLEM - Auth DNS on ns1.wikimedia.org is CRITICAL: CRITICAL - Plugin timed out while executing system call [14:14:33] RECOVERY - Disk space on srv191 is OK: DISK OK [14:19:34] RECOVERY - RAID on srv191 is OK: OK: no RAID installed [14:22:33] RECOVERY - DPKG on srv191 is OK: All packages OK [14:24:03] RECOVERY - Auth DNS on ns1.wikimedia.org is OK: DNS OK: 0.026 seconds response time. www.wikipedia.org returns 208.80.152.201 [14:28:03] RECOVERY - Host sodium is UP: PING OK - Packet loss = 0%, RTA = 30.91 ms [14:32:05] PROBLEM - mailman on sodium is CRITICAL: Connection refused by host [14:32:05] RECOVERY - Memcached on srv191 is OK: TCP OK - 2.994 second response time on port 11000 [14:33:15] PROBLEM - HTTPS on sodium is CRITICAL: Connection refused [14:34:55] PROBLEM - DPKG on sodium is CRITICAL: Connection refused by host [14:34:55] PROBLEM - RAID on sodium is CRITICAL: Connection refused by host [14:36:45] PROBLEM - SSH on sodium is CRITICAL: Connection refused [14:41:25] PROBLEM - spamassassin on sodium is CRITICAL: Connection refused by host [14:41:35] PROBLEM - HTTP on sodium is CRITICAL: Connection refused [15:38:42] PROBLEM - Puppet freshness on db22 is CRITICAL: Puppet has not run in the last 10 hours [15:51:11] guillom: hi? [15:51:40] hello mutante [15:52:09] about to to the requested blog update [15:52:34] mutante, oh, fantastic! That's... fast! (I'm not complaining :) [15:52:37] if you want to check in a minute,ok? [15:52:41] sure [15:52:46] thanks! [15:53:12] theme: Updated to revision 40. [15:53:23] checking [15:53:59] Everything looks ok [15:54:29] WMBlog: Checked out revision 2. [15:54:48] ok, going to test it now [15:55:14] looks good [15:55:22] Danke schön mutante ! [15:55:29] :), de rien [15:56:36] RECOVERY - SSH on sodium is OK: SSH OK - OpenSSH_5.3p1 Debian-3ubuntu7 (protocol 2.0) [15:59:34] guillom: next week if you want, i can also help with the Labs prototype [15:59:46] guillom: re: RT ticket 2032 [16:00:11] mutante, ah, thanks. Yes, that would be great. [16:02:11] !b 33507 [16:02:11] https://bugzilla.wikimedia.org/show_bug.cgi?id=33507 [16:02:36] i don't see a reedy, can someone take a look at that? [16:03:14] seems aswiki's wgMetaNamespace does in fact have a space [16:06:14] drdee: do you have all of the access you need now? [16:07:36] !log updated blog theme and installed a plugin per RT 2271 [16:07:37] Logged the message, Master [16:16:32] any shell ppl around? 33507 is a regression AFAICT... [16:17:25] alswikt and alswikibooks aren't closed properly [16:17:39] http://meta.wikimedia.org/w/api.php?action=sitematrix&format=xml lists them as open [16:18:23] I don't see the space in the config [16:18:57] Platonides: i do [16:19:27] jeremyb, I see it in the api, but not in initialisesettings [16:19:37] Platonides: i see it in initialisesettings [16:20:24] oh, I see it now [16:20:41] I was looking at namespacealiases [16:20:57] shijualex: ping [16:21:43] someone needs to remove that space from 1617 [16:21:51] what's 1617? [16:21:57] *from line 1617 [16:22:35] I'd do it myself, but I don't have access there [16:22:42] and I don't think they are in git either [16:23:20] def not git. ;( ;( [16:23:47] Platonides: where is it? [16:24:00] mutante, where's what? [16:24:10] mutante: InitialiseSettings.php [16:24:16] the space that needs to be removed [16:24:17] in fenari [16:24:22] mutante: 'aswiki' in line 1617 [16:25:23] => 'ৱিকিপিডিয়া ', [16:25:25] mutante: in vim you can convert the line to hex (:.!xxd) edit the hex to rm trailing space and then convert back (:.,+2!xxd -r) [16:26:40] (that's the way i'd do it) [16:27:39] or i can give you a diff if that's easier :) [16:28:35] the part to convert back doesnt seem to work, trying:) [16:29:07] oh, i left out a letter [16:29:20] it's :.,.+2! [16:31:28] i changed it, needs sync? (its not in common) [16:31:45] yes, InitialiseSettings need to be synced [16:32:00] PROBLEM - Disk space on srv219 is CRITICAL: DISK CRITICAL - free space: / 84 MB (1% inode=60%): /var/lib/ureadahead/debugfs 84 MB (1% inode=60%): [16:32:03] oh it is , nevermind [16:33:11] !log dzahn synchronized ./wmf-config/InitialiseSettings.php [16:33:12] Logged the message, Master [16:33:28] !log syncing InitialiseSettings.php after changing as wiki namespace per bz 33507 [16:33:29] Logged the message, Master [16:34:39] did it change? [16:36:05] I see no trailing space in the api now [16:36:30] good, thats what we wanted right [16:37:10] PROBLEM - Disk space on srv221 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=60%): /var/lib/ureadahead/debugfs 0 MB (0% inode=60%): [16:37:59] So... could alswikt and alswikibooks be properly marked as closed now? [16:38:06] Or shall I create a bug [16:38:28] the name itself seems changed, though [16:38:41] # 'alswiki' => 'gem-alsatian', [16:38:42] # 'alswiktionary' => 'gem-alsatian', [16:38:46] they are commented [16:38:51] it was a7 9f e0 a6 be 20 and now I see a6 af e0 a6 bc e0 a6 be [16:39:09] I don't know if it's just an alternative way to code the same glyph [16:39:14] mutante: http://meta.wikimedia.org/w/api.php?action=sitematrix&format=xml lists them as open [16:39:16] jeremyb? [16:39:25] but the only open als project it alswiki [16:43:59] !log dzahn synchronized closed.dblist [16:44:00] Logged the message, Master [16:44:15] !log added alswiktionary & alswikibooks to closed.dblist [16:44:16] Logged the message, Master [16:44:50] thanks, mutante ;) [16:44:51] hoo: [16:44:58] i hope thats all that was needed though [16:45:17] i agree they were obviously closed already..but they werent on that list [16:45:23] Platonides: mutante: sorry internet died [16:45:24] Yes, sitematrix is fine now [16:45:34] k:) [16:46:16] jeremyb: please check if als looks better now [16:46:25] as, !als [16:46:35] arg, yea:) we just talked about als [16:46:36] shijualex: ^^^^ [16:46:39] i know :) [16:46:47] but line 1617 [16:46:50] RECOVERY - Disk space on srv221 is OK: DISK OK [16:48:07] be back in 5 minutes [16:50:01] k [16:51:30] RECOVERY - Disk space on srv219 is OK: DISK OK [16:53:38] mutante: can has svn diff? [16:55:21] - 'aswiki' => 'ৱিকিপিডিয়া ', [16:55:21] + 'aswiki' => 'ৱিকিপিডিয়া', [16:55:41] mutante: can you pipe that to `xxd -ps` ? ;-) [16:56:35] mutante: (shijualex is the native and is here but i guess too idle. this is the third hilight) [16:57:10] jeremyb: http://meta.wikimedia.org/wiki/User:Mutante/pastebin [16:57:24] haha, that's not a pastebin! [16:57:28] let me add a doesn't matter for hex [16:58:04] jeremyb: do we have a place for it, btw? (pastebin) [16:58:25] mutante: idk... i usually use dpaste.com [16:58:31] mutante: or etherpad... [16:59:13] etherpad, so right! yeap [16:59:43] i do like dpaste.com except they have a size limit [16:59:51] anyway... [17:01:40] PROBLEM - NTP on ms1002 is CRITICAL: NTP CRITICAL: No response from NTP server [17:02:44] jeremyb: i should have just pasted to bugzilla in the first place, doing so now [17:11:00] PROBLEM - Puppet freshness on ms1002 is CRITICAL: Puppet has not run in the last 10 hours [17:15:40] hm, what's with the fancy stuff on the account creation form you get with campaign=ACP2? [17:15:46] can i get some more info about that somewhere? [17:19:29] hi, if a wiki is closed and locked, only stewards and staff should be able to log in right? [17:20:30] Thehelpfulone: idk about log in, but edit anything certainly. and idk if it applies to staff. maybe just stewards [17:20:52] okay so I can't edit anything jeremyb but I can change my user rights [17:21:04] idk... [17:21:09] what are you? [17:21:15] just auto confirmed [17:21:17] https://bugzilla.wikimedia.org/show_bug.cgi?id=33644 [17:21:41] but all users can grant 2 rights to themselves [17:21:46] (I'm looking at en.labs.wikimedia.org [17:25:43] i doesn't follow [17:29:47] jeremyb: I updated the bug to reflect the problem [17:37:53] Thehelpfulone: are you wikichaipau? [17:38:10] nope... [17:38:17] where did you see that? [17:38:21] Thehelpfulone: different bug... ;-P [17:38:24] 33507 [17:38:51] * Thehelpfulone is Thehelpfulone :P [17:38:58] how confusing [17:39:22] I know right? [17:40:47] Tanvir: ping? [17:40:57] Jeremyb, yes? [17:41:25] Tanvir: can you read devanagri? [17:41:34] Jeremyb, no. [17:41:41] ;( [19:01:42] RECOVERY - Puppet freshness on ms1002 is OK: puppet ran at Fri Jan 13 19:01:30 UTC 2012 [19:05:53] RECOVERY - NTP on ms1002 is OK: NTP OK: Offset 0.08668124676 secs [19:06:44] RECOVERY - RAID on ms1002 is OK: OK: State is Optimal, checked 2 logical device(s) [19:13:46] no platonides ;( [19:19:07] so, https://bugzilla.wikimedia.org/show_bug.cgi?id=33507#c11 is still broke [19:19:45] * jeremyb has to do other stuff for a while... wish i could read half the chars even! [19:27:01] New patchset: Lcarr; "Fixed some formatting and ensure gmond.conf present" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1905 [19:28:52] New patchset: Lcarr; "Fixed some formatting and ensure gmond.conf present" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1905 [19:29:34] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1905 [19:29:35] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1905 [19:30:18] !log shutting down virt1 to ensure migration was completed [19:30:20] Logged the message, Master [19:31:03] heh [19:36:20] PROBLEM - Host virt1 is DOWN: PING CRITICAL - Packet loss = 100% [19:38:45] !bug 33509 [19:38:45] https://bugzilla.wikimedia.org/show_bug.cgi?id=33509 [19:41:18] New patchset: Lcarr; "separating out the cp machines to make them try and realize they are collectors" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1906 [19:41:39] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/1906 [19:42:34] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1906 [19:42:34] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1906 [19:43:42] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1893 [19:43:42] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1894 [19:43:43] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1893 [19:45:58] RECOVERY - Host virt1 is UP: PING OK - Packet loss = 0%, RTA = 0.24 ms [19:48:07] New patchset: Lcarr; "cp1044 is an aggregator" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1907 [19:48:51] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1907 [19:48:52] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1907 [19:50:53] saper: maybe would help if you got an RT filed for that [19:51:39] New patchset: Bhartshorne; "shouldn't change what the user passed in - range() does what I want instead." [operations/software] (master) - https://gerrit.wikimedia.org/r/1908 [19:53:14] jeremyb: can I file something in RT? [19:53:31] saper: no, but there are ppl here that can for you [19:53:58] that's what I thought. It's totally non-tech. [19:54:11] New patchset: Bhartshorne; "shouldn't change what the user passed in - range() does what I want instead." [operations/software] (master) - https://gerrit.wikimedia.org/r/1908 [20:00:20] New patchset: Asher; "named virthosts on 443. shine on, little star cert." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1909 [20:00:46] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1909 [20:00:47] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1909 [20:16:24] !log synchronized payments cluster to r108833 [20:16:25] Logged the message, Master [20:19:13] New patchset: Asher; "further cluster def cleanup, write a marker file on dbs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1910 [20:20:44] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1910 [20:20:44] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1910 [20:20:59] New patchset: Ryan Lane; "Point recursor to virt0 for wmflabs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1911 [20:21:34] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1911 [20:21:34] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1911 [20:27:59] New patchset: Lcarr; "changing match condition for ganglia_aggregator" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1913 [20:28:06] !log changed recursor to point wmflabs domain to virt0 [20:28:08] Logged the message, Master [20:28:24] !log changed NS records for wmflabs.org and wmflabs to point to virt0 [20:28:26] Logged the message, Master [20:28:41] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1913 [20:28:41] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1913 [20:29:08] Jeremyb, there? [20:29:11] yah [20:29:27] Still needs Devanagari help? [20:29:52] If you can tell me what kind of help you need, I can try to find one. [20:30:11] Jeremyb ^ [20:30:12] New patchset: Lcarr; "Fixing cp1044" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1914 [20:30:14] Tanvir: it's still not fixed... i don't have time right now to figure out exactly what's still broken [20:30:30] Oh, okay. [20:30:52] Tanvir: so, https://bugzilla.wikimedia.org/show_bug.cgi?id=33507#c11 is still broke [20:31:08] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1914 [20:31:08] Change merged: Lcarr; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1914 [20:31:37] Tanvir: if you click the link it shows up ok (but with the wrong title?). but if you then try to visit it's talk page you get a page not found and they say it does already exist [20:32:16] I have no idea, but I will find one guy with that language capability and will tell them. [20:32:17] or, its* ? [20:32:44] !log reedy synchronized php-1.18/extensions/Contest/specials/ 'r108843' [20:32:46] Logged the message, Master [20:32:53] Tanvir: i think they're just going to say what i just did though. i think we're back to needing someone to dig in initialisesettings [20:33:08] oh, Reedy's here now. didn't notice he showed up [20:34:48] jeremyb, been here for an hour or so [20:35:01] 13 16:02:36 < jeremyb> i don't see a reedy, can someone take a look at that? [20:35:03] 13 19:22:29 -!- Reedy [~Reedy@109.224.134.228] has joined #wikimedia-tech [20:35:06] ;) [20:35:57] Reedy: about 33507. mutante did seem to fix something. but now i think there's another issue that's still broken? [20:36:13] * Reedy shrugs [20:36:34] Changing stuff in languages you really can't understand (rtl, non latin script) is hard [20:36:56] in langs you don't have fonts for is worse ;P [20:38:13] All bets are off in that case [20:38:16] * jeremyb runs away for a bit [20:40:26] !log stopped pdns on virt1 [20:40:28] Logged the message, Master [20:56:46] New patchset: Ryan Lane; "Fix mchenry's ldap client config" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1915 [20:57:02] New review: Ryan Lane; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/1915 [20:57:08] Change merged: Ryan Lane; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/1915 [21:04:06] nighty \o [21:17:31] !log killed pdns on virt1 [21:17:33] Logged the message, Master [21:40:59] !log changing virt1 to be a cname of virt0 [21:41:01] Logged the message, Master [21:45:53] Ryan_Lane: ugh! (last !log) [21:46:03] what? [21:46:13] the ttl for the NS record is 24 hours [21:46:30] oh, it's temporary? ok then :) [21:47:02] i thought virt1 would just never be used again. and virt1's body would become e.g. virt5 [21:52:38] apergos: where did we leave large res thumbnailing? multichill was just asking about it [21:52:48] apergos: should i file a bug? [21:53:06] yes [21:55:05] how weird that the wikibugs irc bot has a component but not the image scaling cluster... [21:56:18] hah, "ariel test glenn" [21:56:44] yeah, we were testing changes made to deal with the spammer [21:56:49] oh [21:57:22] I have my "regular" account over there (which is not disabled ;-) ) [21:57:34] i know :) [21:59:24] oh, and I'm the wrong person to get assigned image bugs btw [21:59:32] orly? [21:59:34] yes [21:59:42] i was going to just cc... [21:59:47] add me to cc is fine [21:59:49] but who is [22:00:07] that is a good question [22:00:12] apergos: So I have another rather large collection of images I plan to upload [22:00:24] total size in gb? [22:00:34] But all tiffs. I don't feel like generating thumbs myself so I was wondering how the tiff rendering is going [22:00:47] I have no idea [22:00:58] maybe we should have new keywords? some of glam,uploads,scaling,thumbs,etc. [22:01:02] in bz [22:01:03] I'm basically on the thumb server space train at this point [22:01:18] 3-4 TB? [22:01:22] good grief [22:01:24] really?? [22:01:26] hah [22:01:50] apergos: is the scaling stuff sufficiently puppetized that someone could do teh puppet changes in labs? [22:02:02] well [22:02:13] About 300.000 images. The files are 5-50MB [22:02:13] there's the pending move to swift which complicates things [22:02:38] So the 3TB is just a guess [22:02:49]