[00:05:05] New patchset: Andre Engels; "Test" [analytics] (master) - https://gerrit.wikimedia.org/r/2054 [00:18:17] !log uploaded new rsvg to apt.wikimedia.org, deploying to image scalers [00:18:18] Logged the message, Master [00:19:42] !log running apt-get upgrade on image scalers [00:19:43] Logged the message, Master [00:24:47] New review: Diederik; "Test works." [analytics] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2054 [00:24:47] Change merged: Diederik; [analytics] (master) - https://gerrit.wikimedia.org/r/2054 [00:40:35] PROBLEM - Apache HTTP on srv224 is CRITICAL: Connection refused [00:43:05] PROBLEM - Disk space on db43 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /var/lib/ureadahead/debugfs 0 MB (0% inode=88%): [00:44:34] PROBLEM - MySQL disk space on db43 is CRITICAL: DISK CRITICAL - free space: / 0 MB (0% inode=88%): /var/lib/ureadahead/debugfs 0 MB (0% inode=88%): [00:50:11] !log upgraded rsvg on all mediawiki-installation servers, for some reason it is installed on all of them [00:50:13] Logged the message, Master [00:54:06] !log tstarling synchronized wmf-config/CommonSettings.php [00:54:08] Logged the message, Master [00:54:57] !log tstarling synchronized wmf-config/InitialiseSettings.php 'new rsvg command line option' [00:54:59] Logged the message, Master [00:56:54] RECOVERY - Apache HTTP on srv224 is OK: HTTP OK - HTTP/1.1 301 Moved Permanently - 0.057 second response time [01:03:24] RECOVERY - Disk space on db43 is OK: DISK OK [01:04:34] RECOVERY - MySQL disk space on db43 is OK: DISK OK [01:18:27] gn8 folks [01:27:34] New patchset: Pyoungmeister; "invalid param :/" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2055 [01:27:52] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/2055 [01:28:44] New review: Pyoungmeister; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2055 [01:28:44] Change merged: Pyoungmeister; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2055 [01:32:08] !log catrope synchronized wmf-config/CommonSettings.php 'Add account creation throttle increase for bug 33900' [01:32:10] Logged the message, Master [01:55:39] !log fixed reverse dns for labs instances [01:55:41] Logged the message, Master [02:05:48] !log LocalisationUpdate completed (1.18) at Tue Jan 24 02:05:48 UTC 2012 [02:05:50] Logged the message, Master [02:15:22] PROBLEM - MySQL replication status on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 1585s [02:18:22] RECOVERY - Memcached on srv256 is OK: TCP OK - 0.001 second response time on port 11000 [02:22:02] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 1985s [02:30:32] RECOVERY - check_all_memcacheds on spence is OK: MEMCACHED OK - All memcacheds are online [02:42:22] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [02:45:22] RECOVERY - MySQL replication status on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [04:16:10] RECOVERY - Disk space on es1004 is OK: DISK OK [04:18:00] RECOVERY - MySQL disk space on es1004 is OK: DISK OK [04:44:30] PROBLEM - MySQL slave status on es1004 is CRITICAL: CRITICAL: Slave running: expected Yes, got No [05:24:26] PROBLEM - Puppet freshness on knsq9 is CRITICAL: Puppet has not run in the last 10 hours [08:47:25] PROBLEM - RAID on searchidx2 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [09:06:14] PROBLEM - Puppet freshness on cp1039 is CRITICAL: Puppet has not run in the last 10 hours [09:07:14] RECOVERY - RAID on searchidx2 is OK: OK: State is Optimal, checked 4 logical device(s) [09:08:14] PROBLEM - Puppet freshness on cp1037 is CRITICAL: Puppet has not run in the last 10 hours [09:09:14] PROBLEM - Puppet freshness on srv199 is CRITICAL: Puppet has not run in the last 10 hours [09:14:14] PROBLEM - Puppet freshness on cp1038 is CRITICAL: Puppet has not run in the last 10 hours [09:16:14] PROBLEM - Misc_Db_Lag on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 1397s [09:16:24] PROBLEM - MySQL replication status on storage3 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 1407s [09:26:06] PROBLEM - MySQL replication status on db1025 is CRITICAL: CHECK MySQL REPLICATION - lag - CRITICAL - Seconds_Behind_Master : 1989s [09:36:46] PROBLEM - Puppet freshness on cp1040 is CRITICAL: Puppet has not run in the last 10 hours [09:46:46] PROBLEM - Puppet freshness on cp1036 is CRITICAL: Puppet has not run in the last 10 hours [09:58:46] PROBLEM - Disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 455006 MB (3% inode=99%): [10:05:26] PROBLEM - MySQL disk space on es1004 is CRITICAL: DISK CRITICAL - free space: /a 391287 MB (3% inode=99%): [10:06:16] RECOVERY - MySQL replication status on db1025 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [10:19:34] PROBLEM - Puppet freshness on db43 is CRITICAL: Puppet has not run in the last 10 hours [11:04:14] RECOVERY - MySQL slave status on es1004 is OK: OK: [12:15:53] PROBLEM - Puppet freshness on mw1115 is CRITICAL: Puppet has not run in the last 10 hours [14:41:28] hey! [15:33:54] PROBLEM - Puppet freshness on knsq9 is CRITICAL: Puppet has not run in the last 10 hours [15:50:26] are some images supposed to be broken? [15:54:09] [[en:w:File:USS_Enterprise_(NCC-1701),_ENT1231.jpg]] seems to be broken for me. [15:54:13] * FAdmArcher will brb [16:04:58] FAdmArcher|away, did it work recently? [16:05:21] I dunno, discussion on https://en.wikipedia.org/wiki/Wikipedia_talk:WikiProject_Star_Trek#Broken_image.3F [16:05:54] I'm assuming yes [16:07:40] apergos, ^ [16:32:26] it's in the snapshots [16:32:29] that's really weird [16:33:46] Ugh [16:34:51] care to try loading it now? [16:35:44] wfm [16:35:47] Inded [16:35:48] now, sure [16:36:06] so the reporter could bugzilla it and I could confirm that it was missing on the filesystem and restred from snapshot [16:36:17] after that there's not much I can do [16:36:27] except hope it's not the result of a bug in the code [16:43:51] !log reedy synchronized wmf-config/InitialiseSettings.php 'Fix PHP Notice: Undefined variable: wmgMFCustomLogos in /home/wikipedia/common/wmf-config/CommonSettings.php on line 2354' [16:46:06] !log Creatd wikilove tables on fawiki and fawiktionary [16:47:47] !log reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33541 - activating WikiLove for fa.wiktionary and fa.wikipedia' [16:48:36] :( [16:50:26] RECOVERY - check_job_queue on spence is OK: JOBQUEUE OK - all job queues below 10,000 [16:58:59] Thanks Reedy and apergos it works now :) [16:59:06] sure [16:59:21] so the bad news is that it was actually gone from teh disks, I restored it form a snapshot [17:14:17] RECOVERY - Misc_Db_Lag on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [17:15:12] !log reedy synchronized wmf-config/InitialiseSettings.php 'Bug 33541 - activating WikiLove for fa.wiktionary and fa.wikipedia' [17:17:47] RECOVERY - MySQL replication status on storage3 is OK: CHECK MySQL REPLICATION - lag - OK - Seconds_Behind_Master : 0s [17:18:40] if nobody cares about storage3 replication locking up, can we make it test for 'replication not lagging for more than 1 day' or so [17:18:42] :) [17:27:50] * Dmcdevit pokes Reedy. [17:27:57] run! [17:29:31] Reedy: I think Katie talked to you yesterday about a server-side upload I was hoping to get done today? [17:29:42] yeah [17:30:10] I have a dumb question first, though. Is there a way I can search for a file somewhere on a FTP server? I can't seem to find where odder put it. [17:32:06] ^ Reedy [17:32:53] Errr [17:32:54] Not easily [17:33:49] It's in his folder space on tools.wikimedia.pl, but I was on a different computer when he gave me the path. [17:34:48] http://tools.wikimedia.pl/~odder/ [17:35:17] It doesn't seem to be in any of those public files. [17:50:38] Reedy: On second thought, it turns out this isn't as time-sensitive as I was originally led to believe. I probably don't need to worry about it until next week. Thanks for putting up with me, though. [17:50:50] lol [17:50:51] ook [18:20:28] RECOVERY - Host ms6 is UP: PING OK - Packet loss = 0%, RTA = 109.64 ms [18:22:58] PROBLEM - Puppet freshness on ms6 is CRITICAL: Puppet has not run in the last 10 hours [18:35:23] New patchset: Catrope; "Moving stuff over from analytics.git/reportcard" [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2056 [18:36:58] PROBLEM - Disk space on srv221 is CRITICAL: DISK CRITICAL - free space: / 226 MB (3% inode=60%): /var/lib/ureadahead/debugfs 226 MB (3% inode=60%): [18:37:28] New patchset: Catrope; "Delete stuff that's been moved to analytics/reportcard.git" [analytics] (master) - https://gerrit.wikimedia.org/r/2057 [18:38:48] RECOVERY - Puppet freshness on ms6 is OK: puppet ran at Tue Jan 24 18:38:40 UTC 2012 [18:40:10] New patchset: Catrope; "Add dummy README file" [analytics] (master) - https://gerrit.wikimedia.org/r/2058 [18:43:08] PROBLEM - Host cp1036 is DOWN: PING CRITICAL - Packet loss = 100% [18:43:08] PROBLEM - Host cp1037 is DOWN: PING CRITICAL - Packet loss = 100% [18:43:18] PROBLEM - Host cp1038 is DOWN: PING CRITICAL - Packet loss = 100% [18:43:28] PROBLEM - Host cp1040 is DOWN: PING CRITICAL - Packet loss = 100% [18:44:28] PROBLEM - Host cp1039 is DOWN: PING CRITICAL - Packet loss = 100% [18:44:37] New patchset: ArielGlenn; "clean up tmp on imagescalers more aggressively" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2059 [18:46:18] New patchset: Bhartshorne; "adding country filters for nimish RT-2260" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2060 [18:46:47] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2060 [18:46:48] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2060 [18:47:02] New review: ArielGlenn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2059 [18:47:02] Change merged: ArielGlenn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2059 [18:48:46] New review: Diederik; "Ok." [analytics/reportcard] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2056 [18:48:46] Change merged: Diederik; [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2056 [18:50:08] New patchset: ArielGlenn; "hmm, guess I like */5 better" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2061 [18:50:50] New review: ArielGlenn; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2061 [18:50:50] Change merged: ArielGlenn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2061 [18:51:32] New patchset: Diederik; "Adding readme file" [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2062 [18:52:02] New review: Diederik; "(no comment)" [analytics/reportcard] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2062 [18:52:02] Change merged: Diederik; [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2062 [18:56:38] PHP Notice: Undefined variable: wmgMFCustomLogos in /home/wikipedia/common/wmf-config/CommonSettings.php on line 2354 [19:01:53] apergos, is that still going on? [19:02:19] was showing when trying to use sql foobarwiki [19:02:25] from a couple hours back [19:02:39] I was sseeing it in someone's cron job [19:02:44] Ah [19:02:53] need to poke patrick about it later [19:03:05] mmmmm [19:04:08] looks like sql is working ok now [19:06:03] there was no default set in the config for the wmgMFCustomLoogos [19:06:49] RECOVERY - Puppet freshness on db43 is OK: puppet ran at Tue Jan 24 19:06:43 UTC 2012 [19:08:24] New patchset: Catrope; "Add .gitreview file" [analytics/udp-filters] (master) - https://gerrit.wikimedia.org/r/2063 [19:08:41] New patchset: Asher; "db43 -> s6" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2064 [19:09:37] New review: Diederik; "Ok." [analytics/udp-filters] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2063 [19:09:37] Change merged: Diederik; [analytics/udp-filters] (master) - https://gerrit.wikimedia.org/r/2063 [19:10:10] New review: Diederik; "Ok." [analytics] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2058 [19:10:10] Change merged: Diederik; [analytics] (master) - https://gerrit.wikimedia.org/r/2058 [19:10:28] New review: Diederik; "Ok." [analytics] (master); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2057 [19:10:28] Change merged: Diederik; [analytics] (master) - https://gerrit.wikimedia.org/r/2057 [19:10:29] RECOVERY - DPKG on db43 is OK: All packages OK [19:11:01] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2064 [19:11:02] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2064 [19:16:17] New patchset: Asher; "move db43 to fully puppetized mysql conf" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2065 [19:16:42] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2065 [19:16:42] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2065 [19:18:33] !log tstarling synchronized wmf-config/InitialiseSettings.php 'switched to Preprocessor_Hash on ocwiki only' [19:18:34] Logged the message, Master [19:18:59] PROBLEM - Puppet freshness on srv199 is CRITICAL: Puppet has not run in the last 10 hours [19:19:49] RECOVERY - Disk space on srv221 is OK: DISK OK [19:30:20] RECOVERY - Puppet freshness on srv199 is OK: puppet ran at Tue Jan 24 19:30:12 UTC 2012 [19:33:33] !log asher synchronized wmf-config/db.php 'adding db53 as an enwiki slave at 1/4 normal weight' [19:33:34] Logged the message, Master [19:37:04] !log asher synchronized wmf-config/db.php 'raising db53 weight to 200' [19:37:05] Logged the message, Master [19:48:40] PROBLEM - udp2log processes on emery is CRITICAL: CRITICAL: filters absent: /var/log/squid/filters/countries-100, /var/log/squid/filters/countries-10, /var/log/squid/filters/countries-1, [20:01:36] !log asher synchronized wmf-config/db.php 'raising db53 weight to 400' [20:01:38] Logged the message, Master [20:06:29] !log asher synchronized wmf-config/db.php 'adding db43 back to s6 at a low weight' [20:06:31] Logged the message, Master [20:09:33] !log asher synchronized wmf-config/db.php 're-weighting s6 dbs' [20:09:34] Logged the message, Master [20:14:08] RECOVERY - udp2log processes on emery is OK: OK: all filters present [20:44:08] PROBLEM - udp2log processes on emery is CRITICAL: CRITICAL: filters absent: /var/log/squid/filters/countries-100, /var/log/squid/filters/countries-10, /var/log/squid/filters/countries-1, [20:48:05] !log asher synchronized wmf-config/db.php 'pulling db26 from s1 to reimage' [20:48:07] Logged the message, Master [21:11:48] PROBLEM - RAID on db26 is CRITICAL: Connection refused by host [21:20:46] RECOVERY - RAID on db26 is OK: OK: 1 logical device(s) checked [21:23:03] New patchset: Asher; "rebuilding db26 - enwiki pmtpa snapshot host" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2066 [21:23:25] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2066 [21:23:26] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2066 [21:26:13] New patchset: Lcarr; "Adding in the sw repo as well as symlinking it in files for ease of puppet pulling" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2067 [21:29:43] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: -1; - https://gerrit.wikimedia.org/r/2067 [21:30:00] New patchset: Asher; "rethinking db26 as a snapshot host" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2068 [21:30:18] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2068 [21:30:20] New review: Asher; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2068 [21:30:20] Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2068 [21:37:50] New patchset: Bhartshorne; "removing recently installed country filters - they're crashing" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2069 [21:38:28] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2069 [21:38:29] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2069 [21:40:56] PROBLEM - DPKG on db55 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [21:42:06] RECOVERY - udp2log processes on emery is OK: OK: all filters present [21:45:16] PROBLEM - DPKG on db56 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [22:06:20] New review: Lcarr; "(no comment)" [operations/puppet] (production); V: 0 C: 0; - https://gerrit.wikimedia.org/r/2067 [22:09:12] zzz =_= [22:22:00] New patchset: Lcarr; "Adding in the sw repo as well as symlinking it in files for ease of puppet pulling" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2067 [22:28:03] hey is our old IMAP mail server acting up? I can't connect to it starting about 30-60 min ago. [22:29:35] PROBLEM - Puppet freshness on mw1115 is CRITICAL: Puppet has not run in the last 10 hours [22:32:01] mark, you're doing stuff to sanger, right? [22:32:12] yeah i'm upgrading it [22:32:25] stuwest, maintenance! [22:32:44] since basically noone but ariel, you, me and some board members use it, I just did it ;) [22:32:53] ahh, maintenance. and now I see on twitter. thx. [22:32:53] sorry stu ;-) [22:33:07] the rest moved off to google apps [22:33:11] the joys of being a lagging adopter. [22:33:18] well [22:33:24] you have a kickass imap server all to yourself ;-p [22:33:40] i'm happy to switch to google, btw. i already use it for 2-3 other accounts! whatever works. [22:33:42] New patchset: Lcarr; "Adding in the sw repo as well as making it available via fileserver.conf" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2067 [22:33:49] up to you [22:33:58] we keep the imap server around, I won't switch to google [22:34:31] well now I feel special on the old server so let's leave as is for now. nothing really requiring shift to google i suppose. [22:34:33] me neither [22:35:05] RECOVERY - Puppet freshness on mw1115 is OK: puppet ran at Tue Jan 24 22:35:00 UTC 2012 [22:35:30] New patchset: Lcarr; "Adding in the sw repo as well as making it available via fileserver.conf" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2067 [22:35:38] hmm it's broken right now [22:40:18] New review: Mark Bergsma; "(no comment)" [operations/puppet] (production); V: 0 C: 2; - https://gerrit.wikimedia.org/r/2067 [22:40:18] Change merged: Mark Bergsma; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2067 [22:53:31] mark: i'm not certain this is a regression but it's definitely a bug: look at the charsets on https://lists.wikimedia.org/mailman/listinfo , in particular the zh and uk lists. both HTTP header and say us-ascii [23:18:20] New patchset: Bhartshorne; "moving swift passwords from the public to private git repos" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2071 [23:19:25] New patchset: Bhartshorne; "moving swift passwords from the public to private git repos" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2071 [23:20:04] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2071 [23:20:05] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2071 [23:22:19] New patchset: Lcarr; "addingin cron job to sync software repo in labs" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2072 [23:27:05] New patchset: Ottomata; "launcher.py, pipeline.py - added some documentation, more to come" [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2073 [23:28:23] !log Syncing prod CiviCRM on aluminium to r1209 [23:28:24] Logged the message, Master [23:30:13] !log testing [23:30:15] Logged the message, Master [23:36:06] New patchset: Ottomata; "user_agent.py - added little documentation, this is mainly a test of git push" [analytics/reportcard] (master) - https://gerrit.wikimedia.org/r/2074 [23:37:40] New review: Bhartshorne; "(no comment)" [analytics/reportcard] (master) C: 1; - https://gerrit.wikimedia.org/r/2074 [23:56:36] New patchset: Bhartshorne; "creating configs for the production swift cluster on ms-fe1 and 2" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2075 [23:57:31] Hello everybody, is it possible to upload a video file of 101.8 MB on commons or the systeme will block at 100 MB exactly ? [23:59:13] New review: Bhartshorne; "(no comment)" [operations/puppet] (production); V: 1 C: 2; - https://gerrit.wikimedia.org/r/2075 [23:59:13] Change merged: Bhartshorne; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/2075