[00:01:46] <logmsgbot>	 !log bsitu synchronized php-1.22wmf3/includes/OutputPage.php  'Adding GetNewMessagesAlert hook and wgUserNewMsgRevisionId JS global'
[00:01:54] <morebots>	 Logged the message, Master
[00:02:29] <logmsgbot>	 !log bsitu synchronized php-1.22wmf3/includes/Skin.php  'Adding GetNewMessagesAlert hook and wgUserNewMsgRevisionId JS global'
[00:02:37] <morebots>	 Logged the message, Master
[00:02:59] <logmsgbot>	 !log bsitu synchronized php-1.22wmf3/includes/User.php  'Adding GetNewMessagesAlert hook and wgUserNewMsgRevisionId JS global'
[00:03:06] <morebots>	 Logged the message, Master
[00:03:13] <aude>	 ah, you know about the error ....
[00:03:23] <aude>	 PHP fatal error in /usr/local/apache/common-local/php-1.22wmf3/includes/OutputPage.php line 3071:
[00:03:25] <aude>	 Call to undefined method User::getNewMessageRevisionId()
[00:03:35] <aude>	 got it on a talk pge
[00:03:36] <StevenW>	 +1 to aude's error
[00:03:36] <aude>	 page
[00:03:53] <Susan>	 It should be better now.
[00:03:53] <kaldari>	 we jsut caused a site outage for about 30 seconds
[00:04:07] <kaldari>	 should be fine now
[00:04:09] <aude>	 yikes
[00:04:13] <aude>	 panic :)
[00:04:27] <bsitu>	 aude: I didn't realize I need to sync the file in a particular order
[00:04:29] <ori-l>	 mutante: oops, meant https://gerrit.wikimedia.org/r/#/c/62530/
[00:04:29] <kaldari>	 go the file sync order wrong
[00:04:38] <aude>	 ok, well all is good now
[00:05:20] <logmsgbot>	 !log bsitu synchronized php-1.22wmf3/includes/WikiPage.php  'Adding GetNewMessagesAlert hook and wgUserNewMsgRevisionId JS global'
[00:05:28] <morebots>	 Logged the message, Master
[00:05:53] <logmsgbot>	 !log bsitu synchronized php-1.22wmf3/RELEASE-NOTES-1.22  'Adding GetNewMessagesAlert hook and wgUserNewMsgRevisionId JS global'
[00:06:01] <morebots>	 Logged the message, Master
[00:06:22] <logmsgbot>	 !log bsitu synchronized php-1.22wmf3/docs/hooks.txt  'Adding GetNewMessagesAlert hook and wgUserNewMsgRevisionId JS global'
[00:06:30] <morebots>	 Logged the message, Master
[00:06:53] <logmsgbot>	 !log bsitu synchronized php-1.22wmf3/extensions/Echo  'Update Echo to master'
[00:07:00] <morebots>	 Logged the message, Master
[00:27:27] <icinga-wm_>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[00:28:17] <icinga-wm_>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.127 second response time
[00:29:53] <Reedy>	 Uhhh
[00:30:03] <Reedy>	 Who just deployed something?
[00:30:03] <Reedy>	     441 PHP Fatal error:  Call to a member function getId() on a non-object in /usr/local/apache/common-local/php-1.22wmf3/includes/User.php on line 1847
[00:30:20] <Reedy>	 bsitu: ^^
[00:31:09] <Reedy>	                         if ( count( $newMessageLinks ) === 1 && $newMessageLinks[0]['wiki'] === wfWikiID() ) {
[00:31:09] <Reedy>	                                 $newMessageRevision = $newMessageLinks[0]['rev'];
[00:31:09] <Reedy>	                                 $newMessageRevisionId = $newMessageRevision->getId();
[00:31:29] <bsitu>	 Reedy: looking now
[00:32:56] <bsitu>	 Reedy: double checking with kaldari now
[00:34:50] <Reedy>	 		$rev = $timestamp ? Revision::loadFromTimestamp( $dbr, $utp, $timestamp ) : null;
[00:34:50] <Reedy>	 		return array( array( 'wiki' => wfWikiID(), 'link' => $utp->getLocalURL(), 'rev' => $rev ) );
[00:35:24] <Reedy>	 When rev can end up as null in your calling code, you should really be checking it's value when you try and use it ie $newMessageRevision = $newMessageLinks[0]['rev'];
[00:36:41] <Reedy>	 if ( count( $newMessageLinks ) === 1 && $newMessageLinks[0]['wiki'] === wfWikiID() ) {
[00:36:55] <Reedy>	 if ( count( $newMessageLinks ) === 1 && $newMessageLinks[0]['wiki'] === wfWikiID() && $newMessageLinks[0]['rev'] !== null ) {
[00:37:59] <logmsgbot>	 !log reedy synchronized php-1.22wmf3/includes/User.php  'Live hack to stop fatals'
[00:38:07] <morebots>	 Logged the message, Master
[00:39:28] <kaldari>	 bsitu: https://gerrit.wikimedia.org/r/#/c/62554/
[00:41:11] <gerrit-wm>	 New patchset: Dr0ptp4kt; "Adding Mobilink Pakistan IP addresses to Varnish VCL." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62555
[00:47:51] <bsitu>	 Reedy: thx for the live hack,
[00:47:54] <gerrit-wm>	 New patchset: Dr0ptp4kt; "Constrain default language list for Dialog Sri Lanka to en, ta, si." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62460
[00:48:10] <bsitu>	 Reedy: I am going to merge this change: https://gerrit.wikimedia.org/r/#/c/62554/
[00:48:33] <gerrit-wm>	 New patchset: Dr0ptp4kt; "Adding Mobilink Pakistan IP addresses to Varnish VCL." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62555
[00:56:07] <gerrit-wm>	 New patchset: Dr0ptp4kt; "Adding Mobilink Pakistan IP addresses to Varnish VCL." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62555
[01:28:22] <logmsgbot>	 !log reedy synchronized php-1.22wmf3/extensions/WikimediaMaintenance
[01:28:30] <morebots>	 Logged the message, Master
[01:29:10] <logmsgbot>	 !log bsitu synchronized php-1.22wmf3/includes/User.php  'Two fixes to User::getNewMessageRevisionId() from kaldari'
[01:29:18] <morebots>	 Logged the message, Master
[02:16:06] <logmsgbot>	 !log LocalisationUpdate completed (1.22wmf3) at Tue May  7 02:16:06 UTC 2013
[02:16:14] <morebots>	 Logged the message, Master
[02:18:00] <yurik>	 Reedy, could you +2 https://gerrit.wikimedia.org/r/#/c/62457/
[02:18:17] <yurik>	 i will need to deploy something early morning tomorrow (with max's help)
[02:18:26] <Reedy>	 Unfortunately not, I'm not Ops
[02:18:38] <yurik>	 oh, i keep forgeting who is ops and who is not :)
[02:19:51] <yurik>	 LeslieCarr, ping?
[02:19:57] <gerrit-wm>	 New patchset: Reedy; "Delete all the superfluous wikimania docroots" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62565
[02:20:09] <icinga-wm_>	 PROBLEM - Puppet freshness on db45 is CRITICAL: No successful Puppet run in the last 10 hours
[02:25:12] <logmsgbot>	 !log LocalisationUpdate completed (1.22wmf2) at Tue May  7 02:25:11 UTC 2013
[02:25:20] <morebots>	 Logged the message, Master
[02:26:01] <gerrit-wm>	 New patchset: Reedy; "WIP: Simplify wikimania apache conf" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/62566
[02:34:12] <gerrit-wm>	 New patchset: Reedy; "WIP: Simplify wikimania apache conf" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/62566
[02:35:03] <gerrit-wm>	 New patchset: Reedy; "Simplify wikimania apache conf" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/62566
[02:37:54] <gerrit-wm>	 New patchset: Reedy; "Simplify wikimania apache conf" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/62566
[02:41:51] <Reedy>	 TimStarling: Any reason I can't just move /mnt/upload7/wikipedia/wikimania to /mnt/upload7/wikipedia/wikimania2005, and then symlink wikimania back to wikimania2005 to make them consistent?
[02:42:28] <Reedy>	 It comes to a massive 27M, and I suspect very little of it is actually in the upload squids, so isn't an issue from that perspective
[02:44:35] <Reedy>	 Same on upload6 for consistency
[02:44:36] <gerrit-wm>	 New review: Reedy; "(1 comment)" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/62566
[02:46:15] <Reedy>	 Or even just do the reverse. Create a wikimania2005 symlink that points at wikimania
[02:50:09] * Reedy  just creates the symlink

[02:51:58] <gerrit-wm>	 New patchset: Reedy; "Simplify wikimania apache conf" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/62566
[02:54:28] <TimStarling>	 you probably need to fix the DB name detection in CommonSettings.php if you change the document root
[02:56:21] <Reedy>	 'docRoot' => $_SERVER['DOCUMENT_ROOT'],
[02:56:49] <Reedy>	 Though, SiteConfiguration doesn't use docRoot
[02:57:22] <Krinkle>	 Hm.. wikitech is still on 1.21wmf?
[02:57:39] <icinga-wm_>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[02:57:41] <Reedy>	 Ryan updates it semi periodically
[02:59:29] <icinga-wm_>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time
[03:01:45] <TimStarling>	 Reedy: actually it has moved to multiversion/MWMultiVersion.php now
[03:01:57] <TimStarling>	 but it's still the same code
[03:02:16] <TimStarling>	 			} elseif ( preg_match( "/^\/usr\/local\/apache\/(?:htdocs|common\/docroot)\/([a-z0-9\-_]*)$/", $docRoot, $matches ) ) {
[03:02:16] <TimStarling>	 				$site = "wikipedia";
[03:02:16] <TimStarling>	 				$lang = $matches[1];
[03:05:01] <Reedy>	 Probably should review/merge https://gerrit.wikimedia.org/r/#/c/62461 before making any further changes to save conflicts. And/or depend on that revision
[03:07:44] <icinga-wm_>	 PROBLEM - Puppet freshness on cp3003 is CRITICAL: No successful Puppet run in the last 10 hours
[03:07:44] <icinga-wm_>	 PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours
[03:12:05] <Reedy>	 Shame wikimania don't use their own TLD
[03:13:43] <Reedy>	 I'll look at that post sleep/before attempting to deploy either of those revisions. Thanks
[03:14:12] <gerrit-wm>	 New review: Reedy; "I'll deploy this post sleep if it hasn't been done already" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62461
[03:18:25] <gerrit-wm>	 New patchset: Krinkle; "Add secure.wikimedia.org (old SSL site) redirects to cluster" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/60934
[03:18:43] <gerrit-wm>	 New review: Krinkle; "<dzahn> adds DocumentRoot to existing docroot/secure on cluster, deletes conflicting wikimedia-ssl-b..." [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/60934
[03:19:01] <gerrit-wm>	 New review: Krinkle; "<dzahn> forgot to also remove old config from all.conf" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/60934
[03:19:24] <gerrit-wm>	 New review: Krinkle; "Removed patch set iteration/discussion from the commit message." [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/60934
[03:22:18] <Krinkle>	 Reedy: 2013.wikimania.org, interesting. that *would* be nice indeed.
[03:22:54] <Krinkle>	 I wonder if anything fails if sub/lang contains and starts with only numbers
[03:23:01] <Krinkle>	 would be a good test :)
[03:35:21] <gerrit-wm>	 New patchset: Krinkle; "gerrit: Cleanup/refactor logging" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/8120
[03:46:18] <icinga-wm_>	 PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours
[03:57:08] <paravoid>	 Reedy: upload6? upload7?
[04:19:39] <logmsgbot>	 !log LocalisationUpdate ResourceLoader cache refresh completed at Tue May  7 04:19:39 UTC 2013
[04:19:47] <morebots>	 Logged the message, Master
[04:40:02] <icinga-wm_>	 PROBLEM - Puppet freshness on db44 is CRITICAL: No successful Puppet run in the last 10 hours
[05:52:11] <gerrit-wm>	 New review: Hydriz; "Reasonable, though I think people should always check carefully before committing changes :D" [operations/mediawiki-config] (master) C: 1;  - https://gerrit.wikimedia.org/r/61428
[06:12:21] <icinga-wm_>	 PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours
[06:27:54] <gerrit-wm>	 New review: Ori.livneh; "LGTM; as a deployer you have the right to +2 your own change, so I'm only +1ing it, to give you a ch..." [operations/mediawiki-config] (master) C: 1;  - https://gerrit.wikimedia.org/r/62446
[06:47:13] <gerrit-wm>	 Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62460
[06:47:43] <gerrit-wm>	 New patchset: Faidon; "Adding Mobilink Pakistan IP addresses to Varnish VCL." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62555
[06:48:19] <gerrit-wm>	 Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62555
[06:59:30] <TimStarling>	 !log on holmium: chown'd to root all script files writable by www-data and all directories which are configured to allow PHP execution
[06:59:38] <morebots>	 Logged the message, Master
[07:21:23] <hashar>	 hello :-D
[07:21:24] <hashar>	 apergos: i am awake!
[07:21:33] <apergos>	 congrats!
[07:21:44] <apergos>	 I did not get my gsoc stuff done last night, it was too long a day already
[07:21:51] <apergos>	 so I am doing it now
[07:22:07] <apergos>	 we can check in though about today
[07:22:18] <TimStarling>	 !log on holmium: added some ad hoc iptables rules intended to keep the rest of the site mostly safe from it overnight
[07:22:26] <morebots>	 Logged the message, Master
[07:22:42] <hashar>	 apergos: I will look at the puppet manifests this morning and might attempt something in labs and see how it goes
[07:22:50] <apergos>	 ah ok
[07:26:35] <icinga-wm_>	 PROBLEM - HTTP on holmium is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 722 bytes in 7.051 second response time
[07:26:58] <TimStarling>	 !log on holmium: reverted firewall, too hard
[07:27:06] <morebots>	 Logged the message, Master
[07:27:21] <hashar>	 good nagios!
[07:27:35] <icinga-wm_>	 RECOVERY - HTTP on holmium is OK: HTTP OK: HTTP/1.1 200 OK - 86489 bytes in 0.509 second response time
[07:41:05] <gerrit-wm>	 New review: Hashar; "I have no ideas about the impact of this in production.  Anyway whenever you get it merged, Jenkins ..." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62446
[07:46:04] <icinga-wm_>	 PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours
[07:46:04] <icinga-wm_>	 PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours
[07:46:04] <icinga-wm_>	 PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours
[07:46:04] <icinga-wm_>	 PROBLEM - Puppet freshness on virt1005 is CRITICAL: No successful Puppet run in the last 10 hours
[07:48:04] <icinga-wm_>	 RECOVERY - search indices - check lucene status page on search35 is OK: HTTP OK: HTTP/1.1 200 OK - 747 bytes in 0.055 second response time
[07:58:24] <icinga-wm_>	 PROBLEM - DPKG on ms-fe1002 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[07:59:14] <paravoid>	 !log partially upgrading ceph to 0.56.6
[07:59:23] <morebots>	 Logged the message, Master
[08:01:28] <gerrit-wm>	 Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62347
[08:03:24] <icinga-wm_>	 RECOVERY - DPKG on ms-fe1002 is OK: All packages OK
[08:05:43] <gerrit-wm>	 New patchset: Nikerabbit; "ULS configuration for beta labs" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62446
[08:06:18] <gerrit-wm>	 Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62446
[08:20:31] <paravoid>	 !log ceph: all radosgw & mon + osd.24 upgraded to 0.56.6
[08:20:39] <morebots>	 Logged the message, Master
[08:27:41] <icinga-wm_>	 PROBLEM - SSH on mc15 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:28:31] <icinga-wm_>	 RECOVERY - SSH on mc15 is OK: SSH OK - OpenSSH_5.9p1 Debian-5ubuntu1.1 (protocol 2.0)
[08:30:41] <icinga-wm_>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[08:31:31] <icinga-wm_>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 4.462 second response time
[08:43:11] <gerrit-wm>	 New patchset: Yurik; "Mobile Zero extension shared configuration settings" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62576
[08:44:57] <hashar>	 paravoid: hi. A quick enquiry about the protoproxy (used for nginx ssl termination),  That is a few files and I am willing to make it a puppet module named 'protoproxy' does that sound like a good idea to you ?  (ping apergos as well)
[08:46:10] <hashar>	 or maybe I can make it a role class grin
[08:49:22] <apergos>	 at least some of the protoproxy stuff should go into a module, I agree
[08:50:03] <hashar>	 I am doing that
[08:50:20] <icinga-wm_>	 PROBLEM - Host wikimedia-lb.esams.wikimedia.org_ipv6 is DOWN: PING CRITICAL - Packet loss = 100%
[08:50:21] <icinga-wm_>	 PROBLEM - Host upload-lb.esams.wikimedia.org_ipv6 is DOWN: PING CRITICAL - Packet loss = 100%
[08:51:20] <icinga-wm_>	 RECOVERY - Host wikimedia-lb.esams.wikimedia.org_ipv6 is UP: PING WARNING - Packet loss = 80%, RTA = 86.90 ms
[08:51:22] <icinga-wm_>	 RECOVERY - Host upload-lb.esams.wikimedia.org_ipv6 is UP: PING WARNING - Packet loss = 93%, RTA = 87.89 ms
[08:51:40] <icinga-wm_>	 PROBLEM - LVS HTTPS IPv6 on wikibooks-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection timed out
[08:52:10] <icinga-wm_>	 PROBLEM - LVS HTTPS IPv6 on mediawiki-lb.esams.wikimedia.org_ipv6 is CRITICAL: Connection timed out
[08:52:31] <icinga-wm_>	 RECOVERY - LVS HTTPS IPv6 on wikibooks-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 66326 bytes in 0.697 second response time
[08:53:10] <icinga-wm_>	 RECOVERY - LVS HTTPS IPv6 on mediawiki-lb.esams.wikimedia.org_ipv6 is OK: HTTP OK: HTTP/1.1 200 OK - 66329 bytes in 0.703 second response time
[09:09:12] <logmsgbot>	 !log nikerabbit synchronized php-1.22wmf3/extensions/UniversalLanguageSelector/  'ULS to master'
[09:09:20] <morebots>	 Logged the message, Master
[09:13:58] <logmsgbot>	 !log nikerabbit synchronized wmf-config/InitialiseSettings.php  'New ULS config part 1'
[09:14:06] <morebots>	 Logged the message, Master
[09:15:01] <logmsgbot>	 !log nikerabbit synchronized wmf-config/CommonSettings.php  'New ULS config part 2'
[09:15:08] <morebots>	 Logged the message, Master
[09:28:00] <Nikerabbit>	 action=parse apparently broken on wikidata: https://bugzilla.wikimedia.org/48201
[09:29:39] <aude>	 Nikerabbit: it needs "contentmodel=wikitext"
[09:30:04] <aude>	 appears to assume json content model, perhaps because items are in the main namespace
[09:30:24] <Nikerabbit>	 aude: I would consider that as breaking change
[09:32:29] <aude>	 sure
[10:10:27] <paravoid>	 apergos: any news on those ms-be disks?
[10:10:51] <apergos>	 they didn't have them on site, in theory they would show up today
[10:21:21] <gerrit-wm>	 New patchset: Hashar; "proxy_configuration had a typo" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62581
[10:57:45] <gerrit-wm>	 New patchset: Hashar; "lame work in progress" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62582
[10:59:31] <gerrit-wm>	 New patchset: Hashar; "lame work in progress" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62582
[11:18:54] <gerrit-wm>	 New patchset: Hashar; "lame work in progress" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62582
[11:22:24] <icinga-wm_>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[11:23:14] <icinga-wm_>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.125 second response time
[11:41:43] <gerrit-wm>	 New patchset: Hashar; "lame work in progress" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62582
[11:41:43] <gerrit-wm>	 New patchset: Hashar; "nginx_site now supports boolean values" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62583
[11:48:43] <icinga-wm_>	 PROBLEM - Auth DNS on ns1.wikimedia.org is CRITICAL: CRITICAL - Plugin timed out while executing system call
[12:15:56] <gerrit-wm>	 New patchset: Hashar; "lame work in progress" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62582
[12:20:54] <icinga-wm_>	 PROBLEM - Puppet freshness on db45 is CRITICAL: No successful Puppet run in the last 10 hours
[12:21:24] <icinga-wm_>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[12:22:14] <icinga-wm_>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.128 second response time
[12:57:44] <icinga-wm_>	 PROBLEM - Disk space on mc15 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds.
[12:58:44] <icinga-wm_>	 RECOVERY - Disk space on mc15 is OK: DISK OK
[13:08:17] <icinga-wm_>	 PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours
[13:08:17] <icinga-wm_>	 PROBLEM - Puppet freshness on cp3003 is CRITICAL: No successful Puppet run in the last 10 hours
[13:47:07] <icinga-wm_>	 PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours
[13:53:00] <gerrit-wm>	 New patchset: Mark Bergsma; "Add AFTv5 archive feedback cron job" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62602
[13:58:14] <gerrit-wm>	 New patchset: Mark Bergsma; "Add AFTv5 archive feedback cron job" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62602
[14:01:53] <gerrit-wm>	 New patchset: Hashar; "labs: hardcode nginx types_hash_bucket_size to 64" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62603
[14:03:36] <gerrit-wm>	 New patchset: Hashar; "lame work in progress" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62582
[14:11:24] <gerrit-wm>	 New patchset: Hashar; "labs: hardcode nginx types_hash_bucket_size to 64" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62603
[14:11:24] <gerrit-wm>	 New patchset: Hashar; "lame work in progress" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62582
[14:22:07] <mlitn>	 hi; anyone around who has a clue why ee-prototype (labs instance) tells me it "Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock'"?
[14:29:55] <petan>	 mlitn it seems that mysql is down on that box?
[14:30:25] <petan>	 just check if it's running and if not, eh, that would answer your question
[14:30:42] <gerrit-wm>	 New patchset: Hashar; "beta: reenable commons as a default foreign file repo." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62606
[14:31:31] <gerrit-wm>	 New patchset: Hashar; "beta: reenable commons as a default foreign file repo." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62606
[14:32:38] <cmjohnson1>	 !log authdns update
[14:32:47] <morebots>	 Logged the message, Master
[14:34:20] <gerrit-wm>	 New review: Hashar; "I have no idea how the foreign repo stuff works. I guess it is supposed to make the beta commons wik..." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62606
[14:34:48] <gerrit-wm>	 New patchset: Hashar; "beta: reenable commons as a default foreign file repo." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62606
[14:35:57] <gerrit-wm>	 New patchset: Jgreen; "remove db1025 from production puppet, it's in frack.eqiad now" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62607
[14:37:41] <gerrit-wm>	 Change merged: Jgreen; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62607
[14:40:16] <gerrit-wm>	 New review: Hashar; "Ariel pointed out that we could use $wgUseInstantCommons = true." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62606
[14:40:57] <icinga-wm_>	 PROBLEM - Puppet freshness on db44 is CRITICAL: No successful Puppet run in the last 10 hours
[14:46:28] <gerrit-wm>	 New patchset: Hashar; "lame work in progress" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62582
[14:47:12] <cmjohnson1>	 !log authdns update
[14:47:20] <morebots>	 Logged the message, Master
[14:57:34] <icinga-wm_>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[14:59:24] <icinga-wm_>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.126 second response time
[15:04:27] <gerrit-wm>	 New patchset: Yurik; "Deployment rights for Yuri Astrakhan per RT 5069" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62457
[15:04:58] <yurik>	 hi, need an urgent review pls  -- https://gerrit.wikimedia.org/r/#/c/62457/
[15:06:19] <yurik>	 i have to do a deployment of the new zero extension under MaxSem guidance in a bit
[15:08:08] <Reedy>	 You realise puppet doesn't understand urgent?
[15:08:34] <yurik>	 Reedy, i'm hoping to get a merge now, so that puppet would finish syncing by then
[15:09:59] <Reedy>	 paravoid: Can we alias containers for uploads in swift/ceph/wherever the hell they are going?
[15:10:25] <paravoid>	 uhm, what?
[15:11:06] <Reedy>	 Your comment about upload6/upload7
[15:11:17] <Reedy>	 Currently wikimania2005 wiki use the wikipedia/wikimania container
[15:11:27] <Reedy>	 >= 2006 use wikipedia/wikimaniaYEAR
[15:11:57] <Reedy>	 To help reduce the apache rulesets, if we can move/alias/whatever the 2005 one, that's a seperate vhost definition saved
[15:12:24] <Reedy>	 in addition to minus nearly 10 docroot folders
[15:12:43] <Reedy>	 TBH, just moving it would be fine.. for the amount of files and access they get..
[15:12:44] <paravoid>	 I don't think so
[15:12:54] <paravoid>	 wikipedia-wikimania2005-local-deleted
[15:12:54] <paravoid>	 wikipedia-wikimania2005-local-public
[15:12:54] <paravoid>	 wikipedia-wikimania2005-local-temp
[15:12:54] <paravoid>	 wikipedia-wikimania2005-local-thumb
[15:13:07] <Reedy>	 Uhh
[15:13:08] <paravoid>	 and I don't see any "wikimania" container
[15:13:14] <paravoid>	 I see wikimaniaYEAR and wikimaniateam
[15:13:16] <Reedy>	 That looks like they've already been moved in that system
[15:13:53] <Reedy>	 2005 currently has
[15:13:53] <Reedy>	 RewriteRule ^/upload/(.*)$ http://upload.wikimedia.org/wikipedia/wikimania/$1 [R=302]
[15:14:07] <gerrit-wm>	 Change merged: Andrew Bogott; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62457
[15:16:06] <Reedy>	 paravoid: lol
[15:16:17] <Reedy>	 All images are broken on wikimania2005 wiki
[15:16:17] <Reedy>	 https://wikimania2005.wikimedia.org/wiki/File:Logo_sun_tag_rgb.jpg
[15:16:20] <Reedy>	 https://upload.wikimedia.org/wikipedia/wikimania/a/a6/Logo_sun_tag_rgb.jpg
[15:16:23] <Reedy>	 401 unauthorized
[15:16:44] <paravoid>	 that means the container doesn't exist
[15:16:47] <paravoid>	                         if(lang in ['meta', 'commons', 'internal', 'grants', 'wikimania2006']):
[15:16:50] <paravoid>	                             proj = 'wikimedia'
[15:17:16] <Reedy>	 Add hte 2005 and it works
[15:17:16] <Reedy>	 https://upload.wikimedia.org/wikipedia/wikimania2005/a/a6/Logo_sun_tag_rgb.jpg
[15:17:29] <odder>	 hm — when can I see Agora in action?
[15:17:34] <odder>	 where*
[15:17:54] <yurik>	 paravoid, sorry to interject, could you help with the https://gerrit.wikimedia.org/r/#/c/62457/ pls - our deployment window is pretty soon
[15:17:56] <gerrit-wm>	 New patchset: Reedy; "Fix uploads on wikimania2005wiki" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/62616
[15:18:27] <paravoid>	 I'm in meetings for the rest of the day, sorry
[15:19:19] <yurik>	 paravoid, do you know who else might look at the rights change? Its a tiny change - moving myself from one group to another (to allow for deployments)
[15:19:50] <paravoid>	 rt duty?
[15:20:05] <yurik>	 is mark around? I thought that status is stale
[15:21:22] <Reedy>	 [16:21:09] * Topic set by mark!mark@tilia.nedworks.org on 2013-05-06 10:09:03 UTC
[15:22:22] <gerrit-wm>	 New review: Reedy; "(1 comment)" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/62566
[15:28:07] <paravoid>	 Reedy: do you have any idea what that wikimania2006 exception is?
[15:28:12] <paravoid>	 the comment says                     # and here are all the legacy special cases, imported from thumb_handler.php
[15:29:05] <Reedy>	 It would seem to be a lie
[15:29:06] <Reedy>	     RewriteRule ^/upload/(.*)$ http://upload.wikimedia.org/wikipedia/wikimania2006/$1 [R=302]
[15:29:33] <paravoid>	 right on time :)
[15:29:40] * Reedy  blames AaronSchulz

[15:30:05] <paravoid>	                     # and here are all the legacy special cases, imported from thumb_handler.php
[15:30:08] <Reedy>	 It's not even in /wikimedia on upload7
[15:30:09] <paravoid>	                     if(proj == 'wikipedia'):
[15:30:10] <paravoid>	 Aaron|home: do you know where that comes from?
[15:30:12] <paravoid>	                         if(lang in ['meta', 'commons', 'internal', 'grants', 'wikimania2006']):
[15:30:15] <paravoid>	                             proj = 'wikimedia'
[15:38:39] <brion_>	 Anybody feel like doing a quick merge-and-push on bits? FirefoxOS app fix, shouldn't affect anything else. https://gerrit.wikimedia.org/r/#/c/62522/
[15:38:46] <paravoid>	 Reedy: so your change fixes the apache redirect but this won't actually fix the links generated by MW, right?
[15:38:55] <Reedy>	 Indeed
[15:39:18] <Reedy>	 Because MW uses wikimania2005.wikimedia.org/w/uploads which is rewritten to upload.wm.o
[15:40:02] <gerrit-wm>	 New review: Faidon; "No time to deploy now, but it's correct." [operations/apache-config] (master) C: 2;  - https://gerrit.wikimedia.org/r/62616
[15:40:36] <paravoid>	 <li><a href="#metadata">Metadata</a></li></ul><div class="fullImageLink" id="file"><a href="//upload.wikimedia.org/wikipedia/wikimania/a/a6/Logo_sun_tag_rgb.jpg">
[15:40:39] <paravoid>	 not really
[15:41:21] <paravoid>	 also, we're probably screwed, this is probably in parsercache :)
[15:41:27] <Aaron|home>	 paravoid: what about it?
[15:41:36] <paravoid>	 Aaron|home: what about what?
[15:41:49] <Aaron|home>	 [08:30]	paravoid	Aaron|home: do you know where that comes from?
[15:41:53] <Reedy>	 wikimania2005 will have sooo little traffic
[15:42:04] <gerrit-wm>	 New review: Hashar; "This change probably need to be split in smaller changes, it includes too many things:" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62582
[15:42:11] <paravoid>	 where that list comes from
[15:42:14] <paravoid>	 and why wikimania2006 is there
[15:43:32] <gerrit-wm>	 New patchset: Reedy; "Update Wikipedia FirefoxOS app: fix for langlinks menu" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62522
[15:43:34] <Aaron|home>	 it was in http://svn.wikimedia.org/viewvc/mediawiki/trunk/tools/upload-scripts/thumb-handler.php?revision=111717&view=markup
[15:43:51] <gerrit-wm>	 Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62522
[15:43:53] <brion_>	 \o/ thx Reedy
[15:45:45] <logmsgbot>	 !log reedy synchronized docroot
[15:45:53] <morebots>	 Logged the message, Master
[15:48:48] <paravoid>	 Aaron|home: but is it actually needed?
[15:49:53] <paravoid>	 it works now, so I guess it is
[15:50:12] <paravoid>	 Aaron|home: btw, I'm attending http://wiki.ceph.com/01Planning/Developer_Summit today
[15:50:16] <paravoid>	 in case you're interested
[15:50:43] <paravoid>	 the "RGW Geo-Replication and Disaster Recovery" talk in particular might be the most interesting for you
[15:50:54] <paravoid>	 9:30 your time
[15:51:13] <gerrit-wm>	 New patchset: Reedy; "Add initial apache config for login.wikimedia.org" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/62021
[15:51:38] <gerrit-wm>	 New patchset: Reedy; "Add initial apache config for login.wikimedia.org" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/62021
[15:52:47] <gerrit-wm>	 New patchset: Reedy; "Initial config for login.wikimedia.org (loginwiki)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62020
[15:52:50] <paravoid>	 Reedy:     'wikimania2005wiki' => '//upload.wikimedia.org/wikipedia/wikimania', // back compat
[15:52:55] <paravoid>	 $wgUploadPath
[15:53:04] <Reedy>	 o_0
[15:53:26] <paravoid>	 also     'wikimania2005wiki'     => '//upload.wikimedia.org/wikipedia/wikimania/b/bc/Wiki.png',
[15:53:31] <paravoid>	 for wgLogo
[15:53:46] <Reedy>	 Yup
[15:53:49] * Reedy  cleans up

[15:54:20] <paravoid>	 wgUploadPath too, although I don't think this is used anymore
[15:56:01] <paravoid>	     'wikimania'     => '//upload.wikimedia.org/wikipedia/wikimania2006/c/c0/Mania-logo.gif',
[15:56:12] <logmsgbot>	 !log reedy synchronized wmf-config/InitialiseSettings.php
[15:56:20] <morebots>	 Logged the message, Master
[15:56:44] <Reedy>	 That's better :)
[15:56:56] <gerrit-wm>	 New patchset: Reedy; "Fix wikimania2005wiki logo. Remove upload path exception" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62620
[15:57:04] <Reedy>	 At worst, the pages need a purge
[15:57:17] <gerrit-wm>	 Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62620
[15:57:41] <paravoid>	 the wgLogo for "wikimania" (no year) is wikimania2006?
[15:57:44] <paravoid>	 no idea what that is
[15:58:18] <Reedy>	 wikimania like that is a project group
[15:58:31] <Reedy>	 so any wikimania without it's own explicitly defined will get that, rather than default
[16:00:42] <gerrit-wm>	 New patchset: Reedy; "Remove old secure.wikimedia.org code." [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62461
[16:01:59] <gerrit-wm>	 Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62461
[16:02:45] <odder>	 is there any particular way to close a wiki, or do I just add it to closed.dblist?
[16:03:17] <logmsgbot>	 !log reedy synchronized multiversion/MWMultiVersion.php
[16:03:21] <RobH>	 odder: https://wikitech.wikimedia.org/wiki/Close_a_wiki
[16:03:25] <morebots>	 Logged the message, Master
[16:03:40] <RobH>	 odder: so yes, that is 99% of it, and a few more steps is all.
[16:03:42] <odder>	 ah! knew I missed something
[16:03:47] <RobH>	 (unless its changed recently)
[16:04:28] <odder>	 yeah, I guessed there was no need for additional code in groupOverrides etc.
[16:04:33] <odder>	 thanks RobH
[16:04:46] <paravoid>	 Aaron|home: so wikimania2006 isn't really an exception; imagescalers accept both wikimaniaYEAR.wikimedia.org and .wikipedia.org
[16:05:03] <gerrit-wm>	 New review: Dzahn; "just path fix" [operations/puppet] (production) C: 2;  - https://gerrit.wikimedia.org/r/62530
[16:05:04] <gerrit-wm>	 Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62530
[16:05:23] <Jeff_Green>	 i want to pull db1025 out of puppet so it the related icinga config gets purged--is the 'right' way to do that to add it to decommissioning.pp?
[16:06:11] <mutante>	 ori-l: merged the path fix in analysis.pp
[16:06:37] <gerrit-wm>	 New review: Reedy; "Can't be merged until setSiteInfoForWiki in multiversion/MWMultiVersion.php has been updated to work..." [operations/mediawiki-config] (master) C: -1;  - https://gerrit.wikimedia.org/r/62565
[16:07:08] <RobH>	 glad to help =]
[16:07:19] <gerrit-wm>	 New review: Reedy; "Can't be merged until setSiteInfoForWiki in multiversion/MWMultiVersion.php has been updated to work..." [operations/apache-config] (master) C: -1;  - https://gerrit.wikimedia.org/r/62566
[16:07:29] <RobH>	 Jeff_Green: You can manually purge icinga but yep
[16:07:39] <RobH>	 if its in decomissioning.pp it should remove from monitoring
[16:08:09] <RobH>	 the manual removal is faster if you expect to put it back into service under the same name
[16:08:30] <RobH>	 the adding to decom and waiting it out is less error prone
[16:09:16] <gerrit-wm>	 New review: Dzahn; "don't merge per Reedy's comment, but the +1 is for simplifying it which is nice" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/62566
[16:12:08] <Jeff_Green>	 RobH: ok--does adding it to decom and a puppet -tv effectively force it to happen? or is there another layer I'm missing in there?
[16:12:21] <gerrit-wm>	 New patchset: Odder; "(bug 48026) Close wikimania2012 wiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62626
[16:12:27] <RobH>	 you have to force it to run on the icinga server specifically
[16:12:29] <RobH>	 (puppet)
[16:12:30] <Jeff_Green>	 fwiw it's not coming back, it's moved into frack and switching to entirely-nsca monitoring
[16:12:36] <RobH>	 it doesnt even have to run on 1025
[16:12:38] <Jeff_Green>	 RobH: ok cool, that's what I'll do
[16:12:42] <Jeff_Green>	 yup
[16:12:49] <RobH>	 so you can add to decom and merge, and shut it down and be fine
[16:12:55] <Jeff_Green>	 k
[16:12:58] <RobH>	 you dont even have to force the icinga run, since we know its never coming back
[16:13:03] <icinga-wm_>	 PROBLEM - Puppet freshness on mc15 is CRITICAL: No successful Puppet run in the last 10 hours
[16:13:49] <gerrit-wm>	 New patchset: Jgreen; "pull db1025 b/c it moved to frack.eqiad" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62627
[16:13:54] <RobH>	 i also tend to follow up a few days later and remove my entry from decomissioning.pp
[16:14:03] <RobH>	 since once run it doesnt need to be there
[16:14:08] <Jeff_Green>	 oic, alright
[16:14:15] <RobH>	 (you'll notice file is full of servers that technically dont need to be in there)
[16:14:19] <yurik>	 andrewbogott, thanks!!!
[16:14:23] <gerrit-wm>	 Change merged: Jgreen; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62627
[16:14:32] <RobH>	 i just have not cleaned file, since it means really confirming each item in there is dead and gone
[16:14:38] <RobH>	 (havent gotten to it recently)
[16:15:26] <Jeff_Green>	 RobH: ok. I put a comment for db1025 along with the entry in case I forget
[16:15:37] <RobH>	 coolness
[16:15:59] <gwicke>	 Parsoid deployment is about to start
[16:17:37] <gwicke>	 and done
[16:18:42] <Reedy>	 You might want to !log that you're doing it
[16:19:02] <gwicke>	 !log updated Parsoid to current master
[16:19:10] <morebots>	 Logged the message, Master
[16:19:28] <gwicke>	 Reedy: thanks!
[16:20:12] <LeslieCarr>	 yurik: pong
[16:20:24] <yurik>	 LeslieCarr, it got resolved, thanks :)
[16:20:30] <LeslieCarr>	 cool
[16:20:31] <LeslieCarr>	 :)
[16:24:34] <MaxSem>	 is there a way to profile VCL execution time? I've been messing with mobile detection rules, this time specifically to improve Varnish performance, but need to compare first
[16:24:35] <greg-g>	 so, a person in -tech reported that they can't get to http://arbcom.de.wikipedia.org/ "no wiki found"
[16:25:09] <greg-g>	 Nemo_bis: might want to ask here
[16:25:18] <paravoid>	 MaxSem: you can translate them into C and then profile that with regular tools
[16:25:28] <paravoid>	 (I used gprof)
[16:25:36] <paravoid>	 but what exactly are you intending to work on?
[16:25:58] <MaxSem>	 heh. but they'll use VCL-specific functions
[16:26:23] <paravoid>	 for my example it was very few (logging) and I implemented stubs of my own
[16:26:35] <paravoid>	 and marked them as no inline, since gcc was being way too smart :)
[16:26:46] <MaxSem>	 I work on reducing the number of regexes run and changing their order to make it bail out early for popular devices
[16:27:02] <paravoid>	 I think I ended up building them in a different object and linked it afterwards
[16:27:25] <paravoid>	 that's nice
[16:27:45] <paravoid>	 I'm not sure if regexps is the best way to do all this btw
[16:28:05] <paravoid>	 in the sense that every time you start matching the regexp from the beginning
[16:28:13] <paravoid>	 although it is probably much easier to maintain
[16:28:47] <paravoid>	 bblack might be interested in your work, he has been working on speeding up the zero network mappings
[16:28:51] <logmsgbot>	 !log reedy synchronized multiversion/MWMultiVersion.php
[16:28:53] <MaxSem>	 if it only had str_contains() or something...
[16:28:59] <morebots>	 Logged the message, Master
[16:29:22] <paravoid>	 you know you can do inline C, right?
[16:29:28] <gerrit-wm>	 Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/47026
[16:29:42] <MaxSem>	 poor maintainability:P
[16:29:47] <greg-g>	 gwicke: what you do with parsoid doesn't affect the jobqueue, right?
[16:30:00] <gerrit-wm>	 New patchset: Reedy; "Revert "Remove old secure.wikimedia.org code."" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62635
[16:30:07] <MaxSem>	 of course, VCL will be faster if rewritten in C only
[16:30:09] <gerrit-wm>	 Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62635
[16:30:24] <gwicke>	 greg-g: currently it is JS only without any job queue use
[16:30:39] <paravoid>	 you could contain inline C into a single function
[16:30:50] <paravoid>	 or even create an external library (vmod) that you use from VCL
[16:30:55] <RobH>	 !log professor being worked on, network is offline =(
[16:31:03] <morebots>	 Logged the message, RobH
[16:31:09] <gwicke>	 greg-g: but later this month we'll start tracking current edits and template updates, which is done with a PHP extension and does use the job queue
[16:31:47] <greg-g>	 gwicke: ok, thanks, and nevermind anyways, it doesn't appear to be lined up with your activities
[16:31:50] <greg-g>	 :)
[16:32:19] <greg-g>	 The jobrunners seem to be slacking a lot: https://ganglia.wikimedia.org/latest/?r=custom&cs=05%2F01%2F2013+00%3A00+&ce=05%2F07%2F2013+21%3A00+&m=cpu_report&s=by+name&c=Jobrunners+eqiad&h=&host_regex=&max_graphs=0&tab=m&vn=&sh=1&z=small&hc=4
[16:33:15] <gwicke>	 greg-g: IIRC the queue had grown a lot recently
[16:33:21] <gwicke>	 now it seems to be back to saneish levels
[16:34:02] <gwicke>	 I was chatting about potential issues with our job queue use with Aaron last week when we noticed the recent queue growth
[16:34:28] <logmsgbot>	 !log reedy synchronized multiversion/MWMultiVersion.php
[16:34:35] <morebots>	 Logged the message, Master
[16:35:01] <gerrit-wm>	 New patchset: Reedy; "Re-do Ia29aba6762692f70bd63e07e9eac67c07e477878 after I borked the rebase" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62637
[16:35:25] <mwalker>	 Reedy: do you have any more changes?
[16:35:38] <mwalker>	 I'm about to do a scap
[16:35:42] <gerrit-wm>	 New patchset: Reedy; "Re-do Ia29aba6762692f70bd63e07e9eac67c07e477878 after I borked the rebase" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62637
[16:35:51] <Reedy>	 Should be done in about a minute
[16:36:40] <gerrit-wm>	 Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62637
[16:37:15] <logmsgbot>	 !log reedy synchronized multiversion/MWMultiVersion.php
[16:37:23] <morebots>	 Logged the message, Master
[16:37:27] <Reedy>	 mwalker: Done now. Thanks
[16:37:37] <mwalker>	 check
[16:38:08] <mwalker>	 greg-g: btw -- starting my scap for the CentralNotice deploy
[16:38:20] <greg-g>	 mwalker: sweet
[16:38:31] <mwalker>	 Reedy: actually -- any insight into "Copying to fenari from 10.0.5.8...rsync: send_files failed to open "/multiversion/MWMultiVersion.php.save" (in common): Permission denied (13)"
[16:38:42] <greg-g>	 mwalker: reedy was just unbreaking a wiki real quick :)
[16:38:55] <mwalker>	 yeppers
[16:39:12] <Reedy>	 Deleted
[16:39:46] <mwalker>	 thx
[16:40:30] <icinga-wm_>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:41:20] <icinga-wm_>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.130 second response time
[16:45:37] <gerrit-wm>	 New patchset: Yurik; "Mobile Zero extension shared configuration settings" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62576
[16:49:23] <gerrit-wm>	 New patchset: Reedy; "Delete 1.21wmf1-1.21wmf5 symlinks" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62640
[16:49:42] <gerrit-wm>	 Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62640
[16:56:52] <logmsgbot>	 !log mwalker Started syncing Wikimedia installation... : Update CentralNotice and core for HTMLForms, iframe preview, and devices!
[16:57:00] <morebots>	 Logged the message, Master
[17:01:28] <dr0ptp4kt>	 last week i was on fenari as my deployment id, abaso, trying to git pull as part of an attempted code deploy. but i got an access denied. i then tried modifying my git username to dr0ptp4kt, which is my gerrit username, but still got the same error. any hints on what i can do to avoid such errors?
[17:01:47] <mutante>	 !log DNS update - add login.wikimedia and login.m.wikimedia
[17:01:55] <morebots>	 Logged the message, Master
[17:03:38] <Reedy>	 Are you in the wikidev group?
[17:05:22] <icinga-wm_>	 RECOVERY - Host db1025 is UP: PING OK - Packet loss = 0%, RTA = 0.78 ms
[17:06:24] <MaxSem>	 hey mwalker. are you deploying https://gerrit.wikimedia.org/r/#/c/61988/ too now?
[17:06:32] <icinga-wm_>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[17:06:45] <mwalker>	 MaxSem: yes
[17:06:55] <MaxSem>	 awesome:)
[17:07:22] <icinga-wm_>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.143 second response time
[17:08:42] <gerrit-wm>	 New patchset: Andrew Bogott; "Prevent rt DB setup from running repeatedly in production." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62642
[17:08:54] <andrewbogott>	 mutante:  ^
[17:13:52] <kaldari>	 about to run a maintanence script on en.wiki
[17:14:45] <kaldari>	 !log begin running extensions/Echo/maintenance/copyExistingEmailPreference.php on enwiki db
[17:14:53] <morebots>	 Logged the message, Master
[17:18:12] <icinga-wm_>	 RECOVERY - Auth DNS on ns1.wikimedia.org is OK: DNS OK: 0.032 seconds response time. www.wikipedia.org returns 208.80.154.225
[17:18:46] <mutante>	 !log killed hanging pdns related processes on ns1 and restarted pdns
[17:18:54] <morebots>	 Logged the message, Master
[17:19:03] <dr0ptp4kt>	 Reedy, sorry for the delay, yes, abaso is in group wikidev
[17:19:29] <Reedy>	 Can you edit any of the config files?
[17:19:45] <Reedy>	 Are you forwarding your ssh key and such?
[17:21:01] <matanya>	 Ryan_Lane: busy?
[17:21:10] <Ryan_Lane>	 in 10 minutes I will be
[17:21:13] <Ryan_Lane>	 what's up?
[17:21:22] <icinga-wm_>	 RECOVERY - mysqld processes on db1020 is OK: PROCS OK: 1 process with command name mysqld
[17:21:34] <matanya>	 well, i joined puuper-cleanup instance
[17:21:45] <matanya>	 but not sure what is the porpuse of it
[17:21:57] <Ryan_Lane>	 it's for cleaning up the puppet repo
[17:21:59] <mutante>	 andrewbogott: arr, looks good, but we'd still have to execute that on db9. i'd still merge it though because it's good for labs and doesn't hurt
[17:22:05] <matanya>	 *puppet typo's etc
[17:22:18] <Ryan_Lane>	 turning out spaghetti code puppet repo into modules and such
[17:22:21] <Ryan_Lane>	 *our
[17:22:29] <mutante>	 andrewbogott: but that test won't work if we apply to a new production host like that
[17:22:37] <matanya>	 sounds nice
[17:22:46] <kaldari>	 seeing a bit of lag on the enwiki slaves, but seems to be holding steady
[17:22:49] <matanya>	 who can mentor me around there?
[17:22:57] <Ryan_Lane>	 yeah. so in that project you'd make an instance with puppetmaster::self and chose some modules to work with
[17:23:07] <dr0ptp4kt>	 Reedy, which config files? yes, did agent forwarding. may need to play more with the settings in terms of which key is being plucked. is it unusual for people to have different usernames for gerrit/bastion and fenari?
[17:23:11] <andrewbogott>	 mutante:  Ah, sorry, when you suggested that test I thought you meant that it worked on the RT prod box :)
[17:23:14] <matanya>	 BTW Ryan_Lane why do we use puppet and not say, salt?
[17:23:37] <dr0ptp4kt>	 Reedy, brb…will review response (if you have time to provide one, know lots of deployments going on right now)
[17:23:40] <Ryan_Lane>	 paravoid, mutante, notpeter, binasher, mark, LeslieCarr, andrewbogott: ^^ anyone feel like doing some puppet mentoring?
[17:23:42] <Reedy>	 dr0ptp4kt: /home/wikipedia/common/wmf-config
[17:23:48] <Reedy>	 Though, it's seemingly gerrit going lolno
[17:23:55] <Ryan_Lane>	 ^^ specifically chosing parts of the repo to turn into modules?
[17:24:04] <andrewbogott>	 Ryan_Lane, I can once I get untangled from my current puzzle.
[17:24:19] <Ryan_Lane>	 matanya: we use puppet for config management and have a lot of code written for it
[17:24:25] <Ryan_Lane>	 matanya: we're using salt for remote execution
[17:24:27] <mutante>	 andrewbogott: eh, yeah, my bad, the command line just works on local, let me find a better one
[17:24:36] <Ryan_Lane>	 we don't have plans to switch to salt for config management
[17:25:12] <matanya>	 i must say puppet is used in a very unique way
[17:25:26] <logmsgbot>	 !log mwalker Finished syncing Wikimedia installation... : Update CentralNotice and core for HTMLForms, iframe preview, and devices!
[17:25:33] <morebots>	 Logged the message, Master
[17:25:59] <mutante>	 matanya: if you don't care about things changing everywhere in the same moment, puppet is fine, if you have a situation like roll-outs where id does matter that you change it on all hosts at the same time, you would use salt
[17:26:02] <matanya>	 and i not sure why to use two products that can both do the same
[17:26:14] <matanya>	 *i'm
[17:26:21] <andrewbogott>	 mutante:  That suggestion happened to fix the exact problem I was looking at in another puppet project, though, so I'm happy :)
[17:26:21] <Ryan_Lane>	 because we have a ton of code written in puppet :)
[17:26:35] <Ryan_Lane>	 and salt's config management isn't as good as puppet's
[17:26:35] <mutante>	 andrewbogott: :)
[17:26:36] <dr0ptp4kt>	 Reedy, i'm back, will verify i can touch a file
[17:26:50] <Ryan_Lane>	 puppet doesn't have remote execution capabilities
[17:26:51] <andrewbogott>	 modulo stupid quoting/wildcard problems.  *grumble*
[17:26:52] <matanya>	 good answer Ryan_Lane
[17:26:55] <Ryan_Lane>	 so, we use both
[17:27:09] <mutante>	 matanya: they are not really the same, puppet decribes a state you want to achieve
[17:27:28] <matanya>	 i use salt the same way
[17:27:47] <matanya>	 using sls files
[17:28:06] <Ryan_Lane>	 mutante: salt can do config management, too
[17:28:17] <Ryan_Lane>	 we just don't use that
[17:28:28] <matanya>	 for the reasons above
[17:28:31] <matanya>	 i guess
[17:28:36] <Ryan_Lane>	 yep
[17:28:37] <Reedy>	 !log Killed php-1.21wmf1 through 1.21wmf5 on fenari and mediawiki-installation hosts
[17:28:46] <morebots>	 Logged the message, Master
[17:28:58] <mutante>	 i see.. re: "sls files"
[17:29:21] <mutante>	 looks at the page about state enforcement
[17:29:44] <matanya>	 yes, i manage a cluster of hadoop that way
[17:30:19] <mutante>	 looks up "Jinja template"
[17:30:32] <Ryan_Lane>	 mutante: think erb templates in ruby
[17:30:34] <gerrit-wm>	 Change merged: ArielGlenn; [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/62616
[17:30:41] <matanya>	 sls is more or less like pp files, just humen readable :)
[17:30:46] <Ryan_Lane>	 except that you can also use them in the states
[17:30:46] <dr0ptp4kt>	 Reedy, I just touch'd mc.php~ and the timestamp updated...
[17:31:05] <mutante>	 i see an example, not sure if that is more human-readable
[17:31:12] <mutante>	 Ryan_Lane: ..in python, right
[17:31:22] <dr0ptp4kt>	 Reedy: abaso@fenari:/home/wikipedia/common/wmf-config$ groups  … wikidev
[17:31:26] <matanya>	 python=humen-readable mutante
[17:31:34] <Ryan_Lane>	 I don't think the sls is any more or less readable than puppet's dsl
[17:31:47] <kaldari>	 binasher: We're running a 1-time maintenance script for echo on en.wiki right now. It's causing a moderate amount of lag on the slaves, but they are holding steady and I don't expect it to get worse (the queries are batched and wait for the slaves to catch up after each batch). Despite a good deal of effort to optimize the queries, it still looks like the maintenance script is going to take 2-3 days to finish running. Is it acceptable to h
[17:31:47] <kaldari>	 moderate lag on the slaves for that long, or should I kill it and come up with something else?
[17:31:48] <Ryan_Lane>	 sls, by default, is yaml + jinja
[17:31:56] <matanya>	 yes
[17:32:05] <matanya>	 just becuase i'm used to it more
[17:32:28] <kaldari>	 binasher: Current state of the slaves: http://noc.wikimedia.org/dbtree/
[17:32:41] <matanya>	 anyway, andrewbogott i assign you to the task ;P
[17:33:08] <binasher>	 kaldari: what is the maintenance script doing?
[17:33:35] <kaldari>	 it's inserting records in the user_options table, about 1 million total
[17:33:50] <andrewbogott>	 mutante:  Hm, actually I need a way to detect a table, not a whole db.  Is there a way to use mysqlshow for that?
[17:33:51] <gerrit-wm>	 New patchset: Reedy; "Add initial apache config for login.wikimedia.org" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/62021
[17:34:33] <andrewbogott>	 oh, nm, read the help
[17:35:33] <kaldari>	 binasher: https://gerrit.wikimedia.org/r/gitweb?p=mediawiki/extensions/Echo.git;a=blob;f=maintenance/copyExistingEmailPreference.php;h=39395837bc45e36a285db39508cd1fb351f00d52;hb=HEAD
[17:36:11] <binasher>	 wtf
[17:37:16] <binasher>	 kaldari: kill it
[17:38:06] <kaldari>	 !log killed copyExistingEmailPreference.php on enwiki
[17:38:10] <andrewbogott>	 matanya:  ok.  Are you in SF?
[17:38:13] <morebots>	 Logged the message, Master
[17:38:15] <binasher>	 that is causing all kinds of lock contention on user_properties
[17:38:17] <binasher>	 don't rerun
[17:38:36] <matanya>	 andrewbogott: the other side of the ocean
[17:38:45] <matanya>	 asia
[17:39:06] <andrewbogott>	 Oh, where in particular?
[17:39:07] <andrewbogott>	 I'm in Minneapolis which is I think utc-6
[17:39:12] <andrewbogott>	 So probably the exact opposite of you
[17:39:20] <Aaron|home>	 binasher: eep :s
[17:39:31] <matanya>	 yes, can I pm you please? andrewbogott
[17:39:35] <binasher>	 kaldari: never use "insert … select … " in production
[17:39:37] <andrewbogott>	 sure
[17:39:46] <kaldari>	 binasher: sorry about that, I'll see if we can come up with a better solution
[17:41:30] <binasher>	 kaldari: and no limit + offset
[17:42:00] <binasher>	 rows are ordered by user id, so do where up_user >= X limit Y
[17:42:34] <logmsgbot>	 !log mwalker synchronized php-1.22wmf3/extensions/FundraiserLandingPage/  'Updating FundraiserLandingPage for WMF fundraising in FR'
[17:42:42] <morebots>	 Logged the message, Master
[17:43:14] <kaldari>	 binasher: makes sense, I think we had an earlier version that did that
[17:43:45] <kaldari>	 binasher: but I'm going to see if we can come up with a way to avoid the massive db update altogether
[17:43:48] <mwalker>	 greg-g: I'm done with my window; awjr and jdlrobson sound like they have a quick fix they're going to push to MF in this last 15 minutes of it
[17:44:03] <binasher>	 kaldari: thanks
[17:44:39] <greg-g>	 mwalker: cool
[17:44:45] <kaldari>	 binasher: everything we've looked at so far has either been too aggressive or takes a week to run
[17:47:02] <icinga-wm_>	 PROBLEM - Puppet freshness on lvs1004 is CRITICAL: No successful Puppet run in the last 10 hours
[17:47:02] <icinga-wm_>	 PROBLEM - Puppet freshness on lvs1005 is CRITICAL: No successful Puppet run in the last 10 hours
[17:47:02] <icinga-wm_>	 PROBLEM - Puppet freshness on lvs1006 is CRITICAL: No successful Puppet run in the last 10 hours
[17:47:02] <icinga-wm_>	 PROBLEM - Puppet freshness on virt1005 is CRITICAL: No successful Puppet run in the last 10 hours
[17:47:54] <binasher>	 kaldari: so you want to have both enotifusertalkpages and echo-subscriptions-email-edit-user-talk properties containing the same value?
[17:48:18] <kaldari>	 yeah, we want to migrate away from using the old enotifusertalkpages pref
[17:48:35] <binasher>	 is it necessary to keep enotifusertalkpages ?
[17:48:59] <kaldari>	 that's debatable. Depends on how confident we are that Echo isn't going to be turned off
[17:51:07] <binasher>	 arg user_properties, what a table.. "echo-subscriptions-email-edit-user-talk", etc as part of a primary key
[17:51:46] <binasher>	 kaldari: why doesn't echo just use enotifusertalkpages if echo-subscriptions-email-edit-user-talk takes the same value?
[17:52:16] <kaldari>	 that's what we're doing for now, and we could code it to do that permanently
[17:52:30] <kaldari>	 which might be the route we take
[17:53:51] <awjr>	 mwalker: ok for me to push/
[17:54:03] <mwalker>	 awjr: it's all yours
[17:54:05] <awjr>	 thanks!
[17:54:34] <mutante>	 andrewbogott: mysqlshow -h db9.pmtpa.wmnet -u rt -p<PASSWORD> rt Users
[17:54:39] <mutante>	 andrewbogott: where Users is a table name
[17:55:15] <mutante>	 but would have to put the password into private repo if it isn't and replace it with variable
[17:55:52] <mutante>	  grep Password /etc/request-tracker3.8/RT_SiteConfig.pm
[17:56:20] <Aaron|home>	 paravoid: asking questions? :)
[17:56:28] <paravoid>	 ?
[17:56:32] <Aaron|home>	 ceph
[17:56:35] <andrewbogott>	 mutante:  Presumably the db server name is also defined someplace...
[17:56:58] <bblack>	 MaxSem: re your VCL device regex stuff: makes me think there's a place in the world for a library that efficiently implements all of that as a single pattern-match.  e.g. you provide as input a list of regexes and an output label for whichever one the input string matches, and the engine compiles it down to a single pattern tree that's executed at all once and bails out at the first definitive match.
[17:57:00] <paravoid>	 yes
[17:57:02] <mutante>	 andrewbogott: yes, in the same file.. Set($DatabaseHost, 'db9.pmtpa.wmnet');
[17:57:04] <bblack>	 surely someone's done that somewhere
[17:57:21] <mutante>	 andrewbogott: first step to check if RT_SiteConfig.pm is already a template .. can do
[17:58:28] <bblack>	 maybe you could build such a thing as a sort-of DSL on top of Ragel and have it spit out a Ragel state machine
[17:59:16] <logmsgbot>	 !log awjrichards synchronized php-1.22wmf3/extensions/MobileFrontend/stylesheets/common/ui.css  'Cherry-pick e37e9d9ae6b8a581569c39ede38fc21827983128'
[17:59:23] <morebots>	 Logged the message, Master
[18:02:43] <logmsgbot>	 !log awjrichards synchronized php-1.22wmf2/extensions/MobileFrontend/stylesheets/common/ui.css  'Cherry-pick e37e9d9ae6b8a581569c39ede38fc21827983128'
[18:02:51] <morebots>	 Logged the message, Master
[18:03:02] <awjr>	 mwalker, greg-g: should be all done; thanks!
[18:03:20] <greg-g>	 awjr: great, thanks
[18:03:39] <greg-g>	 mlitn: things should be clear for you now :)
[18:06:05] <mlitn>	 greg-g: thanks; we won't be able to deploy anything today though, so the window yours again :)
[18:06:25] <greg-g>	 oh.. ok :)
[18:06:32] <greg-g>	 mlitn: did I totally miss an email or something?
[18:07:43] <greg-g>	 LeslieCarr: btw, now that you're here, is this true?! https://twitter.com/monkeybrainsnet/status/331622143529984000
[18:08:25] <mlitn>	 greg-g: you did not miss any email; we originally planned to deploy AFT changes today, but did not get enough in
[18:08:49] <greg-g>	 mlitn: "good" (sorry about not getting your changes in)  :-)
[18:09:01] <kaldari>	 greg-g: the rest of us have been too busy with echo to review mlitn's AFT code :(
[18:09:15] * greg-g  nods

[18:09:17] <mlitn>	 that's ok, there's nothing urgent ;)
[18:09:28] <greg-g>	 also good
[18:10:56] <kaldari>	 binasher: BTW, I was curious how you noticed the lock contention. Is that something I can look for in the future?
[18:11:15] <binasher>	 kaldari: show engine innodb status\G
[18:11:46] <kaldari>	 thanks
[18:17:16] <gerrit-wm>	 New patchset: awjrichards; "Enable mobile sitenotice on meta" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62650
[18:22:20] <gerrit-wm>	 New review: Dzahn; "(1 comment)" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62642
[18:22:50] <awjr>	 dr0ptp4kt: get your deployment issues sorted?
[18:30:12] <gerrit-wm>	 New patchset: Demon; "Puppetize gitblit.properties" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/61036
[18:34:23] <gerrit-wm>	 Change merged: Dzahn; [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/62021
[18:34:46] <gerrit-wm>	 New patchset: Asher; "new s4 snapshot host, adding future new mariadb master as a slave" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62651
[18:35:15] <gerrit-wm>	 New patchset: Asher; "moving s4 snapshot host to db1042" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62653
[18:35:56] <gerrit-wm>	 Change merged: Asher; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62651
[18:36:14] <gerrit-wm>	 Change merged: Asher; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62653
[18:37:56] <logmsgbot>	 !log asher synchronized wmf-config/db-eqiad.php  'adding db1042,1059 to s4'
[18:38:04] <morebots>	 Logged the message, Master
[18:39:44] <gerrit-wm>	 New patchset: Dzahn; "add docroot for login.wikimedia.org (depended on by gerrit 62021)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62655
[18:40:34] <gerrit-wm>	 New patchset: Andrew Bogott; "Prevent rt DB setup from running repeatedly." [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62642
[18:40:58] <gerrit-wm>	 New review: Dzahn; "have a docroot before deploying the new ServerName" [operations/mediawiki-config] (master) C: 2;  - https://gerrit.wikimedia.org/r/62655
[18:40:59] <gerrit-wm>	 Change merged: Dzahn; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62655
[18:41:31] <mutante>	 !log sync-docroot for login.wm
[18:41:39] <morebots>	 Logged the message, Master
[18:42:01] <logmsgbot>	 !log dzahn synchronized docroot
[18:42:09] <morebots>	 Logged the message, Master
[18:42:37] <paravoid>	 mutante: while at it, why not do the SSL stuff too?
[18:43:33] <icinga-wm_>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[18:43:49] <paravoid>	 mutante: I'll help debug it if something goes wrong :)
[18:44:22] <icinga-wm_>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.132 second response time
[18:46:46] <mutante>	 paravoid: ok:) let me deploy that so we can test against cluster before making the DNS switch
[18:49:40] <gerrit-wm>	 Change merged: Dzahn; [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/60934
[18:53:08] <mutante>	 paravoid: sync-apache does not appear to delete deleted files :p
[18:53:42] <cmjohnson1>	 !log authdns update
[18:53:50] <morebots>	 Logged the message, Master
[18:56:16] <mutante>	 paravoid: so, here's the thing:  https://secure.wikimedia.org/wikipedia/en/wiki/ * 404 Not Found
[18:56:23] <mutante>	 when testing it on mw1044
[18:56:34] <mutante>	 using apache-fast-test
[18:56:48] <Aaron|home>	 2013-05-07 18:36:10 mw1003 wikidatawiki: Could not acknowledge TTMServerMessageUpdateJob job.
[18:56:58] <Aaron|home>	 hmm, sounds like that job takes a long time somtimes
[18:57:50] <Aaron|home>	 *sometimes
[19:02:58] * Aaron|home  does mwscript extensions/WikimediaMaintenance/getJobQueueLengths.php aawiki

[19:03:05] <Aaron|home>	 that looks kind of...ugh
[19:04:29] <gerrit-wm>	 New patchset: Ori.livneh; "Add login / password to MongoDB client" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62661
[19:09:32] <Aaron|home>	 Nikerabbit: should TTMServerMessageUpdateJob jobs take 1 hour?
[19:10:10] <Nikerabbit>	 Aaron|home: 2 to 30 seconds
[19:11:22] <Aaron|home>	 well sometimes it must take forever it seems
[19:11:42] <Aaron|home>	 the ack() function fails, presumably due to the job being recycled already
[19:12:33] <Nikerabbit>	 there is a problem with the jobs and solr on vanadium... it can't handle the concurrent load from jobqueue apparently
[19:13:00] <gerrit-wm>	 New patchset: Hashar; "beta: syslog-ng on deployment-bastion host" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/51668
[19:13:34] <Aaron|home>	 2013-05-07 19:11:05 mw1012 wikidatawiki: TTMServerMessageUpdateJob Translations:Wikidata:List_of_properties/Terms/231 t=14643449 good
[19:13:38] <Nikerabbit>	 haven't had time to work on it yet, ULS is on priority now
[19:13:41] <Aaron|home>	 ugh, all of these log as slow
[19:13:51] <Aaron|home>	 ("t" is ms though, not sec)
[19:14:36] <Aaron|home>	 4hrs, typical time
[19:15:57] <Aaron|home>	 we should probably ulimit this stuff
[19:16:07] <Nikerabbit>	 yes
[19:16:30] <Nikerabbit>	 and tweak solr settings and try to batch the updates and do it single-threaded or so
[19:19:40] <Aaron|home>	 binasher: http://ganglia.wikimedia.org/latest/?c=Redis%20eqiad&h=rdb1001.eqiad.wmnet&m=cpu_report&r=week&s=by%20name&hc=4&mc=2 seems to be missing some data
[19:19:45] <Aaron|home>	 some of the graphs just cut off
[19:20:34] <Aaron|home>	 binasher: I wonder if the password requirement added around then broke the ganglia checks
[19:20:53] <binasher>	 Aaron|home: that would do it
[19:22:37] <gerrit-wm>	 New patchset: Dzahn; "move the secure.wm conf before the *.wikimedia.org wildcard or that matches first" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/62662
[19:23:22] <icinga-wm_>	 RECOVERY - Host professor is UP: PING OK - Packet loss = 0%, RTA = 26.63 ms
[19:23:22] <icinga-wm_>	 RECOVERY - carbon-cache.py on professor is OK: PROCS OK: 1 process with args carbon-cache.py
[19:23:51] <gerrit-wm>	 New review: Dzahn; "thanks paravoid" [operations/apache-config] (master) C: 2;  - https://gerrit.wikimedia.org/r/62662
[19:23:51] <gerrit-wm>	 Change merged: Dzahn; [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/62662
[19:28:22] <icinga-wm_>	 PROBLEM - carbon-cache.py on professor is CRITICAL: PROCS CRITICAL: 2 processes with args carbon-cache.py
[19:29:55] <mutante>	 !log apache-graceful-all, move old secure.wm redirects
[19:30:03] <morebots>	 Logged the message, Master
[19:32:11] <gerrit-wm>	 New review: Dzahn; "dzahn@fenari:~$ apache-fast-test secure.url mw1043" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/62662
[19:36:42] <mutante>	 !log DNS update - switch old secure.wm
[19:36:50] <morebots>	 Logged the message, Master
[19:51:14] <gerrit-wm>	 New patchset: Demon; "Use JGit's recursive merge" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62669
[19:52:37] <gerrit-wm>	 New patchset: Reedy; "Initial config for login.wikimedia.org (loginwiki)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62020
[19:52:48] <gerrit-wm>	 New patchset: Reedy; "Initial config for login.wikimedia.org (loginwiki)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62020
[19:53:02] <gerrit-wm>	 Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62020
[19:55:19] <logmsgbot>	 !log reedy synchronized database lists files: loginwiki
[19:55:27] <morebots>	 Logged the message, Master
[19:55:51] <mutante>	 Reedy: will it have a database?
[19:55:52] <logmsgbot>	 !log reedy synchronized wmf-config/
[19:55:56] <Reedy>	 mutante: Yup
[19:55:59] <morebots>	 Logged the message, Master
[19:56:21] <mutante>	 Reedy: and private?
[19:56:26] <mutante>	 then we'd need to add to realm.pp
[19:56:41] <logmsgbot>	 !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files: loginwiki
[19:56:49] <morebots>	 Logged the message, Master
[19:56:56] <Reedy>	 Nope
[19:57:02] <mutante>	 ok, just making sure
[19:57:30] <gerrit-wm>	 New patchset: Reedy; "loginwiki" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62671
[19:58:57] <gerrit-wm>	 Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62671
[19:59:43] <ori-l>	 can i get a merge on https://gerrit.wikimedia.org/r/#/c/62661/ ? just adds the username and password to the mongodb connection string. they were added to the puppet private repo by mark per 5101
[19:59:47] <ori-l>	 rt 5101 i mean
[20:01:34] <logmsgbot>	 !log reedy synchronized wmf-config/CommonSettings.php
[20:01:42] <morebots>	 Logged the message, Master
[20:03:11] <mutante>	 ori-l: hmm,, i don't see them in private repo yet
[20:03:35] <ori-l>	 oh, weird. mark replied saying it was done.
[20:03:42] <MaxSem>	 Reedy, ping me when you're done, plz
[20:03:46] <ori-l>	 maybe he forgot to sync or something
[20:04:08] <mutante>	 ori-l: ah, i see, it's that, sync, yea
[20:04:26] <mutante>	 was on stafford but not on sockpuppet
[20:04:34] <logmsgbot>	 !log reedy synchronized wmf-config/InitialiseSettings.php
[20:04:35] <mutante>	 usually it's the other way around
[20:04:42] <morebots>	 Logged the message, Master
[20:05:31] <gerrit-wm>	 New patchset: Reedy; "Couple more loginwiki config items" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62673
[20:05:54] <ori-l>	 mutante: what is the role of each machine? sockpuppet = puppetmaster, stafford = puppet/private?
[20:05:58] <gerrit-wm>	 Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62661
[20:06:14] <dr0ptp4kt>	 awjr, not yet. Reedy, any pointers on the access for git pull on fenari?
[20:06:28] <ori-l>	 mutante: thanks :)
[20:06:54] <dr0ptp4kt>	 ^  awjr, Reedy  - (Reedy, as well as, possibly, sync-*)
[20:07:31] <gerrit-wm>	 Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62673
[20:07:51] <mutante>	 ori-l: stafford is the actual master . puppet.wikimedia.org is an alias for stafford.pmtpa.wmnet.
[20:08:15] <mutante>	 ori-l: but they both have the git repos, public and private, and usually we merge on sockpuppet
[20:08:39] <mutante>	 if it's public it syncs automatically, and if it's private we have to do a manual git pull on stafford after merging on sockpuppet
[20:08:46] <ori-l>	 cool, thanks for the explanation
[20:11:16] <ori-l>	 mutante: looking at the config file, both variables expanded to 'undefined'
[20:11:19] <logmsgbot>	 !log reedy synchronized docroot/login/
[20:11:26] <morebots>	 Logged the message, Master
[20:14:04] <gerrit-wm>	 New patchset: Ori.livneh; "MongoDB on Vanadium: auth => true" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62674
[20:15:11] <ori-l>	 mutante: yt?
[20:16:20] <logmsgbot>	 !log reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
[20:16:28] <morebots>	 Logged the message, Master
[20:27:30] <ori-l>	 andrewbogott / mutante -- after the correct l/p are sync'ed i'll have to ask you to merge this too: https://gerrit.wikimedia.org/r/#/c/62674/ , sets the 'auth' flag to true in the mongo config file. they can't go together because i need to set the credentials in the database first.
[20:27:59] <gerrit-wm>	 New patchset: Reedy; "loginwiki v2" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/62719
[20:29:20] <andrewbogott>	 !log copied passwords from stafford to sockpuppet, then reset --hard on stafford to get things back in sync.
[20:29:28] <morebots>	 Logged the message, Master
[20:30:55] <Aaron|home>	 TTMServerMessageUpdateJob 4906075.843 0.843059938983
[20:31:03] * Aaron|home  just whipped up some python script

[20:31:20] <Aaron|home>	 Nikerabbit: yeah, so if that works, then 84% of job runner time is on those
[20:31:50] <gerrit-wm>	 Change abandoned: Reedy; "(no reason)" [operations/apache-config] (master) - https://gerrit.wikimedia.org/r/62719
[20:34:23] <Aaron|home>	 the fact that it dwarfs refreshLinks by an order of magnitude is impressive
[20:38:37] <yurik>	 do we have a procedure in place to do partial varnish flushes?
[20:41:07] <gerrit-wm>	 New patchset: Reedy; "Update size dblists" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62723
[20:43:34] <gerrit-wm>	 Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62723
[20:44:08] <logmsgbot>	 !log reedy synchronized database lists files:
[20:44:15] <morebots>	 Logged the message, Master
[20:53:18] <logmsgbot>	 !log aaron synchronized php-1.22wmf3/includes/job/JobQueueGroup.php  'Live hack to avoid translationNotificationJob jobs.'
[20:53:26] <morebots>	 Logged the message, Master
[21:08:46] <logmsgbot>	 !log aaron synchronized php-1.22wmf2/includes/job/JobQueueGroup.php
[21:08:53] <morebots>	 Logged the message, Master
[21:15:03] * MaxSem  prepares to scap...

[21:15:28] <gerrit-wm>	 New patchset: Reedy; "Die symlinks, die" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62735
[21:15:39] <gerrit-wm>	 Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62735
[21:27:41] <MaxSem>	 "cannot delete non-empty directory: php-1.21wmf7/.git/modules/extensions/AssertEdit"
[21:28:13] <Reedy>	 Probably stupid permissions errors
[21:28:25] <Reedy>	 Might have to get a root to run a removal for them
[21:32:28] <paravoid>	 Aaron|home: forgot to tell you: one of things the ceph people are working on is sharded indexes for bucket listings
[21:32:36] <paravoid>	 or rather AaronSchulz
[21:32:54] <paravoid>	 so maybe one day we'll have the chance to get rid of our own sharding
[21:32:57] <paravoid>	 if it isn't too late now :)
[21:33:32] <Aaron|home>	 aaron@fluorine:~/mw-log$ grep 'No working slave' dberror.log | wc -l
[21:33:34] <Aaron|home>	 5589
[21:33:54] * Aaron|home  is still trying to figure out wtf is going on

[21:34:26] <Aaron|home>	 paravoid: would be nice I suppose
[21:34:55] <Reedy>	 Aaron|home: Have they all retired?
[21:36:36] <drdee>	 mutant; stat1 is going crazy with a backup; can we reschedule this to a more convenient time (let's say midnight PST) ?
[21:36:50] <drdee>	 mutante, whoops; forget the 'e' ^^
[21:39:31] <Aaron|home>	 2013-05-07 21:39:15 srv238 enwiki: Memcached error for key "enwiki:gadgets-definition:6" on server "10.0.12.15:11211": A TIMEOUT OCCURRED
[21:39:33] <Aaron|home>	 2013-05-07 21:39:15 srv238 enwiki: Memcached error for key "enwiki:gadgets-definition:6" on server "10.0.12.15:11211": SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY
[21:39:37] <Aaron|home>	 spammy
[21:40:02] <Aaron|home>	 paravoid: is there something up with that box?
[21:43:30] <Aaron|home>	 $this->mLastError = 'No working slave server: ' . $this->mLastError;
[21:43:41] <Aaron|home>	 Reedy: that seems to result in the same error a hundred times ;)
[21:43:47] * Aaron|home  looks in LoadBalancer

[21:46:23] <logmsgbot>	 !log maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
[21:46:32] <morebots>	 Logged the message, Master
[21:47:37] <Aaron|home>	 notpeter: I wonder what's up with 10.0.12.15:11211
[21:50:36] <notpeter>	 box hasn't been running puppet
[21:50:39] <notpeter>	 might be crapping out
[21:53:35] <MaxSem>	 uuuugh, deleted branches spam is spammy
[21:54:07] <mutante>	 drdee: we can but it's not done quickly, it's in backup.pp and it uses the files in puppet/files/backup. stat1 has been added like here: https://gerrit.wikimedia.org/r/#/c/42286/1 so you see all hosts are in that file
[21:54:44] <notpeter>	 Aaron|home: it's insanely comically slow. lots of connection errors from everything that it's running
[21:54:48] <notpeter>	 I'm going to guess hw issues
[21:54:56] <notpeter>	 but I haven't found anything to that effect yet
[21:55:21] <notpeter>	 Aaron|home: I'm assuming that taking it down for hardware diagnostics won't be problematic?
[21:55:29] <mutante>	 drdee: more in RT-3098 and https://wikitech.wikimedia.org/wiki/Backup_procedures#Misc_data
[21:55:36] <Aaron|home>	 right
[21:55:40] <drdee>	 mutante: ok thx, i iwll read that
[21:55:47] <ori-l>	 mutante: hey :)
[21:55:49] <notpeter>	 cool. just wanted to doublecheck
[21:55:56] <MaxSem>	 snapshot4: sudo: no tty present and no askpass program specified
[21:56:05] * Aaron|home  trying to figure out what's backing up jobs

[21:56:18] <Reedy>	 They've been appearing for a while
[22:02:31] <Aaron|home>	 notpeter: maybe it should be pulled from wmf-config
[22:02:54] <Aaron|home>	 if it really is hardware or something that you can't just powercycle
[22:03:48] <logmsgbot>	 !log maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/
[22:03:56] <morebots>	 Logged the message, Master
[22:04:04] <logmsgbot>	 !log maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
[22:04:04] <notpeter>	 Aaron|home: well, erratic slowness sounds like hw failure to me
[22:04:12] <morebots>	 Logged the message, Master
[22:04:27] <notpeter>	 Aaron|home: want to pull it, and I'll make a ticket in the pmtpa queue?
[22:05:28] <Aaron|home>	 sure
[22:07:10] <Aaron|home>	 notpeter: wait isn't that in tampa?
[22:07:10] <gerrit-wm>	 Change merged: MaxSem; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62650
[22:07:42] <notpeter>	 Aaron|home: yes
[22:07:50] <Aaron|home>	 why is enwiki hitting it like crazy then?
[22:08:02] <notpeter>	 you know, I'm glad you asked
[22:08:08] <notpeter>	 and I'm sure there's a good answer...
[22:08:17] <Aaron|home>	 I mean seriously, wtf
[22:08:28] <notpeter>	 oh, well, isn't reddis set up to repl off of eqiad?
[22:08:31] <notpeter>	 is that what it's seeing?
[22:08:37] <notpeter>	 or are the connections from apaches?
[22:09:42] <logmsgbot>	 !log maxsem synchronized wmf-config/InitialiseSettings.php  'https://gerrit.wikimedia.org/r/#/c/62650/'
[22:09:46] <Aaron|home>	 this is for memcached, not redis
[22:09:50] <morebots>	 Logged the message, Master
[22:09:55] <Aaron|home>	 so the apaches having errors are from tampa too
[22:11:15] <gerrit-wm>	 New patchset: Ori.livneh; "Add passwords::mongodb::eventlogging to dummy passwords.pp" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62741
[22:11:23] <ori-l>	 ^ mutante, in case that helps
[22:11:41] <mutante>	 yea, it's nice to do in any case
[22:12:01] <gerrit-wm>	 Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62741
[22:12:47] <Aaron|home>	 notpeter: 49 different apaches getting traffic there
[22:12:48] <Krinkle>	 mark: Can you execute an ops request for me? https://bugzilla.wikimedia.org/show_bug.cgi?id=48233
[22:13:26] <Aaron|home>	 each with 2.8-3k of these errors
[22:14:21] <Aaron|home>	 while this seems very broken (and will split-brain memcached) I don't see it being related to the jobq
[22:19:11] <gerrit-wm>	 New patchset: Ori.livneh; "Require the MongoDB password class" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62743
[22:19:22] <mutante>	 aaah:)
[22:19:37] * ori-l  puts on the dunce cap

[22:19:49] <gerrit-wm>	 New review: Dzahn; "heh, right:)" [operations/puppet] (production) C: 2;  - https://gerrit.wikimedia.org/r/62743
[22:19:49] <gerrit-wm>	 Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62743
[22:20:53] <ori-l>	 sorry about the wild goose chase
[22:21:18] <notpeter>	 Aaron|home: well, that's pretty crazy...
[22:21:25] <mutante>	 ori-l: np, it would have been odd
[22:21:34] <icinga-wm_>	 PROBLEM - Puppet freshness on db45 is CRITICAL: No successful Puppet run in the last 10 hours
[22:21:51] <Aaron|home>	 notpeter: who is most familiar with the DC traffic routing?
[22:21:55] <ori-l>	 before you run off though i need to add the credentials to the db and then merge https://gerrit.wikimedia.org/r/#/c/62674/ -- things will be broken until then
[22:22:19] <mutante>	 watching the puppet run
[22:22:35] <icinga-wm_>	 PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:23:04] <mutante>	 ori-l: the password is in the conf file now :)
[22:23:24] <icinga-wm_>	 RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK: Status line output matched 400 - 336 bytes in 0.131 second response time
[22:23:29] <mutante>	 lemme know when to merge the "auth => true"
[22:24:07] <drdee>	 mutante, could you please merge https://gerrit.wikimedia.org/r/#/c/62001/
[22:24:12] <notpeter>	 Aaron|home: LeslieCarr?
[22:24:51] <ori-l>	 mutante: yep, ready. i added it
[22:25:11] <LeslieCarr>	 yep
[22:25:13] <LeslieCarr>	 what's up ?
[22:25:17] <LeslieCarr>	 oh dc traffic routing
[22:25:25] <gerrit-wm>	 New review: Dzahn; "eventlogging now has the credentials and they've been added to db" [operations/puppet] (production) C: 2;  - https://gerrit.wikimedia.org/r/62674
[22:25:26] <gerrit-wm>	 Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62674
[22:25:44] <LeslieCarr>	 Krinkle: mark's in europe so it's pretty late out there
[22:25:56] <Krinkle>	 LeslieCarr: Tell me about it :P
[22:26:15] <Krinkle>	 LeslieCarr: I figured he'd be asleep, idle for 4 hours
[22:26:20] <LeslieCarr>	 ok :)
[22:26:24] <Krinkle>	 LeslieCarr: I defaulted to him since he's in topic for rt duty
[22:26:33] <Krinkle>	 but forgot to check his whois first
[22:26:38] <gerrit-wm>	 New patchset: MaxSem; "Disable Nearby on Wikivoyage" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62744
[22:27:01] * Krinkle  lives in the same country as mark

[22:27:36] <ori-l>	 rootlandia
[22:28:24] <gerrit-wm>	 Change merged: MaxSem; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62744
[22:28:46] <gerrit-wm>	 New review: Dzahn; "RT-5059 has approval and waiting period ended." [operations/puppet] (production) C: 2;  - https://gerrit.wikimedia.org/r/62001
[22:28:47] <gerrit-wm>	 Change merged: Dzahn; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62001
[22:29:18] <mutante>	 !change 62001 | xyzram
[22:29:18] <wm-bot>	 xyzram: https://gerrit.wikimedia.org/r/#q,62001,n,z
[22:29:21] <Aaron|home>	 notpeter: btw ganglia needs to use the password for redis stats
[22:29:24] <mutante>	 !change 62001 | drdee
[22:29:25] <wm-bot>	 drdee: https://gerrit.wikimedia.org/r/#q,62001,n,z
[22:29:38] <notpeter>	 Aaron|home: ah, yes, that makes sense
[22:29:48] <Aaron|home>	 graphs are wonk now :(
[22:29:54] <Aaron|home>	 gah, too much broken stuff today
[22:30:04] <drdee>	 thanks mutante!
[22:30:21] <Aaron|home>	 redis stats, gdash, mc box, traffic routing, queue backup, aarg
[22:30:23] <logmsgbot>	 !log maxsem synchronized wmf-config/InitialiseSettings.php  'https://gerrit.wikimedia.org/r/#/c/62744/'
[22:30:31] <morebots>	 Logged the message, Master
[22:30:34] <gerrit-wm>	 New patchset: Pyoungmeister; "removing mc15 for hw maint" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62746
[22:30:42] <notpeter>	 Aaron|home: ^^
[22:31:38] <Aaron|home>	 looks fine but jenkins fails
[22:31:47] <Aaron|home>	 Reedy: whinging about dblists
[22:31:58] <Aaron|home>	 did someone change those lately?
[22:32:08] <Reedy>	 It happens when we add wikis and stuff
[22:32:19] <Aaron|home>	 the tests fail?
[22:32:26] <ori-l>	 mutante, andrewbogott: everything works well now, thanks very much for everything.
[22:32:28] <Reedy>	 +    301 => ''
[22:32:31] <Reedy>	 That's really fucking useful
[22:32:45] <Reedy>	 Aaron|home: I think that's called a newline
[22:32:54] <mutante>	 ori-l: great, glad to hear
[22:34:01] <gerrit-wm>	 New patchset: Reedy; "Fix trailing newline" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62747
[22:34:03] <Aaron|home>	 yeah, I figured that
[22:34:09] * Aaron|home  was just reading medium.dblist

[22:34:12] <mutante>	 !log upgrading PHP_CodeSniffer on gallium
[22:34:13] <gerrit-wm>	 Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62747
[22:34:21] <morebots>	 Logged the message, Master
[22:34:21] <mutante>	 Krinkle: upgrade ok: channel://pear.php.net/PHP_CodeSniffer-1.4.5
[22:34:24] <notpeter>	 ok, I'll rebase
[22:34:25] <gerrit-wm>	 New patchset: Reedy; "removing mc15 for hw maint" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62746
[22:34:30] <Reedy>	 heh
[22:34:33] <Krinkle>	 mutante: Yes
[22:34:37] <Krinkle>	 mutante: Great
[22:34:44] <notpeter>	 or reedy will :)
[22:35:19] <gerrit-wm>	 Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62746
[22:35:54] <Reedy>	 $list[$projectname] = file( $filename, FILE_IGNORE_NEW_LINES);
[22:35:59] <logmsgbot>	 !log aaron synchronized wmf-config/mc-pmtpa.php  'removing mc15 for hw maint'
[22:36:06] <morebots>	 Logged the message, Master
[22:36:18] <mutante>	 Reedy: and now i'm trying iegcom :)
[22:36:20] <mutante>	 sry
[22:36:24] <Reedy>	 :D
[22:36:49] <notpeter>	 Reedy: Aaron|home ok, made rt 5104 for hw diagnostics
[22:37:44] <gerrit-wm>	 New patchset: Reedy; "Make dblist tests skip empty lines when reading" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62750
[22:38:40] * Reedy  waits for jenkins

[22:39:33] <gerrit-wm>	 Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62750
[22:47:36] <TimStarling>	 !log on marmontel: disabled logrotate
[22:47:44] <morebots>	 Logged the message, Master
[22:52:22] <gerrit-wm>	 New patchset: Ori.livneh; "Redis ganglia metrics: support password param" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/62751
[23:08:34] <icinga-wm_>	 PROBLEM - Puppet freshness on cp3003 is CRITICAL: No successful Puppet run in the last 10 hours
[23:08:34] <icinga-wm_>	 PROBLEM - Puppet freshness on virt3 is CRITICAL: No successful Puppet run in the last 10 hours
[23:12:45] <gerrit-wm>	 New patchset: Dzahn; "add iegcom wiki initial config (RT-5042)" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62756
[23:13:24] <icinga-wm_>	 PROBLEM - DPKG on srv246 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:13:25] <icinga-wm_>	 PROBLEM - DPKG on srv244 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:13:44] <icinga-wm_>	 PROBLEM - DPKG on srv257 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:13:44] <icinga-wm_>	 PROBLEM - DPKG on srv243 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:13:44] <icinga-wm_>	 PROBLEM - DPKG on srv193 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:13:44] <icinga-wm_>	 PROBLEM - DPKG on srv253 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:13:44] <icinga-wm_>	 PROBLEM - DPKG on srv251 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:13:45] <icinga-wm_>	 PROBLEM - DPKG on srv236 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:13:45] <icinga-wm_>	 PROBLEM - DPKG on srv298 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:13:46] <icinga-wm_>	 PROBLEM - DPKG on srv264 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:13:46] <icinga-wm_>	 PROBLEM - DPKG on srv239 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:14:04] <icinga-wm_>	 PROBLEM - DPKG on srv268 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:14:04] <icinga-wm_>	 PROBLEM - DPKG on srv295 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:14:04] <icinga-wm_>	 PROBLEM - DPKG on srv301 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:14:14] <icinga-wm_>	 PROBLEM - DPKG on srv261 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:14:14] <icinga-wm_>	 PROBLEM - DPKG on srv273 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:14:14] <icinga-wm_>	 PROBLEM - DPKG on srv263 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:14:14] <icinga-wm_>	 PROBLEM - DPKG on srv293 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:14:14] <icinga-wm_>	 PROBLEM - DPKG on srv281 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:14:15] <icinga-wm_>	 PROBLEM - DPKG on srv297 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:14:24] <icinga-wm_>	 PROBLEM - DPKG on srv289 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:14:24] <icinga-wm_>	 RECOVERY - DPKG on srv246 is OK: All packages OK
[23:14:24] <icinga-wm_>	 RECOVERY - DPKG on srv244 is OK: All packages OK
[23:14:24] <icinga-wm_>	 PROBLEM - DPKG on srv299 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:14:34] <icinga-wm_>	 PROBLEM - DPKG on srv275 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:14:34] <icinga-wm_>	 PROBLEM - DPKG on srv265 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:14:44] <icinga-wm_>	 RECOVERY - DPKG on srv257 is OK: All packages OK
[23:14:44] <icinga-wm_>	 RECOVERY - DPKG on srv243 is OK: All packages OK
[23:14:44] <icinga-wm_>	 RECOVERY - DPKG on srv193 is OK: All packages OK
[23:14:44] <icinga-wm_>	 RECOVERY - DPKG on srv253 is OK: All packages OK
[23:14:44] <icinga-wm_>	 RECOVERY - DPKG on srv251 is OK: All packages OK
[23:14:45] <icinga-wm_>	 RECOVERY - DPKG on srv236 is OK: All packages OK
[23:14:45] <icinga-wm_>	 PROBLEM - DPKG on srv274 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:14:46] <icinga-wm_>	 PROBLEM - DPKG on srv283 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:14:46] <icinga-wm_>	 PROBLEM - DPKG on srv271 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:14:47] <icinga-wm_>	 PROBLEM - DPKG on srv276 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:14:47] <icinga-wm_>	 PROBLEM - DPKG on srv277 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:14:48] <icinga-wm_>	 PROBLEM - DPKG on srv285 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:14:48] <icinga-wm_>	 PROBLEM - DPKG on srv296 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:14:49] <icinga-wm_>	 RECOVERY - DPKG on srv239 is OK: All packages OK
[23:14:49] <icinga-wm_>	 PROBLEM - DPKG on srv272 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:14:50] <icinga-wm_>	 PROBLEM - DPKG on srv300 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:14:50] <icinga-wm_>	 PROBLEM - DPKG on srv292 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:14:51] <icinga-wm_>	 PROBLEM - DPKG on srv294 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:14:51] <icinga-wm_>	 PROBLEM - DPKG on srv290 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[23:14:52] <icinga-wm_>	 PROBLEM - Apache HTTP on srv246 is CRITICAL: Connection refused
[23:15:04] <icinga-wm_>	 PROBLEM - Apache HTTP on srv243 is CRITICAL: Connection refused
[23:15:14] <icinga-wm_>	 PROBLEM - Apache HTTP on srv244 is CRITICAL: Connection refused
[23:15:14] <icinga-wm_>	 PROBLEM - Apache HTTP on srv248 is CRITICAL: Connection refused
[23:15:14] <icinga-wm_>	 PROBLEM - Apache HTTP on srv255 is CRITICAL: Connection refused
[23:15:14] <icinga-wm_>	 PROBLEM - Apache HTTP on srv252 is CRITICAL: Connection refused
[23:15:24] <icinga-wm_>	 PROBLEM - Apache HTTP on srv193 is CRITICAL: Connection refused
[23:15:24] <icinga-wm_>	 PROBLEM - Apache HTTP on srv247 is CRITICAL: Connection refused
[23:15:34] <icinga-wm_>	 PROBLEM - Apache HTTP on srv251 is CRITICAL: Connection refused
[23:15:34] <icinga-wm_>	 PROBLEM - Apache HTTP on srv264 is CRITICAL: Connection refused
[23:15:34] <icinga-wm_>	 PROBLEM - Apache HTTP on srv236 is CRITICAL: Connection refused
[23:15:34] <icinga-wm_>	 PROBLEM - Apache HTTP on srv249 is CRITICAL: Connection refused
[23:15:34] <icinga-wm_>	 PROBLEM - Apache HTTP on srv256 is CRITICAL: Connection refused
[23:15:44] <icinga-wm_>	 RECOVERY - DPKG on srv264 is OK: All packages OK
[23:15:44] <icinga-wm_>	 PROBLEM - Apache HTTP on srv245 is CRITICAL: Connection refused
[23:15:44] <icinga-wm_>	 PROBLEM - Apache HTTP on srv239 is CRITICAL: Connection refused
[23:15:44] <icinga-wm_>	 PROBLEM - Apache HTTP on srv257 is CRITICAL: Connection refused
[23:15:44] <icinga-wm_>	 PROBLEM - Apache HTTP on srv253 is CRITICAL: Connection refused
[23:15:45] <icinga-wm_>	 PROBLEM - Apache HTTP on srv238 is CRITICAL: Connection refused
[23:15:45] <icinga-wm_>	 PROBLEM - Apache HTTP on srv240 is CRITICAL: Connection refused
[23:15:46] <icinga-wm_>	 PROBLEM - Apache HTTP on srv235 is CRITICAL: Connection refused
[23:15:46] <icinga-wm_>	 PROBLEM - Apache HTTP on srv254 is CRITICAL: Connection refused
[23:15:54] <icinga-wm_>	 PROBLEM - Apache HTTP on srv242 is CRITICAL: Connection refused
[23:16:04] <icinga-wm_>	 PROBLEM - Apache HTTP on srv273 is CRITICAL: Connection refused
[23:16:04] <icinga-wm_>	 PROBLEM - Apache HTTP on srv261 is CRITICAL: Connection refused
[23:16:04] <icinga-wm_>	 RECOVERY - DPKG on srv268 is OK: All packages OK
[23:16:14] <icinga-wm_>	 RECOVERY - Apache HTTP on srv244 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.129 second response time
[23:16:14] <icinga-wm_>	 RECOVERY - Apache HTTP on srv255 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.158 second response time
[23:16:14] <icinga-wm_>	 RECOVERY - DPKG on srv273 is OK: All packages OK
[23:16:14] <icinga-wm_>	 RECOVERY - DPKG on srv261 is OK: All packages OK
[23:16:14] <icinga-wm_>	 RECOVERY - Apache HTTP on srv248 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.286 second response time
[23:16:15] <icinga-wm_>	 RECOVERY - DPKG on srv263 is OK: All packages OK
[23:16:15] <icinga-wm_>	 RECOVERY - DPKG on srv293 is OK: All packages OK
[23:16:16] <icinga-wm_>	 RECOVERY - Apache HTTP on srv252 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.157 second response time
[23:16:16] <icinga-wm_>	 RECOVERY - DPKG on srv281 is OK: All packages OK
[23:16:17] <icinga-wm_>	 RECOVERY - DPKG on srv297 is OK: All packages OK
[23:16:22] <LeslieCarr>	 !log upgrading srv* machines
[23:16:24] <icinga-wm_>	 RECOVERY - DPKG on srv289 is OK: All packages OK
[23:16:24] <icinga-wm_>	 RECOVERY - Apache HTTP on srv193 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.157 second response time
[23:16:24] <icinga-wm_>	 RECOVERY - Apache HTTP on srv247 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.121 second response time
[23:16:24] <icinga-wm_>	 RECOVERY - DPKG on srv299 is OK: All packages OK
[23:16:30] <morebots>	 Logged the message, Mistress of the network gear.
[23:16:34] <icinga-wm_>	 RECOVERY - Apache HTTP on srv236 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.120 second response time
[23:16:34] <icinga-wm_>	 RECOVERY - Apache HTTP on srv264 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.134 second response time
[23:16:34] <icinga-wm_>	 RECOVERY - Apache HTTP on srv251 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.153 second response time
[23:16:34] <icinga-wm_>	 RECOVERY - DPKG on srv275 is OK: All packages OK
[23:16:34] <icinga-wm_>	 RECOVERY - Apache HTTP on srv256 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.160 second response time
[23:16:35] <icinga-wm_>	 RECOVERY - Apache HTTP on srv249 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.291 second response time
[23:16:35] <icinga-wm_>	 RECOVERY - DPKG on srv265 is OK: All packages OK
[23:16:44] <icinga-wm_>	 RECOVERY - DPKG on srv271 is OK: All packages OK
[23:16:44] <icinga-wm_>	 RECOVERY - DPKG on srv283 is OK: All packages OK
[23:16:44] <icinga-wm_>	 RECOVERY - DPKG on srv274 is OK: All packages OK
[23:16:44] <icinga-wm_>	 RECOVERY - DPKG on srv276 is OK: All packages OK
[23:16:44] <icinga-wm_>	 RECOVERY - DPKG on srv285 is OK: All packages OK
[23:16:45] <icinga-wm_>	 RECOVERY - DPKG on srv272 is OK: All packages OK
[23:16:45] <icinga-wm_>	 RECOVERY - DPKG on srv277 is OK: All packages OK
[23:16:46] <icinga-wm_>	 RECOVERY - DPKG on srv300 is OK: All packages OK
[23:16:46] <icinga-wm_>	 RECOVERY - DPKG on srv298 is OK: All packages OK
[23:16:47] <icinga-wm_>	 RECOVERY - DPKG on srv292 is OK: All packages OK
[23:16:47] <icinga-wm_>	 RECOVERY - DPKG on srv296 is OK: All packages OK
[23:16:48] <icinga-wm_>	 RECOVERY - DPKG on srv294 is OK: All packages OK
[23:16:48] <icinga-wm_>	 RECOVERY - DPKG on srv290 is OK: All packages OK
[23:16:49] <icinga-wm_>	 RECOVERY - Apache HTTP on srv239 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.101 second response time
[23:16:49] <icinga-wm_>	 RECOVERY - Apache HTTP on srv257 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.105 second response time
[23:16:50] <icinga-wm_>	 RECOVERY - Apache HTTP on srv235 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.113 second response time
[23:16:50] <icinga-wm_>	 RECOVERY - Apache HTTP on srv246 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.120 second response time
[23:16:51] <icinga-wm_>	 RECOVERY - Apache HTTP on srv245 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.120 second response time
[23:16:51] <icinga-wm_>	 RECOVERY - Apache HTTP on srv240 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.120 second response time
[23:16:52] <icinga-wm_>	 RECOVERY - Apache HTTP on srv238 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.123 second response time
[23:16:52] <icinga-wm_>	 RECOVERY - Apache HTTP on srv253 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.152 second response time
[23:16:53] <icinga-wm_>	 RECOVERY - Apache HTTP on srv254 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.151 second response time
[23:16:54] <icinga-wm_>	 RECOVERY - Apache HTTP on srv242 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.101 second response time
[23:17:01] <logmsgbot>	 !log mflaschen synchronized php-1.22wmf3/extensions/GettingStarted/  'E3 Lightning deploy for GettingStarted'
[23:17:04] <icinga-wm_>	 RECOVERY - Apache HTTP on srv243 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.101 second response time
[23:17:04] <icinga-wm_>	 RECOVERY - Apache HTTP on srv273 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.131 second response time
[23:17:04] <icinga-wm_>	 RECOVERY - Apache HTTP on srv261 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 747 bytes in 0.132 second response time
[23:17:04] <icinga-wm_>	 RECOVERY - DPKG on srv295 is OK: All packages OK
[23:17:04] <icinga-wm_>	 RECOVERY - DPKG on srv301 is OK: All packages OK
[23:17:07] <morebots>	 Logged the message, Master
[23:17:21] <LeslieCarr>	 sorry, i just love making icinga-wm spam the channel
[23:17:29] <superm401>	 Is "mw1041: ssh: connect to host mw1041 port 22: Connection timed out" expected?
[23:17:45] <gerrit-wm>	 Change merged: Dzahn; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62756
[23:17:54] <Reedy>	 Usually
[23:17:55] <LeslieCarr>	 superm401: yeah, mw1041 is having issues
[23:18:01] <LeslieCarr>	 well not usually ;)
[23:18:03] <LeslieCarr>	 just righ tnow
[23:18:05] <Reedy>	 heh
[23:18:12] <Reedy>	 I was meaning there's usually one or 2 hosts with issues
[23:18:22] <Reedy>	 Any more than that erroring, then yes, it's a problem that needs looking at
[23:19:27] <superm401>	 That was it
[23:20:04] <mutante>	 !log mw1041 - RT-4517, RT-5065, RT-5105 ...and broken again :p
[23:20:06] <icinga-wm_>	 ACKNOWLEDGEMENT - Host mw1041 is DOWN: PING CRITICAL - Packet loss = 100% LeslieCarr rt5105
[23:20:12] <morebots>	 Logged the message, Master
[23:20:45] <icinga-wm_>	 ACKNOWLEDGEMENT - Host marmontel is DOWN: PING CRITICAL - Packet loss = 100% LeslieCarr down due to security issue
[23:21:28] <mutante>	 !log removed mw1041 from dsh groups mediawiki-installation and apaches
[23:21:34] <LeslieCarr>	 superm401: but good to notify us here if something fails
[23:21:36] <morebots>	 Logged the message, Master
[23:22:17] <superm401>	 LeslieCarr, thanks, I figure it doesn't hurt to double-check.
[23:23:08] <mutante>	 !log depool mw1041 in pybal
[23:23:16] <morebots>	 Logged the message, Master
[23:23:22] <mutante>	 superm401: deactivated it:)
[23:23:33] <superm401>	 mutante, thanks. :)
[23:25:23] <logmsgbot>	 !log dzahn synchronized wmf-config/InitialiseSettings.php
[23:25:30] <morebots>	 Logged the message, Master
[23:28:40] <mutante>	 Reedy: as you reported with addWiki.php
[23:28:43] <mutante>	 Fatal error: /usr/local/apache/common-local/wikiversions.cdb has no version entry for `en`.
[23:28:49] <mutante>	 or different.. oh
[23:28:58] <Reedy>	 Just put --wiki=aawiki first
[23:29:03] <mutante>	 yea, but that's it, right
[23:29:04] <Reedy>	 Not sure when/why/how that regressed
[23:29:07] <mutante>	 nod
[23:29:29] <mutante>	 !log running addWiki.php for iegcomwiki
[23:29:37] <morebots>	 Logged the message, Master
[23:29:50] <mutante>	 Warning: fopen(/home/wikipedia/common/all.dblist): failed to open stream: Permission denied in /home/wikipedia/common/php-1.22wmf3/extensions/WikimediaMaintenance/addWiki.php on line 170
[23:29:55] <mutante>	 narf
[23:30:01] <Reedy>	 Yeah, they always happen
[23:30:15] <Reedy>	 script runs as apache, owned by wikidev group etc
[23:30:38] <Reedy>	 just add it to all.dblist and private.dblist, then run ./refresh-dblist and it'll do the cluster ones etc
[23:30:55] <mutante>	 cool,k
[23:31:39] <Reedy>	 Oh, and wikiversions.dat needs doing too ofc
[23:32:00] <mutante>	 ok, yea
[23:32:10] <mutante>	 php-1.22wmf2 or php-1.22wmf3
[23:32:15] <Reedy>	 wmf3
[23:33:15] <gerrit-wm>	 New patchset: Dzahn; "add iegcomwiki to dblists and wikiversions" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62761
[23:34:23] <logmsgbot>	 !log mwalker synchronized php-1.22wmf3/extensions/CentralNotice/  'Appling update to fix CentralNotice message editing bug'
[23:34:31] <morebots>	 Logged the message, Master
[23:35:17] <gerrit-wm>	 Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62761
[23:37:38] <mutante>	 Reedy:      updateinterwikicache  for private wikis?
[23:37:42] <mutante>	 nah, hmm
[23:37:52] <Reedy>	 nah
[23:38:03] <Reedy>	 Need to do it for loginwiki when it's actually working though ;)
[23:38:32] <logmsgbot>	 !log dzahn rebuilt wikiversions.cdb and synchronized wikiversions files:
[23:38:40] <morebots>	 Logged the message, Master
[23:41:19] <logmsgbot>	 !log catrope synchronized php-1.22wmf3/extensions/VisualEditor/modules/ve/ve.js  'touch'
[23:41:28] <morebots>	 Logged the message, Master
[23:41:47] <logmsgbot>	 !log catrope synchronized php-1.22wmf3/resources/startup.js  'touch'
[23:41:55] <morebots>	 Logged the message, Master
[23:47:34] <icinga-wm_>	 PROBLEM - Puppet freshness on ms-fe3001 is CRITICAL: No successful Puppet run in the last 10 hours
[23:52:56] <gerrit-wm>	 New patchset: Reedy; "Add iegcomwiki to s3 and special dblists" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62762
[23:53:33] <gerrit-wm>	 Change merged: jenkins-bot; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/62762