[00:13:44] There's a Dallas data center? [00:15:55] Gloria: There isn't. We're reviewing bids for a new datacenter that's off the east-coast power grid, and a site in Dallas is one of the front-runners. [00:16:02] I don't know if the decision is final though. [00:17:05] There's a draft RFP that makes it sound as though there is. :-) Hence the question. Okay, thanks. [00:17:42] Ah, yeah, getting hardware bids for the datacenter-to-be. [00:43:51] PROBLEM - Puppet freshness on labsdb1004 is CRITICAL: Last successful Puppet run was Fri 21 Mar 2014 03:41:16 PM UTC [00:45:22] (03PS1) 10Spage: Organize BetaFeatures labs overrides cleanly [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/120168 [01:04:19] (03PS1) 10Spage: Enable Popups (Hovercards) everywhere [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/120170 [01:14:05] (03PS1) 10Jforrester: Create a dblist for non-Beta Features wikis [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/120171 [01:14:07] (03PS1) 10Jforrester: Fix typo in noncirrus.dblist reference [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/120172 [01:16:37] (03CR) 10Krinkle: [C: 031] Fix typo in noncirrus.dblist reference [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/120172 (owner: 10Jforrester) [01:21:59] (03CR) 10Physikerwelt: [C: 031] Move math related packages to a puppet class [operations/debs/wikimedia-task-appserver] - 10https://gerrit.wikimedia.org/r/115135 (owner: 10Hashar) [01:22:55] (03CR) 10Physikerwelt: [C: 031] Describe Math related packages in a class [operations/puppet] - 10https://gerrit.wikimedia.org/r/115133 (owner: 10Hashar) [01:24:51] PROBLEM - Puppet freshness on labstore2 is CRITICAL: Last successful Puppet run was Fri 21 Mar 2014 01:17:26 AM UTC [01:30:03] (03CR) 10Spage: [C: 031] "I like it, minor quibble on the name." (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/120171 (owner: 10Jforrester) [01:46:08] (03PS1) 10MaxSem: Add a handler for HHVM fatals [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/120180 [01:46:20] (03CR) 10jenkins-bot: [V: 04-1] Add a handler for HHVM fatals [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/120180 (owner: 10MaxSem) [01:47:08] stupid PHP, tries to parse HHVM-only code:P [01:48:24] (03PS2) 10MaxSem: Add a handler for HHVM fatals [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/120180 [01:48:34] (03CR) 10jenkins-bot: [V: 04-1] Add a handler for HHVM fatals [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/120180 (owner: 10MaxSem) [01:49:34] (03CR) 10MaxSem: [C: 04-2] Add a handler for HHVM fatals [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/120180 (owner: 10MaxSem) [01:50:17] (03PS1) 10coren: Revert "Tool Labs: tweak to mail config" [operations/puppet] - 10https://gerrit.wikimedia.org/r/120181 [01:52:36] (03CR) 10coren: [C: 032] Revert "Tool Labs: tweak to mail config" [operations/puppet] - 10https://gerrit.wikimedia.org/r/120181 (owner: 10coren) [01:55:06] (03PS3) 10MaxSem: Add a handler for HHVM fatals [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/120180 [02:11:01] !log LocalisationUpdate completed (1.23wmf18) at 2014-03-22 02:11:01+00:00 [02:11:09] Logged the message, Master [02:18:55] !log LocalisationUpdate completed (1.23wmf19) at 2014-03-22 02:18:55+00:00 [02:19:00] Logged the message, Master [02:20:52] (03CR) 10Jforrester: Create a dblist for non-Beta Features wikis (031 comment) [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/120171 (owner: 10Jforrester) [02:44:00] !log LocalisationUpdate ResourceLoader cache refresh completed at Sat Mar 22 02:43:56 UTC 2014 (duration 43m 55s) [02:44:04] Logged the message, Master [02:57:04] (03CR) 10Ori.livneh: [C: 032] applicationserver::hhvm: add boost backports ppa for beta [operations/puppet] - 10https://gerrit.wikimedia.org/r/119979 (owner: 10Ori.livneh) [02:57:39] Coren: you have changes staged but unmerged; OK to merge? [02:58:15] * ori goes with "yes" [03:44:51] PROBLEM - Puppet freshness on labsdb1004 is CRITICAL: Last successful Puppet run was Fri 21 Mar 2014 03:41:16 PM UTC [03:54:38] (03PS1) 10Tim Landscheidt: WIP: Tools: Puppetize toolwatcher [operations/puppet] - 10https://gerrit.wikimedia.org/r/120186 [04:10:47] (03PS1) 10Tim Landscheidt: Tools: Install package libxml2-utils for xmllint [operations/puppet] - 10https://gerrit.wikimedia.org/r/120187 [04:25:51] PROBLEM - Puppet freshness on labstore2 is CRITICAL: Last successful Puppet run was Fri 21 Mar 2014 01:17:26 AM UTC [04:28:18] (03PS2) 10Tim Landscheidt: Tools: Puppetize toolwatcher [operations/puppet] - 10https://gerrit.wikimedia.org/r/120186 [06:45:51] PROBLEM - Puppet freshness on labsdb1004 is CRITICAL: Last successful Puppet run was Fri 21 Mar 2014 03:41:16 PM UTC [07:23:05] (03CR) 10Calak: [C: 031] Tools: Install package libxml2-utils for xmllint [operations/puppet] - 10https://gerrit.wikimedia.org/r/120187 (owner: 10Tim Landscheidt) [07:26:51] PROBLEM - Puppet freshness on labstore2 is CRITICAL: Last successful Puppet run was Fri 21 Mar 2014 01:17:26 AM UTC [08:45:09] (03PS2) 10Odder: Add new user groups to Spanish Wikiquote [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/120153 [09:34:11] PROBLEM - Varnishkafka Delivery Errors on cp3021 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 236.46666 [09:38:31] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 856.366638 [09:41:11] RECOVERY - Varnishkafka Delivery Errors on cp3021 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [09:43:31] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [09:46:51] PROBLEM - Puppet freshness on labsdb1004 is CRITICAL: Last successful Puppet run was Fri 21 Mar 2014 03:41:16 PM UTC [09:51:11] PROBLEM - Varnishkafka Delivery Errors on cp3021 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1897.766724 [09:51:21] PROBLEM - Varnishkafka Delivery Errors on cp3022 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 2862.199951 [09:53:21] RECOVERY - Varnishkafka Delivery Errors on cp3022 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [09:54:31] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 1218.133301 [09:57:08] RECOVERY - Varnishkafka Delivery Errors on cp3021 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [09:57:31] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [10:04:24] PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Sat 22 Mar 2014 10:00:11 AM UTC [10:06:24] PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Sat 22 Mar 2014 10:00:11 AM UTC [10:08:24] PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Sat 22 Mar 2014 10:00:11 AM UTC [10:10:24] PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Sat 22 Mar 2014 10:00:11 AM UTC [10:11:34] PROBLEM - Varnishkafka Delivery Errors on cp3019 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 2798.100098 [10:12:24] PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Sat 22 Mar 2014 10:00:11 AM UTC [10:13:34] RECOVERY - Varnishkafka Delivery Errors on cp3019 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [10:14:14] PROBLEM - Varnishkafka Delivery Errors on cp3021 is CRITICAL: kafka.varnishkafka.kafka_drerr.per_second CRITICAL: 754.133362 [10:14:24] PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Sat 22 Mar 2014 10:00:11 AM UTC [10:15:14] RECOVERY - Varnishkafka Delivery Errors on cp3021 is OK: kafka.varnishkafka.kafka_drerr.per_second OKAY: 0.0 [10:16:24] PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Sat 22 Mar 2014 10:00:11 AM UTC [10:18:24] PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Sat 22 Mar 2014 10:00:11 AM UTC [10:20:24] PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Sat 22 Mar 2014 10:00:11 AM UTC [10:22:24] PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Sat 22 Mar 2014 10:00:11 AM UTC [10:24:24] PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Sat 22 Mar 2014 10:00:11 AM UTC [10:26:24] PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Sat 22 Mar 2014 10:00:11 AM UTC [10:27:24] PROBLEM - Puppet freshness on labstore2 is CRITICAL: Last successful Puppet run was Fri 21 Mar 2014 01:17:26 AM UTC [10:28:24] PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Sat 22 Mar 2014 10:00:11 AM UTC [10:30:24] PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Sat 22 Mar 2014 10:00:11 AM UTC [10:30:34] RECOVERY - Puppet freshness on mw1097 is OK: puppet ran at Sat Mar 22 10:30:31 UTC 2014 [10:31:43] (03CR) 10Alexandros Kosiaris: "The problem is the default ferm policy. Should bastion hosts include this, all connections to them that are not specifically allowed will " [operations/puppet] - 10https://gerrit.wikimedia.org/r/96424 (owner: 10Dzahn) [10:32:24] PROBLEM - Puppet freshness on mw1097 is CRITICAL: Last successful Puppet run was Sat 22 Mar 2014 10:30:31 AM UTC [11:00:25] RECOVERY - Puppet freshness on mw1097 is OK: puppet ran at Sat Mar 22 11:00:18 UTC 2014 [12:47:25] PROBLEM - Puppet freshness on labsdb1004 is CRITICAL: Last successful Puppet run was Fri 21 Mar 2014 03:41:16 PM UTC [12:51:29] akosiaris: You working on labsdb1004? [13:13:29] (03CR) 10Hashar: "> Could you set up puppet-lint as non-voting for operations/puppet?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/108018 (owner: 10Hashar) [13:27:25] PROBLEM - Puppet freshness on labstore2 is CRITICAL: Last successful Puppet run was Fri 21 Mar 2014 01:17:26 AM UTC [14:04:07] PROBLEM - Puppet freshness on db72 is CRITICAL: Last successful Puppet run was Sat 22 Mar 2014 01:58:41 PM UTC [14:06:07] PROBLEM - Puppet freshness on db72 is CRITICAL: Last successful Puppet run was Sat 22 Mar 2014 01:58:41 PM UTC [14:08:07] PROBLEM - Puppet freshness on db72 is CRITICAL: Last successful Puppet run was Sat 22 Mar 2014 01:58:41 PM UTC [14:10:07] PROBLEM - Puppet freshness on db72 is CRITICAL: Last successful Puppet run was Sat 22 Mar 2014 01:58:41 PM UTC [14:12:07] PROBLEM - Puppet freshness on db72 is CRITICAL: Last successful Puppet run was Sat 22 Mar 2014 01:58:41 PM UTC [14:14:07] PROBLEM - Puppet freshness on db72 is CRITICAL: Last successful Puppet run was Sat 22 Mar 2014 01:58:41 PM UTC [14:16:07] PROBLEM - Puppet freshness on db72 is CRITICAL: Last successful Puppet run was Sat 22 Mar 2014 01:58:41 PM UTC [14:18:07] PROBLEM - Puppet freshness on db72 is CRITICAL: Last successful Puppet run was Sat 22 Mar 2014 01:58:41 PM UTC [14:20:07] PROBLEM - Puppet freshness on db72 is CRITICAL: Last successful Puppet run was Sat 22 Mar 2014 01:58:41 PM UTC [14:22:07] PROBLEM - Puppet freshness on db72 is CRITICAL: Last successful Puppet run was Sat 22 Mar 2014 01:58:41 PM UTC [14:24:07] PROBLEM - Puppet freshness on db72 is CRITICAL: Last successful Puppet run was Sat 22 Mar 2014 01:58:41 PM UTC [14:26:07] PROBLEM - Puppet freshness on db72 is CRITICAL: Last successful Puppet run was Sat 22 Mar 2014 01:58:41 PM UTC [14:28:07] PROBLEM - Puppet freshness on db72 is CRITICAL: Last successful Puppet run was Sat 22 Mar 2014 01:58:41 PM UTC [14:28:57] RECOVERY - Puppet freshness on db72 is OK: puppet ran at Sat Mar 22 14:28:49 UTC 2014 [14:31:07] PROBLEM - Puppet freshness on db72 is CRITICAL: Last successful Puppet run was Sat 22 Mar 2014 02:28:49 PM UTC [14:50:34] Coren: yes. And seems like I will be for a long time still. Dont enable puppet please [14:51:33] akosiaris: No worries, just wanted to know who to point Faidon at. :-) [14:58:34] RECOVERY - Puppet freshness on db72 is OK: puppet ran at Sat Mar 22 14:58:32 UTC 2014 [15:37:18] (03PS1) 10Faidon Liambotis: install-server fix logrotate for squid3 [operations/puppet] - 10https://gerrit.wikimedia.org/r/120211 [15:37:43] (03CR) 10Faidon Liambotis: [C: 032] install-server fix logrotate for squid3 [operations/puppet] - 10https://gerrit.wikimedia.org/r/120211 (owner: 10Faidon Liambotis) [15:43:04] PROBLEM - Host ms-be8 is DOWN: PING CRITICAL - Packet loss = 100% [15:45:14] RECOVERY - Host ms-be8 is UP: PING OK - Packet loss = 0%, RTA = 35.45 ms [15:47:54] PROBLEM - Puppet freshness on labsdb1004 is CRITICAL: Last successful Puppet run was Fri 21 Mar 2014 03:41:16 PM UTC [16:27:54] PROBLEM - Puppet freshness on labstore2 is CRITICAL: Last successful Puppet run was Fri 21 Mar 2014 01:17:26 AM UTC [18:26:28] (03CR) 10Matanya: [C: 04-1] "did you see the request at the top of this file? Please abandon." [operations/puppet] - 10https://gerrit.wikimedia.org/r/120005 (owner: 10Hashar) [18:31:22] (03CR) 10Matanya: [C: 031] package-builder.pp: convert notify to use 'message' [operations/puppet] - 10https://gerrit.wikimedia.org/r/120008 (owner: 10Hashar) [18:37:51] (03CR) 10Tim Landscheidt: "That's absurd. Keeping the repository in a less than optimal shape so that recruiters have something to play around with is very wrong. " [operations/puppet] - 10https://gerrit.wikimedia.org/r/120005 (owner: 10Hashar) [18:47:34] RECOVERY - Puppet freshness on labsdb1004 is OK: puppet ran at Sat Mar 22 18:47:28 UTC 2014 [18:54:31] (03CR) 10Matanya: package-builder.pp: parameterized $pbuilder_root (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/120009 (owner: 10Hashar) [18:59:01] (03CR) 10Matanya: "looks sane, i would wait with those changes until pmtpa is out." [operations/puppet] - 10https://gerrit.wikimedia.org/r/120013 (owner: 10Hashar) [19:03:22] (03CR) 10Matanya: [C: 031] decom: remove pmtpa search pools and searchidx2 [operations/dns] - 10https://gerrit.wikimedia.org/r/120062 (owner: 10Dzahn) [19:04:14] (03CR) 10Matanya: [C: 031] decom Tampa: remove service IPs [operations/dns] - 10https://gerrit.wikimedia.org/r/120063 (owner: 10Dzahn) [19:17:59] (03CR) 10Matanya: [C: 031] "I guess this is a left over from hardy. there was no timidity-daemon in hardy. the timidity-daemon package was added at 17-Sep-2009, which" [operations/puppet] - 10https://gerrit.wikimedia.org/r/118709 (owner: 10Hashar) [19:28:54] PROBLEM - Puppet freshness on labstore2 is CRITICAL: Last successful Puppet run was Fri 21 Mar 2014 01:17:26 AM UTC [19:56:19] (03PS1) 10Tim Landscheidt: Tools: Rename references to local-admin to tools.admin [operations/puppet] - 10https://gerrit.wikimedia.org/r/120241 [20:05:35] (03PS1) 10Tim Landscheidt: Tools: Rename reference to "local-" to "tools." [operations/puppet] - 10https://gerrit.wikimedia.org/r/120242 [20:22:29] (03PS1) 10Tim Landscheidt: Tools: Adapt proxylistener to eqiad naming scheme [operations/puppet] - 10https://gerrit.wikimedia.org/r/120243 [21:00:55] (03CR) 10coren: [C: 032] Tools: Rename reference to "local-" to "tools." [operations/puppet] - 10https://gerrit.wikimedia.org/r/120242 (owner: 10Tim Landscheidt) [21:06:31] scfc_de: i wanted to say i really appreciate your work, thank you. [21:14:12] matanya: I think you're doing a lot more than me :-). [21:14:30] :) [22:29:54] PROBLEM - Puppet freshness on labstore2 is CRITICAL: Last successful Puppet run was Fri 21 Mar 2014 01:17:26 AM UTC