[00:07:55] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [00:07:55] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [00:18:07] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:32:58] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.033 seconds [00:42:52] PROBLEM - Puppet freshness on cp1040 is CRITICAL: Puppet has not run in the last 10 hours [01:05:58] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:20:13] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 3.206 seconds [01:42:16] PROBLEM - MySQL Slave Delay on db1025 is CRITICAL: CRIT replication delay 302 seconds [01:45:16] RECOVERY - MySQL Slave Delay on db1025 is OK: OK replication delay 20 seconds [01:49:55] PROBLEM - Puppet freshness on analytics1001 is CRITICAL: Puppet has not run in the last 10 hours [01:53:40] PROBLEM - Puppetmaster HTTPS on stafford is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:00:48] !log LocalisationUpdate failed: git pull of extensions failed [02:01:04] Logged the message, Master [02:08:58] PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours [02:09:16] RECOVERY - Puppetmaster HTTPS on stafford is OK: HTTP OK HTTP/1.1 400 Bad Request - 336 bytes in 0.017 seconds [02:56:58] PROBLEM - Puppet freshness on ms-be7 is CRITICAL: Puppet has not run in the last 10 hours [03:02:58] PROBLEM - Puppet freshness on db42 is CRITICAL: Puppet has not run in the last 10 hours [03:07:46] RECOVERY - Puppet freshness on analytics1001 is OK: puppet ran at Sat Oct 13 03:07:21 UTC 2012 [04:11:18] Is there a go-to person for getting the status of an RT ticket? [04:11:27] Is that something Andre can do? [04:23:15] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [05:46:04] PROBLEM - Puppet freshness on magnesium is CRITICAL: Puppet has not run in the last 10 hours [05:46:04] PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours [06:09:01] PROBLEM - Puppet freshness on storage3 is CRITICAL: Puppet has not run in the last 10 hours [06:24:55] PROBLEM - MySQL Slave Delay on db1042 is CRITICAL: CRIT replication delay 182 seconds [06:24:55] PROBLEM - MySQL Replication Heartbeat on db1042 is CRITICAL: CRIT replication delay 182 seconds [06:28:04] RECOVERY - MySQL Slave Delay on db1042 is OK: OK replication delay 0 seconds [06:28:04] RECOVERY - MySQL Replication Heartbeat on db1042 is OK: OK replication delay 0 seconds [07:03:56] PROBLEM - MySQL Replication Heartbeat on db1042 is CRITICAL: CRIT replication delay 223 seconds [07:04:14] PROBLEM - MySQL Slave Delay on db1042 is CRITICAL: CRIT replication delay 227 seconds [07:10:05] RECOVERY - MySQL Replication Heartbeat on db1042 is OK: OK replication delay 0 seconds [07:10:32] RECOVERY - MySQL Slave Delay on db1042 is OK: OK replication delay 5 seconds [08:07:01] PROBLEM - MySQL Slave Delay on db1042 is CRITICAL: CRIT replication delay 191 seconds [08:07:10] PROBLEM - MySQL Replication Heartbeat on db1042 is CRITICAL: CRIT replication delay 185 seconds [08:10:10] PROBLEM - MySQL Replication Heartbeat on db1042 is CRITICAL: CRIT replication delay 187 seconds [08:16:19] PROBLEM - MySQL Slave Delay on db1042 is CRITICAL: CRIT replication delay 192 seconds [08:16:28] PROBLEM - MySQL Replication Heartbeat on db1042 is CRITICAL: CRIT replication delay 194 seconds [08:17:58] RECOVERY - MySQL Slave Delay on db1042 is OK: OK replication delay 0 seconds [08:18:07] RECOVERY - MySQL Replication Heartbeat on db1042 is OK: OK replication delay 0 seconds [08:48:57] PROBLEM - MySQL Replication Heartbeat on db1042 is CRITICAL: CRIT replication delay 186 seconds [08:49:15] PROBLEM - MySQL Slave Delay on db1042 is CRITICAL: CRIT replication delay 199 seconds [08:59:54] RECOVERY - MySQL Replication Heartbeat on db1042 is OK: OK replication delay 0 seconds [09:00:12] RECOVERY - MySQL Slave Delay on db1042 is OK: OK replication delay 0 seconds [10:09:08] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [10:09:08] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [10:44:05] PROBLEM - Puppet freshness on cp1040 is CRITICAL: Puppet has not run in the last 10 hours [12:09:47] PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours [12:30:53] https://gerrit.wikimedia.org/r/#/c/27422/ If my calculations are correct this begins in 2 hours... Is there anyone around who can deploy? [12:47:19] I kinda wish throttling could be controlled via the wiki and permissions just granted out to admins rather than hacking up the config every time. [12:48:49] Damianz, so do I, but at the moment it's stuck in the config [12:57:45] PROBLEM - Puppet freshness on ms-be7 is CRITICAL: Puppet has not run in the last 10 hours [13:03:45] PROBLEM - Puppet freshness on db42 is CRITICAL: Puppet has not run in the last 10 hours [13:10:38] it's not really necessary to hack configs [13:15:37] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/27422 [13:16:23] Nemo_bis, hm? Can this be done on the wiki now? [13:16:58] !log reedy synchronized wmf-config/throttle.php [13:17:11] Logged the message, Master [13:22:45] New patchset: Reedy; "Revert "(bug 40911) Raise account creation throttle for Malayalam Wikipedia workshop"" [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/27868 [13:26:16] ... Why are you reverting? o.O [13:31:52] [14:21:46] Reedy: and the workshop has ended [13:34:29] So it began at 4:30 AM and ended at 11:30 AM UTC? damn timezones... [13:34:47] I think all of the exemptions in there have expired, might as well remove them Reedy [14:24:14] PROBLEM - Puppet freshness on neon is CRITICAL: Puppet has not run in the last 10 hours [15:47:07] PROBLEM - Puppet freshness on magnesium is CRITICAL: Puppet has not run in the last 10 hours [15:47:07] PROBLEM - Puppet freshness on zinc is CRITICAL: Puppet has not run in the last 10 hours [16:10:21] PROBLEM - Puppet freshness on storage3 is CRITICAL: Puppet has not run in the last 10 hours [16:16:42] Change merged: Reedy; [operations/mediawiki-config] (master) - https://gerrit.wikimedia.org/r/27868 [16:27:14] !log reedy synchronized php-1.21wmf1/extensions/Collection [16:27:26] Logged the message, Master [16:34:43] !log reedy synchronized php-1.21wmf1/extensions/Collection [16:34:55] Logged the message, Master [18:21:09] !log reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to 9f37005e78a068821c45d9d313394c6728f55b4a' [18:21:20] Logged the message, Master [18:23:27] !log reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to 72b7d4d645489dd092605a2183a944fac33cb40b' [18:23:38] Logged the message, Master [18:25:03] !log reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to 8415a4564bf429b76e113368a3e5e1c19fb84a47' [18:25:15] Logged the message, Master [18:26:33] !log reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to original' [18:26:45] Logged the message, Master [18:34:29] !log reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to 72b7d4d645489dd092605a2183a944fac33cb40b' [18:34:41] Logged the message, Master [18:36:17] !log reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to 4f898444ce5d9c2c9a5491f2303db15a3143a5c5' [18:36:29] Logged the message, Master [18:38:04] !log reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to f980f2380b830e712edfdbce065784d1bcba82fd' [18:38:16] Logged the message, Master [18:41:06] !log reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to original again..' [18:41:18] Logged the message, Master [18:51:20] !log reedy synchronized php-1.21wmf1/extensions/Collection 'Revert to master' [18:51:31] Logged the message, Master [20:10:39] PROBLEM - Puppet freshness on ocg3 is CRITICAL: Puppet has not run in the last 10 hours [20:10:39] PROBLEM - Puppet freshness on virt1004 is CRITICAL: Puppet has not run in the last 10 hours [20:19:41] New patchset: preilly; "Wikipedia Zero Saudi Launch" [operations/puppet] (production) - https://gerrit.wikimedia.org/r/27892 [20:20:22] paravoid: ping [20:20:34] pong [20:20:45] merge this? [20:20:46] New review: gerrit2; "Lint check passed." [operations/puppet] (production); V: 1 - https://gerrit.wikimedia.org/r/27892 [20:21:34] paravoid: this is what we need merged in 4 hours [20:21:42] 4? [20:21:45] I thought 8 [20:21:50] paravoid: will you be around to merge and push it at that time [20:22:01] why not push it now? [20:22:24] in 4 I'll probably be around (3am), in 8 it's getting too late/early but I can make it if none of the SF people can [20:22:25] paravoid: between 5pm-7pm PST Saturday [20:23:02] paravoid: yeah it's 12:22am Sunday (GST) - Time in Dubai [20:23:11] paravoid: let's just do it now [20:23:13] why not push it now? [20:23:16] paravoid: good call [20:23:19] heh [20:23:24] paravoid: yeah right now is good [20:23:37] Change merged: Faidon; [operations/puppet] (production) - https://gerrit.wikimedia.org/r/27892 [20:24:08] done [20:24:20] force-running puppet [20:24:23] paravoid: can you push it too [20:24:27] paravoid: thanks man [20:25:08] is that all you'll need? [20:25:24] should I just reply to the thread and let others know that it's done? [20:26:48] paravoid: I just responded to the thread [20:27:02] paravoid: can you look at my reply and see if you think that is good enough [20:32:21] paravoid: ? [20:33:49] I replied [20:34:28] paravoid: cool thanks for your help [20:34:33] paravoid: talk to you later [20:34:51] hi [20:34:54] what's up? [20:35:39] mark: do you have any issue with me having +2 on /operations/debs/varnish [20:36:25] no [20:36:27] what do you want to do? [20:37:11] mark: just wanted to approve the open change by Tim to varnishncsa [20:37:22] mark: as well as being able to work a bit more on the patch [20:37:39] mark: can you reply on https://rt.wikimedia.org/Ticket/Display.html?id=3708&results=5ddf220ee02537e37df36cb6175455c5 [20:40:15] done [20:40:17] nothing's up, don't worry. they're deploying a new Wikipedia Zero partner tomorrow and they needed ops support to merge ACLs etc. [20:41:12] mark: thanks mark [20:41:15] there's a thread that ops@ isn't Cc'ed and CT has mailed to a few of us [20:41:50] fwiw, I'm okay with Patrick getting +2 too [20:42:03] in case my reply wasn't clear [20:42:58] paravoid, mark: I totally agree that we should always get another set of eyes to review changes [20:45:36] PROBLEM - Puppet freshness on cp1040 is CRITICAL: Puppet has not run in the last 10 hours [21:06:08] !log preilly synchronized php-1.21wmf1/extensions/MobileFrontend 'update for landing page' [21:06:20] Logged the message, Master [21:18:18] !log preilly Started syncing Wikimedia installation... : update zero rated mobile access [21:18:30] Logged the message, Master [21:39:29] !log preilly Finished syncing Wikimedia installation... : update zero rated mobile access [21:39:41] Logged the message, Master [22:10:49] PROBLEM - Puppet freshness on zhen is CRITICAL: Puppet has not run in the last 10 hours [22:58:49] PROBLEM - Puppet freshness on ms-be7 is CRITICAL: Puppet has not run in the last 10 hours [23:04:49] PROBLEM - Puppet freshness on db42 is CRITICAL: Puppet has not run in the last 10 hours