[02:02:39] (03PS3) 10Chad: Create global /etc/gitconfig for deployment hosts [operations/puppet] - 10https://gerrit.wikimedia.org/r/114385 [02:02:45] (03CR) 10Ori.livneh: [C: 032 V: 032] Create global /etc/gitconfig for deployment hosts [operations/puppet] - 10https://gerrit.wikimedia.org/r/114385 (owner: 10Chad) [02:14:26] !log LocalisationUpdate completed (1.23wmf16) at 2014-03-08 02:14:25+00:00 [02:14:37] Logged the message, Master [02:25:00] !log LocalisationUpdate completed (1.23wmf17) at 2014-03-08 02:25:00+00:00 [02:25:09] Logged the message, Master [02:55:58] !log LocalisationUpdate ResourceLoader cache refresh completed at Sat Mar 8 02:55:55 UTC 2014 (duration 55m 54s) [02:56:07] Logged the message, Master [03:04:36] !log shut down Tampa mw hosts 1-20 [03:04:45] Logged the message, Master [03:15:51] !log shut down Tampa mw hosts 21-27,49,58,75-85 (that is sdpta row D:D1) [03:15:58] Logged the message, Master [03:15:59] Reedy: there, check D1 again [03:16:12] get us the good torrus graph:) [03:16:12] For anyone following along http://torrus.wikimedia.org/torrus/Facilities?path=/Power_strips/sdtpa/ps1-d1-sdtpa/System/Power_density [03:16:19] and http://torrus.wikimedia.org/torrus/Facilities?path=/Power_usage/Total_power_usage/Power_per_site&view=longterm-rrd-html [03:21:48] (03CR) 10Dzahn: [C: 032] Add Laura Hale Sochi blog to English Planet [operations/puppet] - 10https://gerrit.wikimedia.org/r/117434 (owner: 10Odder) [03:28:07] !log upgrading nova-compute on virt100[1-6] [03:28:15] Logged the message, Master [03:41:58] !log shutting down srv193,srv235-srv247,srv248,249 [03:42:07] Logged the message, Master [03:42:45] srv193!?!?!? [03:43:02] # srv193 was test.wikipedia.org (precise) [03:43:19] # on 20130711 test has been switched over to mw1017 [03:44:01] heh [03:44:16] it was more it having a bit of significance/history [03:44:26] do you want to have it?:) [03:44:35] i think peter is taking db9 [03:44:37] heh [03:45:04] it'd probably be a good host to have due to it's low life workload [03:49:17] !log shut down srv250-srv247 (this was stdpa row D-D2) [03:49:25] Logged the message, Master [03:49:27] Reedy: there, D2, stopping here for now [03:49:29] gotta run [03:49:42] PROBLEM - MySQL InnoDB on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [03:50:01] D2 is showing 4k -> 2.3k so far [03:50:06] cool [03:50:32] RECOVERY - MySQL InnoDB on db1047 is OK: OK longest blocking idle transaction sleeps for 0 seconds [03:50:48] http://torrus.wikimedia.org/torrus/Facilities?path=/Power_strips/sdtpa/ps1-d2-sdtpa/System/Power_density [03:50:52] PROBLEM - MySQL Slave Running on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [03:53:42] PROBLEM - MySQL Idle Transactions on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [03:53:42] RECOVERY - MySQL Slave Running on db1047 is OK: OK replication Slave_IO_Running: Yes Slave_SQL_Running: Yes Last_Error: [03:53:42] PROBLEM - MySQL InnoDB on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [03:54:33] ^ that's "researchdb" and not in core dbtree, not sure though [03:54:34] springle-away: [03:55:10] Reedy: i'll enjoy that later when it updated a bit:) [03:55:18] nice graph [03:57:12] PROBLEM - MySQL Recent Restart on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [03:58:12] RECOVERY - MySQL Recent Restart on db1047 is OK: OK 4886749 seconds since restart [03:58:32] RECOVERY - MySQL Idle Transactions on db1047 is OK: OK longest blocking idle transaction sleeps for 0 seconds [03:58:32] RECOVERY - MySQL InnoDB on db1047 is OK: OK longest blocking idle transaction sleeps for 0 seconds [04:01:42] PROBLEM - MySQL Idle Transactions on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:02:42] PROBLEM - MySQL InnoDB on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:02:52] PROBLEM - MySQL Slave Running on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:04:32] RECOVERY - MySQL Idle Transactions on db1047 is OK: OK longest blocking idle transaction sleeps for 0 seconds [04:04:32] RECOVERY - MySQL InnoDB on db1047 is OK: OK longest blocking idle transaction sleeps for 0 seconds [04:04:42] RECOVERY - MySQL Slave Running on db1047 is OK: OK replication Slave_IO_Running: Yes Slave_SQL_Running: Yes Last_Error: [04:05:34] (03PS1) 10Andrew Bogott: When migrating instances, use flavors copied over from pmtpa. [operations/puppet] - 10https://gerrit.wikimedia.org/r/117641 [04:09:12] PROBLEM - MySQL Recent Restart on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:09:42] PROBLEM - MySQL InnoDB on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:10:02] RECOVERY - MySQL Recent Restart on db1047 is OK: OK 4887469 seconds since restart [04:11:42] PROBLEM - MySQL Idle Transactions on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:12:22] (03CR) 10Andrew Bogott: [C: 032] When migrating instances, use flavors copied over from pmtpa. [operations/puppet] - 10https://gerrit.wikimedia.org/r/117641 (owner: 10Andrew Bogott) [04:19:32] RECOVERY - MySQL Idle Transactions on db1047 is OK: OK longest blocking idle transaction sleeps for 0 seconds [04:19:32] RECOVERY - MySQL InnoDB on db1047 is OK: OK longest blocking idle transaction sleeps for 0 seconds [04:24:42] PROBLEM - MySQL Idle Transactions on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:25:42] PROBLEM - MySQL InnoDB on db1047 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:29:30] poor old analytics slave. it's a hard life [04:30:02] PROBLEM - MySQL Processlist on db1021 is CRITICAL: CRIT 1 unauthenticated, 0 locked, 0 copy to table, 274 statistics [04:33:12] RECOVERY - MySQL Processlist on db1021 is OK: OK 2 unauthenticated, 0 locked, 1 copy to table, 12 statistics [04:38:52] PROBLEM - Puppet freshness on tin is CRITICAL: Last successful Puppet run was Sat 08 Mar 2014 01:38:04 AM UTC [04:44:02] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: reqstats.5xx [crit=500.000000 [05:42:02] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: reqstats.5xx [warn=250.000 [07:27:21] (03CR) 10Dzahn: "err: Failed to apply catalog: Parameter unless failed: 'git config --global url."ssh://gerrit.wikimedia.org:29418/".insteadOf | grep -q "h" [operations/puppet] - 10https://gerrit.wikimedia.org/r/114385 (owner: 10Chad) [07:30:03] (03CR) 10Dzahn: "<+icinga-wm> PROBLEM - Puppet freshness on tin is CRITICAL: Last successful Puppet run was Sat 08 Mar 2014 01:38:04 AM UTC" [operations/puppet] - 10https://gerrit.wikimedia.org/r/114385 (owner: 10Chad) [07:37:24] (03PS1) 10Ori.livneh: Fix-up for Ia67474053: Create global /etc/gitconfig for deployment hosts [operations/puppet] - 10https://gerrit.wikimedia.org/r/117645 [07:37:50] (03CR) 10Ori.livneh: "@Dzahn: https://gerrit.wikimedia.org/r/#/c/117645/" [operations/puppet] - 10https://gerrit.wikimedia.org/r/114385 (owner: 10Chad) [07:38:19] (03CR) 10Ori.livneh: [C: 032 V: 032] Fix-up for Ia67474053: Create global /etc/gitconfig for deployment hosts [operations/puppet] - 10https://gerrit.wikimedia.org/r/117645 (owner: 10Ori.livneh) [07:38:47] (03CR) 10Dzahn: ":)" [operations/puppet] - 10https://gerrit.wikimedia.org/r/117645 (owner: 10Ori.livneh) [07:39:52] PROBLEM - Puppet freshness on tin is CRITICAL: Last successful Puppet run was Sat 08 Mar 2014 01:38:04 AM UTC [07:41:02] RECOVERY - Puppet freshness on tin is OK: puppet ran at Sat Mar 8 07:40:53 UTC 2014 [07:43:10] (03CR) 10Dzahn: "thanks:) good night! hehe" [operations/puppet] - 10https://gerrit.wikimedia.org/r/117645 (owner: 10Ori.livneh) [07:51:36] (03PS1) 10Ori.livneh: Revert "Create global /etc/gitconfig for deployment hosts" [operations/puppet] - 10https://gerrit.wikimedia.org/r/117646 [07:51:48] (03CR) 10jenkins-bot: [V: 04-1] Revert "Create global /etc/gitconfig for deployment hosts" [operations/puppet] - 10https://gerrit.wikimedia.org/r/117646 (owner: 10Ori.livneh) [07:52:55] (03PS2) 10Ori.livneh: Revert "Create global /etc/gitconfig for deployment hosts" [operations/puppet] - 10https://gerrit.wikimedia.org/r/117646 [07:53:13] (03CR) 10Ori.livneh: [C: 032 V: 032] Revert "Create global /etc/gitconfig for deployment hosts" [operations/puppet] - 10https://gerrit.wikimedia.org/r/117646 (owner: 10Ori.livneh) [08:23:59] (03CR) 10Gabrielchihonglee: [C: 031] Set wmgBabelCategoryNames for Chinese Wikivoyage [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/114970 (owner: 10Odder) [08:33:10] mutante|away: :-) [09:50:25] (03PS1) 10Ori.livneh: mwgrep: add '--user' option for searching NS_USER [operations/puppet] - 10https://gerrit.wikimedia.org/r/117647 [15:30:12] (03PS1) 10Nemo bis: [Planet Wikimedia] Add Vibha Bamba to English planet [operations/puppet] - 10https://gerrit.wikimedia.org/r/117659 [15:30:24] (03CR) 10jenkins-bot: [V: 04-1] [Planet Wikimedia] Add Vibha Bamba to English planet [operations/puppet] - 10https://gerrit.wikimedia.org/r/117659 (owner: 10Nemo bis) [15:33:45] (03PS2) 10Nemo bis: [Planet Wikimedia] Add Vibha Bamba to English planet [operations/puppet] - 10https://gerrit.wikimedia.org/r/117659 [17:00:36] (03CR) 10MZMcBride: [Planet Wikimedia] Add Vibha Bamba to English planet (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/117659 (owner: 10Nemo bis) [17:03:55] (03CR) 10Nemo bis: [Planet Wikimedia] Add Vibha Bamba to English planet (031 comment) [operations/puppet] - 10https://gerrit.wikimedia.org/r/117659 (owner: 10Nemo bis) [17:25:36] (03CR) 10MZMcBride: "I realize that you followed up on the design mailing list, but can you please e-mail the three Tumblr authors directly and confirm that th" [operations/puppet] - 10https://gerrit.wikimedia.org/r/117659 (owner: 10Nemo bis) [17:31:52] (03CR) 10Nemo bis: "No, I can't. Yours looks like mere speculation: the blog has no about page and "this blog which I started for the design team" by Vibha is" [operations/puppet] - 10https://gerrit.wikimedia.org/r/117659 (owner: 10Nemo bis) [17:32:25] (03PS3) 10Nemo bis: [Planet Wikimedia] Add Vibha Bamba to English planet [operations/puppet] - 10https://gerrit.wikimedia.org/r/117659 [17:38:28] (03CR) 10MZMcBride: "As I understand it, inclusion in Planet Wikimedia is typically opt-in. If someone doesn't wish to be listed or wishes to be attributed in " [operations/puppet] - 10https://gerrit.wikimedia.org/r/117659 (owner: 10Nemo bis) [17:38:35] * Gloria eyes Nemo_bis. [17:40:36] (03CR) 10Nemo bis: "[citation needed]" [operations/puppet] - 10https://gerrit.wikimedia.org/r/117659 (owner: 10Nemo bis) [17:41:11] Gloria: stop annoying me [17:41:55] That blog is not so interesting or important as to be worth such a conversation, planet is a simple aggregator [17:42:53] Heh. [17:43:16] Shrug, that change can wait until one of them chimes in. [18:46:45] (03Abandoned) 10Matanya: nagios: follow up fix for I235 [operations/puppet] - 10https://gerrit.wikimedia.org/r/112019 (owner: 10Matanya) [19:33:03] (03PS1) 10Dr0ptp4kt: Overwrite stale ZeroOpts= (blank) cookies if HTTPS zero-rated. [operations/puppet] - 10https://gerrit.wikimedia.org/r/117661 [21:44:05] Nemo: that tumblr blog is kind of rogue. It's kind of Vibha's thing, not the design team's. [22:11:46] jorm: nice, what I was trying to explain Gloria ;) [22:12:59] I just add to the Planet all wikimedia-related blogs, I don't see why aggregate them on my reader only when we have a shared aggregator. Nothing special. [23:02:59] I have logged a bugzilla for a mailing list password change for two lists, if there is someone around to coordinate that, that would be magic [23:03:08] well two list changes [23:04:18] I have no idea how that works (if I even have access... which I also don't know)... better come back during office hours, I guess