[00:15:12] <grrrit-wm>	 (03PS1) 10Ottomata: Set high proxy_read_timeout for archiva nginx proxy [puppet] - 10https://gerrit.wikimedia.org/r/187837 
[00:18:36] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032] Set high proxy_read_timeout for archiva nginx proxy [puppet] - 10https://gerrit.wikimedia.org/r/187837 (owner: 10Ottomata)
[00:58:39] <grrrit-wm>	 (03PS2) 10Spage: Add Dev namespace on mediawiki.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/187278 (https://phabricator.wikimedia.org/T369) 
[00:59:12] <grrrit-wm>	 (03CR) 10Spage: "Dzahn: d'oh, fixed." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/187278 (https://phabricator.wikimedia.org/T369) (owner: 10Spage)
[01:04:46] <jamesofur>	 Folks aware of the wikitech outage yet?
[01:06:00] <marktraceur>	 Maybe connected to the #wikipedia report of 503
[01:06:06] <marktraceur>	 No icinga-wm messages yet though
[01:06:14] <YuviPanda|bus>	 Yes
[01:06:15] <YuviPanda|bus>	 We are
[01:06:22] <jamesofur>	 cool :)
[01:06:27] <YuviPanda|bus>	 marktraceur: there was a shinken message in our inboxes
[01:06:42] <marktraceur>	 So much cabal
[01:08:51] * YuviPanda|bus sends it to marktraceur's inbox instead
[01:09:09] <marktraceur>	 :)
[01:09:17] <marktraceur>	 No thanks, have fun
[01:10:45] <YuviPanda|bus>	 marktraceur: jamesofur back up
[01:13:18] <grrrit-wm>	 (03PS1) 10Ottomata: Also increase nginx archiva proxy send and connect timeout [puppet] - 10https://gerrit.wikimedia.org/r/187852 
[01:14:15] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032] Also increase nginx archiva proxy send and connect timeout [puppet] - 10https://gerrit.wikimedia.org/r/187852 (owner: 10Ottomata)
[01:38:47] <jamesofur>	 YuviPanda|bus: thanks!
[01:55:30] <grrrit-wm>	 (03Abandoned) 10Matthias Mullie: Add $wgParsoid... variables [mediawiki-config] - 10https://gerrit.wikimedia.org/r/187625 (owner: 10Matthias Mullie)
[02:10:55] <logmsgbot>	 !log l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 02s)
[02:11:07] <morebots>	 Logged the message, Master
[02:12:02] <logmsgbot>	 !log LocalisationUpdate completed (1.25wmf14) at 2015-01-31 02:10:58+00:00
[02:12:06] <morebots>	 Logged the message, Master
[02:24:23] <logmsgbot>	 !log l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 02s)
[02:24:28] <morebots>	 Logged the message, Master
[02:25:30] <logmsgbot>	 !log LocalisationUpdate completed (1.25wmf15) at 2015-01-31 02:24:27+00:00
[02:25:34] <morebots>	 Logged the message, Master
[03:11:07] <grrrit-wm>	 (03PS1) 10Mattflaschen: Update Flow Parsoid config comment [mediawiki-config] - 10https://gerrit.wikimedia.org/r/187869 (https://phabricator.wikimedia.org/T86920) 
[03:12:37] <grrrit-wm>	 (03CR) 10Mattflaschen: [C: 031] "This is a simple doc update that can be merged when someone is already deploying a config change." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/187869 (https://phabricator.wikimedia.org/T86920) (owner: 10Mattflaschen)
[03:32:22] <grrrit-wm>	 (03PS1) 10Yuvipanda: shinken: Make puppet failure checks depend on wikitech being up [puppet] - 10https://gerrit.wikimedia.org/r/187872 
[03:33:32] <grrrit-wm>	 (03PS2) 10Yuvipanda: shinken: Make puppet failure checks depend on wikitech being up [puppet] - 10https://gerrit.wikimedia.org/r/187872 
[03:34:09] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032] shinken: Make puppet failure checks depend on wikitech being up [puppet] - 10https://gerrit.wikimedia.org/r/187872 (owner: 10Yuvipanda)
[03:35:22] <icinga-wm>	 PROBLEM - puppet last run on cp4020 is CRITICAL: CRITICAL: puppet fail  
[03:55:22] <icinga-wm>	 RECOVERY - puppet last run on cp4020 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures  
[03:57:33] <wikibugs>	 3Tool-Labs, operations: Replag on labsdb - https://phabricator.wikimedia.org/T88183#1005949 (10yuvipanda) A few hours later! Replag is reducing. I killed a query on labsdb that looked inefficient and had been running for about a hour and a half (and was referencing the rev table, which was also what was implicat...
[04:12:34] <logmsgbot>	 !log LocalisationUpdate ResourceLoader cache refresh completed at Sat Jan 31 04:11:31 UTC 2015 (duration 11m 29s)
[04:12:40] <morebots>	 Logged the message, Master
[04:47:17] <wikibugs>	 3Tool-Labs, operations: Replag on labsdb - https://phabricator.wikimedia.org/T88183#1005975 (10yuvipanda) Lag is 0 everywhere according to tendril.  We really should have lag info public somewhere. Let me try to get it into graphite.
[05:03:41] <wikibugs>	 3Beta-Cluster, operations: Set up an alert for unmerged changes in deployment-prep - https://phabricator.wikimedia.org/T87616#1005980 (10yuvipanda) @Dzahn: We can't, since beta's puppetmaster automatically rebases on production and so we'll need to alert on conflicts as well. Plus that is run on prod as an activ...
[05:32:14] <grrrit-wm>	 (03PS1) 10Yuvipanda: apertium: Remove jenkins_access class [puppet] - 10https://gerrit.wikimedia.org/r/187885 (https://phabricator.wikimedia.org/T86644) 
[05:37:52] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032] apertium: Remove jenkins_access class [puppet] - 10https://gerrit.wikimedia.org/r/187885 (https://phabricator.wikimedia.org/T86644) (owner: 10Yuvipanda)
[05:40:18] <wikibugs>	 3operations: Cannot use dsh-based restart of parsoid from tin anymore - https://phabricator.wikimedia.org/T87803#1005996 (10yuvipanda) Every time @ssastry has asked me to restart parsoid across the cluster I've always used salt (with a -b option) to do rolling restarts. What problems does it cause?  +1 to alex,...
[06:26:59] <grrrit-wm>	 (03PS1) 10Legoktm: Enable GlobalUserPage on test* wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/187888 (https://phabricator.wikimedia.org/T72576) 
[06:29:02] <icinga-wm>	 PROBLEM - puppet last run on db1015 is CRITICAL: CRITICAL: Puppet has 1 failures  
[06:29:19] <grrrit-wm>	 (03CR) 10Legoktm: "Note that the extension currently isn't branched, I4d3ff7a7cac7cba395941b43d524e1403508b232 updates make-wmf-branches for that." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/187888 (https://phabricator.wikimedia.org/T72576) (owner: 10Legoktm)
[06:29:41] <icinga-wm>	 PROBLEM - puppet last run on mw1166 is CRITICAL: CRITICAL: Puppet has 1 failures  
[06:29:52] <icinga-wm>	 PROBLEM - puppet last run on db1051 is CRITICAL: CRITICAL: Puppet has 1 failures  
[06:30:12] <icinga-wm>	 PROBLEM - puppet last run on db1021 is CRITICAL: CRITICAL: Puppet has 1 failures  
[06:30:22] <icinga-wm>	 PROBLEM - puppet last run on cp1056 is CRITICAL: CRITICAL: Puppet has 1 failures  
[06:31:02] <icinga-wm>	 PROBLEM - puppet last run on mw1092 is CRITICAL: CRITICAL: Puppet has 1 failures  
[06:31:12] <icinga-wm>	 PROBLEM - puppet last run on mw1172 is CRITICAL: CRITICAL: Puppet has 1 failures  
[06:31:22] <icinga-wm>	 PROBLEM - puppet last run on mw1118 is CRITICAL: CRITICAL: Puppet has 1 failures  
[06:31:22] <icinga-wm>	 PROBLEM - puppet last run on cp4003 is CRITICAL: CRITICAL: Puppet has 1 failures  
[06:31:33] <icinga-wm>	 PROBLEM - puppet last run on mw1025 is CRITICAL: CRITICAL: Puppet has 1 failures  
[06:31:42] <icinga-wm>	 PROBLEM - puppet last run on mw1054 is CRITICAL: CRITICAL: Puppet has 1 failures  
[06:32:22] <icinga-wm>	 PROBLEM - puppet last run on cp4008 is CRITICAL: CRITICAL: Puppet has 1 failures  
[06:32:55] <YuviPanda>	 hmm
[06:33:01] <YuviPanda>	 is this the puppet matser again?
[06:33:03] * YuviPanda waits for a bit
[06:45:52] <icinga-wm>	 RECOVERY - puppet last run on db1015 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures  
[06:46:12] <icinga-wm>	 RECOVERY - puppet last run on cp1056 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures  
[06:46:12] <icinga-wm>	 RECOVERY - puppet last run on mw1118 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures  
[06:46:21] <icinga-wm>	 RECOVERY - puppet last run on mw1025 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures  
[06:46:31] <icinga-wm>	 RECOVERY - puppet last run on mw1166 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures  
[06:46:31] <icinga-wm>	 RECOVERY - puppet last run on mw1054 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures  
[06:46:41] <icinga-wm>	 RECOVERY - puppet last run on db1051 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures  
[06:46:52] <icinga-wm>	 RECOVERY - puppet last run on mw1092 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures  
[06:47:02] <icinga-wm>	 RECOVERY - puppet last run on db1021 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures  
[06:47:02] <icinga-wm>	 RECOVERY - puppet last run on mw1172 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures  
[06:47:12] <icinga-wm>	 RECOVERY - puppet last run on cp4008 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures  
[06:48:12] <icinga-wm>	 RECOVERY - puppet last run on cp4003 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures  
[08:17:21] <grrrit-wm>	 (03CR) 10Glaisher: Enable GlobalUserPage on test* wikis (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/187888 (https://phabricator.wikimedia.org/T72576) (owner: 10Legoktm)
[08:18:05] <grrrit-wm>	 (03PS1) 10Yuvipanda: tools: Move webservice2 into puppet repo [puppet] - 10https://gerrit.wikimedia.org/r/187891 
[08:18:24] <grrrit-wm>	 (03CR) 10Legoktm: Enable GlobalUserPage on test* wikis (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/187888 (https://phabricator.wikimedia.org/T72576) (owner: 10Legoktm)
[08:18:56] <grrrit-wm>	 (03PS2) 10Yuvipanda: tools: Move webservice2 into puppet repo [puppet] - 10https://gerrit.wikimedia.org/r/187891 
[08:19:54] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032] tools: Move webservice2 into puppet repo [puppet] - 10https://gerrit.wikimedia.org/r/187891 (owner: 10Yuvipanda)
[08:58:23] <grrrit-wm>	 (03PS1) 10Yuvipanda: tools: Add webservice2 to tools-submit as well [puppet] - 10https://gerrit.wikimedia.org/r/187894 
[08:58:41] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032] tools: Add webservice2 to tools-submit as well [puppet] - 10https://gerrit.wikimedia.org/r/187894 (owner: 10Yuvipanda)
[09:11:06] <grrrit-wm>	 (03PS1) 10Yuvipanda: toollabs: Add python3 support to uwsgi web services [puppet] - 10https://gerrit.wikimedia.org/r/187895 
[09:11:15] <PissedPanda>	 legoktm: ^ py3 support
[09:11:49] <legoktm>	 yay!
[09:12:26] <grrrit-wm>	 (03CR) 10Legoktm: [C: 031] toollabs: Add python3 support to uwsgi web services [puppet] - 10https://gerrit.wikimedia.org/r/187895 (owner: 10Yuvipanda)
[09:12:40] <YuviPanda>	 legoktm: will you have a few mins to test this out? :)
[09:12:44] <legoktm>	 sure!
[09:13:11] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032] toollabs: Add python3 support to uwsgi web services [puppet] - 10https://gerrit.wikimedia.org/r/187895 (owner: 10Yuvipanda)
[09:13:16] <YuviPanda>	 legoktm: let me merge. convert your venv to py3? :)
[09:13:50] * legoktm does
[09:14:57] <legoktm>	 YuviPanda: venv is now py3 and stopped the webservice
[09:15:13] <YuviPanda>	 legoktm: ok. gimme a moment, running puppet on both trusty login and uwsgi host
[09:15:46] <YuviPanda>	 legoktm: ok try now
[09:16:45] <legoktm>	 YuviPanda: died. http://fpaste.org/179096/95799142/raw/
[09:17:38] <YuviPanda>	 legoktm: bah https://github.com/unbit/uwsgi/issues/621
[09:18:04] <legoktm>	 awwww
[09:18:11] <YuviPanda>	 legoktm: moment
[09:18:45] <YuviPanda>	 legoktm: try again?
[09:18:46] <YuviPanda>	 py3
[09:19:35] * legoktm tries
[09:22:09] <YuviPanda>	 legoktm: didd ittt work?
[09:22:18] <legoktm>	 YuviPanda: same error :(
[09:22:24] <YuviPanda>	 legoktm: hmm
[09:23:15] <YuviPanda>	 legoktm: looks like uwsgi’s python3 support is buggy :(
[09:23:53] <legoktm>	 :(
[09:24:07] <legoktm>	 back to py2 then :(
[09:24:12] <YuviPanda>	 legoktm: yeah
[09:24:14] <YuviPanda>	 legoktm: sigh
[09:29:31] <legoktm>	 YuviPanda: um, python2 is broken now
[09:29:39] <YuviPanda>	 legoktm: oh?
[09:29:41] <YuviPanda>	 legoktm: what error?
[09:30:35] <grrrit-wm>	 (03PS1) 10Yuvipanda: Revert "toollabs: Add python3 support to uwsgi web services" [puppet] - 10https://gerrit.wikimedia.org/r/187896 
[09:30:37] <YuviPanda>	 legoktm: ^ reverted
[09:31:08] <legoktm>	 open("/usr/lib/uwsgi/plugins/python2_plugin.so"): No such file or directory [core/utils.c line 3436]
[09:31:08] <legoktm>	 !!! UNABLE to load uWSGI plugin: /usr/lib/uwsgi/plugins/python2_plugin.so: cannot open shared object file: No such file or directory !!!
[09:31:08] <legoktm>	 /usr/bin/uwsgi: unrecognized option '--wsgi-file'
[09:31:11] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032 V: 032] Revert "toollabs: Add python3 support to uwsgi web services" [puppet] - 10https://gerrit.wikimedia.org/r/187896 (owner: 10Yuvipanda)
[09:33:32] <YuviPanda>	 legoktm: should be back now
[09:33:33] <YuviPanda>	 try?
[09:34:44] <legoktm>	 YuviPanda: working! :D
[09:34:54] <YuviPanda>	 legoktm: cool, but also sad :(
[09:35:05] <legoktm>	 yeah :(
[09:35:57] <YuviPanda>	 legoktm: so it can still run, but would be slightly more complicated, I guess.
[09:36:01] <YuviPanda>	 would have to be a module rather than a file
[09:39:45] <legoktm>	 it's not super important
[09:39:53] <legoktm>	 it's pretty easy to write py2+py3 compat code
[09:40:26] <YuviPanda>	 legoktm: right
[09:40:40] <YuviPanda>	 legoktm: but having this for py3 would let us write py3 only code :)
[09:40:48] <legoktm>	 :D
[09:41:07] <legoktm>	 like return type hints!
[09:41:29] <YuviPanda>	 isn’t that like much later on?
[09:41:30] <YuviPanda>	 not 3.4?
[09:42:16] <legoktm>	 function return type hints have been included since 3.4.0 I think.
[09:42:18] <legoktm>	 er
[09:42:20] <legoktm>	 3.0*
[09:42:29] <YuviPanda>	 oh
[09:42:30] <YuviPanda>	 interesting
[09:42:32] <YuviPanda>	 haven’t seen any
[09:42:55] <legoktm>	 I occassionally use them
[09:43:33] <legoktm>	 https://github.com/legoktm/composer-proxy/blob/master/app.py#L20
[09:43:40] <legoktm>	 totally broke github's syntax highlighting :P
[09:44:22] <YuviPanda>	 hehe
[09:45:12] <icinga-wm>	 PROBLEM - check_mysql on db1008 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 898  
[09:50:12] <icinga-wm>	 RECOVERY - check_mysql on db1008 is OK: Uptime: 313941  Threads: 2  Questions: 569718  Slow queries: 2148  Opens: 7444  Flush tables: 2  Open tables: 64  Queries per second avg: 1.814 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0  
[10:05:31] <_joe_>	 YuviPanda: we're at a talk about apache mesos
[10:05:40] <YuviPanda>	 _joe_: ooooo
[10:05:46] <YuviPanda>	 _joe_: do tell me what your opinions on that are
[10:05:56] <YuviPanda>	 is it still an incubator?
[10:06:47] <YuviPanda>	 hmm, Java
[10:07:10] <_joe_>	 yeah, that
[10:07:18] <_joe_>	 but gridengine is what?
[10:12:08] <paravoid>	  I think the core is in C++, not Java
[10:14:04] <YuviPanda>	 yeah, I think so. Remember reading through some source code at some point
[10:14:23] <YuviPanda>	 yup, C++
[10:16:46] <YuviPanda>	 this http://mesos.apache.org/assets/img/documentation/architecture3.jpg does not make me feel very happy
[10:19:20] <_joe_>	 YuviPanda: why?
[10:19:33] <_joe_>	 it's pretty complicated in fact
[10:19:40] <YuviPanda>	 it’s a lot of Java.
[10:19:49] <_joe_>	 yes
[10:19:51] <YuviPanda>	 and they describe that as *thin*
[10:20:09] <_joe_>	 well, everyone's full of shit. We already know that
[10:20:13] <YuviPanda>	 :D
[10:20:31] <YuviPanda>	 I would like to play with Docker / LXC-based-things at some point, though
[10:20:33] <_joe_>	 it's thought as a solution for large clusters
[10:20:44] <YuviPanda>	 yeah
[10:20:50] <YuviPanda>	 toollabs isn’t a large cluster, I think.
[10:20:52] <_joe_>	 me too btw, we will
[10:23:18] <paravoid>	 I love how he's mentioned "thousands of servers" a bunch of times
[10:23:24] <paravoid>	 like who in this room has thousands of servers
[10:23:26] <_joe_>	 yes
[10:24:25] <_joe_>	 paravoid: mesos makes sense only if you have a sizable number of services to run
[10:24:29] <YuviPanda>	 how many servers do we have anyway?
[10:24:40] <_joe_>	 also, most people nowadays use clouds
[10:25:01] <_joe_>	 YuviPanda: 1K? something more now with dallas?
[10:25:03] <YuviPanda>	 784?
[10:25:09] <YuviPanda>	 according to https://servermon.wikimedia.org/
[10:25:15] <YuviPanda>	 that’s lesser than I imagined. not bad
[10:25:34] <paravoid>	 we were > 1k when we had tampa
[10:25:40] <paravoid>	 we'll get there again I think
[10:25:43] <_joe_>	 ok
[10:25:48] <YuviPanda>	 right
[10:25:48] <paravoid>	 but right in that ballpark, not too much higher
[10:25:49] <YuviPanda>	 with dallas
[10:26:03] <_joe_>	 I looked quite some time ago
[10:26:14] <_joe_>	 YuviPanda: don't trust servermon, I know the developer
[10:26:25] <YuviPanda>	 hahaha :D
[10:29:23] <_joe_>	 "ruby sucks, let's move to the JVM"
[10:29:34] <_joe_>	 twitter has done a lot of bad to the tech world
[10:29:53] <YuviPanda>	 :D
[10:30:05] <YuviPanda>	 idk, I’d rather the Ruby devs move to JVM than try to write C
[10:30:07] <_joe_>	 (the speaker is from twitter)
[10:37:07] <grrrit-wm>	 (03CR) 10JanZerebecki: [C: 031] add IPv6 interface to dataset1001 (eth2) [puppet] - 10https://gerrit.wikimedia.org/r/187121 (https://phabricator.wikimedia.org/T68996) (owner: 10Dzahn)
[12:31:59] <YuviPanda>	 chasemp: hey! Are you tracking the ‘external monitoring / uptime metrics for cluster’ somewhere?
[12:32:24] <YuviPanda>	 chasemp: doing exactly the same for beta is one of releng’s goals, so I think the metrics should pretty much be the same instead of being duplicated
[13:16:19] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: "Just making sure, does Kartik's "do not merge right now" still holds true or not ?" [puppet] - 10https://gerrit.wikimedia.org/r/186522 (https://phabricator.wikimedia.org/T76674) (owner: 10KartikMistry)
[13:21:33] <grrrit-wm>	 (03PS1) 10QChris: Install MaxMind DBs on Hadoop worker nodes [puppet] - 10https://gerrit.wikimedia.org/r/187901 
[14:07:20] <grrrit-wm>	 (03PS1) 1001tonythomas: Set $domain part override for emails from beta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/187905 (https://phabricator.wikimedia.org/T88204) 
[14:18:53] <grrrit-wm>	 (03CR) 10Hoo man: [C: 032] "Beta only change" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/187905 (https://phabricator.wikimedia.org/T88204) (owner: 1001tonythomas)
[14:19:21] <grrrit-wm>	 (03Merged) 10jenkins-bot: Set $domain part override for emails from beta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/187905 (https://phabricator.wikimedia.org/T88204) (owner: 1001tonythomas)
[14:20:01] <logmsgbot>	 !log hoo Synchronized wmf-config/CommonSettings-labs.php: (no message) (duration: 00m 06s)
[14:20:06] <morebots>	 Logged the message, Master
[14:37:02] <icinga-wm>	 PROBLEM - graphite.wikimedia.org on tungsten is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 525 bytes in 0.045 second response time  
[14:41:21] <icinga-wm>	 RECOVERY - graphite.wikimedia.org on tungsten is OK: HTTP OK: HTTP/1.1 200 OK - 1607 bytes in 0.025 second response time  
[14:49:39] <tonythomas>	 hey YuviPanda : can you take a look at https://phabricator.wikimedia.org/T88215 ?
[14:50:11] <tonythomas>	 on adding MX records for beta ? If you have an example, I can create the patch though 
[14:57:06] <grrrit-wm>	 (03PS2) 1001tonythomas: Made the deployment-mx talk back to deployment wiki on receiving a bounce [puppet] - 10https://gerrit.wikimedia.org/r/186938 
[15:08:01] <wikibugs>	 3ops-eqiad, operations: eqiad housekeeping task -storage room - https://phabricator.wikimedia.org/T86344#1006275 (10Cmjohnson) p:5Triage>3Low
[15:29:53] <icinga-wm>	 PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0]  
[15:37:21] <icinga-wm>	 PROBLEM - puppet last run on cp4002 is CRITICAL: CRITICAL: Puppet has 1 failures  
[15:44:51] <icinga-wm>	 RECOVERY - puppet last run on cp4002 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures  
[15:49:06] <grrrit-wm>	 (03CR) 10KartikMistry: [C: 031] "Go ahead and merge :)" [puppet] - 10https://gerrit.wikimedia.org/r/186522 (https://phabricator.wikimedia.org/T76674) (owner: 10KartikMistry)
[16:06:52] <icinga-wm>	 PROBLEM - puppet last run on cp3014 is CRITICAL: CRITICAL: puppet fail  
[16:11:52] <icinga-wm>	 RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0]  
[16:14:37] <grrrit-wm>	 (03PS1) 10Glaisher: Throttle rule for workshop in Municipal Library of Prague [mediawiki-config] - 10https://gerrit.wikimedia.org/r/187912 (https://phabricator.wikimedia.org/T88203) 
[16:17:21] <grrrit-wm>	 (03PS1) 10Glaisher: Enable Quiz extension at cawikibooks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/187913 (https://phabricator.wikimedia.org/T88208) 
[16:19:42] <grrrit-wm>	 (03PS1) 10Glaisher: Add www.doria.fi to $wgCopyUploadsDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/187914 (https://phabricator.wikimedia.org/T87104) 
[16:24:31] <icinga-wm>	 PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0]  
[16:24:45] <grrrit-wm>	 (03PS1) 10Glaisher: Create 'interface-editor' user group on cawikibooks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/187915 (https://phabricator.wikimedia.org/T85713) 
[16:26:42] <icinga-wm>	 RECOVERY - puppet last run on cp3014 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures  
[16:31:26] <grrrit-wm>	 (03CR) 10Dan-nl: [C: 031] Add www.doria.fi to $wgCopyUploadsDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/187914 (https://phabricator.wikimedia.org/T87104) (owner: 10Glaisher)
[16:35:46] <wikibugs>	 3Multimedia, operations, Wikimedia-Site-requests, MediaWiki-extensions-GWToolset: Updating broken 'request' link at GWToolset - https://phabricator.wikimedia.org/T75732#1006317 (10Glaisher) 5Resolved>3Open #operations project is wrong. Please use #wikimedia-site-requests instead. Also "Whitelist a Domain" fo...
[16:42:13] <icinga-wm>	 RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0]  
[16:47:32] <icinga-wm>	 PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0]  
[17:01:11] <icinga-wm>	 RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0]  
[17:04:12] <icinga-wm>	 PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 7.14% of data above the critical threshold [500.0]  
[17:16:41] <icinga-wm>	 RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0]  
[18:06:47] <wikibugs>	 3operations: dysprosium failed idrac - https://phabricator.wikimedia.org/T88129#1006382 (10Krenair)
[18:14:52] <Qcoder00>	 Hi
[18:15:00] <Qcoder00>	 Is Eruope a bit sllow right now?
[18:15:12] <Qcoder00>	 (Owing to the server that went down the other night?)
[18:25:22] <Krenair>	 seems fine to me, Qcoder00 
[18:25:32] <Qcoder00>	 Must be a local issye
[18:25:42] <Qcoder00>	 did have some icy rain earlier
[18:25:45] <Qcoder00>	 :(
[18:26:26] <Krenair>	 Can ping text-lb.eqiad.wikimedia.org at ~80ms from a Paris-based server
[18:27:50] <Krenair>	 tried from a server in wikimedia's amsterdam location as well, ~90ms
[18:31:00] <wikibugs>	 3operations: Cannot use dsh-based restart of parsoid from tin anymore - https://phabricator.wikimedia.org/T87803#1006387 (10GWicke) @yuvipanda, try without root.
[23:30:11] <NotASpy>	 who would like to re-attribute some edits for me to remove a series of significant copyright violations ?
[23:31:18] <hoo>	 NotASpy: Better open a task in phabricator for this, hard to find someone during the weekend
[23:31:20] <hoo>	 feel free to CC me
[23:31:33] <NotASpy>	 needing everything from 9 October 2006 on this list of contributions moved from User:Nick to User:Nick@de.wp https://de.wikipedia.org/w/index.php?title=Spezial:Beitr%C3%A4ge/Nick&offset=20061213221108&target=Nick
[23:31:39] <NotASpy>	 can that be done with a phab task ?
[23:33:50] <hoo>	 NotASpy: If at all, yes... just poking people on IRC for that is way less likely to work
[23:34:29] <NotASpy>	 I've been trying to get it done for years and the usual promise is SUL will fix it. 
[23:35:05] <hoo>	 SUL will not fix old records/ history entries
[23:35:13] <hoo>	 just we wont run into any *new* issues
[23:35:20] <NotASpy>	 yeap, those contributions will move when User:Nick on the de.wp is forcibly renamed
[23:35:57] <hoo>	 but only on dewiki
[23:36:04] <hoo>	 the enwiki records wont update
[23:36:15] <NotASpy>	 yeah