[00:08:02] PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: There are 2 unmerged changes in puppet (dir /var/lib/git/operations/puppet). [00:09:32] RECOVERY - puppet last run on lvs3004 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [00:55:22] RECOVERY - Unmerged changes on repository puppet on strontium is OK: No changes to merge. [00:57:58] !log dump db2017 to dbstore2001 [01:00:54] Logged the message, Master [01:07:53] mutante: https://gerrit.wikimedia.org/r/#/c/96424/ Achievement unlocked! [01:09:25] !log dump db2018 to dbstore2001 [01:09:35] Logged the message, Master [01:11:45] !log dump db2019 to dbstore2001 [01:11:52] Logged the message, Master [01:12:00] !log applied base::firewall to all bastions hosts. This is bast*, iron, hooft [01:12:04] Logged the message, Master [01:25:16] !log ongoing schema changes T86415 externallinks [01:25:24] Logged the message, Master [02:11:03] !log l10nupdate Synchronized php-1.25wmf14/cache/l10n: (no message) (duration: 00m 01s) [02:11:09] !log LocalisationUpdate completed (1.25wmf14) at 2015-01-22 02:11:09+00:00 [02:11:18] Logged the message, Master [02:11:24] Logged the message, Master [02:19:15] !log l10nupdate Synchronized php-1.25wmf15/cache/l10n: (no message) (duration: 00m 04s) [02:19:19] !log LocalisationUpdate completed (1.25wmf15) at 2015-01-22 02:19:19+00:00 [02:19:24] Logged the message, Master [02:19:32] Logged the message, Master [03:41:22] PROBLEM - puppet last run on cp4016 is CRITICAL: CRITICAL: Puppet has 1 failures [03:57:52] RECOVERY - puppet last run on cp4016 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [04:15:02] !log LocalisationUpdate ResourceLoader cache refresh completed at Thu Jan 22 04:15:02 UTC 2015 (duration 15m 1s) [04:15:11] Logged the message, Master [04:58:42] 3operations: Re-image osmium and repurpose into VE testing host - https://phabricator.wikimedia.org/T87215#989575 (10yuvipanda) This should technically go through procurement, which we decided to move from RT to Phab but haven't set up a process for it yet. Poking @robh to figure out how to do this. Still needs... [04:59:16] 3operations: Re-image osmium and repurpose into VE testing host - https://phabricator.wikimedia.org/T87215#989577 (10yuvipanda) p:5Triage>3Normal [05:23:55] 3operations: Re-image osmium and repurpose into VE testing host - https://phabricator.wikimedia.org/T87215#989587 (10RobH) a:3RobH Well, since I have to detail and work out that work flow post all-staff, I imagine this is going to sit until then. We're in the process of re-tooling how to request servers, and... [05:24:26] 3operations: Re-image osmium and repurpose into VE testing host - https://phabricator.wikimedia.org/T87215#989589 (10RobH) If you guys need this before the end of the all staff, we can break procedure and just get it approved, but I prefer to do it properly. (Please advise.) [05:25:01] PROBLEM - git.wikimedia.org on antimony is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:51:38] Any jobrunner experts around? There is a bad LocalRenameUserJob for vewikimedia stuck somehow and causing literally thousands of exceptions to be logged per second [05:52:24] mwscript showJobs.php --wiki=vewikimedia --group doesn't show it but tailing execption.log certainly does [05:57:52] RECOVERY - git.wikimedia.org on antimony is OK: HTTP OK: HTTP/1.1 200 OK - 59701 bytes in 0.069 second response time [06:28:43] PROBLEM - puppet last run on cp3016 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:52] PROBLEM - puppet last run on db1046 is CRITICAL: CRITICAL: Puppet has 2 failures [06:30:11] PROBLEM - puppet last run on cp1056 is CRITICAL: CRITICAL: Puppet has 2 failures [06:30:31] PROBLEM - puppet last run on ms-fe2003 is CRITICAL: CRITICAL: Puppet has 2 failures [06:31:52] PROBLEM - puppet last run on mw1144 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:22] PROBLEM - puppet last run on cp4008 is CRITICAL: CRITICAL: Puppet has 1 failures [06:45:21] RECOVERY - puppet last run on cp3016 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [06:45:31] RECOVERY - puppet last run on cp1056 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [06:47:42] RECOVERY - puppet last run on cp4008 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:50:30] Hi, I lost my two-factor authentication tokens to Phabricator a while back. I asked about this on IRC a few weeks ago, and quiddity said he emailed two admins, but I haven't heard from them since. [06:56:40] hi Zhaofeng_Li. all sensible people who could help you are probably sleeping or partying right now, i'd try emailing quiddity if i were you. :) [07:05:21] RECOVERY - puppet last run on db1046 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [07:06:02] RECOVERY - puppet last run on ms-fe2003 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [07:06:12] RECOVERY - puppet last run on mw1144 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [11:22:42] PROBLEM - HTTP 5xx req/min on tungsten is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0] [11:34:32] RECOVERY - HTTP 5xx req/min on tungsten is OK: OK: Less than 1.00% above the threshold [250.0] [12:40:22] PROBLEM - Slow CirrusSearch query rate on fluorine is CRITICAL: CirrusSearch-slow.log_line_rate CRITICAL: 0.00332225913621 [12:45:31] RECOVERY - Slow CirrusSearch query rate on fluorine is OK: CirrusSearch-slow.log_line_rate OKAY: 0.0 [12:58:09] !log Run https://phabricator.wikimedia.org/T87040#984282 again to stop GWToolset extension flooding commons log [12:58:19] Logged the message, Master [13:10:28] (03PS1) 10Calak: Allow sysops/botadmins to grant/revoke 'confirmed' user group on fawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186230 (https://phabricator.wikimedia.org/T87348) [14:34:51] (03PS1) 10QChris: Alert analytics on obvious EventLogging validation issues [puppet] - 10https://gerrit.wikimedia.org/r/186233 [15:21:50] (03PS1) 10Glaisher: Add 'move-categorypages' right to ruwiki bots [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186236 (https://phabricator.wikimedia.org/T87230) [15:52:12] (03CR) 10Glaisher: "Note that wwwportals adapt according to the screen resolution so a mobile friendly portal is shown instead of the regular desktop version " [puppet] - 10https://gerrit.wikimedia.org/r/185461 (https://phabricator.wikimedia.org/T78421) (owner: 10Glaisher) [16:00:04] Dear anthropoid, the time has come. Please deploy WMF All Hands (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20150122T1600). [16:03:17] (03PS1) 1001tonythomas: Added BounceHandler extension to group0 wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186242 [16:07:25] (03CR) 10Hoo man: [C: 04-1] Added BounceHandler extension to group0 wikis (033 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186242 (owner: 1001tonythomas) [16:07:27] (03CR) 10Glaisher: [C: 04-1] Added BounceHandler extension to group0 wikis (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186242 (owner: 1001tonythomas) [16:09:35] (03CR) 10Hoo man: "Also you'd need to add group0 to the group loading stuffs in CommonSettings (around line 200)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186242 (owner: 1001tonythomas) [16:19:17] (03PS2) 1001tonythomas: Added BounceHandler extension to group0 wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186242 [16:31:39] 3Project-Creators, operations: Create #site-incident tag and use it for incident reports - https://phabricator.wikimedia.org/T85889#990027 (10greg) >>! In T85889#987106, @GWicke wrote: > @aklapper, we did bring it up & heard some positive noises, but I think a final decision was not yet made. @greg should know m... [16:47:14] (03CR) 10Glaisher: "Huh. It looks like http://wikibook.org/ is not hosted by WMF but the config has been there all this time for some reason. Should I remove " [puppet] - 10https://gerrit.wikimedia.org/r/185474 (https://phabricator.wikimedia.org/T87039) (owner: 10Glaisher) [16:53:15] 3operations, Wikimedia-DNS: wikibook.org should be hosted by WMF and should redirect to wikibooks.org - https://phabricator.wikimedia.org/T87375#990047 (10Glaisher) [16:53:40] (03CR) 10Glaisher: "T87375" [puppet] - 10https://gerrit.wikimedia.org/r/185474 (https://phabricator.wikimedia.org/T87039) (owner: 10Glaisher) [18:07:57] !log Deployed patch for T87304 [18:08:13] Logged the message, Master [18:39:09] !log ran 'hdel jobqueue:aggregator:h-ready-queues:v2 LocalRenameUserJob/vewikimedia' on rdb1001 [18:39:19] Logged the message, Master [18:54:07] (03PS5) 10QChris: Mail webrequest partition status summaries to analytics ops [puppet] - 10https://gerrit.wikimedia.org/r/185442 [18:54:09] (03PS1) 10QChris: For stats user's cron jobs on stat1002, make empty MAILTO explicit [puppet] - 10https://gerrit.wikimedia.org/r/186253 [18:54:11] (03PS1) 10QChris: Grant stats user access to private analytics data [puppet] - 10https://gerrit.wikimedia.org/r/186254 [18:54:48] (03CR) 10QChris: Grant stats user access to private analytics data (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/186254 (owner: 10QChris) [18:55:03] (03CR) 10jenkins-bot: [V: 04-1] Mail webrequest partition status summaries to analytics ops [puppet] - 10https://gerrit.wikimedia.org/r/185442 (owner: 10QChris) [18:55:21] (03CR) 10jenkins-bot: [V: 04-1] Grant stats user access to private analytics data [puppet] - 10https://gerrit.wikimedia.org/r/186254 (owner: 10QChris) [19:02:08] (03CR) 10QChris: Grant stats user access to private analytics data (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/186254 (owner: 10QChris) [19:04:18] (03CR) 10QChris: Mail webrequest partition status summaries to analytics ops (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/185442 (owner: 10QChris) [19:05:30] (03CR) 10QChris: "The V-1 stems from the parent commit." [puppet] - 10https://gerrit.wikimedia.org/r/185442 (owner: 10QChris) [19:32:06] <_joe_> ori: oh that's the job bd808 mentioned as the reason for logstash crashes [19:32:21] <_joe_> (renameuser on vewikimedia) [19:32:29] yeah i debugged it because he asked me to [19:32:41] Is that still a thing? :/ [19:32:50] debugging? alas, yes :) [19:32:57] Nah, these jobs [19:33:09] I hoped runJobs would give up after like a few tries [19:33:51] <_joe_> hoo: are you coming to the dev summit? [19:33:59] <_joe_> I may have already asked this [19:34:03] Yep, will fly tomorrow [19:34:04] Hate that I can't bring and use my laptop at places like these [19:34:08] * YuviPanda grrrs [19:34:08] hehe, yes you did [19:34:18] (03CR) 10Chad: [C: 032] Remove "Protection for unusual entry points" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/164008 (owner: 10PleaseStand) [19:34:22] YuviPanda: ? [19:34:27] why can't you? [19:34:39] paravoid: need to use that keyboard [19:34:52] Can't type on laptop keyboard [19:34:53] oh [19:35:02] even for this short period of time? [19:35:08] hoo: long story that bd808|BUFFER / ori can explain better (re why the job wasn't just dumped after 3 tries versus the 1000/sec tries) [19:35:25] If I type 5sentences on it I have pain for 2days [19:35:33] YuviPanda: btw, you keep saying you're an introvert, but I don't think your actions confirm that :) [19:36:06] <_joe_> greg-g: let's reassess that after he's in ops for 1 year [19:36:24] Heh [19:36:37] I don't like interacting with people I don't know at all [19:36:44] <_joe_> I am trying to teach him to be anti-social and hateful [19:36:47] Smalltalk and eugh and eh [19:36:51] <_joe_> it will take time [19:37:05] _joe_: hahaha :D [19:37:07] <_joe_> but let enough shitstom hit him [19:37:14] _joe_: "you will assimilate" [19:37:22] (03CR) 10Chad: [V: 032] Remove "Protection for unusual entry points" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/164008 (owner: 10PleaseStand) [19:37:42] greg-g: you should see Jon if you think i am not an introvert :) [19:37:51] <_joe_> greg-g: http://i47.tinypic.com/15pnzib.jpg [19:37:52] !log demon Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 05s) [19:38:01] Logged the message, Master [19:38:13] <_joe_> (but he is definitely not an intern) [19:38:13] _joe_: Hehe :) [19:38:54] _joe_: I'm an intern until I break prod? [19:38:56] :) [19:39:03] YuviPanda: oh, you're not the most extrovert, just not very introverted in reality :) [19:40:16] greg-g: heh true. It is a scale I guess. But wikimedia is special - something I care about enough to not feel inhibited. [19:40:18] <_joe_> greg-g: I think there is a big cultural meaning to "introvert" [19:40:36] I hate people who go 'dude do not talk shop we are at a party' [19:40:38] <_joe_> I mean I am really introvert for being italian [19:40:41] And then I go quiet [19:41:09] <_joe_> YuviPanda: you can talk shop, just not about actual work :) [19:41:27] Potato, potato :D [19:41:31] <_joe_> nah [19:42:01] <_joe_> like if I now start doing commits [19:42:12] <_joe_> which is like what I am very tempted to do [19:42:32] You should! Let's merge that realm branching patch? ;) [19:42:46] For the appservers [19:43:54] <_joe_> right [19:43:54] * ^d is merging patches [19:44:08] <_joe_> is it up for review? [19:44:11] <_joe_> let's do that [19:44:14] :D [19:44:45] _joe_: it is [19:45:27] _joe_: there is a realm branch and a feature branch [19:45:36] One for the apache difference and one for the lvs [19:45:50] _joe_: I'll fix the apache real branch later [19:46:01] Our entire apache config [19:46:03] Is meh [19:46:13] _joe_: aren't we supposed to switch to nginx? [19:47:01] Also I need to figure out how to do pools on beta. DNS maybe [19:47:32] <_joe_> ok [19:47:32] <_joe_> where it is it [19:47:35] is greg-g on IRC? [19:47:48] 3Analytics, operations, ops-core: Deprecate HTTPS udp2log stream? - https://phabricator.wikimedia.org/T86656#990203 (10mark) @TNegrin confirmed that Analytics is not using this data (anymore), and it can be turned off. (Let's turn off this data ASAP, so we still have a bit of time if someone suddenly starts scr... [19:47:56] <^d> Krenair: Well you just pinged him :p [19:48:09] yes, so we'll find out :p [19:48:25] _joe_: https://gerrit.wikimedia.org/r/#/c/185952/ [19:48:31] 3Analytics, operations, ops-core: Deprecate HTTPS udp2log stream? - https://phabricator.wikimedia.org/T86656#990205 (10mark) [19:48:45] _joe_: and https://gerrit.wikimedia.org/r/#/c/185965/ [19:53:45] <_joe_> YuviPanda: looking [19:57:50] _joe_: cool [19:58:10] <_joe_> YuviPanda: I'm going to make a little amend to the first patch [19:58:14] <_joe_> if you don't mind [19:58:21] Sure [19:58:41] _joe_: are you going to make it a feature flag branch? [19:58:51] (03CR) 10Giuseppe Lavagetto: [C: 04-1] "I like the direction you're headed to, but I'd rework this a little." (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/185952 (https://phabricator.wikimedia.org/T87210) (owner: 10Yuvipanda) [19:58:58] <_joe_> no [19:59:06] * YuviPanda looks [19:59:11] <_joe_> just make the default the prod behaviour [20:00:16] _joe_: hmm the mediawiki roles aren't used anywhere outside beta [20:00:27] Other MW installs use labs vagrant [20:00:45] _joe_: so what do you mean by 'default' [20:00:58] Branch on instance project? [20:01:38] <_joe_> can I show you? [20:01:40] <_joe_> :) [20:01:50] <_joe_> if you come sit here, it's actually easier :P [20:01:56] _joe_: haha definitely [20:49:44] (03PS1) 10BBlack: disable nginx udplog for SSL clusters [puppet] - 10https://gerrit.wikimedia.org/r/186257 [20:49:46] (03PS1) 10BBlack: use /run/ directly for pidfile under systemd [puppet] - 10https://gerrit.wikimedia.org/r/186258 [20:51:06] (03PS2) 10BBlack: disable nginx udplog for SSL clusters [puppet] - 10https://gerrit.wikimedia.org/r/186257 [20:52:29] (03CR) 10Dzahn: [C: 031] "per "@Tnegrin confirmed that Analytics is not using this data (anymore), and it can be turned off."" [puppet] - 10https://gerrit.wikimedia.org/r/186257 (owner: 10BBlack) [20:56:31] PROBLEM - HTTP error ratio anomaly detection on tungsten is CRITICAL: CRITICAL: Anomaly detected: 10 data above and 0 below the confidence bounds [21:09:49] (03PS3) 10Giuseppe Lavagetto: mediawiki: Load prod/beta apache configurabion based on realm [puppet] - 10https://gerrit.wikimedia.org/r/185952 (https://phabricator.wikimedia.org/T87210) (owner: 10Yuvipanda) [21:10:14] (03PS4) 10Giuseppe Lavagetto: mediawiki: Load prod/beta apache configurabion based on realm [puppet] - 10https://gerrit.wikimedia.org/r/185952 (https://phabricator.wikimedia.org/T87210) (owner: 10Yuvipanda) [21:10:49] (03CR) 10Yuvipanda: [C: 031] mediawiki: Load prod/beta apache configurabion based on realm [puppet] - 10https://gerrit.wikimedia.org/r/185952 (https://phabricator.wikimedia.org/T87210) (owner: 10Yuvipanda) [21:13:56] (03CR) 10Giuseppe Lavagetto: [C: 032] mediawiki: Load prod/beta apache configurabion based on realm [puppet] - 10https://gerrit.wikimedia.org/r/185952 (https://phabricator.wikimedia.org/T87210) (owner: 10Yuvipanda) [21:20:42] (03CR) 10Faidon Liambotis: "Why is this needed? Isn't /var/run a symlink to /run?" [puppet] - 10https://gerrit.wikimedia.org/r/186258 (owner: 10BBlack) [21:23:04] _joe_: there is one more patch :D [21:25:12] <_joe_> I know [21:26:10] (03PS2) 10Giuseppe Lavagetto: mediawiki: Introduce feature flag to enable/disable lvs [puppet] - 10https://gerrit.wikimedia.org/r/185965 (https://phabricator.wikimedia.org/T87210) (owner: 10Yuvipanda) [21:26:49] _joe_: ty [21:34:19] (03CR) 10Giuseppe Lavagetto: [C: 04-1] "Change is good, but "has_lvs" is a bit too generic as a name for the feature branch right now." (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/185965 (https://phabricator.wikimedia.org/T87210) (owner: 10Yuvipanda) [21:35:40] _joe_: hmm I made it generic because I wasn't sure if that was the only lvs related branch we would have [21:39:17] 3operations: Remove the auto +v for users joining -operations from WMF IP space - https://phabricator.wikimedia.org/T87270#990277 (10RobH) 5Open>3Resolved a:3RobH I just ran: flags #wikimedia-operations *!*@*.wikimedia.org -V So now there isn't auto +V for wikimedia.org. I agree the IRC best practices co... [21:39:21] <_joe_> YuviPanda: no, but I'm not sure we want to tie them together [21:39:30] <_joe_> lemme think about it for a moment [21:39:41] _joe_: ok [21:39:51] <_joe_> because it instinctively seemed like it was wrong [21:39:57] <_joe_> but maybe it's not [21:40:10] We do that with has_ganglia. [21:40:17] I'm not sure where else we use lvs tho [21:40:22] Ganglia was everywhere [21:40:30] <_joe_> yeah [21:40:40] <_joe_> but that seemed more simple to isolate [21:40:42] So maybe I should look at where else lvs is used [21:40:45] Before doing [21:40:53] This as a global flag vs a role param [21:40:57] <_joe_> and yes, it should be one single variable [21:41:01] <_joe_> yes [21:41:15] <_joe_> but well, is_lvs_realserver is better [21:41:30] as a role param? [21:43:34] I've been wondering if the patch for https://phabricator.wikimedia.org/T87040 should be deployed (so that next time someone uses GWToolset on commons it doesn't get stuck in an indefinite job/log spam loop) but greg has not been available on IRC... [21:56:40] (03CR) 10Giuseppe Lavagetto: [C: 032] "Actually changed my mind. This is in fact a global feature flag like has_ganglia" [puppet] - 10https://gerrit.wikimedia.org/r/185965 (https://phabricator.wikimedia.org/T87210) (owner: 10Yuvipanda) [21:57:20] _joe_: wheee [21:57:27] I'll re image the beta stuff tomorrow [21:58:10] _joe_: well once we get the new labs images at least [22:01:29] 3operations: Rolling restart for Elasticsearch to pick up new version of wikimedia-extra plugin - https://phabricator.wikimedia.org/T86602#990314 (10RobH) a:3greg I'm assigning this task to Greg, once he supplies answers, if the next person is unclear, he can kick it back to unassigned. (I don't want to leave... [22:03:15] 3operations: Decomission svn.wikimedia.org - https://phabricator.wikimedia.org/T86655#990319 (10RobH) Chad: If it is imported to Phab, will it cause an issue requiring odd ports or will it simply present on web ports? [22:06:34] 3operations: Decomission svn.wikimedia.org - https://phabricator.wikimedia.org/T86655#990320 (10Chad) >>! In T86655#990319, @RobH wrote: > Chad: If it is imported to Phab, will it cause an issue requiring odd ports or will it simply present on web ports? It should just be the web ports, nothing special. We won'... [22:08:05] 3ops-core: backup old blog server/holmium with bacula - server will be wiped post backup - https://phabricator.wikimedia.org/T86975#990322 (10RobH) It seems that the db1001 is backed up, but possibly not the blog database. Would you be the person to check that Alex or should I speak with Sean? (Thanks for back... [22:13:12] PROBLEM - Host holmium is DOWN: CRITICAL - Host Unreachable (208.80.154.49) [22:13:20] 3operations, ops-eqiad: wipe holmium disks - https://phabricator.wikimedia.org/T87391#990328 (10RobH) 3NEW a:3Cmjohnson [22:19:56] 3operations: Decomission svn.wikimedia.org - https://phabricator.wikimedia.org/T86655#990340 (10Dzahn) importing into phab would be a great solution as well. then we should also get rid of viewvc completely and can delete the entire puppet role [22:33:51] 3ops-core: backup old blog server/holmium with bacula - server will be wiped post backup - https://phabricator.wikimedia.org/T86975#990367 (10akosiaris) 5Open>3Resolved Actually db1001 is not backed up right now, but it can be. I will talk with @springle about doing it at some point but this should not block... [22:36:42] RECOVERY - HTTP error ratio anomaly detection on tungsten is OK: OK: No anomaly detected [23:11:17] 3operations: mysql boxes not in ganglia - https://phabricator.wikimedia.org/T87209#990415 (10Reedy) 5Resolved>3Open @springle It seems that only the mysql masters are showing any sort of mysql stats See s4 master vs s4 slave https://ganglia.wikimedia.org/latest/?c=MySQL%20eqiad&h=db1040.eqiad.wmnet&m=cpu_r... [23:27:46] !log restarted the job runner service [23:27:50] Logged the message, Master [23:30:22] PROBLEM - Redis on rdb1001 is CRITICAL: Connection timed out [23:32:32] RECOVERY - Redis on rdb1001 is OK: TCP OK - 0.010 second response time on port 6379 [23:32:54] 3MediaWiki-extensions-Flow, operations: Non-Flow pages can be moved to the Topic namespace, and cannot be moved back - https://phabricator.wikimedia.org/T87396#990430 (10MrStradivarius) 3NEW [23:42:21] 3MediaWiki-extensions-Flow, operations: Non-Flow pages can be moved to the Topic namespace, and cannot be moved back - https://phabricator.wikimedia.org/T87396#990451 (10MrStradivarius) [23:43:03] (03PS1) 10Mjbmr: Add extra namespaces for fawikibooks bug T76663 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/186317