[00:22:30] Every time I leave SSH to fluorine open in the background, when I come back it's locked up [00:23:42] Krenair: I had no problem when I left it on tail -f, though it wasn't that long... [00:23:51] Maybe some timeout on the ssh server...? [00:56:02] (03Abandoned) 10Tim Landscheidt: Tools: Remove last references to pmtpa [puppet] - 10https://gerrit.wikimedia.org/r/138480 (owner: 10Tim Landscheidt) [02:02:09] !log l10nupdate Synchronized php-1.25wmf16/cache/l10n: (no message) (duration: 00m 02s) [02:02:18] Logged the message, Master [02:03:16] !log LocalisationUpdate completed (1.25wmf16) at 2015-02-15 02:02:13+00:00 [02:03:21] Logged the message, Master [02:03:41] !log l10nupdate Synchronized php-1.25wmf17/cache/l10n: (no message) (duration: 00m 01s) [02:03:44] Logged the message, Master [02:04:48] !log LocalisationUpdate completed (1.25wmf17) at 2015-02-15 02:03:44+00:00 [02:04:51] Logged the message, Master [02:13:57] !log LocalisationUpdate ResourceLoader cache refresh completed at Sun Feb 15 02:12:53 UTC 2015 (duration 12m 52s) [02:14:04] Logged the message, Master [03:41:00] PROBLEM - puppet last run on amssq33 is CRITICAL: CRITICAL: Puppet has 1 failures [03:52:51] PROBLEM - puppet last run on cp1048 is CRITICAL: CRITICAL: Puppet has 1 failures [03:54:47] (03CR) 10AndyRussG: [C: 04-1] "This change is no longer appropriate, because in the end we deployed a mixed system in which Special:RecordImpression is sampled on the cl" [puppet] - 10https://gerrit.wikimedia.org/r/188395 (https://phabricator.wikimedia.org/T45250) (owner: 10Ejegg) [03:55:41] RECOVERY - puppet last run on amssq33 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [04:09:50] RECOVERY - puppet last run on cp1048 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [04:41:55] (03CR) 10GWicke: [C: 031] "Thanks, Ori." [puppet] - 10https://gerrit.wikimedia.org/r/190688 (owner: 10Ori.livneh) [06:27:41] PROBLEM - puppet last run on cp3016 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:40] PROBLEM - puppet last run on mw1092 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:41] PROBLEM - puppet last run on db1018 is CRITICAL: CRITICAL: Puppet has 1 failures [06:28:51] PROBLEM - puppet last run on elastic1027 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:11] PROBLEM - puppet last run on db1002 is CRITICAL: CRITICAL: Puppet has 2 failures [06:34:01] PROBLEM - HTTP 5xx req/min on graphite1001 is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0] [06:45:40] RECOVERY - puppet last run on db1018 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [06:45:50] RECOVERY - puppet last run on cp3016 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [06:45:50] RECOVERY - puppet last run on elastic1027 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [06:45:51] RECOVERY - HTTP 5xx req/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] [06:46:10] RECOVERY - puppet last run on db1002 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [06:46:40] RECOVERY - puppet last run on mw1092 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [08:11:22] 3operations: Enable TRIM for SSDs - https://phabricator.wikimedia.org/T89584#1039857 (10GWicke) 3NEW [08:14:04] is there a ticket for enabling 2FA/MFA/OATH on WM wikis, somewhere? Couldn't find it after a quick phab search. [08:15:03] 3operations: Enable TRIM for SSDs - https://phabricator.wikimedia.org/T89584#1039864 (10GWicke) [09:50:11] PROBLEM - check_mysql on db1008 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 624 [09:55:11] RECOVERY - check_mysql on db1008 is OK: Uptime: 485941 Threads: 95 Questions: 7199249 Slow queries: 3808 Opens: 8041 Flush tables: 2 Open tables: 64 Queries per second avg: 14.815 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 [16:14:40] PROBLEM - puppet last run on db1063 is CRITICAL: CRITICAL: Puppet has 1 failures [16:32:41] RECOVERY - puppet last run on db1063 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [17:40:04] 3operations: Enable TRIM for SSDs - https://phabricator.wikimedia.org/T89584#1040222 (10faidon) Which hosts are you talking about specifically? SSDs that participate in HW RAID (even single-disk RAID0 hacks on H710s) do not passthrough TRIM to the underlying device last time I checked. [17:44:07] (03PS1) 10Gerardduenas: Create 'autopatrolled' user group on maiwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/190721 (https://phabricator.wikimedia.org/T89346) [17:45:53] mai mai mai [17:46:31] (03PS2) 10Gerardduenas: Create 'autopatrolled' user group on maiwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/190721 (https://phabricator.wikimedia.org/T89346) [17:48:06] (03PS1) 10Faidon Liambotis: protoproxy: mute nginx reload cron's output [puppet] - 10https://gerrit.wikimedia.org/r/190722 [17:48:28] (03CR) 10Faidon Liambotis: [C: 032] protoproxy: mute nginx reload cron's output [puppet] - 10https://gerrit.wikimedia.org/r/190722 (owner: 10Faidon Liambotis) [18:11:54] 3operations, Wikimedia-General-or-Unknown: DMARC: Users cannot send emails via a wiki's [[Special:EmailUser]] - https://phabricator.wikimedia.org/T66795#1040246 (10Krenair) [18:45:20] PROBLEM - Slow CirrusSearch query rate on fluorine is CRITICAL: CirrusSearch-slow.log_line_rate CRITICAL: 0.00333333333333 [18:55:21] RECOVERY - Slow CirrusSearch query rate on fluorine is OK: CirrusSearch-slow.log_line_rate OKAY: 0.0 [19:58:51] PROBLEM - Varnishkafka Delivery Errors per minute on cp3022 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [20000.0] [20:05:20] RECOVERY - Varnishkafka Delivery Errors per minute on cp3022 is OK: OK: Less than 1.00% above the threshold [0.0] [22:22:53] (03PS1) 10Gerardduenas: Enable Extension:UploadWizard on idwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/190744 (https://phabricator.wikimedia.org/T88918) [22:58:04] O_o