[00:59:42] PROBLEM - mobileapps endpoints health on scb1002 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [01:02:11] RECOVERY - mobileapps endpoints health on scb1002 is OK: All endpoints are healthy [01:30:55] 06Operations, 10Traffic, 07HTTPS, 07Tracking: SSL related (tracking) - https://phabricator.wikimedia.org/T29946#2445045 (10Danny_B) [02:20:12] !log mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.9) (duration: 08m 34s) [02:20:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:25:55] !log l10nupdate@tin ResourceLoader cache refresh completed at Sun Jul 10 02:25:55 UTC 2016 (duration 5m 43s) [02:26:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [03:22:49] PROBLEM - puppet last run on mw2182 is CRITICAL: CRITICAL: Puppet has 1 failures [03:36:19] PROBLEM - puppet last run on mw1212 is CRITICAL: CRITICAL: Puppet has 2 failures [03:37:09] PROBLEM - puppet last run on mw1248 is CRITICAL: CRITICAL: Puppet has 1 failures [03:43:58] PROBLEM - puppet last run on cp3005 is CRITICAL: CRITICAL: puppet fail [03:49:39] RECOVERY - puppet last run on mw2182 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [03:52:19] PROBLEM - puppet last run on mw1025 is CRITICAL: CRITICAL: Puppet has 1 failures [04:02:59] RECOVERY - puppet last run on mw1212 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:03:59] RECOVERY - puppet last run on mw1248 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:10:48] RECOVERY - puppet last run on cp3005 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:16:50] RECOVERY - puppet last run on mw1025 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [06:22:06] PROBLEM - puppet last run on labvirt1010 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:56] PROBLEM - Eqiad HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [1000.0] [06:32:05] PROBLEM - puppet last run on elastic2007 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:06] PROBLEM - puppet last run on mw2208 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:35] PROBLEM - puppet last run on lvs1003 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:46] PROBLEM - Text HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [1000.0] [06:33:25] PROBLEM - puppet last run on elastic1042 is CRITICAL: CRITICAL: Puppet has 1 failures [06:33:46] PROBLEM - puppet last run on es2018 is CRITICAL: CRITICAL: puppet fail [06:36:25] RECOVERY - Eqiad HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] [06:37:16] RECOVERY - Text HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] [06:46:37] RECOVERY - puppet last run on labvirt1010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:50:16] PROBLEM - tools homepage -admin tool- on tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Not Available - 531 bytes in 0.035 second response time [06:56:35] RECOVERY - puppet last run on mw2208 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:56:55] RECOVERY - puppet last run on lvs1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:56:56] RECOVERY - tools homepage -admin tool- on tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 3670 bytes in 0.046 second response time [06:57:46] RECOVERY - puppet last run on elastic1042 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:58:16] RECOVERY - puppet last run on es2018 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:58:45] RECOVERY - puppet last run on elastic2007 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:23:55] PROBLEM - puppet last run on ms-be3002 is CRITICAL: CRITICAL: puppet fail [07:47:51] RECOVERY - puppet last run on ms-be3002 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [07:58:36] 06Operations, 06Commons, 10MediaWiki-Page-deletion, 10media-storage, and 5 others: Unable to delete file pages on commons: MWException/LocalFileLockError: "Could not acquire lock" - https://phabricator.wikimedia.org/T132921#2445196 (10Pokefan95) [09:22:14] 06Operations, 10Traffic, 07HTTPS, 07Tracking: SSL related (tracking) - https://phabricator.wikimedia.org/T29946#2445243 (10Danny_B) [09:22:37] 06Operations, 10Traffic, 07HTTPS, 07Tracking: SSL related (tracking) - https://phabricator.wikimedia.org/T29946#319239 (10Danny_B) [10:15:30] 07Blocked-on-Operations, 06Operations, 10Wikimedia-Site-requests, 07Community-consensus-needed, 13Patch-For-Review: Add the Kartographer extension to Metawiki - https://phabricator.wikimedia.org/T139787#2445293 (10Ash_Crow) @Urbanecm : the discussion is currently running at https://meta.wikimedia.org/wik... [10:49:12] 07Blocked-on-Operations, 06Operations, 10Wikimedia-Site-requests, 07Community-consensus-needed, 13Patch-For-Review: Add the Kartographer extension to Metawiki - https://phabricator.wikimedia.org/T139787#2445431 (10Urbanecm) Thanks @Ash_Crow. [11:38:20] PROBLEM - puppet last run on ganeti2006 is CRITICAL: CRITICAL: puppet fail [12:03:00] RECOVERY - puppet last run on ganeti2006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:29:30] PROBLEM - puppet last run on mw1141 is CRITICAL: CRITICAL: Puppet has 1 failures [12:54:10] RECOVERY - puppet last run on mw1141 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [16:24:44] PROBLEM - puppet last run on cp3009 is CRITICAL: CRITICAL: Puppet has 1 failures [16:49:42] RECOVERY - puppet last run on cp3009 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [17:07:00] 06Operations, 06Commons, 10media-storage, 07User-notice: Some fonts not anti-aliasing in SVG thumbnails after upgrade of scaling servers - https://phabricator.wikimedia.org/T139543#2445717 (10kaldari) Yeah, sounds like we should make sure the whole stack is up to date: Pango, Cairo, Harfbuzz, Ghostscript. [17:40:58] 06Operations: (www.)wmfusercontent.org should respond to HTTP - https://phabricator.wikimedia.org/T104735#2445797 (10Bawolff) 05stalled>03Resolved both http://phab.wmfusercontent.org and http://wmfusercontent.org respond with 301 Moved Permanently. Closing as fixed. [17:46:39] PROBLEM - puppet last run on cp2021 is CRITICAL: CRITICAL: Puppet has 1 failures [18:12:05] RECOVERY - puppet last run on cp2021 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [18:36:16] (03PS1) 10Urbanecm: Add possibility to disable CompactLink in default state and disable it on enwikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/298187 (https://phabricator.wikimedia.org/T139903) [18:38:47] (03CR) 10Dereckson: Add possibility to disable CompactLink in default state and disable it on enwikivoyage (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/298187 (https://phabricator.wikimedia.org/T139903) (owner: 10Urbanecm) [18:43:23] PROBLEM - puppet last run on mw2236 is CRITICAL: CRITICAL: puppet fail [18:45:13] (03PS2) 10Urbanecm: Add possibility to disable CompactLink in default state and disable it on enwikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/298187 (https://phabricator.wikimedia.org/T139903) [18:45:57] (03CR) 10Urbanecm: "Fixed." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/298187 (https://phabricator.wikimedia.org/T139903) (owner: 10Urbanecm) [19:08:03] RECOVERY - puppet last run on mw2236 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [19:08:12] PROBLEM - puppet last run on wtp2013 is CRITICAL: CRITICAL: puppet fail [19:09:13] PROBLEM - puppet last run on hassaleh is CRITICAL: CRITICAL: puppet fail [19:11:51] 06Operations, 10Analytics, 06Zero, 05Security, 07audits-data-retention: Purge > 90 days stat1002:/a/squid/archive/sampled - https://phabricator.wikimedia.org/T92342#2445892 (10Bawolff) p:05Low>03Lowest [19:34:59] RECOVERY - puppet last run on wtp2013 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:35:49] RECOVERY - puppet last run on hassaleh is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:07:57] my attempts to log in to any of the wikis is resulting in: Fatal exception of type "Exception" [21:09:52] twentyafterfour, try again? [21:11:12] (03PS1) 10Reedy: Alphasort extension-list-labs and swap to using extension.json [mediawiki-config] - 10https://gerrit.wikimedia.org/r/298236 [21:12:11] Krenair: tried several times. I'm looking for logs now [21:12:44] which wiki are you trying to log in on? [21:13:15] ah it's the cannot find local user data bug again. [21:13:51] Krenair: tried en.wikipedia.org, mediawiki.org and wikitech [21:13:54] (03PS2) 10Reedy: Alphasort extension-list-labs, use extension.json [mediawiki-config] - 10https://gerrit.wikimedia.org/r/298236 [21:14:02] ok [21:17:13] (03PS3) 10Reedy: SpecialNuke -> Nuke/extension.json [mediawiki-config] - 10https://gerrit.wikimedia.org/r/298054 [21:32:34] :-/ [21:35:23] (03PS4) 10Reedy: [WIP] Swap to using extension.json where it exists in extension-list [mediawiki-config] - 10https://gerrit.wikimedia.org/r/298054 (https://phabricator.wikimedia.org/T139800) [22:00:34] (03PS5) 10Reedy: Swap to using extension.json where it exists in extension-list [mediawiki-config] - 10https://gerrit.wikimedia.org/r/298054 (https://phabricator.wikimedia.org/T139800) [22:09:10] (03PS1) 10Reedy: Use extension.json in extension-list-wikitech [mediawiki-config] - 10https://gerrit.wikimedia.org/r/298238 (https://phabricator.wikimedia.org/T139800) [22:59:41] 06Operations, 10MediaWiki-JobQueue: Restore 30 minutes delayed list update to no waiting, to stop killing sandbox functionality - https://phabricator.wikimedia.org/T139893#2446085 (10Peachey88) [23:41:36] PROBLEM - puppet last run on mw2017 is CRITICAL: CRITICAL: puppet fail [23:59:47] PROBLEM - puppet last run on ms-be2022 is CRITICAL: CRITICAL: puppet fail