[00:00:06] !log icinga1001 - using wmf-auto-reimage to reinstall gets stuck at initial puppet run after reboot - Still waiting for Puppet after 105.0 minutes - aborting on cumin, loggin in directly and manually running puppet (T202782 T208100) [00:00:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:00:11] T208100: cumin tries to downtime Icinga even with --no-downtime - https://phabricator.wikimedia.org/T208100 [00:00:12] T202782: upgrade icinga server to stretch and replace einsteinium - https://phabricator.wikimedia.org/T202782 [00:00:13] 10Operations, 10monitoring, 10Patch-For-Review: upgrade icinga server to stretch and replace einsteinium - https://phabricator.wikimedia.org/T202782 (10ops-monitoring-bot) Completed auto-reimage of hosts: ``` ['icinga1001.wikimedia.org'] ``` Of which those **FAILED**: ``` ['icinga1001.wikimedia.org'] ``` [00:00:44] PROBLEM - Maps - OSM synchronization lag - codfw on einsteinium is CRITICAL: 1.728e+05 ge 1.728e+05 https://grafana.wikimedia.org/dashboard/db/maps-performances?panelId=12&fullscreen&orgId=1 [00:01:04] PROBLEM - Maps - OSM synchronization lag - eqiad on einsteinium is CRITICAL: 1.729e+05 ge 1.728e+05 https://grafana.wikimedia.org/dashboard/db/maps-performances?panelId=11&fullscreen&orgId=1 [00:09:03] 10Operations: httpd class and php7.0 - conflict with mpm_event module - https://phabricator.wikimedia.org/T208108 (10Dzahn) [00:09:52] 10Operations: httpd class and php7.0 - conflict with mpm_event module - https://phabricator.wikimedia.org/T208108 (10Dzahn) [00:13:12] 10Operations: httpd class and php7.0 - conflict with mpm_event module - https://phabricator.wikimedia.org/T208108 (10Dzahn) Nevermind, exact same issue already described in role::webperf::profiling_tools ``` # class httpd installs mpm_event by default, and once installed, # it cannot easily be uninstal... [00:17:02] (03PS1) 10Dzahn: icinga/alerting_host: ensure mpm_prefork is selected for httpd and php7.0 [puppet] - 10https://gerrit.wikimedia.org/r/470106 (https://phabricator.wikimedia.org/T208108) [00:31:28] (03PS2) 10Dzahn: icinga/alerting_host: ensure mpm_prefork is selected for httpd and php7.0 [puppet] - 10https://gerrit.wikimedia.org/r/470106 (https://phabricator.wikimedia.org/T208108) [00:33:23] (03PS3) 10Dzahn: icinga/alerting_host: ensure mpm_prefork is selected for httpd and php7.0 [puppet] - 10https://gerrit.wikimedia.org/r/470106 (https://phabricator.wikimedia.org/T208108) [00:33:32] (03CR) 10Dzahn: [C: 032] "https://puppet-compiler.wmflabs.org/compiler1002/13228/" [puppet] - 10https://gerrit.wikimedia.org/r/470106 (https://phabricator.wikimedia.org/T208108) (owner: 10Dzahn) [00:38:46] 10Operations, 10Patch-For-Review: httpd class and php7.0 - conflict with mpm_event module - https://phabricator.wikimedia.org/T208108 (10Dzahn) 05Open>03Resolved a:03Dzahn [00:39:00] 10Operations, 10Patch-For-Review: httpd class and php7.0 - conflict with mpm_event module - https://phabricator.wikimedia.org/T208108 (10Dzahn) [00:39:03] 10Operations, 10monitoring, 10Patch-For-Review: upgrade icinga server to stretch and replace einsteinium - https://phabricator.wikimedia.org/T202782 (10Dzahn) [00:42:57] (03PS2) 10Dzahn: icinga: tune check_result_reaper_frequency and _time on stretch [puppet] - 10https://gerrit.wikimedia.org/r/470077 (https://phabricator.wikimedia.org/T208066) [00:45:32] (03CR) 10Dzahn: [C: 032] "https://puppet-compiler.wmflabs.org/compiler1002/13229/" [puppet] - 10https://gerrit.wikimedia.org/r/470077 (https://phabricator.wikimedia.org/T208066) (owner: 10Dzahn) [00:45:43] (03PS3) 10Dzahn: icinga: tune check_result_reaper_frequency and _time on stretch [puppet] - 10https://gerrit.wikimedia.org/r/470077 (https://phabricator.wikimedia.org/T208066) [00:46:34] RECOVERY - pdfrender on scb1003 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 8.384 second response time [00:50:03] PROBLEM - pdfrender on scb1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:50:17] (03PS5) 10Dzahn: icinga: on stretch, use systemd::service, unit file by systemd-sysv-generator [puppet] - 10https://gerrit.wikimedia.org/r/462600 (https://phabricator.wikimedia.org/T202782) [00:51:03] (03CR) 10jerkins-bot: [V: 04-1] icinga: on stretch, use systemd::service, unit file by systemd-sysv-generator [puppet] - 10https://gerrit.wikimedia.org/r/462600 (https://phabricator.wikimedia.org/T202782) (owner: 10Dzahn) [00:55:22] (03CR) 10Dzahn: [C: 04-1] "ignoring jerkins, but nevertheless this is an issue: https://puppet-compiler.wmflabs.org/compiler1002/13230/icinga1001.wikimedia.org/chang" [puppet] - 10https://gerrit.wikimedia.org/r/462600 (https://phabricator.wikimedia.org/T202782) (owner: 10Dzahn) [01:04:53] (03PS6) 10Dzahn: icinga: on stretch, use systemd::service, unit file by systemd-sysv-generator [puppet] - 10https://gerrit.wikimedia.org/r/462600 (https://phabricator.wikimedia.org/T202782) [01:05:35] (03CR) 10jerkins-bot: [V: 04-1] icinga: on stretch, use systemd::service, unit file by systemd-sysv-generator [puppet] - 10https://gerrit.wikimedia.org/r/462600 (https://phabricator.wikimedia.org/T202782) (owner: 10Dzahn) [01:08:13] 10Puppet, 10Cloud-VPS (Ubuntu Trusty Deprecation): cloudvps: puppet project trusty deprecation - https://phabricator.wikimedia.org/T204558 (10Dzahn) a:05Dzahn>03akosiaris @akosiaris just confirming it can be deleted...right? [01:10:02] 10Operations, 10Patch-For-Review: Reallocate former image scalers - https://phabricator.wikimedia.org/T192457 (10Dzahn) @Joe I think you have a preference already what these should be used for, right? [01:11:03] (03CR) 10Dzahn: [C: 04-1] icinga: on stretch, use systemd::service, unit file by systemd-sysv-generator (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/462600 (https://phabricator.wikimedia.org/T202782) (owner: 10Dzahn) [01:13:25] 10Operations, 10Patch-For-Review: Reallocate former image scalers - https://phabricator.wikimedia.org/T192457 (10Dzahn) a:05Dzahn>03Joe I had this to get former mwmaint1001 back into the "spare" pool. That is done. Happy to also help reinstalling the others but you know which role you wanted them for. Feel... [01:25:13] RECOVERY - pdfrender on scb1003 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 9.591 second response time [01:28:43] PROBLEM - pdfrender on scb1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:12:24] RECOVERY - pdfrender on scb1003 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 9.140 second response time [02:15:53] PROBLEM - pdfrender on scb1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:33:56] !log smalyshev@deploy1001 Started deploy [wdqs/wdqs@e9392f4]: Re-deploy Updater to deal with performance issues [02:33:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:34:01] !log smalyshev@deploy1001 Finished deploy [wdqs/wdqs@e9392f4]: Re-deploy Updater to deal with performance issues (duration: 00m 05s) [02:34:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:34:33] !log smalyshev@deploy1001 Started deploy [wdqs/wdqs@7eeede7]: Re-deploy Updater to deal with performance issues [02:34:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:35:11] !log smalyshev@deploy1001 Finished deploy [wdqs/wdqs@7eeede7]: Re-deploy Updater to deal with performance issues (duration: 00m 38s) [02:35:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:39:54] RECOVERY - Maps - OSM synchronization lag - codfw on einsteinium is OK: (C)1.728e+05 ge (W)9e+04 ge 9591 https://grafana.wikimedia.org/dashboard/db/maps-performances?panelId=12&fullscreen&orgId=1 [02:40:13] RECOVERY - Maps - OSM synchronization lag - eqiad on einsteinium is OK: (C)1.728e+05 ge (W)9e+04 ge 9606 https://grafana.wikimedia.org/dashboard/db/maps-performances?panelId=11&fullscreen&orgId=1 [03:08:53] RECOVERY - pdfrender on scb1003 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 6.082 second response time [03:12:13] PROBLEM - pdfrender on scb1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:31:44] PROBLEM - MariaDB Slave Lag: s1 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 877.52 seconds [03:41:41] !log depool wdqs1003 again to let it catch up some more [03:41:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [04:33:24] RECOVERY - MariaDB Slave Lag: s1 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 265.29 seconds [06:15:33] RECOVERY - pdfrender on scb1003 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 7.205 second response time [06:18:53] PROBLEM - pdfrender on scb1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:25:34] RECOVERY - pdfrender on scb1003 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 4.889 second response time [06:28:54] PROBLEM - pdfrender on scb1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:31:03] RECOVERY - pdfrender on scb1003 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 1.522 second response time [06:33:04] PROBLEM - puppet last run on labvirt1014 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apparmor.d/abstractions/ssl_certs] [06:34:23] PROBLEM - pdfrender on scb1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:58:35] RECOVERY - puppet last run on labvirt1014 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [08:15:34] (03PS1) 10Addshore: Wikibase, Set siteLinkGroups settings on all wikis again [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470139 (https://phabricator.wikimedia.org/T208048) [08:19:13] (03PS2) 10Addshore: Wikibase, Set siteLinkGroups settings on all wikis again [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470139 (https://phabricator.wikimedia.org/T208048) [08:20:03] RECOVERY - pdfrender on scb1003 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 9.780 second response time [08:20:44] (03PS1) 10Addshore: BETA: wmgUseWikibaseMediaInfo false (again) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470140 (https://phabricator.wikimedia.org/T208043) [08:21:12] (03PS2) 10Addshore: BETA: wmgUseWikibaseMediaInfo false (again) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470140 (https://phabricator.wikimedia.org/T208043) [08:21:20] (03CR) 10Addshore: [C: 032] BETA: wmgUseWikibaseMediaInfo false (again) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470140 (https://phabricator.wikimedia.org/T208043) (owner: 10Addshore) [08:21:39] ^^ beta only patch (-labs only file) to fix updating on beta [08:23:07] (03Merged) 10jenkins-bot: BETA: wmgUseWikibaseMediaInfo false (again) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470140 (https://phabricator.wikimedia.org/T208043) (owner: 10Addshore) [08:23:24] PROBLEM - pdfrender on scb1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:24:45] !log addshore@deploy1001 Synchronized wmf-config/InitialiseSettings-labs.php: BETA ONLY T208043 (duration: 01m 06s) [08:24:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:24:49] T208043: Unknown entity type mediainfo - https://phabricator.wikimedia.org/T208043 [08:25:03] RECOVERY - Disk space on notebook1004 is OK: DISK OK [08:25:13] (03CR) 10Tarrow: [C: 031] Wikibase, Set siteLinkGroups settings on all wikis again [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470139 (https://phabricator.wikimedia.org/T208048) (owner: 10Addshore) [08:26:56] addshore: tarrow good morning. Do you want me to deploy/ [08:26:58] *? [08:26:58] (03PS3) 10Addshore: Wikibase, Set siteLinkGroups settings on all wikis again [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470139 (https://phabricator.wikimedia.org/T208048) [08:27:02] hi Amir1 [08:27:06] hey! [08:27:13] (03CR) 10Addshore: [C: 032] Wikibase, Set siteLinkGroups settings on all wikis again [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470139 (https://phabricator.wikimedia.org/T208048) (owner: 10Addshore) [08:27:18] why are we all here on a saturday? [08:27:34] (03CR) 10jenkins-bot: BETA: wmgUseWikibaseMediaInfo false (again) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470140 (https://phabricator.wikimedia.org/T208043) (owner: 10Addshore) [08:27:34] Amir1: I'm going to do it now, as its clearly causing issues :) [08:27:43] * tarrow is here only because he saw the tickets last night :) [08:27:44] I'm at Wiki Techstorm in the Hague, I don't know about you :P [08:27:55] Amir1: ooooh, im in Schipol [08:28:04] I'm in https://avointiede.fi/wide-hackathon [08:28:06] addshore: sure, let me know if you need anything [08:28:13] Amir1: ack :) [08:28:28] cool, come visit us :P [08:28:34] Nemo_bis: ooooh [08:28:41] (03Merged) 10jenkins-bot: Wikibase, Set siteLinkGroups settings on all wikis again [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470139 (https://phabricator.wikimedia.org/T208048) (owner: 10Addshore) [08:28:44] Amir1: I nearly did, but I'm very tried, didn't sleep much on the flight back from tech conf.... [08:29:03] oh yeah :( [08:29:10] I let you rest [08:29:18] right, the change is on mwdebug *goes to check it* [08:32:10] looks like I seem more entries in getSites() :) [08:33:02] yup, i just did a test edit and it fixes the problem too [08:33:06] cool, will sync [08:34:52] !log addshore@deploy1001 Synchronized wmf-config/Wikibase.php: Wikibase, Set siteLinkGroups settings on all wikis again T208048 T208077 T208074 (duration: 00m 54s) [08:34:57] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:34:58] T208074: Other projects interwiki block does not work - https://phabricator.wikimedia.org/T208074 [08:34:58] T208048: [BUG] Sidebar interwiki linking failed - https://phabricator.wikimedia.org/T208048 [08:34:58] T208077: Connecting pages failed at Wkidata with wrong message - https://phabricator.wikimedia.org/T208077 [08:41:07] Amir1: have fun at wikitechstorm [08:41:13] or whatever it is called :P [08:41:45] hehe. You too. Get rest [08:41:51] yupp [08:43:14] (03CR) 10jenkins-bot: Wikibase, Set siteLinkGroups settings on all wikis again [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470139 (https://phabricator.wikimedia.org/T208048) (owner: 10Addshore) [09:05:37] (03PS1) 10Addshore: Remove wgArticlePlaceholderSearchIntegrationBackend BETA override [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470144 [09:06:14] (03CR) 10Addshore: [C: 032] Remove wgArticlePlaceholderSearchIntegrationBackend BETA override [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470144 (owner: 10Addshore) [09:07:56] (03Merged) 10jenkins-bot: Remove wgArticlePlaceholderSearchIntegrationBackend BETA override [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470144 (owner: 10Addshore) [09:09:33] !log addshore@deploy1001 Synchronized wmf-config/InitialiseSettings-labs.php: Remove wgArticlePlaceholderSearchIntegrationBackend BETA override (duration: 01m 00s) [09:09:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:11:07] (03PS1) 10Addshore: BETA remove wmgWikibaseAllowLocalShortDesc override (same as prod) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470145 [09:11:23] (03CR) 10Addshore: [C: 032] BETA remove wmgWikibaseAllowLocalShortDesc override (same as prod) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470145 (owner: 10Addshore) [09:12:11] (03PS1) 10Addshore: BETA, use wmgWikibaseAllowDataAccessInUserLanguage from prod [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470146 [09:12:45] (03PS1) 10Addshore: BETA, use wmgWikibaseDisabledDataTypes from prod [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470147 [09:12:53] (03Merged) 10jenkins-bot: BETA remove wmgWikibaseAllowLocalShortDesc override (same as prod) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470145 (owner: 10Addshore) [09:13:07] (03CR) 10Addshore: [C: 032] BETA, use wmgWikibaseAllowDataAccessInUserLanguage from prod [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470146 (owner: 10Addshore) [09:13:15] (03CR) 10Addshore: [C: 032] BETA, use wmgWikibaseDisabledDataTypes from prod [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470147 (owner: 10Addshore) [09:14:09] (03Merged) 10jenkins-bot: BETA, use wmgWikibaseAllowDataAccessInUserLanguage from prod [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470146 (owner: 10Addshore) [09:14:15] (03PS1) 10Addshore: BETA, use wgArticlePlaceholderSearchIntegrationEnabled from prod [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470148 [09:14:29] (03Merged) 10jenkins-bot: BETA, use wmgWikibaseDisabledDataTypes from prod [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470147 (owner: 10Addshore) [09:14:31] (03CR) 10Addshore: [C: 032] BETA, use wgArticlePlaceholderSearchIntegrationEnabled from prod [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470148 (owner: 10Addshore) [09:14:37] (03CR) 10jenkins-bot: Remove wgArticlePlaceholderSearchIntegrationBackend BETA override [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470144 (owner: 10Addshore) [09:14:39] (03CR) 10jenkins-bot: BETA remove wmgWikibaseAllowLocalShortDesc override (same as prod) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470145 (owner: 10Addshore) [09:14:41] (03CR) 10jenkins-bot: BETA, use wmgWikibaseAllowDataAccessInUserLanguage from prod [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470146 (owner: 10Addshore) [09:14:43] (03CR) 10jenkins-bot: BETA, use wmgWikibaseDisabledDataTypes from prod [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470147 (owner: 10Addshore) [09:15:33] (03Merged) 10jenkins-bot: BETA, use wgArticlePlaceholderSearchIntegrationEnabled from prod [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470148 (owner: 10Addshore) [09:17:08] !log addshore@deploy1001 Synchronized wmf-config/InitialiseSettings-labs.php: BETA ONLY (4x patches) (duration: 00m 55s) [09:17:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [09:30:11] (03CR) 10jenkins-bot: BETA, use wgArticlePlaceholderSearchIntegrationEnabled from prod [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470148 (owner: 10Addshore) [09:36:44] (03PS1) 10Addshore: Wikibase, create and use wmgWikibaseMaxSerializedEntitySize [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470149 [09:36:46] (03PS1) 10Addshore: Wikibase, Split specialSiteLinkGroups and manage from IS.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470150 [09:37:01] (03PS1) 10Addshore: Wikibase, move wmgWBSiteLinkGroups to IS.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470151 [09:41:26] (03PS1) 10Addshore: Wikibase, kill $wmgWBSharedSettings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470152 [09:41:38] (03PS2) 10Addshore: Wikibase, kill $wmgWBSharedSettings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470152 [09:46:56] (03PS1) 10Addshore: Wikibase, define $wgExtraNamespaces in IS.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470153 [09:48:30] 10Operations, 10Operations-Software-Development: cumin tries to downtime Icinga even with --no-downtime - https://phabricator.wikimedia.org/T208100 (10Volans) >>! In T208100#4699083, @Dzahn wrote: > Isn't the issue that despite saying --no-downtime it tries to set a downtime? No, the `--no-downtime` option do... [09:49:48] (03PS1) 10Addshore: Wikibase, put all wgNamespaceAliases in IS.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470154 [09:53:23] 10Operations, 10monitoring, 10Patch-For-Review: upgrade icinga server to stretch and replace einsteinium - https://phabricator.wikimedia.org/T202782 (10Volans) >>! In T202782#4699100, @Stashbot wrote: > {nav icon=file, name=Mentioned in SAL (#wikimedia-operations), href=https://tools.wmflabs.org/sal/log/AWay... [09:55:00] (03PS1) 10Addshore: Wikibase, create and use wmgWikibaseClientInjectRecentChanges [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470155 [09:55:44] RECOVERY - pdfrender on scb1003 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 8.523 second response time [09:58:24] (03PS1) 10Addshore: Wikibase, remove unused wmgWikibaseClientSettings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470156 [09:59:04] PROBLEM - pdfrender on scb1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:32:59] (03PS1) 10Addshore: Wikibase, Remove unused wmgUseWikibaseQualityExternalValidation [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470158 [10:33:01] (03PS1) 10Addshore: Wikibase, add IS.php setting for each possible extension [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470159 [10:33:03] (03PS1) 10Addshore: Wikibase.php, move a bunch of config into 'clean' area [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470160 [10:33:05] (03PS1) 10Addshore: Wikibase, Create and use wmgWikibaseRepoStatementSections [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470161 [10:36:24] (03CR) 10jerkins-bot: [V: 04-1] Wikibase, Create and use wmgWikibaseRepoStatementSections [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470161 (owner: 10Addshore) [11:10:13] (03CR) 10Urbanecm: "urbanecm@notebook ~" [dns] - 10https://gerrit.wikimedia.org/r/467087 (https://phabricator.wikimedia.org/T206923) (owner: 10Urbanecm) [11:17:10] 10Operations, 10DNS, 10Traffic, 10WMCZ-General, and 4 others: Remove *.cz domains from WMF's infrastructure - https://phabricator.wikimedia.org/T206923 (10Urbanecm) ``` urbanecm@notebook ~ $ host wikizdroje.cz wikizdroje.cz has address 81.95.96.75 urbanecm@notebook ~ $ ``` Look like it propagated. [11:17:28] 10Operations, 10DNS, 10Traffic, 10WMCZ-General, and 4 others: Remove *.cz domains from WMF's infrastructure - https://phabricator.wikimedia.org/T206923 (10Urbanecm) ``` urbanecm@notebook ~ $ host wikizdroje.cz wikizdroje.cz has address 81.95.96.75 urbanecm@notebook ~ $ ``` Look like it propagated. [11:23:52] 10Operations, 10DNS, 10Traffic, 10WMCZ-General, and 4 others: Remove *.cz domains from WMF's infrastructure - https://phabricator.wikimedia.org/T206923 (10Urbanecm) >>! In T206923#4698418, @Dzahn wrote: > Also i was wondering what about mail? It looks like the MX records are pointing to wikimedia.org for... [12:10:57] 10Operations, 10New-Readers: Create URL for Mexico Awareness Campaign - https://phabricator.wikimedia.org/T207816 (10Prtksxna) >>! In T207816#4699091, @Dzahn wrote: > @Prtksxna Is this a dynamic page with scripting or is it a static page with just HTML/CSS and some images? Yep. But, one of the main requirem... [12:26:53] RECOVERY - pdfrender on scb1003 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 9.010 second response time [12:29:13] !log resuming replication on s1@dbstore2002 as table compression is finished (T204930) [12:29:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:29:17] T204930: dbstore2002 tables compression status check - https://phabricator.wikimedia.org/T204930 [12:30:13] PROBLEM - pdfrender on scb1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:32:10] !log Deployed patch for T207576 [12:32:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:44:34] PROBLEM - Nginx local proxy to apache on mw1285 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:45:34] RECOVERY - Nginx local proxy to apache on mw1285 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 617 bytes in 0.050 second response time [12:55:14] (03PS6) 10Banyek: mariadb: table checker for monitoring data drift [puppet] - 10https://gerrit.wikimedia.org/r/469889 (https://phabricator.wikimedia.org/T207253) [12:55:52] (03CR) 10jerkins-bot: [V: 04-1] mariadb: table checker for monitoring data drift [puppet] - 10https://gerrit.wikimedia.org/r/469889 (https://phabricator.wikimedia.org/T207253) (owner: 10Banyek) [12:56:44] (03CR) 10GTirloni: git-sync-upstream: Send cron mail in case of failures (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/468865 (https://phabricator.wikimedia.org/T184261) (owner: 10GTirloni) [13:06:15] (03PS7) 10Banyek: mariadb: table checker for monitoring data drift [puppet] - 10https://gerrit.wikimedia.org/r/469889 (https://phabricator.wikimedia.org/T207253) [13:16:52] (03CR) 10GTirloni: git-sync-upstream: Send cron mail in case of failures (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/468865 (https://phabricator.wikimedia.org/T184261) (owner: 10GTirloni) [13:39:38] (03Abandoned) 10Reedy: Enable CSP in Report Only Mode everywhere [mediawiki-config] - 10https://gerrit.wikimedia.org/r/439471 (owner: 10Reedy) [15:10:11] (03PS1) 10Tarrow: Revert "Wikibase, Set siteLinkGroups settings on all wikis again" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470170 [15:11:01] (03PS1) 10Tarrow: Revert "Wikibase.php, don't load wikidata repo settings on other repos (take 2)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470172 [15:11:11] (03CR) 10jerkins-bot: [V: 04-1] Revert "Wikibase.php, don't load wikidata repo settings on other repos (take 2)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470172 (owner: 10Tarrow) [15:11:43] (03PS2) 10Tarrow: Revert "Wikibase, Set siteLinkGroups settings on all wikis again" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470170 [15:13:43] (03Abandoned) 10Tarrow: Revert "Wikibase, Set siteLinkGroups settings on all wikis again" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470170 (owner: 10Tarrow) [15:13:58] (03Abandoned) 10Tarrow: Revert "Wikibase.php, don't load wikidata repo settings on other repos (take 2)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470172 (owner: 10Tarrow) [15:48:54] PROBLEM - HHVM jobrunner on mw1296 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 473 bytes in 0.001 second response time [15:49:30] (03PS1) 10Addshore: Wikibase, make sure specialSiteLinkGroups has wikidata group [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470175 [15:49:54] RECOVERY - HHVM jobrunner on mw1296 is OK: HTTP OK: HTTP/1.1 200 OK - 206 bytes in 0.006 second response time [15:50:17] (03CR) 10Addshore: [C: 032] Wikibase, make sure specialSiteLinkGroups has wikidata group [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470175 (owner: 10Addshore) [15:50:22] (03PS2) 10Addshore: Wikibase, make sure specialSiteLinkGroups has wikidata group [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470175 [15:50:28] (03CR) 10Addshore: [C: 032] Wikibase, make sure specialSiteLinkGroups has wikidata group [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470175 (owner: 10Addshore) [15:51:33] (03Merged) 10jenkins-bot: Wikibase, make sure specialSiteLinkGroups has wikidata group [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470175 (owner: 10Addshore) [15:57:24] !log addshore@deploy1001 Synchronized wmf-config/Wikibase.php: Wikibase, make sure specialSiteLinkGroups has wikidata group (duration: 00m 54s) [15:57:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:01:15] (03CR) 10jenkins-bot: Wikibase, make sure specialSiteLinkGroups has wikidata group [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470175 (owner: 10Addshore) [16:14:17] (03PS1) 10Addshore: Wikibase, fix duplicate specialSiteLinkGroups key [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470176 (https://phabricator.wikimedia.org/T208124) [16:14:41] (03CR) 10Addshore: [C: 032] Wikibase, fix duplicate specialSiteLinkGroups key [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470176 (https://phabricator.wikimedia.org/T208124) (owner: 10Addshore) [16:16:02] (03Merged) 10jenkins-bot: Wikibase, fix duplicate specialSiteLinkGroups key [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470176 (https://phabricator.wikimedia.org/T208124) (owner: 10Addshore) [16:16:24] (03CR) 10jenkins-bot: Wikibase, fix duplicate specialSiteLinkGroups key [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470176 (https://phabricator.wikimedia.org/T208124) (owner: 10Addshore) [16:18:22] !log addshore@deploy1001 Synchronized wmf-config/Wikibase.php: Wikibase, fix duplicate specialSiteLinkGroups key T208124 (duration: 00m 54s) [16:18:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:18:26] T208124: Can't edit "Other sites" on Wikidata - https://phabricator.wikimedia.org/T208124 [17:21:12] (03PS2) 10Addshore: Wikibase, create and use wmgWikibaseMaxSerializedEntitySize [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470149 [17:25:12] (03PS2) 10Addshore: Wikibase, Split specialSiteLinkGroups and manage from IS.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470150 [17:26:56] (03PS2) 10Addshore: Wikibase, move wmgWBSiteLinkGroups to IS.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470151 [17:27:03] (03PS3) 10Addshore: Wikibase, kill $wmgWBSharedSettings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470152 [17:27:09] (03PS2) 10Addshore: Wikibase, define $wgExtraNamespaces in IS.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470153 [17:27:16] (03PS2) 10Addshore: Wikibase, put all wgNamespaceAliases in IS.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470154 [17:27:22] (03PS2) 10Addshore: Wikibase, create and use wmgWikibaseClientInjectRecentChanges [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470155 [17:27:28] (03PS2) 10Addshore: Wikibase, remove unused wmgWikibaseClientSettings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470156 [17:27:35] (03PS2) 10Addshore: Wikibase, Remove unused wmgUseWikibaseQualityExternalValidation [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470158 [17:27:41] (03PS2) 10Addshore: Wikibase, add IS.php setting for each possible extension [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470159 [17:27:51] (03PS2) 10Addshore: Wikibase.php, move a bunch of config into 'clean' area [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470160 [17:28:38] (03PS2) 10Addshore: Wikibase, Create and use wmgWikibaseRepoStatementSections [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470161 [17:30:14] PROBLEM - puppet last run on sodium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [17:40:53] PROBLEM - HP RAID on db2048 is CRITICAL: CRITICAL: Slot 0: OK: 1I:1:1, 1I:1:3, 1I:1:4, 1I:1:5, 1I:1:6, 1I:1:7, 1I:1:8, 1I:1:9, 1I:1:10, 1I:1:11, 1I:1:12 - Failed: 1I:1:2 - Controller: OK - Battery/Capacitor: OK [17:40:55] ACKNOWLEDGEMENT - HP RAID on db2048 is CRITICAL: CRITICAL: Slot 0: OK: 1I:1:1, 1I:1:3, 1I:1:4, 1I:1:5, 1I:1:6, 1I:1:7, 1I:1:8, 1I:1:9, 1I:1:10, 1I:1:11, 1I:1:12 - Failed: 1I:1:2 - Controller: OK - Battery/Capacitor: OK nagiosadmin RAID handler auto-ack: https://phabricator.wikimedia.org/T208141 [17:41:00] 10Operations, 10ops-codfw: Degraded RAID on db2048 - https://phabricator.wikimedia.org/T208141 (10ops-monitoring-bot) [17:55:44] RECOVERY - puppet last run on sodium is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [18:22:04] RECOVERY - pdfrender on scb1003 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 4.836 second response time [18:25:34] PROBLEM - pdfrender on scb1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:28:48] (03CR) 10Elukey: git-sync-upstream: Send cron mail in case of failures (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/468865 (https://phabricator.wikimedia.org/T184261) (owner: 10GTirloni) [20:16:53] RECOVERY - pdfrender on scb1003 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 8.684 second response time [20:20:23] PROBLEM - pdfrender on scb1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:36:03] PROBLEM - High lag on wdqs1003 is CRITICAL: 5042 ge 3600 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [20:56:57] !log depooling wdqs1003 to catch up with others [20:56:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:05:21] (03CR) 10Krinkle: [C: 031] "LGTM, remember to swat IS before WB (or to be able to stage it, separate patch)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470149 (owner: 10Addshore) [21:05:26] addshore: nice going :) [21:10:01] 10Operations, 10ops-codfw: Degraded RAID on db2048 - https://phabricator.wikimedia.org/T208141 (10Banyek) p:05Triage>03Normal a:03Papaul Can we get a new disk here @Papaul? Thanks in advance! [21:12:23] PROBLEM - Host db1117 is DOWN: PING CRITICAL - Packet loss = 100% [21:13:03] PROBLEM - haproxy failover on dbproxy1007 is CRITICAL: CRITICAL check_failover servers up 1 down 1 [21:13:04] PROBLEM - haproxy failover on dbproxy1002 is CRITICAL: CRITICAL check_failover servers up 1 down 1 [21:13:13] PROBLEM - haproxy failover on dbproxy1003 is CRITICAL: CRITICAL check_failover servers up 1 down 1 [21:13:23] PROBLEM - haproxy failover on dbproxy1006 is CRITICAL: CRITICAL check_failover servers up 1 down 1 [21:13:24] PROBLEM - haproxy failover on dbproxy1008 is CRITICAL: CRITICAL check_failover servers up 1 down 1 [21:13:53] PROBLEM - haproxy failover on dbproxy1001 is CRITICAL: CRITICAL check_failover servers up 1 down 1 [21:14:01] marostegui banyek|away ^^ [21:14:13] I am just arrived, checking [21:14:31] thanks. [21:14:37] np [21:15:43] ACKNOWLEDGEMENT - HP RAID on db2048 is CRITICAL: CRITICAL: Slot 0: OK: 1I:1:1, 1I:1:3, 1I:1:4, 1I:1:5, 1I:1:6, 1I:1:7, 1I:1:8, 1I:1:9, 1I:1:10, 1I:1:11, 1I:1:12 - Failed: 1I:1:2 - Controller: OK - Battery/Capacitor: OK Banyek T208141 [21:22:06] The host db1117 is not available, and the mgmt interface's serial console is not showing anything, I do a power cycly [21:22:12] *cycle [21:23:03] PROBLEM - High lag on wdqs1003 is CRITICAL: 4688 ge 3600 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [21:24:44] !log resetting power on db1117 as the host is DOWN and the serial console shows nothing [21:24:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:25:13] 10Operations, 10MediaWiki-Page-deletion, 10Performance-Team, 10MW-1.32-notes, and 3 others: Deleting pages on the English Wikipedia is very slow - https://phabricator.wikimedia.org/T207530 (10Krinkle) I'm not sure if it is related or not, but I'm still experiencing problems deleting things. For example,... [21:25:19] 10Operations, 10MediaWiki-Page-deletion, 10Performance-Team, 10MW-1.32-notes, and 3 others: Deleting pages on the English Wikipedia is very slow - https://phabricator.wikimedia.org/T207530 (10Krinkle) [21:25:47] 10Operations, 10MediaWiki-Page-deletion, 10Performance-Team, 10MW-1.32-notes, and 3 others: Deleting pages on the English Wikipedia is very slow - https://phabricator.wikimedia.org/T207530 (10Krinkle) 05Resolved>03Open Re-opening for now, but if it turns out to be a separate issue we should re-close an... [21:28:04] ACKNOWLEDGEMENT - High lag on wdqs1003 is CRITICAL: 4566 ge 3600 Mathew.onipe This is a known issue. Server has been depooled to catch up https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [21:29:13] RECOVERY - Host db1117 is UP: PING OK - Packet loss = 0%, RTA = 0.84 ms [21:30:13] RECOVERY - pdfrender on scb1003 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 2.403 second response time [21:31:33] PROBLEM - mysqld processes on db1117 is CRITICAL: PROCS CRITICAL: 0 processes with command name mysqld [21:31:33] PROBLEM - MariaDB Slave IO: m3 on db1117 is CRITICAL: CRITICAL slave_io_state could not connect [21:31:33] PROBLEM - MariaDB Slave IO: m1 on db1117 is CRITICAL: CRITICAL slave_io_state could not connect [21:31:34] PROBLEM - MariaDB read only m1 on db1117 is CRITICAL: Could not connect to localhost:3321 [21:31:34] PROBLEM - MariaDB read only m3 on db1117 is CRITICAL: Could not connect to localhost:3323 [21:31:34] PROBLEM - MariaDB read only m5 on db1117 is CRITICAL: Could not connect to localhost:3325 [21:31:53] PROBLEM - MariaDB Slave IO: m5 on db1117 is CRITICAL: CRITICAL slave_io_state could not connect [21:31:54] PROBLEM - Check systemd state on db1117 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [21:32:03] PROBLEM - MariaDB Slave SQL: m3 on db1117 is CRITICAL: CRITICAL slave_sql_state could not connect [21:32:04] PROBLEM - MariaDB Slave SQL: m1 on db1117 is CRITICAL: CRITICAL slave_sql_state could not connect [21:32:04] PROBLEM - MariaDB Slave SQL: m2 on db1117 is CRITICAL: CRITICAL slave_sql_state could not connect [21:32:14] PROBLEM - MariaDB read only m2 on db1117 is CRITICAL: Could not connect to localhost:3322 [21:32:33] PROBLEM - MariaDB Slave IO: m2 on db1117 is CRITICAL: CRITICAL slave_io_state could not connect [21:32:33] PROBLEM - MariaDB Slave SQL: m5 on db1117 is CRITICAL: CRITICAL slave_sql_state could not connect [21:33:14] RECOVERY - MariaDB Slave SQL: m1 on db1117 is OK: OK slave_sql_state Slave_SQL_Running: Yes [21:33:24] RECOVERY - haproxy failover on dbproxy1006 is OK: OK check_failover servers up 2 down 0 [21:33:43] PROBLEM - pdfrender on scb1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:33:53] RECOVERY - MariaDB Slave IO: m1 on db1117 is OK: OK slave_io_state Slave_IO_Running: Yes [21:33:54] RECOVERY - MariaDB read only m1 on db1117 is OK: Version 10.1.33-MariaDB, Uptime 72s, read_only: True, 5249.31 QPS, connection latency: 0.002104s, query latency: 0.000788s [21:34:03] RECOVERY - haproxy failover on dbproxy1001 is OK: OK check_failover servers up 2 down 0 [21:36:33] RECOVERY - haproxy failover on dbproxy1007 is OK: OK check_failover servers up 2 down 0 [21:36:34] RECOVERY - haproxy failover on dbproxy1002 is OK: OK check_failover servers up 2 down 0 [21:36:34] RECOVERY - MariaDB Slave SQL: m2 on db1117 is OK: OK slave_sql_state Slave_SQL_Running: Yes [21:36:44] RECOVERY - MariaDB read only m2 on db1117 is OK: Version 10.1.33-MariaDB, Uptime 55s, read_only: True, 20.86 QPS, connection latency: 0.003210s, query latency: 0.000809s [21:37:03] RECOVERY - MariaDB Slave IO: m2 on db1117 is OK: OK slave_io_state Slave_IO_Running: Yes [21:38:14] RECOVERY - MariaDB Slave IO: m3 on db1117 is OK: OK slave_io_state Slave_IO_Running: Yes [21:38:24] RECOVERY - MariaDB read only m3 on db1117 is OK: Version 10.1.33-MariaDB, Uptime 29s, read_only: True, 4491.56 QPS, connection latency: 0.002484s, query latency: 0.000404s [21:38:44] RECOVERY - MariaDB Slave SQL: m3 on db1117 is OK: OK slave_sql_state Slave_SQL_Running: Yes [21:38:54] RECOVERY - haproxy failover on dbproxy1003 is OK: OK check_failover servers up 2 down 0 [21:39:04] RECOVERY - haproxy failover on dbproxy1008 is OK: OK check_failover servers up 2 down 0 [21:39:43] RECOVERY - MariaDB Slave IO: m5 on db1117 is OK: OK slave_io_state Slave_IO_Running: Yes [21:39:54] RECOVERY - mysqld processes on db1117 is OK: PROCS OK: 4 processes with command name mysqld [21:40:23] RECOVERY - MariaDB Slave SQL: m5 on db1117 is OK: OK slave_sql_state Slave_SQL_Running: Yes [21:40:34] RECOVERY - MariaDB read only m5 on db1117 is OK: Version 10.1.33-MariaDB, Uptime 79s, read_only: True, 32.82 QPS, connection latency: 0.003140s, query latency: 0.000597s [22:01:51] * Krinkle staging on mwdebug1002 [22:04:14] RECOVERY - High lag on wdqs1003 is OK: (C)3600 ge (W)1200 ge 198 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [22:09:44] PROBLEM - High lag on wdqs1003 is CRITICAL: 3687 ge 3600 https://grafana.wikimedia.org/dashboard/db/wikidata-query-service?orgId=1&panelId=8&fullscreen [22:22:20] !log krinkle@deploy1001 Synchronized php-1.33.0-wmf.1/resources/src: T208093- I25012a2c6f (duration: 00m 58s) [22:22:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:22:25] T208093: RLQ.push callbacks added after mediawiki.base arrives don't get called - https://phabricator.wikimedia.org/T208093 [22:31:53] RECOVERY - pdfrender on scb1003 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 5.730 second response time [22:35:13] PROBLEM - pdfrender on scb1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:23:55] (03PS1) 10Hoo man: Enable Wikidata data access on trwiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/470224 (https://phabricator.wikimedia.org/T204419) [23:51:24] RECOVERY - pdfrender on scb1003 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 8.151 second response time [23:54:54] PROBLEM - pdfrender on scb1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:56:45] * Krinkle staging on mwdebug1002 [23:59:23] RECOVERY - pdfrender on scb1003 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 4.384 second response time