[00:03:53] RECOVERY - HTTP error ratio anomaly detection on graphite1001 is OK: OK: No anomaly detected [00:05:25] RECOVERY - puppet last run on cp3037 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [00:14:05] PROBLEM - IPsec on cp2021 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v6 [00:15:44] RECOVERY - IPsec on cp2021 is OK: Strongswan OK - 8 ESP OK [00:22:13] (03CR) 10MZMcBride: "Given the merge of , this will need to be rebased." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/243517 (https://phabricator.wikimedia.org/T114613) (owner: 10Alex Monk) [00:23:34] PROBLEM - IPsec on cp4020 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v6 [00:25:14] RECOVERY - IPsec on cp4020 is OK: Strongswan OK - 8 ESP OK [00:41:34] PROBLEM - nutcracker port on silver is CRITICAL: CRITICAL - Socket timeout after 2 seconds [00:43:14] RECOVERY - nutcracker port on silver is OK: TCP OK - 0.000 second response time on port 11212 [00:55:34] PROBLEM - IPsec on cp4011 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v4 [00:57:14] RECOVERY - IPsec on cp4011 is OK: Strongswan OK - 8 ESP OK [01:11:44] PROBLEM - puppet last run on mw2149 is CRITICAL: CRITICAL: puppet fail [01:26:45] PROBLEM - nutcracker port on silver is CRITICAL: CRITICAL - Socket timeout after 2 seconds [01:28:25] RECOVERY - nutcracker port on silver is OK: TCP OK - 0.000 second response time on port 11212 [01:40:15] RECOVERY - puppet last run on mw2149 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [01:41:58] 6operations, 10Deployment-Systems, 10Salt: service-restart or git deploy service restart does not wait between batches - https://phabricator.wikimedia.org/T114583#1718690 (10mmodell) 5Open>3declined git-deploy is completely deprecated at this point. #scap3 supports proper batches. [01:46:04] PROBLEM - IPsec on cp4012 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v4 [01:49:25] RECOVERY - IPsec on cp4012 is OK: Strongswan OK - 8 ESP OK [01:56:04] PROBLEM - IPsec on cp4012 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v6 [01:57:45] RECOVERY - IPsec on cp4012 is OK: Strongswan OK - 8 ESP OK [02:03:49] (03CR) 10Alex Monk: "-> T100990" [puppet] - 10https://gerrit.wikimedia.org/r/243357 (owner: 10Alex Monk) [02:14:46] PROBLEM - IPsec on cp3016 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v6 [02:18:13] RECOVERY - IPsec on cp3016 is OK: Strongswan OK - 8 ESP OK [02:29:56] !log l10nupdate@tin Synchronized php-1.27.0-wmf.2/cache/l10n: l10nupdate for 1.27.0-wmf.2 (duration: 06m 50s) [02:30:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:31:14] PROBLEM - IPsec on cp2015 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v4 [02:32:55] RECOVERY - IPsec on cp2015 is OK: Strongswan OK - 8 ESP OK [02:33:13] !log l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.2) at 2015-10-12 02:33:13+00:00 [02:33:13] !log l10nupdate@tin ResourceLoader cache refresh completed at Mon Oct 12 02:33:13 UTC 2015 (duration 33m 12s) [02:33:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:33:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:39:51] (03CR) 10Alex Monk: "https://wikitech.wikimedia.org/wiki/User:Itsmeprabha is already the owner of this UID, causing puppet on deployment-fluorine to fail." [puppet] - 10https://gerrit.wikimedia.org/r/240334 (owner: 10ArielGlenn) [02:44:54] PROBLEM - Kafka Broker Replica Max Lag on kafka1018 is CRITICAL: CRITICAL: 85.71% of data above the critical threshold [5000000.0] [02:45:15] PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL: CRITICAL: Anomaly detected: 10 data above and 2 below the confidence bounds [02:53:14] RECOVERY - Kafka Broker Replica Max Lag on kafka1018 is OK: OK: Less than 1.00% above the threshold [1000000.0] [03:07:16] 7Puppet, 10Beta-Cluster-Infrastructure: Puppet failures across all beta caches due to *.wmflabs.org certificate - https://phabricator.wikimedia.org/T115238#1718737 (10Krenair) 3NEW [03:33:24] PROBLEM - IPsec on cp4011 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v4 [03:38:24] RECOVERY - IPsec on cp4011 is OK: Strongswan OK - 8 ESP OK [03:38:25] PROBLEM - IPsec on cp4020 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v4 [03:46:44] RECOVERY - IPsec on cp4020 is OK: Strongswan OK - 8 ESP OK [03:53:04] PROBLEM - IPsec on cp2015 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v6 [03:56:24] RECOVERY - IPsec on cp2015 is OK: Strongswan OK - 8 ESP OK [04:04:13] RECOVERY - HTTP error ratio anomaly detection on graphite1001 is OK: OK: No anomaly detected [04:05:33] PROBLEM - IPsec on cp2021 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v4 [04:08:54] RECOVERY - IPsec on cp2021 is OK: Strongswan OK - 8 ESP OK [04:10:46] anybody around? I'm trying to deploy something on tin and git deploy is basically ignoring me... 0/2 minions completed fetch [04:11:33] ah, looks like it worked the second time... not sure what's going on [04:16:14] PROBLEM - IPsec on cp2009 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v4 [04:17:55] RECOVERY - IPsec on cp2009 is OK: Strongswan OK - 8 ESP OK [04:20:36] 6operations, 10ContentTranslation-Deployments, 10ContentTranslation-cxserver, 10MediaWiki-extensions-ContentTranslation, and 6 others: Standardise CXServer deployment - https://phabricator.wikimedia.org/T101272#1718825 (10santhosh) [04:27:03] PROBLEM - IPsec on cp4012 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v6 [04:28:44] RECOVERY - IPsec on cp4012 is OK: Strongswan OK - 8 ESP OK [04:36:14] PROBLEM - DPKG on stat1003 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:36:35] PROBLEM - Disk space on stat1003 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:36:44] PROBLEM - Check size of conntrack table on stat1003 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:36:44] PROBLEM - dhclient process on stat1003 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:36:45] PROBLEM - RAID on stat1003 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:37:04] PROBLEM - SSH on stat1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [04:37:04] PROBLEM - salt-minion processes on stat1003 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:37:04] PROBLEM - puppet last run on stat1003 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:37:05] PROBLEM - configured eth on stat1003 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [04:37:44] RECOVERY - DPKG on stat1003 is OK: All packages OK [04:38:13] RECOVERY - Disk space on stat1003 is OK: DISK OK [04:38:14] RECOVERY - Check size of conntrack table on stat1003 is OK: OK: nf_conntrack is 0 % full [04:38:14] RECOVERY - dhclient process on stat1003 is OK: PROCS OK: 0 processes with command name dhclient [04:38:23] RECOVERY - RAID on stat1003 is OK: OK: Active: 8, Working: 8, Failed: 0, Spare: 0 [04:38:34] RECOVERY - SSH on stat1003 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.3 (protocol 2.0) [04:38:34] RECOVERY - salt-minion processes on stat1003 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion [04:38:43] RECOVERY - puppet last run on stat1003 is OK: OK: Puppet is currently enabled, last run 10 minutes ago with 0 failures [04:38:44] RECOVERY - configured eth on stat1003 is OK: OK - interfaces up [04:45:54] PROBLEM - IPsec on cp3016 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v6 [04:50:54] RECOVERY - IPsec on cp3016 is OK: Strongswan OK - 8 ESP OK [05:02:24] PROBLEM - Outgoing network saturation on labstore1002 is CRITICAL: CRITICAL: 12.00% of data above the critical threshold [100000000.0] [05:02:33] PROBLEM - IPsec on cp2003 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v6 [05:07:03] PROBLEM - IPsec on cp2015 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v4 [05:08:43] RECOVERY - IPsec on cp2015 is OK: Strongswan OK - 8 ESP OK [05:14:13] PROBLEM - IPsec on cp2021 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v6 [05:22:24] RECOVERY - Outgoing network saturation on labstore1002 is OK: OK: Less than 10.00% above the threshold [75000000.0] [05:22:25] RECOVERY - IPsec on cp2021 is OK: Strongswan OK - 8 ESP OK [05:24:05] RECOVERY - IPsec on cp2003 is OK: Strongswan OK - 8 ESP OK [05:51:44] (03PS1) 10BryanDavis: vagarnt::mediawiki: Ensure clone before adding config [puppet] - 10https://gerrit.wikimedia.org/r/245207 [05:56:15] PROBLEM - IPsec on cp3017 is CRITICAL: Strongswan CRITICAL - ok: 8 connecting: (unnamed) [05:58:03] RECOVERY - IPsec on cp3017 is OK: Strongswan OK - 8 ESP OK [05:59:15] PROBLEM - IPsec on cp4019 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v4 [06:02:45] RECOVERY - IPsec on cp4019 is OK: Strongswan OK - 8 ESP OK [06:03:14] PROBLEM - IPsec on cp3017 is CRITICAL: Strongswan CRITICAL - ok: 8 connecting: (unnamed) [06:11:25] RECOVERY - IPsec on cp3017 is OK: Strongswan OK - 8 ESP OK [06:30:33] PROBLEM - puppet last run on mw1090 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:43] PROBLEM - puppet last run on restbase2006 is CRITICAL: CRITICAL: puppet fail [06:30:53] PROBLEM - puppet last run on db1046 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:03] PROBLEM - puppet last run on mw2021 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:24] PROBLEM - puppet last run on mw1226 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:43] PROBLEM - puppet last run on wtp2017 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:44] PROBLEM - puppet last run on mw2036 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:44] PROBLEM - puppet last run on ms-be1010 is CRITICAL: CRITICAL: Puppet has 2 failures [06:31:54] PROBLEM - puppet last run on mw1135 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:34] PROBLEM - puppet last run on mw1110 is CRITICAL: CRITICAL: Puppet has 2 failures [06:32:35] PROBLEM - puppet last run on terbium is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:43] PROBLEM - puppet last run on mw2126 is CRITICAL: CRITICAL: Puppet has 3 failures [06:32:43] PROBLEM - puppet last run on mw2073 is CRITICAL: CRITICAL: Puppet has 1 failures [06:44:04] PROBLEM - puppet last run on mw2160 is CRITICAL: CRITICAL: puppet fail [06:44:45] PROBLEM - IPsec on cp2021 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v4 [06:46:33] PROBLEM - IPsec on cp4012 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v4 [06:50:35] PROBLEM - IPsec on cp2009 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v4 [06:53:34] PROBLEM - IPsec on cp3018 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v4 [06:54:03] RECOVERY - IPsec on cp2009 is OK: Strongswan OK - 8 ESP OK [06:54:43] RECOVERY - puppet last run on mw1226 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [06:54:45] RECOVERY - IPsec on cp2021 is OK: Strongswan OK - 8 ESP OK [06:55:53] RECOVERY - puppet last run on db1046 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [06:55:54] RECOVERY - puppet last run on terbium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:56:43] RECOVERY - puppet last run on wtp2017 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [06:56:44] RECOVERY - puppet last run on ms-be1010 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [06:56:44] RECOVERY - puppet last run on mw2036 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [06:56:45] RECOVERY - puppet last run on mw1135 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [06:57:14] RECOVERY - puppet last run on mw1090 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:57:24] RECOVERY - puppet last run on restbase2006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:57:34] RECOVERY - puppet last run on mw1110 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [06:57:44] RECOVERY - puppet last run on mw2126 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:57:44] RECOVERY - puppet last run on mw2073 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:57:45] RECOVERY - puppet last run on mw2021 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:58:35] RECOVERY - IPsec on cp3018 is OK: Strongswan OK - 8 ESP OK [06:58:43] PROBLEM - IPsec on cp3015 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v6 [06:59:54] PROBLEM - IPsec on cp2003 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v4 [07:00:25] RECOVERY - IPsec on cp3015 is OK: Strongswan OK - 8 ESP OK [07:08:45] PROBLEM - IPsec on cp3017 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v6 [07:12:34] RECOVERY - puppet last run on mw2160 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [07:13:55] PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL: CRITICAL: Anomaly detected: 10 data above and 0 below the confidence bounds [07:14:23] PROBLEM - puppet last run on mw1227 is CRITICAL: CRITICAL: Puppet has 1 failures [07:16:43] RECOVERY - IPsec on cp2003 is OK: Strongswan OK - 8 ESP OK [07:16:45] RECOVERY - IPsec on cp4012 is OK: Strongswan OK - 8 ESP OK [07:20:14] PROBLEM - Kafka Broker Replica Max Lag on kafka1018 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [5000000.0] [07:22:15] RECOVERY - IPsec on cp3017 is OK: Strongswan OK - 8 ESP OK [07:25:54] PROBLEM - IPsec on cp2009 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v6 [07:26:53] PROBLEM - IPsec on cp4019 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v6 [07:27:16] PROBLEM - IPsec on cp3018 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v6 [07:30:34] RECOVERY - Kafka Broker Replica Max Lag on kafka1018 is OK: OK: Less than 1.00% above the threshold [1000000.0] [07:41:13] RECOVERY - puppet last run on mw1227 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:42:45] RECOVERY - IPsec on cp2009 is OK: Strongswan OK - 8 ESP OK [07:44:05] PROBLEM - IPsec on cp3016 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v6 [07:48:45] PROBLEM - IPsec on cp4011 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v6 [07:49:14] RECOVERY - HTTP error ratio anomaly detection on graphite1001 is OK: OK: No anomaly detected [07:50:45] RECOVERY - IPsec on cp3018 is OK: Strongswan OK - 8 ESP OK [07:53:45] RECOVERY - IPsec on cp4019 is OK: Strongswan OK - 8 ESP OK [07:56:55] PROBLEM - IPsec on cp2021 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v6 [07:58:35] RECOVERY - IPsec on cp2021 is OK: Strongswan OK - 8 ESP OK [07:58:35] PROBLEM - IPsec on cp2003 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v6 [08:02:24] RECOVERY - IPsec on cp4011 is OK: Strongswan OK - 8 ESP OK [08:07:44] RECOVERY - IPsec on cp3016 is OK: Strongswan OK - 8 ESP OK [08:12:44] PROBLEM - IPsec on cp3016 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v4 [08:22:04] RECOVERY - IPsec on cp2003 is OK: Strongswan OK - 8 ESP OK [08:22:43] RECOVERY - IPsec on cp3016 is OK: Strongswan OK - 8 ESP OK [08:39:15] PROBLEM - IPsec on cp4020 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v4 [08:44:15] PROBLEM - IPsec on cp4011 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v4 [08:44:16] !log Zuul CI in trouble. zuul-merger can't not apply patches anymore https://phabricator.wikimedia.org/T115243 [08:44:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [08:47:43] RECOVERY - IPsec on cp4020 is OK: Strongswan OK - 8 ESP OK [08:50:55] RECOVERY - IPsec on cp4011 is OK: Strongswan OK - 8 ESP OK [08:54:38] !log zuul-merger process leaked file descriptors and ended up unable to open any more files. Fixed by restarting the service on gallium. https://phabricator.wikimedia.org/T115243 [08:54:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [09:02:13] PROBLEM - IPsec on cp2015 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v6 [09:08:45] PROBLEM - Host mw1154 is DOWN: PING CRITICAL - Packet loss = 100% [09:14:15] PROBLEM - IPsec on cp2021 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v6 [09:16:24] PROBLEM - IPsec on cp3018 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v6 [09:29:45] PROBLEM - IPsec on cp4012 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v4 [09:30:33] PROBLEM - IPsec on cp2009 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v4 [09:30:45] RECOVERY - IPsec on cp2015 is OK: Strongswan OK - 8 ESP OK [09:31:33] RECOVERY - IPsec on cp4012 is OK: Strongswan OK - 8 ESP OK [09:33:25] RECOVERY - IPsec on cp3018 is OK: Strongswan OK - 8 ESP OK [09:34:54] PROBLEM - IPsec on cp4011 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v4 [09:35:13] PROBLEM - IPsec on cp3015 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v6 [09:37:13] RECOVERY - IPsec on cp2009 is OK: Strongswan OK - 8 ESP OK [09:39:58] (03CR) 10Phuedx: [C: 032] Don't require QuickSurveys to use HTTPS links in labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/245188 (https://phabricator.wikimedia.org/T114485) (owner: 10Alex Monk) [09:40:29] (03Merged) 10jenkins-bot: Don't require QuickSurveys to use HTTPS links in labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/245188 (https://phabricator.wikimedia.org/T114485) (owner: 10Alex Monk) [09:42:45] PROBLEM - IPsec on cp2003 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v4 [09:43:14] PROBLEM - IPsec on cp4012 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v6 [09:47:45] RECOVERY - IPsec on cp2021 is OK: Strongswan OK - 8 ESP OK [09:47:45] RECOVERY - IPsec on cp2003 is OK: Strongswan OK - 8 ESP OK [09:54:54] RECOVERY - IPsec on cp4012 is OK: Strongswan OK - 8 ESP OK [09:56:54] PROBLEM - Unmerged changes on repository mediawiki_config on tin is CRITICAL: There are 2 unmerged changes in mediawiki_config (dir /srv/mediawiki-staging/). [09:58:33] PROBLEM - IPsec on cp3017 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v6 [10:00:14] PROBLEM - IPsec on cp3018 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v6 [10:01:55] RECOVERY - IPsec on cp3018 is OK: Strongswan OK - 8 ESP OK [10:03:24] RECOVERY - IPsec on cp4011 is OK: Strongswan OK - 8 ESP OK [10:08:24] PROBLEM - IPsec on cp4011 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v6 [10:09:15] PROBLEM - IPsec on cp2015 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v4 [10:11:55] RECOVERY - IPsec on cp3015 is OK: Strongswan OK - 8 ESP OK [10:15:55] RECOVERY - IPsec on cp2015 is OK: Strongswan OK - 8 ESP OK [10:19:04] PROBLEM - IPsec on cp2009 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v6 [10:20:04] PROBLEM - IPsec on cp4019 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v6 [10:23:24] RECOVERY - IPsec on cp4011 is OK: Strongswan OK - 8 ESP OK [10:29:04] RECOVERY - IPsec on cp2009 is OK: Strongswan OK - 8 ESP OK [10:34:48] <_joe_> uhm what happened here? [10:35:25] <_joe_> it looks like cp1059 went completely down in the end? [10:40:33] PROBLEM - IPsec on cp3016 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v6 [10:40:55] (03CR) 10Hashar: [C: 031] contint: stop gerrit replication to gallium [puppet] - 10https://gerrit.wikimedia.org/r/244498 (https://phabricator.wikimedia.org/T86661) (owner: 10Hashar) [10:46:14] PROBLEM - IPsec on cp2015 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v6 [10:46:25] PROBLEM - IPsec on cp2003 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v6 [10:47:15] PROBLEM - IPsec on cp3015 is CRITICAL: Strongswan CRITICAL - ok: 8 connecting: (unnamed) [10:50:23] RECOVERY - IPsec on cp4019 is OK: Strongswan OK - 8 ESP OK [10:50:35] RECOVERY - IPsec on cp3015 is OK: Strongswan OK - 8 ESP OK [10:55:23] PROBLEM - IPsec on cp4019 is CRITICAL: Strongswan CRITICAL - ok: 6 connecting: cp1059_v4, cp1059_v6 [10:55:34] PROBLEM - IPsec on cp3015 is CRITICAL: Strongswan CRITICAL - ok: 8 connecting: (unnamed) [10:57:24] RECOVERY - IPsec on cp3017 is OK: Strongswan OK - 8 ESP OK [11:11:44] PROBLEM - IPsec on cp2021 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v6 [11:12:14] PROBLEM - IPsec on cp4012 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v6 [11:12:35] PROBLEM - IPsec on cp3017 is CRITICAL: Strongswan CRITICAL - ok: 8 connecting: (unnamed) [11:17:53] PROBLEM - IPsec on cp2009 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v6 [11:18:55] PROBLEM - IPsec on cp4011 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v6 [11:20:35] PROBLEM - IPsec on cp4020 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v4 [11:21:13] akosiaris, morning, can we deploy tileratorui? [11:24:23] PROBLEM - puppet last run on cp3031 is CRITICAL: CRITICAL: puppet fail [11:25:09] _joe_, cp1059 has had issues for a while already, see https://phabricator.wikimedia.org/T114870 [11:25:39] yurik: ops are not around this week so there is no deployments. [11:25:53] hashar, gone for the whole week? wow ) [11:26:23] yurik, see postings on engineering@ from Wednesday. [11:26:27] yurik: see mark mail on engineering list [11:26:37] * yurik looks [11:26:47] if everyone is out, why is everyone here? :D [11:26:53] * andre__ recognizes a general "please read mail" attitude. :P [11:26:59] yurik: so you will want to schedule it somewhere after the middle of next week since most will recover from jet lag on the first days of next week [11:27:00] just lurking! :) [11:27:37] yurik: at least you can deploy on beta-cluster :-} [11:27:42] no deployments? [11:27:46] That was not how I read the email. [11:28:22] "we will make sure to respond to site incidents and other emergencies, we will not be attending to any other tasks." [11:28:56] yeah, i think deployments are as usual starting tomorrow (columbus day for the states) [11:29:00] Trying to logon to Phab and getting errors, was like that last night. Any suggestions, is it a known issue ? [11:29:46] yurik: nop [11:30:04] As far as I'm aware there are deployments happening today. [11:30:11] that's what i said :) [11:30:22] yurik: for ops, there is no deployment this week. Only response to issues/outages, unless you agreed something with them :-D [11:30:51] agreed something with ops?! me??!? that's nearly impossible :))))))) [11:31:15] hashar, i meant general depl, not ops. E.g. swat is probably on for tomorrow [11:31:27] Krenair: yeah the usual train swat etc are happening. But quoting the calendar "please don't do anything stupid" [11:31:55] PROBLEM - nutcracker port on silver is CRITICAL: CRITICAL - Socket timeout after 2 seconds [11:31:57] * yurik thinks "don't be stupid" is a mantra for every day, not just when ops are out [11:33:43] RECOVERY - IPsec on cp2021 is OK: Strongswan OK - 8 ESP OK [11:34:16] yurik: indeed ;-} So in short, don't expect much replies from ops this week [11:34:26] food time! [11:35:03] * yurik thinks ops will get so bored with talking to each other, they will jump back here and do all sorts of pending mini tasks )) [11:37:03] RECOVERY - nutcracker port on silver is OK: TCP OK - 0.000 second response time on port 11212 [11:40:54] RECOVERY - IPsec on cp4019 is OK: Strongswan OK - 8 ESP OK [11:41:04] getting Request: POST http://phabricator.wikimedia.org/auth/login/ldap:self/, from 10.64.0.107 via cp1070 cp1070 ([10.64.0.107]:80), Varnish XID 1614268393 when trying to logon to Phab [11:45:34] Blobbity, well, I can log in using the normal form... [11:46:13] I can't even get the login page to load at the moment [11:46:15] Blobbity, and via MediaWiki.org OAuth [11:49:43] OAuth not coming up as an option, and still giving me the same Varnish errors. I'll try again later when I've got more time. [11:50:09] On the login page you should have the LDAP login form and the MediaWiki OAuth button at the bottom [11:51:15] RECOVERY - puppet last run on cp3031 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:54:24] RECOVERY - IPsec on cp4020 is OK: Strongswan OK - 8 ESP OK [11:54:46] (03PS3) 10Alex Monk: Update DB size lists [mediawiki-config] - 10https://gerrit.wikimedia.org/r/243517 (https://phabricator.wikimedia.org/T114613) [11:59:24] RECOVERY - IPsec on cp4011 is OK: Strongswan OK - 8 ESP OK [11:59:55] RECOVERY - IPsec on cp2009 is OK: Strongswan OK - 8 ESP OK [12:00:41] (03PS1) 10Dereckson: Throttle rule for Ada Lovelace Day editathon 2015 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/245472 (https://phabricator.wikimedia.org/T115245) [12:01:34] RECOVERY - IPsec on cp3016 is OK: Strongswan OK - 8 ESP OK [12:01:34] RECOVERY - IPsec on cp3015 is OK: Strongswan OK - 8 ESP OK [12:02:23] RECOVERY - IPsec on cp2003 is OK: Strongswan OK - 8 ESP OK [12:03:14] RECOVERY - IPsec on cp3017 is OK: Strongswan OK - 8 ESP OK [12:10:47] !log deployed kartotherian & tilerator to maps-test200{1-4} [12:10:51] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [12:10:59] (03CR) 10Alex Monk: [C: 031] Throttle rule for Ada Lovelace Day editathon 2015 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/245472 (https://phabricator.wikimedia.org/T115245) (owner: 10Dereckson) [12:12:42] (03CR) 10Alex Monk: [C: 031] Naming standardization from 'flooder' to 'flood' [mediawiki-config] - 10https://gerrit.wikimedia.org/r/245194 (https://phabricator.wikimedia.org/T115200) (owner: 10MarcoAurelio) [12:15:24] RECOVERY - IPsec on cp2015 is OK: Strongswan OK - 8 ESP OK [12:15:44] PROBLEM - IPsec on cp2003 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v4 [12:19:44] PROBLEM - IPsec on cp4020 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v6 [12:22:13] PROBLEM - IPsec on cp2015 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v6 [12:28:54] RECOVERY - IPsec on cp2015 is OK: Strongswan OK - 8 ESP OK [12:31:54] PROBLEM - IPsec on cp3018 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v6 [12:40:03] PROBLEM - IPsec on cp4011 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v4 [12:42:04] RECOVERY - Unmerged changes on repository mediawiki_config on tin is OK: No changes to merge. [12:45:23] PROBLEM - IPsec on cp3015 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v6 [12:47:25] PROBLEM - IPsec on cp2015 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v4 [12:50:34] (03CR) 10Luke081515: [C: 031] Throttle rule for Ada Lovelace Day editathon 2015 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/245472 (https://phabricator.wikimedia.org/T115245) (owner: 10Dereckson) [13:00:19] (03PS4) 10Zfilipin: rubocop: enforcing comma after the last element of a multiline list [puppet] - 10https://gerrit.wikimedia.org/r/238779 (https://phabricator.wikimedia.org/T112651) [13:03:53] PROBLEM - IPsec on cp3016 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v6 [13:05:25] RECOVERY - IPsec on cp3016 is OK: Strongswan OK - 8 ESP OK [13:07:18] (03PS5) 10Zfilipin: rubocop: enforcing comma after the last element of a multiline list [puppet] - 10https://gerrit.wikimedia.org/r/238779 (https://phabricator.wikimedia.org/T112651) [13:08:57] <_joe_> andre__: I kinda opened that ticket I guess :P [13:13:00] (03PS1) 10BBlack: remove cp1059 from ipsec hostlists - T114870 [puppet] - 10https://gerrit.wikimedia.org/r/245477 [13:13:25] (03CR) 10BBlack: [C: 032 V: 032] remove cp1059 from ipsec hostlists - T114870 [puppet] - 10https://gerrit.wikimedia.org/r/245477 (owner: 10BBlack) [13:16:13] RECOVERY - IPsec on cp2003 is OK: Strongswan OK - 6 ESP OK [13:17:50] (03PS3) 10Revi: Modify timezone for cswiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/244649 (https://phabricator.wikimedia.org/T115048) [13:18:44] RECOVERY - IPsec on cp3018 is OK: Strongswan OK - 6 ESP OK [13:18:53] PROBLEM - IPsec on cp3017 is CRITICAL: Strongswan CRITICAL - ok: 7 connecting: cp1059_v6 [13:20:33] RECOVERY - IPsec on cp3017 is OK: Strongswan OK - 6 ESP OK [13:20:56] RECOVERY - IPsec on cp2015 is OK: Strongswan OK - 6 ESP OK [13:23:34] RECOVERY - IPsec on cp4011 is OK: Strongswan OK - 6 ESP OK [13:25:35] RECOVERY - IPsec on cp3015 is OK: Strongswan OK - 6 ESP OK [13:26:55] RECOVERY - IPsec on cp4012 is OK: Strongswan OK - 6 ESP OK [13:26:55] RECOVERY - IPsec on cp4020 is OK: Strongswan OK - 6 ESP OK [13:31:36] (03PS6) 10Zfilipin: rubocop: Ignore Style/TrailingComma offense [puppet] - 10https://gerrit.wikimedia.org/r/238779 (https://phabricator.wikimedia.org/T112651) [13:50:29] (03PS3) 10Thiemo Mättig (WMDE): Add pageImagesPropertyIds configuration for Wikibase servers [mediawiki-config] - 10https://gerrit.wikimedia.org/r/244165 (https://phabricator.wikimedia.org/T112865) [13:56:35] PROBLEM - nutcracker port on silver is CRITICAL: CRITICAL - Socket timeout after 2 seconds [14:00:05] RECOVERY - nutcracker port on silver is OK: TCP OK - 0.000 second response time on port 11212 [14:02:13] yurik: we are in an offsite, I 'll ping you when I got some availability [14:02:25] akosiaris, sure, thx )) [14:02:27] enjoy! [14:02:39] where is the offsite? [14:05:03] oh, hi yurik. [14:05:06] mutante said I have to ask you, so... How can I use maps.wm.o in wikivoyage/incubator? [14:06:57] yurik: PR [14:08:05] revi: the map gadgets typically have a selector for the map layer (mapnik/OSM/labs), and they should get a maps.wm.o added (or rather, they should replace labs I think) [14:08:32] ah, that way [14:08:37] revi: actually, WV seems to link to https://tools.wmflabs.org/wikivoyage/w/poimap2.php?lat=38.13&lon=125.65&zoom=13&layer=M&lang=en&name=Hwanghae [14:08:51] so that tool should be changed to refer to the right map source [14:09:01] I'm planning to use it on incubator so :-p [14:09:18] (ko.wikivoyage.org is poor incubator project :-p) [14:09:54] akosiaris: so... how's the weather? :P [14:10:15] revi, i think it has been enabled already [14:10:24] yes [14:10:29] I was asking how to use it [14:11:07] I was trying to use it from scratch and there was no help for such users [14:11:16] revi, the problem is that we don't have a client-side component (yet). so you have to use one of the wmflabs instances for that. E.g. the one that the prod wikivoyage uses [14:11:17] JohnFLewis: hot and humid ;-) [14:11:38] hmm, ok [14:11:47] revi: I'm also not sure *what* you want. en.wv just has a template that create a link to tools [14:11:51] at this point we only provide the server side -- the actual tile images [14:11:53] you can just copy that template [14:11:56] akosiaris: bet you wished you went with Iceland now ;) [14:12:17] (at least, it looks like that to me) [14:12:29] JohnFLewis: I was wishing that, I no longer do though [14:13:35] hm, no, it seems to be added by javascript later on. [14:13:37] akosiaris: Iceland would have been more fun - no Icelandic opsen and different weather pattern = super fun and great work conditions! [14:13:56] revi, so i think at this point you have to modify common.js/css in the incubator to use the tools.wmflabs/poimap2 [14:14:05] JohnFLewis: and let's not forget CCP. we could have payed them a visit [14:14:09] hmm [14:14:17] I need test-sysop there then [14:14:53] anyway, thanks [14:14:55] akosiaris: true :) [14:15:39] revi, i already started implementing Kartographer extension - https://git.wikimedia.org/log/mediawiki%2Fextensions%2FKartographer/HEAD [14:15:43] revi: https://en.wikivoyage.org/wiki/Template:Geo [14:15:46] so at some point i hope it will be easier [14:16:19] valhallasw`cloud, i am not sure that template will work without some hacks in the common.js [14:16:26] wikitech is loosing my session every few minutes :( [14:16:46] yurik: there's nothing in en.wv's common.js or vector.js :/ [14:16:57] gadgets? [14:17:04] just for inline maps as far as I cansee [14:17:06] valhallasw`cloud, weird, are we allowing arbitrary iframes? [14:17:23] it doesn't give me an iframe, just a popup to tools.wm.o [14:17:50] eh, normal link if I don't middle-click [14:18:03] (03PS4) 10Thiemo Mättig (WMDE): Add pageImagesPropertyIds configuration for Wikibase servers [mediawiki-config] - 10https://gerrit.wikimedia.org/r/244165 (https://phabricator.wikimedia.org/T112865) [14:18:42] jzerebecki: ok, we will be having a quick look [14:18:43] which is why I asked what revi wanted to have ;-) [14:18:59] thx [14:19:03] (03PS5) 10Thiemo Mättig (WMDE): Add pageImagesPropertyIds configuration for Wikibase servers [mediawiki-config] - 10https://gerrit.wikimedia.org/r/244165 (https://phabricator.wikimedia.org/T75482) [14:19:56] valhallasw`cloud, https://en.wikivoyage.org/wiki/Salzburg#Get_around is an iframe [14:20:12] yes, that's a gadget [14:20:30] the template is for the link on the top right [14:21:00] jzerebecki: can you tell me a bit more about what’s happening? [14:21:00] valhallasw`cloud, gotcha. I was thinking of the embedded scenario [14:22:16] andrewbogott: only that I log in and after a few seconds / minutes i'm not logged in anymore, as if the server forgot my session. [14:22:45] jzerebecki: ok. I apologize in advance for asking this question: [14:22:57] Are you clicking the ‘Keep me logged in’ box when you log in? [14:23:21] andrewbogott: no :) [14:23:51] jzerebecki: ok! I can’t guarantee that the behavior without that checkbox is sensible, but ticking that box will probably help :) [14:24:05] I think otherwise it only lasts for the lifespan of that browser tab. [14:24:41] valhallasw`cloud, i need to have a discussion on what to include in the maps extenson. Are you interested in participating? [14:24:55] yurik: I know very little about the subject [14:25:06] ok [14:25:12] yurik: but I would mostly take a look around different wikis to see how they use maps? [14:26:02] e.g. nlwiki uses a page-wide embedded iframe at the top (expanded when you click a link), while enwiki has a div-popup, enwv has a link plus special embeds [14:27:02] andrewbogott: it should last for the browser session though [14:27:11] wikitech has often had this issue for people [14:27:19] (jan isnt the first person i've heard complain) [14:27:27] not the tab life. [14:28:13] PROBLEM - puppet last run on mw1085 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [14:28:21] andrewbogott: I don't think it is deleting the session via javascript on onunload or something like that. I didn't close the browser in between. i lost the session between login and then clicking edit. then again I lost it between editing and saving. I'm on a static ip. [14:28:44] !log rebooting mw1154 [14:28:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [14:29:57] jzerebecki: usually I don't have that problem with wikitech. I didn't change my browser settings, version nor addons. [14:30:54] RECOVERY - Host mw1154 is UP: PING OK - Packet loss = 0%, RTA = 0.53 ms [14:31:08] andrewbogott: ^^. anyway feel free to not look into it until I see the problem again. [14:31:40] jzerebecki: ok. It’s possible that the behavior without checking that box is totally broken and just no one one ever does that :) [14:32:03] Nah, I never check that box [14:32:13] But we had these exact problems on wikitech before [14:32:21] jsut can't remember what we did to fix them (if anything) [14:32:36] Could be related to the flaky nutcracker stuff [14:32:46] (if it uses production's session redises) [14:32:50] hoo: yeah, could be, but I’d expect it to be bad for everyone [14:33:08] If it’s happening to >1 person then I can purge things and force all users to re-log in. [14:33:48] Kicking nutcracker is harmless all the time [14:33:53] it's just a proxy [14:34:56] valhallasw`cloud, thanks for the NL info, didn't know they used cross the top map. do you know who's maintaining it? [14:35:08] yurik: no-one, probably ;-) [14:35:28] i mean who could revise it to possibly use wmf maps? [14:36:08] I think it's https://nl.wikipedia.org/wiki/MediaWiki:Gadget-OpenStreetMapFrame.js , so Krinkle. [14:36:33] oh, hmm, its actually gives wmf maps as one of the options [14:36:34] via https://meta.wikimedia.org/wiki/User:Krinkle/OpenStreetMapFrame.js [14:36:35] (03PS6) 10JanZerebecki: Add GeoData and PageImages configuration for Wikibase repo wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/244165 (https://phabricator.wikimedia.org/T75482) (owner: 10Thiemo Mättig (WMDE)) [14:37:28] (03CR) 10JanZerebecki: [C: 031] Add GeoData and PageImages configuration for Wikibase repo wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/244165 (https://phabricator.wikimedia.org/T75482) (owner: 10Thiemo Mättig (WMDE)) [14:39:04] ugh, this is really one of those rabbit hole situations. Gadget on nlwiki linking to JS on meta which adds an iframe hosted on tools which loads map tiles from various sources [14:39:37] jzerebecki: That sometimes happen to me when I switch browsers/computers. [14:44:01] it is supposed to happen when you switch browser profiles that don't sync cookies. but not within the same minute of logging in when you didn't even close the tab. anyway lets see if it happens again. for now the same session is still working. [14:45:27] (03CR) 10Luke081515: Add patrol, autopatrol, flood group to itwikiversity (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/244896 (https://phabricator.wikimedia.org/T114930) (owner: 10Gerrit Patch Uploader) [14:54:12] !log closing unused cirrus indices in eqiad (T112863) [14:54:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [14:55:06] Hi. [14:57:32] I'm making a list minute add of https://gerrit.wikimedia.org/r/#/c/245472 as it's a throttle rule for tomorrow. [14:57:54] Oh, it's already in the series of Krenair patches, fine. [15:00:04] anomie ostriches thcipriani marktraceur Krenair: Respected human, time to deploy Morning SWAT(Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20151012T1500). Please do the needful. [15:00:04] revi yurik jdlrobson jhobs Krenair jzerebecki: A patch you scheduled for Morning SWAT(Max 8 patches) is about to be deployed. Please be available during the process. [15:00:09] pong [15:03:23] hmm no SWAT? I want mine to be done soon so I can go to bed [15:03:29] that is a lot of changes...ok, I can SWAT. revi you're up. [15:03:30] midnight here [15:03:36] good [15:04:25] (03CR) 10Thcipriani: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/244649 (https://phabricator.wikimedia.org/T115048) (owner: 10Revi) [15:04:43] o/ [15:04:50] (03Merged) 10jenkins-bot: Modify timezone for cswiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/244649 (https://phabricator.wikimedia.org/T115048) (owner: 10Revi) [15:07:18] Wiki default now: Europe/Prague. Good [15:07:30] !log thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Modify timezone for cswiktionary [[gerrit:244649]] (duration: 01m 12s) [15:07:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [15:08:07] revi: should be sync'd. Looks like you checked it already. Everything looking good? [15:08:12] yeah [15:08:19] awesome. Thank you! [15:08:26] Thanks! Goodnight! [15:08:49] yurik: jdlrobson jhobs ping for SWAT. [15:09:19] jzerebecki: let's get yours done. [15:09:25] k [15:09:54] andre__: ich finde die doku nicht. kannst du bitte https://phabricator.wikimedia.org/T115263 in --> https://phabricator.wikimedia.org/T115261 mergen. danke. [15:10:08] jzerebecki: does order matter on these? [15:10:13] no [15:10:28] kk [15:11:18] (03CR) 10Thcipriani: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/244591 (https://phabricator.wikimedia.org/T114869) (owner: 10Aude) [15:11:27] Steinsplitter: merge dups is just go to T115261 and merge duplicate in T115263. [15:11:33] not other way around [15:11:40] (03Merged) 10jenkins-bot: Explicitly set wmgMFNearby = false for wikidata [mediawiki-config] - 10https://gerrit.wikimedia.org/r/244591 (https://phabricator.wikimedia.org/T114869) (owner: 10Aude) [15:11:43] ah, ok. thx revi. [15:11:59] you have to go to "older task" and merge "newer task" [15:12:07] it's bit confusing at first time :-p [15:14:30] !log thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Explicitly set wmgMFNearby = false for wikidata [[gerrit:244591]] (duration: 01m 14s) [15:14:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [15:14:38] there is one server that is evidently very slow today :\ [15:14:43] ^ jzerebecki sync'd [15:16:01] (03CR) 10Thcipriani: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/244165 (https://phabricator.wikimedia.org/T75482) (owner: 10Thiemo Mättig (WMDE)) [15:16:16] 6operations, 10OTRS: Apply security patch to OTRS (Scheduler Process ID File Access vulnerability) - https://phabricator.wikimedia.org/T114132#1719390 (10Aklapper) @Jgreen / #Operations: is there a vague timeframe / ETA? [15:16:23] (03Merged) 10jenkins-bot: Add GeoData and PageImages configuration for Wikibase repo wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/244165 (https://phabricator.wikimedia.org/T75482) (owner: 10Thiemo Mättig (WMDE)) [15:17:14] 6operations, 6Phabricator, 7Database, 5Patch-For-Review, 7WorkType-Maintenance: Phabricator creates MySQL connection spikes: Attempt to connect to phuser@m3-master.eqiad.wmnet failed with error #1040: Too many connections. - https://phabricator.wikimedia.org/T109279#1719398 (10Aklapper) Has anyone still... [15:17:48] 6operations: cp2017 is down - https://phabricator.wikimedia.org/T114022#1719399 (10Aklapper) >>! In T114022#1682734, @BBlack wrote: > Leaving it up and depooled for now to see if it stays stable without traffic or not... @BBlack: Any updates? [15:19:12] thcipriani: looks good. thx. [15:19:45] !log thcipriani@tin Synchronized wmf-config/Wikibase-production.php: SWAT: Add GeoData and PageImages configuration for Wikibase repo wikis [[gerrit:244165]] (duration: 01m 13s) [15:19:49] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [15:20:12] jzerebecki: awesome, thanks for checking! ^ Wikibase-prod config sync'd now, too, FYI. [15:20:34] thcipriani, hey [15:20:55] Krenair: howdy, just getting ready to ping you for your stuff :) [15:21:08] ok [15:22:12] (03CR) 10Thcipriani: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/242096 (https://phabricator.wikimedia.org/T114002) (owner: 10Siebrand) [15:22:35] (03Merged) 10jenkins-bot: Rename Azerbaijani Wikisource project and namespaces [mediawiki-config] - 10https://gerrit.wikimedia.org/r/242096 (https://phabricator.wikimedia.org/T114002) (owner: 10Siebrand) [15:23:30] thcipriani, jzerebecki: umm... we appear to have more than 8 patches listed? [15:23:46] I only had 2 [15:24:10] AFAIK it's not a per-person limit [15:24:40] yeah, it's supposed to be a per-SWAT limit. Since most of these are small config changes I was just going to roll with it this time. [15:24:53] oops [15:26:16] !log thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Rename Azerbaijani Wikisource project and namespaces [[gerrit:242096]] (duration: 01m 13s) [15:26:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [15:26:27] ^ Krenair check please [15:26:28] We can also deploy the stuff ourselves [15:26:36] If there's still time left at the end of the SWAT window [15:27:40] 6operations, 6Performance-Team, 10Traffic, 7Performance: enwiki Main_Page timeouts - https://phabricator.wikimedia.org/T104225#1719429 (10Aklapper) @ori: Any news? Or should this have lower priority? [15:27:48] hoo: our patches are already done [15:27:57] oh, nice [15:28:16] thcipriani, looks good so far [15:28:39] (03CR) 10Thcipriani: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/244435 (owner: 10Glaisher) [15:28:42] (03CR) 10jenkins-bot: [V: 04-1] Remove duplicate entries from commonsuploads.dblist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/244435 (owner: 10Glaisher) [15:29:43] thcipriani, needs rebasing, will fix and we can come back to it later [15:29:47] kk [15:29:56] Steinsplitter: click T115261, click "Merge Into", add T115263 in that dialog [15:30:44] (03CR) 10Thcipriani: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/241079 (https://phabricator.wikimedia.org/T67306) (owner: 10Florianschmidtwelzow) [15:31:07] (03Merged) 10jenkins-bot: Use new page name for wmf release notes [mediawiki-config] - 10https://gerrit.wikimedia.org/r/241079 (https://phabricator.wikimedia.org/T67306) (owner: 10Florianschmidtwelzow) [15:33:55] !log thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Use new page name for wmf release notes [[gerrit:241079]] (duration: 01m 14s) [15:34:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [15:34:09] ^ Krenair sync'd [15:34:24] thcipriani, i'm around [15:34:27] (03PS3) 10Alex Monk: Remove duplicate entries from commonsuploads.dblist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/244435 (owner: 10Glaisher) [15:34:41] thcipriani, sorry didn't see your post earlier [15:34:47] for some reason I could rebase that one locally but not via gerrit [15:35:10] yurik: no problem, let me finish up this list of quick changes and we'll come back around. [15:35:15] Krenair: weird. [15:35:17] ok, thx [15:35:30] (03PS1) 10Ori.livneh: reprepro: import from grafana apt [puppet] - 10https://gerrit.wikimedia.org/r/245490 [15:35:54] (03CR) 10Thcipriani: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/244435 (owner: 10Glaisher) [15:36:04] (03Merged) 10jenkins-bot: Remove duplicate entries from commonsuploads.dblist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/244435 (owner: 10Glaisher) [15:38:33] (03PS2) 10Ori.livneh: reprepro: import from grafana apt [puppet] - 10https://gerrit.wikimedia.org/r/245490 [15:39:41] !log thcipriani@tin Synchronized dblists/commonsuploads.dblist: SWAT: Remove duplicate entries from commsuploads.dblist [[gerrit:244435]] (duration: 01m 12s) [15:39:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [15:40:08] Krenair: ^ sync'd. Timezone for cswiki went out first in SWAT, FYI. [15:40:45] (03CR) 10Thcipriani: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/244953 (https://phabricator.wikimedia.org/T62956) (owner: 10MarcoAurelio) [15:41:13] (03Merged) 10jenkins-bot: Enable Extension:ShortURL on bnwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/244953 (https://phabricator.wikimedia.org/T62956) (owner: 10MarcoAurelio) [15:42:11] thcipriani, did you make the tables for shorturl? [15:42:52] Krenair: no, looking now. [15:45:59] (03PS1) 10Ori.livneh: misc varnish: proxy grafana-testing.wm.o to krypton as well [puppet] - 10https://gerrit.wikimedia.org/r/245494 [15:46:49] oh thanks Krenair and thcipriani [15:48:06] Krenair: : is it just php /srv/mediawiki-staging/php-1.27.0-wmf.2/extensions/ShortUrl/populateShortUrlTable.php --wiki bnwiki am I missing a script? [15:48:27] did you make the table? [15:48:58] Krenair: no, I'm not sure which script to run for that [15:50:28] (03CR) 10Faidon Liambotis: [C: 04-1] reprepro: import from grafana apt (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/245490 (owner: 10Ori.livneh) [15:50:35] I ran "mwscript sql.php bnwiki /srv/mediawiki-staging/php-1.27.0-wmf.2/extensions/ShortUrl/schemas/shorturls.sql" [15:51:48] Krenair: ok, table created then? Thank you. [15:51:57] yes [15:54:29] (03CR) 10Ori.livneh: reprepro: import from grafana apt (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/245490 (owner: 10Ori.livneh) [15:55:15] (03PS3) 10Ori.livneh: reprepro: import from grafana apt [puppet] - 10https://gerrit.wikimedia.org/r/245490 [15:55:23] !log thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Extension:ShortURL on bnwiki [[gerrit:244953]] (duration: 01m 14s) [15:55:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [15:55:34] ^ Krenair check please [15:56:01] thcipriani, looks good [15:56:16] cool. Thanks! [15:56:57] (03CR) 10Thcipriani: [C: 032] "SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/245472 (https://phabricator.wikimedia.org/T115245) (owner: 10Dereckson) [15:57:21] (03Merged) 10jenkins-bot: Throttle rule for Ada Lovelace Day editathon 2015 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/245472 (https://phabricator.wikimedia.org/T115245) (owner: 10Dereckson) [16:00:41] !log thcipriani@tin Synchronized wmf-config/throttle.php: SWAT: Throttle rule for Ada Lovelace Day editathon 2015 [[gerrit:245472]] (duration: 01m 13s) [16:00:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [16:00:54] ^ Krenair sync'd [16:01:10] yurik: still around? [16:01:49] ty [16:02:13] Krenair: thank you for your help. Appreciated. [16:02:38] (03PS1) 10Ori.livneh: Rename php_ini() to ini_file() [puppet] - 10https://gerrit.wikimedia.org/r/245496 [16:02:51] thcipriani, yep [16:03:22] yurik: ok, doesn't look like there's another deployment for a bit, so SWAT can run a touch long. You ready? [16:03:31] yep [16:03:45] jhobs, ^ [16:04:06] jdlrobson, ^ [16:13:20] 6operations, 10Analytics, 10Deployment-Systems, 6Services, 3Scap3: Use Scap3 for deploying AQS - https://phabricator.wikimedia.org/T114999#1719499 (10mmodell) [16:13:34] !log thcipriani@tin Synchronized php-1.27.0-wmf.2/extensions/ZeroBanner: SWAT: Defer loading of ZeroOverlay until needed [[gerrit:244737]] (duration: 01m 13s) [16:13:40] ^ yurik check please [16:13:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [16:16:15] * yurik looks [16:19:04] PROBLEM - puppet last run on ganeti2004 is CRITICAL: CRITICAL: puppet fail [16:32:04] PROBLEM - puppet last run on mw1079 is CRITICAL: CRITICAL: puppet fail [16:32:13] PROBLEM - puppet last run on labmon1001 is CRITICAL: CRITICAL: puppet fail [16:32:14] PROBLEM - puppet last run on mw1133 is CRITICAL: CRITICAL: puppet fail [16:32:14] PROBLEM - puppet last run on ms-be1008 is CRITICAL: CRITICAL: puppet fail [16:32:23] PROBLEM - puppet last run on mw1246 is CRITICAL: CRITICAL: puppet fail [16:32:24] PROBLEM - puppet last run on cp4013 is CRITICAL: CRITICAL: puppet fail [16:32:24] PROBLEM - puppet last run on cp1048 is CRITICAL: CRITICAL: puppet fail [16:32:33] PROBLEM - puppet last run on db1039 is CRITICAL: CRITICAL: puppet fail [16:32:35] PROBLEM - puppet last run on mc2006 is CRITICAL: CRITICAL: puppet fail [16:32:45] PROBLEM - puppet last run on mw1259 is CRITICAL: CRITICAL: puppet fail [16:32:46] PROBLEM - puppet last run on mw1014 is CRITICAL: CRITICAL: puppet fail [16:32:46] PROBLEM - puppet last run on restbase1008 is CRITICAL: CRITICAL: puppet fail [16:32:47] PROBLEM - puppet last run on db2049 is CRITICAL: CRITICAL: puppet fail [16:32:47] PROBLEM - puppet last run on cp1057 is CRITICAL: CRITICAL: puppet fail [16:32:47] PROBLEM - puppet last run on mw1035 is CRITICAL: CRITICAL: puppet fail [16:32:48] PROBLEM - puppet last run on mw2191 is CRITICAL: CRITICAL: puppet fail [16:32:53] PROBLEM - puppet last run on osmium is CRITICAL: CRITICAL: puppet fail [16:32:54] PROBLEM - puppet last run on analytics1045 is CRITICAL: CRITICAL: puppet fail [16:32:54] PROBLEM - puppet last run on mc1014 is CRITICAL: CRITICAL: puppet fail [16:32:54] PROBLEM - puppet last run on mw2125 is CRITICAL: CRITICAL: puppet fail [16:33:03] PROBLEM - puppet last run on rdb1001 is CRITICAL: CRITICAL: puppet fail [16:33:04] PROBLEM - puppet last run on mw1036 is CRITICAL: CRITICAL: puppet fail [16:33:04] PROBLEM - puppet last run on mw1013 is CRITICAL: CRITICAL: puppet fail [16:33:04] PROBLEM - puppet last run on mc2013 is CRITICAL: CRITICAL: puppet fail [16:33:05] PROBLEM - puppet last run on mw2051 is CRITICAL: CRITICAL: puppet fail [16:33:05] PROBLEM - puppet last run on wtp2003 is CRITICAL: CRITICAL: puppet fail [16:33:05] PROBLEM - puppet last run on mw2035 is CRITICAL: CRITICAL: puppet fail [16:33:05] PROBLEM - puppet last run on restbase1002 is CRITICAL: CRITICAL: puppet fail [16:33:13] PROBLEM - puppet last run on mw1156 is CRITICAL: CRITICAL: puppet fail [16:33:13] PROBLEM - puppet last run on mw1034 is CRITICAL: CRITICAL: puppet fail [16:33:14] PROBLEM - puppet last run on mw2177 is CRITICAL: CRITICAL: puppet fail [16:33:14] PROBLEM - puppet last run on db1060 is CRITICAL: CRITICAL: puppet fail [16:33:15] PROBLEM - puppet last run on mw2181 is CRITICAL: CRITICAL: puppet fail [16:33:15] PROBLEM - puppet last run on es2007 is CRITICAL: CRITICAL: puppet fail [16:33:24] PROBLEM - puppet last run on hooft is CRITICAL: CRITICAL: puppet fail [16:33:24] PROBLEM - puppet last run on mw1151 is CRITICAL: CRITICAL: puppet fail [16:33:25] PROBLEM - puppet last run on analytics1034 is CRITICAL: CRITICAL: puppet fail [16:33:25] PROBLEM - puppet last run on mw1257 is CRITICAL: CRITICAL: puppet fail [16:33:33] PROBLEM - puppet last run on db1041 is CRITICAL: CRITICAL: puppet fail [16:33:33] PROBLEM - puppet last run on mw1096 is CRITICAL: CRITICAL: puppet fail [16:33:34] PROBLEM - puppet last run on praseodymium is CRITICAL: CRITICAL: puppet fail [16:33:34] PROBLEM - puppet last run on mw1258 is CRITICAL: CRITICAL: puppet fail [16:33:34] PROBLEM - puppet last run on analytics1037 is CRITICAL: CRITICAL: puppet fail [16:33:34] PROBLEM - puppet last run on mw1183 is CRITICAL: CRITICAL: puppet fail [16:33:35] PROBLEM - puppet last run on mw1050 is CRITICAL: CRITICAL: puppet fail [16:33:35] PROBLEM - puppet last run on mw1029 is CRITICAL: CRITICAL: puppet fail [16:33:44] PROBLEM - puppet last run on francium is CRITICAL: CRITICAL: puppet fail [16:33:44] PROBLEM - puppet last run on mw1098 is CRITICAL: CRITICAL: puppet fail [16:33:53] PROBLEM - puppet last run on analytics1050 is CRITICAL: CRITICAL: puppet fail [16:33:54] PROBLEM - puppet last run on mw1097 is CRITICAL: CRITICAL: puppet fail [16:34:03] PROBLEM - puppet last run on elastic1014 is CRITICAL: CRITICAL: puppet fail [16:34:04] PROBLEM - puppet last run on db2066 is CRITICAL: CRITICAL: puppet fail [16:34:04] PROBLEM - puppet last run on logstash1003 is CRITICAL: CRITICAL: puppet fail [16:34:04] PROBLEM - puppet last run on mw2162 is CRITICAL: CRITICAL: puppet fail [16:34:04] PROBLEM - puppet last run on mw1244 is CRITICAL: CRITICAL: puppet fail [16:34:13] PROBLEM - puppet last run on cp3032 is CRITICAL: CRITICAL: puppet fail [16:34:13] PROBLEM - puppet last run on cp3015 is CRITICAL: CRITICAL: puppet fail [16:34:13] PROBLEM - puppet last run on eeden is CRITICAL: CRITICAL: puppet fail [16:34:14] PROBLEM - puppet last run on elastic1028 is CRITICAL: CRITICAL: puppet fail [16:34:14] PROBLEM - puppet last run on cp1060 is CRITICAL: CRITICAL: puppet fail [16:34:14] PROBLEM - puppet last run on mw2072 is CRITICAL: CRITICAL: puppet fail [16:34:14] PROBLEM - puppet last run on mw2214 is CRITICAL: CRITICAL: puppet fail [16:34:15] PROBLEM - puppet last run on wtp1019 is CRITICAL: CRITICAL: puppet fail [16:34:15] PROBLEM - puppet last run on db1057 is CRITICAL: CRITICAL: puppet fail [16:34:16] PROBLEM - puppet last run on mw2012 is CRITICAL: CRITICAL: puppet fail [16:34:16] PROBLEM - puppet last run on db2051 is CRITICAL: CRITICAL: puppet fail [16:34:17] PROBLEM - puppet last run on mw2054 is CRITICAL: CRITICAL: puppet fail [16:34:17] PROBLEM - puppet last run on mw1248 is CRITICAL: CRITICAL: puppet fail [16:34:18] PROBLEM - puppet last run on mc1010 is CRITICAL: CRITICAL: puppet fail [16:34:21] hm [16:34:33] PROBLEM - puppet last run on mw1163 is CRITICAL: CRITICAL: puppet fail [16:34:33] PROBLEM - puppet last run on mc2008 is CRITICAL: CRITICAL: puppet fail [16:34:34] PROBLEM - puppet last run on analytics1042 is CRITICAL: CRITICAL: puppet fail [16:34:34] PROBLEM - puppet last run on ms-be1013 is CRITICAL: CRITICAL: puppet fail [16:34:34] PROBLEM - puppet last run on mw1023 is CRITICAL: CRITICAL: puppet fail [16:34:34] PROBLEM - puppet last run on mw1192 is CRITICAL: CRITICAL: puppet fail [16:34:44] PROBLEM - puppet last run on mw2122 is CRITICAL: CRITICAL: puppet fail [16:34:44] PROBLEM - puppet last run on cp2009 is CRITICAL: CRITICAL: puppet fail [16:34:44] PROBLEM - puppet last run on ytterbium is CRITICAL: CRITICAL: puppet fail [16:34:45] PROBLEM - puppet last run on mw1212 is CRITICAL: CRITICAL: puppet fail [16:34:45] PROBLEM - puppet last run on mw2038 is CRITICAL: CRITICAL: puppet fail [16:34:45] PROBLEM - puppet last run on mw2028 is CRITICAL: CRITICAL: puppet fail [16:34:53] PROBLEM - puppet last run on elastic1031 is CRITICAL: CRITICAL: puppet fail [16:34:54] PROBLEM - puppet last run on restbase2004 is CRITICAL: CRITICAL: puppet fail [16:34:54] PROBLEM - puppet last run on mw2074 is CRITICAL: CRITICAL: puppet fail [16:34:54] PROBLEM - puppet last run on restbase1007 is CRITICAL: CRITICAL: puppet fail [16:34:54] PROBLEM - puppet last run on mc1013 is CRITICAL: CRITICAL: puppet fail [16:34:54] PROBLEM - puppet last run on mw1109 is CRITICAL: CRITICAL: puppet fail [16:34:54] PROBLEM - puppet last run on rcs1002 is CRITICAL: CRITICAL: puppet fail [16:34:55] PROBLEM - puppet last run on rdb2004 is CRITICAL: CRITICAL: puppet fail [16:34:55] PROBLEM - puppet last run on db2035 is CRITICAL: CRITICAL: puppet fail [16:34:56] PROBLEM - puppet last run on wtp1009 is CRITICAL: CRITICAL: puppet fail [16:34:56] PROBLEM - puppet last run on wtp1011 is CRITICAL: CRITICAL: puppet fail [16:34:57] PROBLEM - puppet last run on mw2155 is CRITICAL: CRITICAL: puppet fail [16:34:57] PROBLEM - puppet last run on mw1216 is CRITICAL: CRITICAL: puppet fail [16:35:03] PROBLEM - puppet last run on elastic1002 is CRITICAL: CRITICAL: puppet fail [16:35:04] PROBLEM - puppet last run on kafka1020 is CRITICAL: CRITICAL: puppet fail [16:35:04] PROBLEM - puppet last run on mw2210 is CRITICAL: CRITICAL: puppet fail [16:35:04] PROBLEM - puppet last run on mw2169 is CRITICAL: CRITICAL: puppet fail [16:35:04] PROBLEM - puppet last run on mw2034 is CRITICAL: CRITICAL: puppet fail [16:35:04] PROBLEM - puppet last run on restbase2003 is CRITICAL: CRITICAL: puppet fail [16:35:04] PROBLEM - puppet last run on wtp1017 is CRITICAL: CRITICAL: puppet fail [16:35:05] PROBLEM - puppet last run on mw1006 is CRITICAL: CRITICAL: puppet fail [16:35:06] PROBLEM - git.wikimedia.org on antimony is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:35:23] PROBLEM - puppet last run on db1030 is CRITICAL: CRITICAL: puppet fail [16:35:24] PROBLEM - puppet last run on db1054 is CRITICAL: CRITICAL: puppet fail [16:35:24] PROBLEM - puppet last run on graphite1001 is CRITICAL: CRITICAL: puppet fail [16:35:33] PROBLEM - puppet last run on mw1077 is CRITICAL: CRITICAL: puppet fail [16:35:33] PROBLEM - puppet last run on analytics1054 is CRITICAL: CRITICAL: puppet fail [16:35:34] PROBLEM - puppet last run on mw1148 is CRITICAL: CRITICAL: puppet fail [16:35:34] PROBLEM - puppet last run on mw1005 is CRITICAL: CRITICAL: puppet fail [16:35:34] PROBLEM - puppet last run on mw1105 is CRITICAL: CRITICAL: puppet fail [16:35:35] PROBLEM - puppet last run on wtp2006 is CRITICAL: CRITICAL: puppet fail [16:35:39] the puppet alerts were me. my commit to the private repo introduced a syntax error. fixed now, but there will be spam :( [16:35:43] PROBLEM - puppet last run on mw1028 is CRITICAL: CRITICAL: puppet fail [16:35:44] PROBLEM - puppet last run on elastic1005 is CRITICAL: CRITICAL: puppet fail [16:35:44] PROBLEM - puppet last run on mw2124 is CRITICAL: CRITICAL: puppet fail [16:35:44] PROBLEM - puppet last run on db2063 is CRITICAL: CRITICAL: puppet fail [16:35:45] PROBLEM - puppet last run on ocg1003 is CRITICAL: CRITICAL: puppet fail [16:35:45] PROBLEM - puppet last run on cp2004 is CRITICAL: CRITICAL: puppet fail [16:35:50] yeah [16:35:51] I just saw [16:35:54] PROBLEM - puppet last run on cp4006 is CRITICAL: CRITICAL: puppet fail [16:35:54] PROBLEM - puppet last run on cp3049 is CRITICAL: CRITICAL: puppet fail [16:35:54] PROBLEM - puppet last run on nescio is CRITICAL: CRITICAL: puppet fail [16:35:54] PROBLEM - puppet last run on cp2024 is CRITICAL: CRITICAL: puppet fail [16:35:55] PROBLEM - puppet last run on mw1239 is CRITICAL: CRITICAL: puppet fail [16:35:55] PROBLEM - puppet last run on mw2071 is CRITICAL: CRITICAL: puppet fail [16:35:56] ok [16:36:00] we're going out [16:36:04] PROBLEM - puppet last run on mw1167 is CRITICAL: CRITICAL: puppet fail [16:36:04] PROBLEM - puppet last run on mw1043 is CRITICAL: CRITICAL: puppet fail [16:36:04] PROBLEM - puppet last run on mw2159 is CRITICAL: CRITICAL: puppet fail [16:36:04] PROBLEM - puppet last run on mw2154 is CRITICAL: CRITICAL: puppet fail [16:36:05] PROBLEM - puppet last run on db1065 is CRITICAL: CRITICAL: puppet fail [16:36:05] PROBLEM - puppet last run on neptunium is CRITICAL: CRITICAL: puppet fail [16:36:05] PROBLEM - puppet last run on xenon is CRITICAL: CRITICAL: puppet fail [16:36:05] PROBLEM - puppet last run on cp1070 is CRITICAL: CRITICAL: puppet fail [16:36:05] call if anything breaks badly [16:36:05] PROBLEM - puppet last run on carbon is CRITICAL: CRITICAL: puppet fail [16:36:09] nothing will [16:36:12] enjoy, bye [16:36:14] PROBLEM - puppet last run on db2053 is CRITICAL: CRITICAL: puppet fail [16:36:14] PROBLEM - puppet last run on nitrogen is CRITICAL: CRITICAL: puppet fail [16:36:14] PROBLEM - puppet last run on analytics1055 is CRITICAL: CRITICAL: puppet fail [16:36:15] PROBLEM - puppet last run on mw2148 is CRITICAL: CRITICAL: puppet fail [16:36:15] PROBLEM - puppet last run on pc1003 is CRITICAL: CRITICAL: puppet fail [16:36:15] PROBLEM - puppet last run on db2012 is CRITICAL: CRITICAL: puppet fail [16:36:23] PROBLEM - puppet last run on db2005 is CRITICAL: CRITICAL: puppet fail [16:36:24] PROBLEM - puppet last run on mw1067 is CRITICAL: CRITICAL: puppet fail [16:36:24] PROBLEM - puppet last run on mw2014 is CRITICAL: CRITICAL: puppet fail [16:36:24] PROBLEM - puppet last run on mw1012 is CRITICAL: CRITICAL: puppet fail [16:36:25] PROBLEM - puppet last run on mw2108 is CRITICAL: CRITICAL: puppet fail [16:36:33] PROBLEM - puppet last run on mw1229 is CRITICAL: CRITICAL: puppet fail [16:36:34] PROBLEM - puppet last run on db2030 is CRITICAL: CRITICAL: puppet fail [16:36:34] PROBLEM - puppet last run on db2028 is CRITICAL: CRITICAL: puppet fail [16:36:34] PROBLEM - puppet last run on mw2121 is CRITICAL: CRITICAL: puppet fail [16:36:35] PROBLEM - puppet last run on ganeti2006 is CRITICAL: CRITICAL: puppet fail [16:36:35] PROBLEM - puppet last run on wtp1003 is CRITICAL: CRITICAL: puppet fail [16:36:35] PROBLEM - puppet last run on restbase-test2001 is CRITICAL: CRITICAL: puppet fail [16:36:35] PROBLEM - puppet last run on mw2175 is CRITICAL: CRITICAL: puppet fail [16:36:35] PROBLEM - puppet last run on mw2205 is CRITICAL: CRITICAL: puppet fail [16:36:36] PROBLEM - puppet last run on mw1186 is CRITICAL: CRITICAL: puppet fail [16:36:36] PROBLEM - puppet last run on oxygen is CRITICAL: CRITICAL: puppet fail [16:36:43] PROBLEM - puppet last run on mw1152 is CRITICAL: CRITICAL: puppet fail [16:36:44] PROBLEM - puppet last run on mw1108 is CRITICAL: CRITICAL: puppet fail [16:36:44] PROBLEM - puppet last run on db1061 is CRITICAL: CRITICAL: puppet fail [16:36:44] PROBLEM - puppet last run on ganeti1001 is CRITICAL: CRITICAL: puppet fail [16:36:44] PROBLEM - puppet last run on mw2185 is CRITICAL: CRITICAL: puppet fail [16:36:44] PROBLEM - puppet last run on mw2180 is CRITICAL: CRITICAL: puppet fail [16:36:44] PROBLEM - puppet last run on mw2204 is CRITICAL: CRITICAL: puppet fail [16:36:45] PROBLEM - puppet last run on mw2213 is CRITICAL: CRITICAL: puppet fail [16:36:45] PROBLEM - puppet last run on dbstore2001 is CRITICAL: CRITICAL: puppet fail [16:36:46] PROBLEM - puppet last run on mw1160 is CRITICAL: CRITICAL: puppet fail [16:36:46] PROBLEM - puppet last run on cp1047 is CRITICAL: CRITICAL: puppet fail [16:36:47] PROBLEM - puppet last run on elastic1007 is CRITICAL: CRITICAL: puppet fail [16:36:53] PROBLEM - puppet last run on mw1224 is CRITICAL: CRITICAL: puppet fail [16:37:03] PROBLEM - puppet last run on es1019 is CRITICAL: CRITICAL: puppet fail [16:37:03] PROBLEM - puppet last run on labnodepool1001 is CRITICAL: CRITICAL: puppet fail [16:37:04] PROBLEM - puppet last run on mw1121 is CRITICAL: CRITICAL: puppet fail [16:37:05] PROBLEM - puppet last run on ms-be1011 is CRITICAL: CRITICAL: puppet fail [16:37:05] PROBLEM - puppet last run on labvirt1008 is CRITICAL: CRITICAL: puppet fail [16:37:14] PROBLEM - puppet last run on mw1099 is CRITICAL: CRITICAL: puppet fail [16:37:15] PROBLEM - puppet last run on cp4020 is CRITICAL: CRITICAL: puppet fail [16:37:15] PROBLEM - puppet last run on cp1067 is CRITICAL: CRITICAL: puppet fail [16:37:43] So who is the puppet master? [16:37:48] <_< [16:37:57] everything is fine [16:47:34] ori: icinga isn't fine - its annoyed that it's had to post all the messages and is having a nervous breakdown. [17:20:16] (03PS1) 10Ori.livneh: Add grafana-test.wikimedia.org, behind misc-web-lb [dns] - 10https://gerrit.wikimedia.org/r/245503 [17:20:53] (03PS2) 10Ori.livneh: misc varnish: proxy grafana-testing.wm.o to krypton as well [puppet] - 10https://gerrit.wikimedia.org/r/245494 [17:22:21] (03PS3) 10Nuria: Mark incoming requests without cookies in x-analytics [puppet] - 10https://gerrit.wikimedia.org/r/244626 [17:36:19] (03PS1) 10Ori.livneh: Provision Grafana 2 on grafana-test.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/245504 [17:40:50] (03PS4) 10Ori.livneh: reprepro: import from grafana apt [puppet] - 10https://gerrit.wikimedia.org/r/245490 [17:41:53] (03PS5) 10Ori.livneh: reprepro: import from grafana apt [puppet] - 10https://gerrit.wikimedia.org/r/245490 [17:42:45] (03PS3) 10Ori.livneh: misc varnish: proxy grafana-testing.wm.o to krypton as well [puppet] - 10https://gerrit.wikimedia.org/r/245494 [17:43:47] (03PS2) 10Ori.livneh: Add grafana-test.wikimedia.org, behind misc-web-lb [dns] - 10https://gerrit.wikimedia.org/r/245503 [17:43:56] (03PS3) 10Ori.livneh: Add grafana-test.wikimedia.org, behind misc-web-lb [dns] - 10https://gerrit.wikimedia.org/r/245503 [17:45:51] (03PS2) 10Ori.livneh: Provision Grafana 2 on grafana-test.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/245504 (https://phabricator.wikimedia.org/T104738) [17:48:20] 6operations, 7Graphite, 5Patch-For-Review: Upgrade to Grafana v2.x - https://phabricator.wikimedia.org/T104738#1719666 (10ori) Patches: * [Idd37460aa82](https://gerrit.wikimedia.org/r/#/c/245504/): Provision Grafana 2 on grafana-test.wikimedia.org * [Ife115e0c902](https://gerrit.wikimedia.org/r/#/c/245494/)... [17:52:50] ori, have you been getting session data losses on wikitech? [17:53:17] no, but i haven't been using it heavily. i used it a fair bit about 12 hours ago. [17:57:01] (03PS3) 10Ori.livneh: Revert "Route Bug40009 logs to fluorine" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/244141 (owner: 10TTO) [17:57:06] (03CR) 10Ori.livneh: [C: 032] "thanks" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/244141 (owner: 10TTO) [17:57:12] (03Merged) 10jenkins-bot: Revert "Route Bug40009 logs to fluorine" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/244141 (owner: 10TTO) [18:16:58] (03CR) 10Hashar: [C: 031] rubocop: Ignore Style/TrailingComma offense [puppet] - 10https://gerrit.wikimedia.org/r/238779 (https://phabricator.wikimedia.org/T112651) (owner: 10Zfilipin) [18:27:32] (03PS2) 10Hashar: contint: install npm/grunt-cli with npm [puppet] - 10https://gerrit.wikimedia.org/r/244748 (https://phabricator.wikimedia.org/T113903) [18:33:43] 6operations, 10RESTBase, 6Services: Switch RESTBase to use Node.js 4 - https://phabricator.wikimedia.org/T107762#1719777 (10Pchelolo) LTS version 4.2.0 was released: https://github.com/nodejs/node/blob/v4.2.0/CHANGELOG.md [19:09:19] !log on ruthenium installed iotop for stall investigation [19:09:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [19:20:44] (03CR) 10Southparkfan: "Did you forgot to rename modules/wmflib/lib/puppet/parser/functions/php_ini.rb to modules/wmflib/lib/puppet/parser/functions/ini_file.rb?" [puppet] - 10https://gerrit.wikimedia.org/r/245496 (owner: 10Ori.livneh) [19:22:07] (03PS1) 10Catrope: Remove override of $wgEchoDefaultNotificationTypes [mediawiki-config] - 10https://gerrit.wikimedia.org/r/245578 (https://phabricator.wikimedia.org/T113367) [19:43:55] PROBLEM - puppet last run on mw1190 is CRITICAL: CRITICAL: Puppet has 1 failures [19:51:31] (03Abandoned) 10Reedy: Move dblists to dblist folder [mediawiki-config] - 10https://gerrit.wikimedia.org/r/244727 (owner: 10Reedy) [19:52:08] (03PS5) 10Reedy: Delete dblist symlinks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/244728 (https://phabricator.wikimedia.org/T115144) [19:52:12] (03CR) 10jenkins-bot: [V: 04-1] Delete dblist symlinks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/244728 (https://phabricator.wikimedia.org/T115144) (owner: 10Reedy) [19:52:35] RECOVERY - puppet last run on mw1085 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [19:54:24] RECOVERY - puppet last run on mw1190 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:55:43] (03Abandoned) 10Reedy: Delete dblist symlinks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/244728 (https://phabricator.wikimedia.org/T115144) (owner: 10Reedy) [19:57:57] (03PS1) 10Reedy: Add new dblist symlinks for noc conf [mediawiki-config] - 10https://gerrit.wikimedia.org/r/245584 [20:00:04] gwicke cscott arlolra subbu bearND mdholloway: Respected human, time to deploy Services – Parsoid / OCG / Citoid / Mobileapps / … (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20151012T2000). Please do the needful. [20:06:56] (03CR) 10Ori.livneh: "Southparkfan: Maybe. >_>" [puppet] - 10https://gerrit.wikimedia.org/r/245496 (owner: 10Ori.livneh) [20:10:02] (03PS2) 10Ori.livneh: Rename php_ini() to ini() [puppet] - 10https://gerrit.wikimedia.org/r/245496 [20:21:02] log MobileApps deployed sha1 95293e5 [20:21:09] !log MobileApps deployed sha1 95293e5 [20:21:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [20:21:48] (03CR) 10Krinkle: [C: 04-1] contint: install npm/grunt-cli with npm (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/244748 (https://phabricator.wikimedia.org/T113903) (owner: 10Hashar) [20:24:08] (03CR) 10Krinkle: contint: install npm/grunt-cli with npm (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/244748 (https://phabricator.wikimedia.org/T113903) (owner: 10Hashar) [20:26:35] (03PS1) 10Catrope: Flow-occupy all talk namespaces on sewikimedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/245588 (https://phabricator.wikimedia.org/T106302) [20:27:24] (03PS1) 10Catrope: Flow-occupy new talk namespaces from Gadgets on mediawikiwiki and sewikimedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/245589 [20:50:14] 10Ops-Access-Requests, 6operations, 7Icinga: give John Lewis permissions to send commands in icinga for fermium/mailman - https://phabricator.wikimedia.org/T105229#1720004 (10Dzahn) >>! In T105229#1717266, @JohnLewis wrote: > May be irrelevant but the username seemingly used for the icings set up is johnfle... [21:16:37] (03PS2) 10BryanDavis: vagarnt::mediawiki: Ensure clone before adding config [puppet] - 10https://gerrit.wikimedia.org/r/245207 (https://phabricator.wikimedia.org/T115229) [22:23:16] 6operations, 10Datasets-General-or-Unknown, 7HHVM, 5Patch-For-Review: Convert snapshot hosts to use HHVM and trusty - https://phabricator.wikimedia.org/T94277#1720231 (10ArielGlenn) Moving to trusty and php 5.5 didn't impact the dumps particularly. There was an issue caused by my pylint changes, unrelated... [22:26:06] 6operations, 10Salt: Move salt master to separate host from puppet master - https://phabricator.wikimedia.org/T115287#1720233 (10ArielGlenn) 3NEW a:3ArielGlenn [22:28:40] 6operations, 10hardware-requests: Allocate hardware for salt master in eqiad - https://phabricator.wikimedia.org/T115288#1720246 (10ArielGlenn) 3NEW [22:29:25] 6operations, 10hardware-requests: Allocate hardware for salt master in eqiad - https://phabricator.wikimedia.org/T115288#1720255 (10ArielGlenn) [22:29:26] 6operations, 10Salt: Move salt master to separate host from puppet master - https://phabricator.wikimedia.org/T115287#1720254 (10ArielGlenn) [22:44:13] 6operations, 10Salt: take steps outlined at techops offiste to (try to) address salt reliability - https://phabricator.wikimedia.org/T115292#1720299 (10ArielGlenn) 3NEW a:3ArielGlenn [22:44:27] 6operations, 10Salt: take steps outlined at techops offiste to (try to) address salt reliability - https://phabricator.wikimedia.org/T115292#1720307 (10ArielGlenn) [22:44:28] 6operations, 10Salt: Move salt master to separate host from puppet master - https://phabricator.wikimedia.org/T115287#1720308 (10ArielGlenn) [23:00:04] RoanKattouw ostriches Krenair: Dear anthropoid, the time has come. Please deploy Evening SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20151012T2300). [23:00:04] RoanKattouw Krenair: A patch you scheduled for Evening SWAT (Max 8 patches) is about to be deployed. Please be available during the process. [23:02:05] Good to see that 10 is less than 8 [23:02:55] RoanKattouw, want to do your parts first? [23:02:58] or shall I? [23:03:42] 6operations, 10Datasets-General-or-Unknown, 7HHVM, 5Patch-For-Review: Convert snapshot hosts to use HHVM and trusty - https://phabricator.wikimedia.org/T94277#1720326 (10Hydriz) Thanks for the information, hopefully we are progressing moving towards a faster dump generation process. Thanks! [23:04:04] Go ahead [23:04:21] oh, there's something I missed from the list [23:04:36] (03PS4) 10Alex Monk: Update DB size lists [mediawiki-config] - 10https://gerrit.wikimedia.org/r/243517 (https://phabricator.wikimedia.org/T114613) [23:04:42] (03CR) 10Alex Monk: [C: 032] Update DB size lists [mediawiki-config] - 10https://gerrit.wikimedia.org/r/243517 (https://phabricator.wikimedia.org/T114613) (owner: 10Alex Monk) [23:04:48] (03Merged) 10jenkins-bot: Update DB size lists [mediawiki-config] - 10https://gerrit.wikimedia.org/r/243517 (https://phabricator.wikimedia.org/T114613) (owner: 10Alex Monk) [23:06:00] one host stuck? [23:06:07] Not mira again hopefully? [23:06:15] Use ps ajxf | less in a second shell to figure out which host it us [23:06:17] *is [23:06:18] !log krenair@tin Synchronized database lists: (no message) (duration: 01m 13s) [23:06:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:06:31] it got there eventually [23:06:36] was just slow [23:07:52] ummm [23:07:55] I don't think that worked [23:08:06] ori, did you test sync-dblist works? [23:09:35] PROBLEM - nutcracker port on silver is CRITICAL: CRITICAL - Socket timeout after 2 seconds [23:10:00] sync-dir dblist [23:10:01] ;) [23:10:24] that's what I ended up doing [23:11:19] https://github.com/wikimedia/mediawiki-tools-scap/blob/9da7653650bd57d07b9cdfceb2223d5ca0901056/scap/main.py#L331-L332 [23:11:24] I presume that wants updating [23:11:31] !log krenair@tin Synchronized dblists: https://gerrit.wikimedia.org/r/#/c/243517/ (duration: 01m 13s) [23:11:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:12:44] (03CR) 10Reedy: [C: 04-1] "dblist -> dblists" [puppet] - 10https://gerrit.wikimedia.org/r/244743 (owner: 10Reedy) [23:12:54] RECOVERY - nutcracker port on silver is OK: TCP OK - 0.000 second response time on port 11212 [23:13:22] I guess ori might be the person to ask about those silver nutcracker warnings too [23:13:58] (03PS1) 10Reedy: Fix sync-dblist to go with dblist moves to folder [tools/scap] - 10https://gerrit.wikimedia.org/r/245606 [23:14:03] (03PS2) 10Alex Monk: Naming standardization from 'flooder' to 'flood' [mediawiki-config] - 10https://gerrit.wikimedia.org/r/245194 (https://phabricator.wikimedia.org/T115200) (owner: 10MarcoAurelio) [23:14:12] (03CR) 10Alex Monk: [C: 032] Naming standardization from 'flooder' to 'flood' [mediawiki-config] - 10https://gerrit.wikimedia.org/r/245194 (https://phabricator.wikimedia.org/T115200) (owner: 10MarcoAurelio) [23:14:18] (03Merged) 10jenkins-bot: Naming standardization from 'flooder' to 'flood' [mediawiki-config] - 10https://gerrit.wikimedia.org/r/245194 (https://phabricator.wikimedia.org/T115200) (owner: 10MarcoAurelio) [23:16:36] !log krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/245194/ (duration: 01m 13s) [23:16:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:16:49] It does look like mira is being slow [23:18:53] (03CR) 10Alex Monk: Add patrol, autopatrol, flood group to itwikiversity (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/244896 (https://phabricator.wikimedia.org/T114930) (owner: 10Gerrit Patch Uploader) [23:21:02] (03PS9) 10Alex Monk: Add patrol, autopatrol, flood group to itwikiversity [mediawiki-config] - 10https://gerrit.wikimedia.org/r/244896 (https://phabricator.wikimedia.org/T114930) (owner: 10Gerrit Patch Uploader) [23:21:11] (03CR) 10Alex Monk: [C: 032] Add patrol, autopatrol, flood group to itwikiversity [mediawiki-config] - 10https://gerrit.wikimedia.org/r/244896 (https://phabricator.wikimedia.org/T114930) (owner: 10Gerrit Patch Uploader) [23:21:55] (03Merged) 10jenkins-bot: Add patrol, autopatrol, flood group to itwikiversity [mediawiki-config] - 10https://gerrit.wikimedia.org/r/244896 (https://phabricator.wikimedia.org/T114930) (owner: 10Gerrit Patch Uploader) [23:22:32] (03PS2) 10Reedy: Add dblist to many paths [puppet] - 10https://gerrit.wikimedia.org/r/244743 [23:22:58] (03PS3) 10Reedy: Add dblist to many paths [puppet] - 10https://gerrit.wikimedia.org/r/244743 [23:23:40] !log krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/244896/ (duration: 01m 14s) [23:23:44] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:24:24] (03PS3) 10Alex Monk: Portal namespace for fawikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/244378 (https://phabricator.wikimedia.org/T113593) [23:24:31] (03CR) 10Alex Monk: [C: 032] Portal namespace for fawikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/244378 (https://phabricator.wikimedia.org/T113593) (owner: 10Alex Monk) [23:24:37] (03Merged) 10jenkins-bot: Portal namespace for fawikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/244378 (https://phabricator.wikimedia.org/T113593) (owner: 10Alex Monk) [23:26:07] !log krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/244378/ (duration: 01m 13s) [23:26:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:27:53] (03PS2) 10Alex Monk: Remove $wgLanguageCode for special wikis in CommonSettings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/243921 (owner: 10Glaisher) [23:28:06] (03CR) 10Alex Monk: [C: 032] Remove $wgLanguageCode for special wikis in CommonSettings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/243921 (owner: 10Glaisher) [23:28:12] (03Merged) 10jenkins-bot: Remove $wgLanguageCode for special wikis in CommonSettings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/243921 (owner: 10Glaisher) [23:29:26] (03PS2) 10Reedy: Fix sync-dblist to go with dblist moves to folder [tools/scap] - 10https://gerrit.wikimedia.org/r/245606 [23:29:27] fscking plurals [23:31:04] PROBLEM - YARN NodeManager Node-State on analytics1034 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [23:32:19] Hmm [23:32:26] Might have found an issue with that last commit I merged [23:32:43] RECOVERY - YARN NodeManager Node-State on analytics1034 is OK: OK: YARN NodeManager analytics1034.eqiad.wmnet:8041 Node-State: RUNNING [23:33:10] or... maybe not [23:33:50] oh, nope, seems fine [23:35:22] !log krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/243921/ (duration: 01m 13s) [23:35:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:36:19] (03PS3) 10Alex Monk: Set $wgUploadNavigationUrl to use uselang=$lang for commonsuploads wikis by default [mediawiki-config] - 10https://gerrit.wikimedia.org/r/243920 (https://phabricator.wikimedia.org/T111335) (owner: 10Glaisher) [23:36:26] (03CR) 10Alex Monk: [C: 032] Set $wgUploadNavigationUrl to use uselang=$lang for commonsuploads wikis by default [mediawiki-config] - 10https://gerrit.wikimedia.org/r/243920 (https://phabricator.wikimedia.org/T111335) (owner: 10Glaisher) [23:36:32] (03Merged) 10jenkins-bot: Set $wgUploadNavigationUrl to use uselang=$lang for commonsuploads wikis by default [mediawiki-config] - 10https://gerrit.wikimedia.org/r/243920 (https://phabricator.wikimedia.org/T111335) (owner: 10Glaisher) [23:38:15] !log krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/243920/ (duration: 01m 14s) [23:38:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:38:55] looks good [23:39:17] RoanKattouw, around? [23:39:22] Yup [23:39:31] ok. want to do your commits? [23:39:59] or shall I? [23:42:33] I'd be happy to do them myself, but if you don't mind doing them that would be cool [23:43:24] PROBLEM - puppet last run on ganeti2005 is CRITICAL: CRITICAL: puppet fail [23:49:49] (03CR) 10Alex Monk: [C: 032] Remove override of $wgEchoDefaultNotificationTypes [mediawiki-config] - 10https://gerrit.wikimedia.org/r/245578 (https://phabricator.wikimedia.org/T113367) (owner: 10Catrope) [23:50:17] (03Merged) 10jenkins-bot: Remove override of $wgEchoDefaultNotificationTypes [mediawiki-config] - 10https://gerrit.wikimedia.org/r/245578 (https://phabricator.wikimedia.org/T113367) (owner: 10Catrope) [23:53:49] !log krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/245578/ (duration: 01m 12s) [23:53:50] (03CR) 10Alex Monk: [C: 032] Fix sync-dblist to go with dblist moves to folder [tools/scap] - 10https://gerrit.wikimedia.org/r/245606 (owner: 10Reedy) [23:53:53] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:54:35] (03Merged) 10jenkins-bot: Fix sync-dblist to go with dblist moves to folder [tools/scap] - 10https://gerrit.wikimedia.org/r/245606 (owner: 10Reedy) [23:57:34] PROBLEM - puppet last run on mw2157 is CRITICAL: CRITICAL: puppet fail [23:57:41] (03CR) 10Alex Monk: [C: 032] Flow-occupy all talk namespaces on sewikimedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/245588 (https://phabricator.wikimedia.org/T106302) (owner: 10Catrope) [23:58:03] (03Merged) 10jenkins-bot: Flow-occupy all talk namespaces on sewikimedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/245588 (https://phabricator.wikimedia.org/T106302) (owner: 10Catrope) [23:58:24] (03CR) 10Alex Monk: [C: 032] Flow-occupy new talk namespaces from Gadgets on mediawikiwiki and sewikimedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/245589 (owner: 10Catrope) [23:58:30] (03Merged) 10jenkins-bot: Flow-occupy new talk namespaces from Gadgets on mediawikiwiki and sewikimedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/245589 (owner: 10Catrope)