[00:00:04] addshore, hashar, anomie, ostriches, aude, MaxSem, twentyafterfour, RoanKattouw, Dereckson, and thcipriani: Respected human, time to deploy Evening SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20170111T0000). Please do the needful. [00:07:58] RECOVERY - puppet last run on cp4001 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [00:39:19] <_joe_> !log restart hhvm on mw1182, stuck on HPHP::Treadmill::getAgeOldestRequest [00:39:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:40:46] (03CR) 10Andrew Bogott: [C: 032] labs dnsrecursor: add wikimania2018 [puppet] - 10https://gerrit.wikimedia.org/r/331526 (https://phabricator.wikimedia.org/T155038) (owner: 10Dzahn) [00:40:48] RECOVERY - Apache HTTP on mw1182 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 612 bytes in 0.027 second response time [00:40:54] (03PS4) 10Andrew Bogott: labs dnsrecursor: add wikimania2018 [puppet] - 10https://gerrit.wikimedia.org/r/331526 (https://phabricator.wikimedia.org/T155038) (owner: 10Dzahn) [00:40:58] RECOVERY - Nginx local proxy to apache on mw1182 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 613 bytes in 0.028 second response time [00:41:28] RECOVERY - HHVM rendering on mw1182 is OK: HTTP OK: HTTP/1.1 200 OK - 73661 bytes in 0.107 second response time [00:49:04] (03CR) 10Andrew Bogott: [C: 032] toollabs: bigbrother: stop tracking jobs when rcfile is deleted [puppet] - 10https://gerrit.wikimedia.org/r/330265 (https://phabricator.wikimedia.org/T94500) (owner: 10BryanDavis) [00:49:09] (03PS2) 10Andrew Bogott: toollabs: bigbrother: stop tracking jobs when rcfile is deleted [puppet] - 10https://gerrit.wikimedia.org/r/330265 (https://phabricator.wikimedia.org/T94500) (owner: 10BryanDavis) [00:54:50] 06Operations, 10Continuous-Integration-Infrastructure, 13Patch-For-Review, 07WorkType-Maintenance: Jenkins master / client ssh connection fails due to missing ssh algorithm - https://phabricator.wikimedia.org/T100509#2932738 (10Paladox) [01:34:58] PROBLEM - puppet last run on sca1004 is CRITICAL: CRITICAL: Puppet has 21 failures. Last run 2 minutes ago with 21 failures. Failed resources (up to 3 shown): Exec[eth0_v6_token],Package[zotero/translators],Package[zotero/translation-server],Exec[chown /srv/deployment/zotero for deploy-service] [01:38:58] 06Operations, 13Patch-For-Review: Remote IPMI doesn't work for ~17% of the fleet - https://phabricator.wikimedia.org/T150160#2932824 (10Dzahn) I checked P4379 for distro version and all the affected hosts are trusty, so there is that pattern. [01:43:46] 06Operations, 10Continuous-Integration-Infrastructure, 13Patch-For-Review, 07WorkType-Maintenance: Jenkins master / client ssh connection fails due to missing ssh algorithm - https://phabricator.wikimedia.org/T100509#2932838 (10Paladox) [01:52:39] 06Operations, 13Patch-For-Review: Remote IPMI doesn't work for ~17% of the fleet - https://phabricator.wikimedia.org/T150160#2932874 (10Dzahn) @volans looked more, it's like it is simply not installed on _any trusty_, that's a lot more than in P4379 though, it's 272 per salt ..'G@lsb_distrib_codename:trusty' a... [01:58:38] RECOVERY - Check systemd state on elastic2030 is OK: OK - running: The system is fully operational [02:00:13] (03PS1) 10Dzahn: (debug) test replacing require_package with package for ipmi [puppet] - 10https://gerrit.wikimedia.org/r/331574 [02:01:58] RECOVERY - puppet last run on sca1004 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [02:07:58] PROBLEM - puppet last run on cp3014 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:14:11] (03PS2) 10Dzahn: (debug) test removing is_virtual check for ipmi [puppet] - 10https://gerrit.wikimedia.org/r/331574 [02:23:59] 06Operations, 13Patch-For-Review: Remote IPMI doesn't work for ~17% of the fleet - https://phabricator.wikimedia.org/T150160#2932906 (10Dzahn) I tested replacing require_package with package, no difference. Then i tried removing the "is_virtual == false" check, and oh look that does it: http://puppet-compiler... [02:30:48] PROBLEM - Unmerged changes on repository mediawiki_config on mira is CRITICAL: There is one unmerged change in mediawiki_config (dir /srv/mediawiki-staging/, ref HEAD..readonly/master). [02:31:15] !log l10nupdate@tin scap sync-l10n completed (1.29.0-wmf.7) (duration: 10m 47s) [02:31:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:34:50] PROBLEM - puppet last run on maps1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:34:58] RECOVERY - puppet last run on cp3014 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [02:35:46] !log l10nupdate@tin ResourceLoader cache refresh completed at Wed Jan 11 02:35:46 UTC 2017 (duration 4m 31s) [02:35:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:50:28] PROBLEM - puppet last run on db1026 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [03:03:48] RECOVERY - puppet last run on maps1003 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [03:18:28] RECOVERY - puppet last run on db1026 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures [03:20:38] RECOVERY - Check systemd state on elastic2028 is OK: OK - running: The system is fully operational [03:55:28] PROBLEM - puppet last run on sca1003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [04:09:38] PROBLEM - puppet last run on sca2004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [04:22:28] RECOVERY - puppet last run on sca1003 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [04:37:38] RECOVERY - puppet last run on sca2004 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [04:42:38] PROBLEM - puppet last run on analytics1015 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [05:10:38] RECOVERY - puppet last run on analytics1015 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [05:14:05] (03PS1) 10Dereckson: Fix University of Mumbai event throttle rule [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331577 (https://phabricator.wikimedia.org/T154312) [05:56:18] (03PS1) 10Dzahn: base: missing quotes around is_virtual 'false' for ipmi [puppet] - 10https://gerrit.wikimedia.org/r/331579 (https://phabricator.wikimedia.org/T150160) [05:57:15] (03CR) 10jerkins-bot: [V: 04-1] base: missing quotes around is_virtual 'false' for ipmi [puppet] - 10https://gerrit.wikimedia.org/r/331579 (https://phabricator.wikimedia.org/T150160) (owner: 10Dzahn) [05:59:39] (03PS2) 10Dzahn: base: missing quotes around is_virtual 'false' for ipmi [puppet] - 10https://gerrit.wikimedia.org/r/331579 (https://phabricator.wikimedia.org/T150160) [06:00:37] (03CR) 10jerkins-bot: [V: 04-1] base: missing quotes around is_virtual 'false' for ipmi [puppet] - 10https://gerrit.wikimedia.org/r/331579 (https://phabricator.wikimedia.org/T150160) (owner: 10Dzahn) [06:02:28] (03PS3) 10Dzahn: base: missing quotes around is_virtual 'false' for ipmi [puppet] - 10https://gerrit.wikimedia.org/r/331579 (https://phabricator.wikimedia.org/T150160) [06:04:58] PROBLEM - puppet last run on sca1004 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 2 minutes ago with 7 failures. Failed resources (up to 3 shown): Package[tzdata],Service[zotero],Exec[zotero-admin_ensure_members],Exec[sc-admins_ensure_members] [06:07:57] (03CR) 10Dzahn: "this compiler run is an example where it claims there is "no change" but when you actually look at the production/change catalog links and" [puppet] - 10https://gerrit.wikimedia.org/r/331579 (https://phabricator.wikimedia.org/T150160) (owner: 10Dzahn) [06:12:03] 06Operations, 13Patch-For-Review: Remote IPMI doesn't work for ~17% of the fleet - https://phabricator.wikimedia.org/T150160#2933024 (10Dzahn) see change above in context of: ``` 19:43 < paravoid> mutante: it's "false", not false, be careful 19:43 < paravoid> the facts are strings still 19:43 < paravoid> and... [06:29:48] PROBLEM - Check HHVM threads for leakage on mw1168 is CRITICAL: CRITICAL: HHVM has more than double threads running or queued than apache has busy workers [06:30:38] PROBLEM - Check HHVM threads for leakage on mw1260 is CRITICAL: CRITICAL: HHVM has more than double threads running or queued than apache has busy workers [06:31:58] PROBLEM - puppet last run on sca2003 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [06:32:59] RECOVERY - puppet last run on sca1004 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [06:46:38] PROBLEM - Check HHVM threads for leakage on mw1260 is CRITICAL: CRITICAL: HHVM has more than double threads running or queued than apache has busy workers [06:46:48] PROBLEM - Check HHVM threads for leakage on mw1259 is CRITICAL: CRITICAL: HHVM has more than double threads running or queued than apache has busy workers [06:52:38] PROBLEM - Check HHVM threads for leakage on mw1169 is CRITICAL: CRITICAL: HHVM has more than double threads running or queued than apache has busy workers [06:57:38] PROBLEM - Check HHVM threads for leakage on mw1260 is CRITICAL: CRITICAL: HHVM has more than double threads running or queued than apache has busy workers [06:59:58] RECOVERY - puppet last run on sca2003 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [07:03:38] PROBLEM - Check HHVM threads for leakage on mw1260 is CRITICAL: CRITICAL: HHVM has more than double threads running or queued than apache has busy workers [07:10:38] PROBLEM - Check HHVM threads for leakage on mw1260 is CRITICAL: CRITICAL: HHVM has more than double threads running or queued than apache has busy workers [07:10:38] RECOVERY - Check systemd state on elastic2029 is OK: OK - running: The system is fully operational [07:17:48] PROBLEM - Check HHVM threads for leakage on mw1259 is CRITICAL: CRITICAL: HHVM has more than double threads running or queued than apache has busy workers [07:18:38] RECOVERY - Check HHVM threads for leakage on mw1260 is OK: OK [07:49:56] (03PS1) 10Urbanecm: Add bnwiki, enwiki and commons as import source in bd.wikimedia.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331584 (https://phabricator.wikimedia.org/T154990) [07:59:48] RECOVERY - Check HHVM threads for leakage on mw1168 is OK: OK [07:59:48] RECOVERY - Check HHVM threads for leakage on mw1259 is OK: OK [08:05:58] PROBLEM - Router interfaces on cr2-eqiad is CRITICAL: CRITICAL: host 208.80.154.197, interfaces up: 212, down: 1, dormant: 0, excluded: 0, unused: 0BRxe-3/2/3: down - Core: cr2-codfw:xe-5/0/1 (Zayo, OGYX/120003//ZYO) 36ms {#2909} [10Gbps wave]BR [08:11:58] RECOVERY - Router interfaces on cr2-eqiad is OK: OK: host 208.80.154.197, interfaces up: 214, down: 0, dormant: 0, excluded: 0, unused: 0 [08:37:18] PROBLEM - citoid endpoints health on scb2001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [08:37:18] PROBLEM - citoid endpoints health on scb2004 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [08:38:08] RECOVERY - citoid endpoints health on scb2004 is OK: All endpoints are healthy [08:38:08] RECOVERY - citoid endpoints health on scb2001 is OK: All endpoints are healthy [08:42:38] RECOVERY - Check HHVM threads for leakage on mw1169 is OK: OK [09:07:38] RECOVERY - Check systemd state on elastic2031 is OK: OK - running: The system is fully operational [09:12:27] (03CR) 10Alexandros Kosiaris: "Yeah, seems like a bug in the upstream project. Even running puppet catalog diff manually produces a "nodiff" output. The upstream project" [puppet] - 10https://gerrit.wikimedia.org/r/331579 (https://phabricator.wikimedia.org/T150160) (owner: 10Dzahn) [09:12:43] (03CR) 10Alexandros Kosiaris: [C: 031] base: missing quotes around is_virtual 'false' for ipmi [puppet] - 10https://gerrit.wikimedia.org/r/331579 (https://phabricator.wikimedia.org/T150160) (owner: 10Dzahn) [09:26:11] 06Operations, 10puppet-compiler: puppet compiler claims "no change" when catalogs are actually different - https://phabricator.wikimedia.org/T149432#2752413 (10akosiaris) I 've also reported the bug upstream at https://github.com/acidprime/puppet-catalog-diff/issues/48 [09:28:53] (03PS1) 10Alexandros Kosiaris: puppet-facts-export: Use correct regular expression [puppet] - 10https://gerrit.wikimedia.org/r/331592 [09:29:38] (03CR) 10Alexandros Kosiaris: [V: 032 C: 032] puppet-facts-export: Use correct regular expression [puppet] - 10https://gerrit.wikimedia.org/r/331592 (owner: 10Alexandros Kosiaris) [09:45:38] PROBLEM - puppet last run on analytics1057 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [09:50:23] (03CR) 10Alexandros Kosiaris: [C: 031] "Indeed. Fixed in https://gerrit.wikimedia.org/r/#/c/331592/ and https://puppet-compiler.wmflabs.org/5077/ is now happy." [puppet] - 10https://gerrit.wikimedia.org/r/331516 (owner: 10Dzahn) [09:55:29] (03CR) 10Alexandros Kosiaris: "Icinga complains this is unmerged on mira and tin. I am not entirely sure if I should just pull the change, please advise" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331093 (https://phabricator.wikimedia.org/T85947) (owner: 10Krinkle) [10:05:08] RECOVERY - PyBal backends health check on lvs2002 is OK: PYBAL OK - All pools are healthy [10:11:48] (03PS1) 10Juniorsys: admin module: include sudo class's full name [puppet] - 10https://gerrit.wikimedia.org/r/331595 [10:14:38] RECOVERY - puppet last run on analytics1057 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [10:17:56] (03CR) 10Alexandros Kosiaris: [C: 032] ruby-httpclient callers: Use the operating system's certificate store [puppet] - 10https://gerrit.wikimedia.org/r/311048 (https://phabricator.wikimedia.org/T145808) (owner: 10Alex Monk) [10:18:02] (03PS3) 10Alexandros Kosiaris: ruby-httpclient callers: Use the operating system's certificate store [puppet] - 10https://gerrit.wikimedia.org/r/311048 (https://phabricator.wikimedia.org/T145808) (owner: 10Alex Monk) [10:18:06] (03CR) 10Alexandros Kosiaris: [V: 032 C: 032] ruby-httpclient callers: Use the operating system's certificate store [puppet] - 10https://gerrit.wikimedia.org/r/311048 (https://phabricator.wikimedia.org/T145808) (owner: 10Alex Monk) [10:18:55] 06Operations, 10Traffic, 13Patch-For-Review: convert wikitech.wikimedia.org from globalsign to letsencrypt certificate (deadline 2017-02-24) - https://phabricator.wikimedia.org/T154913#2933100 (10akosiaris) Great! thanks! Patch merged [10:22:04] (03CR) 10Alexandros Kosiaris: [C: 04-1] "minor comment inline. LGTM otherwise" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/331595 (owner: 10Juniorsys) [10:26:54] (03PS1) 10Juniorsys: apache module: lint fixes [puppet] - 10https://gerrit.wikimedia.org/r/331596 [10:29:55] (03PS2) 10Juniorsys: apache module: lint fixes [puppet] - 10https://gerrit.wikimedia.org/r/331596 [10:31:08] PROBLEM - citoid endpoints health on scb1004 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [10:31:58] RECOVERY - citoid endpoints health on scb1004 is OK: All endpoints are healthy [10:32:38] (03PS3) 10Juniorsys: apache module: lint fixes [puppet] - 10https://gerrit.wikimedia.org/r/331596 [10:34:58] RECOVERY - PyBal backends health check on lvs2005 is OK: PYBAL OK - All pools are healthy [10:35:15] (03PS2) 10Juniorsys: admin module: include sudo class's full name [puppet] - 10https://gerrit.wikimedia.org/r/331595 [10:40:00] (03CR) 10Alexandros Kosiaris: [C: 032] "LGTM, thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/331595 (owner: 10Juniorsys) [10:40:03] (03CR) 10Alexandros Kosiaris: [V: 032 C: 032] admin module: include sudo class's full name [puppet] - 10https://gerrit.wikimedia.org/r/331595 (owner: 10Juniorsys) [10:43:38] (03PS1) 10Juniorsys: apertium/archiva modules [puppet] - 10https://gerrit.wikimedia.org/r/331597 [10:44:48] PROBLEM - puppet last run on elastic1017 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [10:45:17] (03PS2) 10Juniorsys: apertium/archiva module linting [puppet] - 10https://gerrit.wikimedia.org/r/331597 [10:50:09] (03PS1) 10Hashar: build: add xmlrpc for ruby2.4 [puppet] - 10https://gerrit.wikimedia.org/r/331598 [10:51:14] (03CR) 10jerkins-bot: [V: 04-1] build: add xmlrpc for ruby2.4 [puppet] - 10https://gerrit.wikimedia.org/r/331598 (owner: 10Hashar) [10:52:14] (03CR) 10Hashar: "And on Jenkins which use a different ruby version, we end up with:" [puppet] - 10https://gerrit.wikimedia.org/r/331598 (owner: 10Hashar) [10:53:50] (03PS1) 10Juniorsys: authdns module: Use full class names not relative [puppet] - 10https://gerrit.wikimedia.org/r/331599 [10:54:18] PROBLEM - citoid endpoints health on scb2001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [10:54:18] PROBLEM - citoid endpoints health on scb2004 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [10:55:08] PROBLEM - restbase endpoints health on restbase2005 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [10:55:18] PROBLEM - restbase endpoints health on restbase2004 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [10:55:58] RECOVERY - restbase endpoints health on restbase2005 is OK: All endpoints are healthy [10:56:08] RECOVERY - restbase endpoints health on restbase2004 is OK: All endpoints are healthy [10:57:08] RECOVERY - citoid endpoints health on scb2004 is OK: All endpoints are healthy [10:57:08] RECOVERY - citoid endpoints health on scb2001 is OK: All endpoints are healthy [11:00:58] PROBLEM - puppet last run on sca2003 is CRITICAL: CRITICAL: Puppet has 27 failures. Last run 2 minutes ago with 27 failures. Failed resources (up to 3 shown): Exec[eth0_v6_token],Package[zotero/translators],Package[zotero/translation-server],Exec[chown /srv/deployment/zotero for deploy-service] [11:12:48] RECOVERY - puppet last run on elastic1017 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [11:14:15] (03PS1) 10Alexandros Kosiaris: Add .sh extension to check_mailman_queue.sh [puppet] - 10https://gerrit.wikimedia.org/r/331600 [11:15:58] RECOVERY - puppet last run on sca2003 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [11:20:31] (03PS2) 10Alexandros Kosiaris: Add .sh extension to check_mailman_queue.sh [puppet] - 10https://gerrit.wikimedia.org/r/331600 [11:25:47] (03CR) 10Alexandros Kosiaris: [V: 032 C: 032] "Noop per https://puppet-compiler.wmflabs.org/5078/, merging" [puppet] - 10https://gerrit.wikimedia.org/r/331599 (owner: 10Juniorsys) [11:26:49] (03CR) 10Alexandros Kosiaris: [C: 032] Add .sh extension to check_mailman_queue.sh [puppet] - 10https://gerrit.wikimedia.org/r/331600 (owner: 10Alexandros Kosiaris) [11:26:54] (03PS3) 10Alexandros Kosiaris: Add .sh extension to check_mailman_queue.sh [puppet] - 10https://gerrit.wikimedia.org/r/331600 [11:26:57] (03CR) 10Alexandros Kosiaris: [V: 032 C: 032] Add .sh extension to check_mailman_queue.sh [puppet] - 10https://gerrit.wikimedia.org/r/331600 (owner: 10Alexandros Kosiaris) [11:28:58] RECOVERY - puppet last run on fermium is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [12:04:33] (03Abandoned) 10Hashar: build: add xmlrpc for ruby2.4 [puppet] - 10https://gerrit.wikimedia.org/r/331598 (owner: 10Hashar) [12:12:36] (03CR) 10Alexandros Kosiaris: [C: 032] apertium/archiva module linting [puppet] - 10https://gerrit.wikimedia.org/r/331597 (owner: 10Juniorsys) [12:12:41] (03PS3) 10Alexandros Kosiaris: apertium/archiva module linting [puppet] - 10https://gerrit.wikimedia.org/r/331597 (owner: 10Juniorsys) [12:12:44] (03CR) 10Alexandros Kosiaris: [V: 032 C: 032] apertium/archiva module linting [puppet] - 10https://gerrit.wikimedia.org/r/331597 (owner: 10Juniorsys) [12:26:05] (03CR) 10Alexandros Kosiaris: [C: 032] vagrant: add sudo rules for Vagrant 1.9.1 [puppet] - 10https://gerrit.wikimedia.org/r/329724 (https://phabricator.wikimedia.org/T122735) (owner: 10BryanDavis) [12:26:59] (03CR) 10Alexandros Kosiaris: [C: 032] vagrant: Update LXC packages and apparmor conf for systemd [puppet] - 10https://gerrit.wikimedia.org/r/329702 (https://phabricator.wikimedia.org/T154294) (owner: 10BryanDavis) [12:27:06] (03PS3) 10Alexandros Kosiaris: vagrant: Update LXC packages and apparmor conf for systemd [puppet] - 10https://gerrit.wikimedia.org/r/329702 (https://phabricator.wikimedia.org/T154294) (owner: 10BryanDavis) [12:27:10] (03CR) 10Alexandros Kosiaris: [V: 032 C: 032] vagrant: Update LXC packages and apparmor conf for systemd [puppet] - 10https://gerrit.wikimedia.org/r/329702 (https://phabricator.wikimedia.org/T154294) (owner: 10BryanDavis) [12:27:52] (03CR) 10Alexandros Kosiaris: [C: 032] vagrant: remove setup.sh call [puppet] - 10https://gerrit.wikimedia.org/r/329723 (owner: 10BryanDavis) [12:27:58] (03PS3) 10Alexandros Kosiaris: vagrant: remove setup.sh call [puppet] - 10https://gerrit.wikimedia.org/r/329723 (owner: 10BryanDavis) [12:28:01] (03CR) 10Alexandros Kosiaris: [V: 032 C: 032] vagrant: remove setup.sh call [puppet] - 10https://gerrit.wikimedia.org/r/329723 (owner: 10BryanDavis) [12:28:23] (03PS3) 10Alexandros Kosiaris: vagrant: add sudo rules for Vagrant 1.9.1 [puppet] - 10https://gerrit.wikimedia.org/r/329724 (https://phabricator.wikimedia.org/T122735) (owner: 10BryanDavis) [12:28:27] (03CR) 10Alexandros Kosiaris: [V: 032 C: 032] vagrant: add sudo rules for Vagrant 1.9.1 [puppet] - 10https://gerrit.wikimedia.org/r/329724 (https://phabricator.wikimedia.org/T122735) (owner: 10BryanDavis) [12:43:43] PROBLEM - puppet last run on californium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [12:45:01] (03PS1) 10Alexandros Kosiaris: admin: Add the staff group in the managed ones [puppet] - 10https://gerrit.wikimedia.org/r/331602 [12:45:53] (03CR) 10jerkins-bot: [V: 04-1] admin: Add the staff group in the managed ones [puppet] - 10https://gerrit.wikimedia.org/r/331602 (owner: 10Alexandros Kosiaris) [12:53:21] (03PS2) 10Alexandros Kosiaris: admin: Add the staff group in the managed ones [puppet] - 10https://gerrit.wikimedia.org/r/331602 [13:03:47] (03PS1) 10Alexandros Kosiaris: Add an oresrdb.svc.eqiad.wmnet RR [dns] - 10https://gerrit.wikimedia.org/r/331603 [13:10:00] (03CR) 10Alexandros Kosiaris: [C: 032] Add an oresrdb.svc.eqiad.wmnet RR [dns] - 10https://gerrit.wikimedia.org/r/331603 (owner: 10Alexandros Kosiaris) [13:11:43] RECOVERY - puppet last run on californium is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [13:22:31] (03PS1) 10Alexandros Kosiaris: ores: Use the service DNS for ores datastore [puppet] - 10https://gerrit.wikimedia.org/r/331604 [13:23:34] (03PS1) 10Urbanecm: Turn off patrolling in ruwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331605 (https://phabricator.wikimedia.org/T154285) [13:26:44] jouncebot, next [13:26:44] In 0 hour(s) and 33 minute(s): European Mid-day SWAT(Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20170111T1400) [13:33:22] (03PS2) 10Urbanecm: Turn off patrolling in ruwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331605 (https://phabricator.wikimedia.org/T154285) [13:35:03] (03PS3) 10Urbanecm: Turn off patrolling in ruwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331605 (https://phabricator.wikimedia.org/T154285) [13:38:51] (03CR) 10Alexandros Kosiaris: [C: 032] "https://puppet-compiler.wmflabs.org/5080/ says noop on multiple hosts, I am merging. Thanks!" [puppet] - 10https://gerrit.wikimedia.org/r/331596 (owner: 10Juniorsys) [13:38:57] (03PS4) 10Alexandros Kosiaris: apache module: lint fixes [puppet] - 10https://gerrit.wikimedia.org/r/331596 (owner: 10Juniorsys) [13:39:01] (03CR) 10Alexandros Kosiaris: [V: 032 C: 032] apache module: lint fixes [puppet] - 10https://gerrit.wikimedia.org/r/331596 (owner: 10Juniorsys) [13:42:23] PROBLEM - mailman I/O stats on fermium is CRITICAL: CRITICAL - I/O stats: Transfers/Sec=2232.80 Read Requests/Sec=2502.50 Write Requests/Sec=0.70 KBytes Read/Sec=33473.60 KBytes_Written/Sec=24.00 [13:56:23] RECOVERY - mailman I/O stats on fermium is OK: OK - I/O stats: Transfers/Sec=197.60 Read Requests/Sec=144.70 Write Requests/Sec=6.20 KBytes Read/Sec=5481.60 KBytes_Written/Sec=1406.40 [13:58:42] jouncebot: next [13:58:42] In 0 hour(s) and 1 minute(s): European Mid-day SWAT(Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20170111T1400) [13:59:36] (03PS2) 10Hashar: Add bnwiki, enwiki and commons as import source in bd.wikimedia.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331584 (https://phabricator.wikimedia.org/T154990) (owner: 10Urbanecm) [13:59:38] (03PS4) 10Hashar: Turn off patrolling in ruwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331605 (https://phabricator.wikimedia.org/T154285) (owner: 10Urbanecm) [14:00:04] addshore, hashar, anomie, ostriches, aude, MaxSem, twentyafterfour, RoanKattouw, Dereckson, and thcipriani: Respected human, time to deploy European Mid-day SWAT(Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20170111T1400). Please do the needful. [14:00:04] Urbanecm: A patch you scheduled for European Mid-day SWAT(Max 8 patches) is about to be deployed. Please be available during the process. [14:00:11] o/ [14:01:25] Around [14:01:39] (03CR) 10Hashar: [C: 032] Add bnwiki, enwiki and commons as import source in bd.wikimedia.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331584 (https://phabricator.wikimedia.org/T154990) (owner: 10Urbanecm) [14:01:45] (03CR) 10Hashar: [C: 032] Turn off patrolling in ruwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331605 (https://phabricator.wikimedia.org/T154285) (owner: 10Urbanecm) [14:01:56] Urbanecm: hello! I am going to push both on mwdebug1001 [14:02:29] Okay but I won't be able to test the import one as I have no sysop bit. [14:02:45] (03Merged) 10jenkins-bot: Add bnwiki, enwiki and commons as import source in bd.wikimedia.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331584 (https://phabricator.wikimedia.org/T154990) (owner: 10Urbanecm) [14:02:56] (03CR) 10jenkins-bot: Add bnwiki, enwiki and commons as import source in bd.wikimedia.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331584 (https://phabricator.wikimedia.org/T154990) (owner: 10Urbanecm) [14:03:18] (03Merged) 10jenkins-bot: Turn off patrolling in ruwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331605 (https://phabricator.wikimedia.org/T154285) (owner: 10Urbanecm) [14:03:53] RECOVERY - Unmerged changes on repository mediawiki_config on tin is OK: No changes to merge. [14:04:15] there was an undeployed change :/ [14:04:34] hashar, is this a problem for me? [14:04:38] na [14:04:43] (or am I required to do something) [14:04:49] I have pulled everything on mwdebug1001 [14:04:56] so you can test there :} [14:04:56] Okay, going to test testable. [14:05:04] (03CR) 10jenkins-bot: Turn off patrolling in ruwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331605 (https://phabricator.wikimedia.org/T154285) (owner: 10Urbanecm) [14:06:05] !log scap pull on terbium [14:06:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:06:21] Are you sure both of them are on mwdebug1001? [14:07:01] hashar, ^ [14:07:05] oh no sorry [14:07:24] I have been distracted by the other patch [14:07:27] Urbanecm: they are now :} [14:08:06] hashar, that's better. Both of them works. [14:09:03] RECOVERY - Unmerged changes on repository mediawiki_config on mira is OK: No changes to merge. [14:09:08] !log hashar@tin Synchronized composer.lock: build: Update PHPUnit from 3.7 to 4.8, add phplint to composer-test - T85947 (duration: 00m 55s) [14:09:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:09:12] T85947: Convert operations/mediawiki-config to use composer for phpunit and php linting - https://phabricator.wikimedia.org/T85947 [14:10:05] !log hashar@tin Synchronized composer.json: build: Update PHPUnit from 3.7 to 4.8, add phplint to composer-test - T85947 (duration: 00m 45s) [14:10:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:11:26] Urbanecm: deploying :) thank you! [14:11:43] !log hashar@tin Synchronized wmf-config/InitialiseSettings.php: Import source on bd.wikimedia.org T154990 + Turn of patrolling on ruwiki T154285 (duration: 00m 42s) [14:11:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:11:47] T154990: Add bnwiki, enwiki and commons as import source in bd.wikimedia.org - https://phabricator.wikimedia.org/T154990 [14:11:48] T154285: Turn off patrolling in ruwiki - https://phabricator.wikimedia.org/T154285 [14:12:35] hashar, thank you as well! [14:12:57] !log European SWAT completed [14:13:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:14:13] (03CR) 10Ladsgroup: [C: 031] ores: Use the service DNS for ores datastore [puppet] - 10https://gerrit.wikimedia.org/r/331604 (owner: 10Alexandros Kosiaris) [14:19:07] (03CR) 10Alexandros Kosiaris: [C: 032] ores: Use the service DNS for ores datastore [puppet] - 10https://gerrit.wikimedia.org/r/331604 (owner: 10Alexandros Kosiaris) [14:19:13] (03CR) 10Alexandros Kosiaris: [C: 032] "thanks! merging" [puppet] - 10https://gerrit.wikimedia.org/r/331604 (owner: 10Alexandros Kosiaris) [14:19:19] (03PS2) 10Alexandros Kosiaris: ores: Use the service DNS for ores datastore [puppet] - 10https://gerrit.wikimedia.org/r/331604 [14:19:21] (03CR) 10Alexandros Kosiaris: [V: 032 C: 032] ores: Use the service DNS for ores datastore [puppet] - 10https://gerrit.wikimedia.org/r/331604 (owner: 10Alexandros Kosiaris) [14:19:50] :) [14:58:43] PROBLEM - puppet last run on mw1289 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [15:21:09] 06Operations, 10ops-esams: Degraded RAID on bast3001 - https://phabricator.wikimedia.org/T154603#2933347 (10akosiaris) 05Resolved>03Open Reopening, smartd just sent the following for bast3001 ``` Device: /dev/sdb [SAT], 1 Currently unreadable (pending) sectors Device info: WDC WD5002ABYS-18B1B0, S/N:WD-W... [15:27:43] RECOVERY - puppet last run on mw1289 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [15:31:03] PROBLEM - puppet last run on sca2003 is CRITICAL: CRITICAL: Puppet has 27 failures. Last run 2 minutes ago with 27 failures. Failed resources (up to 3 shown): Exec[eth0_v6_token],Package[zotero/translators],Package[zotero/translation-server],Exec[chown /srv/deployment/zotero for deploy-service] [15:34:53] PROBLEM - puppet last run on db1061 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [15:45:50] (03PS1) 10Alexandros Kosiaris: osm: Use LABS_NETWORKS in ferm rsync rule [puppet] - 10https://gerrit.wikimedia.org/r/331622 [15:45:52] (03PS1) 10Alexandros Kosiaris: osm: Add a prometheus textfile exporter [puppet] - 10https://gerrit.wikimedia.org/r/331623 [16:00:03] RECOVERY - puppet last run on sca2003 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [16:01:22] 06Operations, 10ops-eqiad: Rack and set up ms-fe100[5-7] - https://phabricator.wikimedia.org/T155095#2933393 (10Cmjohnson) [16:01:55] (03PS1) 10Cmjohnson: Adding mgmt and production dns entries for ms-fe100[5-7] T155095 [dns] - 10https://gerrit.wikimedia.org/r/331625 [16:03:53] RECOVERY - puppet last run on db1061 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [16:35:22] (03Abandoned) 10Umherirrender: Expand .gitignore for more editors [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331138 (owner: 10Umherirrender) [16:38:03] PROBLEM - Outgoing network saturation on labstore1003 is CRITICAL: CRITICAL: 13.33% of data above the critical threshold [106250000.0] [16:38:40] (03CR) 10Alex Monk: "do we really want to use this group when its name is always going to be slightly misleading? maybe add a description to it at least?" [puppet] - 10https://gerrit.wikimedia.org/r/331602 (owner: 10Alexandros Kosiaris) [16:45:03] RECOVERY - Outgoing network saturation on labstore1003 is OK: OK: Less than 10.00% above the threshold [93750000.0] [16:45:43] PROBLEM - check mtime mod from tools cron job on checker.tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 SERVICE UNAVAILABLE - string OK not found on http://checker.tools.wmflabs.org:80/toolscron - 185 bytes in 0.008 second response time [16:48:03] PROBLEM - Outgoing network saturation on labstore1003 is CRITICAL: CRITICAL: 13.33% of data above the critical threshold [106250000.0] [16:49:21] (03CR) 10Cmjohnson: [C: 032] Adding mgmt and production dns entries for ms-fe100[5-7] T155095 [dns] - 10https://gerrit.wikimedia.org/r/331625 (owner: 10Cmjohnson) [16:52:43] RECOVERY - check mtime mod from tools cron job on checker.tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 166 bytes in 0.005 second response time [17:02:32] 06Operations, 06Commons, 10TimedMediaHandler-Transcode, 10Wikimedia-Video, and 3 others: Commons video transcoders have over 6500 tasks in the backlog. - https://phabricator.wikimedia.org/T153488#2933535 (10matmarex) [17:03:16] (03PS1) 10Hashar: aptrepo: fix spec on Mac OS X [puppet] - 10https://gerrit.wikimedia.org/r/331632 [17:04:03] PROBLEM - Outgoing network saturation on labstore1003 is CRITICAL: CRITICAL: 13.33% of data above the critical threshold [106250000.0] [17:11:45] 06Operations, 06Commons, 10TimedMediaHandler-Transcode, 10Wikimedia-Video, and 3 others: Commons video transcoders have over 6500 tasks in the backlog. - https://phabricator.wikimedia.org/T153488#2933561 (10brion) [17:20:24] (03PS1) 10Hashar: install_server: mock standard class for tests [puppet] - 10https://gerrit.wikimedia.org/r/331635 [17:20:58] (03PS1) 10Alex Monk: Revert "Revert "labtest hiera: use labtestwikitech, not wikitech"" [puppet] - 10https://gerrit.wikimedia.org/r/331636 (https://phabricator.wikimedia.org/T145808) [17:21:03] RECOVERY - Outgoing network saturation on labstore1003 is OK: OK: Less than 10.00% above the threshold [93750000.0] [17:26:38] 06Operations, 07Wikimedia-Multiple-active-datacenters: Prepare and improve the datacenter switchover procedure - https://phabricator.wikimedia.org/T154658#2933589 (10Marostegui) [17:27:13] 06Operations, 07Epic, 07Wikimedia-Multiple-active-datacenters: Prepare and improve the datacenter switchover procedure - https://phabricator.wikimedia.org/T154658#2933591 (10mark) [17:27:38] (03PS1) 10Alex Monk: Use LE for wikitech [puppet] - 10https://gerrit.wikimedia.org/r/331638 (https://phabricator.wikimedia.org/T154913) [17:28:53] PROBLEM - puppet last run on db1028 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [17:30:00] (03Abandoned) 10Dereckson: Unassign 'transcode-reset' from Commons autoconfirmed and sysop groups [mediawiki-config] - 10https://gerrit.wikimedia.org/r/330835 (https://phabricator.wikimedia.org/T154733) (owner: 10Zhuyifei1999) [17:32:19] 06Operations, 05DC-Switchover-Prep-Q3-2016, 07Epic, 07Wikimedia-Multiple-active-datacenters: Prepare and improve the datacenter switchover procedure - https://phabricator.wikimedia.org/T154658#2933600 (10Joe) [17:32:33] PROBLEM - citoid endpoints health on scb1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [17:34:23] RECOVERY - citoid endpoints health on scb1001 is OK: All endpoints are healthy [17:34:43] PROBLEM - puppet last run on sca1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [17:38:18] (03PS1) 10Hashar: (WP)mirrors: fix spec [puppet] - 10https://gerrit.wikimedia.org/r/331639 [17:39:07] (03CR) 10Hashar: "That is a WIP :} Guess I should start writing documentation about rspec-puppet and the puppet labs spec helper!" [puppet] - 10https://gerrit.wikimedia.org/r/331639 (owner: 10Hashar) [17:40:43] PROBLEM - puppet last run on db1020 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [17:52:47] (03PS2) 10Alex Monk: designate-sink nova_ldap: set l to the correct site [puppet] - 10https://gerrit.wikimedia.org/r/312115 [17:56:53] RECOVERY - puppet last run on db1028 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [18:01:43] RECOVERY - puppet last run on sca1004 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [18:03:43] (03CR) 10RobH: [C: 031] "This looks good to me. I'll plan to coordinate with someone in the labs team to put together an announce for a maint window, so we can pu" [puppet] - 10https://gerrit.wikimedia.org/r/331638 (https://phabricator.wikimedia.org/T154913) (owner: 10Alex Monk) [18:04:04] (03PS1) 10Filippo Giunchedi: prometheus: add cassandra jobs in beta [puppet] - 10https://gerrit.wikimedia.org/r/331644 [18:07:30] (03CR) 10Andrew Bogott: [C: 032] designate-sink nova_ldap: set l to the correct site [puppet] - 10https://gerrit.wikimedia.org/r/312115 (owner: 10Alex Monk) [18:07:34] CI should get into "get stuff done" mode too heh [18:07:38] (03PS3) 10Andrew Bogott: designate-sink nova_ldap: set l to the correct site [puppet] - 10https://gerrit.wikimedia.org/r/312115 (owner: 10Alex Monk) [18:08:10] !log restart elasticsaerch on relforge100[12] for new test version of ltr plugin [18:08:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:09:43] RECOVERY - puppet last run on db1020 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [18:10:22] godog: Do you think we could poke https://gerrit.wikimedia.org/r/#/c/330709/? [18:10:33] 06Operations, 10Traffic, 13Patch-For-Review: convert wikitech.wikimedia.org from globalsign to letsencrypt certificate (deadline 2017-02-24) - https://phabricator.wikimedia.org/T154913#2933695 (10RobH) p:05Triage>03High a:05Krenair>03Andrew [18:10:40] Trying to standardize the docroots to all be *.org [18:10:49] wikidata is a special snowflake for no particular reason [18:11:33] (03PS2) 10Reedy: Get rid of $wmg hack for MassMessage settings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/237686 (owner: 10Legoktm) [18:12:15] 06Operations, 10Traffic, 13Patch-For-Review: convert wikitech.wikimedia.org from globalsign to letsencrypt certificate (deadline 2017-02-24) - https://phabricator.wikimedia.org/T154913#2928272 (10RobH) Next steps: We need to schedule a time with the labs team (likely @andrew) for a maint window for wikitech... [18:17:00] (03CR) 10Filippo Giunchedi: [C: 032] "In production both wikidata and wikidata.org already point to standard-docroot" [puppet] - 10https://gerrit.wikimedia.org/r/330709 (owner: 10Chad) [18:19:46] ostriches: yeah LGTM! I'm sure it is a noop effectively in production, I can babysit it at the next puppet swat [18:21:13] (03CR) 10Filippo Giunchedi: [C: 032] prometheus: add cassandra jobs in beta [puppet] - 10https://gerrit.wikimedia.org/r/331644 (owner: 10Filippo Giunchedi) [18:21:19] (03PS2) 10Filippo Giunchedi: prometheus: add cassandra jobs in beta [puppet] - 10https://gerrit.wikimedia.org/r/331644 [18:21:24] (03CR) 10Filippo Giunchedi: [V: 032 C: 032] prometheus: add cassandra jobs in beta [puppet] - 10https://gerrit.wikimedia.org/r/331644 (owner: 10Filippo Giunchedi) [18:23:49] godog: Ty! [18:24:39] (03Abandoned) 10Reedy: Enforce password policies on labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/276518 (https://phabricator.wikimedia.org/T119100) (owner: 10CSteipp) [18:26:01] (03PS2) 10Reedy: Redo local password enforcement [mediawiki-config] - 10https://gerrit.wikimedia.org/r/289780 (https://phabricator.wikimedia.org/T119736) (owner: 10CSteipp) [18:26:13] (03CR) 10jerkins-bot: [V: 04-1] Redo local password enforcement [mediawiki-config] - 10https://gerrit.wikimedia.org/r/289780 (https://phabricator.wikimedia.org/T119736) (owner: 10CSteipp) [18:26:36] (03CR) 10Chad: [C: 032] Reinstate "Remove MWVersion, fold its two functions into MWMultiVersion" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331552 (owner: 10Reedy) [18:28:34] (03Merged) 10jenkins-bot: Reinstate "Remove MWVersion, fold its two functions into MWMultiVersion" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331552 (owner: 10Reedy) [18:33:40] (03CR) 10Legoktm: "Yes, it can go forward." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/237686 (owner: 10Legoktm) [18:36:23] (03CR) 10jenkins-bot: Reinstate "Remove MWVersion, fold its two functions into MWMultiVersion" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331552 (owner: 10Reedy) [18:36:46] (03CR) 10Reedy: [C: 04-1] "I'd like to stop tracking master of PageForms if we're going to switch..." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/327307 (owner: 1020after4) [18:56:11] !log demon@tin Synchronized multiversion/MWMultiVersion.php: Attempt #2 for Multiversion cleanup (duration: 00m 41s) [18:56:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:56:46] Nikerabbit: fyi... "Translation search server unavailable:Result window is too large, from + size must be less than or equal to: [10000] but was [11000]. See the scroll api for a more efficient way to request large data sets. This limit can be set by changing the [index.max_result_window] index level parameter." [18:56:50] Seeing a ton of that on logstash [18:58:18] (03CR) 10Muehlenhoff: [C: 031] "Seems fine" [puppet] - 10https://gerrit.wikimedia.org/r/331494 (owner: 10Hashar) [19:00:04] addshore, hashar, anomie, ostriches, aude, MaxSem, twentyafterfour, RoanKattouw, Dereckson, and thcipriani: Respected human, time to deploy Morning SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20170111T1900). Please do the needful. [19:00:18] Nothing on swat, I'm stealing the window with Reedy [19:04:13] PROBLEM - puppet last run on mc1034 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [19:04:27] (03CR) 10Muehlenhoff: [C: 031] "Looks good to me" [puppet] - 10https://gerrit.wikimedia.org/r/331622 (owner: 10Alexandros Kosiaris) [19:09:18] 07Puppet, 06Labs, 10MediaWiki-Vagrant, 13Patch-For-Review, 15User-bd808: Make role::labs::mediawiki_vagrant work on Debian Jessie host systems - https://phabricator.wikimedia.org/T154340#2933915 (10bd808) [19:11:03] PROBLEM - DPKG on labtestneutron2001 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [19:11:04] 06Operations, 10MediaWiki-Vagrant: Upgrade Vagrant to 1.9.1 in Wikimedia apt for both Trusty and Jessie - https://phabricator.wikimedia.org/T155112#2933901 (10bd808) @akosiaris is this something that you could help me solve? [19:14:03] RECOVERY - DPKG on labtestneutron2001 is OK: All packages OK [19:15:11] ostriches: ack, seen that in translatewiki.net sometimes too but I have been unable to reproduce [19:17:13] PROBLEM - Check systemd state on notebook1001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [19:17:45] Nikerabbit: I'm dealing with something else right now, but let's see if we can repro in prod. It's trending really high on the error logs. [19:18:31] ostriches: I wonder if it is some kind of bot crawling that page... [19:18:49] Hmm, I dunno. I know nothing about this service [19:23:31] (03PS1) 10Brion VIBBER: Split TMH transcode queue into two for prioritization [puppet] - 10https://gerrit.wikimedia.org/r/331668 (https://phabricator.wikimedia.org/T155098) [19:25:32] (03PS1) 10Brion VIBBER: Exclude new high-priority video transcode jobs from default queue [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331669 (https://phabricator.wikimedia.org/T155098) [19:27:17] !log demon@tin Synchronized php-1.29.0-wmf.7/extensions/FlaggedRevs: Stupid errors (duration: 00m 46s) [19:27:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:27:53] notebook1001 is me - i got it [19:28:13] RECOVERY - Check systemd state on notebook1001 is OK: OK - running: The system is fully operational [19:31:13] RECOVERY - puppet last run on mc1034 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [19:34:45] !log demon@tin Synchronized multiversion: MWVersion fallbacks & such (duration: 00m 56s) [19:35:20] ok, who did it [19:35:20] I just saw a 500 error [19:35:35] ostriches [19:35:41] Already rolling back [19:35:43] it's stopped now [19:35:46] something about stdClass::get [19:35:50] Yeah bizzare [19:35:50] hmm wikitech is giving me 500... [19:35:58] I wonder if that was just transient [19:35:59] Call to undefined method stdClass::get() [19:36:02] Away? [19:36:02] Hello guys, is there a technical problem? [19:36:06] not just wikitech, SMalyshev [19:36:10] enwiki and everything [19:36:12] Penskins: should be good now [19:36:13] So Wikisource just went? [19:36:15] Yes, we know [19:36:16] enterprisey: I see [19:36:16] Just wait [19:36:19] everything, ShakespeareFan00 [19:36:19] !log demon@tin Synchronized multiversion: rollback (duration: 00m 56s) [19:36:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:36:23] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:36:23] PROBLEM - Mobileapps LVS codfw on mobileapps.svc.codfw.wmnet is CRITICAL: /{domain}/v1/page/mobile-sections-lead/{title} (retrieve lead section of en.wp Altrincham page via mobile-sections-lead) is CRITICAL: Test retrieve lead section of en.wp Altrincham page via mobile-sections-lead returned the unexpected status 500 (expecting: 200) [19:36:42] enwiki is fine for me (maybe its cached) [19:37:07] it was just rolled back SMalyshev [19:37:23] RECOVERY - Mobileapps LVS codfw on mobileapps.svc.codfw.wmnet is OK: All endpoints are healthy [19:37:23] PROBLEM - restbase endpoints health on restbase1012 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [19:37:40] ah wikitech is back to normal [19:37:43] PROBLEM - Apache HTTP on mw1263 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 50419 bytes in 0.004 second response time [19:37:43] PROBLEM - Nginx local proxy to apache on mw1263 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 50461 bytes in 0.011 second response time [19:37:53] PROBLEM - HHVM rendering on mwdebug1002 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 50424 bytes in 0.008 second response time [19:37:53] PROBLEM - HHVM rendering on mw1263 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 50419 bytes in 0.004 second response time [19:38:23] RECOVERY - restbase endpoints health on restbase1012 is OK: All endpoints are healthy [19:38:43] PROBLEM - Nginx local proxy to apache on mwdebug1002 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 50472 bytes in 0.016 second response time [19:38:53] PROBLEM - Eqiad HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [1000.0] [19:38:53] PROBLEM - Apache HTTP on mwdebug1002 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 50424 bytes in 0.005 second response time [19:38:53] PROBLEM - Esams HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [1000.0] [19:39:33] PROBLEM - citoid endpoints health on scb1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [19:39:52] Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find data item network::subnets in any Hiera data file and no default supplied at /etc/puppet/modules/network/manifests/constants.pp:19 on node wdq-beta.wikidata-query.eqiad.wmflabs [19:39:52] Looks like my puppet in labs is broken :( [19:40:15] SMalyshev, when wikitech breaks that happens [19:40:17] 06Operations, 10MediaWiki-Configuration, 06Performance-Team, 06Services (watching), and 5 others: Integrating MediaWiki (and other services) with dynamic configuration - https://phabricator.wikimedia.org/T149617#2933969 (10Joe) p:05Triage>03Normal [19:40:23] RECOVERY - citoid endpoints health on scb1001 is OK: All endpoints are healthy [19:40:43] RECOVERY - Nginx local proxy to apache on mwdebug1002 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 618 bytes in 0.060 second response time [19:40:43] PROBLEM - Ulsfo HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [1000.0] [19:40:51] bit too slow there icinga [19:40:53] RECOVERY - Apache HTTP on mwdebug1002 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 617 bytes in 0.056 second response time [19:40:53] RECOVERY - Esams HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] [19:40:53] RECOVERY - HHVM rendering on mwdebug1002 is OK: HTTP OK: HTTP/1.1 200 OK - 73540 bytes in 0.109 second response time [19:41:09] (03PS1) 10Chad: Fuck you Multiversion [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331673 [19:41:33] (03CR) 10Chad: [V: 032 C: 032] Fuck you Multiversion [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331673 (owner: 10Chad) [19:41:47] (03CR) 10jenkins-bot: Fuck you Multiversion [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331673 (owner: 10Chad) [19:42:43] PROBLEM - Text HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [1000.0] [19:43:53] PROBLEM - Esams HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [1000.0] [19:47:33] PROBLEM - citoid endpoints health on scb1002 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [19:47:33] PROBLEM - citoid endpoints health on scb1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [19:47:43] RECOVERY - Text HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] [19:48:33] PROBLEM - citoid endpoints health on scb1003 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [19:48:53] RECOVERY - Esams HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] [19:49:23] RECOVERY - citoid endpoints health on scb1003 is OK: All endpoints are healthy [19:49:23] RECOVERY - citoid endpoints health on scb1002 is OK: All endpoints are healthy [19:49:23] RECOVERY - citoid endpoints health on scb1001 is OK: All endpoints are healthy [19:49:43] RECOVERY - Ulsfo HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] [19:50:53] RECOVERY - Eqiad HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] [19:51:58] (03PS2) 10Reedy: Add recent dblist files on noc. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331436 (owner: 10Dereckson) [19:52:00] (03CR) 10Reedy: [C: 032] Add recent dblist files on noc. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331436 (owner: 10Dereckson) [19:52:06] (03CR) 10Reedy: [C: 032] Add recent dblist files on noc. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331436 (owner: 10Dereckson) [19:53:13] (03Abandoned) 10Chad: WIP: Work towards not needed MWMinimalScriptInit.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/309601 (owner: 10Chad) [19:53:33] PROBLEM - citoid endpoints health on scb2001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [19:54:23] RECOVERY - citoid endpoints health on scb2001 is OK: All endpoints are healthy [19:54:55] (03CR) 10Chad: [C: 032] Add a test to prevent people from making new dblist files without appropriate noc symlinks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331441 (owner: 10Alex Monk) [19:55:40] (03Merged) 10jenkins-bot: Add recent dblist files on noc. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331436 (owner: 10Dereckson) [19:55:53] (03CR) 10jenkins-bot: Add recent dblist files on noc. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331436 (owner: 10Dereckson) [19:56:12] (03PS3) 10Chad: Labs: remove wmgUseGWToolset [mediawiki-config] - 10https://gerrit.wikimedia.org/r/328870 (owner: 10MaxSem) [19:56:36] (03CR) 10Chad: [C: 032] Labs: remove wmgUseGWToolset [mediawiki-config] - 10https://gerrit.wikimedia.org/r/328870 (owner: 10MaxSem) [19:57:25] (03CR) 10Chad: [C: 032] Labs: remove wmgUseCommonsMetadata [mediawiki-config] - 10https://gerrit.wikimedia.org/r/328868 (owner: 10MaxSem) [19:57:27] (03CR) 10Chad: [C: 032] Labs: remove unused wmgUseWebFonts [mediawiki-config] - 10https://gerrit.wikimedia.org/r/328864 (owner: 10MaxSem) [19:57:35] (03CR) 10Chad: [C: 032] Labs: remove TMH and MwEmbedSupport override [mediawiki-config] - 10https://gerrit.wikimedia.org/r/328867 (owner: 10MaxSem) [19:57:51] (03CR) 10Chad: [C: 032] Labs: remove wmgEchoMentionStatusNotifications [mediawiki-config] - 10https://gerrit.wikimedia.org/r/328866 (owner: 10MaxSem) [19:57:59] (03CR) 10Chad: [C: 032] Labs: remove unused wmgNoticeProject [mediawiki-config] - 10https://gerrit.wikimedia.org/r/328863 (owner: 10MaxSem) [19:58:03] (03CR) 10Chad: [C: 032] Labs: remove wmgUseEcho overrides [mediawiki-config] - 10https://gerrit.wikimedia.org/r/328865 (owner: 10MaxSem) [19:58:23] PROBLEM - citoid endpoints health on scb2003 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [19:58:33] what's going on? [19:58:33] PROBLEM - restbase endpoints health on restbase2011 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [19:58:43] (03Merged) 10jenkins-bot: Labs: remove wmgUseGWToolset [mediawiki-config] - 10https://gerrit.wikimedia.org/r/328870 (owner: 10MaxSem) [19:58:53] (03CR) 10jenkins-bot: Labs: remove wmgUseGWToolset [mediawiki-config] - 10https://gerrit.wikimedia.org/r/328870 (owner: 10MaxSem) [19:59:13] RECOVERY - citoid endpoints health on scb2003 is OK: All endpoints are healthy [19:59:18] greg-g: Just clearing a backlog :) [19:59:23] RECOVERY - restbase endpoints health on restbase2011 is OK: All endpoints are healthy [19:59:31] greg-g, a multiversion change caused some HTTP 500s [19:59:38] (03CR) 10jerkins-bot: [V: 04-1] Labs: remove unused wmgUseWebFonts [mediawiki-config] - 10https://gerrit.wikimedia.org/r/328864 (owner: 10MaxSem) [19:59:40] (03CR) 10jerkins-bot: [V: 04-1] Labs: remove wmgUseEcho overrides [mediawiki-config] - 10https://gerrit.wikimedia.org/r/328865 (owner: 10MaxSem) [19:59:42] (03CR) 10jerkins-bot: [V: 04-1] Labs: remove wmgEchoMentionStatusNotifications [mediawiki-config] - 10https://gerrit.wikimedia.org/r/328866 (owner: 10MaxSem) [19:59:44] (03CR) 10jerkins-bot: [V: 04-1] Labs: remove TMH and MwEmbedSupport override [mediawiki-config] - 10https://gerrit.wikimedia.org/r/328867 (owner: 10MaxSem) [19:59:44] ostriches: I just got a ping from a friend telling me wikipedia was down :) [19:59:46] (03CR) 10jerkins-bot: [V: 04-1] Labs: remove wmgUseCommonsMetadata [mediawiki-config] - 10https://gerrit.wikimedia.org/r/328868 (owner: 10MaxSem) [19:59:48] ahhh, thanks Krenair [19:59:55] greg-g: just? :P It's been fixed ;) [19:59:57] greg-g: That was already rolled back, your friend was late :p [19:59:58] a bunch of users noticed [20:00:09] 19:35 < jrwren> greg-g: don't get fired! [20:00:17] helpful advice [20:00:29] (03PS2) 10Chad: Labs: remove unused wmgNoticeProject [mediawiki-config] - 10https://gerrit.wikimedia.org/r/328863 (owner: 10MaxSem) [20:00:39] :) so yeah, same time as in here, I'm just in the MW Stakeholder's meeting [20:00:50] (03PS2) 10Chad: Labs: remove wmgUseEcho overrides [mediawiki-config] - 10https://gerrit.wikimedia.org/r/328865 (owner: 10MaxSem) [20:01:00] so users are stakeholders :) [20:01:12] (03PS2) 10Chad: Labs: remove wmgEchoMentionStatusNotifications [mediawiki-config] - 10https://gerrit.wikimedia.org/r/328866 (owner: 10MaxSem) [20:01:19] (03PS2) 10Chad: Labs: remove TMH and MwEmbedSupport override [mediawiki-config] - 10https://gerrit.wikimedia.org/r/328867 (owner: 10MaxSem) [20:01:25] (03PS2) 10Chad: Labs: remove wmgUseCommonsMetadata [mediawiki-config] - 10https://gerrit.wikimedia.org/r/328868 (owner: 10MaxSem) [20:01:52] (03PS2) 10Chad: Labs: remove unused wmgUseWebFonts [mediawiki-config] - 10https://gerrit.wikimedia.org/r/328864 (owner: 10MaxSem) [20:04:27] matanya: meh :P [20:04:43] PROBLEM - puppet last run on sca1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [20:04:49] If greg-g got fired because of what his team did... [20:04:53] He would've been fired a long time ago [20:05:26] (03PS3) 10Chad: Labs: remove wmgUseEcho overrides [mediawiki-config] - 10https://gerrit.wikimedia.org/r/328865 (owner: 10MaxSem) [20:05:30] (03CR) 10jenkins-bot: Labs: remove unused wmgNoticeProject [mediawiki-config] - 10https://gerrit.wikimedia.org/r/328863 (owner: 10MaxSem) [20:07:51] quote of the day Reedy [20:11:39] (03CR) 10jenkins-bot: Labs: remove wmgUseEcho overrides [mediawiki-config] - 10https://gerrit.wikimedia.org/r/328865 (owner: 10MaxSem) [20:12:44] (03PS3) 10Chad: Labs: remove TMH and MwEmbedSupport override [mediawiki-config] - 10https://gerrit.wikimedia.org/r/328867 (owner: 10MaxSem) [20:18:45] (03PS4) 10Dzahn: base: missing quotes around is_virtual 'false' for ipmi [puppet] - 10https://gerrit.wikimedia.org/r/331579 (https://phabricator.wikimedia.org/T150160) [20:19:34] (03CR) 10jenkins-bot: Labs: remove TMH and MwEmbedSupport override [mediawiki-config] - 10https://gerrit.wikimedia.org/r/328867 (owner: 10MaxSem) [20:19:36] (03PS3) 10Chad: Labs: remove unused wmgUseWebFonts [mediawiki-config] - 10https://gerrit.wikimedia.org/r/328864 (owner: 10MaxSem) [20:22:47] (03CR) 10Reedy: [C: 04-1] "Also needs extension-list-labs fixing" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/327307 (owner: 1020after4) [20:22:53] (03Abandoned) 10Reedy: Wikitech: Switching from using SemanticForms to PageForms extension [mediawiki-config] - 10https://gerrit.wikimedia.org/r/319131 (https://phabricator.wikimedia.org/T149749) (owner: 10Paladox) [20:24:10] (03CR) 10Dzahn: [C: 032] base: missing quotes around is_virtual 'false' for ipmi [puppet] - 10https://gerrit.wikimedia.org/r/331579 (https://phabricator.wikimedia.org/T150160) (owner: 10Dzahn) [20:27:00] (03PS1) 10Hashar: nrpe: fix spec [puppet] - 10https://gerrit.wikimedia.org/r/331677 [20:27:18] (03PS3) 10Chad: Labs: remove wmgUseCommonsMetadata [mediawiki-config] - 10https://gerrit.wikimedia.org/r/328868 (owner: 10MaxSem) [20:27:23] 06Operations, 13Patch-For-Review: Remote IPMI doesn't work for ~17% of the fleet - https://phabricator.wikimedia.org/T150160#2934122 (10Dzahn) @volans After the merge above, i could confirm that the 3 freeipmi packages got installed on iridium (which was on the list) and nothing broke on phab2001 which already... [20:28:29] (03CR) 10jenkins-bot: Labs: remove unused wmgUseWebFonts [mediawiki-config] - 10https://gerrit.wikimedia.org/r/328864 (owner: 10MaxSem) [20:30:26] (03CR) 10Dzahn: "oh! thank you very much for this! had the bug on several runs and this should fix all of them :)" [puppet] - 10https://gerrit.wikimedia.org/r/331592 (owner: 10Alexandros Kosiaris) [20:32:43] RECOVERY - puppet last run on sca1004 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [20:32:46] (03PS2) 10Dzahn: icinga: missing trailing commas [puppet] - 10https://gerrit.wikimedia.org/r/331516 [20:34:01] (03PS3) 10Chad: Labs: remove wmgEchoMentionStatusNotifications [mediawiki-config] - 10https://gerrit.wikimedia.org/r/328866 (owner: 10MaxSem) [20:34:05] (03CR) 10jenkins-bot: Labs: remove wmgUseCommonsMetadata [mediawiki-config] - 10https://gerrit.wikimedia.org/r/328868 (owner: 10MaxSem) [20:35:32] (03PS5) 10MarcoAurelio: Removing 'technican' user group from tr.wikiquote [mediawiki-config] - 10https://gerrit.wikimedia.org/r/326354 (https://phabricator.wikimedia.org/T152911) [20:36:24] (03CR) 10Dzahn: "thank you! love it that einsteinium changes compile again" [puppet] - 10https://gerrit.wikimedia.org/r/331516 (owner: 10Dzahn) [20:37:17] (03CR) 10Dzahn: [C: 032] icinga: missing trailing commas [puppet] - 10https://gerrit.wikimedia.org/r/331516 (owner: 10Dzahn) [20:39:06] (03CR) 10jenkins-bot: Labs: remove wmgEchoMentionStatusNotifications [mediawiki-config] - 10https://gerrit.wikimedia.org/r/328866 (owner: 10MaxSem) [20:40:23] !log demon@tin Synchronized wmf-config/InitialiseSettings-labs.php: no-op (duration: 00m 40s) [20:40:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:42:16] (03CR) 10jenkins-bot: Add a test to prevent people from making new dblist files without appropriate noc symlinks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331441 (owner: 10Alex Monk) [20:43:15] !log demon@tin Synchronized tests/noc-conf/NOCDblistTest.php: No-op (duration: 00m 40s) [20:43:18] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:43:37] (03PS2) 10Reedy: Fix University of Mumbai event throttle rule [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331577 (https://phabricator.wikimedia.org/T154312) (owner: 10Dereckson) [20:43:43] (03CR) 10Reedy: [C: 032] Fix University of Mumbai event throttle rule [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331577 (https://phabricator.wikimedia.org/T154312) (owner: 10Dereckson) [20:44:51] !log Reset user e-mail for account for Panam2014 [20:44:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:45:05] (03Merged) 10jenkins-bot: Fix University of Mumbai event throttle rule [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331577 (https://phabricator.wikimedia.org/T154312) (owner: 10Dereckson) [20:45:19] (03CR) 10jenkins-bot: Fix University of Mumbai event throttle rule [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331577 (https://phabricator.wikimedia.org/T154312) (owner: 10Dereckson) [20:45:24] ah thanks Reedy, I was going to include that at evening SWAT [20:46:17] (03PS3) 10Reedy: Upgrade Collection's license URL to HTTPS [mediawiki-config] - 10https://gerrit.wikimedia.org/r/329023 (owner: 10MaxSem) [20:46:22] (03CR) 10Reedy: [C: 032] Upgrade Collection's license URL to HTTPS [mediawiki-config] - 10https://gerrit.wikimedia.org/r/329023 (owner: 10MaxSem) [20:47:43] (03Merged) 10jenkins-bot: Upgrade Collection's license URL to HTTPS [mediawiki-config] - 10https://gerrit.wikimedia.org/r/329023 (owner: 10MaxSem) [20:48:17] (03CR) 10jenkins-bot: Upgrade Collection's license URL to HTTPS [mediawiki-config] - 10https://gerrit.wikimedia.org/r/329023 (owner: 10MaxSem) [20:49:19] (03CR) 10Dzahn: [C: 04-1] "modules/openstack/templates/common/wikitech.wikimedia.org.erb needs to be adjusted as well. It needs the new /etc/acme/ pathes for cert/ch" [puppet] - 10https://gerrit.wikimedia.org/r/331638 (https://phabricator.wikimedia.org/T154913) (owner: 10Alex Monk) [20:49:27] (03PS2) 10Reedy: Set Translation namespace on ml.wikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331313 (https://phabricator.wikimedia.org/T154087) (owner: 10Dereckson) [20:49:31] (03CR) 10Reedy: [C: 032] Set Translation namespace on ml.wikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331313 (https://phabricator.wikimedia.org/T154087) (owner: 10Dereckson) [20:51:50] (03CR) 10Dzahn: [C: 04-1] "i probably would not have set the old cert absent (in the same change) and instead manually deleted it after confirming it all works. but " [puppet] - 10https://gerrit.wikimedia.org/r/331638 (https://phabricator.wikimedia.org/T154913) (owner: 10Alex Monk) [20:51:51] (03PS2) 10Reedy: Set $wgCategoryCollation for Finnish Wikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/326409 (https://phabricator.wikimedia.org/T151570) (owner: 10Odder) [20:51:59] (03Merged) 10jenkins-bot: Set Translation namespace on ml.wikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331313 (https://phabricator.wikimedia.org/T154087) (owner: 10Dereckson) [20:52:10] (03CR) 10jenkins-bot: Set Translation namespace on ml.wikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331313 (https://phabricator.wikimedia.org/T154087) (owner: 10Dereckson) [20:52:23] !log reedy@tin Synchronized wmf-config/throttle.php: Fix throttle (duration: 00m 42s) [20:52:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:53:54] !log reedy@tin Synchronized wmf-config/CommonSettings.php: Upgrade Collections license URL to HTTPS (duration: 00m 57s) [20:53:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:54:04] (03CR) 10Reedy: [C: 032] Set $wgCategoryCollation for Finnish Wikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/326409 (https://phabricator.wikimedia.org/T151570) (owner: 10Odder) [20:55:39] (03CR) 10Dzahn: "sorry, you are right we should have added more info why exactly we removed it. i just remember after migration to jessie and Apache 2.4 we" [puppet] - 10https://gerrit.wikimedia.org/r/331558 (https://phabricator.wikimedia.org/T150727) (owner: 10Krinkle) [20:56:50] (03PS13) 10Paladox: Gerrit: Add support for logstash in gerrit [puppet] - 10https://gerrit.wikimedia.org/r/330832 (https://phabricator.wikimedia.org/T141324) [20:57:06] (03CR) 10Paladox: "Are we going to give this another go today?" [puppet] - 10https://gerrit.wikimedia.org/r/330832 (https://phabricator.wikimedia.org/T141324) (owner: 10Paladox) [20:57:10] (03PS3) 10Dzahn: mailman: Indent @ssl_settings in Apache configuration [puppet] - 10https://gerrit.wikimedia.org/r/329742 (owner: 10Tim Landscheidt) [20:57:57] (03CR) 10Hashar: "Alexandros: I took care of deploying the change during the European SWAT Window." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331093 (https://phabricator.wikimedia.org/T85947) (owner: 10Krinkle) [20:57:59] (03PS3) 10Reedy: Set $wgCategoryCollation for Finnish Wikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/326409 (https://phabricator.wikimedia.org/T151570) (owner: 10Odder) [20:58:02] (03CR) 10Reedy: Set $wgCategoryCollation for Finnish Wikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/326409 (https://phabricator.wikimedia.org/T151570) (owner: 10Odder) [20:58:06] (03CR) 10Reedy: [C: 032] Set $wgCategoryCollation for Finnish Wikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/326409 (https://phabricator.wikimedia.org/T151570) (owner: 10Odder) [20:58:14] (03CR) 10Dzahn: [C: 032] mailman: Indent @ssl_settings in Apache configuration [puppet] - 10https://gerrit.wikimedia.org/r/329742 (owner: 10Tim Landscheidt) [20:59:29] (03Merged) 10jenkins-bot: Set $wgCategoryCollation for Finnish Wikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/326409 (https://phabricator.wikimedia.org/T151570) (owner: 10Odder) [20:59:39] (03CR) 10jenkins-bot: Set $wgCategoryCollation for Finnish Wikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/326409 (https://phabricator.wikimedia.org/T151570) (owner: 10Odder) [21:00:04] gwicke, cscott, arlolra, subbu, bearND, halfak, Amir1, and yurik: Dear anthropoid, the time has come. Please deploy Services – Parsoid / OCG / Citoid / Mobileapps / ORES / … (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20170111T2100). [21:00:12] (03CR) 10Dzahn: [C: 031] puppetdb: Do not set up Ganglia in Labs [puppet] - 10https://gerrit.wikimedia.org/r/329329 (https://phabricator.wikimedia.org/T154104) (owner: 10Tim Landscheidt) [21:01:09] !log reedy@tin Synchronized wmf-config/InitialiseSettings.php: Translation namespace for mlwikisource. fiwikivoyage collation (duration: 00m 40s) [21:01:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:01:41] (03CR) 10Dzahn: [C: 031] mirrors: Indent @ssl_settings in NGINX configuration [puppet] - 10https://gerrit.wikimedia.org/r/329743 (owner: 10Tim Landscheidt) [21:01:45] (03CR) 10Dzahn: [C: 031] dynamicproxy: Indent @ssl_settings in NGINX configurations [puppet] - 10https://gerrit.wikimedia.org/r/329750 (owner: 10Tim Landscheidt) [21:01:52] (03CR) 10Dzahn: [V: 031 C: 031] docker: Indent @ssl_settings in NGINX configuration [puppet] - 10https://gerrit.wikimedia.org/r/329735 (owner: 10Tim Landscheidt) [21:01:55] (03CR) 10Dzahn: [V: 031 C: 031] Tools: Indent @ssl_settings in NGINX configuration [puppet] - 10https://gerrit.wikimedia.org/r/329749 (owner: 10Tim Landscheidt) [21:01:59] (03CR) 10Dzahn: [V: 031 C: 031] toolserver_legacy: Indent @ssl_settings in Apache configuration [puppet] - 10https://gerrit.wikimedia.org/r/329748 (owner: 10Tim Landscheidt) [21:02:03] (03CR) 10Dzahn: [V: 031 C: 031] puppetmaster: Indent @ssl_settings in Apache and NGINX configurations [puppet] - 10https://gerrit.wikimedia.org/r/329745 (owner: 10Tim Landscheidt) [21:02:39] (03CR) 10Dzahn: [V: 031 C: 031] openstack: Indent @ssl_settings in Apache configuration [puppet] - 10https://gerrit.wikimedia.org/r/329744 (owner: 10Tim Landscheidt) [21:02:56] !log reedy@tin Synchronized php-1.29.0-wmf.7/extensions/CentralAuth/extension.json: fix name (duration: 00m 41s) [21:03:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:03:11] !log update collation of fiwikivoyage T151570 [21:03:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:03:20] T151570: Create Wikivoyage Finnish - https://phabricator.wikimedia.org/T151570 [21:03:44] PROBLEM - puppet last run on ms-be1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [21:04:10] (03PS6) 10Reedy: Removing 'technican' user group from tr.wikiquote [mediawiki-config] - 10https://gerrit.wikimedia.org/r/326354 (https://phabricator.wikimedia.org/T152911) (owner: 10MarcoAurelio) [21:04:13] (03CR) 10Reedy: [C: 032] Removing 'technican' user group from tr.wikiquote [mediawiki-config] - 10https://gerrit.wikimedia.org/r/326354 (https://phabricator.wikimedia.org/T152911) (owner: 10MarcoAurelio) [21:04:42] Reedy: here for SWAT ^^^ if you need tester [21:05:39] (03Merged) 10jenkins-bot: Removing 'technican' user group from tr.wikiquote [mediawiki-config] - 10https://gerrit.wikimedia.org/r/326354 (https://phabricator.wikimedia.org/T152911) (owner: 10MarcoAurelio) [21:05:47] (03PS2) 10Reedy: Add pawiki's HD logo and fix two typos [mediawiki-config] - 10https://gerrit.wikimedia.org/r/330401 (https://phabricator.wikimedia.org/T150618) (owner: 10Urbanecm) [21:05:51] (03CR) 10Reedy: [C: 032] Add pawiki's HD logo and fix two typos [mediawiki-config] - 10https://gerrit.wikimedia.org/r/330401 (https://phabricator.wikimedia.org/T150618) (owner: 10Urbanecm) [21:05:53] (03CR) 10jenkins-bot: Removing 'technican' user group from tr.wikiquote [mediawiki-config] - 10https://gerrit.wikimedia.org/r/326354 (https://phabricator.wikimedia.org/T152911) (owner: 10MarcoAurelio) [21:08:05] (03Merged) 10jenkins-bot: Add pawiki's HD logo and fix two typos [mediawiki-config] - 10https://gerrit.wikimedia.org/r/330401 (https://phabricator.wikimedia.org/T150618) (owner: 10Urbanecm) [21:08:17] (03CR) 10jenkins-bot: Add pawiki's HD logo and fix two typos [mediawiki-config] - 10https://gerrit.wikimedia.org/r/330401 (https://phabricator.wikimedia.org/T150618) (owner: 10Urbanecm) [21:09:11] !log reedy@tin Synchronized static/images: pawiki (duration: 00m 42s) [21:09:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:10:37] !log reedy@tin Synchronized wmf-config/InitialiseSettings.php: pawiki logos. remove technican from trwikiquote (duration: 00m 38s) [21:10:37] (03CR) 10Dzahn: "@Papaul should this still be reviewed and merged?" [dns] - 10https://gerrit.wikimedia.org/r/325856 (owner: 10Papaul) [21:10:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:11:12] (03CR) 10Dzahn: "asking because on ticket it looks like already done but open in gerrit" [dns] - 10https://gerrit.wikimedia.org/r/325856 (owner: 10Papaul) [21:12:10] (03PS4) 10Chad: Make notification logos high-density [mediawiki-config] - 10https://gerrit.wikimedia.org/r/319968 (https://phabricator.wikimedia.org/T147219) (owner: 10Catrope) [21:12:38] (03CR) 10Dzahn: [C: 031] osm: Use LABS_NETWORKS in ferm rsync rule [puppet] - 10https://gerrit.wikimedia.org/r/331622 (owner: 10Alexandros Kosiaris) [21:13:51] (03CR) 10Chad: [C: 032] Make notification logos high-density [mediawiki-config] - 10https://gerrit.wikimedia.org/r/319968 (https://phabricator.wikimedia.org/T147219) (owner: 10Catrope) [21:14:40] 06Operations, 10Wikimedia-Language-setup, 10Wikimedia-Site-requests, 05MW-1.28-release-notes, 13Patch-For-Review: Create Wikipedia Tulu - https://phabricator.wikimedia.org/T140898#2934232 (10Krenair) @Dereckson [21:15:32] (03Merged) 10jenkins-bot: Make notification logos high-density [mediawiki-config] - 10https://gerrit.wikimedia.org/r/319968 (https://phabricator.wikimedia.org/T147219) (owner: 10Catrope) [21:15:52] (03CR) 10jenkins-bot: Make notification logos high-density [mediawiki-config] - 10https://gerrit.wikimedia.org/r/319968 (https://phabricator.wikimedia.org/T147219) (owner: 10Catrope) [21:15:59] (03PS3) 10Reedy: Match 'editcontentmodel' permission with 'move' [mediawiki-config] - 10https://gerrit.wikimedia.org/r/309066 (https://phabricator.wikimedia.org/T85847) (owner: 10Legoktm) [21:16:57] !log demon@tin Synchronized static/images/project-logos/notifications: New HD logos (duration: 00m 38s) [21:17:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:17:06] (03CR) 10Legoktm: [C: 04-2] "Sorry, I need to announce this first before it goes out." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/309066 (https://phabricator.wikimedia.org/T85847) (owner: 10Legoktm) [21:17:38] (03PS2) 10Alex Monk: Use LE for wikitech [puppet] - 10https://gerrit.wikimedia.org/r/331638 (https://phabricator.wikimedia.org/T154913) [21:17:51] !log demon@tin Synchronized wmf-config/InitialiseSettings.php: use new HD logos (duration: 00m 38s) [21:17:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:18:16] James_F: You're live [21:18:33] (03CR) 10jerkins-bot: [V: 04-1] Match 'editcontentmodel' permission with 'move' [mediawiki-config] - 10https://gerrit.wikimedia.org/r/309066 (https://phabricator.wikimedia.org/T85847) (owner: 10Legoktm) [21:19:41] (03PS2) 10Reedy: Enable signature button in Portal: on de.wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/315119 (https://phabricator.wikimedia.org/T145619) (owner: 10Dereckson) [21:19:45] (03CR) 10Reedy: [C: 032] Enable signature button in Portal: on de.wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/315119 (https://phabricator.wikimedia.org/T145619) (owner: 10Dereckson) [21:21:21] (03Merged) 10jenkins-bot: Enable signature button in Portal: on de.wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/315119 (https://phabricator.wikimedia.org/T145619) (owner: 10Dereckson) [21:21:31] (03CR) 10jenkins-bot: Enable signature button in Portal: on de.wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/315119 (https://phabricator.wikimedia.org/T145619) (owner: 10Dereckson) [21:21:58] (03Abandoned) 10Reedy: Make $wgMessageCacheType on beta cluster the same as on production [mediawiki-config] - 10https://gerrit.wikimedia.org/r/316204 (https://phabricator.wikimedia.org/T144952) (owner: 10AndyRussG) [21:23:15] (03PS3) 10Reedy: Get rid of $wmg hack for MassMessage settings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/237686 (owner: 10Legoktm) [21:23:19] (03CR) 10Reedy: [C: 032] Get rid of $wmg hack for MassMessage settings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/237686 (owner: 10Legoktm) [21:23:53] (03PS4) 10Chad: Follow-up I049fa67: Remind people not to enable wgKartographerWikivoyageMode [mediawiki-config] - 10https://gerrit.wikimedia.org/r/284483 (owner: 10Jforrester) [21:24:17] (03PS3) 10Reedy: Enable sitenotice banners for arwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/327253 (https://phabricator.wikimedia.org/T152826) (owner: 10Florianschmidtwelzow) [21:24:21] (03CR) 10Reedy: [C: 032] Enable sitenotice banners for arwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/327253 (https://phabricator.wikimedia.org/T152826) (owner: 10Florianschmidtwelzow) [21:25:12] (03Merged) 10jenkins-bot: Get rid of $wmg hack for MassMessage settings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/237686 (owner: 10Legoktm) [21:25:23] (03CR) 10jenkins-bot: Get rid of $wmg hack for MassMessage settings [mediawiki-config] - 10https://gerrit.wikimedia.org/r/237686 (owner: 10Legoktm) [21:26:25] (03CR) 10Chad: [C: 032] Follow-up I049fa67: Remind people not to enable wgKartographerWikivoyageMode [mediawiki-config] - 10https://gerrit.wikimedia.org/r/284483 (owner: 10Jforrester) [21:26:42] (03PS5) 10Chad: Follow-up I049fa67: Remind people not to enable wgKartographerWikivoyageMode [mediawiki-config] - 10https://gerrit.wikimedia.org/r/284483 (owner: 10Jforrester) [21:26:47] (03CR) 10Chad: [V: 032 C: 032] Follow-up I049fa67: Remind people not to enable wgKartographerWikivoyageMode [mediawiki-config] - 10https://gerrit.wikimedia.org/r/284483 (owner: 10Jforrester) [21:27:03] PROBLEM - puppet last run on elastic1021 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [21:28:08] (03CR) 10jenkins-bot: Follow-up I049fa67: Remind people not to enable wgKartographerWikivoyageMode [mediawiki-config] - 10https://gerrit.wikimedia.org/r/284483 (owner: 10Jforrester) [21:28:55] !log demon@tin Synchronized wmf-config: massmessage hack cleanup + comments on kartographer wikivoyage mode (duration: 00m 41s) [21:28:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:29:34] (03PS2) 10Reedy: Setting $wgPageAssessmentsOnTalkPages to false for enwikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/328874 (owner: 10Kaldari) [21:29:38] (03CR) 10Reedy: [C: 032] Setting $wgPageAssessmentsOnTalkPages to false for enwikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/328874 (owner: 10Kaldari) [21:30:23] Reedy: Cool, are you deploying that too? [21:30:45] kaldari: Yeah, we're deploying as much as we can from mw-config [21:30:55] awesome! [21:31:36] (03PS3) 10Reedy: beta: Remove duplicate entry for wikidata in $wgLogo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/327439 (owner: 10Legoktm) [21:31:43] RECOVERY - puppet last run on ms-be1001 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [21:31:45] (03CR) 10Reedy: [C: 032] beta: Remove duplicate entry for wikidata in $wgLogo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/327439 (owner: 10Legoktm) [21:31:47] (03Merged) 10jenkins-bot: Setting $wgPageAssessmentsOnTalkPages to false for enwikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/328874 (owner: 10Kaldari) [21:32:03] (03CR) 10jenkins-bot: Setting $wgPageAssessmentsOnTalkPages to false for enwikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/328874 (owner: 10Kaldari) [21:32:43] (03PS2) 10Reedy: Content namespaces configuration for lt.wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/307455 (https://phabricator.wikimedia.org/T144118) (owner: 10Dereckson) [21:32:46] (03CR) 10Reedy: [C: 032] Content namespaces configuration for lt.wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/307455 (https://phabricator.wikimedia.org/T144118) (owner: 10Dereckson) [21:33:20] (03Merged) 10jenkins-bot: beta: Remove duplicate entry for wikidata in $wgLogo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/327439 (owner: 10Legoktm) [21:34:09] (03CR) 10jenkins-bot: beta: Remove duplicate entry for wikidata in $wgLogo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/327439 (owner: 10Legoktm) [21:34:14] Reedy: 307455 is linked to a task blocked on community consensus [21:34:25] Oh.. [21:34:53] PROBLEM - puppet last run on db1030 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [21:34:56] Reedy: well, we can enable it, and see how the community reacts, and disable it if they don't like it [21:35:09] (03PS3) 10Reedy: Remove individual wikis' config for wgOresModels, use 'default' instead [mediawiki-config] - 10https://gerrit.wikimedia.org/r/312168 (owner: 10Catrope) [21:35:12] (03CR) 10Reedy: [C: 032] Remove individual wikis' config for wgOresModels, use 'default' instead [mediawiki-config] - 10https://gerrit.wikimedia.org/r/312168 (owner: 10Catrope) [21:35:14] but the initial plan was to reach first the community to ask them an opinion [21:35:14] (03Merged) 10jenkins-bot: Content namespaces configuration for lt.wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/307455 (https://phabricator.wikimedia.org/T144118) (owner: 10Dereckson) [21:35:29] (03CR) 10jenkins-bot: Content namespaces configuration for lt.wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/307455 (https://phabricator.wikimedia.org/T144118) (owner: 10Dereckson) [21:35:36] as lithuanian isn't a language we speak a lot, nobody volunteered [21:36:01] https://lt.wikipedia.org/wiki/Vikipedija:Forumas#Visual_editor_for_S.C4.85ra.C5.A1as:_namespace [21:36:07] hmmmm I notified them actually [21:36:12] (03PS4) 10Reedy: Enable sitenotice banners for arwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/327253 (https://phabricator.wikimedia.org/T152826) (owner: 10Florianschmidtwelzow) [21:36:51] Dereckson: ask domas ;) [21:41:55] 06Operations, 10Wikimedia-Language-setup, 10Wikimedia-Site-requests, 05MW-1.28-release-notes, 13Patch-For-Review: Create Wikipedia Tulu - https://phabricator.wikimedia.org/T140898#2934293 (10Dereckson) 05Open>03Resolved a:03Dereckson Yes, I checked the steps on Add wiki documentation, all is now in... [21:42:10] (03CR) 10Papaul: "This is already done the systems are installed" [dns] - 10https://gerrit.wikimedia.org/r/325856 (owner: 10Papaul) [21:42:19] (03PS2) 10Chad: Allow users to enable wikieditor-preview on beta cluster [mediawiki-config] - 10https://gerrit.wikimedia.org/r/322136 (owner: 10Dereckson) [21:42:24] 06Operations, 10Wikimedia-Language-setup, 10Wikimedia-Site-requests, 05MW-1.28-release-notes, 13Patch-For-Review: Create Wikipedia Tulu - https://phabricator.wikimedia.org/T140898#2934297 (10Dereckson) a:05Dereckson>03None [21:42:26] (03CR) 10Chad: [C: 032] Allow users to enable wikieditor-preview on beta cluster [mediawiki-config] - 10https://gerrit.wikimedia.org/r/322136 (owner: 10Dereckson) [21:43:11] (03PS5) 10Reedy: Properly point eventbus.svc to codfw endpoint in codfw [mediawiki-config] - 10https://gerrit.wikimedia.org/r/328212 (owner: 10Ottomata) [21:43:16] (03CR) 10Reedy: [C: 032] Properly point eventbus.svc to codfw endpoint in codfw [mediawiki-config] - 10https://gerrit.wikimedia.org/r/328212 (owner: 10Ottomata) [21:43:49] (03Merged) 10jenkins-bot: Allow users to enable wikieditor-preview on beta cluster [mediawiki-config] - 10https://gerrit.wikimedia.org/r/322136 (owner: 10Dereckson) [21:43:53] Reedy: :-) [21:44:13] (03CR) 10jenkins-bot: Allow users to enable wikieditor-preview on beta cluster [mediawiki-config] - 10https://gerrit.wikimedia.org/r/322136 (owner: 10Dereckson) [21:45:00] (03Merged) 10jenkins-bot: Properly point eventbus.svc to codfw endpoint in codfw [mediawiki-config] - 10https://gerrit.wikimedia.org/r/328212 (owner: 10Ottomata) [21:45:08] !log demon@tin Synchronized wmf-config/CommonSettings-labs.php: no-op (duration: 00m 38s) [21:45:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:45:12] (03CR) 10jenkins-bot: Properly point eventbus.svc to codfw endpoint in codfw [mediawiki-config] - 10https://gerrit.wikimedia.org/r/328212 (owner: 10Ottomata) [21:45:33] PROBLEM - citoid endpoints health on scb1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [21:45:49] Dereckson: Lithuanian used to be third most important language in Wikipedias [21:45:55] Dereckson: after English and Esperanto, of course [21:46:19] (03PS3) 10Reedy: he.wiki images size configuration [mediawiki-config] - 10https://gerrit.wikimedia.org/r/31580 (https://phabricator.wikimedia.org/T43712) (owner: 10Dereckson) [21:46:23] RECOVERY - citoid endpoints health on scb1001 is OK: All endpoints are healthy [21:46:30] (03CR) 10jerkins-bot: [V: 04-1] he.wiki images size configuration [mediawiki-config] - 10https://gerrit.wikimedia.org/r/31580 (https://phabricator.wikimedia.org/T43712) (owner: 10Dereckson) [21:47:47] (03PS3) 10Alex Monk: Use LE for wikitech [puppet] - 10https://gerrit.wikimedia.org/r/331638 (https://phabricator.wikimedia.org/T154913) [21:50:06] (03CR) 10Andrew Bogott: "You're correct on both counts. The 'labspuppet' db is used by the labs puppet backend to store instance state, so should be preserver and" [puppet] - 10https://gerrit.wikimedia.org/r/328476 (owner: 10Jcrespo) [21:51:48] (03CR) 10Chad: [C: 032] Rewrite wmf-beta-autoupdate as a scap3 plugin [mediawiki-config] - 10https://gerrit.wikimedia.org/r/325875 (https://phabricator.wikimedia.org/T151519) (owner: 10Chad) [21:52:18] (03PS1) 10Madhuvishy: paws-internal: Fix custom ldap authenticator [puppet] - 10https://gerrit.wikimedia.org/r/331694 [21:53:30] (03Merged) 10jenkins-bot: Rewrite wmf-beta-autoupdate as a scap3 plugin [mediawiki-config] - 10https://gerrit.wikimedia.org/r/325875 (https://phabricator.wikimedia.org/T151519) (owner: 10Chad) [21:53:50] (03CR) 10jenkins-bot: Rewrite wmf-beta-autoupdate as a scap3 plugin [mediawiki-config] - 10https://gerrit.wikimedia.org/r/325875 (https://phabricator.wikimedia.org/T151519) (owner: 10Chad) [21:53:53] PROBLEM - puppet last run on aqs1004 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [21:54:35] !log demon@tin Synchronized scap/plugins/wmf-beta-autoupdate.py: no-op, not yet used (duration: 00m 38s) [21:54:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [21:55:03] RECOVERY - puppet last run on elastic1021 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [21:56:05] (03PS2) 10Andrew Bogott: Labs: build images without pam.d and access.conf hacks [puppet] - 10https://gerrit.wikimedia.org/r/257411 (https://phabricator.wikimedia.org/T120710) (owner: 10Coren) [21:57:15] RoanKattouw: https://gerrit.wikimedia.org/r/#/c/312867/ has merge conflicts. Is it still needed? [21:58:07] Ah, grep says no [21:59:36] (03Abandoned) 10Chad: Follow-up fd8998a4ec9: remove another stray $wmgMFUseCentralAuthToken reference [mediawiki-config] - 10https://gerrit.wikimedia.org/r/312867 (owner: 10Catrope) [22:01:37] (03CR) 10Chad: [C: 032] Rename $wgFlagRestrctions to $wgFlaggedRevsTagsRestrictions [mediawiki-config] - 10https://gerrit.wikimedia.org/r/325456 (owner: 10Catrope) [22:02:53] RECOVERY - puppet last run on db1030 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [22:03:22] (03Merged) 10jenkins-bot: Rename $wgFlagRestrctions to $wgFlaggedRevsTagsRestrictions [mediawiki-config] - 10https://gerrit.wikimedia.org/r/325456 (owner: 10Catrope) [22:03:37] (03CR) 10jenkins-bot: Rename $wgFlagRestrctions to $wgFlaggedRevsTagsRestrictions [mediawiki-config] - 10https://gerrit.wikimedia.org/r/325456 (owner: 10Catrope) [22:04:47] !log demon@tin Synchronized wmf-config/flaggedrevs.php: Deprecated variable cleanup (duration: 00m 38s) [22:04:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:05:08] (03CR) 10Andrew Bogott: [C: 032] Labs: build images without pam.d and access.conf hacks [puppet] - 10https://gerrit.wikimedia.org/r/257411 (https://phabricator.wikimedia.org/T120710) (owner: 10Coren) [22:05:53] RECOVERY - HHVM rendering on mw1263 is OK: HTTP OK: HTTP/1.1 200 OK - 73502 bytes in 0.141 second response time [22:06:43] RECOVERY - Apache HTTP on mw1263 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 612 bytes in 0.027 second response time [22:06:43] RECOVERY - Nginx local proxy to apache on mw1263 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 613 bytes in 0.037 second response time [22:06:49] (03CR) 10Chad: [C: 032] Set $wgAbuseFilterNotificationsPrivate = true; for Meta-Wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/329600 (https://phabricator.wikimedia.org/T154358) (owner: 10MarcoAurelio) [22:07:07] o-O [22:07:18] :) [22:07:34] well, I don't have to schedule it for SWAT now :D [22:07:57] Nah, simple enough I'll do it in our mass of stuff we've been doing :) [22:08:35] (03Merged) 10jenkins-bot: Set $wgAbuseFilterNotificationsPrivate = true; for Meta-Wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/329600 (https://phabricator.wikimedia.org/T154358) (owner: 10MarcoAurelio) [22:09:38] (03CR) 10jenkins-bot: Set $wgAbuseFilterNotificationsPrivate = true; for Meta-Wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/329600 (https://phabricator.wikimedia.org/T154358) (owner: 10MarcoAurelio) [22:13:02] !log demon@tin Synchronized wmf-config/abusefilter.php: Set $wgAbuseFilterNotificationsPrivate = true; for Meta-Wiki (duration: 00m 40s) [22:13:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:15:31] I'll let the community know this is now resolved. Unanimous support for it fwiw [22:15:33] PROBLEM - restbase endpoints health on restbase2001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [22:15:53] PROBLEM - Apache HTTP on mw1198 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:16:13] PROBLEM - Nginx local proxy to apache on mw1198 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:16:13] PROBLEM - HHVM rendering on mw1198 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:16:33] PROBLEM - restbase endpoints health on restbase2011 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [22:17:20] mw1198 seems unhappy... [22:17:27] HHVM go boom? [22:18:23] RECOVERY - restbase endpoints health on restbase2011 is OK: All endpoints are healthy [22:18:23] RECOVERY - restbase endpoints health on restbase2001 is OK: All endpoints are healthy [22:19:44] ostriches: I am checking it, hhvm seems busted [22:20:31] (03PS2) 10Madhuvishy: paws-internal: Fix custom ldap authenticator [puppet] - 10https://gerrit.wikimedia.org/r/331694 [22:20:44] (03CR) 10Madhuvishy: [V: 032 C: 032] paws-internal: Fix custom ldap authenticator [puppet] - 10https://gerrit.wikimedia.org/r/331694 (owner: 10Madhuvishy) [22:20:56] !log restarting hhvm on mw1198 (dump-debug in /tmp/hhvm.9737.bt) [22:20:59] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:22:43] RECOVERY - Apache HTTP on mw1198 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 612 bytes in 0.027 second response time [22:22:53] RECOVERY - puppet last run on aqs1004 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [22:23:03] RECOVERY - Nginx local proxy to apache on mw1198 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 613 bytes in 0.062 second response time [22:23:03] RECOVERY - HHVM rendering on mw1198 is OK: HTTP OK: HTTP/1.1 200 OK - 73527 bytes in 1.163 second response time [22:23:07] can see the usual HPHP::Treadmill::getAgeOldestRequest () from /usr/bin/hhvm [22:23:10] :( [22:24:13] Boo :( [22:26:34] !log added mw1239.eqiad.wmnet back to service - T148421 [22:26:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:26:40] T148421: mw1239: memory scrubbing error - https://phabricator.wikimedia.org/T148421 [22:27:41] (I did scap pull before doing it) [22:29:41] (03Draft2) 10MarcoAurelio: Fix Task number in abusefilter.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331785 [22:29:48] (03Draft1) 10MarcoAurelio: Fix Task number in abusefilter.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331785 [22:30:06] (03PS3) 10MarcoAurelio: Fix Task number in abusefilter.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331785 (https://phabricator.wikimedia.org/T154358) [22:30:40] ostriches: ^^ [22:31:31] (03CR) 10Chad: [V: 032 C: 032] Fix Task number in abusefilter.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331785 (https://phabricator.wikimedia.org/T154358) (owner: 10MarcoAurelio) [22:31:49] so long jenkins bot :P [22:32:27] !log demon@tin Synchronized wmf-config/abusefilter.php: comment fix (duration: 00m 39s) [22:32:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:33:03] (03PS4) 10Reedy: Remove individual wikis' config for wgOresModels, use 'default' instead [mediawiki-config] - 10https://gerrit.wikimedia.org/r/312168 (owner: 10Catrope) [22:33:08] (03CR) 10Reedy: Remove individual wikis' config for wgOresModels, use 'default' instead [mediawiki-config] - 10https://gerrit.wikimedia.org/r/312168 (owner: 10Catrope) [22:33:12] (03CR) 10Reedy: [C: 032] Remove individual wikis' config for wgOresModels, use 'default' instead [mediawiki-config] - 10https://gerrit.wikimedia.org/r/312168 (owner: 10Catrope) [22:35:23] 06Operations, 05Prometheus-metrics-monitoring: Create prometheus nutcracker exporter - https://phabricator.wikimedia.org/T155129#2934402 (10elukey) [22:35:37] (03Merged) 10jenkins-bot: Remove individual wikis' config for wgOresModels, use 'default' instead [mediawiki-config] - 10https://gerrit.wikimedia.org/r/312168 (owner: 10Catrope) [22:35:54] PROBLEM - puppet last run on mw1250 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [22:35:59] (03PS5) 10Reedy: Enable sitenotice banners for arwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/327253 (https://phabricator.wikimedia.org/T152826) (owner: 10Florianschmidtwelzow) [22:36:04] (03CR) 10Reedy: Enable sitenotice banners for arwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/327253 (https://phabricator.wikimedia.org/T152826) (owner: 10Florianschmidtwelzow) [22:36:09] (03CR) 10Reedy: [C: 032] Enable sitenotice banners for arwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/327253 (https://phabricator.wikimedia.org/T152826) (owner: 10Florianschmidtwelzow) [22:36:10] 06Operations, 05Prometheus-metrics-monitoring: Create prometheus nutcracker exporter - https://phabricator.wikimedia.org/T155129#2934415 (10elukey) [22:36:12] (03PS2) 10Chad: Add a logo for beta Meta-Wiki on Labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/326842 (https://phabricator.wikimedia.org/T125942) (owner: 10Odder) [22:36:45] (03PS1) 10Raimond Spekking: Replace outdated comment with a short description [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331788 [22:38:05] (03Merged) 10jenkins-bot: Enable sitenotice banners for arwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/327253 (https://phabricator.wikimedia.org/T152826) (owner: 10Florianschmidtwelzow) [22:38:26] (03CR) 10Reedy: [C: 04-1] "Needs rebase" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/327438 (owner: 10Legoktm) [22:38:28] (03CR) 10jerkins-bot: [V: 04-1] Add a logo for beta Meta-Wiki on Labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/326842 (https://phabricator.wikimedia.org/T125942) (owner: 10Odder) [22:39:41] !log reedy@tin Synchronized wmf-config/InitialiseSettings.php: Simplify ores config. Enable sitenotice banners for arwiki (duration: 00m 39s) [22:39:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:40:00] (03PS1) 10BBlack: [WIP] discovery stuff [puppet] - 10https://gerrit.wikimedia.org/r/331789 [22:40:41] (03CR) 10Reedy: [C: 04-1] "Also needs rebasing" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281973 (https://phabricator.wikimedia.org/T27397) (owner: 10Matanya) [22:40:59] (03Abandoned) 10Reedy: [DO NOT MERGE] Revert "Capture the "CentralNotice" log bucket" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/312310 (owner: 10Awight) [22:41:05] (03PS3) 10Chad: Add a logo for beta Meta-Wiki on Labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/326842 (https://phabricator.wikimedia.org/T125942) (owner: 10Odder) [22:41:35] (03CR) 10Chad: [C: 032] Replace outdated comment with a short description [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331788 (owner: 10Raimond Spekking) [22:43:15] (03Merged) 10jenkins-bot: Replace outdated comment with a short description [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331788 (owner: 10Raimond Spekking) [22:43:52] (03CR) 10jerkins-bot: [V: 04-1] [WIP] discovery stuff [puppet] - 10https://gerrit.wikimedia.org/r/331789 (owner: 10BBlack) [22:43:59] (03CR) 10Chad: [C: 032] Add a logo for beta Meta-Wiki on Labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/326842 (https://phabricator.wikimedia.org/T125942) (owner: 10Odder) [22:44:23] !log demon@tin Synchronized wmf-config/CommonSettings.php: commentfix (duration: 00m 38s) [22:44:26] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:46:03] (03Merged) 10jenkins-bot: Add a logo for beta Meta-Wiki on Labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/326842 (https://phabricator.wikimedia.org/T125942) (owner: 10Odder) [22:46:09] (03CR) 10jenkins-bot: Fix Task number in abusefilter.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331785 (https://phabricator.wikimedia.org/T154358) (owner: 10MarcoAurelio) [22:46:11] (03CR) 10jenkins-bot: Remove individual wikis' config for wgOresModels, use 'default' instead [mediawiki-config] - 10https://gerrit.wikimedia.org/r/312168 (owner: 10Catrope) [22:46:13] (03CR) 10jenkins-bot: Enable sitenotice banners for arwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/327253 (https://phabricator.wikimedia.org/T152826) (owner: 10Florianschmidtwelzow) [22:46:15] (03CR) 10jenkins-bot: Replace outdated comment with a short description [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331788 (owner: 10Raimond Spekking) [22:46:17] (03CR) 10jenkins-bot: Add a logo for beta Meta-Wiki on Labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/326842 (https://phabricator.wikimedia.org/T125942) (owner: 10Odder) [22:47:11] !log demon@tin Synchronized static/images/project-logos: beta logos (duration: 00m 40s) [22:47:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:48:08] (03PS2) 10Matanya: webp: enabled by default - remove old dead code [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281973 (https://phabricator.wikimedia.org/T27397) [22:48:19] !log demon@tin Synchronized wmf-config/InitialiseSettings-labs.php: Use new metawiki logo, no-op in prod (duration: 00m 38s) [22:48:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:48:25] Huge replag spike on db1028 [22:48:41] eg: Server db1028 (#1) has >= 218.60211396217 seconds of lag [22:51:36] (03PS3) 10Matanya: webp: enabled by default - remove old dead code [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281973 (https://phabricator.wikimedia.org/T27397) [22:52:50] (03PS3) 10Reedy: Update gallery image bounding box on svwiki to 150x150 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/304991 (https://phabricator.wikimedia.org/T113877) (owner: 10Gilles) [22:54:05] (03CR) 10Reedy: [C: 032] webp: enabled by default - remove old dead code [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281973 (https://phabricator.wikimedia.org/T27397) (owner: 10Matanya) [22:55:33] (03Merged) 10jenkins-bot: webp: enabled by default - remove old dead code [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281973 (https://phabricator.wikimedia.org/T27397) (owner: 10Matanya) [22:58:32] (03CR) 10jenkins-bot: webp: enabled by default - remove old dead code [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281973 (https://phabricator.wikimedia.org/T27397) (owner: 10Matanya) [22:59:35] (03CR) 10Chad: [C: 032] static.php should use deployed branch for invalid hashes [mediawiki-config] - 10https://gerrit.wikimedia.org/r/312254 (https://phabricator.wikimedia.org/T146363) (owner: 10Brion VIBBER) [22:59:48] (03Abandoned) 10Reedy: Add "composer test" command to lint files and run tests [mediawiki-config] - 10https://gerrit.wikimedia.org/r/189148 (https://phabricator.wikimedia.org/T85947) (owner: 10Legoktm) [23:02:14] (03CR) 10Krinkle: [C: 031] "LGTM - it's a labs-only change that'll make it easier to safely try this in the unlikely event that it goes wrong again - https://wikitech" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/330612 (owner: 10Aaron Schulz) [23:02:33] (03Merged) 10jenkins-bot: static.php should use deployed branch for invalid hashes [mediawiki-config] - 10https://gerrit.wikimedia.org/r/312254 (https://phabricator.wikimedia.org/T146363) (owner: 10Brion VIBBER) [23:02:44] (03CR) 10jenkins-bot: static.php should use deployed branch for invalid hashes [mediawiki-config] - 10https://gerrit.wikimedia.org/r/312254 (https://phabricator.wikimedia.org/T146363) (owner: 10Brion VIBBER) [23:03:53] RECOVERY - puppet last run on mw1250 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [23:04:58] Reedy: https://gerrit.wikimedia.org/r/#/q/status:open+project:operations/mediawiki-config+-label:Verified%253C%253D-1+-label:Code-Review%253D-2 [23:05:49] anybody have encountered java throwing "SSLHandshakeException: Unsupported curveId: 29" when trying to talk to Wikimedia servers? [23:06:32] SMalyshev: bblack may be a good person to talk to [23:06:48] bblack: ping? [23:10:18] !log demon@tin Synchronized w/static.php: For Timo <3 (duration: 00m 40s) [23:10:21] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:11:27] (03PS2) 10Chad: Exclude new high-priority video transcode jobs from default queue [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331669 (https://phabricator.wikimedia.org/T155098) (owner: 10Brion VIBBER) [23:13:06] (03CR) 10Chad: [C: 032] Exclude new high-priority video transcode jobs from default queue [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331669 (https://phabricator.wikimedia.org/T155098) (owner: 10Brion VIBBER) [23:14:38] (03Merged) 10jenkins-bot: Exclude new high-priority video transcode jobs from default queue [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331669 (https://phabricator.wikimedia.org/T155098) (owner: 10Brion VIBBER) [23:14:48] (03CR) 10jenkins-bot: Exclude new high-priority video transcode jobs from default queue [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331669 (https://phabricator.wikimedia.org/T155098) (owner: 10Brion VIBBER) [23:16:12] !log demon@tin Synchronized wmf-config/CommonSettings.php: video transcode jobqueue stuff for Brion (duration: 00m 38s) [23:16:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:18:24] (03PS4) 10Reedy: Re-add wgCommonsMetadataForceRecalculate [mediawiki-config] - 10https://gerrit.wikimedia.org/r/330318 (owner: 10Gergő Tisza) [23:18:39] (03CR) 10Reedy: [C: 032] Re-add wgCommonsMetadataForceRecalculate [mediawiki-config] - 10https://gerrit.wikimedia.org/r/330318 (owner: 10Gergő Tisza) [23:20:02] (03Merged) 10jenkins-bot: Re-add wgCommonsMetadataForceRecalculate [mediawiki-config] - 10https://gerrit.wikimedia.org/r/330318 (owner: 10Gergő Tisza) [23:20:12] (03CR) 10jenkins-bot: Re-add wgCommonsMetadataForceRecalculate [mediawiki-config] - 10https://gerrit.wikimedia.org/r/330318 (owner: 10Gergő Tisza) [23:40:13] PROBLEM - MariaDB Slave Lag: m3 on db1048 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 400.08 seconds [23:40:26] 06Operations, 10Ops-Access-Requests: Requesting access to analytics-privatedata-users for anomie - https://phabricator.wikimedia.org/T155143#2935206 (10Anomie) [23:40:54] can someone see which task is the subtask of https://phabricator.wikimedia.org/T43492 ? [23:41:11] it's private task so I can't see the title nor remove it as subtask either [23:42:37] TabbyCat: Subtask is a dupe of T41343 [23:42:43] I'll see if I can open it [23:42:52] ostriches: can you add me to it too please? [23:43:06] One second, the dupe target is public already [23:43:09] I'll just open it up I think [23:43:28] okay so edit the task and set visible to public and edit to public [23:43:41] maybe set also security to none [23:43:41] Yep one sec, just skimming to make sure there's no PII [23:43:49] ah, very true [23:44:27] Ah crud, there's some IPs and shiz [23:44:48] then I don't think we should make it public [23:44:53] Yeah, so 41343 is a dupe of 31583 (the latter is public) [23:45:05] 41343 is the subtask on T43492 [23:45:05] If you want us to do some tag maintenance on it we can handle that without making it public [23:45:06] T43492: [DO NOT USE] Steward, global sysop and SWMT tasks bugs (tracking) [superseded by #Stewards-and-global-tools] - https://phabricator.wikimedia.org/T43492 [23:46:25] just remove it as subtask then, finished tracking tasks shouldn't have subtasks attached as per mediawiki guideline on tracking task transition [23:46:41] system won't let me do that because I can't access that restricted task [23:47:22] !log reedy@tin Synchronized wmf-config: consistency (duration: 00m 41s) [23:47:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:48:09] (03PS2) 10Chad: Minerva should apply known template hacks in production [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331425 (https://phabricator.wikimedia.org/T94102) (owner: 10Jdlrobson) [23:48:29] TabbyCat, done [23:48:39] meow-thanks [23:49:01] want me to add the new project? [23:49:25] if you can, sure [23:49:35] so everything is there now [23:49:59] geez, brooming on tracking tasks generates a lot of noise, sorry fellas [23:50:53] (03CR) 10Chad: [C: 032] Minerva should apply known template hacks in production [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331425 (https://phabricator.wikimedia.org/T94102) (owner: 10Jdlrobson) [23:52:35] (03PS2) 10Reedy: Add transitionary config for EducationProgram [mediawiki-config] - 10https://gerrit.wikimedia.org/r/303383 [23:52:53] PROBLEM - puppet last run on dbproxy1006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [23:53:12] (03Merged) 10jenkins-bot: Minerva should apply known template hacks in production [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331425 (https://phabricator.wikimedia.org/T94102) (owner: 10Jdlrobson) [23:54:13] RECOVERY - MariaDB Slave Lag: m3 on db1048 is OK: OK slave_sql_lag Replication lag: 0.17 seconds [23:54:49] (03CR) 10jenkins-bot: Minerva should apply known template hacks in production [mediawiki-config] - 10https://gerrit.wikimedia.org/r/331425 (https://phabricator.wikimedia.org/T94102) (owner: 10Jdlrobson) [23:55:10] !log demon@tin Synchronized wmf-config/InitialiseSettings.php: Minerva hacks (duration: 00m 38s) [23:55:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:55:22] (03CR) 10Chad: "Needs manual rebase" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/315983 (owner: 10Dereckson) [23:57:23] (03Abandoned) 10Chad: Restore mobile formatting for enwiki mdot [mediawiki-config] - 10https://gerrit.wikimedia.org/r/295926 (https://phabricator.wikimedia.org/T138578) (owner: 10Dr0ptp4kt) [23:58:13] (03PS2) 10Chad: Adding language name configuration for Wikidata [mediawiki-config] - 10https://gerrit.wikimedia.org/r/315912 (https://phabricator.wikimedia.org/T113408) (owner: 10Jon Harald Søby) [23:59:22] (03PS3) 10Reedy: Update Kafka analytics broker list for deployment-prep [mediawiki-config] - 10https://gerrit.wikimedia.org/r/287741 (owner: 10Ottomata)