[00:01:59] (03PS2) 10Dzahn: Add ferm rules for rsyncd/scap master [puppet] - 10https://gerrit.wikimedia.org/r/240074 (https://phabricator.wikimedia.org/T113351) (owner: 10Muehlenhoff) [00:03:31] (03CR) 10Dzahn: [C: 032] "looks good, not applied on tin or mira right now, will be enabled on mira first" [puppet] - 10https://gerrit.wikimedia.org/r/240074 (https://phabricator.wikimedia.org/T113351) (owner: 10Muehlenhoff) [00:09:11] (03PS2) 10Dzahn: remove sodium.wm.o (leaving sodium.mgmt.eqiad.wmnet) [dns] - 10https://gerrit.wikimedia.org/r/239414 (https://phabricator.wikimedia.org/T110142) (owner: 10John F. Lewis) [00:09:34] (03PS3) 10Dzahn: remove sodium.wm.o (leaving sodium.mgmt.eqiad.wmnet) [dns] - 10https://gerrit.wikimedia.org/r/239414 (https://phabricator.wikimedia.org/T110142) (owner: 10John F. Lewis) [00:20:17] (03PS2) 10Ori.livneh: xenon-log: make retention count apply per-entrypoint [puppet] - 10https://gerrit.wikimedia.org/r/242411 [00:21:49] 6operations, 10MediaWiki-Cache, 6Performance-Team, 7Availability: Setup a 3 server Kafka instance in both eqiad and codfw for reliable purge streams - https://phabricator.wikimedia.org/T114191#1687880 (10Ottomata) Aye cool! Quick note before I leave for the day: These Kafka clusters will likely be used f... [00:22:11] (03PS3) 10Ori.livneh: xenon-log: make retention count apply per-entrypoint [puppet] - 10https://gerrit.wikimedia.org/r/242411 [00:27:41] (03CR) 10Ori.livneh: [C: 032 V: 032] xenon-log: make retention count apply per-entrypoint [puppet] - 10https://gerrit.wikimedia.org/r/242411 (owner: 10Ori.livneh) [00:28:15] PROBLEM - puppet last run on fluorine is CRITICAL: CRITICAL: puppet fail [00:29:54] RECOVERY - puppet last run on fluorine is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [00:31:09] 10Ops-Access-Requests, 6operations: Requesting access to stat1002 for Zhou Zhou - https://phabricator.wikimedia.org/T113325#1687894 (10ZhouZ) Thanks @RobH. Are we all set then or are there additional steps? [00:40:39] (03CR) 10GWicke: "See https://phabricator.wikimedia.org/T107762 for the discussion on going straight to node 4.1." [puppet] - 10https://gerrit.wikimedia.org/r/229304 (owner: 10GWicke) [00:41:14] PROBLEM - puppet last run on mw2002 is CRITICAL: CRITICAL: puppet fail [01:02:43] !log ori@tin Synchronized php-1.26wmf24/vendor: 940124a7db: Updated mediawiki/core Project: mediawiki/vendor ff5e254f7eddf811f6f66b4a4063b1a8cc70f265 (duration: 00m 21s) [01:02:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [01:06:40] (03PS1) 10Yuvipanda: dynamicproxy: Enable redis replication for novaproxy [puppet] - 10https://gerrit.wikimedia.org/r/242430 [01:06:44] (03CR) 10jenkins-bot: [V: 04-1] dynamicproxy: Enable redis replication for novaproxy [puppet] - 10https://gerrit.wikimedia.org/r/242430 (owner: 10Yuvipanda) [01:07:01] (03PS2) 10Yuvipanda: dynamicproxy: Enable redis replication for novaproxy [puppet] - 10https://gerrit.wikimedia.org/r/242430 [01:08:10] (03CR) 10Alex Monk: "I tried fiddling with NovaInstance to test the other order, no luck though. Maybe we could change ldap-yaml-enc.py (in the puppetmaster mo" [puppet] - 10https://gerrit.wikimedia.org/r/241526 (owner: 10Alex Monk) [01:09:44] RECOVERY - puppet last run on mw2002 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [01:10:25] (03CR) 10Yuvipanda: [C: 032] dynamicproxy: Enable redis replication for novaproxy [puppet] - 10https://gerrit.wikimedia.org/r/242430 (owner: 10Yuvipanda) [01:22:17] 10Ops-Access-Requests, 6operations, 5Patch-For-Review: Expand shell access for aklapper on Phabricator - https://phabricator.wikimedia.org/T113124#1688117 (10Dzahn) Then let's assign it to one of them [01:29:13] 10Ops-Access-Requests, 6operations: RESTBase Admin access on aqs1001, aqs1002, and aqs1003 for Joseph and Dan - https://phabricator.wikimedia.org/T113416#1688129 (10Dzahn) We should add a new admin group "aqs-admins" and give them the right to control this service, (start/stop/restart etc, ) as well as any com... [01:29:24] PROBLEM - MariaDB Slave Lag: m3 on db1048 is CRITICAL: CRITICAL slave_sql_lag Seconds_Behind_Master: 1621 [01:31:14] RECOVERY - MariaDB Slave Lag: m3 on db1048 is OK: OK slave_sql_lag Seconds_Behind_Master: 0 [01:34:25] 6operations, 10Traffic, 5Patch-For-Review: Fix ethernet startup race on HP LVS w/ jessie - https://phabricator.wikimedia.org/T110530#1688133 (10Dzahn) These kinds of tickets come to mind when the additional price of supporting multiple hardware vendors is discussed. [01:41:20] 6operations, 10Analytics, 10Analytics-Cluster, 10Fundraising Tech Backlog, 10Fundraising-Backlog: Verify kafkatee use for fundraising logs on erbium - https://phabricator.wikimedia.org/T97676#1688147 (10Jgreen) >>! In T97676#1687605, @awight wrote: > Furthermore, when we do make the change, the `count` c... [02:00:24] PROBLEM - HTTP 5xx req/min on graphite1001 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [500.0] [02:05:25] RECOVERY - HTTP 5xx req/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] [02:20:39] 6operations, 7Mail: Remove Aliases in Exim Mail Routing Config [SemiUrgent] - https://phabricator.wikimedia.org/T114173#1688164 (10Mjohnson_WMF) Thank you! *Marti JohnsonProgram Officer* *Individual Grants* *Wikimedia Foundation * +1 415-839-6885 Skype: Mjohnson_WMF... [02:22:34] PROBLEM - puppet last run on lvs3004 is CRITICAL: CRITICAL: puppet fail [02:32:22] !log l10nupdate@tin Synchronized php-1.26wmf24/cache/l10n: l10nupdate for 1.26wmf24 (duration: 07m 21s) [02:32:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:36:38] !log l10nupdate@tin LocalisationUpdate completed (1.26wmf24) at 2015-09-30 02:36:37+00:00 [02:36:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:49:34] RECOVERY - puppet last run on lvs3004 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [03:03:45] !log l10nupdate@tin Synchronized php-1.27.0-wmf.1/cache/l10n: l10nupdate for 1.27.0-wmf.1 (duration: 10m 45s) [03:03:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [03:05:24] PROBLEM - Disk space on labstore1002 is CRITICAL: DISK CRITICAL - /run/lock/storage-replicate-labstore-others/snapshot is not accessible: Permission denied [03:08:45] 6operations: How to page when a host is down? - https://phabricator.wikimedia.org/T113834#1688186 (10Dzahn) >>! In T113834#1681111, @Andrew wrote: > Does that mean that those other alerts didn't fire, or is it just that the IRC system was somehow smart about it? Since history rotates so quickly on icinga it's... [03:10:39] !log l10nupdate@tin LocalisationUpdate completed (1.27.0-wmf.1) at 2015-09-30 03:10:39+00:00 [03:10:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [03:13:25] PROBLEM - puppet last run on mw2194 is CRITICAL: CRITICAL: puppet fail [03:22:13] RECOVERY - Disk space on labstore1002 is OK: DISK OK [03:39:43] 6operations: Ensure that there are no firewall rules in modules - https://phabricator.wikimedia.org/T114209#1688206 (10yuvipanda) 3NEW [03:41:53] RECOVERY - puppet last run on mw2194 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:50:39] (03PS1) 10Yuvipanda: dynamicproxy: Firewall redis more restrictively [puppet] - 10https://gerrit.wikimedia.org/r/242434 (https://phabricator.wikimedia.org/T114209) [03:51:03] (03PS2) 10Yuvipanda: dynamicproxy: Firewall redis more restrictively [puppet] - 10https://gerrit.wikimedia.org/r/242434 (https://phabricator.wikimedia.org/T114209) [03:52:59] (03CR) 10Yuvipanda: [C: 032] dynamicproxy: Firewall redis more restrictively [puppet] - 10https://gerrit.wikimedia.org/r/242434 (https://phabricator.wikimedia.org/T114209) (owner: 10Yuvipanda) [04:01:53] PROBLEM - Incoming network saturation on labstore1003 is CRITICAL: CRITICAL: 12.00% of data above the critical threshold [100000000.0] [04:04:04] PROBLEM - Disk space on labstore1002 is CRITICAL: DISK CRITICAL - /run/lock/storage-replicate-labstore-maps/snapshot is not accessible: Permission denied [04:09:05] PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet). [04:09:54] PROBLEM - Unmerged changes on repository puppet on palladium is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet). [04:26:02] (03PS1) 10Yuvipanda: labstore: Disable some NFS mounts on project-proxy [puppet] - 10https://gerrit.wikimedia.org/r/242437 (https://phabricator.wikimedia.org/T102369) [04:26:30] (03PS2) 10Yuvipanda: labstore: Disable some NFS mounts on project-proxy [puppet] - 10https://gerrit.wikimedia.org/r/242437 (https://phabricator.wikimedia.org/T102369) [04:28:30] (03CR) 10Yuvipanda: [C: 032] labstore: Disable some NFS mounts on project-proxy [puppet] - 10https://gerrit.wikimedia.org/r/242437 (https://phabricator.wikimedia.org/T102369) (owner: 10Yuvipanda) [04:33:25] RECOVERY - Unmerged changes on repository puppet on palladium is OK: No changes to merge. [04:34:15] RECOVERY - Unmerged changes on repository puppet on strontium is OK: No changes to merge. [04:37:03] RECOVERY - Incoming network saturation on labstore1003 is OK: OK: Less than 10.00% above the threshold [75000000.0] [04:37:35] 75000000.0!!!!!!!!!!!!111oneone [04:52:41] ori: there's a bug for it :P [05:02:00] 6operations, 10Gerrit: Wikimedia Gerrit doesn't work if OpenSSH version is higher than 7.0 - https://phabricator.wikimedia.org/T112025#1688288 (10devunt) [05:20:51] 6operations, 7Database: decom ishmael? - https://phabricator.wikimedia.org/T109777#1688303 (10Dzahn) @jcrespo up to you, do you want to keep ishmael or nah? [05:21:43] 6operations, 7Database: decom ishmael? - https://phabricator.wikimedia.org/T109777#1688305 (10Dzahn) [05:21:44] 6operations, 10Wikimedia-General-or-Unknown, 7Database, 7Performance: ishmael shows blank graphs - https://phabricator.wikimedia.org/T66581#1688304 (10Dzahn) [05:23:34] 6operations, 7Database: decom ishmael? - https://phabricator.wikimedia.org/T109777#1558908 (10Dzahn) yea, he basically answered already on T82225#1556293 so i i see no reason to keep ishmael. that would mean reject tickets to fix stuff inside ishmael. [05:25:46] (03PS1) 10Yuvipanda: tools: Make webproxy role not inherit from toollabs [puppet] - 10https://gerrit.wikimedia.org/r/242439 [05:26:18] 6operations: package and puppetize ishmael - https://phabricator.wikimedia.org/T82225#1688307 (10Dzahn) suggest to set as "declined" in favor of decom ishmael [05:26:58] (03PS2) 10Yuvipanda: tools: Make webproxy role not inherit from toollabs [puppet] - 10https://gerrit.wikimedia.org/r/242439 [05:27:48] (03PS3) 10Yuvipanda: tools: Make webproxy role not inherit from toollabs [puppet] - 10https://gerrit.wikimedia.org/r/242439 [05:27:56] (03CR) 10Yuvipanda: [C: 032 V: 032] tools: Make webproxy role not inherit from toollabs [puppet] - 10https://gerrit.wikimedia.org/r/242439 (owner: 10Yuvipanda) [05:33:14] (03PS1) 10Yuvipanda: tools: Move 'web_domain' param to proxy manifest too [puppet] - 10https://gerrit.wikimedia.org/r/242441 [05:33:18] (03CR) 10jenkins-bot: [V: 04-1] tools: Move 'web_domain' param to proxy manifest too [puppet] - 10https://gerrit.wikimedia.org/r/242441 (owner: 10Yuvipanda) [05:34:12] (03PS2) 10Yuvipanda: tools: Move 'web_domain' param to proxy manifest too [puppet] - 10https://gerrit.wikimedia.org/r/242441 [05:36:26] (03CR) 10Yuvipanda: [C: 032] tools: Move 'web_domain' param to proxy manifest too [puppet] - 10https://gerrit.wikimedia.org/r/242441 (owner: 10Yuvipanda) [05:43:20] !log l10nupdate@tin ResourceLoader cache refresh completed at Wed Sep 30 05:43:19 UTC 2015 (duration 43m 18s) [05:43:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [05:44:42] (03PS1) 10Yuvipanda: tools: Do not have tools webproxy inherit the common role [puppet] - 10https://gerrit.wikimedia.org/r/242442 [05:45:20] (03PS2) 10Yuvipanda: tools: Do not have tools webproxy inherit the common role [puppet] - 10https://gerrit.wikimedia.org/r/242442 [05:46:36] (03CR) 10Yuvipanda: [C: 032] tools: Do not have tools webproxy inherit the common role [puppet] - 10https://gerrit.wikimedia.org/r/242442 (owner: 10Yuvipanda) [05:49:54] PROBLEM - puppet last run on mw2212 is CRITICAL: CRITICAL: puppet fail [05:50:55] PROBLEM - HTTP 5xx req/min on graphite1001 is CRITICAL: CRITICAL: 6.67% of data above the critical threshold [500.0] [05:59:52] (03PS1) 10Yuvipanda: toollabs: Make proxylistener support systemd [puppet] - 10https://gerrit.wikimedia.org/r/242443 [06:00:25] (03PS2) 10Yuvipanda: toollabs: Make proxylistener support systemd [puppet] - 10https://gerrit.wikimedia.org/r/242443 [06:01:13] RECOVERY - HTTP 5xx req/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] [06:01:17] (03CR) 10jenkins-bot: [V: 04-1] toollabs: Make proxylistener support systemd [puppet] - 10https://gerrit.wikimedia.org/r/242443 (owner: 10Yuvipanda) [06:02:05] (03PS3) 10Yuvipanda: toollabs: Make proxylistener support systemd [puppet] - 10https://gerrit.wikimedia.org/r/242443 [06:03:04] (03CR) 10Yuvipanda: [C: 032] toollabs: Make proxylistener support systemd [puppet] - 10https://gerrit.wikimedia.org/r/242443 (owner: 10Yuvipanda) [06:13:17] <_joe_> yuvipanda: you have a -1 from jenkins-bot [06:13:28] <_joe_> oh the preceding PS [06:13:31] <_joe_> sorry, scratch that [06:13:46] _joe_: yup I updated [06:18:24] RECOVERY - puppet last run on mw2212 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:23:54] (03PS1) 10Yuvipanda: labsdns: Update tools.wmflabs.org internal value [puppet] - 10https://gerrit.wikimedia.org/r/242447 [06:24:48] (03CR) 10Yuvipanda: [C: 032] labsdns: Update tools.wmflabs.org internal value [puppet] - 10https://gerrit.wikimedia.org/r/242447 (owner: 10Yuvipanda) [06:27:30] (03PS1) 10Yuvipanda: tools: Add k8s::webproxy to tools::proxy [puppet] - 10https://gerrit.wikimedia.org/r/242448 (https://phabricator.wikimedia.org/T111916) [06:29:10] (03CR) 10Yuvipanda: [C: 032] tools: Add k8s::webproxy to tools::proxy [puppet] - 10https://gerrit.wikimedia.org/r/242448 (https://phabricator.wikimedia.org/T111916) (owner: 10Yuvipanda) [06:30:23] PROBLEM - puppet last run on chromium is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:44] PROBLEM - puppet last run on cp2013 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:55] PROBLEM - puppet last run on mw1119 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:55] PROBLEM - puppet last run on db2055 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:24] PROBLEM - puppet last run on mw2158 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:24] PROBLEM - puppet last run on mw2207 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:40] legoktm: you there? [06:32:44] PROBLEM - puppet last run on mw1170 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:45] PROBLEM - puppet last run on mw2045 is CRITICAL: CRITICAL: Puppet has 1 failures [06:33:24] PROBLEM - puppet last run on mw1110 is CRITICAL: CRITICAL: Puppet has 1 failures [06:34:32] (03PS1) 10Yuvipanda: tools: Move webproxy code into a role [puppet] - 10https://gerrit.wikimedia.org/r/242449 [06:36:13] (03CR) 10Yuvipanda: [C: 032] tools: Move webproxy code into a role [puppet] - 10https://gerrit.wikimedia.org/r/242449 (owner: 10Yuvipanda) [06:40:11] (03PS1) 10Yuvipanda: tools: Remove redundant k8s::proxy include [puppet] - 10https://gerrit.wikimedia.org/r/242451 [06:40:25] PROBLEM - puppet last run on iron is CRITICAL: CRITICAL: Puppet has 1 failures [06:42:06] (03CR) 10Yuvipanda: [C: 032] tools: Remove redundant k8s::proxy include [puppet] - 10https://gerrit.wikimedia.org/r/242451 (owner: 10Yuvipanda) [06:44:53] RECOVERY - mysqld processes on labsdb1004 is OK: PROCS OK: 1 process with command name mysqld [06:50:00] (03PS1) 10Alexandros Kosiaris: actually use is_critical in monitor_replication [puppet/mariadb] - 10https://gerrit.wikimedia.org/r/242452 [06:50:02] (03PS1) 10Alexandros Kosiaris: Add a .gitreview file [puppet/mariadb] - 10https://gerrit.wikimedia.org/r/242453 [06:51:28] (03PS1) 10Yuvipanda: tools: Provision puppet CA for k8s webproxy [puppet] - 10https://gerrit.wikimedia.org/r/242454 [06:52:23] (03CR) 10Yuvipanda: [C: 032] tools: Provision puppet CA for k8s webproxy [puppet] - 10https://gerrit.wikimedia.org/r/242454 (owner: 10Yuvipanda) [06:55:13] RECOVERY - puppet last run on chromium is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [06:57:23] RECOVERY - puppet last run on cp2013 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [06:59:54] In what cases would the default skin be set for Monobook users? [07:00:24] I'm asking because my skin got changed to the default on mediawiki.org and simple.wikipedia [07:00:25] o_O [07:00:30] But no others [07:00:48] Bsadowski1: if you reset your preferences [07:00:54] I did not. [07:01:00] It just... happened. [07:01:38] I knew I had Monobook set, but for some odd reason it changed to the default skin, which would be Vector. [07:02:45] (03CR) 10Yuvipanda: [C: 04-1] dynamicproxy: add support for kubernetes WIP (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/241908 (owner: 10Giuseppe Lavagetto) [07:02:48] Oh I see [07:02:49] https://phabricator.wikimedia.org/T114208 [07:02:53] ugh :p [07:03:11] It's a known bug I see. [07:04:48] (03CR) 10Yuvipanda: "I'll also note that tools-k8s-master-01.tools.eqiad.wmflabs:6443 works for me when called from wherever." (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/241908 (owner: 10Giuseppe Lavagetto) [07:05:33] RECOVERY - puppet last run on iron is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [07:07:19] (03CR) 10Yuvipanda: "Should also do a check based on hiera('active_proxy_host') and make sure it is running only on that host and stopped on other hosts. activ" [puppet] - 10https://gerrit.wikimedia.org/r/241908 (owner: 10Giuseppe Lavagetto) [07:07:34] (03CR) 10Alexandros Kosiaris: [C: 032] Add a .gitreview file [puppet/mariadb] - 10https://gerrit.wikimedia.org/r/242453 (owner: 10Alexandros Kosiaris) [07:09:13] (03CR) 10ArielGlenn: [C: 032 V: 032] worker.py: fix indentation issues, many camelcase names [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/242396 (owner: 10ArielGlenn) [07:10:20] (03PS1) 10ArielGlenn: CommandManagement: fix indentation and many camelcase issues [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/242456 [07:12:57] (03CR) 10Yuvipanda: "Also put any k8s related proxy role code in manifests/role/tools's proxy role class." [puppet] - 10https://gerrit.wikimedia.org/r/241908 (owner: 10Giuseppe Lavagetto) [07:13:15] (03CR) 10ArielGlenn: [C: 032 V: 032] CommandManagement: fix indentation and many camelcase issues [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/242456 (owner: 10ArielGlenn) [07:14:07] (03PS1) 10ArielGlenn: fileutils: pylint, clean up a bunch of camelcases [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/242457 [07:16:22] (03CR) 10Yuvipanda: "Run it as a 'kubernetes' user maybe? That's what the apiserver runs as and you can get the user by including k8s::users." [puppet] - 10https://gerrit.wikimedia.org/r/241908 (owner: 10Giuseppe Lavagetto) [07:19:49] (03PS1) 10ArielGlenn: runnerutils: pylint, convert many camelcases [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/242458 [07:22:50] (03CR) 10ArielGlenn: [C: 032 V: 032] runnerutils: pylint, convert many camelcases [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/242458 (owner: 10ArielGlenn) [07:23:07] (03PS2) 10Alexandros Kosiaris: Add a .gitreview file [puppet/mariadb] - 10https://gerrit.wikimedia.org/r/242453 [07:23:09] (03PS2) 10Alexandros Kosiaris: actually use is_critical in monitor_replication [puppet/mariadb] - 10https://gerrit.wikimedia.org/r/242452 [07:27:14] RECOVERY - puppet last run on mw2158 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [07:27:14] RECOVERY - puppet last run on mw2207 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [07:27:24] RECOVERY - puppet last run on mw1170 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:27:24] RECOVERY - puppet last run on mw1119 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [07:27:33] RECOVERY - puppet last run on db2055 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:27:43] RECOVERY - puppet last run on mw2045 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [07:28:14] RECOVERY - puppet last run on mw1110 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:28:31] (03PS1) 10ArielGlenn: utils.py: pylint, fix many camelcase names. worker.py, fix indent issue [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/242464 [07:32:21] (03CR) 10ArielGlenn: [C: 032 V: 032] utils.py: pylint, fix many camelcase names. worker.py, fix indent issue [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/242464 (owner: 10ArielGlenn) [07:33:41] (03PS1) 10ArielGlenn: jobs.py: clean up many camelcase issues [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/242465 [07:46:53] (03CR) 10ArielGlenn: [C: 032 V: 032] jobs.py: clean up many camelcase issues [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/242465 (owner: 10ArielGlenn) [07:48:12] (03PS1) 10ArielGlenn: get rid of last camelcase in CommandManagement [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/242467 [07:50:19] (03CR) 10ArielGlenn: [C: 032 V: 032] get rid of last camelcase in CommandManagement [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/242467 (owner: 10ArielGlenn) [07:51:43] (03PS1) 10ArielGlenn: remove last of camelcase from fileutils [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/242470 [07:56:33] (03CR) 10ArielGlenn: [C: 032 V: 032] remove last of camelcase from fileutils [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/242470 (owner: 10ArielGlenn) [07:57:51] (03PS1) 10ArielGlenn: dumps: remove last camelcase from jobs.py [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/242474 [07:58:52] (03CR) 10ArielGlenn: [C: 032 V: 032] dumps: remove last camelcase from jobs.py [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/242474 (owner: 10ArielGlenn) [08:04:54] PROBLEM - mailman_queue_size on fermium is CRITICAL: CRITICAL: 1 mailman queue(s) above 100 [08:07:16] (03PS1) 10ArielGlenn: remove last camelcase from runnerutils.py [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/242479 [08:09:10] (03CR) 10ArielGlenn: [C: 032 V: 032] remove last camelcase from runnerutils.py [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/242479 (owner: 10ArielGlenn) [08:19:11] (03PS1) 10ArielGlenn: fix up a couple stray camelcases in Runner attributes [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/242485 [08:19:44] RECOVERY - mailman_queue_size on fermium is OK: OK: mailman queues are below 100 [08:20:41] (03CR) 10ArielGlenn: [C: 032 V: 032] fix up a couple stray camelcases in Runner attributes [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/242485 (owner: 10ArielGlenn) [08:38:28] (03CR) 10Alexandros Kosiaris: "As pointed out in the corresponding task, we will probably have to end up with more than one package to differentiate between package vers" [puppet] - 10https://gerrit.wikimedia.org/r/229304 (owner: 10GWicke) [08:40:05] (03PS1) 10ArielGlenn: fix last camelcases in utils.py [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/242490 [08:41:10] apergos: are you doing some linting on operations/dumps ? I noticed the jobs are not voting :D [08:41:26] they are not voting [08:41:34] yes I am doing a huge amount of pylint cleanup [08:42:03] I need to be able to pylint some complex changes I make later, and righ tnow the state of that code is... well now it's better but it was horrid. and still not great [08:42:04] hashar_: [08:42:24] <_joe_> apergos: so, about snapshots... we can't convert them to HHVM, right? [08:42:31] not until upstream gets done [08:42:39] there is a patch for internal review apparently [08:42:41] <_joe_> we can still go with trusty and just fix the php.ini and use zend [08:42:43] yes [08:42:53] <_joe_> so that we can ditch precise there [08:43:04] well no need to fix php.ini [08:43:19] I have that one config change for dumps I can push out, then revert when we have the packages with the fix [08:43:26] apergos: let me add the CI entry point based on tox :D [08:43:28] and I can do te last snapshot host today (to trust) [08:43:47] three are converted. one left. [08:43:56] _joe_: [08:44:17] hashar_: what would that do? I hope it wouldn't make that voting already, too soon! [08:44:28] <_joe_> apergos: ok, what's the patch in question? [08:44:57] to add compress.bzip2 capability to their implementation of bzip2, they left it out by mistake :-D [08:45:19] righ tnow there is not yet a public patch for us tolook at [08:46:05] _joe_: [08:46:27] <_joe_> no I mean your patch to revert use of hhvm [08:46:45] oh. it's just a config setting in the dump configs files [08:47:06] I haven't pshed it yet but it will go today just before I convert the last snapshot [08:47:16] it will use php5 instead of php [08:47:40] https://gerrit.wikimedia.org/r/#/c/241612/ [08:47:51] apergos: That's php 5.5 then? [08:47:58] let me have a look [08:48:40] indeed it is [08:48:48] Ok, nice [08:48:57] yep [08:49:30] so hashar_, what does adding that CI entry point do? [08:54:25] (03CR) 10ArielGlenn: [C: 032 V: 032] fix last camelcases in utils.py [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/242490 (owner: 10ArielGlenn) [08:54:28] (03CR) 1020after4: [C: 031] Add config deployment [tools/scap] - 10https://gerrit.wikimedia.org/r/240292 (https://phabricator.wikimedia.org/T109512) (owner: 10Thcipriani) [08:54:33] apergos: we have a Jenkins job named 'fox-jessie' which clone your repo, checkout the proposed patch and simply invokes 'tox' [08:54:56] apergos: tox is a wrapper around virtualenv to define a bunch of venv to run commands into. Such as flake8 that runs both pep8 and pyflakes [08:54:57] <_joe_> apergos: the mediawiki module for trusty doesn't have the correct php ini settings, but maybe you're not using it [08:55:13] apergos: so once the Jenkins job is configured, you can tweak the job run by editing the tox.ini file and add more commands as needed [08:55:45] <_joe_> apart from that, I don't see any gain we could have from running it in hhvm, given jit is out of the question in this case [09:02:42] (03PS1) 10Hashar: tox integration to run flake8 [dumps] - 10https://gerrit.wikimedia.org/r/242494 [09:03:32] (03CR) 10Hashar: "I have configured CI to have it run tox in a Jessie instance by commenting 'check experimental'. Once it passes, we can make the job to b" [dumps] - 10https://gerrit.wikimedia.org/r/242494 (owner: 10Hashar) [09:03:40] (03CR) 10Hashar: "check experimental" [dumps] - 10https://gerrit.wikimedia.org/r/242494 (owner: 10Hashar) [09:04:16] (03PS2) 10Hashar: tox integration to run flake8 [dumps] - 10https://gerrit.wikimedia.org/r/242494 (https://phabricator.wikimedia.org/T55354) [09:04:26] apergos: example run https://integration.wikimedia.org/ci/job/tox-jessie/17/console :) [09:09:20] (03PS1) 10ArielGlenn: last camelcase gone except for WikiDump.py [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/242495 [09:10:33] _joe_: I'm using it as far as I know [09:10:41] mediawiki::packages::php5 is in the puppet classes on the hosts [09:10:57] <_joe_> apergos: ok, what host is still not converted? [09:11:05] <_joe_> so that I can take a look at the ini files [09:11:06] snapshot1003 [09:11:34] <_joe_> ok thanks [09:11:39] <_joe_> I'll take a look in a few [09:13:24] hashar: gotcha, nice [09:15:48] _joe_: remember I'm using the cli php.ini, maybe that makes the difference [09:15:59] <_joe_> yeah I know [09:16:18] <_joe_> apergos: I just want to check there is no screwup in precise => trusty [09:16:25] right [09:23:29] (03PS2) 10ArielGlenn: last camelcase gone except for WikiDump.py [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/242495 [09:27:32] (03CR) 10ArielGlenn: [C: 032 V: 032] last camelcase gone except for WikiDump.py [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/242495 (owner: 10ArielGlenn) [09:28:57] (03PS2) 10ArielGlenn: Fix URL to interwiki cache on noc.wikimedia.org [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/220075 (owner: 10Hydriz) [09:29:01] (03CR) 10Phuedx: [C: 031] Add Extension:RelatedArticles to beta labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/242362 (https://phabricator.wikimedia.org/T113770) (owner: 10Jdlrobson) [09:30:06] (03CR) 10ArielGlenn: [C: 032] Fix URL to interwiki cache on noc.wikimedia.org [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/220075 (owner: 10Hydriz) [09:30:15] (03CR) 10ArielGlenn: [V: 032] Fix URL to interwiki cache on noc.wikimedia.org [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/220075 (owner: 10Hydriz) [09:39:44] <_joe_> apergos: so we miss php5-igbinary [09:39:59] <_joe_> which is debian-backported on precise, but not on trusty [09:40:06] <_joe_> this is the first difference I see [09:40:10] (03PS1) 10ArielGlenn: dump change tags table, T68700 [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/242499 [09:42:12] <_joe_> then we have to convert what we had in apc.ini in what is in opcache.ini I guess [09:42:27] <_joe_> as php 5.5 got rid of the apc extension IIRC [09:43:07] <_joe_> as an opcode cache I mean [09:43:16] <_joe_> we don't even install the extension I guess [09:44:03] <_joe_> so yeah, we might need to install php-apc there and use a dedicated config [09:45:06] (03PS5) 10Ori.livneh: WIP: Add etcd configuration client [debs/pybal] - 10https://gerrit.wikimedia.org/r/225649 [09:45:30] _joe_: almost ^. but i ran out of it steam. tomorrrrowwwww [09:45:47] <_joe_> ori: \o/ [09:45:54] <_joe_> and yes, sleep! [09:46:01] good grief you are stil here? go go go [09:46:03] <_joe_> can I peek at it in the meanwhile? :) [09:46:42] <_joe_> apergos: I cannot really work on this as I have to finsh up the kubernetes work today [09:46:48] _joe_: without that how are my jobs going to be impacted? [09:46:54] <_joe_> but I hope my pointers are enough [09:46:55] 6operations, 10RESTBase: enable restbase syslog/file logging - https://phabricator.wikimedia.org/T112648#1688645 (10Pchelolo) [09:47:04] <_joe_> apergos: they might fail if they use apc [09:47:14] <_joe_> I mean it was enabled on the old snapshot hosts [09:47:21] well I can surely check that; I've done some php5 runs already and they went through ok [09:47:28] but I can make sure I test the few jobs I did not [09:47:31] <_joe_> so you should enable it BUT disable its use as an opcode cache [09:47:36] <_joe_> oh ok [09:48:18] this is why I think the switch will be ok; for igbinary if it's a matter of a little slower this run, it's fine [09:49:14] so go do yer kubernetes stuff and thanks for looking at this with me [09:51:57] <_joe_> :) [09:52:00] <_joe_> ok nice [09:54:34] (03PS3) 10Hashar: lint: fix 'variable not enclosed' pt2 [puppet] - 10https://gerrit.wikimedia.org/r/242057 (owner: 10Dzahn) [09:55:41] (03CR) 10Hashar: [C: 031] "+ Alexandros for the resolve() parts" [puppet] - 10https://gerrit.wikimedia.org/r/242057 (owner: 10Dzahn) [09:58:02] (03CR) 10Hashar: [C: 031] lint: fix 'variable not enclosed' warnings [puppet] - 10https://gerrit.wikimedia.org/r/242055 (owner: 10Dzahn) [09:58:14] yep verified I need neither of those, good to go. [10:10:53] (03PS1) 10ArielGlenn: lost a camelcase conversion in utils.py, fixed [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/242503 [10:13:02] (03PS2) 10Muehlenhoff: Disable the ferm rules cache [puppet] - 10https://gerrit.wikimedia.org/r/240335 (https://phabricator.wikimedia.org/T113380) [10:14:54] (03CR) 10Alexandros Kosiaris: [C: 031] lint: fix 'variable not enclosed' pt2 [puppet] - 10https://gerrit.wikimedia.org/r/242057 (owner: 10Dzahn) [10:15:37] (03CR) 10Alexandros Kosiaris: [C: 031] Disable the ferm rules cache [puppet] - 10https://gerrit.wikimedia.org/r/240335 (https://phabricator.wikimedia.org/T113380) (owner: 10Muehlenhoff) [10:18:28] 6operations, 6Phabricator, 6Project-Creators: create acl*operationsteam & acl*procurement projects, cease using #operations for access control - https://phabricator.wikimedia.org/T114135#1688709 (10Liuxinyu970226) [10:35:41] (03CR) 10Bmansurov: [C: 031] Add Extension:RelatedArticles to beta labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/242362 (https://phabricator.wikimedia.org/T113770) (owner: 10Jdlrobson) [10:36:40] (03CR) 10Bmansurov: "Please, someone with a +2 power, merge." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/242362 (https://phabricator.wikimedia.org/T113770) (owner: 10Jdlrobson) [10:36:51] (03PS2) 10ArielGlenn: fix outliers from camelcase conversion [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/242503 [10:38:28] (03PS3) 10ArielGlenn: fix outliers from camelcase conversion and from a bad merge [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/242503 [10:41:19] (03CR) 10ArielGlenn: [C: 032 V: 032] fix outliers from camelcase conversion and from a bad merge [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/242503 (owner: 10ArielGlenn) [10:42:09] (03PS2) 10ArielGlenn: dump change tags table, T68700 [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/242499 [10:42:55] (03CR) 10Muehlenhoff: [C: 032 V: 032] Disable the ferm rules cache [puppet] - 10https://gerrit.wikimedia.org/r/240335 (https://phabricator.wikimedia.org/T113380) (owner: 10Muehlenhoff) [10:45:54] (03PS3) 10ArielGlenn: dump change tag table, T68700 [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/242499 [10:47:50] (03CR) 10ArielGlenn: [C: 032 V: 032] dump change tag table, T68700 [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/242499 (owner: 10ArielGlenn) [10:48:44] PROBLEM - puppet last run on mw2082 is CRITICAL: CRITICAL: puppet fail [10:50:49] (03PS1) 10ArielGlenn: more camelcase and merge outliers, not commited with prev changeset [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/242506 [10:51:49] (03CR) 10ArielGlenn: [C: 032 V: 032] more camelcase and merge outliers, not commited with prev changeset [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/242506 (owner: 10ArielGlenn) [10:51:53] (03PS1) 10Alexandros Kosiaris: Set analytics mariadb replication monitor to non critical [puppet] - 10https://gerrit.wikimedia.org/r/242507 [10:52:28] 6operations, 6Discovery, 7Elasticsearch, 5Patch-For-Review: Ferm doesn't update @resolve hostnames on IP change - https://phabricator.wikimedia.org/T113380#1688763 (10MoritzMuehlenhoff) 5Open>3Resolved The ferm cache has now been disabled, i.e. after an IP address change a simple restart of ferm will p... [11:03:13] (03PS2) 10ArielGlenn: dumps: fall back to php5 instead of hhvm for now [puppet] - 10https://gerrit.wikimedia.org/r/241612 [11:06:04] (03CR) 10ArielGlenn: [C: 032] "until our upstream bug in hvm is fixed." [puppet] - 10https://gerrit.wikimedia.org/r/241612 (owner: 10ArielGlenn) [11:11:58] (03PS1) 10Alexandros Kosiaris: Introduce ticket-test.wikimedia.org [dns] - 10https://gerrit.wikimedia.org/r/242512 [11:14:25] 6operations, 7user-notice: schedule maintenance for IRC server - https://phabricator.wikimedia.org/T105804#1688808 (10MoritzMuehlenhoff) Judging from the current ferm rules it seems we can schedule this now? [11:16:44] RECOVERY - puppet last run on mw2082 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:21:27] PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet). [11:22:34] PROBLEM - Unmerged changes on repository puppet on palladium is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet). [11:22:34] (03PS2) 10Alexandros Kosiaris: Introduce ticket-test.wikimedia.org [dns] - 10https://gerrit.wikimedia.org/r/242512 [11:25:52] (03PS3) 10Alexandros Kosiaris: Introduce otrs-test.wikimedia.org [dns] - 10https://gerrit.wikimedia.org/r/242512 [11:25:58] (03PS1) 10Alexandros Kosiaris: misc-web: Get otrs-test.wikimedia.org behind it [puppet] - 10https://gerrit.wikimedia.org/r/242516 [11:39:29] (03PS4) 10Giuseppe Lavagetto: dynamicproxy: add support for kubernetes [puppet] - 10https://gerrit.wikimedia.org/r/241908 (https://phabricator.wikimedia.org/T111916) [11:40:18] (03CR) 10jenkins-bot: [V: 04-1] dynamicproxy: add support for kubernetes [puppet] - 10https://gerrit.wikimedia.org/r/241908 (https://phabricator.wikimedia.org/T111916) (owner: 10Giuseppe Lavagetto) [11:44:14] (03PS5) 10Giuseppe Lavagetto: dynamicproxy: add support for kubernetes [puppet] - 10https://gerrit.wikimedia.org/r/241908 (https://phabricator.wikimedia.org/T111916) [11:57:11] (03PS2) 10Alexandros Kosiaris: Set analytics mariadb replication monitor to non critical [puppet] - 10https://gerrit.wikimedia.org/r/242507 [11:57:17] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] Set analytics mariadb replication monitor to non critical [puppet] - 10https://gerrit.wikimedia.org/r/242507 (owner: 10Alexandros Kosiaris) [11:59:04] RECOVERY - Unmerged changes on repository puppet on strontium is OK: No changes to merge. [12:00:14] RECOVERY - Unmerged changes on repository puppet on palladium is OK: No changes to merge. [12:02:55] (03PS2) 10Alexandros Kosiaris: otrs: add missing perl modules for OTRS 4.0.13 [puppet] - 10https://gerrit.wikimedia.org/r/242205 [12:02:57] (03PS2) 10Alexandros Kosiaris: Add the new OTRS scheduler watchdog cron entry [puppet] - 10https://gerrit.wikimedia.org/r/242184 [12:02:59] (03PS2) 10Alexandros Kosiaris: otrs: Update apache configuration [puppet] - 10https://gerrit.wikimedia.org/r/242192 [12:03:52] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] otrs: add missing perl modules for OTRS 4.0.13 [puppet] - 10https://gerrit.wikimedia.org/r/242205 (owner: 10Alexandros Kosiaris) [12:04:37] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] otrs: Update apache configuration [puppet] - 10https://gerrit.wikimedia.org/r/242192 (owner: 10Alexandros Kosiaris) [12:15:02] (03PS1) 10KartikMistry: Enable CX suggestions in ar, eo, hi, nl, vi and dawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/242528 [12:25:54] _joe_, akosiaris: How do I get https://phabricator.wikimedia.org/T114208 escalated? [12:27:03] It's really not acceptable to muck with user preferences like that. Is there a plan to restore the previous preferences? [12:28:19] Katie: I just became aware of this, thanks to you. let me have a look [12:28:54] ostriches and Krenair were cleaning up the userprefs tables [12:29:34] Katie: https://phabricator.wikimedia.org/T54778 [12:29:44] akosiaris: Cool, thanks. I thought it was only applicable to group0 wikis, but it seems like maybe it's larger than that. :-/ [12:30:22] Glaisher: Oh, I see. small and medium wikis... blah. [12:34:48] !log added debdeploy 0.0.8 to carbon [12:34:53] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [12:35:38] Katie: the way I read this, the best people would be ostriches and Krenair. If there is some restoration of data that needs to be done and they haven't kept a backup anyway, we do have backups to revert to and will have for some time (safe bet is about a month). But before anything else, they should be made aware [12:36:59] All right. They're subscribed to that task. I was annoyed that it was difficult to figure out what happened and why. [12:37:44] yeah, understandable [12:37:46] Thanks for taking a look. :-) [12:37:56] yw [12:45:11] 6operations, 7Database: New hardware for production core mysql cluster - https://phabricator.wikimedia.org/T106847#1688978 (10jcrespo) Procurement request was sent, RT https://rt.wikimedia.org/Ticket/Display.html?id=9660 [12:45:56] (03PS1) 10Rush: phab: update name of apache stats collector [puppet] - 10https://gerrit.wikimedia.org/r/242531 [12:55:52] (03PS1) 10Rush: elasticsearch: update row for master eligibles [puppet] - 10https://gerrit.wikimedia.org/r/242532 [12:56:10] (03CR) 10Jcrespo: [C: 031] "Ready to merge, but it requires human oversight, will schedule it soon." [puppet] - 10https://gerrit.wikimedia.org/r/242211 (owner: 10Jcrespo) [12:58:44] 6operations, 10RESTBase, 6Services: Switch RESTBase to use iojs - https://phabricator.wikimedia.org/T107762#1689047 (10mobrovac) [12:59:15] Hey Ops! In Git, does Operations have any path structure for "repositories which we only pull from upstream and never do any downstream changes to their codebase"? [12:59:17] Especially wondering about operations/debs/* and operations/software/* here [12:59:30] (03CR) 10Rush: [C: 032] elasticsearch: update row for master eligibles [puppet] - 10https://gerrit.wikimedia.org/r/242532 (owner: 10Rush) [13:01:46] Hmm. Probably not operations/software/*, looking at recent commits by WM. Maybe operations/debs/* ? [13:02:18] andre__: are you asking is there some path for a repo in gerrit that says we deploy only the vanilla upstream? [13:03:01] 6operations, 10ops-eqiad, 5Patch-For-Review: Swap two elasticsearch servers in row D with an elasticsearch server in racks A3 and C5. - https://phabricator.wikimedia.org/T112559#1689056 (10chasemp) [13:03:19] 6operations, 10RESTBase, 6Services: Switch RESTBase to use iojs - https://phabricator.wikimedia.org/T107762#1689059 (10akosiaris) >>! In T107762#1689046, @mobrovac wrote: >>>! In T107762#1689006, @akosiaris wrote: >> I am also worried about those as well. So, depending to what @MoritzMuehlenhoff reports back... [13:03:43] (03PS2) 10Rush: phab: update name of apache stats collector [puppet] - 10https://gerrit.wikimedia.org/r/242531 [13:03:54] (03CR) 10Rush: [C: 032 V: 032] phab: update name of apache stats collector [puppet] - 10https://gerrit.wikimedia.org/r/242531 (owner: 10Rush) [13:04:33] chasemp, yes [13:04:39] thanks, that's better words [13:05:31] there is no such alignment, some things in ops/software we are upstream, and if we are packaging it ourselves decent chance we have modifications [13:05:43] there is no such alignment -- that I am aware of [13:05:50] what is this for? [13:06:11] chasemp: so I'm looking at committer/author statistics for our Git repos. [13:06:19] chasemp: And I'm trying to answer the question if numbers could get polluted by e.g "we pull HHVM from upstream and many of the commits we imported are just from upstream devs" [13:06:47] so it's shown as author activity in Wikimedia Git, but of course not in Wikimedia Gerrit (as those chances never went thru WM Gerrit as they were "just" imported) [13:06:56] chasemp, Does that question make sense? [13:07:50] <_joe_> andre__: so specifically about HHVM [13:07:57] <_joe_> what we have is a deb package repo [13:08:06] (HHVM is just one obvious example that came to my mind) [13:08:10] <_joe_> so no, none of the upstream commits are in it's git history [13:08:21] ah. Alright, that's already helpful to know [13:08:27] <_joe_> it's just imports of specific tags from upstream [13:08:41] <_joe_> and then building the package and our patches onto it [13:09:08] <_joe_> authorship of said patches is best determined by looking at each one in debian/patches as they should be in DEP-3 format [13:09:35] <_joe_> and this is valid for any debian package for which we don't maintain the upstream as well [13:09:53] <_joe_> so, if we are the ones doing the packaging, we usually do like I did for HHVM [13:10:02] What I basically want to know: If I "git clone" that repo from Wikimedia Git and run "git log" will it also list upstream edits [13:10:12] (*cough*, I could just try this myself. Sorry) [13:10:16] <_joe_> andre__: it depends, but usually nope [13:10:34] I was thinking along the lines of phabricator or whatever where we have commits in upstream itself [13:10:50] andre__: what is this for? [13:11:03] chasemp: https://phabricator.wikimedia.org/T103292 [13:11:31] One theory I simply had is that commits pulled from upstream pollute our stats for Git [13:11:44] ah, well taht is interesting [13:12:09] <_joe_> andre__: the only way to see that correctly is to whitelist repos [13:12:17] chasemp, yeah. Though before thinking about reasons we need to make sure that the data is actually correct. Another fun step. [13:12:30] _joe_, yeah but for whitelisting I need criteria plus understand workflows. hence I asked here :) [13:12:36] <_joe_> anything else is pointless and will give you false results [13:12:51] <_joe_> andre__: exclude any debian packaging effort from your numbers, ofc [13:13:08] <_joe_> because well, YMMV with ops [13:13:26] _joe_, heh, yeah, but for that I need to identify criteria and understand workflows first :P [13:13:28] <_joe_> andre__: exclude whatever is under operations/debs but a few that I can research for you [13:13:35] _joe_, can you generalize factors for the "it depends" in "it depends, but usually nope"? :) [13:13:38] ah, maybe you just did :D [13:13:51] <_joe_> like pybal, it's under operations/debs [13:13:55] <_joe_> but it's homemade [13:14:08] <_joe_> (and actually went from 1 dev to 3/4 last year :P) [13:14:46] Note to myself: Wondering if the same workflow to import specific tags from upstream is also used by other teams that might use quite some upstream/3rd party like Analytics. I should ask them too. [13:15:47] chasemp, _joe_: Alright, that's already very helpful. Now I'll try to summarize this conversation :) [13:15:48] And while I know it's not your area you're of course also welcome to comment on that task if that topic interests you, if you feel like, and if you have the time. [13:15:50] <_joe_> I'm not saying that we don't work on those - actually a lot of effort goes into that packaging [13:16:05] <_joe_> but some of them could have upstream commits [13:16:16] <_joe_> and should be discarded from your counts [13:16:21] it's also probably a bad indicator of this problem in that time frame [13:16:37] <_joe_> also, you should keep in mind some wmf-related projects don't use gerrit anymore [13:16:47] oh yeah, like mobile apps [13:16:48] <_joe_> and more are following [13:16:58] <_joe_> andre__: mobile apps, services team, maps... [13:17:47] <_joe_> so yeah, that data are garbage, basically [13:17:50] Uh. Some of that is actually news to me. And might definitely explain things. [13:17:51] some analytics too I think [13:18:02] That's a very very good point. Thanks, seriously [13:18:12] <_joe_> andre__: blame mobrovac urandom and gwicke :P [13:18:36] <_joe_> oh, and MaxSem and yurik too I guess [13:18:37] Tss, I don't blame. If at all, I gently lart people. ;) [13:18:50] <_joe_> andre__: I was joking, actually [13:18:55] euh? [13:19:03] <_joe_> I'm just trying to ping them to annoy em [13:19:04] <_joe_> :P [13:19:06] And I wasn't joking. As usual! ;) [13:19:06] <_joe_> see ^^ [13:19:13] <_joe_> mission accomplished [13:19:14] Thanks guys. This was a very helpful conversation. [13:19:22] <_joe_> hi marko :) how are you? [13:19:49] hahaha _joe_ [13:19:56] it's not friday!!! [13:20:06] <_joe_> oh i feel like it is [13:20:06] End of quarter is Friday'ish enough! [13:20:12] <_joe_> exactly [13:20:15] good point andre__ [13:21:09] heh [13:21:12] andre__: re gerrit, i think i pointed out to you that we don't use gerrit right before the clean-up day [13:21:24] we == services team [13:21:35] * MaxSem bites _joe_ [13:21:57] (03PS6) 10Giuseppe Lavagetto: dynamicproxy: add support for kubernetes [puppet] - 10https://gerrit.wikimedia.org/r/241908 (https://phabricator.wikimedia.org/T111916) [13:22:31] mobrovac: you very likely did, and I very likely should have made that line in my notes. Sorry (many teams telling me many things). :-/ [13:22:49] :) [13:23:20] mobrovac, Still what I'm after is declining *Wikimedia Git* author activity, not Gerrit. [13:23:34] (Though I still don't really know how much you can bypass Gerrit with our setup. /me not a coder) [13:24:31] (03PS1) 10Rush: phab: apachetop for debugging [puppet] - 10https://gerrit.wikimedia.org/r/242538 [13:25:11] 6operations, 7Database: implement performance_schema for wmf prod - https://phabricator.wikimedia.org/T99485#1689102 (10jcrespo) p:5Normal>3High [13:25:29] mobile apps do use gerrit [13:26:38] 6operations, 7Database: decom ishmael? - https://phabricator.wikimedia.org/T109777#1689116 (10jcrespo) [13:26:40] 6operations: package and puppetize ishmael - https://phabricator.wikimedia.org/T82225#1689113 (10jcrespo) 5Open>3declined a:3jcrespo Use current tendril functionality (based on `SHOW PROCESSLIST`) + pt-query-digest manual runs until a proper functionality replacement is implemented: T99485 [13:26:58] andre__: for github git activity, there's an api for getting the feed for a team/member, but not sure if that'd help [13:27:18] https://developer.github.com/v3/activity/ [13:27:28] (03CR) 10Giuseppe Lavagetto: "I am not sure about yuvi's comment on readonly slaves, if it has to do with redis or what. I don't see any harm in having N proxies all wa" [puppet] - 10https://gerrit.wikimedia.org/r/241908 (https://phabricator.wikimedia.org/T111916) (owner: 10Giuseppe Lavagetto) [13:28:43] mobrovac, noted. thanks! [13:29:43] 6operations, 10Wikimedia-General-or-Unknown, 7Database, 7Performance: ishmael shows blank graphs - https://phabricator.wikimedia.org/T66581#1689119 (10jcrespo) Use current tendril functionality (`SHOW PROCESSLIST`) + pt-query-digest manual runs until a proper functionality replacement is implemented: T99485 [13:29:55] 6operations, 10Wikimedia-General-or-Unknown, 7Database, 7Performance: ishmael shows blank graphs - https://phabricator.wikimedia.org/T66581#1689121 (10jcrespo) 5Open>3declined [13:37:56] 10Ops-Access-Requests, 6operations: RESTBase Admin access on aqs1001, aqs1002, and aqs1003 for Joseph and Dan - https://phabricator.wikimedia.org/T113416#1689133 (10Milimetric) +1 Dzahn, that sounds perfect. Thanks for explaining. [13:46:00] (03CR) 10Tim Landscheidt: "proxy.pp sets up a Redis server on the host pointed to by $active_proxy_host and a Redis slave pulling from $active_proxy_host everywhere " [puppet] - 10https://gerrit.wikimedia.org/r/241908 (https://phabricator.wikimedia.org/T111916) (owner: 10Giuseppe Lavagetto) [13:52:14] (03CR) 10Giuseppe Lavagetto: "@Tim: I already figured that out, thanks for pointing it out anyways; I am disabling the service wherever we're not running on the master." [puppet] - 10https://gerrit.wikimedia.org/r/241908 (https://phabricator.wikimedia.org/T111916) (owner: 10Giuseppe Lavagetto) [13:52:31] (03CR) 10Giuseppe Lavagetto: dynamicproxy: add support for kubernetes (034 comments) [puppet] - 10https://gerrit.wikimedia.org/r/241908 (https://phabricator.wikimedia.org/T111916) (owner: 10Giuseppe Lavagetto) [13:53:34] !log rebooted stat1001/stat1002/stat1003 for kernel updates (already happened between 13:00 UTC and 13:10 UTC, but forgot to log earlier) [13:53:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [13:58:53] jouncebot: next [13:58:53] In 1 hour(s) and 1 minute(s): Morning SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20150930T1500) [14:00:24] (03PS1) 10Filippo Giunchedi: cassandra: ship systemd service file [puppet] - 10https://gerrit.wikimedia.org/r/242548 (https://phabricator.wikimedia.org/T108306) [14:01:26] (03PS2) 10Filippo Giunchedi: cassandra: ship systemd service file [puppet] - 10https://gerrit.wikimedia.org/r/242548 (https://phabricator.wikimedia.org/T108306) [14:03:47] (03PS7) 10Giuseppe Lavagetto: dynamicproxy: add support for kubernetes [puppet] - 10https://gerrit.wikimedia.org/r/241908 (https://phabricator.wikimedia.org/T111916) [14:05:27] (03PS1) 10Muehlenhoff: Remove NMI backports, all folded into 3.19.8-ckt5 [debs/linux] - 10https://gerrit.wikimedia.org/r/242550 [14:05:58] (03CR) 10Muehlenhoff: [C: 032 V: 032] Remove NMI backports, all folded into 3.19.8-ckt5 [debs/linux] - 10https://gerrit.wikimedia.org/r/242550 (owner: 10Muehlenhoff) [14:11:40] 6operations, 10Traffic, 5Patch-For-Review: LVS HTTPS IPv6 on mobile-lb.eqiad alert occasionally flapping - https://phabricator.wikimedia.org/T113154#1689237 (10BBlack) 5Open>3Resolved a:3BBlack No flap paged in the usual timeframe last night. icinga logs are clear of the usual raft of 1/3 soft fails t... [14:12:09] \o/ [14:14:04] (03PS1) 10Muehlenhoff: Update to 3.19.8-ckt5 [debs/linux] - 10https://gerrit.wikimedia.org/r/242554 [14:18:24] (03CR) 10Muehlenhoff: [C: 032 V: 032] Update to 3.19.8-ckt5 [debs/linux] - 10https://gerrit.wikimedia.org/r/242554 (owner: 10Muehlenhoff) [14:25:17] (03CR) 10Eevans: [C: 031] "I like the more incremental approach!" [puppet] - 10https://gerrit.wikimedia.org/r/242548 (https://phabricator.wikimedia.org/T108306) (owner: 10Filippo Giunchedi) [14:29:43] 6operations, 6Analytics-Backlog, 10Analytics-EventLogging, 10MediaWiki-extensions-CentralNotice, 10Traffic: Eventlogging should transparently split large event payloads - https://phabricator.wikimedia.org/T114078#1689330 (10mforns) I'm with Nuria in that we have to evaluate whether the complexity in the... [14:30:42] (03PS1) 10Muehlenhoff: Update to 3.19.8-ckt6 [debs/linux] - 10https://gerrit.wikimedia.org/r/242562 [14:33:29] (03CR) 10Giuseppe Lavagetto: [C: 032] "Let's merge and test this first." [puppet] - 10https://gerrit.wikimedia.org/r/241908 (https://phabricator.wikimedia.org/T111916) (owner: 10Giuseppe Lavagetto) [14:35:12] (03PS1) 10Andrew Bogott: Add labs-recursor1 as a secondary nameserver for labs instances. [puppet] - 10https://gerrit.wikimedia.org/r/242567 (https://phabricator.wikimedia.org/T106142) [14:40:46] 6operations, 10Continuous-Integration-Infrastructure, 10Dumps-Generation, 7WorkType-Maintenance: operations/dumps repo should pass flake8 - https://phabricator.wikimedia.org/T114249#1689407 (10hashar) [14:41:15] (03PS2) 10Andrew Bogott: Add labs-recursor1 as a secondary nameserver for labs instances. [puppet] - 10https://gerrit.wikimedia.org/r/242567 (https://phabricator.wikimedia.org/T106142) [14:41:19] (03PS3) 10Hashar: tox integration to run flake8 [dumps] - 10https://gerrit.wikimedia.org/r/242494 (https://phabricator.wikimedia.org/T55354) [14:42:29] 6operations, 10RESTBase, 10RESTBase-Cassandra: column family cassandra metrics size - https://phabricator.wikimedia.org/T113733#1689412 (10Eevans) What would metrics filtering in [[https://github.com/wikimedia/cassandra-metrics-collector|cmc]] look like? A "patterns file" of some kind? Is it enough to excl... [14:43:16] (03CR) 10Andrew Bogott: [C: 032] Add labs-recursor1 as a secondary nameserver for labs instances. [puppet] - 10https://gerrit.wikimedia.org/r/242567 (https://phabricator.wikimedia.org/T106142) (owner: 10Andrew Bogott) [14:45:52] (03PS4) 10Hashar: tox integration to run flake8 [dumps] - 10https://gerrit.wikimedia.org/r/242494 (https://phabricator.wikimedia.org/T55354) [14:46:07] 6operations, 7Database: decom ishmael? - https://phabricator.wikimedia.org/T109777#1689420 (10Dzahn) a:3Dzahn [14:46:09] (03CR) 10Hashar: "check experimental" [dumps] - 10https://gerrit.wikimedia.org/r/242494 (https://phabricator.wikimedia.org/T55354) (owner: 10Hashar) [14:47:22] (03CR) 10Hashar: "Made it to ignore the most frequent errors (via tox.ini [flake8] section):" [dumps] - 10https://gerrit.wikimedia.org/r/242494 (https://phabricator.wikimedia.org/T55354) (owner: 10Hashar) [14:48:06] (03CR) 10Hashar: "See also https://gerrit.wikimedia.org/r/#/c/242494/ which switch to flake8." [dumps] - 10https://gerrit.wikimedia.org/r/207504 (owner: 10Dereckson) [14:49:04] 6operations: dataset1001/dumps rsync setup should use rsync::server from module - https://phabricator.wikimedia.org/T108992#1689429 (10Dzahn) But maybe besides any firewalling issue, it would still be nice if the dumps setup would use the puppet module? [14:49:29] (03PS2) 10Filippo Giunchedi: restbase-test2001 additional cassandra instances [dns] - 10https://gerrit.wikimedia.org/r/242117 (https://phabricator.wikimedia.org/T95253) [14:49:35] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] restbase-test2001 additional cassandra instances [dns] - 10https://gerrit.wikimedia.org/r/242117 (https://phabricator.wikimedia.org/T95253) (owner: 10Filippo Giunchedi) [14:52:56] matanya: any interest in looking at a dozen quote-mark-related puppet patches? [14:53:08] 6operations, 10RESTBase, 6Services: Switch RESTBase to use iojs - https://phabricator.wikimedia.org/T107762#1689441 (10GWicke) >>! In T107762#1688852, @mobrovac wrote: >>>! In T107762#1687929, @GWicke wrote: >> As a basic test, I have upgraded node to the sid 4.1 packages on deployment-restbase02. RESTBase s... [14:53:55] (03PS1) 10Muehlenhoff: Update to 3.19.8-ckt7 [debs/linux] - 10https://gerrit.wikimedia.org/r/242573 [14:57:03] (03CR) 10Muehlenhoff: [C: 032 V: 032] Update to 3.19.8-ckt6 [debs/linux] - 10https://gerrit.wikimedia.org/r/242562 (owner: 10Muehlenhoff) [14:57:34] (03CR) 10Muehlenhoff: [C: 032 V: 032] Update to 3.19.8-ckt7 [debs/linux] - 10https://gerrit.wikimedia.org/r/242573 (owner: 10Muehlenhoff) [14:57:38] jouncebot: next [14:57:39] In 0 hour(s) and 2 minute(s): Morning SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20150930T1500) [14:57:48] 6operations, 7user-notice: schedule maintenance for IRC server - https://phabricator.wikimedia.org/T105804#1689469 (10Dzahn) The blocker is kind of the knowledge how to properly bring the IRC bot back up. [14:58:34] PROBLEM - Unmerged changes on repository puppet on palladium is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet). [14:58:54] PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet). [15:00:04] anomie ostriches thcipriani marktraceur Krenair: Respected human, time to deploy Morning SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20150930T1500). Please do the needful. [15:00:04] MatmaRex: A patch you scheduled for Morning SWAT (Max 8 patches) is about to be deployed. Please be available during the process. [15:01:14] RECOVERY - Disk space on labstore1002 is OK: DISK OK [15:01:47] 6operations, 10RESTBase, 10RESTBase-Cassandra: column family cassandra metrics size - https://phabricator.wikimedia.org/T113733#1689500 (10fgiunchedi) good question! yeah I think a blacklist would be fine for now re: metric storage space I concur, related is {T85451} even though it got better since May with... [15:02:24] *crickets* [15:02:35] I'm here [15:02:35] (03CR) 10Eminn: "I looked at, there is no problem." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/242096 (https://phabricator.wikimedia.org/T114002) (owner: 10Siebrand) [15:02:45] MatmaRex, hi [15:02:58] morning [15:03:40] afternoon [15:05:14] MatmaRex, oh, you want i18n changes? :/ [15:05:52] bd808, only a full scap will sync those properly, right? [15:06:01] Krenair: not really. no need to rebuild the cache and stuff [15:06:22] Krenair: the messages are changed but mostly reusable… i can revert the i18n changes from those patches, if you want [15:09:33] MatmaRex, does this change really need to go out in swat? [15:10:21] Krenair: we've had at least two people report it on phabricator. like i said, i can easily back out the i18n changes from it, if that's an issue, the rest can stand by itself. [15:10:39] shall i? [15:10:46] I can run a full scap of everything, that's not really the issue [15:11:00] I just wonder whether this is an important enough change to be backporting [15:11:35] i think it's worth it, and it's not like this swat is crowded. :P [15:11:48] but you're deploying, so you decide [15:16:33] (03PS1) 10Giuseppe Lavagetto: kube2proxy: fix systemd declaration [puppet] - 10https://gerrit.wikimedia.org/r/242578 [15:17:38] (03CR) 10Giuseppe Lavagetto: [C: 032] kube2proxy: fix systemd declaration [puppet] - 10https://gerrit.wikimedia.org/r/242578 (owner: 10Giuseppe Lavagetto) [15:18:23] <_joe_> andrewbogott: I merged your change too [15:18:43] RECOVERY - Unmerged changes on repository puppet on palladium is OK: No changes to merge. [15:18:54] RECOVERY - Unmerged changes on repository puppet on strontium is OK: No changes to merge. [15:19:11] Thanks! [15:20:01] Krenair: concerned about the IE9 one or the special:movepage one? [15:20:22] it was the movepage one [15:20:30] (03PS1) 10Giuseppe Lavagetto: kube2proxy: fix typo [puppet] - 10https://gerrit.wikimedia.org/r/242580 [15:21:03] 6operations, 6Analytics-Backlog, 10Analytics-EventLogging, 10MediaWiki-extensions-CentralNotice, 10Traffic: Eventlogging should transparently split large event payloads - https://phabricator.wikimedia.org/T114078#1689560 (10BBlack) Logging events and stats without having significant complexity or perf is... [15:21:07] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] kube2proxy: fix typo [puppet] - 10https://gerrit.wikimedia.org/r/242580 (owner: 10Giuseppe Lavagetto) [15:21:12] * greg-g shrugs [15:21:20] whatever, it's fine, imo [15:22:07] (03PS1) 10RobH: adding spage to analytics-privatedata-users [puppet] - 10https://gerrit.wikimedia.org/r/242581 [15:22:57] 10Ops-Access-Requests, 6operations: add spage to analytics-privatedata-users group for hive access - https://phabricator.wikimedia.org/T114150#1689565 (10RobH) a:3RobH I've gone ahead and created patchset https://gerrit.wikimedia.org/r/#/c/242581/ If there are no objections raised, I'll merge this live afte... [15:23:37] <_joe_> oh, I suck... [15:23:42] (03PS1) 10Giuseppe Lavagetto: kube2proxy: fix hiera variable name [puppet] - 10https://gerrit.wikimedia.org/r/242582 [15:25:46] (03CR) 10Giuseppe Lavagetto: [C: 032] kube2proxy: fix hiera variable name [puppet] - 10https://gerrit.wikimedia.org/r/242582 (owner: 10Giuseppe Lavagetto) [15:27:05] !log krenair@tin Started scap: swat [15:27:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [15:27:16] MatmaRex, ^ [15:27:56] thanks Krenair. will verify when it's done. doing both at once? [15:28:06] all 4 commits at once [15:35:42] snapshot1003 ignore whines from iciinga, I'll put ack in 1 minute [15:36:41] (still scapping?) [15:36:58] MatmaRex, yeah, it takes like 20-30 minutes IIRC [15:37:26] sorry about that, icinga notified now [15:50:35] 6operations, 10Analytics, 5Patch-For-Review: Moving analysis data from flourine to analytics cluster - https://phabricator.wikimedia.org/T112744#1689684 (10Addshore) 5Open>3Resolved a:3Addshore [15:57:25] MatmaRex, FYI it's about 50% of the way through sync-common [15:57:37] sync-common: 50% (ok: 237; fail: 0; left: 228) [15:58:08] hmph [15:58:16] guess i really should have taken the i18n chages out [15:59:44] (03PS1) 10Giuseppe Lavagetto: kube2proxy: change service name [puppet] - 10https://gerrit.wikimedia.org/r/242593 [16:00:32] oh dear [16:00:42] one of the hosts failed [16:00:45] oh, it's snapshot1003 [16:01:20] not fully set up yet, prompts for password [16:01:29] and yet, still in mediawiki-installation... sigh [16:02:08] <_joe_> Krenair: it's being reimaged, I guess [16:02:21] <_joe_> so that's why it's not removed [16:02:54] (03CR) 10Giuseppe Lavagetto: [C: 032] kube2proxy: change service name [puppet] - 10https://gerrit.wikimedia.org/r/242593 (owner: 10Giuseppe Lavagetto) [16:04:49] _joe_, is it normal to reimage hosts that are still in the installation group? [16:04:55] <_joe_> yes [16:05:09] <_joe_> but I am not reimaging it now [16:09:13] (03PS4) 10Faidon Liambotis: Introduce otrs-test.wikimedia.org [dns] - 10https://gerrit.wikimedia.org/r/242512 (owner: 10Alexandros Kosiaris) [16:09:18] (03CR) 10Faidon Liambotis: [C: 032] Introduce otrs-test.wikimedia.org [dns] - 10https://gerrit.wikimedia.org/r/242512 (owner: 10Alexandros Kosiaris) [16:10:09] jenkins broken for DNS again? [16:10:33] (03CR) 10Faidon Liambotis: [V: 032] Introduce otrs-test.wikimedia.org [dns] - 10https://gerrit.wikimedia.org/r/242512 (owner: 10Alexandros Kosiaris) [16:10:36] hasharMeeting: ^ [16:11:17] (03CR) 10Alex Monk: "Despite it being for labs only, these changes still need to be merged in production, you can't just +2 and leave it like in some developme" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/242362 (https://phabricator.wikimedia.org/T113770) (owner: 10Jdlrobson) [16:11:37] (03PS2) 10Faidon Liambotis: misc-web: Get otrs-test.wikimedia.org behind it [puppet] - 10https://gerrit.wikimedia.org/r/242516 (owner: 10Alexandros Kosiaris) [16:12:11] (03CR) 10Faidon Liambotis: [C: 032] misc-web: Get otrs-test.wikimedia.org behind it [puppet] - 10https://gerrit.wikimedia.org/r/242516 (owner: 10Alexandros Kosiaris) [16:12:56] (03CR) 10Alex Monk: [C: 032] "There should probably be a beta wikivoyage but I don't have time to set that up right now." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/242362 (https://phabricator.wikimedia.org/T113770) (owner: 10Jdlrobson) [16:13:25] (03Merged) 10jenkins-bot: Add Extension:RelatedArticles to beta labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/242362 (https://phabricator.wikimedia.org/T113770) (owner: 10Jdlrobson) [16:14:17] Krenair: yeh a beta wikivoyage would be nice :) [16:14:24] Krenair: thanks for the merge [16:15:20] I'm going to merge it on tin once I'm done with this scap. [16:15:26] I think ori is planning to deploy something afterwards. [16:15:54] jdlrobson, also I'm pretty sure adding a new beta project requires labs ops intervention or something silly like that [16:16:36] (03PS1) 10Giuseppe Lavagetto: kube2proxy: remove explicit dependency on python-requests [puppet] - 10https://gerrit.wikimedia.org/r/242596 [16:16:49] Because of the way DNS works :( [16:16:55] (I mean, DNS within labs) [16:17:33] (03PS2) 10Giuseppe Lavagetto: kube2proxy: remove explicit dependency on python-requests [puppet] - 10https://gerrit.wikimedia.org/r/242596 [16:17:47] (03CR) 10Giuseppe Lavagetto: [C: 032 V: 032] kube2proxy: remove explicit dependency on python-requests [puppet] - 10https://gerrit.wikimedia.org/r/242596 (owner: 10Giuseppe Lavagetto) [16:18:29] if anyone's looking for me what the scap finishes, then i'm not here right now. [16:19:16] MatmaRex, did you at least test the changes? [16:19:29] snapshot1003 is doing its initial scap right now [16:19:34] (from the reinstall) [16:19:35] there are some seriously slow rsync common times in the scap logs right now. Lots of 15m+ [16:20:06] I wonder if we have somehow overloaded one of the rsync slaves? [16:20:38] !log krenair@tin Finished scap: swat (duration: 53m 33s) [16:20:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [16:20:43] MatmaRex, ^ [16:21:31] bd808, snapshot1003 failed in sync-apaches and then again in scap-rebuild-cdbs (it's in the middle of reimaging), but I'm not aware of other issues [16:22:48] Krenair: i'm back, looking [16:22:49] *nod* seeing times >3m for a single host always makes me jumpy. we may have just be the victim of a bad shuffle that put too many clients on a few slaves concurrently [16:23:00] Krenair: i did test them on master [16:24:39] Krenair: all's in order. [16:24:48] great [16:27:03] Krenair: how far along is it now? [16:27:14] ah thanks :) [16:27:20] no !log? [16:27:24] It logged earlier [16:27:27] !log krenair@tin Finished scap: swat (duration: 53m 33s) [16:27:32] nod [16:27:39] I don't think I need to sync the -labs file [16:27:44] I just merged it on tin [16:27:48] thanks [16:28:41] !log ori@tin Synchronized php-1.26wmf24/includes/resourceloader: 89595ba49a: Cherry-pick I173a9820b and I7c7546ec (duration: 00m 18s) [16:28:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [16:34:34] 6operations, 10ops-ulsfo: Move NTT @ ulsfo to a different cross-connect - https://phabricator.wikimedia.org/T112154#1689868 (10RobH) Mark's approved our new order for the migration, and now I'm simply awaiting sync up with Kevin @ NTT before I submit the work order and get it migrated. Once I chat with Kevin... [16:38:48] 10Ops-Access-Requests, 6operations: add spage to analytics-privatedata-users group for hive access - https://phabricator.wikimedia.org/T114150#1689874 (10RobH) p:5Triage>3Normal [16:40:56] gilles: thanks for taking notes :) [16:50:52] np [16:53:28] (03PS3) 10Alexandros Kosiaris: Add the new OTRS scheduler watchdog cron entry [puppet] - 10https://gerrit.wikimedia.org/r/242184 [16:53:30] (03PS1) 10Alexandros Kosiaris: otrs: Have otrs-test.wikimedia.org served as well [puppet] - 10https://gerrit.wikimedia.org/r/242601 [16:55:31] (03PS4) 10Alexandros Kosiaris: Add the new OTRS scheduler watchdog cron entry [puppet] - 10https://gerrit.wikimedia.org/r/242184 [16:55:33] (03PS2) 10Alexandros Kosiaris: otrs: Have otrs-test.wikimedia.org served as well [puppet] - 10https://gerrit.wikimedia.org/r/242601 [16:57:44] (03CR) 10Alexandros Kosiaris: [C: 032] otrs: Have otrs-test.wikimedia.org served as well [puppet] - 10https://gerrit.wikimedia.org/r/242601 (owner: 10Alexandros Kosiaris) [16:59:15] (03PS1) 10ArielGlenn: wikidata dumps job: use group www-data which exists, apache doesn't [puppet] - 10https://gerrit.wikimedia.org/r/242604 [17:03:29] (03CR) 10Hoo man: [C: 031] "This will work." [puppet] - 10https://gerrit.wikimedia.org/r/242604 (owner: 10ArielGlenn) [17:03:36] (03PS1) 10Dzahn: Revert "mailman: Korean encoding fixes" [puppet] - 10https://gerrit.wikimedia.org/r/242605 [17:03:42] (03CR) 10jenkins-bot: [V: 04-1] Revert "mailman: Korean encoding fixes" [puppet] - 10https://gerrit.wikimedia.org/r/242605 (owner: 10Dzahn) [17:05:11] (03PS2) 10ArielGlenn: wikidata dumps job: use group www-data which exists, apache doesn't [puppet] - 10https://gerrit.wikimedia.org/r/242604 [17:05:41] 6operations, 10ops-eqiad, 10hardware-requests: disk degausser for eqiad - https://phabricator.wikimedia.org/T112780#1690006 (10RobH) [17:05:53] (03PS2) 10Dzahn: Revert "mailman: Korean encoding fixes" [puppet] - 10https://gerrit.wikimedia.org/r/242605 (https://phabricator.wikimedia.org/T72180) [17:05:57] (03CR) 10jenkins-bot: [V: 04-1] Revert "mailman: Korean encoding fixes" [puppet] - 10https://gerrit.wikimedia.org/r/242605 (https://phabricator.wikimedia.org/T72180) (owner: 10Dzahn) [17:06:01] (03CR) 10ArielGlenn: [C: 032] wikidata dumps job: use group www-data which exists, apache doesn't [puppet] - 10https://gerrit.wikimedia.org/r/242604 (owner: 10ArielGlenn) [17:11:29] 6operations, 6Analytics-Backlog, 10Analytics-EventLogging, 10MediaWiki-extensions-CentralNotice, 10Traffic: Eventlogging should transparently split large event payloads - https://phabricator.wikimedia.org/T114078#1690049 (10Nuria) There are many good points on @BBlack reply. We could give a little more... [17:13:23] 6operations: clean up admins module data file - https://phabricator.wikimedia.org/T109516#1690058 (10RobH) 5Open>3declined [17:14:05] 6operations: clean up admins module data file - https://phabricator.wikimedia.org/T109516#1550833 (10RobH) I disagree since they are not in any order now. Folks don't append them reliably to the very end of the array, merely where they fit. That being said, I don't really think either answer is proper, its jus... [17:16:50] 6operations, 7HHVM, 7Tracking: Complete the use of HHVM over Zend PHP on the Wikimedia cluster (tracking) - https://phabricator.wikimedia.org/T86081#1690071 (10ArielGlenn) [17:16:51] 6operations, 7Tracking: Upgrade Wikimedia servers to Ubuntu Trusty (14.04) (tracking) - https://phabricator.wikimedia.org/T65899#1690072 (10ArielGlenn) [17:16:53] 6operations, 10Datasets-General-or-Unknown, 7HHVM, 5Patch-For-Review: Convert snapshot hosts to use HHVM and trusty - https://phabricator.wikimedia.org/T94277#1690069 (10ArielGlenn) 5Open>3stalled all snapshot hosts now converted to use trusty. Joe found two differences in php5 from precise to trusty,... [17:17:09] 6operations, 7Tracking: Upgrade Wikimedia servers to Ubuntu Trusty (14.04) (tracking) - https://phabricator.wikimedia.org/T65899#705851 (10ArielGlenn) [17:17:11] 6operations, 10Datasets-General-or-Unknown, 7HHVM, 5Patch-For-Review: Convert snapshot hosts to use HHVM and trusty - https://phabricator.wikimedia.org/T94277#1690074 (10ArielGlenn) [17:21:23] 6operations, 7Tracking: Upgrade Wikimedia servers to Ubuntu Trusty (14.04) (tracking) - https://phabricator.wikimedia.org/T65899#1690093 (10Krenair) [17:21:25] 6operations, 10Datasets-General-or-Unknown, 7HHVM, 5Patch-For-Review: Convert snapshot hosts to use HHVM and trusty - https://phabricator.wikimedia.org/T94277#1690094 (10Krenair) [17:22:31] (03PS3) 10Dzahn: Revert "mailman: Korean encoding fixes" [puppet] - 10https://gerrit.wikimedia.org/r/242605 (https://phabricator.wikimedia.org/T72180) [17:26:11] (03CR) 10Chad: [C: 031] Add config deployment [tools/scap] - 10https://gerrit.wikimedia.org/r/240292 (https://phabricator.wikimedia.org/T109512) (owner: 10Thcipriani) [17:30:32] (03Abandoned) 10BryanDavis: Force use of IPv4 addresses with ssh and rsync [tools/scap] - 10https://gerrit.wikimedia.org/r/234687 (owner: 10BryanDavis) [17:31:45] (03PS4) 10Dzahn: Revert "mailman: Korean encoding fixes" [puppet] - 10https://gerrit.wikimedia.org/r/242605 (https://phabricator.wikimedia.org/T72180) [17:32:32] 6operations, 10Dumps-Generation: sql dump schemata - seven tables should have their columns reordered - https://phabricator.wikimedia.org/T103583#1690129 (10Umherirrender) You cannot assume the order of columns in a relational database to be always the same as on other instance. The concept of a relational dat... [17:33:03] (03PS1) 10Ottomata: Provision 12 eventlogging client side processors [puppet] - 10https://gerrit.wikimedia.org/r/242613 [17:34:48] (03CR) 10Ottomata: [C: 032] Provision 12 eventlogging client side processors [puppet] - 10https://gerrit.wikimedia.org/r/242613 (owner: 10Ottomata) [17:35:24] (03CR) 10Dzahn: [C: 032] "this was ISO-8859 encoded. going back to UTF-8. the bug was reopened after recent mailman upgrade." [puppet] - 10https://gerrit.wikimedia.org/r/242605 (https://phabricator.wikimedia.org/T72180) (owner: 10Dzahn) [17:35:44] (03PS5) 10Dzahn: Revert "mailman: Korean encoding fixes" [puppet] - 10https://gerrit.wikimedia.org/r/242605 (https://phabricator.wikimedia.org/T72180) [17:36:15] and on goes the rebasing .. [17:39:02] JohnFLewis: ^ ... and that looks ok (Korean characters in shell) but breaks puppet :p [17:39:13] invalid byte sequence in UTF-8 [17:39:22] heh [17:39:29] what a great combo, mailman and puppet both have their own issues with encoding [17:39:40] and open upstream bugs, meh [17:39:43] PROBLEM - Check status of defined EventLogging jobs on eventlog1001 is CRITICAL: CRITICAL: Stopped EventLogging jobs: processor/client-side-11 processor/client-side-10 processor/client-side-09 processor/client-side-08 processor/client-side-07 processor/client-side-06 processor/client-side-05 processor/client-side-04 processor/client-side-03 processor/client-side-02 processor/client-side-01 [17:39:45] !log restarting eventlogging with 12 client side processors [17:39:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [17:40:00] JohnFLewis: but the diff looked so promising: [17:40:05] so fast icinga! [17:40:11] -

������ ������ that looks all Korean to me, puppet output and also in IRC [17:41:23] RECOVERY - Check status of defined EventLogging jobs on eventlog1001 is OK: OK: All defined EventLogging jobs are runnning. [17:41:55] PROBLEM - puppet last run on fermium is CRITICAL: CRITICAL: Puppet has 1 failures [17:44:30] 6operations, 6Analytics-Backlog, 10Analytics-EventLogging, 10MediaWiki-extensions-CentralNotice, 10Traffic: Eventlogging should transparently split large event payloads - https://phabricator.wikimedia.org/T114078#1690191 (10mforns) In the particular case of the CentralNoticeBannerHistory schema, I see th... [17:45:52] JohnFLewis: ah, apparently "iconv from utf8 to utf8" actually makes sense and should remove invalid chars [17:46:14] -c skips any invalid sequence [17:47:58] except that doing that is not a change ..hrmmm [17:51:53] RECOVERY - puppet last run on fermium is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [17:53:14] (03PS1) 10Faidon Liambotis: Mark wiki-mail IPv6 addresses as deprecated [puppet] - 10https://gerrit.wikimedia.org/r/242623 [17:53:20] bblack: ^ more ipv6 horror :) [17:54:23] nice [17:55:09] JohnFLewis: partial fix/ less broken than before? https://lists.wikimedia.org/mailman/listinfo/otrs-ko [17:55:15] bblack: what do you think? [17:55:37] mutante: yep. now it is just the description of the list [17:55:38] (03CR) 10BBlack: [C: 031] Mark wiki-mail IPv6 addresses as deprecated [puppet] - 10https://gerrit.wikimedia.org/r/242623 (owner: 10Faidon Liambotis) [17:55:49] paravoid: I think for this kind of use case, it makes sense [17:55:51] JohnFLewis: the fun part is i could "Trick" puppet into accepting this :p [17:56:15] JohnFLewis: i changed it manually with iconv on the server, then let puppet run again, it changed it to what is in repo and the error is gone :p [17:56:33] heh :) [17:56:40] it's going to be yet another complication in how we solve the whole general problem with autoconfig and interface::ip and add_v6_mapped and whatnot, but we can deal [17:58:26] really at this point I wish the kernel and ip tools had some simple parameter for both v6 and v4 like "sas-priority 123", which defaults to zero, and for sockets that don't specify a source-address, lowest sas-priority wins as the tie-breaker after the RFC-standard algorithms, or something [17:58:40] (and the ability to set sas-priority as "lowest" for anything autoconfig'd) [17:59:03] we shouldn't have to be implicitly doing that through lifetime/dynamic flags, etc [17:59:28] (03PS1) 10ArielGlenn: dumps: provide sha1 checksums of all files along with the old md5s [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/242626 [18:00:03] (03CR) 10BryanDavis: [C: 04-1] "Needs a manual rebase and then somebody to merge and deploy I think." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/189148 (https://phabricator.wikimedia.org/T85947) (owner: 10Legoktm) [18:00:04] twentyafterfour: Respected human, time to deploy MediaWiki train (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20150930T1800). Please do the needful. [18:00:11] (03CR) 10Faidon Liambotis: [C: 032] Mark wiki-mail IPv6 addresses as deprecated [puppet] - 10https://gerrit.wikimedia.org/r/242623 (owner: 10Faidon Liambotis) [18:04:17] 10Ops-Access-Requests, 6operations: RESTBase Admin access on aqs1001, aqs1002, and aqs1003 for Joseph and Dan - https://phabricator.wikimedia.org/T113416#1690239 (10mobrovac) +1 for T113416#1688129 . To clarify, @Milimetric and @JAllemandou will need to be able to: - `sudo systemctl * restbase` - `sudo system... [18:11:50] (03PS1) 1020after4: group1 wikis to 1.27.0-wmf.1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/242629 [18:12:07] (03CR) 1020after4: [C: 032] group1 wikis to 1.27.0-wmf.1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/242629 (owner: 1020after4) [18:12:13] (03Merged) 10jenkins-bot: group1 wikis to 1.27.0-wmf.1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/242629 (owner: 1020after4) [18:14:22] !log twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.1 [18:15:33] PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet). [18:16:15] (03PS1) 10Ottomata: Blacklist Analytics schema from being produced to the eventlogging-valid-mixed topic [puppet] - 10https://gerrit.wikimedia.org/r/242630 [18:16:53] (03PS2) 10Ottomata: Blacklist Analytics schema from being produced to the eventlogging-valid-mixed topic [puppet] - 10https://gerrit.wikimedia.org/r/242630 [18:17:07] what happened? [18:17:48] (03PS3) 10Ottomata: Blacklist Analytics schema from being produced to the eventlogging-valid-mixed topic [puppet] - 10https://gerrit.wikimedia.org/r/242630 [18:18:43] (03CR) 10Ottomata: [C: 032 V: 032] Blacklist Analytics schema from being produced to the eventlogging-valid-mixed topic [puppet] - 10https://gerrit.wikimedia.org/r/242630 (owner: 10Ottomata) [18:24:23] 6operations, 10Datasets-General-or-Unknown, 7HHVM, 5Patch-For-Review: Convert snapshot hosts to use HHVM and trusty - https://phabricator.wikimedia.org/T94277#1690393 (10ArielGlenn) T65899 is not a blocked task; all snapshots are converted to trusty. that's why I removed this from blockers. [18:26:53] (03CR) 10Dduvall: Add config deployment (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/240292 (https://phabricator.wikimedia.org/T109512) (owner: 10Thcipriani) [18:30:58] (03PS1) 10Faidon Liambotis: Brown paper bag fix for interface::ip $options [puppet] - 10https://gerrit.wikimedia.org/r/242633 [18:31:07] bblack: ^ [18:32:50] (03CR) 10BBlack: [C: 04-1] "With no default for the "options" argument, it's going to puppetfail on every host that doesn't set an options argument, right? Maybe def" [puppet] - 10https://gerrit.wikimedia.org/r/242633 (owner: 10Faidon Liambotis) [18:33:07] ... [18:33:10] I suck, clearly [18:33:56] (03PS2) 10Faidon Liambotis: Brown paper bag fix for interface::ip $options [puppet] - 10https://gerrit.wikimedia.org/r/242633 [18:33:57] done [18:34:11] bblack, eta for puppet disablement reenablement? [18:34:18] ironically the old default of '' would have worked, since that gets casted to "false" in puppet (3) [18:34:26] (03PS6) 10Thcipriani: Add config deployment [tools/scap] - 10https://gerrit.wikimedia.org/r/240292 (https://phabricator.wikimedia.org/T109512) [18:34:39] (03CR) 10BBlack: [C: 031] Brown paper bag fix for interface::ip $options [puppet] - 10https://gerrit.wikimedia.org/r/242633 (owner: 10Faidon Liambotis) [18:34:55] (03CR) 10Faidon Liambotis: [C: 032] Brown paper bag fix for interface::ip $options [puppet] - 10https://gerrit.wikimedia.org/r/242633 (owner: 10Faidon Liambotis) [18:35:06] ottomata: after the brown paper bag gets merged we can I think [18:35:09] and tested [18:35:10] k [18:35:11] danke [18:35:24] RECOVERY - Unmerged changes on repository puppet on strontium is OK: No changes to merge. [18:35:35] twentyafterfour: i am getting js errors on some pages, not just wikidata but also wikisource [18:35:38] Exception in module-execute in module mediawiki.page.watch.ajax: [18:36:36] aude: odd, that makes two things... [18:36:46] I am going to roll back the deployment [18:36:52] paravoid: I have a command prepper to selectively re-enable them without touching existing disables [18:36:58] whenever [18:37:06] also, my irc bouncer is lagging out :P [18:37:29] (03PS1) 1020after4: group1 wikis to 1.26wmf24 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/242635 [18:37:36] bblack: just tried it on a few hosts, seems to work as intended [18:37:38] https://phabricator.wikimedia.org/T114288 [18:37:39] so, yes, hit it [18:37:42] ok [18:37:43] and I'll clean up the broken ones [18:37:46] i bet it is a missing resourceloader dependency [18:37:51] it works with debug=true [18:38:12] ok [18:38:18] agent re-enable is salting around now [18:38:29] thanks man :) [18:39:22] (03CR) 1020after4: [C: 032] "rolling back the train deploy because of:" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/242635 (owner: 1020after4) [18:39:27] (03Merged) 10jenkins-bot: group1 wikis to 1.26wmf24 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/242635 (owner: 1020after4) [18:39:56] probably one of those things we should stick somewhere on wikitech or something: if you have to globally-disable puppet agents, use a unique reason, and then the way to undo it. [18:40:03] !log twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.26wmf24 [18:40:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [18:40:11] basically I did a salt on all-targets with "puppet agent --disable bblack" [18:40:22] * aude tries to reproduce and debug on test wikis [18:40:28] and then to undo I salted them all with: 'grep -q bblack /var/lib/puppet/state/agent_disabled.lock && puppet agent --enable' [18:40:41] heh smart [18:40:59] that leaves the ones that were already disabled for other reasons still-disabled [18:41:26] we could really use some salt-wrappers [18:41:50] even a "run-cp upload /bin/true" would be interesting [18:41:57] or run-cp --cluster=upload --site=eqiad or something [18:42:01] yeah [18:42:02] aude: wanna file a phabricator ticket for that error or should I file it? [18:42:23] https://phabricator.wikimedia.org/T114288 [18:42:39] 11 hosts affected by that change [18:43:00] not bad [18:43:06] I wonder if it's related to https://phabricator.wikimedia.org/T114283 [18:45:32] fixed [18:46:24] 6operations, 10RESTBase, 6Services: Switch RESTBase to use Node.js 4 - https://phabricator.wikimedia.org/T107762#1690485 (10Ricordisamoa) [18:47:20] * aude looks [18:47:45] twentyafterfour: i can't see how it's related but who knows [18:48:26] i can reproduce on test.wikidata [18:48:30] https://test.wikidata.org/wiki/Q22 [18:48:41] though had no problems with that item earlier today [18:49:01] 6operations, 10RESTBase, 6Services: Switch RESTBase to use Node.js 4 - https://phabricator.wikimedia.org/T107762#1690492 (10Ricordisamoa) >>! In T107762#1687339, @GWicke wrote: > Now that [node 4.1 is available in Debian unstable](https://packages.debian.org/sid/nodejs), and [the next node LTS release is goi... [18:49:44] 6operations, 10RESTBase, 6Services: Switch RESTBase to use Node.js 4 - https://phabricator.wikimedia.org/T107762#1690493 (10mobrovac) >>! In T107762#1689441, @GWicke wrote: > This is the secondary Cassandra node. As you know, only deployment-restbase01 is used for requests. In any case, deployment-restbase02... [18:50:50] !log restarted eventlogging with blacklist=^Analytics$ [18:50:54] 6operations, 10RESTBase, 6Services: Switch RESTBase to use Node.js 4 - https://phabricator.wikimedia.org/T107762#1690499 (10mobrovac) Also, let's not forget that the Analytics team will soon have their own RESTBase cluster, and I think we should keep these two on the same nodejs/iojs version to preserve our... [18:50:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [18:51:34] !log stopping replication on s2, s3 and s7 for dbstore1001 [18:51:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [18:52:30] (03PS1) 10Dzahn: mailman: fix German listinfo template [puppet] - 10https://gerrit.wikimedia.org/r/242639 (https://phabricator.wikimedia.org/T114289) [18:55:07] (03PS2) 10Dzahn: mailman: fix German listinfo template [puppet] - 10https://gerrit.wikimedia.org/r/242639 (https://phabricator.wikimedia.org/T114289) [18:55:33] 6operations, 10Wikimedia-Mailing-lists: mailman's public list index (listinfo) has the wrong encoding in its Content-Type header - https://phabricator.wikimedia.org/T42971#1690521 (10JohnLewis) [18:56:48] (03PS1) 10Mattflaschen: Freeze LQT on Swedish Wikimedia chapter wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/242640 (https://phabricator.wikimedia.org/T114277) [18:57:05] PROBLEM - puppet last run on db2067 is CRITICAL: CRITICAL: puppet fail [19:00:36] (03PS3) 10Dzahn: mailman: fix German listinfo template [puppet] - 10https://gerrit.wikimedia.org/r/242639 (https://phabricator.wikimedia.org/T114289) [19:01:32] (03PS4) 10Dzahn: mailman: fix German listinfo template [puppet] - 10https://gerrit.wikimedia.org/r/242639 (https://phabricator.wikimedia.org/T114289) [19:10:06] (03PS1) 10Rush: phab: phabtools.conf add slave option [puppet] - 10https://gerrit.wikimedia.org/r/242645 [19:12:45] (03CR) 10Andrew Bogott: [C: 031] "I'll run a compiler test for good measure..." [puppet] - 10https://gerrit.wikimedia.org/r/242055 (owner: 10Dzahn) [19:13:48] twentyafterfour: aude: Found the cause for the watch-undefined bug. OK to deploy? [19:14:26] andrewbogott: :) thanks! i was not sure how small or large to make it. it was tempting to fix them all at once, then it seemed to much at once. so i merged a small part already and then this and one smaller one [19:14:47] not too many altogether to fix though given the size of the repo [19:15:16] Krinkle: sure [19:15:45] I rolled back the wikis but go ahead and deploy the fix, once we get this other thing fixed I'll re-deploy the train [19:16:47] twentyafterfour: OK. testwikis remains on 1.27 yes [19:16:50] ? [19:17:05] yes [19:17:34] RECOVERY - puppet last run on db2067 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:17:54] Krinkle: great [19:18:09] !log krinkle@tin Synchronized php-1.27.0-wmf.1/resources/Resources.php: T114288 (duration: 00m 17s) [19:18:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [19:19:28] twentyafterfour: i would also like to ideally fix https://phabricator.wikimedia.org/T114290 if it's easy [19:19:59] aude: OK. Fix confirmed on https://test.wikidata.org/wiki/Q22 [19:20:12] aude: I now see a different error though, "Existing entitytermsforlanguagelistview DOM does not match configured languages" [19:20:20] Is that specific to test wikidata? [19:20:43] i am not sure [19:20:47] i see that locally [19:22:19] I'm gonna microwave lunch real quick, I'll check back on the status of these tasks and if we're looking good in 30 minutes I'll re-deploy [19:24:28] ok [19:41:05] (03PS2) 10Rush: phab: apachetop for debugging [puppet] - 10https://gerrit.wikimedia.org/r/242538 [19:43:44] (03CR) 10Rush: [C: 032] phab: apachetop for debugging [puppet] - 10https://gerrit.wikimedia.org/r/242538 (owner: 10Rush) [19:44:33] (03PS2) 10Rush: phab: phabtools.conf add slave option [puppet] - 10https://gerrit.wikimedia.org/r/242645 [19:46:04] (03CR) 10Rush: [C: 032] phab: phabtools.conf add slave option [puppet] - 10https://gerrit.wikimedia.org/r/242645 (owner: 10Rush) [19:47:34] PROBLEM - etherpad.wikimedia.org HTTP on etherpad1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:48:55] 6operations, 6Phabricator, 7Database, 5Patch-For-Review, 7WorkType-Maintenance: Phabricator creates MySQL connection spikes: Attempt to connect to phuser@m3-master.eqiad.wmnet failed with error #1040: Too many connections. - https://phabricator.wikimedia.org/T109279#1690736 (10chasemp) [19:49:13] RECOVERY - etherpad.wikimedia.org HTTP on etherpad1001 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 522 bytes in 0.012 second response time [19:52:31] (03PS5) 10Dzahn: mailman: fix German listinfo template [puppet] - 10https://gerrit.wikimedia.org/r/242639 (https://phabricator.wikimedia.org/T114289) [19:52:50] (03PS6) 10Dzahn: mailman: fix German listinfo template [puppet] - 10https://gerrit.wikimedia.org/r/242639 (https://phabricator.wikimedia.org/T114289) [19:53:32] (03CR) 10Dzahn: [C: 032] mailman: fix German listinfo template [puppet] - 10https://gerrit.wikimedia.org/r/242639 (https://phabricator.wikimedia.org/T114289) (owner: 10Dzahn) [19:54:47] 10Ops-Access-Requests, 6operations: Jkatz can't sign into Grafana with LDAP password - https://phabricator.wikimedia.org/T114300#1690757 (10JKatzWMF) 3NEW [19:55:01] 10Ops-Access-Requests, 6operations: Jkatz can't sign into Grafana with LDAP password - https://phabricator.wikimedia.org/T114300#1690765 (10JKatzWMF) [19:59:24] PROBLEM - puppet last run on fermium is CRITICAL: CRITICAL: Puppet has 1 failures [19:59:28] 10Ops-Access-Requests, 6operations: Jkatz can't sign into Grafana with LDAP password - https://phabricator.wikimedia.org/T114300#1690778 (10Dzahn) I think this is due to rOPUP0257652f1f969c76a2f17e60a7589d9d51938c78 and that means membership in the "wmf" LDAP group is needed and the root cause is missing onboa... [19:59:47] 10Ops-Access-Requests, 6operations: Jkatz can't sign into Grafana with LDAP password - https://phabricator.wikimedia.org/T114300#1690780 (10Krenair) It looks like you're in ldap/wmf. Are you using your UID (labs shell name) or CN (wikitech username)? [19:59:53] 6operations, 10CirrusSearch, 6Discovery: Only use newer (elastic10{16..31}) servers as master capable elasticsearch nodes - https://phabricator.wikimedia.org/T112556#1690783 (10chasemp) [19:59:54] 6operations, 10ops-eqiad, 5Patch-For-Review: Swap two elasticsearch servers in row D with an elasticsearch server in racks A3 and C5. - https://phabricator.wikimedia.org/T112559#1690781 (10chasemp) 5Open>3Resolved this is done [20:00:04] gwicke cscott arlolra subbu bearND mdholloway: Dear anthropoid, the time has come. Please deploy Services – Parsoid / OCG / Citoid / Mobileapps / … (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20150930T2000). [20:01:09] mutante, did you just add him to ldap/wmf? [20:01:37] 10Ops-Access-Requests, 6operations: Jkatz can't sign into Grafana with LDAP password - https://phabricator.wikimedia.org/T114300#1690789 (10Dzahn) I take that back. I checked on terbium and: member: uid=jkatz,ou=people,dc=wikimedia,dc=org is already in "wmf". so it must be something different [20:02:05] mutante, when I use ldaplist he is at the bottom though. [20:02:20] Almost all other users are sorted alphabetically by UID [20:02:25] Krenair: no, i just checked if he is in it like you did [20:02:28] and noticed the same [20:02:40] 4 others are at the bottom as well [20:02:44] (03PS1) 10Rush: phab: turn dump cron back on as it hits slave only [puppet] - 10https://gerrit.wikimedia.org/r/242663 [20:03:04] I think newly added people are at the bottom... I don't know how the rest have been sorted [20:03:24] 6operations, 10ops-eqiad: db1050 raid degraded - https://phabricator.wikimedia.org/T103110#1690811 (10Cmjohnson) 5Open>3Resolved a:3Cmjohnson RAID has been restored cmjohnson@db1050:~$ sudo megacli -PDList -aALL |grep "Firmware state:" Firmware state: Online, Spun Up Firmware state: Online, Spun Up Firm... [20:04:29] Krenair: confirmed that alpha/sorting thing. it's odd. i wonder why. but i know that logins like icinga work for pt1979 [20:04:49] Icinga is weird [20:04:55] We login via LDAP credentials [20:05:07] But then there's some separate thing to do to get the ability to change stuff there [20:05:43] yes, the separate thing is icinga's own permissions and the LDAP login is just on top of all that [20:05:49] once icinga was open [20:05:55] but that didnt mean anyone could send commands [20:05:58] that would be dangerous [20:07:11] 6operations, 10RESTBase, 10RESTBase-Cassandra: column family cassandra metrics size - https://phabricator.wikimedia.org/T113733#1690824 (10Eevans) >>! In T113733#1689500, @fgiunchedi wrote: > good question! yeah I think a blacklist would be fine for now OK, this is implemented in https://github.com/wikimedi... [20:07:30] mutante, what does "send commands" mean exactly? [20:07:46] you get to tell icinga to hide some alerts? other stuff? [20:08:37] Krenair: schedule downtime, acknowledge alert, send custom notification.. [20:08:42] stuff like that [20:09:00] Krenair: re: grafana,actually.. what login ?:) [20:09:09] do you see one? [20:09:14] Er. I should [20:09:27] Oh, I know what it was [20:09:40] That was at one point put entirely behind auth [20:09:52] Then it was changed to only require auth to POST/DELETE/PUT/etc. [20:10:09] yes, then there was the ticket about "why graphite this way and grafana the other way" [20:10:27] so yes, what you said, he only talks about changing config though [20:10:31] so this means POST [20:12:48] (03PS1) 10Giuseppe Lavagetto: kube2proxy: various puppetization fixes [puppet] - 10https://gerrit.wikimedia.org/r/242712 [20:13:30] (03CR) 10Giuseppe Lavagetto: [C: 032] kube2proxy: various puppetization fixes [puppet] - 10https://gerrit.wikimedia.org/r/242712 (owner: 10Giuseppe Lavagetto) [20:13:58] Krenair: https://gerrit.wikimedia.org/r/#/c/237761/ [20:14:37] yeah, that was the commit I had in mind [20:16:35] 10Ops-Access-Requests, 6operations: Jkatz can't sign into Grafana with LDAP password - https://phabricator.wikimedia.org/T114300#1690853 (10Dzahn) Since rOPUPb0793141c2ff GET request are allowed without authentication (where before an LDAP group was needed), but POST requests require authentication and an LDAP... [20:17:10] Krenair: yea, so .. <% @auth_ldap['groups'].each do |group| -%> .. [20:17:25] so unless "wmf" is missing from that list .. [20:17:31] it should work [20:18:39] bd808, what is sync-common actually doing after "Finished rsync common (duration: 00m 06s)" ? [20:19:10] confirmed on server. Require ldap-group cn=wmf,ou=groups,dc=wikimedia,dc=org and he is in it [20:19:51] 18:40 logmsgbot: twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.26wmf24 [20:20:00] twentyafterfour: Ahm, that appears to have been a no-op? [20:20:02] Krenair: scap-rebuild-cdbs [20:20:11] The new version is wmf-1, group1 was already on wmf24, right? [20:20:19] Krenair: s://github.com/wikimedia/mediawiki-tools-scap/blob/master/scap/main.py#L309-L313 [20:22:20] 10Ops-Access-Requests, 6operations: Jkatz can't sign into Grafana with LDAP password - https://phabricator.wikimedia.org/T114300#1690873 (10Dzahn) I confirmed on `krypton.eqiad.wmnet`, that the `<% @auth_ldap['groups'].each do |group| -%>` from the puppet template turns into: Require ldap-group cn... [20:23:37] RoanKattouw: I believe that correct. Seems like the group1 deploy has been delayed? [20:23:52] RoanKattouw: we are fixing some bugs [20:24:25] hopefully won't take much longer [20:25:02] RoanKattouw: rolled back [20:25:09] Oh OK [20:25:55] AaronSchulz, "CAS update failed on user_touched for user ID '87' (read from slave); the version of the user to be saved is older than the current version." [20:26:03] aude, twentyafterfour : could one of you ping me when it's done. I need to test a change on Commons once it gets wmf1. [20:26:13] kaldari: ok [20:26:15] thanks [20:27:03] I don't think that should be happening from https://phabricator.wikimedia.org/diffusion/EVED/browse/master/autodisablePref.php;HEAD$50-52 [20:27:18] ok [20:28:12] Krenair: newFromId() could use READ_LATEST there, since it's a maintenance script anyway [20:29:08] AaronSchulz, newFromId doesn't take flags [20:29:49] 6operations: Puppet Compiler: Support wildcards, regexps, or 'all hosts' - https://phabricator.wikimedia.org/T114305#1690937 (10Andrew) 3NEW a:3Joe [20:30:08] yeah but can just call load() after making it [20:30:21] it doesn't actually touch the DB when build the User object at first [20:31:40] $user = User::newFromId( $userRow->user_id ); [20:31:40] + $user->load( User::READ_LATEST ); [20:31:40] $user->setOption( 'visualeditor-autodisable', true ); [20:32:02] like that? [20:32:15] 10Ops-Access-Requests, 6operations: Jkatz can't sign into Grafana with LDAP password - https://phabricator.wikimedia.org/T114300#1690980 (10Dzahn) @jkatzwmf quoting Krenair's question from above. "Are you using your UID (labs shell name) or CN (wikitech username)?" Can you confirm you are using "jkatz" ? [20:33:09] yeah [20:34:20] * AaronSchulz also notices a bug in loadFromId() [20:40:50] !log fixing puppet run on fermium, needs manual fix because puppet cant replace existing illegal character in some templates [20:40:53] RECOVERY - puppet last run on fermium is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [20:40:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [20:41:33] 10Ops-Access-Requests, 6operations: Jkatz can't sign into Grafana with LDAP password - https://phabricator.wikimedia.org/T114300#1691011 (10JKatzWMF) @Dzahn yes, I am using Jkatz and jkatz (no luck with either) [20:43:06] !log updated Parsoid to version 39c60c67 [20:43:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [20:44:00] AaronSchulz, so I still get the error [20:44:11] but with "read from master" [20:45:05] Krenair: even with 242718 ? [20:45:25] no, I'll try [20:45:33] But I'm not really qualified to merge that, so... [20:46:35] Krenair: without that you might be stuck on the problem https://gerrit.wikimedia.org/r/#/c/241898/ fixed [20:47:11] Krenair: though that might have only affected master, I'd have to check the timing [20:47:33] PROBLEM - puppet last run on wtp2020 is CRITICAL: CRITICAL: Puppet has 1 failures [20:48:03] legoktm: does https://gerrit.wikimedia.org/r/#/c/242718/ look sane? [20:48:16] AaronSchulz, I can't really tell whether this fixed the problem or not because it occurred randomly [20:49:26] AaronSchulz: does that still handle the locking case? [20:49:42] locking implies latest [20:49:54] the locking flags also has the latest bit set [20:50:10] READ_LOCKING = 3 [21:13:19] (03CR) 10Ottomata: [WIP] Consume EventLogging validation logs from Logstash (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/241984 (https://phabricator.wikimedia.org/T113627) (owner: 10Mforns) [21:14:04] RECOVERY - puppet last run on wtp2020 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [21:14:14] 7Blocked-on-Operations, 6operations, 6Phabricator, 6Release-Engineering-Team, and 2 others: Phabricator needs to expose ssh - https://phabricator.wikimedia.org/T100519#1691100 (10greg) 5stalled>3Open [21:20:43] mutante: I'm just following up on https://phabricator.wikimedia.org/T113416#1690239 [21:21:23] would like to know if I can help with anything and drive it forward, right now that's the last bridge on a long road to the Pageview API [21:22:32] milimetric: ok.i'll make a patch to add the group itself to move forward [21:23:12] milimetric: about the "log access" part, we allowed to run "journalctl" in other groups to achieve that [21:23:14] mutante: should it be part of this patch: https://gerrit.wikimedia.org/r/#/c/231574/ [21:23:38] (the rest of that sets up the aqs itself) [21:24:12] milimetric: i think that one is large enough and would actually prefer to have a separate one [21:24:34] k, cool, just wanted to know if we could merge [21:24:39] i can add the group so that it exists and can be referred to [21:24:52] and then we can still change what the group is allowed to do exactly [21:25:46] then your patch can use it in the "admin::groups" part [21:26:22] cool [21:26:24] where it says analytics now. that doesn't exist under the name [21:26:47] just analytics-admins and others [21:27:24] right, I'll wait for your patch then and update this [21:27:31] !log ori@tin Synchronized php-1.26wmf24/includes/resourceloader/ResourceLoaderFileModule.php: a1e1619461: Fix LESS file dependency tracking in ResourceLoader (duration: 00m 17s) [21:27:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [21:28:04] (03CR) 10Milimetric: [C: 04-1] "On hold for now, we're waiting on a patch from mutante to add a new aqs admins group, which I'll reference here as soon as it's ready" [puppet] - 10https://gerrit.wikimedia.org/r/231574 (https://phabricator.wikimedia.org/T107056) (owner: 10Milimetric) [21:33:12] milimetric: what is the service of the new service going to be? [21:33:27] milimetric: like on the shell, "service foo start" [21:34:43] looks for a puppet service "ensure running" or so in the upcoming role [21:35:16] or is this simply 'restbase'? [21:35:59] oh, cassandra and restbase you said on the ticket, nevermind [21:36:19] for a minute i expected an "aqs" service [21:37:05] Krenair: around? [21:37:11] paravoid, hi, yes [21:37:13] hi :) [21:37:30] akosiaris and I were looking your Znuny4OTRS-WikimediaDTL change [21:37:38] Okay [21:37:38] how exactly did you do that? :) [21:37:44] I had forgotten about that [21:37:46] hm [21:38:19] So this was https://gerrit.wikimedia.org/r/#/c/165472/ [21:38:22] Is it ok to deploy the train now? [21:38:51] correct [21:39:22] paravoid, I may have just decoded the file contents, made my changes, converted it all to base64 and replaced the entry in the file [21:39:57] heh ok [21:40:08] I was asking if you did it that way or the right way :) [21:40:10] I was testing on the labs otrs project [21:40:19] apparently OTRS has "opm" packages and "sopm" source packages [21:40:31] and I can't find the sopms for our packages anywhere [21:40:33] hence my asking [21:40:56] Given that the build date didn't change, that's probably what I did [21:41:08] (03PS6) 10Mforns: [WIP] Consume EventLogging validation logs from Logstash [puppet] - 10https://gerrit.wikimedia.org/r/241984 (https://phabricator.wikimedia.org/T113627) [21:41:08] yeah, I figured as well [21:41:27] that btw, breaks the verification process of OTRS [21:41:40] oops [21:41:52] (03CR) 10Mforns: [C: 04-1] "Still WIP" [puppet] - 10https://gerrit.wikimedia.org/r/241984 (https://phabricator.wikimedia.org/T113627) (owner: 10Mforns) [21:41:54] as in the "my oh my, you have a package that is not really ours, we will not support it" [21:42:03] otherwise everything is ok [21:42:18] Sorry about that [21:42:52] (03CR) 10Mforns: [WIP] Consume EventLogging validation logs from Logstash (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/241984 (https://phabricator.wikimedia.org/T113627) (owner: 10Mforns) [21:43:33] twentyafterfour: if we want to put wikibase / wikidata back on our previous branch, i'd be ok with that [21:44:37] i figured out where/why for our bug but not 100% sure the best fix and not sure anyone to review it [21:48:01] (03PS1) 10Dzahn: admin: add admin group for analytics query service [puppet] - 10https://gerrit.wikimedia.org/r/242735 (https://phabricator.wikimedia.org/T113416) [21:49:14] 6operations, 6Phabricator, 6Release-Engineering-Team: Enable mod_remoteip and ensure logs follow retention guidelines - https://phabricator.wikimedia.org/T114014#1691207 (10chasemp) Phabricator stores client IP addresses in a few places: 1. user_log activity (https://phabricator.wikimedia.org/settings/pane... [21:50:28] aude: put it back to 1.26wmf24? [21:50:44] (03PS2) 10Dzahn: admin: add admin group for analytics query service [puppet] - 10https://gerrit.wikimedia.org/r/242735 (https://phabricator.wikimedia.org/T113416) [21:51:23] mutante: beat me to the -1 there, I wondered why I got added to it automatically :) [21:52:13] (03CR) 10Dzahn: "https://gerrit.wikimedia.org/r/#/c/242735" [puppet] - 10https://gerrit.wikimedia.org/r/231574 (https://phabricator.wikimedia.org/T107056) (owner: 10Milimetric) [21:52:34] JohnFLewis: the price of context switching :p [21:54:32] 10Ops-Access-Requests, 6operations, 5Patch-For-Review: RESTBase Admin access on aqs1001, aqs1002, and aqs1003 for Joseph and Dan - https://phabricator.wikimedia.org/T113416#1691221 (10Dzahn) @mobrovac how about this for now https://gerrit.wikimedia.org/r/#/c/242735/2/modules/admin/data/data.yaml "journalctl... [21:55:50] twentyafterfour: the issue also affects wikis that have wikibase client (e.g. wikisource) [21:56:25] 10Ops-Access-Requests, 6operations, 5Patch-For-Review: RESTBase Admin access on aqs1001, aqs1002, and aqs1003 for Joseph and Dan - https://phabricator.wikimedia.org/T113416#1691223 (10Dzahn) I can merge the above now to unblock your role change so that you can use the group name. Then this access request wi... [21:57:14] so, what i mean is use the previous "Wikidata" extension branch for the submodule [21:57:23] i think it's wmf/1.26wmf22 [21:57:44] or [21:58:05] (03CR) 10Mobrovac: admin: add admin group for analytics query service (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/242735 (https://phabricator.wikimedia.org/T113416) (owner: 10Dzahn) [21:58:41] i'm trying to make a patch now [22:02:07] Krenair: yeah, no worries. I am already following the same path as you [22:02:25] 10Ops-Access-Requests, 6operations, 5Patch-For-Review: RESTBase Admin access on aqs1001, aqs1002, and aqs1003 for Joseph and Dan - https://phabricator.wikimedia.org/T113416#1691227 (10mobrovac) >>! In T113416#1691221, @Dzahn wrote: > @mobrovac how about this for now https://gerrit.wikimedia.org/r/#/c/242735/... [22:02:36] akosiaris, what, patching it the same way I did? [22:03:18] yup [22:03:53] heh, ok [22:06:43] (03PS1) 10Alexandros Kosiaris: Add bumped packages for 3.3.x and 4.0.x [software/otrs] - 10https://gerrit.wikimedia.org/r/242740 [22:06:59] (03CR) 10Dzahn: [C: 032] "adds empty group only" [puppet] - 10https://gerrit.wikimedia.org/r/242735 (https://phabricator.wikimedia.org/T113416) (owner: 10Dzahn) [22:07:58] milimetric: mobrovac ^ there, just to unblock it, but not to break the acces request rules [22:08:07] but you can use it now [22:08:37] mutante: we waited 3 days and got approval and all that as far as the access request is concerned [22:08:47] I'll go ahead and use this but we can add me and joal on here [22:09:01] (03PS1) 10Yuvipanda: toollabs: Fix redis requiremnet for kube2proxy [puppet] - 10https://gerrit.wikimedia.org/r/242741 [22:09:24] ok, wikidata is broken [22:09:26] (03PS2) 10Yuvipanda: toollabs: Fix redis requiremnet for kube2proxy [puppet] - 10https://gerrit.wikimedia.org/r/242741 [22:09:31] (Cannot access the database: Can't connect to MySQL server on '10.64.48.25' (4) (10.64.48.25)) [22:09:34] 10Ops-Access-Requests, 6operations, 5Patch-For-Review: RESTBase Admin access on aqs1001, aqs1002, and aqs1003 for Joseph and Dan - https://phabricator.wikimedia.org/T113416#1691233 (10Dzahn) I added the empty group to unblock development of the puppet role. This access request should continue as normal, by... [22:09:53] akosiaris: mutante doing anything that could cause that? [22:10:25] 25.48.64.10.in-addr.arpa domain name pointer db1070.eqiad.wmnet. [22:10:37] (03CR) 10Yuvipanda: [C: 032] toollabs: Fix redis requiremnet for kube2proxy [puppet] - 10https://gerrit.wikimedia.org/r/242741 (owner: 10Yuvipanda) [22:10:52] s5 slave [22:10:52] now it seems ok [22:11:03] but i clicked special:random once and then again [22:11:13] hmmm [22:11:15] aude: not from my side, nope [22:11:18] transient [22:11:33] not anything I know of being done on s5 right now [22:12:13] ok [22:12:31] !log ori@tin Synchronized php-1.26wmf24/includes/User.php: 0ed7cc8526: Made User::loadFromId() skip cache with READ_LATEST (duration: 00m 17s) [22:12:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [22:12:59] milimetric: i believe it still needs to be approved in ops meeting on monday though and Rob would follow-up as the on duty guy [22:13:31] !log ori@tin Synchronized php-1.27.0-wmf.1/includes/User.php: 0ed7cc8526: Made User::loadFromId() skip cache with READ_LATEST (duration: 00m 17s) [22:13:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [22:13:36] Krenair: ^ [22:14:04] jouncebot: next [22:14:04] In 0 hour(s) and 45 minute(s): Evening SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20150930T2300) [22:15:07] twentyafterfour: i have a patch and trying to get it reviewed [22:15:39] if not, then we go back to the previous branch for the "Wikidata" extension and suppose can try again next week [22:15:40] thanks ori [22:17:55] (03PS4) 10Alex Monk: [WIP] Move from ircecho to tcpircbot [puppet] - 10https://gerrit.wikimedia.org/r/240945 [22:17:57] (03PS4) 10Alex Monk: tcpircbot: Allow per-infile channel lists [puppet] - 10https://gerrit.wikimedia.org/r/240939 [22:18:05] aude: ok .. link? I can try to review it if it's straightforward [22:18:42] it's not straightforward, hence why it took so long [22:18:53] https://gerrit.wikimedia.org/r/#/c/242739/ [22:19:47] 6operations, 7Graphite: scale graphite deployment (tracking) - https://phabricator.wikimedia.org/T85451#1691273 (10GWicke) @fgiunchedi, sorry if I'm naggy on this, but I'm still wondering what the plan is for scaling graphite storage, especially with codfw and new services adding even more metrics. [22:20:16] 10Ops-Access-Requests, 6operations, 5Patch-For-Review: RESTBase Admin access on aqs1001, aqs1002, and aqs1003 for Joseph and Dan - https://phabricator.wikimedia.org/T113416#1691275 (10mobrovac) a:5Milimetric>3RobH Thank you @DZahn! Assigning to @RobH for adding the needed users so the ticket makes the ne... [22:21:30] 10Ops-Access-Requests, 6operations, 5Patch-For-Review: RESTBase Admin access on aqs1001, aqs1002, and aqs1003 for Joseph and Dan - https://phabricator.wikimedia.org/T113416#1691278 (10RobH) Indeed, Daniel's work on the implementation means that all that is left is the ops meeting review for @JAllemandou (joa... [22:22:04] aude: looks good to me, is it testable? [22:22:24] 6operations, 10RESTBase, 6Services: Switch RESTBase to use Node.js 4 - https://phabricator.wikimedia.org/T107762#1691279 (10GWicke) >>! In T107762#1688386, @MoritzMuehlenhoff wrote: > I can take care of it at the beginning of next week. I'll contact Jeremy Lal on his plans, ideally Debian sid/testing with st... [22:22:27] I mean, I see you added unit tests, but we could just merge it and roll back again if it's still broken? [22:23:05] twentyafterfour: i have someone to review it, but not immediately [22:23:27] to not block you, i'd say let's use the previous branch of wikidata for now [22:23:50] aude: ok [22:24:12] idk if we would want to try again tomorrow with our branch or wait until next week would be ok [22:24:18] (03PS1) 10Yuvipanda: toollabs: Cleanup kube2proxy [puppet] - 10https://gerrit.wikimedia.org/r/242743 [22:24:34] (03PS13) 10Milimetric: Add Analytics Query Service role [puppet] - 10https://gerrit.wikimedia.org/r/231574 (https://phabricator.wikimedia.org/T107056) [22:24:52] (03PS2) 10Yuvipanda: toollabs: Cleanup kube2proxy [puppet] - 10https://gerrit.wikimedia.org/r/242743 [22:26:24] 10Ops-Access-Requests, 6operations, 5Patch-For-Review: RESTBase Admin access on aqs1001, aqs1002, and aqs1003 for Joseph and Dan - https://phabricator.wikimedia.org/T113416#1691290 (10Milimetric) Thanks much everyone, this is great. [22:27:09] (03CR) 10Yuvipanda: [C: 032] toollabs: Cleanup kube2proxy [puppet] - 10https://gerrit.wikimedia.org/r/242743 (owner: 10Yuvipanda) [22:34:54] (03PS1) 10Yuvipanda: tools: Mark kube2proxy as python3 [puppet] - 10https://gerrit.wikimedia.org/r/242744 [22:34:56] (03PS1) 10Yuvipanda: tools: Fix ensure checking in kube2proxy [puppet] - 10https://gerrit.wikimedia.org/r/242745 [22:35:54] (03CR) 10Yuvipanda: [C: 032] tools: Mark kube2proxy as python3 [puppet] - 10https://gerrit.wikimedia.org/r/242744 (owner: 10Yuvipanda) [22:36:08] (03CR) 10Yuvipanda: [C: 032] tools: Fix ensure checking in kube2proxy [puppet] - 10https://gerrit.wikimedia.org/r/242745 (owner: 10Yuvipanda) [22:38:09] (03PS1) 10Yuvipanda: tools: Fix missed rename [puppet] - 10https://gerrit.wikimedia.org/r/242746 [22:38:16] (03PS7) 10BryanDavis: [WIP] Consume EventLogging validation logs from Logstash [puppet] - 10https://gerrit.wikimedia.org/r/241984 (https://phabricator.wikimedia.org/T113627) (owner: 10Mforns) [22:38:33] aude: https://gerrit.wikimedia.org/r/#/c/242747/ [22:38:40] (03CR) 10Yuvipanda: [C: 032 V: 032] tools: Fix missed rename [puppet] - 10https://gerrit.wikimedia.org/r/242746 (owner: 10Yuvipanda) [22:40:18] twentyafterfour: looking [22:41:49] twentyafterfour: looks good [22:42:14] rather not rush to fix this and then cause some other problem [22:44:24] !log perf testing eventlogging in production by hammering https://bits.wikimedia.org/beacon/event.gif [22:44:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [22:46:08] (03PS1) 10Yuvipanda: tools: Python3 compatibility fixes for kube2proxy [puppet] - 10https://gerrit.wikimedia.org/r/242748 [22:46:16] (03PS1) 10Dzahn: mailman: convert language templates, ca,es,fi,fr [puppet] - 10https://gerrit.wikimedia.org/r/242749 [22:47:01] aude: thanks! [22:47:15] I'm gonna deploy it as soon as jenkins finishes doing it's thing [22:47:20] (03CR) 10Yuvipanda: [C: 032] tools: Python3 compatibility fixes for kube2proxy [puppet] - 10https://gerrit.wikimedia.org/r/242748 (owner: 10Yuvipanda) [22:48:20] wow @ tech news is always translated to "ksh" but not "de". [22:48:43] ksh is German dialect [22:49:38] (03PS8) 10BryanDavis: [WIP] Consume EventLogging validation logs from Logstash [puppet] - 10https://gerrit.wikimedia.org/r/241984 (https://phabricator.wikimedia.org/T113627) (owner: 10Mforns) [22:50:33] that's like you have the newsletter only in Jamaican English but not en en.. for lack of a better comparison [22:51:43] where's that spoken ? [22:52:09] Jamaica? [22:53:39] Why can't the en version be written in Jamaican English? [22:54:36] actually no one speaks it in Jamaica [22:54:41] https://en.wikipedia.org/wiki/Jamaica#Language [22:54:45] so says wikipedia [22:55:07] mutante: like how we decided to not take anything from your comparison and instead focus on the comparison? ;] [22:55:48] !log twentyafterfour@tin Synchronized php-1.27.0-wmf.1/: deploying several fixes to the branch (duration: 01m 52s) [22:55:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [22:57:02] akosiaris: ksh? in Cologne and the Rhineland http://www.ethnologue.com/language/ksh [22:57:27] robh: :) [22:57:32] 10Ops-Access-Requests, 6operations: Jkatz can't sign into Grafana with LDAP password - https://phabricator.wikimedia.org/T114300#1691474 (10RobH) So just as a test, have you attempted to change your wikitech/ldap password and see if that perhaps fixes it? [22:58:50] 10Ops-Access-Requests, 6operations: Requesting access to stat1002 for VBaranetsky - https://phabricator.wikimedia.org/T114308#1691481 (10Revi) [22:59:58] (03PS1) 1020after4: group1 wikis to 1.27.0-wmf.1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/242752 [23:00:04] RoanKattouw ostriches rmoen Krenair: Dear anthropoid, the time has come. Please deploy Evening SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20150930T2300). [23:00:12] (03CR) 1020after4: [C: 032] group1 wikis to 1.27.0-wmf.1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/242752 (owner: 1020after4) [23:00:18] (03Merged) 10jenkins-bot: group1 wikis to 1.27.0-wmf.1 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/242752 (owner: 1020after4) [23:00:31] 6operations, 6Analytics-Backlog, 10Analytics-EventLogging, 10MediaWiki-extensions-CentralNotice, 10Traffic: Eventlogging should transparently split large event payloads - https://phabricator.wikimedia.org/T114078#1691516 (10Nuria) In the case of this schema: https://meta.wikimedia.org/wiki/Schema:Centra... [23:00:37] !log twentyafterfour@tin rebuilt wikiversions.cdb and synchronized wikiversions files: group1 wikis to 1.27.0-wmf.1 [23:01:39] ok swatters, I'm out of your way now [23:02:02] thanks twentyafterfour [23:02:23] twentyafterfour, well we have nothing to do, so... [23:02:47] 6operations, 6Analytics-Backlog, 10Analytics-EventLogging, 10MediaWiki-extensions-CentralNotice, 10Traffic: Eventlogging should transparently split large event payloads - https://phabricator.wikimedia.org/T114078#1691538 (10Nuria) You can keep track of client side length errors on dashboard: https://graf... [23:03:33] (03PS9) 10BryanDavis: [WIP] Consume EventLogging validation logs from Logstash [puppet] - 10https://gerrit.wikimedia.org/r/241984 (https://phabricator.wikimedia.org/T113627) (owner: 10Mforns) [23:09:23] (03PS1) 10Faidon Liambotis: Kill "git-setup" [software/otrs] - 10https://gerrit.wikimedia.org/r/242756 [23:09:25] (03PS1) 10Faidon Liambotis: Kill patches/2.4.x and patches/3.2.14 [software/otrs] - 10https://gerrit.wikimedia.org/r/242757 [23:09:27] (03PS1) 10Faidon Liambotis: Kill Znuny4OTRS-QuickClose and Znuny4OTRS-Repo OPM [software/otrs] - 10https://gerrit.wikimedia.org/r/242758 [23:09:29] (03PS1) 10Faidon Liambotis: Reverse-package our two OPMs into Source OPMs [software/otrs] - 10https://gerrit.wikimedia.org/r/242759 [23:09:31] (03PS1) 10Faidon Liambotis: Remove var/, unused for a long time [software/otrs] - 10https://gerrit.wikimedia.org/r/242760 [23:10:19] (03CR) 10Alexandros Kosiaris: [C: 032] Kill "git-setup" [software/otrs] - 10https://gerrit.wikimedia.org/r/242756 (owner: 10Faidon Liambotis) [23:10:24] (03CR) 10Alexandros Kosiaris: [V: 032] Kill "git-setup" [software/otrs] - 10https://gerrit.wikimedia.org/r/242756 (owner: 10Faidon Liambotis) [23:11:07] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] Kill patches/2.4.x and patches/3.2.14 [software/otrs] - 10https://gerrit.wikimedia.org/r/242757 (owner: 10Faidon Liambotis) [23:11:25] wait [23:11:29] the reverse-package needs a fix [23:11:51] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] Kill Znuny4OTRS-QuickClose and Znuny4OTRS-Repo OPM [software/otrs] - 10https://gerrit.wikimedia.org/r/242758 (owner: 10Faidon Liambotis) [23:12:17] (03PS2) 10Faidon Liambotis: Reverse-package our two OPMs into Source OPMs [software/otrs] - 10https://gerrit.wikimedia.org/r/242759 [23:12:19] (03PS2) 10Faidon Liambotis: Remove var/, unused for a long time [software/otrs] - 10https://gerrit.wikimedia.org/r/242760 [23:12:27] done [23:13:39] (03PS1) 10Yuvipanda: tools: Use StrictRedis for kube2proxy [puppet] - 10https://gerrit.wikimedia.org/r/242761 [23:16:10] 6operations, 10RESTBase, 10RESTBase-Cassandra, 5Patch-For-Review: Test multiple Cassandra instances per hardware node - https://phabricator.wikimedia.org/T95253#1691567 (10GWicke) As an aside, http://www.scylladb.com/ are working on a C++ Cassandra clone with a one-instance-per-core architecture. Part of t... [23:16:16] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] Reverse-package our two OPMs into Source OPMs [software/otrs] - 10https://gerrit.wikimedia.org/r/242759 (owner: 10Faidon Liambotis) [23:16:40] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] Remove var/, unused for a long time [software/otrs] - 10https://gerrit.wikimedia.org/r/242760 (owner: 10Faidon Liambotis) [23:18:42] 10Ops-Access-Requests, 6operations: Jkatz can't sign into Grafana with LDAP password - https://phabricator.wikimedia.org/T114300#1691589 (10JKatzWMF) @RobH yes. thanks [23:21:31] (03PS10) 10BryanDavis: [WIP] Consume EventLogging validation logs from Logstash [puppet] - 10https://gerrit.wikimedia.org/r/241984 (https://phabricator.wikimedia.org/T113627) (owner: 10Mforns) [23:23:16] (03PS1) 10Alexandros Kosiaris: Update WikimediaEnableMultiLines to support OTRS 4.0.x [software/otrs] - 10https://gerrit.wikimedia.org/r/242762 [23:25:22] (03Abandoned) 10Alexandros Kosiaris: a bit of reorg for clarity [software/otrs] - 10https://gerrit.wikimedia.org/r/228292 (owner: 10Jgreen) [23:25:29] (03CR) 10BryanDavis: "Cherry-picked to beta cluster for testing. See https://logstash-beta.wmflabs.org/#/dashboard/elasticsearch/eventlogging for results." [puppet] - 10https://gerrit.wikimedia.org/r/241984 (https://phabricator.wikimedia.org/T113627) (owner: 10Mforns) [23:26:12] (03Abandoned) 10Alexandros Kosiaris: Add bumped packages for 3.3.x and 4.0.x [software/otrs] - 10https://gerrit.wikimedia.org/r/242740 (owner: 10Alexandros Kosiaris) [23:29:20] (03PS2) 10Faidon Liambotis: Port WikimediaEnableMultiLines to OTRS 4.0.x [software/otrs] - 10https://gerrit.wikimedia.org/r/242762 (owner: 10Alexandros Kosiaris) [23:29:22] (03PS1) 10Faidon Liambotis: Znuny4OTRS-WikimediaDTL: convert CRLFs to LFs [software/otrs] - 10https://gerrit.wikimedia.org/r/242763 [23:29:24] (03PS1) 10Faidon Liambotis: Rename Znuny4OTRS-WikimediaDTL to WikimediaTemplates [software/otrs] - 10https://gerrit.wikimedia.org/r/242764 [23:29:26] (03PS1) 10Faidon Liambotis: Port WikimediaTemplates to OTRS 4.0.x [software/otrs] - 10https://gerrit.wikimedia.org/r/242765 [23:29:37] (03PS3) 10Alexandros Kosiaris: Update WikimediaEnableMultiLines to support OTRS 4.0.x [software/otrs] - 10https://gerrit.wikimedia.org/r/242762 [23:29:43] nooo [23:29:49] ahahaha [23:30:26] I prefer mine [23:30:30] I'll repush :P [23:30:39] sure [23:30:40] hm, don't think I can at this point! [23:31:19] oh wait, mine was broken too [23:31:30] break all the things, etc [23:32:20] (03CR) 10Alexandros Kosiaris: [C: 04-1] Znuny4OTRS-WikimediaDTL: convert CRLFs to LFs (031 comment) [software/otrs] - 10https://gerrit.wikimedia.org/r/242763 (owner: 10Faidon Liambotis) [23:32:22] (03PS2) 10Faidon Liambotis: Port WikimediaTemplates to OTRS 4.0.x [software/otrs] - 10https://gerrit.wikimedia.org/r/242765 [23:32:24] (03PS4) 10Faidon Liambotis: Port WikimediaEnableMultiLines to OTRS 4.0.x [software/otrs] - 10https://gerrit.wikimedia.org/r/242762 (owner: 10Alexandros Kosiaris) [23:32:26] (03CR) 10BryanDavis: [C: 031] [WIP] Consume EventLogging validation logs from Logstash [puppet] - 10https://gerrit.wikimedia.org/r/241984 (https://phabricator.wikimedia.org/T113627) (owner: 10Mforns) [23:32:49] akosiaris: I actually just did "flip -u" [23:32:54] akosiaris: that's how it is in the original source [23:33:21] (03CR) 10Alexandros Kosiaris: [C: 031] Rename Znuny4OTRS-WikimediaDTL to WikimediaTemplates [software/otrs] - 10https://gerrit.wikimedia.org/r/242764 (owner: 10Faidon Liambotis) [23:34:45] (03CR) 10Alexandros Kosiaris: [C: 031] Port WikimediaEnableMultiLines to OTRS 4.0.x (031 comment) [software/otrs] - 10https://gerrit.wikimedia.org/r/242762 (owner: 10Alexandros Kosiaris) [23:35:34] (03CR) 10Alexandros Kosiaris: [C: 031] Port WikimediaTemplates to OTRS 4.0.x (031 comment) [software/otrs] - 10https://gerrit.wikimedia.org/r/242765 (owner: 10Faidon Liambotis) [23:36:10] paravoid: yeah, not surprised. but we pretty much take ownership, let's just fix it [23:36:29] I 'll abandon my change [23:36:34] no [23:36:37] no wait [23:36:47] which change? [23:37:02] I cherry-picked yours and fixed it [23:37:09] fixed it as in made it more consistent etc. [23:37:17] I didn't actually create a new one [23:37:24] oh, yes [23:37:25] lol [23:37:28] I just noticed [23:37:50] (03CR) 10Faidon Liambotis: [C: 032] Znuny4OTRS-WikimediaDTL: convert CRLFs to LFs [software/otrs] - 10https://gerrit.wikimedia.org/r/242763 (owner: 10Faidon Liambotis) [23:37:56] (03CR) 10Faidon Liambotis: [V: 032] Znuny4OTRS-WikimediaDTL: convert CRLFs to LFs [software/otrs] - 10https://gerrit.wikimedia.org/r/242763 (owner: 10Faidon Liambotis) [23:38:09] (03CR) 10Faidon Liambotis: [C: 032 V: 032] Rename Znuny4OTRS-WikimediaDTL to WikimediaTemplates [software/otrs] - 10https://gerrit.wikimedia.org/r/242764 (owner: 10Faidon Liambotis) [23:38:18] (03CR) 10Faidon Liambotis: [C: 032 V: 032] Port WikimediaEnableMultiLines to OTRS 4.0.x [software/otrs] - 10https://gerrit.wikimedia.org/r/242762 (owner: 10Alexandros Kosiaris) [23:38:27] (03CR) 10Faidon Liambotis: [C: 032 V: 032] Port WikimediaTemplates to OTRS 4.0.x [software/otrs] - 10https://gerrit.wikimedia.org/r/242765 (owner: 10Faidon Liambotis) [23:38:33] 10Ops-Access-Requests, 6operations: Requesting access to stat1002 for Dan Foy - https://phabricator.wikimedia.org/T113324#1691661 (10DFoy) @RobH - my manager is not on Phabricator. I had my manager (Sheree Chang) send approval via email to coren earlier this week, but I will copy it below. Let me or Sheree... [23:38:37] Oh, I 'll eat that whitespace, it will not escape me [23:38:44] there is no whitespace anymore [23:38:49] that template doesn't exist anymore [23:38:54] it was a .dtl, now it's a .tt [23:39:02] 10Ops-Access-Requests, 6operations: Requesting access to stat1002 for Dan Foy - https://phabricator.wikimedia.org/T113324#1691666 (10DFoy) a:5DFoy>3RobH [23:39:21] and now we understand why I need to not be working so late [23:39:28] 2 oversights in 3 minutes [23:39:35] which is why I didn't fix it, btw :) [23:39:44] yup, it makes sense [23:39:57] Writing /tmp/WikimediaTemplates-1.0.4.opm [23:39:59] Writing /tmp/WikimediaEnableMultiLines-1.0.2.opm [23:41:57] (03CR) 10Yuvipanda: [C: 032 V: 032] tools: Use StrictRedis for kube2proxy [puppet] - 10https://gerrit.wikimedia.org/r/242761 (owner: 10Yuvipanda) [23:45:10] 10Ops-Access-Requests, 6operations: Requesting access to stat1002 for Dan Foy - https://phabricator.wikimedia.org/T113324#1691696 (10RobH) @DFoy, You pasting your managers approval won't quite cut it, no offense intended. Most mangers approving shell access would then go and create a phabricator account as a... [23:45:38] 10Ops-Access-Requests, 6operations: Requesting access to stat1002 for Dan Foy - https://phabricator.wikimedia.org/T113324#1691704 (10RobH) I asked @Coren about the email, but he is likely gone for the day since its late in his timezone =] [23:50:54] (03PS1) 10Faidon Liambotis: Move WikimediaTemplates' files under Custom/ [software/otrs] - 10https://gerrit.wikimedia.org/r/242768 [23:51:11] (03CR) 10Faidon Liambotis: [C: 032 V: 032] Move WikimediaTemplates' files under Custom/ [software/otrs] - 10https://gerrit.wikimedia.org/r/242768 (owner: 10Faidon Liambotis) [23:52:17] 10Ops-Access-Requests, 6operations: Requesting access to stat1002 for Dan Foy - https://phabricator.wikimedia.org/T113324#1691725 (10DFoy) + Sheree Hi Sheree, we need to repeat the approval process for the analytics access. Can you review and respond to Rob's request below? Thanks, Dan [23:55:06] 10Ops-Access-Requests, 6operations: Requesting access to stat1002 for Dan Foy - https://phabricator.wikimedia.org/T113324#1691727 (10RobH) Also note the patchset is ready for rollout, so as soon as I am able to chat with Sheree we'll get this sorted! (Sorry its taken so long!) [23:55:43] 10Ops-Access-Requests, 6operations: Jkatz can't sign into Grafana with LDAP password - https://phabricator.wikimedia.org/T114300#1691728 (10Dzahn) @ori could this be a bug in rOPUPb0793141c2ff87116 ? [23:56:13] (03PS11) 10Mforns: Consume EventLogging validation logs from Logstash [puppet] - 10https://gerrit.wikimedia.org/r/241984 (https://phabricator.wikimedia.org/T113627) [23:57:16] (03CR) 10Mforns: [C: 031] "Removed the [WIP] flag. LGTM!" [puppet] - 10https://gerrit.wikimedia.org/r/241984 (https://phabricator.wikimedia.org/T113627) (owner: 10Mforns)