[00:00:08] cool [00:00:37] I didn't see anything at all in journalctl... which probably means my command is wrong [00:00:38] what are logstash100[456]? [00:00:50] those are the elasticsearch instances [00:00:58] ah [00:01:21] ES runs on 123 as well but jsut as a pass-through to the backing cluster [00:01:48] nod, I just did a 'service logstash status' on logstash* and was surprised to see it wasn't running on those hosts [00:02:12] *nod* [00:03:41] (03PS3) 10Ori.livneh: foreachwikiindblist: Fix sudo guard and cleanup script [puppet] - 10https://gerrit.wikimedia.org/r/290863 (https://phabricator.wikimedia.org/T136258) (owner: 10BryanDavis) [00:10:21] (03CR) 10Ori.livneh: [C: 032 V: 032] foreachwikiindblist: Fix sudo guard and cleanup script [puppet] - 10https://gerrit.wikimedia.org/r/290863 (https://phabricator.wikimedia.org/T136258) (owner: 10BryanDavis) [00:17:32] RECOVERY - Router interfaces on cr2-codfw is OK: OK: host 208.80.153.193, interfaces up: 122, down: 0, dormant: 0, excluded: 0, unused: 0 [00:27:12] RECOVERY - puppet last run on mw2081 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:28:03] PROBLEM - Unmerged changes on repository puppet on palladium is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet, ref HEAD..origin/production). [00:28:13] PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet, ref HEAD..origin/production). [00:30:15] RECOVERY - Ensure legal html en.m.wp on en.m.wikipedia.org is OK: all html is present. [00:40:15] 06Operations, 13Patch-For-Review: Audit uses of package=>latest - https://phabricator.wikimedia.org/T115348#2335866 (10Dzahn) Things that still use package=>latest today: | package | location | | percona-toolkit | role::mariadb::maintenance | | python3-* | modules/ircyall | | php-apc, php5-cli | modules/media... [00:50:24] (03PS1) 10Dzahn: wikistats: remove orain and pardus updates [puppet] - 10https://gerrit.wikimedia.org/r/291387 (https://phabricator.wikimedia.org/T136460) [00:58:05] (03CR) 10Dzahn: [C: 032] wikistats: remove orain and pardus updates [puppet] - 10https://gerrit.wikimedia.org/r/291387 (https://phabricator.wikimedia.org/T136460) (owner: 10Dzahn) [01:00:05] ori: i merged yours too [01:00:12] thanks [01:01:23] RECOVERY - Unmerged changes on repository puppet on palladium is OK: No changes to merge. [01:01:33] RECOVERY - Unmerged changes on repository puppet on strontium is OK: No changes to merge. [01:30:30] (03PS1) 10Yuvipanda: Revert "Revert "base: Provide better error messages for service_unit"" [puppet] - 10https://gerrit.wikimedia.org/r/291479 [01:30:53] (03PS2) 10Yuvipanda: Revert "Revert "base: Provide better error messages for service_unit"" [puppet] - 10https://gerrit.wikimedia.org/r/291479 [01:31:13] (03CR) 10Yuvipanda: [C: 032 V: 032] Revert "Revert "base: Provide better error messages for service_unit"" [puppet] - 10https://gerrit.wikimedia.org/r/291479 (owner: 10Yuvipanda) [01:32:39] I think I need to restart puppetmaster for that [01:39:10] (I'm watching ircecho to make sure it isn't missing anything important) [02:20:23] !log mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.3) (duration: 08m 20s) [02:20:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:26:10] !log l10nupdate@tin ResourceLoader cache refresh completed at Sat May 28 02:26:10 UTC 2016 (duration 5m 47s) [02:26:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:27:03] (03PS1) 10Dzahn: remove pardus table and orain remnants [debs/wikistats] - 10https://gerrit.wikimedia.org/r/291481 (https://phabricator.wikimedia.org/T136460) [02:31:56] (03CR) 10Dzahn: [C: 04-2] remove pardus table and orain remnants [debs/wikistats] - 10https://gerrit.wikimedia.org/r/291481 (https://phabricator.wikimedia.org/T136460) (owner: 10Dzahn) [03:16:33] PROBLEM - puppet last run on elastic1034 is CRITICAL: CRITICAL: Puppet has 1 failures [03:42:22] RECOVERY - puppet last run on elastic1034 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [03:46:01] PROBLEM - puppet last run on mw2189 is CRITICAL: CRITICAL: Puppet has 1 failures [04:12:14] RECOVERY - puppet last run on mw2189 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [04:22:19] (03PS4) 10Ori.livneh: DBUtil.py: Fix PEP8 violations [puppet] - 10https://gerrit.wikimedia.org/r/291175 (owner: 10BryanDavis) [04:28:33] (03CR) 10Ori.livneh: [C: 04-1] DBUtil.py: Fix PEP8 violations (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/291175 (owner: 10BryanDavis) [05:25:15] (03PS5) 10BryanDavis: DBUtil.py: Fix PEP8 violations [puppet] - 10https://gerrit.wikimedia.org/r/291175 [05:26:25] (03CR) 10BryanDavis: DBUtil.py: Fix PEP8 violations (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/291175 (owner: 10BryanDavis) [05:32:31] (03PS1) 10BryanDavis: letsencrypt: Fix flake8 exclusion [puppet] - 10https://gerrit.wikimedia.org/r/291487 [05:52:29] (03CR) 10BryanDavis: "Gut feel is that templating Dockerfiles is probably worth the trouble. I'd keep the template dead simple to start with (like just using st" (033 comments) [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/290793 (owner: 10Yuvipanda) [06:01:31] (03CR) 10Mobrovac: [C: 04-1] "Yurik also needs to be added to the deploy-service group in admin/data/admin.yaml" [puppet] - 10https://gerrit.wikimedia.org/r/291268 (https://phabricator.wikimedia.org/T129146) (owner: 10Thcipriani) [06:31:47] PROBLEM - puppet last run on mw2073 is CRITICAL: CRITICAL: Puppet has 2 failures [06:32:06] PROBLEM - puppet last run on mw2129 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:35] PROBLEM - puppet last run on wtp1005 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:55] PROBLEM - puppet last run on mw2207 is CRITICAL: CRITICAL: Puppet has 1 failures [06:33:57] PROBLEM - puppet last run on mw2095 is CRITICAL: CRITICAL: Puppet has 1 failures [06:46:26] RECOVERY - cassandra-c CQL 10.64.0.232:9042 on restbase1007 is OK: TCP OK - 0.024 second response time on port 9042 [06:57:16] RECOVERY - puppet last run on mw2207 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [06:57:35] RECOVERY - puppet last run on mw2073 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [06:58:05] RECOVERY - puppet last run on mw2129 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [06:58:37] RECOVERY - puppet last run on mw2095 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [06:58:45] RECOVERY - puppet last run on wtp1005 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [07:27:19] (03CR) 10Yuvipanda: "Yes, I agree on only string.format 'templating'" [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/290793 (owner: 10Yuvipanda) [07:34:30] (03PS1) 10Yuvipanda: tools: Allow deployment & configmap resources in k8s [puppet] - 10https://gerrit.wikimedia.org/r/291490 [08:15:45] (03PS1) 10Alexandros Kosiaris: rsync::module: Replace obsolete to_a calls [puppet] - 10https://gerrit.wikimedia.org/r/291491 [08:18:15] PROBLEM - Verify internal DNS from within Tools on checker.tools.wmflabs.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:19:53] PROBLEM - puppet last run on labvirt1010 is CRITICAL: CRITICAL: Puppet has 1 failures [08:20:03] RECOVERY - Verify internal DNS from within Tools on checker.tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 166 bytes in 0.774 second response time [08:27:44] PROBLEM - HHVM rendering on mw1107 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 50404 bytes in 0.353 second response time [08:29:34] RECOVERY - HHVM rendering on mw1107 is OK: HTTP OK: HTTP/1.1 200 OK - 64903 bytes in 0.120 second response time [08:32:03] PROBLEM - Text HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [1000.0] [08:32:54] PROBLEM - Ulsfo HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [1000.0] [08:44:24] RECOVERY - Ulsfo HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] [08:46:54] RECOVERY - puppet last run on labvirt1010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:47:33] RECOVERY - Text HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] [08:49:22] (03PS23) 10Alexandros Kosiaris: network: add $production_networks [puppet] - 10https://gerrit.wikimedia.org/r/260926 (https://phabricator.wikimedia.org/T122396) (owner: 10Faidon Liambotis) [09:12:50] (03CR) 10jenkins-bot: [V: 04-1] network: add $production_networks [puppet] - 10https://gerrit.wikimedia.org/r/260926 (https://phabricator.wikimedia.org/T122396) (owner: 10Faidon Liambotis) [09:37:35] 06Operations, 06Performance-Team, 07Availability: Apache <=> mariadb SSL/TLS for cross-datacenter writes - https://phabricator.wikimedia.org/T134809#2336110 (10jcrespo) I've been playing around with some open source SQL proxies lately. These support persistent connections. I wonder how much of the overhead o... [11:41:50] PROBLEM - puppet last run on mw2017 is CRITICAL: CRITICAL: Puppet has 1 failures [11:46:10] PROBLEM - puppet last run on cp3048 is CRITICAL: CRITICAL: Puppet last ran 2 days ago [11:48:01] RECOVERY - puppet last run on cp3048 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [12:02:25] 06Operations, 10Traffic, 13Patch-For-Review: Raise cache frontend memory sizes significantly - https://phabricator.wikimedia.org/T135384#2336369 (10BBlack) After ~1d at the new settings, seeing 140/73 for virt/rss on cp3048. Things are still moving in the right direction, and there doesn't seem to be any no... [12:07:52] RECOVERY - puppet last run on mw2017 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [13:42:15] (03PS1) 10Eevans: enable instance restbase1011-c.eqiad.wmnet [puppet] - 10https://gerrit.wikimedia.org/r/291504 (https://phabricator.wikimedia.org/T134016) [13:43:46] Is there anyone around with +2 on Puppet that would mind merging https://gerrit.wikimedia.org/r/#/c/291504/ for me? It will start a bootstrap of a RESTBase Cassandra instance, it's been lined up forever, totally safe/routine at this point. [13:44:13] they take awhile, so i'm just trying to keep the process running [13:46:34] (03CR) 10Ori.livneh: [C: 032] enable instance restbase1011-c.eqiad.wmnet [puppet] - 10https://gerrit.wikimedia.org/r/291504 (https://phabricator.wikimedia.org/T134016) (owner: 10Eevans) [13:47:05] urandom: ^ [13:47:53] ori: thank you! [13:59:15] !log Bootstrapping restbase1011-c.eqiad.wmnet : T134016 [13:59:17] T134016: RESTBase Cassandra cluster: Increase instance count to 3 - https://phabricator.wikimedia.org/T134016 [13:59:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [14:06:10] PROBLEM - cassandra-c CQL 10.64.0.119:9042 on restbase1011 is CRITICAL: Connection refused [14:07:34] ACKNOWLEDGEMENT - cassandra-c CQL 10.64.0.119:9042 on restbase1011 is CRITICAL: Connection refused eevans Node is bootstrapping - The acknowledgement expires at: 2016-05-29 14:07:20. [15:00:06] (03CR) 10Thcipriani: "yurik was already added to the deploy-service group in: I76428c93f0da1fb445acb48776fe1e848b159ffd" [puppet] - 10https://gerrit.wikimedia.org/r/291268 (https://phabricator.wikimedia.org/T129146) (owner: 10Thcipriani) [15:00:43] thcipriani|afk, ? [15:01:27] yurik: just posted a patch for moving tilerator to scap3 [15:01:43] thcipriani|afk, wasn't there a patch i made a while ago for that same thing? [15:01:56] or is it different? [15:02:02] * yurik looks in history [15:02:12] yurik: there was one for the tilerator repo [15:02:22] there wasn't one for puppet that I could find [15:02:44] ah, i see [15:02:45] (which, of course, doesn't mean it doesn't exist :)) [15:02:54] i was thinking about https://gerrit.wikimedia.org/r/#/c/285979/ [15:03:05] and the matching kartotherian [15:03:12] https://gerrit.wikimedia.org/r/#/c/285980/ [15:04:37] ah, yeah, I definitely saw that one. So the normal procedure for moving repos has been: merge the scap config in the repo, then find an opsen to merge the ops/puppet patch, run puppet on tin, do a deploy that will fail, run puppet on the targets, the run another deploy on tin that should succeed. [15:05:32] I meant to make a matching puppet patch for kartotherian as well. [15:58:46] PROBLEM - Host heka is DOWN: PING CRITICAL - Packet loss = 100% [15:58:53] PROBLEM - Host rigel is DOWN: PING CRITICAL - Packet loss = 100% [15:59:17] <_joe_> what's up? [15:59:42] PROBLEM - Host payments2003 is DOWN: PING CRITICAL - Packet loss = 100% [15:59:44] <_joe_> uh FR [15:59:47] <_joe_> srx again [15:59:50] PROBLEM - Host payments2001 is DOWN: PING CRITICAL - Packet loss = 100% [15:59:57] PROBLEM - Host saiph is DOWN: PING CRITICAL - Packet loss = 100% [16:00:05] PROBLEM - Host pay-lvs2001 is DOWN: PING CRITICAL - Packet loss = 100% [16:00:05] nice! [16:00:14] PROBLEM - Host fdb2001 is DOWN: PING CRITICAL - Packet loss = 100% [16:00:38] This is that same crashing-switch issue that FR always has, right? [16:01:12] <_joe_> yes [16:01:23] <_joe_> it's also not serving traffic AFAIK [16:02:23] PROBLEM - Router interfaces on pfw-codfw is CRITICAL: CRITICAL: host 208.80.153.195, interfaces up: 86, down: 2, dormant: 0, excluded: 0, unused: 0BRxe-6/0/0: down - cr1-codfw:xe-5/0/3 {#10900}BRge-2/0/2: down - pay-lvs2001BR [16:02:59] yup, is it rebooted manually in these cases? [16:03:14] Ok, it usually recovers ina minute on its own afaik [16:03:17] <_joe_> godog: I don't know the details [16:03:29] <_joe_> chasemp: not now I'd say :) [16:03:39] me neither heh [16:04:20] Is Jeff_Green around? I dont believe I have creds [16:04:23] RECOVERY - Router interfaces on pfw-codfw is OK: OK: host 208.80.153.195, interfaces up: 90, down: 0, dormant: 0, excluded: 0, unused: 0 [16:04:55] RECOVERY - Host heka is UP: PING OK - Packet loss = 0%, RTA = 37.21 ms [16:05:03] RECOVERY - Host rigel is UP: PING OK - Packet loss = 0%, RTA = 38.06 ms [16:05:23] RECOVERY - Host saiph is UP: PING OK - Packet loss = 0%, RTA = 36.94 ms [16:05:32] RECOVERY - Host pay-lvs2001 is UP: PING OK - Packet loss = 0%, RTA = 37.65 ms [16:05:40] RECOVERY - Host fdb2001 is UP: PING OK - Packet loss = 0%, RTA = 37.29 ms [16:05:48] RECOVERY - Host payments2001 is UP: PING OK - Packet loss = 0%, RTA = 36.98 ms [16:05:58] RECOVERY - Host payments2003 is UP: PING OK - Packet loss = 0%, RTA = 38.56 ms [16:06:35] Is there a task to drop a note it happened again? Otherwise not sure what can be done atm [16:15:08] PROBLEM - check_puppetrun on saiph is CRITICAL: CRITICAL: Puppet has 9 failures [16:15:17] PROBLEM - check_puppetrun on payments2003 is CRITICAL: CRITICAL: Puppet has 14 failures [16:20:08] RECOVERY - check_puppetrun on saiph is OK: OK: Puppet is currently enabled, last run 131 seconds ago with 0 failures [16:20:17] RECOVERY - check_puppetrun on payments2003 is OK: OK: Puppet is currently enabled, last run 165 seconds ago with 0 failures [17:21:10] PROBLEM - puppet last run on labvirt1010 is CRITICAL: CRITICAL: Puppet has 1 failures [17:46:49] RECOVERY - puppet last run on labvirt1010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:04:25] (03PS1) 10BryanDavis: Make the builder script less simple [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/291525 [18:08:56] (03CR) 10BryanDavis: "See Ifbf155de741dda25636989269bc66c332bc62f6e for a follow up that introduces the template idea and tries to address the other things I wh" [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/290793 (owner: 10Yuvipanda) [18:10:53] (03PS1) 10Ladsgroup: service: Let other methods of deployment work in uwsgi [puppet] - 10https://gerrit.wikimedia.org/r/291527 [18:16:38] 07Puppet, 10ORES, 06Revision-Scoring-As-A-Service: ORES-staging is broken due to service::uwsgi mandatory scap::target invoke - https://phabricator.wikimedia.org/T136488#2336803 (10Ladsgroup) [18:20:29] PROBLEM - puppet last run on labvirt1010 is CRITICAL: CRITICAL: Puppet has 2 failures [18:27:14] (03PS6) 10Ori.livneh: DBUtil.py: Fix PEP8 violations [puppet] - 10https://gerrit.wikimedia.org/r/291175 (owner: 10BryanDavis) [18:27:22] (03CR) 10Ori.livneh: [C: 032 V: 032] DBUtil.py: Fix PEP8 violations [puppet] - 10https://gerrit.wikimedia.org/r/291175 (owner: 10BryanDavis) [18:28:08] (03PS4) 10Ori.livneh: wdqs_updater.py: Fix PEP8 violations [puppet] - 10https://gerrit.wikimedia.org/r/291188 (owner: 10BryanDavis) [18:28:16] (03CR) 10Ori.livneh: [C: 032 V: 032] wdqs_updater.py: Fix PEP8 violations [puppet] - 10https://gerrit.wikimedia.org/r/291188 (owner: 10BryanDavis) [18:28:49] (03PS4) 10Ori.livneh: udp2log: Fix PEP8 violations [puppet] - 10https://gerrit.wikimedia.org/r/291186 (owner: 10BryanDavis) [18:28:59] (03CR) 10Ori.livneh: [C: 032 V: 032] udp2log: Fix PEP8 violations [puppet] - 10https://gerrit.wikimedia.org/r/291186 (owner: 10BryanDavis) [18:29:07] (03PS4) 10Ori.livneh: postgresql.py: Fix PEP8 violations [puppet] - 10https://gerrit.wikimedia.org/r/291182 (owner: 10BryanDavis) [18:29:14] (03CR) 10Ori.livneh: [C: 032 V: 032] postgresql.py: Fix PEP8 violations [puppet] - 10https://gerrit.wikimedia.org/r/291182 (owner: 10BryanDavis) [18:30:25] (03CR) 10Ori.livneh: [C: 031] "the mailman service resource subscribes to this file, meaning the service will get restarted, which is why I'm leaving it for someone else" [puppet] - 10https://gerrit.wikimedia.org/r/291180 (owner: 10BryanDavis) [18:30:40] (03PS4) 10Ori.livneh: ganglia: Fix PEP8 violations [puppet] - 10https://gerrit.wikimedia.org/r/291179 (owner: 10BryanDavis) [18:30:47] (03CR) 10Ori.livneh: [C: 032 V: 032] ganglia: Fix PEP8 violations [puppet] - 10https://gerrit.wikimedia.org/r/291179 (owner: 10BryanDavis) [18:31:49] (03PS4) 10Ori.livneh: rolematcher.py: Fix PEP8 violations [puppet] - 10https://gerrit.wikimedia.org/r/291177 (owner: 10BryanDavis) [18:31:58] (03CR) 10Ori.livneh: [C: 032 V: 032] rolematcher.py: Fix PEP8 violations [puppet] - 10https://gerrit.wikimedia.org/r/291177 (owner: 10BryanDavis) [18:32:52] (03PS4) 10Ori.livneh: wmfelastic.py: Fix PEP8 violations [puppet] - 10https://gerrit.wikimedia.org/r/291178 (owner: 10BryanDavis) [18:32:59] (03CR) 10Ori.livneh: [C: 032 V: 032] wmfelastic.py: Fix PEP8 violations [puppet] - 10https://gerrit.wikimedia.org/r/291178 (owner: 10BryanDavis) [18:34:36] (03CR) 10jenkins-bot: [V: 04-1] service: Let other methods of deployment work in uwsgi [puppet] - 10https://gerrit.wikimedia.org/r/291527 (owner: 10Ladsgroup) [18:35:08] (03PS4) 10Ori.livneh: gmond_memcached.py: Fix PEP8 violations [puppet] - 10https://gerrit.wikimedia.org/r/291176 (owner: 10BryanDavis) [18:35:46] (03CR) 10Ori.livneh: [C: 04-1] "the '}))' line should have the same indent level as the line on which the (({ appear" [puppet] - 10https://gerrit.wikimedia.org/r/291176 (owner: 10BryanDavis) [18:36:51] (03PS2) 10Ori.livneh: letsencrypt: Fix flake8 exclusion [puppet] - 10https://gerrit.wikimedia.org/r/291487 (owner: 10BryanDavis) [18:37:01] (03CR) 10Ori.livneh: [C: 032 V: 032] letsencrypt: Fix flake8 exclusion [puppet] - 10https://gerrit.wikimedia.org/r/291487 (owner: 10BryanDavis) [18:37:41] (03PS4) 10Ori.livneh: servermon: Fix PEP8 violations [puppet] - 10https://gerrit.wikimedia.org/r/291185 (owner: 10BryanDavis) [18:39:56] (03PS4) 10Ori.livneh: ircd_stats.py: Fix PEP8 violations [puppet] - 10https://gerrit.wikimedia.org/r/291181 (owner: 10BryanDavis) [18:40:06] (03CR) 10Ori.livneh: [C: 032 V: 032] ircd_stats.py: Fix PEP8 violations [puppet] - 10https://gerrit.wikimedia.org/r/291181 (owner: 10BryanDavis) [18:42:31] (03CR) 10Ori.livneh: [C: 031] "needs to be merged with someone more comfortable with swift (cc @godog)" [puppet] - 10https://gerrit.wikimedia.org/r/291173 (owner: 10BryanDavis) [18:42:43] (03CR) 10Ori.livneh: "*merged by" [puppet] - 10https://gerrit.wikimedia.org/r/291173 (owner: 10BryanDavis) [18:44:20] (03PS4) 10Ori.livneh: salt: Fix PEP8 violations [puppet] - 10https://gerrit.wikimedia.org/r/291184 (owner: 10BryanDavis) [18:44:51] (03CR) 10Ori.livneh: [C: 032 V: 032] "yuck, salt's magic dunders in global scope are pretty gross" [puppet] - 10https://gerrit.wikimedia.org/r/291184 (owner: 10BryanDavis) [18:46:38] (03PS2) 10Ori.livneh: kafkatee: submodule bump for pep8 fix [puppet] - 10https://gerrit.wikimedia.org/r/291366 (owner: 10BryanDavis) [18:46:47] (03CR) 10Ori.livneh: [C: 032 V: 032] kafkatee: submodule bump for pep8 fix [puppet] - 10https://gerrit.wikimedia.org/r/291366 (owner: 10BryanDavis) [18:46:53] (03PS2) 10Ori.livneh: varnishkafka: submodule bump for pep8 fix [puppet] - 10https://gerrit.wikimedia.org/r/291365 (owner: 10BryanDavis) [18:46:57] RECOVERY - puppet last run on labvirt1010 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [18:47:00] (03CR) 10Ori.livneh: [C: 032 V: 032] varnishkafka: submodule bump for pep8 fix [puppet] - 10https://gerrit.wikimedia.org/r/291365 (owner: 10BryanDavis) [18:51:28] (03CR) 10Ori.livneh: [C: 04-1] varnish: Fix PEP8 violations (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/291187 (owner: 10BryanDavis) [18:53:05] (03CR) 10Ladsgroup: "recheck" [puppet] - 10https://gerrit.wikimedia.org/r/291527 (owner: 10Ladsgroup) [19:00:17] (03CR) 10Ori.livneh: [C: 031] rsync::module: Replace obsolete to_a calls [puppet] - 10https://gerrit.wikimedia.org/r/291491 (owner: 10Alexandros Kosiaris) [19:07:41] PROBLEM - too much code across the tree merged is CRITICAL: today is Saturday [19:08:42] heh [19:09:01] i left the delicate ones alone [19:10:29] the ones that update swift / mailman / pybal python files that notify services [19:17:24] I actually wanted to ask ori to merge some patches :D [19:17:43] i'm not going to, the ones above were cosmetic [19:19:57] ori: I have some cosmetic patches too [19:20:01] for flake8 [19:20:10] remember you merged one of mine [19:21:17] could you ask me again on Monday? [19:21:31] sure [19:21:33] thanks [19:21:37] thank you [19:47:45] !log Updated Wikidata's property suggester with data from Monday's json dump and removed the external identifiers as a workaround for T132839 [19:47:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [19:48:06] T132839: Property suggester suggests human properties for non-human items - https://phabricator.wikimedia.org/T132839 [20:19:08] PROBLEM - puppet last run on labvirt1010 is CRITICAL: CRITICAL: Puppet has 5 failures [20:33:18] PROBLEM - puppet last run on cp3003 is CRITICAL: CRITICAL: Puppet has 1 failures [20:44:47] RECOVERY - cassandra-b CQL 10.192.32.135:9042 on restbase2003 is OK: TCP OK - 0.037 second response time on port 9042 [20:46:27] RECOVERY - puppet last run on labvirt1010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:50:54] (03PS1) 10Ladsgroup: dynamicproxy: make invisible-unicorn.py python3 compatible [puppet] - 10https://gerrit.wikimedia.org/r/291562 [20:51:28] (03PS1) 10Eevans: enable instance restbase2004-b.codfw.wmnet [puppet] - 10https://gerrit.wikimedia.org/r/291563 (https://phabricator.wikimedia.org/T134016) [20:52:09] Is there anyone around with +2 on Puppet that could merge https://gerrit.wikimedia.org/r/#/c/291563/ for me? It's (yet) another Cassandra bootstrap [20:58:54] (03PS1) 10Ladsgroup: dynamicproxy: Migrate to python3 [puppet] - 10https://gerrit.wikimedia.org/r/291565 [21:00:29] RECOVERY - puppet last run on cp3003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:21:21] (03CR) 10jenkins-bot: [V: 04-1] dynamicproxy: Migrate to python3 [puppet] - 10https://gerrit.wikimedia.org/r/291565 (owner: 10Ladsgroup) [21:24:38] (03PS4) 10BryanDavis: varnish: Fix PEP8 violations [puppet] - 10https://gerrit.wikimedia.org/r/291187 [21:24:40] (03CR) 10Ladsgroup: "recheck" [puppet] - 10https://gerrit.wikimedia.org/r/291565 (owner: 10Ladsgroup) [21:24:50] (03CR) 10BryanDavis: varnish: Fix PEP8 violations (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/291187 (owner: 10BryanDavis) [21:28:49] (03PS5) 10BryanDavis: gmond_memcached.py: Fix PEP8 violations [puppet] - 10https://gerrit.wikimedia.org/r/291176 [21:29:18] (03CR) 10BryanDavis: "> the '}))' line should have the same indent level as the line on which the (({ appear" [puppet] - 10https://gerrit.wikimedia.org/r/291176 (owner: 10BryanDavis) [22:43:26] (03PS1) 10Ladsgroup: wikilabels: make file settings recursive [puppet] - 10https://gerrit.wikimedia.org/r/291572 [22:49:41] (03CR) 10Ladsgroup: "recheck" [puppet] - 10https://gerrit.wikimedia.org/r/291565 (owner: 10Ladsgroup) [22:50:54] PROBLEM - puppet last run on db2038 is CRITICAL: CRITICAL: puppet fail [23:17:35] RECOVERY - puppet last run on db2038 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [23:22:45] PROBLEM - puppet last run on wasat is CRITICAL: CRITICAL: puppet fail [23:30:48] 06Operations, 10Wikimedia-Language-setup, 10Wikimedia-Site-requests: Rename zh-classical -> lzh - https://phabricator.wikimedia.org/T30443#2337160 (10Liuxinyu970226) Perhaps this is still #community-consensus-needed, because I have looked a large number of users, they're still using "zh-classical" on their u... [23:42:10] 06Operations, 10Ops-Access-Requests, 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: Allow RelEng access to labnet servers (was: Allow RelEng nova log access) - https://phabricator.wikimedia.org/T133992#2337167 (10Paladox) [23:44:43] Is there anyone around with +2 on Puppet I could interest in mergin https://gerrit.wikimedia.org/r/#/c/291563/ for me? It's just Cassandra bootstrap, totally safe/routine [23:49:32] PROBLEM - puppet last run on mw2079 is CRITICAL: CRITICAL: puppet fail [23:51:52] RECOVERY - puppet last run on wasat is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures