[00:41:54] PROBLEM - Puppet errors on tools-exec-1414 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [01:16:55] RECOVERY - Puppet errors on tools-exec-1414 is OK: OK: Less than 1.00% above the threshold [0.0] [01:18:23] PROBLEM - Puppet errors on tools-exec-1419 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [01:54:46] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1416 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [02:14:45] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [02:58:23] RECOVERY - Puppet errors on tools-exec-1419 is OK: OK: Less than 1.00% above the threshold [0.0] [03:24:03] kloeri, a.k.a. Bryan Ostergaard, is a known pedophile, child molester, child porn producer, and black-hat hacker. Report all sightings or contacts to ##kloeri at once! [03:25:58] Oshwah: [03:55:43] PROBLEM - Puppet errors on tools-exec-1427 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [04:15:46] RECOVERY - Puppet errors on tools-exec-1427 is OK: OK: Less than 1.00% above the threshold [0.0] [04:32:36] PROBLEM - Puppet errors on tools-webgrid-generic-1404 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [05:12:36] RECOVERY - Puppet errors on tools-webgrid-generic-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [07:09:47] (03CR) 10jenkins-bot: Localisation updates from https://translatewiki.net. [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/375727 (owner: 10L10n-bot) [07:25:32] 10Cloud-Services, 10Cloud-VPS, 10Operations, 10Patch-For-Review, 10Wikimedia-Incident: Some labs instances IP have multiple PTR entries in DNS - https://phabricator.wikimedia.org/T115194#3576434 (10hashar) @andrew is there still a lot of leaks happening? Current Nodepool id is 806287 [07:40:32] (03PS1) 10Legoktm: Add 'libraries' repo group [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/375738 [07:40:34] (03PS1) 10Legoktm: Log stderr from `docker logs` too [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/375739 [07:40:36] (03PS1) 10Legoktm: Don't try to sanitize logs if no password is given [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/375740 [07:43:17] (03CR) 10Legoktm: [C: 032] Add 'libraries' repo group [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/375738 (owner: 10Legoktm) [07:43:22] (03CR) 10Legoktm: [C: 032] Log stderr from `docker logs` too [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/375739 (owner: 10Legoktm) [07:43:26] (03CR) 10Legoktm: [C: 032] Don't try to sanitize logs if no password is given [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/375740 (owner: 10Legoktm) [07:45:06] (03Merged) 10jenkins-bot: Add 'libraries' repo group [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/375738 (owner: 10Legoktm) [07:45:08] (03Merged) 10jenkins-bot: Log stderr from `docker logs` too [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/375739 (owner: 10Legoktm) [07:45:10] (03Merged) 10jenkins-bot: Don't try to sanitize logs if no password is given [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/375740 (owner: 10Legoktm) [13:26:36] (03PS1) 10Lokal Profil: Harvest more russian fields into Monument database [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/375796 (https://phabricator.wikimedia.org/T174934) [13:29:09] (03CR) 10Jean-Frédéric: [C: 032] Harvest more russian fields into Monument database [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/375796 (https://phabricator.wikimedia.org/T174934) (owner: 10Lokal Profil) [13:30:05] (03Merged) 10jenkins-bot: Harvest more russian fields into Monument database [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/375796 (https://phabricator.wikimedia.org/T174934) (owner: 10Lokal Profil) [13:32:24] (03CR) 10jenkins-bot: Harvest more russian fields into Monument database [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/375796 (https://phabricator.wikimedia.org/T174934) (owner: 10Lokal Profil) [14:01:30] (03PS1) 10Hashar: Jenkins job validation (DO NOT SUBMIT) [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/375806 [14:01:41] (03PS1) 10Giuseppe Lavagetto: Add profile::openstack::main::rabbit_monitor_pass [labs/private] - 10https://gerrit.wikimedia.org/r/375807 [14:02:02] (03CR) 10Giuseppe Lavagetto: [V: 032 C: 032] Add profile::openstack::main::rabbit_monitor_pass [labs/private] - 10https://gerrit.wikimedia.org/r/375807 (owner: 10Giuseppe Lavagetto) [14:02:04] (03PS1) 10Hashar: Jenkins job validation (DO NOT SUBMIT) [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/375808 [14:02:22] (03CR) 10Hashar: "check experimental" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/375808 (owner: 10Hashar) [14:02:26] (03CR) 10Hashar: "check experimental" [labs/tools/guc] - 10https://gerrit.wikimedia.org/r/375806 (owner: 10Hashar) [14:03:23] (03PS1) 10Giuseppe Lavagetto: Add secrets for jobrunner.svc [labs/private] - 10https://gerrit.wikimedia.org/r/375809 [14:03:48] (03CR) 10Giuseppe Lavagetto: [V: 032 C: 032] Add secrets for jobrunner.svc [labs/private] - 10https://gerrit.wikimedia.org/r/375809 (owner: 10Giuseppe Lavagetto) [14:05:13] !log tools.heritage Deploy latest from Git master: 9f59c59 (T174245), bc94498, d5b2006 (T174934) [14:05:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.heritage/SAL [14:05:18] T174245: Add Peru in Spanish to monuments database - https://phabricator.wikimedia.org/T174245 [14:05:18] T174934: Harvest more russian fields - https://phabricator.wikimedia.org/T174934 [14:05:44] 10Cloud-VPS (Project-requests): Request creation of deep-learning-services VPS project - https://phabricator.wikimedia.org/T172421#3577653 (10Ebraminio) Hey guys, is this discussed last week or will be? [14:34:58] PROBLEM - Puppet errors on tools-worker-1020 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [14:45:24] (03CR) 10Aklapper: [C: 04-1] "Congratulations to your patch in Wikimedia Gerrit! If you are looking for further help or ideas, please check https://www.mediawiki.org/wi" [labs/tools/WikiConvFR-training-2016] - 10https://gerrit.wikimedia.org/r/361629 (owner: 10Minhtq15) [14:45:28] (03CR) 10Aklapper: [C: 04-1] "Congratulations to your patch in Wikimedia Gerrit! If you are looking for further help or ideas, please check https://www.mediawiki.org/wi" [labs/tools/WikiConvFR-training-2016] - 10https://gerrit.wikimedia.org/r/361628 (owner: 10Minhtq15) [14:45:34] (03CR) 10Aklapper: [C: 04-1] "Congratulations to your patch in Wikimedia Gerrit! If you are looking for further help or ideas, please check https://www.mediawiki.org/wi" [labs/tools/WikiConvFR-training-2016] - 10https://gerrit.wikimedia.org/r/361626 (owner: 10Minhtq15) [14:45:39] (03CR) 10Aklapper: [C: 04-1] "Congratulations to your patch in Wikimedia Gerrit! If you are looking for further help or ideas, please check https://www.mediawiki.org/wi" [labs/tools/WikiConvFR-training-2016] - 10https://gerrit.wikimedia.org/r/361021 (owner: 10Minhtq15) [14:45:43] (03CR) 10Aklapper: [C: 04-1] "Congratulations to your patch in Wikimedia Gerrit! If you are looking for further help or ideas, please check https://www.mediawiki.org/wi" [labs/tools/WikiConvFR-training-2016] - 10https://gerrit.wikimedia.org/r/361020 (owner: 10Minhtq15) [14:45:48] (03CR) 10Aklapper: [C: 04-1] "Congratulations to your patch in Wikimedia Gerrit! If you are looking for further help or ideas, please check https://www.mediawiki.org/wi" [labs/tools/WikiConvFR-training-2016] - 10https://gerrit.wikimedia.org/r/361022 (owner: 10Minhtq15) [14:45:59] (03CR) 10Aklapper: [C: 04-1] "Please abandon this change, as it seems to be a test commit. Thanks!" [labs/tools/WikiConvFR-training-2016] - 10https://gerrit.wikimedia.org/r/305865 (owner: 10Xavier Combelle) [14:51:01] andrewbogott: hi! labvirt1015 has been down for a few days now according to icinga. Nothing useful in console, it might need to be rebooted [14:51:27] ema: hm, another datapoint for https://phabricator.wikimedia.org/T171473 [14:52:22] ema: it's in downtime, right? Or did you only just do that? [14:52:56] 10cloud-services-team (Kanban), 10DC-Ops, 10Operations, 10ops-eqiad: labvirt1015 crashes - https://phabricator.wikimedia.org/T171473#3577868 (10Andrew) It's down again, icinga says since 2017-08-30 16:14:18 [14:53:39] oh, it's not. I'll mark it [14:53:48] thanks [15:14:56] RECOVERY - Puppet errors on tools-worker-1020 is OK: OK: Less than 1.00% above the threshold [0.0] [15:15:57] 10VPS-Projects, 10Social-Tools: social-tools.wmflabs.org is running MediaWiki v1.28.0 - https://phabricator.wikimedia.org/T174958#3577958 (10SamanthaNguyen) [16:03:54] 10Cloud-Services, 10Cloud-VPS, 10Operations, 10Patch-For-Review, 10Wikimedia-Incident: Some labs instances IP have multiple PTR entries in DNS - https://phabricator.wikimedia.org/T115194#3578005 (10Andrew) This no longer happens as a matter of course, but anytime designate-sink locks up we leak things fo... [16:10:35] 10Data-Services, 10XTools, 10Patch-For-Review: s51187 and p50380g50692 database users are generating excessive lag on replica service - https://phabricator.wikimedia.org/T172882#3578032 (10jcrespo) 05Resolved>03Open This is still ongoing- the CREATE ... SELECT on p50380g50692__DPL_p is the main cause why... [16:38:37] 10Cloud-Services, 10Cloud-VPS: Investigate rabbitmq tcp_listen_options setting (and others) - https://phabricator.wikimedia.org/T140175#3578124 (10Andrew) 05Open>03declined I've read this guide again and don't see any obvious things for us to change. [17:09:33] 10cloud-services-team, 10Analytics, 10Project-Admins, 10Research: Create a phabricator project called "wikireplica-datasets" - https://phabricator.wikimedia.org/T173512#3578176 (10Aklapper) @Halfak: Could you please answer the last comment? [17:25:57] 10Toolforge, 10Tools, 10Developer-Relations, 10Toolforge-standards-committee: Make sure abandoned useful tools are properly advertised so potentially interested new maintainers could find them - https://phabricator.wikimedia.org/T159595#3578193 (10Aklapper) Wondering what's left to do here? Should this be... [17:46:24] 10Data-Services, 10XTools: s51187 and p50380g50692 database users are generating excessive lag on replica service - https://phabricator.wikimedia.org/T172882#3578204 (10bd808) a:05jcrespo>03russblau The `p50380g50692` tables seem to belong to http://tools.wmflabs.org/admin/tool/dplbot [18:11:15] 10Data-Services, 10XTools: s51187 and p50380g50692 database users are generating excessive lag on replica service - https://phabricator.wikimedia.org/T172882#3578213 (10russblau) What is the specific query that seems to be causing the problem now? [18:13:35] 10Data-Services, 10XTools: s51187 and p50380g50692 database users are generating excessive lag on replica service - https://phabricator.wikimedia.org/T172882#3578214 (10jcrespo) CREATE TABLE ch_pl SELECT ... was running at the time, not sure if it was the only one. [18:41:28] (03CR) 10Merlijn van Deen: [C: 032] Add Mediawiki-extension-SendGrid work board to wikibugs [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/374053 (https://phabricator.wikimedia.org/T174260) (owner: 10D3r1ck01) [18:41:50] (03Merged) 10jenkins-bot: Add Mediawiki-extension-SendGrid work board to wikibugs [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/374053 (https://phabricator.wikimedia.org/T174260) (owner: 10D3r1ck01) [18:42:01] (03CR) 10jenkins-bot: Add Mediawiki-extension-SendGrid work board to wikibugs [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/374053 (https://phabricator.wikimedia.org/T174260) (owner: 10D3r1ck01) [18:45:56] (03CR) 10Merlijn van Deen: [C: 032] Connect wikibugs to irc over ssl [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/328663 (https://phabricator.wikimedia.org/T141089) (owner: 10Paladox) [18:46:43] (03CR) 10Paladox: "thanks." [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/328663 (https://phabricator.wikimedia.org/T141089) (owner: 10Paladox) [18:47:02] (03Merged) 10jenkins-bot: Connect wikibugs to irc over ssl [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/328663 (https://phabricator.wikimedia.org/T141089) (owner: 10Paladox) [18:47:10] (03CR) 10jenkins-bot: Connect wikibugs to irc over ssl [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/328663 (https://phabricator.wikimedia.org/T141089) (owner: 10Paladox) [18:55:37] (03PS1) 10Merlijn van Deen: Logging no longer supported -- fall back to stdout [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/375858 [18:56:07] (03PS2) 10Merlijn van Deen: Logging to irc no longer supported -- fall back to stdout [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/375858 [18:56:09] (03CR) 10jerkins-bot: [V: 04-1] Logging to irc no longer supported -- fall back to stdout [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/375858 (owner: 10Merlijn van Deen) [18:56:43] (03PS3) 10Merlijn van Deen: Logging to irc no longer supported -- fall back to stdout [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/375858 [19:00:56] 10Toolforge, 10cloud-services-team (Kanban), 10Documentation, 10User-bd808: Update code and/or docs for "How can I detect if I'm running in Labs?" - https://phabricator.wikimedia.org/T174082#3550488 (10valhallasw) There are three related files in `/etc`: `wmflabs_imageversion`, `wmflabs-instancename`, `wmf... [19:14:01] 10Toolforge, 10cloud-services-team (Kanban), 10Documentation, 10User-bd808: Update code and/or docs for "How can I detect if I'm running in Labs?" - https://phabricator.wikimedia.org/T174082#3578285 (10valhallasw) I have updated the docs to clarify the situation; if we want to go for renaming, https://gerr... [19:18:39] 10Toolforge, 10Documentation: Investigate and document xdebug usage for PHP projects on Toolforge - https://phabricator.wikimedia.org/T171419#3464354 (10valhallasw) It was enabled for SGE webservices (T72313), but seems to be disabled for kubernetes. Setting it up for kubernetes might be nontrivial, as this w... [19:24:44] 10Wikibugs, 10Patch-For-Review: Make wikibugs use SSL in IRC - https://phabricator.wikimedia.org/T141089#3578311 (10valhallasw) 05Open>03Resolved a:03valhallasw [19:49:43] 10Cloud-Services, 10Toolforge, 10Tracking: Packages to be added to toollabs puppet - https://phabricator.wikimedia.org/T55704#3578347 (10valhallasw) [19:49:45] 10Toolforge, 10cloud-services-team (Kanban): No module named 'pymysql' -- python3-mysql installed on toolforge bastions but not on exec nodes - https://phabricator.wikimedia.org/T174439#3578345 (10valhallasw) 05Open>03Resolved a:03valhallasw [20:07:27] (03PS1) 10Merlijn van Deen: Ignore libraryupgrader in grrrit-wm [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/375870 (https://phabricator.wikimedia.org/T174772) [20:07:34] legoktm: ^ I'm on a roll! :-p [20:07:50] tox will probably complain though [20:07:57] :D [20:08:51] and the code base looks like a good practice ground for learning how to refactor procedural code to OOP :D [20:08:55] it didn't! [20:09:01] heh [20:09:12] I blame the nodejs version that I ported to Python [20:09:20] feel free to refactor to your hearts content :) [20:09:22] haha [20:09:39] http://www.monkeyuser.com/assets/images/2017/56-refactor-man.png [20:10:03] Is my config for my channel wrong no notifications for my phab project for tools-zppixbot work for a while now [20:10:09] valhallasw`cloud: [20:12:20] Zppix: https://phabricator.wikimedia.org/project/view/2325/ [20:12:24] find the spelling mistake :P [20:12:51] (tool vs tools) [20:13:06] should be OK if you rename it to be tools- (or change the channel config) [20:14:33] (03Draft2) 10Zppix: fix typo in ##Zppix-Wikipedia chan config for phab [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/375871 [20:15:18] (03CR) 10Merlijn van Deen: [C: 032] fix typo in ##Zppix-Wikipedia chan config for phab [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/375871 (owner: 10Zppix) [20:15:34] Ty [20:15:42] (03Merged) 10jenkins-bot: fix typo in ##Zppix-Wikipedia chan config for phab [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/375871 (owner: 10Zppix) [20:15:50] (03CR) 10jenkins-bot: fix typo in ##Zppix-Wikipedia chan config for phab [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/375871 (owner: 10Zppix) [20:19:36] !log tools.wikibugs Updated channels.yaml to: 37300576b8c28804ddccfb864f7071c564af4f5f fix typo in ##Zppix-Wikipedia chan config for phab [20:26:31] !log git upgrading gerrit from 2.14.3-pre to 2.14-4-pre (2.14.3 was released, but i built from the 2.14 branch). [20:26:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Git/SAL [20:36:39] 10Cloud-Services, 10Toolforge, 10Labs-Team-Backlog, 10Mail: Set up A-based SPF for tools.wmflabs.org - https://phabricator.wikimedia.org/T104733#3578445 (10valhallasw) 05Open>03declined This was part of an implementation of {T96299}, {T97574} et al., but the implementation got stalled for various reaso... [21:05:17] 10Wikibugs, 10Africa-Wikimedia-Developers, 10MediaWiki-extensions-SendGrid, 10Patch-For-Review: Integrating SendGrid extension work board to wikibugs - https://phabricator.wikimedia.org/T174260#3578492 (10D3r1ck01) 05Open>03Resolved Thanks for merging @valhallasw. :) [21:50:44] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1416 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [22:15:43] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [23:21:04] 10Tools, 10Patch-For-Review: Tool "wikinity" loads assets from bootstrapcdn and ajax.googleapis.com - https://phabricator.wikimedia.org/T173065#3578578 (10Ricordisamoa) https://github.com/urbanecm/wikinity/pull/16