[01:10:11] 06Labs, 10Wikimedia-Extension-setup, 10wikitech.wikimedia.org, 07I18n, and 2 others: Install Translate extension on wikitech - https://phabricator.wikimedia.org/T100313#2618057 (10Dereckson) @Ladsgroup What are the blockers? [01:26:53] 06Labs, 10Tool-Labs-tools-Other, 06Community-Tech-Tool-Labs, 06Developer-Relations: Create an authoritative and well promoted catalog of Wikimedia tools - https://phabricator.wikimedia.org/T115650#1729232 (10Dereckson) @husky Could you give an opinion on this task as thge maintainer of http://tools.wmflabs... [01:35:43] 06Labs, 10Tool-Labs, 06Collaboration-Team-Triage, 06Community-Tech-Tool-Labs, and 3 others: Enable Flow on wikitech (labswiki and labtestwiki), then turn on for Tool talk namespace - https://phabricator.wikimedia.org/T127792#2618106 (10Dereckson) @Mattflaschen-WMF So I confirm `$wgDefaultExternalStore` is... [01:38:58] 06Labs, 10BetaFeatures, 10Wikimedia-Extension-setup, 10wikitech.wikimedia.org: Enable beta features for Wikitech - https://phabricator.wikimedia.org/T125941#2618109 (10Dereckson) [01:40:46] 06Labs, 10BetaFeatures, 10Wikimedia-Extension-setup, 10wikitech.wikimedia.org: Enable beta features for Wikitech - https://phabricator.wikimedia.org/T125941#2001068 (10Dereckson) And do we currently have a working job queue for wikitech? [01:51:13] 06Labs, 10Tool-Labs, 06Collaboration-Team-Triage, 06Community-Tech-Tool-Labs, and 5 others: Enable Flow on wikitech (labswiki and labtestwiki), then turn on for Tool talk namespace - https://phabricator.wikimedia.org/T127792#2618119 (10Dereckson) @jcrespo Do you prefer to create a new labswiki_flow databas... [01:53:47] 06Labs, 10Tool-Labs, 06Collaboration-Team-Triage, 06Community-Tech-Tool-Labs, and 5 others: Enable Flow on wikitech (labswiki and labtestwiki), then turn on for Tool talk namespace - https://phabricator.wikimedia.org/T127792#2054159 (10Catrope) >>! In T127792#2618119, @Dereckson wrote: > @jcrespo Do you pr... [01:56:57] 06Labs, 10Tool-Labs, 06Collaboration-Team-Triage, 06Community-Tech-Tool-Labs, and 5 others: Enable Flow on wikitech (labswiki and labtestwiki), then turn on for Tool talk namespace - https://phabricator.wikimedia.org/T127792#2618125 (10Dereckson) [01:57:52] 06Labs, 10Tool-Labs, 06Collaboration-Team-Triage, 06Community-Tech-Tool-Labs, and 5 others: Enable Flow on wikitech (labswiki and labtestwiki), then turn on for Tool talk namespace - https://phabricator.wikimedia.org/T127792#2054159 (10Dereckson) [02:25:53] !log deployment-prep deployed the latest version of mediawiki/services/parsoid/deploy.git to get https://gerrit.wikimedia.org/r/#/c/309001/ see T144884 [02:25:54] T144884: On beta cluster, images are replaced with failed (404) links to Special:FilePath when you load VE - https://phabricator.wikimedia.org/T144884 [02:26:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL, Master [02:37:17] 10Labs-project-other: Successful pilot of Discourse on https://discourse.wmflabs.org/ as an alternative to wikimedia-l mailinglist - https://phabricator.wikimedia.org/T124690#2618182 (10Samwilson) https://discourse.wmflabs.org seems to not be working at the moment ("504 Gateway Time-out"). [02:52:23] 06Labs, 10BetaFeatures, 10Wikimedia-Extension-setup, 10wikitech.wikimedia.org: Enable beta features for Wikitech - https://phabricator.wikimedia.org/T125941#2618193 (10Krenair) ```krenair@silver:~$ mwscript showJobs.php labswiki --group refreshLinksPrioritized: 0 queued; 507 claimed (0 active, 507 abandone... [06:06:17] 06Labs, 10Wikimedia-Extension-setup, 10wikitech.wikimedia.org, 07I18n, and 2 others: Install Translate extension on wikitech - https://phabricator.wikimedia.org/T100313#2618310 (10Ladsgroup) Get [[https://gerrit.wikimedia.org/r/#/c/214893/9 | this]] merged? [06:13:54] 06Labs, 10Wikimedia-Extension-setup, 10wikitech.wikimedia.org, 07I18n, and 2 others: Install Translate extension on wikitech - https://phabricator.wikimedia.org/T100313#2618318 (10Dereckson) @Ladsgroup schedule it @Jdforrester-WMF What are your fear and what would you like to avoid? What we could do is a... [06:31:41] 06Labs, 10Tool-Labs, 06Collaboration-Team-Triage, 06Community-Tech-Tool-Labs, and 5 others: Enable Flow on wikitech (labswiki and labtestwiki), then turn on for Tool talk namespace - https://phabricator.wikimedia.org/T127792#2618336 (10jcrespo) >>! In T127792#2618122, @Catrope wrote: >>>! In T127792#261811... [06:41:35] 06Labs, 10Tool-Labs: templatetiger is using 613G in Tools out of 8T - https://phabricator.wikimedia.org/T136192#2326337 (10madhuvishy) @Kolossos Hi! templatetiger is back up to using 594G - this is a big chunk of our available disk space. Could you please - Clean up old data - Let us know your cleanup strateg... [08:02:36] 10Tool-Labs-tools-Erwin's-tools: Unknown Error/MySQL errors - https://phabricator.wikimedia.org/T140421#2618392 (10Supernino) Related changed tool seem stuck, doesn't show new edits since yesterday before 15:00 UTC http://tools.wmflabs.org/erwin85/relatedchanges.php Can someone restart it? [08:45:52] PROBLEM - Puppet run on tools-services-02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [09:20:51] RECOVERY - Puppet run on tools-services-02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:24:11] 06Labs, 10Tool-Labs: dplbot webservice on Tools Labs fails repeatedly - https://phabricator.wikimedia.org/T115231#2618731 (10russblau) It is down again and not restarting. It just went down at approx. 10:20 GMT today (Thursday). [11:36:53] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:34:03] (03Draft1) 10Paladox: Hack to make sure the bot uses npm 2 instead of 1 [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/309297 [12:35:14] (03PS2) 10Paladox: Hack to make sure the bot uses npm 2 instead of 1 [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/309297 [12:35:25] (03CR) 10jenkins-bot: [V: 04-1] Hack to make sure the bot uses npm 2 instead of 1 [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/309297 (owner: 10Paladox) [12:36:07] (03PS3) 10Paladox: Hack to make sure the bot uses npm 2 instead of 1 [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/309297 [12:37:08] !log tools.lolrrrit-wm apply https://gerrit.wikimedia.org/r/#/c/309297/ [12:37:09] tools.lolrrrit-wm is not a valid project. [12:37:19] !log tools.lolrrit-wm apply https://gerrit.wikimedia.org/r/#/c/309297/ [12:37:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lolrrit-wm/SAL, Master [12:51:42] (03Abandoned) 10Paladox: Hack to make sure the bot uses npm 2 instead of 1 [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/309297 (owner: 10Paladox) [13:14:53] (03Draft1) 10Paladox: Update npm to npm 2 [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/309307 [13:15:32] (03PS2) 10Paladox: Update npm to npm 2 [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/309307 [13:23:45] (03PS3) 10Paladox: Update npm to npm 2 [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/309307 [13:23:59] (03PS4) 10Paladox: Update npm to npm 2 [labs/tools/grrrit] - 10https://gerrit.wikimedia.org/r/309307 [13:24:31] so there are issues on s6 labs replication right now [13:24:37] sorry [13:24:38] s2 [13:24:55] we are working on it, we may have hit a mariadb bug [13:25:27] data is available, lag may increase until it is fixed [13:25:34] *but [13:26:23] thanks jynus [13:28:21] cannot modify the topic [13:28:24] sorry [13:31:44] me neither apparently :) that's new [13:34:28] RECOVERY - Host tools-secgroup-test-103 is UP: PING OK - Packet loss = 0%, RTA = 0.63 ms [13:39:22] PROBLEM - Host tools-secgroup-test-103 is DOWN: CRITICAL - Host Unreachable (10.68.21.22) [13:44:04] 06Labs, 10Labs-Infrastructure, 10DBA: mysqld process hang in db1069 - S2 mysql instance - https://phabricator.wikimedia.org/T145077#2619183 (10Marostegui) [13:56:10] RECOVERY - Host secgroup-lag-102 is UP: PING OK - Packet loss = 0%, RTA = 0.74 ms [14:17:23] 06Labs, 10Labs-Infrastructure, 10DBA, 07Upstream: mysqld process hang in db1069 - S2 mysql instance - https://phabricator.wikimedia.org/T145077#2619249 (10jcrespo) This happened much more often before the latest upgrade (10.0.16, I think?). Since the upgrade to 10.0.22, this hadn't happened for months (6 m... [14:20:54] RECOVERY - Host tools-secgroup-test-102 is UP: PING OK - Packet loss = 0%, RTA = 0.72 ms [14:21:58] PROBLEM - Host secgroup-lag-102 is DOWN: CRITICAL - Host Unreachable (10.68.17.218) [14:35:25] heyall, can I get a confirmation that this is the file used to redact data from labsdb: https://github.com/wikimedia/operations-software-redactatron/blob/master/scripts/cols.txt ? [14:35:53] We're currently loading all mediawiki data into hadoop and want to follow the same standard [14:36:22] (we're not making it public, but want to reserve that option) [14:37:43] milimetric: may want to try in -databases, I'm not entirely sure that's the /whole/ story and don't want to mislead you [14:38:10] ok, thx, good call, I'll ask jaime [14:38:31] PROBLEM - Host tools-secgroup-test-102 is DOWN: CRITICAL - Host Unreachable (10.68.21.170) [14:51:21] (03PS1) 10Jean-Frédéric: Extract method get_categories_from_upper_categories [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/309324 [14:51:23] (03PS1) 10Jean-Frédéric: Audit-trail of categorization method [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/309325 (https://phabricator.wikimedia.org/T114166) [15:06:12] yuvipanda hi, im wondering could you review https://gerrit.wikimedia.org/r/#/c/309307/ and merge please? [15:06:29] Also would you be able to build the image to go onto it please? [15:06:47] It allows us to use npm2 instead of npm1, since ci had issues with using npm2 and nodejs4 [15:06:51] i mean npm 1 [15:06:59] please? [15:16:19] chasemp: ^ If I knew how to take the topic lock off I would :/ [15:16:43] bd808: thanks man, I think it was s6 tho? [15:16:47] you are supposed to have +t but maybe you have to op yourself to actually use it? [15:17:04] no, it's s2. 24h behind right now [15:17:21] https://tools.wmflabs.org/replag/ [15:17:32] ah right, it was a mistake the first mention from j.ynus I see it now [15:18:57] I'm not sure on op rights, I'll tinker w/ it [15:19:13] irc arcane things I have to lookup every time :) [15:19:24] one of these days I guess I should learn irc stuff :) [15:19:38] mostly I just know how to connect my client to freenode [15:56:30] (03PS2) 10BryanDavis: jsub: Add warnings for precise deprecation [labs/toollabs] - 10https://gerrit.wikimedia.org/r/307461 (https://phabricator.wikimedia.org/T143282) [16:00:00] (03CR) 10BryanDavis: [C: 032] jsub: Add warnings for precise deprecation [labs/toollabs] - 10https://gerrit.wikimedia.org/r/307461 (https://phabricator.wikimedia.org/T143282) (owner: 10BryanDavis) [16:00:26] (03Merged) 10jenkins-bot: jsub: Add warnings for precise deprecation [labs/toollabs] - 10https://gerrit.wikimedia.org/r/307461 (https://phabricator.wikimedia.org/T143282) (owner: 10BryanDavis) [16:36:53] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:47:43] Change on 12wikitech.wikimedia.org a page Nova Resource:Tools/Access Request/Rapsar was created, changed by Rapsar link https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Access_Request/Rapsar edit summary: Created page with "{{Tools Access Request |Justification=I am an active editor in Turkish Wikipedia and I would like to adopt some of these tools to Turkish Wikipedia. |Completed=false |User Nam..." [16:52:29] 06Labs, 07LDAP: Many (approx 265) labs instances in LDAP with associatedDomain: i-00000[0-9a-f]{3}\.(\b.*\b)?eqiad\.wmflabs - https://phabricator.wikimedia.org/T142449#2619720 (10AlexMonk-WMF) It's gone down a little bit over the past month: ```krenair@tools-bastion-03:~$ ldapsearch -x ou:dn:=hosts | grep -E "... [16:53:57] 06Labs, 07LDAP: Many (approx 265) labs instances in LDAP with associatedDomain: i-00000[0-9a-f]{3}\.(\b.*\b)?eqiad\.wmflabs - https://phabricator.wikimedia.org/T142449#2535532 (10Andrew) I think unless we're able to identify some of these as definite leaks, I'm inclined to leave them. There's some chance (alt... [17:01:44] (03PS1) 10BryanDavis: Bump Striker submodule [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/309353 [17:01:56] (03CR) 10BryanDavis: [C: 032] Bump Striker submodule [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/309353 (owner: 10BryanDavis) [17:02:02] (03Merged) 10jenkins-bot: Bump Striker submodule [labs/striker/deploy] - 10https://gerrit.wikimedia.org/r/309353 (owner: 10BryanDavis) [17:11:21] !log toolsbeta Added BryanDavis (self) to project as admin [17:11:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL, Master [17:13:52] andrewbogott: any eta for creating security groups ? [17:14:40] matanya: it should work fine on horizon [17:14:43] unless you know otherwise... [17:15:04] so i'd suggest modifying the error message in wikitech [17:16:09] (03CR) 10Jean-Frédéric: "The one in /test_fill_table.py is more helpful I would say:" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/302887 (owner: 10Lokal Profil) [17:16:39] hm, true [17:16:51] andrewbogott: or better, remove the option completly [17:18:03] Or redirect them to horzon [17:18:17] that is an option as well [17:20:04] (03PS7) 10Jean-Frédéric: Refactor database configuration handling [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/303428 [17:20:06] (03PS6) 10Jean-Frédéric: Setup local development environment for ErfgoedBot [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/303498 [17:22:26] 10Tool-Labs-tools-Pageviews: Improve Topviews interface - https://phabricator.wikimedia.org/T142802#2619804 (10MusikAnimal) [17:22:28] 10Tool-Labs-tools-Pageviews, 03Community-Tech-Sprint: Restrict Topviews to showing data only for individual days or months - https://phabricator.wikimedia.org/T142403#2619801 (10MusikAnimal) 05Open>03Resolved Deployed. Not even going to begin to ask for code review, it was a massive update that involved a... [17:24:02] 10Tool-Labs-tools-Pageviews, 03Community-Tech-Sprint: Restrict Topviews to showing data only for individual days or months - https://phabricator.wikimedia.org/T142403#2619808 (10MusikAnimal) [17:24:03] 10Tool-Labs-tools-Pageviews, 06Community-Tech, 07I18n: Topviews in the Pageviews labs tool doesn't auto-exclude special pages with localized names - https://phabricator.wikimedia.org/T139725#2619806 (10MusikAnimal) 05Open>03Resolved Fixed! You may need to clear your cache, though it will clear out on its... [17:32:21] 10Striker: Request connection of heritage repo to Striker - https://phabricator.wikimedia.org/T145096#2619827 (10JeanFred) [17:32:38] 10Striker, 10Wiki-Loves-Monuments: Request connection of heritage repo to Striker - https://phabricator.wikimedia.org/T145096#2619841 (10JeanFred) [17:39:30] andrewbogott: dns setup times out with 504 in horizon [17:45:11] 10Striker, 07Epic, 07Tracking: Manually connect existing Diffusion repos to tools in toolsadmin - https://phabricator.wikimedia.org/T144282#2619888 (10bd808) [17:45:13] 10Striker, 10Wiki-Loves-Monuments, 15User-bd808: Request connection of heritage repo to Striker - https://phabricator.wikimedia.org/T145096#2619885 (10bd808) 05Open>03Resolved a:03bd808 https://toolsadmin.wikimedia.org/tools/id/heritage/repos/id/labs-tool-heritage [17:45:31] 10Striker, 10Wiki-Loves-Monuments, 15User-bd808: Request connection of heritage repo to Striker - https://phabricator.wikimedia.org/T145096#2619890 (10Krenair) @EdouardHue, @PierreSelim and @Ynhockey should link their Phabricator accounts to LDAP Can't find user Spider [17:46:32] 10Striker, 10Wiki-Loves-Monuments, 15User-bd808: Request connection of heritage repo to Striker - https://phabricator.wikimedia.org/T145096#2619895 (10Krenair) @bd808, you gave those users edit access to the repository without their phabricator accounts being linked to LDAP? [17:51:33] 10Striker, 10Wiki-Loves-Monuments, 15User-bd808: Request connection of heritage repo to Striker - https://phabricator.wikimedia.org/T145096#2619902 (10bd808) >>! In T145096#2619895, @Krenair wrote: > @bd808, you gave those users edit access to the repository without their phabricator accounts being linked to... [17:53:04] 10Striker, 10Wiki-Loves-Monuments, 15User-bd808: Request connection of heritage repo to Striker - https://phabricator.wikimedia.org/T145096#2619904 (10Krenair) Okay. Is Striker going to be okay with that? What about the remaining user who doesn't appear to have a Phabricator account? [17:56:33] 10Striker, 10Wiki-Loves-Monuments, 15User-bd808: Request connection of heritage repo to Striker - https://phabricator.wikimedia.org/T145096#2619967 (10bd808) >>! In T145096#2619904, @Krenair wrote: > Okay. Is Striker going to be okay with that? For now, yes. Striker doesn't currently provide additional edit... [18:13:12] 06Labs, 10Labs-Infrastructure, 10DBA, 07Upstream: mysqld process hang in db1069 - S2 mysql instance - https://phabricator.wikimedia.org/T145077#2620064 (10jcrespo) It happened again. I am going to kill the server again, then convert the table to innodb, as that seemed to do the trick before. ``` Id:... [18:45:16] 06Labs, 10Labs-Infrastructure, 10DBA, 07Upstream: mysqld process hang in db1069 - S2 mysql instance - https://phabricator.wikimedia.org/T145077#2620315 (10Marostegui) Same query again - that is interesting. When I hit the MariaDB bug in the past, it used to be _always_ the same kind of query (a LOAD DATA i... [18:49:23] 06Labs, 10Labs-Infrastructure, 10DBA, 07Upstream: mysqld process hang in db1069 - S2 mysql instance - https://phabricator.wikimedia.org/T145077#2620335 (10jcrespo) BTW, it indeed worked, which I think almost confirms it is a Toku issue (again). [19:08:52] o/ [19:14:42] PROBLEM - Puppet run on tools-docker-builder-03 is CRITICAL: CRITICAL: 16.67% of data above the critical threshold [0.0] [19:19:40] RECOVERY - Puppet run on tools-docker-builder-03 is OK: OK: Less than 1.00% above the threshold [0.0] [19:26:40] matanya: it works for me. If you can reproduce it will you make me a bug? [19:37:30] 06Labs, 10Labs-Infrastructure, 06Operations, 10Traffic, 07HTTPS: update *.wmflabs.org certificate - https://phabricator.wikimedia.org/T145120#2620595 (10RobH) [19:40:21] 06Labs, 10Labs-Infrastructure, 06Operations, 10Traffic, and 2 others: update *.wmflabs.org certificate (existing expires on 2016-09-16) - https://phabricator.wikimedia.org/T145120#2620630 (10RobH) a:05RobH>03chasemp [19:41:47] 06Labs, 10Labs-Infrastructure, 06Operations, 10Traffic, and 2 others: update *.wmflabs.org certificate (existing expires on 2016-09-16) - https://phabricator.wikimedia.org/T145120#2620595 (10RobH) p:05Normal>03High [19:45:01] andrewbogott: thanks, it works now, not sure what happened [20:11:10] 10Tool-Labs-tools-Pageviews: Make automatically selecting chart type work without refreshing - https://phabricator.wikimedia.org/T145129#2620783 (10MusikAnimal) [20:12:52] PROBLEM - Puppet run on tools-services-01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [20:15:40] PROBLEM - Puppet run on tools-docker-builder-03 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [20:16:57] (03PS1) 10BryanDavis: jsub: Fix test suite [labs/toollabs] - 10https://gerrit.wikimedia.org/r/309390 [20:18:26] (03PS2) 10BryanDavis: jsub: Fix test suite [labs/toollabs] - 10https://gerrit.wikimedia.org/r/309390 [20:22:46] (03CR) 10BryanDavis: [C: 032] jsub: Fix test suite [labs/toollabs] - 10https://gerrit.wikimedia.org/r/309390 (owner: 10BryanDavis) [20:23:03] (03Merged) 10jenkins-bot: jsub: Fix test suite [labs/toollabs] - 10https://gerrit.wikimedia.org/r/309390 (owner: 10BryanDavis) [20:25:41] RECOVERY - Puppet run on tools-docker-builder-03 is OK: OK: Less than 1.00% above the threshold [0.0] [20:33:48] bd808: nice, a cloak :D [20:34:25] neat. I think paladox put in the paperwork for it [20:34:40] I did have to send some messages somewhere to ack the request [20:34:46] Yep [20:43:25] i haz a cloak too [20:43:47] Oh [20:43:54] When did you get yours? [20:45:00] paladox, 3 minutes ago :P [20:47:20] Ok :) [20:47:25] thanks [20:51:23] !log tools forcing puppet runs to install jobutils_1.15_all.deb [20:51:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [20:52:53] RECOVERY - Puppet run on tools-services-01 is OK: OK: Less than 1.00% above the threshold [0.0] [21:20:15] Successfully added .*@wikimedia/mviswanathan-wmf [21:20:15] @trustadd .*@wikimedia/mviswanathan-wmf admin [21:20:39] madhuvishy, ^ you can use @q, etc on the bot now. [21:20:57] tom29739: nice! thank you :) [21:49:47] !log tools forcing puppet runs to install toollabs-webservice_0.26_all.deb [21:49:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL, Master [21:57:07] PROBLEM - Puppet run on tools-webgrid-lighttpd-1208 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [21:58:12] PROBLEM - Puppet run on tools-webgrid-lighttpd-1204 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [22:02:38] PROBLEM - Puppet run on tools-webgrid-lighttpd-1207 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [22:07:16] PROBLEM - Puppet run on tools-precise-dev is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [22:09:23] PROBLEM - Puppet run on tools-webgrid-lighttpd-1206 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [22:18:05] 10Striker, 10Wiki-Loves-Monuments, 15User-bd808: Request connection of heritage repo to Striker - https://phabricator.wikimedia.org/T145096#2621164 (10PierreSelim) ??? and also ??? [22:18:21] PROBLEM - Puppet run on tools-webgrid-lighttpd-1210 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [22:18:51] PROBLEM - Puppet run on tools-webgrid-lighttpd-1205 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [22:19:23] PROBLEM - Puppet run on tools-webgrid-lighttpd-1209 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [22:22:03] 06Labs, 10Labs-Infrastructure: Make labs proxies https only - https://phabricator.wikimedia.org/T131288#2621194 (10AlexMonk-WMF) [22:22:11] 06Labs, 06Operations, 10Traffic, 07HTTPS: wmflabs.org should enforce HTTPS - https://phabricator.wikimedia.org/T144790#2610285 (10AlexMonk-WMF) [22:22:50] 06Labs, 10Labs-Infrastructure: Make labs proxies https only - https://phabricator.wikimedia.org/T131288#2162108 (10AlexMonk-WMF) See also T120486 [22:23:13] 06Labs, 06Operations, 10Traffic, 07HTTPS: wmflabs.org should enforce HTTPS - https://phabricator.wikimedia.org/T144790#2610285 (10AlexMonk-WMF) There's also T102367, but that's specific to tools [22:23:18] PROBLEM - Puppet run on tools-webgrid-lighttpd-1202 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [22:23:48] PROBLEM - Puppet run on tools-webgrid-lighttpd-1201 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [22:27:12] PROBLEM - Puppet run on tools-webgrid-lighttpd-1203 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [22:27:19] 06Labs, 10Labs-Infrastructure, 10Horizon: Disassociating floating IP does not make it available to other instances in the project - https://phabricator.wikimedia.org/T136872#2621213 (10AlexMonk-WMF) >>! In T136872#2552424, @bd808 wrote: >>>! In T136872#2552307, @AlexMonk-WMF wrote: >> I think this is because... [22:29:04] 06Labs, 10Labs-Infrastructure, 10Tool-Labs, 07Wikimedia-Incident: Write a simple script that handles failovering proxies - https://phabricator.wikimedia.org/T143639#2574425 (10AlexMonk-WMF) > Right now, the way to failover tools / nova proxy is: > Get on Horizon > De-allocate IP from active > Re-allocate t... [22:31:30] 06Labs, 10Continuous-Integration-Infrastructure, 06Operations, 07Nodepool: Upgrade Nodepool to 0.1.1-wmf5 to reduce requests made to OpenStack API - https://phabricator.wikimedia.org/T145142#2621216 (10hashar) [22:39:04] 06Labs, 10Labs-Infrastructure, 10Horizon: Disassociating floating IP does not make it available to other instances in the project - https://phabricator.wikimedia.org/T136872#2621271 (10bd808) >>! In T136872#2621213, @AlexMonk-WMF wrote: > I was just looking around Horizon for unrelated reasons and ran into P... [22:42:52] 06Labs, 10Labs-Infrastructure, 10Horizon: Disassociating floating IP does not make it available to other instances in the project - https://phabricator.wikimedia.org/T136872#2621301 (10AlexMonk-WMF) >>! In T136872#2621271, @bd808 wrote: >>>! In T136872#2621213, @AlexMonk-WMF wrote: >> I was just looking arou... [22:52:39] 10Striker, 10Wiki-Loves-Monuments, 15User-bd808: Request connection of heritage repo to Striker - https://phabricator.wikimedia.org/T145096#2621354 (10bd808) >>! In T145096#2621164, @PierreSelim wrote: > ??? and also ??? @JeanFred asked for the existing {rTHER} Diffusion repository that is a mirror of the [... [22:56:15] 06Labs, 10Tool-Labs: Make trusty the default release target for jsub commands - https://phabricator.wikimedia.org/T143284#2621381 (10bd808) [22:56:17] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 07Epic, 15User-bd808: Make jsub / qsub default to trusty instances - https://phabricator.wikimedia.org/T94792#2621382 (10bd808) [22:56:20] 06Labs, 10Tool-Labs, 06Community-Tech-Tool-Labs, 13Patch-For-Review, 15User-bd808: Make jsub warn when run without `-l release=...` - https://phabricator.wikimedia.org/T143282#2621379 (10bd808) 05Open>03Resolved Updated versions of `jsub` and `webservice` have been deployed. [22:58:25] PROBLEM - Puppet run on tools-bastion-03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [23:21:56] 06Labs: Build and package traefik - https://phabricator.wikimedia.org/T143294#2621476 (10yuvipanda) a:05yuvipanda>03None un cookie licking, I don't think I'll have any time to do this for a while :( [23:22:21] 06Labs, 13Patch-For-Review: Don't set instance root passwords if using a local puppetmaster - https://phabricator.wikimedia.org/T142531#2621478 (10yuvipanda) I think @andrew got this done? [23:22:43] 06Labs, 10Labs-Kubernetes, 10Tool-Labs: Puppet fails with 'no user syslog' when setting up syslog receiver role on Debian Jessie - https://phabricator.wikimedia.org/T141498#2621479 (10yuvipanda) a:05yuvipanda>03None [23:23:37] 06Labs, 06Operations, 13Patch-For-Review: Phase out the 'puppet' module with fire, make self hosted puppetmasters use the puppetmaster module - https://phabricator.wikimedia.org/T120159#2621490 (10yuvipanda) Only things left are etcd and integration. [23:28:25] RECOVERY - Puppet run on tools-bastion-03 is OK: OK: Less than 1.00% above the threshold [0.0] [23:32:53] 06Labs, 06Operations, 13Patch-For-Review: Phase out the 'puppet' module with fire, make self hosted puppetmasters use the puppetmaster module - https://phabricator.wikimedia.org/T120159#2621517 (10yuvipanda) Hi @joe. Do you still intend on using the etcd project as is? I tried looking through it, puppet has... [23:45:40] bd808: I created the Labs VPN proposal, it's here: https://wikitech.wikimedia.org/wiki/Labs_VPN_proposal [23:57:13] PROBLEM - SSH on tools-exec-1219 is CRITICAL: Server answer