[00:00:20] RECOVERY - Puppet errors on tools-webgrid-generic-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [00:39:16] PROBLEM - Puppet errors on tools-exec-1438 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [00:53:13] 10Tool-Labs-tools-XTools, 10Community-Tech-Sprint: Planning for Xtools beta - https://phabricator.wikimedia.org/T167217#3433889 (10MZMcBride) >>! In T167217#3432300, @Niharika wrote: >>>! In T167217#3430537, @MZMcBride wrote: >> If you're going to spend time rewriting these tools, why not just make them proper... [00:55:34] Heh, I've had .hushlogin and LogLevel QUIET set for forever. [00:55:38] The hell is a Toolforge? [00:56:07] https://lists.wikimedia.org/pipermail/labs-l/2017-July/005036.html [00:56:09] Oh wow. [00:56:24] 10Cloud-VPS, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install labtestpuppetmaster2001 - https://phabricator.wikimedia.org/T167157#3433897 (10Dzahn) It has IPv6 and reverse now. [00:56:50] RIP, Labs. [00:57:19] it was a good name but the project has outgrown it [00:57:35] Good thing I named my bash alias 'tools'. ;-) [00:57:41] :) [00:59:51] mzmcbride@tools-bastion-03:~$ mv .hushlogin2 .hushlogin [00:59:56] * Esther takes a deep breath. [01:01:07] It's not at `/etc/motd`, which is new to me. Managed by Puppet via `/etc/init/mounted-run.conf` and `/etc/update-motd.d/50-bastion-banner` I guess. TIL. [01:14:14] RECOVERY - Puppet errors on tools-exec-1438 is OK: OK: Less than 1.00% above the threshold [0.0] [01:43:58] PROBLEM - Puppet errors on tools-exec-1418 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [01:58:13] 10Tool-Labs-tools-XTools, 10Community-Tech-Sprint: Form state not reset after using browser's back button in Safari/Firefox - https://phabricator.wikimedia.org/T170499#3433963 (10MusikAnimal) PR at https://github.com/x-tools/xtools-rebirth/pull/42 [02:30:12] 10Cloud-VPS, 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install labtestpuppetmaster2001 - https://phabricator.wikimedia.org/T167157#3434011 (10Dzahn) root@labtestpuppetmaster2001:~# ip a s | grep inet6 ... inet6 2620:0:860:4:208:80:153:108/64 scope global mngtmpaddr dynamic --- host l... [02:56:20] PROBLEM - Free space - all mounts on tools-logs-02 is CRITICAL: CRITICAL: tools.tools-logs-02.diskspace._srv.byte_percentfree (<55.56%) [02:57:20] PROBLEM - Puppet errors on tools-worker-1021 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [03:19:03] RECOVERY - Puppet errors on tools-exec-1418 is OK: OK: Less than 1.00% above the threshold [0.0] [03:24:11] 10Tool-Labs-tools-XTools, 10Community-Tech-Sprint: Remove edit size information in the Edit Counter in new XTools - https://phabricator.wikimedia.org/T170103#3419463 (10MusikAnimal) a:03MusikAnimal [03:24:33] is there a maintainers' email address for VPS maintainers, like there is for toolforge project maintainers? [03:32:20] RECOVERY - Puppet errors on tools-worker-1021 is OK: OK: Less than 1.00% above the threshold [0.0] [04:01:22] 10Toolforge: The Labs unicorn when you ssh into Labs is gone - https://phabricator.wikimedia.org/T170467#3434047 (10MZMcBride) I think this task is related to T168480 and . The unicorn you're referring to is now living in Git history, I believe. If you clone the `operations... [04:01:39] samwilson: not really. you should be able to mostly find people, though, via https://tools.wmflabs.org/openstack-browser/project/ [04:02:28] andrewbogott: cool. no, it's okay, i can use the matching maintainers email from the toolforge projects i reckon, and keep them up to date [04:02:38] it's for scripts to send things to, etc. [04:24:27] 10Tool-Labs-tools-XTools, 10Community-Tech-Sprint: Add a slow query killer to new XTools - https://phabricator.wikimedia.org/T170013#3434077 (10Samwilson) `pt-kill` is now active on the prod and staging servers. It's set up to run every minute and kill queries running longer than 120 seconds. It's logging wha... [04:44:48] 10Tool-Labs-tools-XTools, 10Community-Tech-Sprint: Export Page History top-editors as wikitext table - https://phabricator.wikimedia.org/T170098#3434089 (10Samwilson) PR merged. [04:58:20] 10Tool-Labs-tools-XTools: Set up XTools servers with Puppet - https://phabricator.wikimedia.org/T170514#3434103 (10Samwilson) [05:01:32] 10Tool-Labs-tools-XTools, 10Community-Tech-Sprint: Planning for Xtools beta - https://phabricator.wikimedia.org/T167217#3434121 (10Samwilson) The prod server now uses the prod DB credentials, and I've created T170514 to track the Puppetization of the server config. (@bd808 I'll ping you to figure out a time wh... [05:12:43] could someone please drop the s53003__xtools_prod database on toolsdb? it was created under the wrong account [05:14:15] 10Cloud-Services, 10DBA, 10User-Urbanecm: Prepare and check storage layer for dinwiki - https://phabricator.wikimedia.org/T169193#3434164 (10Marostegui) The sanitization was done correctly and no private data was reported, so I have created the views on labsdb1001 and labsdb1003. The new labs hosts have repl... [05:28:21] PROBLEM - Puppet errors on tools-exec-1407 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [05:53:22] PROBLEM - Puppet errors on tools-worker-1021 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [06:00:48] 10Tool-Labs-tools-XTools, 10Community-Tech-Sprint: Add a slow query killer to new XTools - https://phabricator.wikimedia.org/T170013#3434208 (10Samwilson) a:03Samwilson [06:03:20] RECOVERY - Puppet errors on tools-exec-1407 is OK: OK: Less than 1.00% above the threshold [0.0] [06:28:21] RECOVERY - Puppet errors on tools-worker-1021 is OK: OK: Less than 1.00% above the threshold [0.0] [07:11:21] RECOVERY - Free space - all mounts on tools-logs-02 is OK: OK: All targets OK [07:38:25] PROBLEM - Free space - all mounts on tools-worker-1020 is CRITICAL: CRITICAL: tools.tools-worker-1020.diskspace.root.byte_percentfree (<100.00%) [08:33:19] 10Cloud-VPS, 10cloud-services-team (Kanban), 10User-bd808: Redirect wdq.wmflabs.org to query.wikidata.org - https://phabricator.wikimedia.org/T169653#3434407 (10Magnus) I have removed the webproxy and the instance from the wdq project. [10:14:50] 10Tool-Labs-tools-Attribution-Generator, 10TCB-Team: Add license attribution directly to pictures - https://phabricator.wikimedia.org/T170535#3434734 (10Katja_Ullrich_WMDE) [12:07:03] bd808: it's thursday ;). What's the status of https://phabricator.wikimedia.org/T170174 ? [13:52:20] 10Cloud-Services, 10DBA, 10User-Urbanecm: Prepare and check storage layer for maiwikimedia - https://phabricator.wikimedia.org/T168788#3435384 (10Marostegui) Is this supposed to be like this (s3 master)? ``` root@db1075[(none)]> use maiwikimedia; Database changed root@db1075[maiwikimedia]> select count(*) fr... [14:17:17] 10Cloud-VPS, 10Continuous-Integration-Infrastructure, 10Nodepool, 10Patch-For-Review: figure out if nodepool is overwhelming rabbitmq and/or nova - https://phabricator.wikimedia.org/T170492#3435469 (10Andrew) So far things seem quieter and more stable with the nodepool rate change. If we get another good... [14:23:01] 10Cloud-Services, 10DBA, 10User-Urbanecm: Prepare and check storage layer for maiwikimedia - https://phabricator.wikimedia.org/T168788#3435546 (10Urbanecm) >>! In T168788#3435384, @Marostegui wrote: > Is this supposed to be like this (s3 master)? > ``` > root@db1075[(none)]> use maiwikimedia; > Database chan... [14:24:08] 10Cloud-Services, 10DBA, 10User-Urbanecm: Prepare and check storage layer for maiwikimedia - https://phabricator.wikimedia.org/T168788#3435566 (10Marostegui) >>! In T168788#3435546, @Urbanecm wrote: >>>! In T168788#3435384, @Marostegui wrote: >> Is this supposed to be like this (s3 master)? >> ``` >> root@db... [14:52:37] PROBLEM - Puppet errors on tools-prometheus-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [14:57:28] Sagan: on my list of things for today! Now I just need to start working the list [14:58:34] bd808: absolutely no problem :). I'm totally satisfied when I'm unblocked tomorow, or at the weekend [14:59:20] ...but you said thursday :D [14:59:28] samwilson: Please open a phab ticket for dropping s53003__xtools_prod [14:59:39] Sagan: I did, and thanks for poking [14:59:48] :) [14:59:56] I hoped to do it yesterday, but such is life [14:59:59] !log phabricator mariadb 10.2.7 segfaults php, downgrading to mariadb 10.1. [15:00:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Phabricator/SAL [15:01:20] 10Cloud-VPS, 10Discovery, 10Wikidata, 10Wikidata-Query-Service: Sunset of WDQ - https://phabricator.wikimedia.org/T153439#2880526 (10bd808) Should the empty wdq-mm project be deleted as well? [15:05:45] 10Tool-Labs-tools-XTools, 10Community-Tech-Sprint: Form state not reset after using browser's back button in Safari/Firefox - https://phabricator.wikimedia.org/T170499#3435878 (10MusikAnimal) PR merged. You can test this out now at https://xtools-dev.wmflabs.org [15:07:56] PROBLEM - Puppet errors on tools-prometheus-02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:04:15] 10Cloud-Services, 10wikitech.wikimedia.org, 10User-bd808: Update wikitech Titleblacklist - https://phabricator.wikimedia.org/T170178#3436136 (10bd808) I [[https://wikitech.wikimedia.org/w/index.php?title=MediaWiki:Titleblacklist&diff=1764462&oldid=1763321|added all of the accounts]] from: * https://anonscm.d... [16:05:23] 10Cloud-Services, 10cloud-services-team (Kanban), 10wikitech.wikimedia.org, 10User-bd808: Update wikitech Titleblacklist - https://phabricator.wikimedia.org/T170178#3436142 (10bd808) [16:26:33] 10Cloud-Services, 10cloud-services-team (Kanban), 10wikitech.wikimedia.org, 10User-bd808: Update wikitech Titleblacklist - https://phabricator.wikimedia.org/T170178#3436273 (10bd808) [16:26:35] 10Cloud-Services, 10cloud-services-team (Kanban), 10LDAP, 10User-bd808: LDAP is blocking a local service user account - https://phabricator.wikimedia.org/T170174#3436269 (10bd808) 05Open>03Resolved a:03bd808 The shell account `git` has been renamed to `mintao`. ``` name=T170174-rename-git-to-mintao.... [16:26:38] Sagan: ^ [16:28:36] reminds me i want to rename logstash.wikimedis.org one of these days [16:28:47] maybe call it logs.wikimedia.org or something [16:28:59] yeah. it was a shitty choice on my part [16:29:57] TIL we have a shell user named "hostmaster" [16:30:30] :) [16:30:46] lol. and it's blocked on wikitech. I guess I can nuke it [16:34:30] 10Cloud-Services, 10cloud-services-team (Kanban), 10wikitech.wikimedia.org, 10User-bd808: Update wikitech Titleblacklist - https://phabricator.wikimedia.org/T170178#3436300 (10bd808) Deleted `uid=hostmaster,ou=people,dc=wikimedia,dc=org`. Matches blacklist and associated Wikitech account was blocked 2015-0... [16:37:38] 10Cloud-Services, 10cloud-services-team (Kanban), 10wikitech.wikimedia.org, 10User-bd808: Update wikitech Titleblacklist - https://phabricator.wikimedia.org/T170178#3436340 (10Framawiki) (perhaps we can close this task ?) [16:38:31] 10Cloud-Services, 10cloud-services-team (Kanban), 10wikitech.wikimedia.org, 10User-bd808: Update wikitech Titleblacklist - https://phabricator.wikimedia.org/T170178#3436346 (10bd808) >>! In T170178#3436340, @Framawiki wrote: > (perhaps we can close this task ?) I will close it when I'm sure I am done with... [16:41:01] 10Cloud-Services, 10cloud-services-team (Kanban), 10LDAP, 10User-bd808: LDAP is blocking a local service user account - https://phabricator.wikimedia.org/T170174#3436378 (10bd808) https://wikitech.wikimedia.org/w/index.php?title=User_talk:Mintao&diff=1764469&oldid=1764468 [16:46:56] 10Cloud-Services, 10cloud-services-team (Kanban), 10wikitech.wikimedia.org, 10User-bd808: Update wikitech Titleblacklist - https://phabricator.wikimedia.org/T170178#3436416 (10bd808) Renamed `uid=prometheus` to `uid=jberkley` and left [[ https://wikitech.wikimedia.org/w/index.php?title=User_talk:Jermainebe... [16:50:09] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1417 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [16:53:00] PROBLEM - Puppet errors on tools-exec-1430 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [16:53:47] bd808: thinking about it? the ldap username can be something other than the wikitech username right? and titleblacklist only matches against the wikitech username not the ldap username right? [16:54:20] PROBLEM - Puppet errors on tools-exec-1407 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [16:54:22] yes to the first, no to the second [16:54:35] we check the title blacklist for both [16:54:40] I can't connect to tools.wmflabs.org [16:54:43] On the browser [16:54:48] oh that's good [16:55:02] zhuyifei1999_: why is that good? [16:55:05] I assumed that from the User: prefix [16:55:06] Cyberpower678: same here. looking [16:55:16] Cyberpower678: was replying to bd808 [16:55:22] zhuyifei1999_: oh. :p [16:55:24] bd808: thanks [16:55:39] * Cyberpower678 uses his localhost tool [16:56:01] yeah I got connection reset from http://tools.wmflabs.org/ [16:56:17] where does that damn proxy run? [16:57:02] * Cyberpower678 shrugs [16:57:08] the time when openstask-browser is inaccessible... [16:57:16] yeah. good times [16:57:24] bd808: try smacking it. Always worked in the past. :p [16:57:41] Or hitting it with a hammer. :p [16:58:21] * bd808 forces some puppet runs [16:58:31] oh fuck [16:58:40] I did this with an ldap change [16:58:53] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1407 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [16:58:56] PROBLEM - Puppet errors on tools-bastion-05 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [16:59:00] PROBLEM - ToolLabs Home Page on toollabs is CRITICAL: Connection refused [16:59:08] PROBLEM - Puppet errors on tools-exec-1431 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:59:55] PROBLEM - Puppet errors on tools-exec-1420 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [17:00:21] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1406 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [17:03:28] PROBLEM - Puppet errors on tools-proxy-01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [17:03:58] RECOVERY - ToolLabs Home Page on toollabs is OK: HTTP OK: HTTP/1.1 200 OK - 3570 bytes in 0.004 second response time [17:04:05] bd808: need help? (bought to hop into a meeting fyi -- but http://tools.wmflabs.org/ is up for me) [17:04:18] maybe just fixed :) [17:04:46] working on it [17:05:38] !log tools Upgraded nginx-common on tools-proxy-01 [17:05:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:08:23] 10Tool-Labs-tools-XTools, 10Community-Tech-Sprint: Remove edit size information in the Edit Counter in new XTools - https://phabricator.wikimedia.org/T170103#3436516 (10MusikAnimal) I couldn't resist but move forward with my idea to add the data for the past 5,000 edits. I ran this against my own account (~108... [17:09:00] bd808: did you get caught up in the change brandon made? [17:09:01] !log tools Upgraded nginx-common on tools-proxy-02 [17:09:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:09:07] yeah. [17:09:21] bd808: did that also hit the main cloud proxies? [17:09:29] novaproxy project rather than tools-proxy hosts [17:09:36] probably yes [17:09:42] I'll check them next [17:10:26] bd808: [17:10:27] | 9930528e-c824-48e0-932e-862ffe582412 | novaproxy-02 | ACTIVE | public=10.68.21.69 | [17:10:27] | 6e83879e-336a-4b01-98f8-5084b41e54b7 | novaproxy-01 | ACTIVE | public=10.68.21.68, 208.80.155.156 | [17:11:32] bd808: fyi bblack reverted [17:11:40] bd808: Thank you very much :) [17:11:42] could wait ou tthe change there (even tho upgrade may be the better option?) [17:11:48] yeah. I'm still going to upgrade nginx [17:11:54] kk [17:11:57] +1 [17:14:55] 10Tool-Labs-tools-XTools, 10Community-Tech-Sprint: Remove automated edits interface in Edit Counter - https://phabricator.wikimedia.org/T170185#3436550 (10MusikAnimal) a:03MusikAnimal [17:15:34] !log project-proxy Upgraded nginx-common [17:15:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Project-proxy/SAL [17:18:28] RECOVERY - Puppet errors on tools-proxy-01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:18:42] bd808: spot checks on a few novaproxy sites look good atm [17:18:58] yeah I think its ok [17:19:06] * bd808 checks paws [17:19:45] paws and quarry are up too [17:21:30] looks ok to me [17:21:40] bd808: are both instances upgraded? [17:22:19] in project-proxy and tools, yes (4 total nodes) [17:25:10] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1417 is OK: OK: Less than 1.00% above the threshold [0.0] [17:26:40] 10Cloud-Services, 10cloud-services-team (Kanban), 10wikitech.wikimedia.org, 10User-bd808: Update wikitech Titleblacklist - https://phabricator.wikimedia.org/T170178#3436626 (10bd808) >>! In T170178#3436416, @bd808 wrote: > Renamed `uid=prometheus` to `uid=jberkley` and left [[ https://wikitech.wikimedia.or... [17:27:59] RECOVERY - Puppet errors on tools-exec-1430 is OK: OK: Less than 1.00% above the threshold [0.0] [17:29:24] 10Tool-Labs-tools-XTools, 10Community-Tech: EditCounter throws warnings when user has zero edits - https://phabricator.wikimedia.org/T170608#3436636 (10MusikAnimal) [17:30:16] 10Tool-Labs-tools-XTools, 10Community-Tech: EditCounter throws warnings when user has zero edits - https://phabricator.wikimedia.org/T170608#3436636 (10MusikAnimal) [17:34:20] RECOVERY - Puppet errors on tools-exec-1407 is OK: OK: Less than 1.00% above the threshold [0.0] [17:34:57] RECOVERY - Puppet errors on tools-exec-1420 is OK: OK: Less than 1.00% above the threshold [0.0] [17:38:54] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1407 is OK: OK: Less than 1.00% above the threshold [0.0] [17:38:56] RECOVERY - Puppet errors on tools-bastion-05 is OK: OK: Less than 1.00% above the threshold [0.0] [17:39:08] RECOVERY - Puppet errors on tools-exec-1431 is OK: OK: Less than 1.00% above the threshold [0.0] [17:39:13] bd808: thanks. [17:40:21] Cyberpower678: there will be an incident report at some point today/tomorrow I'm sure. It turned out to be a prod puppet change that had unexpected effects [17:40:24] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1406 is OK: OK: Less than 1.00% above the threshold [0.0] [17:40:55] The consequences of technology and software. Changing something always has a side effect. ;p [17:41:14] 10Cloud-Services, 10cloud-services-team (Kanban), 10wikitech.wikimedia.org, 10User-bd808: Update wikitech Titleblacklist - https://phabricator.wikimedia.org/T170178#3436763 (10bd808) [17:41:44] 10Cloud-Services, 10cloud-services-team (Kanban), 10LDAP, 10Security, 10User-bd808: Shell user name "shadow" is not blocked - https://phabricator.wikimedia.org/T131408#3436769 (10bd808) [17:59:20] !log tools Puppet is disabled on tools-proxy-02 with no reason specified. [17:59:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:13:22] 10Tool-Labs-tools-XTools: Allow use of the profiler on xtools-dev.wmflabs.org - https://phabricator.wikimedia.org/T170615#3437010 (10Matthewrbowker) [18:31:23] (03PS6) 10Dzahn: Gerrit: Add gerrit pub key for ssh [labs/private] - 10https://gerrit.wikimedia.org/r/363755 (owner: 10Paladox) [18:32:19] (03CR) 10Dzahn: [C: 032] "per IRC talk - yea...i didn't intend to make you change all of scap for just gerrit" [labs/private] - 10https://gerrit.wikimedia.org/r/363755 (owner: 10Paladox) [18:33:26] (03CR) 10Dzahn: [V: 032 C: 032] Gerrit: Add gerrit pub key for ssh [labs/private] - 10https://gerrit.wikimedia.org/r/363755 (owner: 10Paladox) [18:41:15] paladox: ^ you can try again... and i'm out again [19:08:39] mutante thanks :) [19:08:54] works (i already cherrypicked it and replaced it with my keys) [19:38:19] PROBLEM - Puppet errors on tools-exec-1414 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [19:39:31] PROBLEM - Puppet errors on tools-exec-1442 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [19:46:22] 10Cloud-Services, 10cloud-services-team (Kanban), 10LDAP, 10User-bd808: 'prometheus' service user vs. actual human account 'prometheus' - https://phabricator.wikimedia.org/T158968#3437432 (10bd808) 05Open>03Resolved a:03bd808 I rediscovered this problem while working on {T170178}. My first change was... [19:50:16] 10Cloud-Services, 10cloud-services-team (Kanban), 10wikitech.wikimedia.org, 10User-bd808: Update wikitech Titleblacklist - https://phabricator.wikimedia.org/T170178#3437459 (10bd808) 05Open>03Resolved [19:55:49] 10Tool-Labs-tools-XTools, 10Community-Tech-Sprint: Remove edit size information in the Edit Counter in new XTools - https://phabricator.wikimedia.org/T170103#3437463 (10kaldari) >I couldn't resist but move forward with my idea to add the data for the past 5,000 edits (50,000 was a stretch). Fine with me, altho... [19:58:38] 10Cloud-VPS, 10Puppet: Invesitgate use of Puppet "environments" for per-project Puppet manifests - https://phabricator.wikimedia.org/T170370#3437479 (10bd808) [20:13:20] RECOVERY - Puppet errors on tools-exec-1414 is OK: OK: Less than 1.00% above the threshold [0.0] [20:14:31] RECOVERY - Puppet errors on tools-exec-1442 is OK: OK: Less than 1.00% above the threshold [0.0] [20:16:49] 10Cloud-Services, 10Toolforge, 10cloud-services-team (Kanban): Cron Spam: Package 'sudo-ldap' has conffile prompt and needs to be upgraded manually - https://phabricator.wikimedia.org/T168094#3355995 (10bd808) Is it safe to close this now @Andrew? [20:17:03] 10Cloud-Services, 10Cloud-VPS, 10cloud-services-team (Kanban), 10Patch-For-Review: designate: on instance deletion, remove proxy dns entries as well as proxy entries - https://phabricator.wikimedia.org/T168313#3360958 (10bd808) Is is safe to close this now @andrew? [20:23:59] PROBLEM - Puppet errors on tools-exec-1437 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [20:40:31] 10Tool-Labs-tools-XTools, 10Community-Tech-Sprint: Remove edit size information in the Edit Counter in new XTools - https://phabricator.wikimedia.org/T170103#3419463 (10Matthewrbowker) >>! In T170103#3437463, @kaldari wrote: >>I couldn't resist but move forward with my idea to add the data for the past 5,000 e... [20:45:23] 10Tool-Labs-tools-XTools: Resolve intermittent 500 and 503 errors with xtools - https://phabricator.wikimedia.org/T144783#3437647 (10Matthewrbowker) 05Open>03Invalid a:03Matthewrbowker I am closing this, {T153112} should fix this by eliminating the need for the web watcher at all. {T167217} is the plan fo... [20:47:21] PROBLEM - Puppet errors on tools-webgrid-generic-1401 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [20:52:00] 10Cloud-Services, 10Cloud-VPS, 10Tool-Labs-tools-XTools: Database upgrade MariaDB 10: Lock wait timeouts / deadlocks in a row - https://phabricator.wikimedia.org/T70753#3437702 (10Matthewrbowker) 05Open>03Invalid a:03Matthewrbowker I am closing this, work on {T153112} has removed temporary tables. {T1... [20:54:14] 10Cloud-Services, 10Cloud-VPS, 10cloud-services-team (Kanban), 10Patch-For-Review: designate: on instance deletion, remove proxy dns entries as well as proxy entries - https://phabricator.wikimedia.org/T168313#3437718 (10Andrew) 05Open>03Resolved yep! [20:54:20] 10Cloud-Services, 10Toolforge, 10cloud-services-team (Kanban): Cron Spam: Package 'sudo-ldap' has conffile prompt and needs to be upgraded manually - https://phabricator.wikimedia.org/T168094#3437720 (10Andrew) 05Open>03Resolved It is. [20:58:59] RECOVERY - Puppet errors on tools-exec-1437 is OK: OK: Less than 1.00% above the threshold [0.0] [21:16:32] 10Cloud-Services, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install labweb100[12].wikimedia.org - https://phabricator.wikimedia.org/T167820#3437793 (10RobH) [21:24:20] PROBLEM - Puppet errors on tools-worker-1021 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [21:25:29] !log tools Upgrading ElasticSearch cluster for T164842. There will be service interruptions [21:25:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [21:25:34] T164842: Upgrade Tool Labs elasticsearch to 5.x - https://phabricator.wikimedia.org/T164842 [21:27:23] RECOVERY - Puppet errors on tools-webgrid-generic-1401 is OK: OK: Less than 1.00% above the threshold [0.0] [21:36:07] 10Cloud-VPS, 10cloud-services-team (Kanban), 10User-bd808: Redirect wdq.wmflabs.org to query.wikidata.org - https://phabricator.wikimedia.org/T169653#3437813 (10Multichill) Thanks Bryan! [21:48:09] !log tools Elasticsearch cluster upgraded to 5.3.2 [21:48:44] that's not awesome :/ [21:56:31] (03PS1) 10BryanDavis: Update for ElasticSearch 5.x [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/365178 (https://phabricator.wikimedia.org/T164842) [21:59:53] !log tools Elasticsearch cluster upgraded to 5.3.2 [21:59:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [22:00:28] (03CR) 10BryanDavis: [V: 031 C: 032] "Tested via cherry-pick" [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/365178 (https://phabricator.wikimedia.org/T164842) (owner: 10BryanDavis) [22:00:28] sounds awsome :) [22:00:58] I had to fix stashbot :) [22:01:14] it was using the removed consistency flag [22:01:25] (03Merged) 10jenkins-bot: Update for ElasticSearch 5.x [labs/tools/stashbot] - 10https://gerrit.wikimedia.org/r/365178 (https://phabricator.wikimedia.org/T164842) (owner: 10BryanDavis) [22:02:33] !log tools.stashbot Updated to 3c3810c for T164842 [22:02:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stashbot/SAL [22:02:36] T164842: Upgrade Tool Labs elasticsearch to 5.x - https://phabricator.wikimedia.org/T164842 [22:04:20] RECOVERY - Puppet errors on tools-worker-1021 is OK: OK: Less than 1.00% above the threshold [0.0] [22:10:14] ebernhardson: ugh. what did you do about elastica wanting php >=5.6? [22:11:53] * bd808 sees the hack [22:15:58] bd808: evil, evil hax [22:16:11] yeah :/ [22:16:11] My ears are burning. [22:16:36] it turns out there is nothing in elastica that actually requires 5.6, except they updated their phpunit dependency to one that requires 5.6 [22:16:43] (and the author doesn't want to support EOL php) [22:17:30] Zend/php not supporting php while in heavy use is really annoying [22:36:59] 10Cloud-Services, 10Operations: rack/setup/install labweb100[12].wikimedia.org - https://phabricator.wikimedia.org/T167820#3437954 (10RobH) [22:44:11] 10Tool-Labs-tools-XTools, 10Community-Tech-Sprint: Form state not reset after using browser's back button in Safari/Firefox - https://phabricator.wikimedia.org/T170499#3433388 (10Samwilson) This is working for me in Firefox on Linux. I also checked in "Ubuntu Web Browser" but the timer didn't even work and th... [22:47:51] 10Toolforge: Drop database s53003__xtools_prod - https://phabricator.wikimedia.org/T170645#3437979 (10Samwilson) [23:08:31] !log tools.sal Updated to 1d4d39c for Elasticsearch 5.x compat [23:08:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.sal/SAL [23:25:00] PROBLEM - Puppet errors on tools-exec-1437 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [23:29:39] 10Tools: MediaWiki instance on Tool Labs used as a spam relay - https://phabricator.wikimedia.org/T169040#3438031 (10Luke081515) 05Resolved>03Open @Magnus I still see spam at the wiki, just by clicking on random page. [23:42:34] bd808: I think the labs TOU need a links more. It took me about two minutes from "Porta:Wikimedia VPS" to find them [23:43:28] yeah. there is an epic task to make that all better and update the terms. Its been stalled for a year :/ [23:43:43] I poked some people about it last week [23:59:50] bd808: In case I'm logging IP and email if a user creates an account, I guess I need only the first disclaimer for account creation, right? Not the second one for private data?