[00:32:51] !log tools migrating tools-worker-1022, 1023, 1025, 1026 to eqiad1-r [00:32:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [08:39:50] !log tools T218216 disable puppet in all tools-sgeexec-XXXX nodes for controlled sssd rollout [08:39:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [08:39:53] T218216: When the link is to a section, show snippet of the section - https://phabricator.wikimedia.org/T218216 [09:03:01] !log tools T218216 do a controlled rollover of sssd, depooling sgeexec nodes, reboot and repool [09:03:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:03:04] T218216: When the link is to a section, show snippet of the section - https://phabricator.wikimedia.org/T218216 [09:04:46] !log tools T218216 add `profile::ldap::client::labs::client_stack: sssd` to prefix puppet for sge-exec nodes [09:04:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:26:45] !log tools T218216 hard reboot tools-sgeexec-0932 [09:26:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:26:49] T218216: When the link is to a section, show snippet of the section - https://phabricator.wikimedia.org/T218216 [09:27:24] I just noticed is T218126 (typo) [09:27:24] T218126: LDAP: try how sssd works with our servers - https://phabricator.wikimedia.org/T218126 [09:27:47] !log tools T218126 hard reboot tools-sgeexec-0932 [09:27:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:41:00] !log tools T218126 hard reboot tools-sgeexec-0918 [09:41:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:41:03] T218126: LDAP: try how sssd works with our servers - https://phabricator.wikimedia.org/T218126 [09:56:49] !log tools.quickstatements force deleted job 853139 because it was stucked (trying to depool exec node for T218126) [09:56:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.quickstatements/SAL [09:56:52] T218126: LDAP: try how sssd works with our servers - https://phabricator.wikimedia.org/T218126 [09:57:40] !log tools.wmcounter force deleted job 853968 because it was stucked (trying to depool exec node for T218126) [09:57:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wmcounter/SAL [09:58:08] !log tools.citationhunt force deleted job 871945 because it was stucked (trying to depool exec node for T218126) [09:58:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.citationhunt/SAL [10:01:59] !log tools T218126 hard reboot tools-sgeexec-0907 [10:02:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:02:02] T218126: LDAP: try how sssd works with our servers - https://phabricator.wikimedia.org/T218126 [10:15:43] !log tools.listeria force deleted job 1044941 because it was stucked (trying to depool exec node for T218126) [10:15:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.listeria/SAL [10:15:46] T218126: LDAP: try how sssd works with our servers - https://phabricator.wikimedia.org/T218126 [10:16:39] !log tools.jarbot force deleted job 1045059 because it was stucked (trying to depool exec node for T218126) [10:16:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.jarbot/SAL [10:19:15] !log tools T218126 hard reboot tools-sgeexec-0914 [10:19:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:26:13] !log tools.aivanalysis force deleted job 1739135 because it was stucked (trying to depool exec node for T218126) [10:26:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.aivanalysis/SAL [10:26:17] T218126: LDAP: try how sssd works with our servers - https://phabricator.wikimedia.org/T218126 [10:26:55] !log tools.jarbot force deleted job 1739159 because it was stucked (trying to depool exec node for T218126) [10:26:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.jarbot/SAL [10:27:03] !log tools T218126 hard reboot tools-sgeexec-0935 [10:27:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:42:10] !log tools.himo force deleted job 853604 because it was stucked (trying to depool exec node for T218126) [10:42:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.himo/SAL [10:42:17] T218126: LDAP: try how sssd works with our servers - https://phabricator.wikimedia.org/T218126 [10:43:57] !log tools T218126 hard reboot tools-sgeexec-0915 [10:44:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:49:14] !log tools T218126 hard reboot tools-sgeexec-0923 [10:49:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:49:18] T218126: LDAP: try how sssd works with our servers - https://phabricator.wikimedia.org/T218126 [10:53:42] !log tools.ukbot force deleted job 1264656 because it was stucked (trying to depool exec node for T218126) [10:53:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.ukbot/SAL [11:03:31] !log tools T218126 hard reboot tools-sgeexec-0928 [11:03:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:03:34] T218126: LDAP: try how sssd works with our servers - https://phabricator.wikimedia.org/T218126 [11:23:46] !log tools T218126 hard reboot tools-sgeexec-0940 [11:23:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:23:50] T218126: LDAP: try how sssd works with our servers - https://phabricator.wikimedia.org/T218126 [11:42:35] !log tools.musikbot force deleted job 1262132 because it was stucked (trying to depool exec node for T218126) [11:42:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.musikbot/SAL [11:42:39] T218126: LDAP: try how sssd works with our servers - https://phabricator.wikimedia.org/T218126 [11:47:35] !log tools T218126 hard reboot tools-sgeexec-0921 [11:47:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:55:01] arturo: is that stuck musikbot job anything I should be concerned about? [11:55:38] musikanimal: I don't think so, is probably related to a NFS issue in the exec node [11:55:56] !log tools T218126 hard reboot tools-sgeexec-0924 [11:55:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:55:59] T218126: LDAP: try how sssd works with our servers - https://phabricator.wikimedia.org/T218126 [11:56:25] Okay thank you :) [12:00:02] you are welcome :-) [12:02:16] !log tools.grapedog force deleted job 1044978 because it was stucked (trying to depool exec node for T218126) [12:02:21] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.grapedog/SAL [12:02:23] T218126: LDAP: try how sssd works with our servers - https://phabricator.wikimedia.org/T218126 [12:06:05] !log tools T218126 hard reboot tools-sgeexec-0901 [12:06:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:27:26] !log tools T218126 hard reboot tools-sgeexec-0925 [12:27:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:27:31] T218126: LDAP: try how sssd works with our servers - https://phabricator.wikimedia.org/T218126 [12:31:47] !log tools T218126 hard reboot tools-sgeexec-0926 [12:31:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [12:42:09] !log tools.jarbot force deleted job 1045055 because it was stucked (trying to depool exec node for T218126) [12:42:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.jarbot/SAL [12:42:12] T218126: LDAP: try how sssd works with our servers - https://phabricator.wikimedia.org/T218126 [12:43:37] Hi, wikibugs seems to have stopped working. [12:43:48] Could someone restart it please? (https://www.mediawiki.org/wiki/Wikibugs) [12:44:02] wikibugs: is in here? [12:44:17] Yes, but it's not outputting anything :) [12:44:30] isn't that better? :-P [12:44:35] lol [12:44:45] just kidding, truth is that I /ignore wikibugs [12:44:49] too verbose for me [12:44:58] poor wikibugs [12:48:22] !log tools.wikibugs restart wikibugs following docs at https://www.mediawiki.org/wiki/Wikibugs#Restarting_wikibugs [12:48:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [12:48:28] thanks arturo! [12:48:58] yw [12:53:09] !log tools.vltools force deleted job 1011036 because it was stucked (trying to depool exec node for T218126) [12:53:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.vltools/SAL [12:53:11] T218126: LDAP: try how sssd works with our servers - https://phabricator.wikimedia.org/T218126 [12:53:26] arturo: "stucked"? [12:53:46] typo? [12:54:16] arturo: thats what i thought i just noticed you keep doing it so i thought to think it wasnt :P [12:54:24] started* [12:54:53] ok, apparently it should be simply `stuck` https://www.ibm.com/support/knowledgecenter/en/SSGMGV_3.1.0/com.ibm.cics.ts31.doc/dfhp9/dfhp94g.htm [12:55:08] * arturo not very precise in his english [12:55:25] *shrug* [12:55:27] I keep copy/pasting the same message [13:02:50] Do you guys have a clue why I am having `error: cannot open .git/FETCH_HEAD: Permission denied` when trying to `git pull` code in `deployment-prep.eqiad.wmflabs` instances? [13:03:35] mateusbs17: that sounds like typical permission error, perhaps someone git pulled as root before [13:03:56] (or you should as root, depending on the puppet tree and the context) [13:06:37] !log tools T218126 hard reboot tools-sgeexec-0906 [13:06:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:06:40] T218126: LDAP: try how sssd works with our servers - https://phabricator.wikimedia.org/T218126 [14:01:12] Technical Advice IRC meeting starting in 60 minutes in channel #wikimedia-tech, hosts: @amir1 & @subbu - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [14:49:03] !log tools cleared E state from 5 queues [14:49:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:50:56] Technical Advice IRC meeting starting in 10 minutes in channel #wikimedia-tech, hosts: @amir1 & @subbu - all questions welcome, more infos: https://www.mediawiki.org/wiki/Technical_Advice_IRC_Meeting [16:43:25] bd808: once a cloud vps project is +1’d at the cloud services meeting whats the next step? (I cant find any info on it past this) [16:44:21] Zppix: for you the process is "wait to be told the project has been created". For the Cloud VPS admin team its typically a task taken care of by that week's on-call person. [16:44:43] Ah ok just making sure i didnt need to do any additional steps [16:45:22] Zppix: sure. it was a good question I think :) [16:45:46] * bd808 will try to add something about this on https://phabricator.wikimedia.org/project/view/2875/ [16:45:54] Cool [18:46:48] !log tools depooled and rebooted tools-sgewebgrid-lighttpd-0913 because high load was caused by ancient lsof processes [18:46:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:52:39] !log depooled and rebooted tools-sgeexec-0929 because systemd was in a weird state [18:52:39] bstorm_: Unknown project "depooled" [18:52:46] oops [18:52:54] !log tools depooled and rebooted tools-sgeexec-0929 because systemd was in a weird state [18:52:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:17:44] bstorm_: im having an issue with sshing into puppet-lta it says public key denied [19:18:04] what's the instance name? [19:18:12] bstorm_: "puppet-lta" [19:18:14] It may not be done with initial puppet run [19:18:21] bstorm_: ah that may be it [19:18:27] bstorm_: i didnt think about that [19:18:44] In horizon, you can check the console log tab for the instance [19:19:12] bstorm_: i am its running the puppet run now, i guess i just need to learn patience :P [19:19:42] :) [19:20:07] No worries. I hate waiting for the longer runs. [19:21:01] bstorm_: im in now thanks anyways :P [19:21:11] 👍🏻 [19:23:23] !log tools.dplbot T220646 shut down webservice because it is just in a restart loop and cannot run [19:23:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.dplbot/SAL [19:23:26] T220646: Tool dplbot not running correctly on the stretch grid - https://phabricator.wikimedia.org/T220646 [21:35:50] !log tools.mediaviews-api shut down webservice. It never runs at this time. It appears to need venv rebuilt. [21:35:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.mediaviews-api/SAL [23:58:52] !log tools moving tools-clushmaster-02, tools-elastic-03 and tools-paws-worker-1001 to eqiad1-r [23:58:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL