[00:05:46] hi, it appears that the cloud puppet master is throwing: [00:05:47] Error: /Stage[main]/Geoip::Data::Puppet/File[/usr/share/GeoIP]: Failed to generate additional resources using 'eval_generate': Error 500 on SERVER: Server Error: Permission denied @ rb_sysopen - /var/lib/puppet/volatile/GeoIP/.geoipupdate.lock [00:12:49] paladox: https://phabricator.wikimedia.org/T83447 :/ [00:13:01] i know it's WMF-NDA but it's just because it came from RT [00:13:12] oh [00:13:13] anyways there used to be this: [00:13:22] https://gerrit.wikimedia.org/r/c/operations/puppet/+/121677 [00:13:32] include puppet::self::geoip [00:13:38] but that is years old and not there anymore now [00:14:17] new docs on how to create a deployment_server (stretch) in a cloud VPS project https://wikitech.wikimedia.org/wiki/Deployment_server [00:21:42] thanks bd808 [00:30:07] !log devtools deploy-1002 live hack /srv/deployment/phabricator/deployment/scap/phabricator-targets and replace prod server with cloud instances; scap deploy in phabricator repo [00:30:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL [00:53:34] !log devtools deploy-1002 - become 'trebuchet' user and ssh to phabricator scap targets. to fix ssh host key verification issue on first deploy [00:53:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL [02:35:03] mutante: I haven't checked out your version yet, but I have https://wikitech.wikimedia.org/wiki/User:BryanDavis/Scap3_in_a_Cloud_VPS_project as well [02:35:34] which has been a new adventure in "fun" every time I have built a new one [08:03:16] Help [08:58:10] lamas? [12:07:18] !log toolsbeta live-hack tools-webservice in tools-sgebastion-04 to test https://gerrit.wikimedia.org/r/c/565259 (T242719) [12:07:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [12:07:21] T242719: https://tools.wmflabs.org/{toolname} no longer redirects to https://tools.wmflabs.org/{toolname}/ on new k8s cluster - https://phabricator.wikimedia.org/T242719 [13:11:38] andrewbogott: could you restart phebot again when you have a minute, please? [13:14:59] I can [13:15:16] Coren: could you give me additional details on what shall I do? [13:16:47] It's the only bot of Phe/Tpt I don't have access to (they forgot that one, and they've been inactive in ages) so I never did it myself directly; but I'm going to bet you anything it's literally the last command typed in the history. :-) [13:17:30] IIRC andrew said it was a shell wrapper, it should be named something like run-match-and-split or something close. :-) [13:18:07] "Hi, nice to meet you" by the way. I don't think we've met. :-) [13:18:07] is this a tool running on toolforge? [13:18:19] Yes, sorry, implied context. :-) [13:18:20] likewise :-) [13:18:24] aborrero@tools-sgebastion-07:~$ sudo become phebot [13:18:24] become: no such tool 'phebot' [13:19:06] phetools [13:19:48] * Coren sighs. [13:19:54] phetools, yeah [13:20:06] the history isn't clear on what to do [13:20:31] It's a single wrapper script with no argument IIRC [13:20:41] https://www.irccloud.com/pastebin/c7KC2mU7/ [13:20:51] that's an excerpt from the history Coren ^^^ [13:21:10] Oh! Either tpt or phe have been active?! [13:21:48] Oh, no, that looks like Andrew keeping an eye on the logs. [13:21:54] Go back up just a bit? [13:22:08] It really is just a wrapper script. [13:22:19] Jan3 is when Andrew restarted it the last time. :-( [13:23:18] this is what happened in Jan 3 [13:23:21] https://www.irccloud.com/pastebin/4JJS3yKg/ [13:24:16] Nope. Got the date wrong, so tpt and or phe were there. That's good news I suppose except for the fact that nobody's able to contact them. :-( [13:24:56] there is no SAL for this tool apparently [13:25:00] And no, the match-and-split restart isn't in that bit of history. Can you grep 'split' in the history? [13:25:24] ok, might be this? `~/phe/run_service.sh restart match_and_split` [13:26:02] If that's not it, I lose all faith in naming things. [13:26:06] :-D [13:26:16] ok, let's try that! [13:26:42] !log tools.phetools running `~/phe/run_service.sh restart match_and_split` per Coren request on IRC [13:26:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.phetools/SAL [13:26:59] (first entry in the SAL, heh) [13:27:09] Coren: [13:27:12] https://www.irccloud.com/pastebin/8ipZjfO8/ [13:27:51] Yeah, that looks good. Interesting, it got wedged rather that simply crash this time? [13:28:37] I'm noting down the actual restart command for future reference. Thanks a lot, arturo [13:28:48] any time :-) [13:28:58] having stuff registered in the SAL also helps [13:29:12] it is the first resource I check for this kind of operations [13:29:28] * Coren grins. [13:29:57] In *my* days, we only used the SAL for opsen stuff; having it for individual tools is newfangled. :-P [13:30:16] * Coren waves his cane around. [13:30:44] SALaaS, true cloud, isn't it? [13:31:38] * arturo probably doesn't understand the true meaning of either `aaS` or `cloud` [13:32:11] Tsk. I see we abandonned our old rule of refusing to use 'cloud'. Pity. [13:32:14] :-) [13:33:21] we could do better, concat buzzwords together, like `IA blockchain bigdata cloud for IoT` [13:33:39] I'm sure if you search for it, there are product with such names [13:33:52] Thanks again, gotta run. I honestly still find it amusing that for some projects I'm still the first person they contact re labs. Or whatever it's called nowadays. :-) [13:34:06] o/ [13:34:08] o/ [16:31:53] bd808: ok, cool. i'll take a look. thx [16:45:24] !log tools ran configurator to set the gridengine web queues to `rerun FALSE` T242397 [16:45:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:45:27] T242397: Make webservice grid jobs "non-rerunable" - https://phabricator.wikimedia.org/T242397 [20:17:57] !log codesearch legoktm@codesearch5:~$ sudo crontab -u codesearch -r [20:17:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Codesearch/SAL [21:44:26] !log tools.integraality Perform webservice restart for T242967 [21:44:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [22:58:22] !log wikidiff2-wmde-dev Added BryanDavis (self) as projectadmin for cleanup before project deletion (T236562) [22:58:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidiff2-wmde-dev/SAL [22:58:25] T236562: "wikidiff2-wmde-dev" Cloud VPS project jessie deprecation - https://phabricator.wikimedia.org/T236562 [23:09:29] !log wikidiff2-wmde-dev Deleting project (T236562) [23:09:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidiff2-wmde-dev/SAL [23:09:32] T236562: "wikidiff2-wmde-dev" Cloud VPS project jessie deprecation - https://phabricator.wikimedia.org/T236562 [23:13:52] !log toolsbeta updated toollabs-webservice to 0.58 for stretch to test things out [23:13:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [23:41:47] !log tools deployed toollabs-webservice 0.58 to everything that isn't a container [23:41:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [23:45:07] !log tools rebuilding docker containers to include new webservice version (0.58) [23:45:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [23:54:47] !log tools rebooting tools-docker-builder-06 because there are a couple running containers that don't want to die cleanly [23:54:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL