[01:12:10] !log tools.svgtranslate-test Updated TRUSTED_PROXIES=127.0.0.1,172.16.0.43,172.16.0.17 and cleared cache. [01:12:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.svgtranslate-test/SAL [01:14:27] !log deployment-prep importing a bunch of pages from production cswiki via importDump.php for T236823 [01:14:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [01:14:33] T236823: Newcomer tasks: create test pages on beta enwiki - https://phabricator.wikimedia.org/T236823 [01:16:25] ...or maybe not [01:16:36] anyone here familiar with importDump.php? [09:02:36] !log tools.integraality Deploy latest from Git master: 2d71154, d0d2937, 905007b, effee36, 7c15083 [09:02:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [09:06:06] !log tools.integraality Deploy latest from Git master: 46484f5 (T224226) [09:06:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [10:00:51] !log icinga downtime toolschecker for 1h for replacing SSL certs in tools-static and tools-k8s-master (T236962) [10:00:54] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Icinga/SAL [10:00:55] T236962: Migrate away from legacy star.tools.wmflabs.org certificate - https://phabricator.wikimedia.org/T236962 [10:01:38] !log icinga s/tools-static/tools-docker-registry/g in last SAL entry (T236962) [10:01:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Icinga/SAL [10:02:04] !log tools icinga downtime toolschecker for 1h for replacing SSL certs in tools-docker-registry and tools-k8s-master (T236962) [10:02:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:02:22] !log icinga please ignore last 2 SAL entries, wrong project [10:02:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Icinga/SAL [10:15:25] !log tools SSL cert replacement for tools-docker-registry and tools-k8s-master went fine apparently (T236962) [10:15:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:15:30] T236962: Migrate away from legacy star.tools.wmflabs.org certificate - https://phabricator.wikimedia.org/T236962 [11:01:01] !log admin icinga-downtimed cloudvirt1030 and cloudservices1003 for 1h due to PDU upgrade operations T227543 [11:01:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [11:01:04] T227543: b8-eqiad pdu refresh (Thursday 10/31 @11am UTC) - https://phabricator.wikimedia.org/T227543 [13:41:51] !log tools disabling puppet in tools-k8s-etcd- nodes to test https://gerrit.wikimedia.org/r/c/operations/puppet/+/546995 [13:41:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:59:30] !log tools update puppet prefix `tools-k8s-etcd-` to use the `role::wmcs::toolforge::k8s::etcd` T236826 [13:59:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:59:35] T236826: Toolforge: new k8s: initial build of the new kubernetes cluster - https://phabricator.wikimedia.org/T236826 [16:42:35] Could someone restart wikibugs please? it's not reporting tasks into the feed, though the gerrit function of the bot works. [17:04:06] !log tools.wikibugs Restarted bot at request of paladox via irc [17:04:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [17:23:19] bd808 thanks! [18:46:57] !log tools deleted and/or truncated a bunch of logfiles on tools-worker-1001. Runaway logfiles filled up the drive which prevented puppet from running. If puppet had run, it would have prevented the runaway logfiles. [18:47:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:47:26] andrewbogott: heh. the chicken ate all the eggs on that one I guess [18:47:46] I assume that was the k8s to sylog bug? [18:48:02] yep. As far as I can tell that's the only worker where that happened. [18:48:04] *syslog [19:20:20] !log tools.paws moved PAWS back to toolsdb and restarted the Hub for the change to take effect [19:20:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.paws/SAL [19:20:54] Hi guys, my tool's webserver is down and I can not make it restart again [19:21:00] Could someone help me? [19:21:54] hi Dvorapa [19:22:22] Dvorapa, what's your tool called? [19:22:27] Hi, tools.wmflabs.org/kmlexport is down [19:23:49] Dvorapa, what are you trying to make it restart? [19:23:58] everythng [19:24:39] can you give an example [19:24:42] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#Default_web_server_(lighttpd_+_PHP) [19:24:52] no grid jobs running, no Kubernetes pods, and no $HOME/service.manifest [19:25:15] + webservice --backend=gridengine generic start [19:25:32] yeah, I tried to remove service.manifest, but it didn't help [19:26:14] Dvorapa: so you are trying to get it running as a "normal" grid engine webservice? [19:26:31] I'm trying everything, nothing works [19:26:46] gridengine, kube, whatever will work [19:26:49] saying those words does not help us help you find the answer [19:26:58] error messages are good [19:27:05] and so are commands you have tried [19:28:06] ok. looking at $HOME/public_html this is a perl cgi webservice [19:28:13] yes [19:28:26] I would expect `webservice --backend=gridengine lighttpd start` to start it [19:29:05] Dvorapa: what error do you get when you run that command? [19:29:18] no error [19:29:36] just 503 [19:29:54] Dvorapa: and qstat returns something? [19:30:43] your $HOME/service.manifest is back, but in the state that means the webservice was stopped [19:31:00] or at least usually means that [19:32:14] !log tools.kmlexport Deleted 6.5G $HOME/public_html/core core dump file [19:32:16] qstat returns nothing [19:32:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.kmlexport/SAL [19:32:45] I'm trying to make it work at any cost during our conversation, every second down is bad [19:32:46] lighttpd is failing to start [19:32:57] $HOME/error.log shows why [19:33:00] "2019-10-31 19:30:19: (log.c.171) opening errorlog '~/access.log' failed: No such file or directory" [19:33:11] Yeah, I'm trying to make this line work [19:33:27] In https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web/Lighttpd [19:33:36] you suggested accesslog.filename = "{home}/access.log" [19:33:53] but it does not work and maybe this is the cause, why it is not working [19:34:11] I'm trying any value to this variable [19:34:13] so `touch access.log` and then try to restart? [19:34:17] to make it work [19:34:52] what does touch do? [19:35:07] It should literally be `accesslog.filename = "{home}/access.log"` in your $HOME/.lighttpd.conf. [19:35:09] creates an empty file Dvorapa [19:35:23] access.log is existing there [19:35:29] Ah, okay [19:35:37] I've been using it for a long time [19:36:25] Dvorapa: can I ask what you use the 4.2G of access.log data for? [19:36:58] you have a record of http requests going back to March 2017 [19:37:29] which is a lot of data to hang on to and a long to to do so if you do not have a clear purpose for the log data [19:37:42] *a long time to do so [19:41:56] Ok, it was caused by the accesslog.filename = "{home}/access.log" line [19:42:17] Any value of this variable is not working [19:43:07] Not sure what to do to make access.log work for my tool again, but not make the whole tool fail becuase of broken .lighttpd.conf [19:43:37] Yeah, these records are sometimes useful when fixing bugs [19:44:03] Dvorapa: hmm... that's not great. I know Hieu tested those instructions. I can do some more investigation [19:44:39] Any value of this variable makes the whole tool return 503 server error [19:44:49] Should I make a Phab task? [19:45:09] Dvorapa:yes please [19:45:35] the 503 is because lighttpd was failing to start which ended the grid job [19:47:00] I am confused why it was logging "2019-10-31 19:36:54: (log.c.171) opening errorlog '/project/kmlexport/access.log' failed: No such file or directory" [19:50:38] bd808: if it logged the message like you quote it, I don't think `/project` exists? [19:52:07] yeah. the path should have been /data/project/... [19:52:41] but of course I doesn't log the config file so I don't know what Dvorapa had actually added that made that error [19:53:17] the other confusion is that it says it was opening it as an errorlog file [19:54:03] which might be a bug in the logging, or it might be something about the config Dvorapa was trying to apply [19:57:02] okay, so only accesslog.filename = "/data/project/kmlexport/access.log" works currently [19:57:13] This should be mentioned in the manual [19:57:37] Dvorapa: so the full path and not the "{home}/access.log" notation? [19:57:42] yes [19:57:58] I made T237051 [19:57:59] T237051: accesslog.filename makes lighttpd fail starting - https://phabricator.wikimedia.org/T237051 [20:07:48] anyway, thank you guys for help [20:29:26] !log deployment-prep importing a bunch of pages from production cswiki via importDump.php for T236823 (for reals now) [20:29:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/SAL [20:29:33] T236823: Newcomer tasks: create test pages on beta enwiki - https://phabricator.wikimedia.org/T236823 [21:38:13] Has anybody ever got in contact with this one: https://en.wikipedia.org/wiki/BeeGFS [21:45:31] Wurgl, haven't looked into it deeply but it seems only the client is openly licensed [21:46:06] rest is under a custom EULA