[00:40:32] PROBLEM - Puppet errors on tools-exec-1423 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [01:10:34] PROBLEM - Puppet errors on tools-exec-1442 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [01:15:33] RECOVERY - Puppet errors on tools-exec-1423 is OK: OK: Less than 1.00% above the threshold [0.0] [01:25:56] 10Wikibugs: Request to add "wikibugs" bot to Africa Wikimedia Developers IRC channel - https://phabricator.wikimedia.org/T173555#3532771 (10D3r1ck01) [01:26:12] 10Wikibugs, 10Africa-Wikimedia-Developers: Request to add "wikibugs" bot to Africa Wikimedia Developers IRC channel - https://phabricator.wikimedia.org/T173555#3532784 (10D3r1ck01) [01:36:21] (03PS1) 10D3r1ck01: Add #wiki-dev-africa to channel list [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/372497 (https://phabricator.wikimedia.org/T173555) [01:50:33] RECOVERY - Puppet errors on tools-exec-1442 is OK: OK: Less than 1.00% above the threshold [0.0] [02:41:32] PROBLEM - Puppet errors on tools-exec-1442 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [02:55:04] PROBLEM - Puppet errors on tools-exec-1413 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [03:30:02] RECOVERY - Puppet errors on tools-exec-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [03:38:09] PROBLEM - Puppet errors on tools-exec-1407 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [03:46:34] RECOVERY - Puppet errors on tools-exec-1442 is OK: OK: Less than 1.00% above the threshold [0.0] [04:33:09] RECOVERY - Puppet errors on tools-exec-1407 is OK: OK: Less than 1.00% above the threshold [0.0] [06:32:52] PROBLEM - Puppet errors on tools-exec-1431 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [06:59:04] PROBLEM - Puppet errors on tools-exec-1439 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [07:07:52] RECOVERY - Puppet errors on tools-exec-1431 is OK: OK: Less than 1.00% above the threshold [0.0] [07:34:06] RECOVERY - Puppet errors on tools-exec-1439 is OK: OK: Less than 1.00% above the threshold [0.0] [08:12:46] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1416 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [08:47:46] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [09:43:46] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1416 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [10:13:46] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [10:44:35] (03CR) 10Legoktm: [C: 032] Add #wiki-dev-africa to channel list [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/372497 (https://phabricator.wikimedia.org/T173555) (owner: 10D3r1ck01) [10:44:58] (03Merged) 10jenkins-bot: Add #wiki-dev-africa to channel list [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/372497 (https://phabricator.wikimedia.org/T173555) (owner: 10D3r1ck01) [10:45:06] (03CR) 10jenkins-bot: Add #wiki-dev-africa to channel list [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/372497 (https://phabricator.wikimedia.org/T173555) (owner: 10D3r1ck01) [10:46:54] (03CR) 10D3r1ck01: "Thanks so much @legoktm for merging this PS. Now we have Wikibugs available in our channel right? Or will it have to restarted for the cha" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/372497 (https://phabricator.wikimedia.org/T173555) (owner: 10D3r1ck01) [10:52:58] !log tools.wikibugs Updated channels.yaml to: 86fb4210dbbd41ae5e55c8c5bae58d8454289941 Add #wiki-dev-africa to channel list [10:58:02] 10Wikibugs, 10Africa-Wikimedia-Developers, 10Patch-For-Review: Request to add "wikibugs" bot to Africa Wikimedia Developers IRC channel - https://phabricator.wikimedia.org/T173555#3533208 (10D3r1ck01) Thanks @Legoktm for merging this patch. Marking this as resolved. :) [10:58:23] (03CR) 10Legoktm: "The next time a task is touched that should be sent to that channel, wikibugs will join and notify the channel." [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/372497 (https://phabricator.wikimedia.org/T173555) (owner: 10D3r1ck01) [10:59:22] (03CR) 10D3r1ck01: "Exactly. That just happened as I commented on the ticket related to this PS now :). Thanks again. :)" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/372497 (https://phabricator.wikimedia.org/T173555) (owner: 10D3r1ck01) [10:59:48] 10Wikibugs, 10Africa-Wikimedia-Developers, 10Patch-For-Review: Request to add "wikibugs" bot to Africa Wikimedia Developers IRC channel - https://phabricator.wikimedia.org/T173555#3533210 (10D3r1ck01) 05Open>03Resolved [11:18:02] 10Tool-fatameh: Fails to return an object that exists - https://phabricator.wikimedia.org/T173567#3533224 (10Tarrow) [12:17:25] Are there any known issues connecting to tools-db via ssh port forwarding using python & oursql [13:29:59] Jogo-obb: afaik no [13:30:15] "via ssh port forwarding" part [13:51:08] I am able to use the connection with mysql client in terminal but oursql complains about 'SSL connection error: unknown error number' [14:05:36] Jogo-obb: mysql client over ssh port forwarding? [14:05:54] I don't know a lot about oursql though [14:08:46] zhuyifei1999_: Yes exactly, this works without any problem, also a local adminer (php) [14:12:03] dunno then [14:16:30] Hm, I also tried the same thing with another (commercial) Server and there it worked as expected [15:40:25] !help Hi ! I'm trying to start the webservice for http://tools.wmflabs.org/best-image/ but it looks like there is a lock somewhere [15:40:25] Lena: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [15:40:50] https://thepasteb.in/p/NxhVmjjz96KIN [15:41:44] and obviously when I try to stop gridengine it says "Your webservice is not running" [16:11:04] Lena: hello :) I'm looking in a sec [16:15:36] thanks madhuvishy ! [16:18:23] Lena: where you trying to port it to k8s? [16:18:52] I removed the service.manifest file which is what I think was causing it to keep saying that you are running your job on gridengine [16:19:06] https://www.irccloud.com/pastebin/aLhJTGmL/ [16:20:02] if you were trying to get to the shell and install requirements etc, you can stop the webservice now, and do that [16:20:37] thank you :) [16:21:01] Lena: np :) [16:21:27] Jogo-obb: did you try using the oursql client without the tunnel? [16:22:08] I know people have run into trouble with oursql in the past, I can't recall exactly what, but we recommend using pymysql [16:22:22] madhuvishy: You mean directly on labs host? [16:22:52] If so, yes it works without any problems [16:22:55] Jogo-obb: yeah, which host are you tunneling via? [16:24:46] madhuvishy: I used the command from https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#Connecting_to_the_database_replicas_from_your_own_computer [16:25:14] ssh -Nf -L 4744:tools-db:3306 jogo@tools-login.wmflabs.org [16:25:47] I see, can you log in to tools-login as your user and try to run your program? [16:27:18] That is no problem, it works as expected. My aim was to develop locally and work with the db from labs. [16:28:28] Jogo-obb: what are you running within the tools host to verify that it works? [16:29:00] (what command) [16:36:15] Using my tool account in an venv with python3: db = oursql.connect(host="tools-db", read_default_file=os.path.expanduser("~/replica.my.cnf"), db="s52395__bot" ) [16:37:09] After importing oursql and os, of course [16:50:18] madhuvishy: I'm lost, I followed https://wikitech.wikimedia.org/wiki/Help:Toolforge/My_first_Flask_OAuth_tool#Step_2:_Create_a_basic_Flask_WSGI_webservice and now I have https://thepasteb.in/p/g5hPZGw1yKqhr : is this a python 2.7 vs python 3. thing ? [16:55:09] Lena: what python version is your app [16:56:38] madhuvishy: I developped it with python2.7 -_-" gosh it's obvious [16:58:22] Lena: ah then you have to explicitly say python2 shell [17:01:13] https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#python2_.28Python2_.2B_Kubernetes.29 [17:01:39] The tutorial you are following assumes python3 [17:13:32] Jogo-obb: are you trying something similar with the ssh tunnel on your local too? It would be useful if you paste what exactly you are trying to run, since 'unknown error number' isn't very telling :) [17:19:37] madhuvishy: That meaningless error is exactly my problem ;) [17:20:14] Here is my local command: import oursql [17:20:14] db = oursql.connect(host="127.0.0.1", user="s52395" , passwd="...", port=4744, db="s52395__bot" ) [17:20:44] 10Tool-wikiloves: Bug on a page of http://tools.wmflabs.org/wikiloves - https://phabricator.wikimedia.org/T173583#3533845 (10VIGNERON) [17:22:23] PROBLEM - Puppet errors on tools-exec-1430 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [17:27:20] madhuvishy: Well OK, it was a bit hidden but I found out that I can also use pymysql with sqlalchemy. That seems to be working. [17:28:16] Jogo-obb: oh cool! [17:28:29] madhuvishy: Thanks for your assistance [17:28:51] np, I wish I knew why oursql is weird! [17:32:31] Well I remember I already had some problems installing it against the mariadb mysql-client lib. I think I had to patch something back then. [17:33:53] 10Cloud-Services, 10Operations, 10ops-eqiad, 10Patch-For-Review: rack/setup/install labstore100[67].wikimedia.org - https://phabricator.wikimedia.org/T167984#3352141 (10ayounsi) asw2-d-eqiad:ge-6/0/3 (Description: labstore1007, MAC: 30:e1:71:5f:9d:94 ) has been flapping at a rate of ~40 up/downs per hour.... [17:34:41] 10Tool-wikiloves: Bug on a page of http://tools.wmflabs.org/wikiloves - https://phabricator.wikimedia.org/T173583#3533845 (10JeanFred) It appears that all the 2016 db entries are missing both the `start` and `end` fields... This is definitely something broken in the data processing (`database.py`), not in the w... [17:35:12] I did not remarked PyMySQL until today. And SQLalchemy hides it on a subpage instead of the overall view page. [17:36:08] 10Tool-wikiloves: Bug on a page of http://tools.wmflabs.org/wikiloves - https://phabricator.wikimedia.org/T173583#3533898 (10JeanFred) On top of this, the `oursql` keeps failing at connecting to the Commons Labs DB... >_> [17:57:24] RECOVERY - Puppet errors on tools-exec-1430 is OK: OK: Less than 1.00% above the threshold [0.0] [18:20:00] Jogo-obb: if your oursql connect command was related to JeanFred's command ^: host="127.0.0.1" should probably be 'commonswiki.labsdb' or something along those lines [18:20:25] but I'm also responding in the middle of a conversation so I may just be spewing nonsense ;-) [18:31:15] PROBLEM - Puppet errors on tools-exec-1420 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [19:06:15] RECOVERY - Puppet errors on tools-exec-1420 is OK: OK: Less than 1.00% above the threshold [0.0] [20:10:55] valhallasw`cloud: 127.0.0.1 was indeed correct as I tried to connect from my laptop via ssh tunneling, but oursql has thrown meaningless ssl errors. As it works via PyMySQL I am satisfied for the moment. ;) [20:21:03] Jogo-obb: ahhhh! Ok, good to hear :-) [20:34:58] PROBLEM - Puppet errors on tools-worker-1020 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [21:14:57] RECOVERY - Puppet errors on tools-worker-1020 is OK: OK: Less than 1.00% above the threshold [0.0] [21:25:54] (03CR) 10Dzahn: [C: 031] "@Paladox got a wikibugs maintainer to deploy/restart/check on it?" [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/328663 (https://phabricator.wikimedia.org/T141089) (owner: 10Paladox) [21:28:37] (03Draft1) 10Paladox: Fix base::service [labs/icinga2] - 10https://gerrit.wikimedia.org/r/372571 [21:28:39] (03PS2) 10Paladox: Fix base::service [labs/icinga2] - 10https://gerrit.wikimedia.org/r/372571 [21:28:42] (03CR) 10Paladox: [V: 032 C: 032] Fix base::service [labs/icinga2] - 10https://gerrit.wikimedia.org/r/372571 (owner: 10Paladox) [21:31:12] legoktm hi, im wondering could you review / merge https://gerrit.wikimedia.org/r/328663 please? [21:35:26] redis2irc, i wonder what that does [21:58:04] harej: redis2irc is the component that connects to irc. wikibugs and grrrit are the components that connect to the corresponding services. Redis is used to connect all together [21:58:26] Redis is good for that [22:04:34] (03Abandoned) 10Paladox: Add grrrit-wm project to #wikimedia-labs channel [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/307441 (owner: 10Paladox) [22:12:17] (03Restored) 10Paladox: Add grrrit-wm project to #wikimedia-labs channel [labs/tools/wikibugs2] - 10https://gerrit.wikimedia.org/r/307441 (owner: 10Paladox) [22:49:09] Popped in to say was nice chatting with the team this summer... I think you folks are bees knees [22:49:50] s/bees knees/the bees knees/ [23:56:55] narwhal: <3 thank you