[00:06:05] 10Toolforge, 10Kubernetes: namespaces "wdq_checker" not found error when trying to start webservice shell - https://phabricator.wikimedia.org/T176027#3611596 (10bd808) ``` $ journalctl -u maintain-kubeusers --no-pager | grep wdq_checker Sep 15 17:42:55 tools-k8s-master-01 maintain-kubeusers[13111]: Wrote confi... [00:09:56] 10Toolforge, 10Developer-Relations, 10Developer-Wishlist (2017), 10Documentation: Run a documentation sprint for Cloud VPS and Toolforge - https://phabricator.wikimedia.org/T101659#3611599 (10bd808) [00:09:59] 10Toolforge, 10Tools-Kubernetes, 10Documentation, 10Kubernetes, 10User-bd808: Document basics of using a custom Kubernetes "Deployment" to operate a tool - https://phabricator.wikimedia.org/T165403#3611597 (10bd808) 05Open>03Resolved a:03bd808 [00:35:04] RECOVERY - Puppet errors on tools-exec-1439 is OK: OK: Less than 1.00% above the threshold [0.0] [01:20:59] PROBLEM - Puppet errors on tools-exec-1433 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [01:38:34] PROBLEM - Puppet errors on tools-exec-1423 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [01:56:22] !log suggestbot installed libmysqlclient-dev on suggestbot-prod [01:56:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Suggestbot/SAL [02:01:01] RECOVERY - Puppet errors on tools-exec-1433 is OK: OK: Less than 1.00% above the threshold [0.0] [02:23:33] RECOVERY - Puppet errors on tools-exec-1423 is OK: OK: Less than 1.00% above the threshold [0.0] [02:30:36] PROBLEM - Puppet errors on tools-exec-1411 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [02:48:09] !log suggestbot installed SciPy stack on suggestbot-prod [02:48:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Suggestbot/SAL [03:05:37] RECOVERY - Puppet errors on tools-exec-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [04:58:40] PROBLEM - Puppet errors on tools-exec-1440 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [05:38:40] RECOVERY - Puppet errors on tools-exec-1440 is OK: OK: Less than 1.00% above the threshold [0.0] [06:45:42] 10Toolforge, 10Kubernetes: namespaces "wdq_checker" not found error when trying to start webservice shell - https://phabricator.wikimedia.org/T176027#3611706 (10Frettie) it is possible to rename or delete and create new tool. But it is not correct, it will not repaired. [08:46:02] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1413 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [09:31:03] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [09:47:03] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1413 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [10:06:05] PROBLEM - Puppet errors on tools-exec-1439 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [10:17:03] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [10:36:05] RECOVERY - Puppet errors on tools-exec-1439 is OK: OK: Less than 1.00% above the threshold [0.0] [11:17:19] !support Database error on en.wikinews Spectial:Usermerge . Error: [Wb0IEApEE4AAAAK@xdIAAAAN] 2017-09-16 11:16:33: Fatal exception of type "Wikimedia\Rdbms\DBQueryError" [11:17:31] !help Database error on en.wikinews Spectial:Usermerge . Error: [Wb0IEApEE4AAAAK@xdIAAAAN] 2017-09-16 11:16:33: Fatal exception of type "Wikimedia\Rdbms\DBQueryError" [11:17:31] sau226: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [11:17:36] Ok [11:22:28] sau226: could you walk us through what caused the error? What's your local username on enwikinews, as SUL doesn't show an attached account there [11:22:52] ITs the beta cluster ThersNoTime [11:27:35] That's definitely something worth mentioning :) what time roughly did you get the error, have you tried again since? [11:38:41] sau226: last log entry was at 11:16:33 UTC, have you managed to get this working? [11:41:35] TheresNoTime Not yet [11:48:52] sau226: who are you trying to merge, it appears both accounts have the same page in their watchlists (`Error: 1062 Duplicate entry '175-0-If_you_are_a_owner_of' for key 'wl_user`) [11:49:19] Old user: Vikylia2029 New user: Sania2037 [11:49:32] Basically trying to make all the spamn edits into one user [11:49:39] So its easier for me to delete [11:52:18] Ah right, you won't be able to do that, as the current workaround is to get one of the two accounts to remove the offending page from the watchlist [11:52:26] T162563 related [11:52:26] T162563: Fatal error of type DBQueryError, with previous PHP Notice: Undefined offset: 1 - https://phabricator.wikimedia.org/T162563 [11:53:17] Oh [11:53:34] They're spam accounts [11:54:07] Can you try merging two other accounts (I imagine you have a fair few spam accounts to choose from?) [11:55:00] Successful [11:55:21] But not the 2 I'm metging [11:55:50] It is probably the same origin (large writes) but please report it separatelly [11:55:51] Is there an automate script to delete all pages made by a certain user? [11:56:08] as it will have to be solved separatelly [11:56:23] Hmm, you're not going to be able to merge those two I'm afraid, unless someone is willing to remove the page from 119/175's watchlist [11:56:40] (as in, on the db, which is unlikely) [11:57:10] Oh [11:57:26] Well I just suppose my job'll be harder [11:57:52] Is there any good scripts for mass deleting all their contributions that's relatively easy to set up? [11:58:48] Twinkle's D-batch might do it? [11:59:17] Is there a link? [11:59:22] Thanks [11:59:31] https://en.wikipedia.org/wiki/Wikipedia:Twinkle/doc#Modules_for_administrators [11:59:53] Thanks [12:02:21] Haven't got it on beta cluster [12:02:55] pretty sure Twinkle will work on beta [12:03:25] There is no checkbox on wikinews [12:10:18] Back [12:12:48] Is pywikibot good for de-cluttering a spam-ridden wiki (beta cluster)? [12:20:46] sau226: just seeing if Auto Wiki Browser works there [12:21:24] Ok [12:22:03] yup it does, that can read a list of user contribs and bulk delete pages [12:22:25] Why does my user get constantly logged out? [12:23:06] There seems to be a problem with your login session; this action has been canceled as a precaution against session hijacking. Go back to the previous page, reload that page and then try again. Every time I try [12:26:55] sau226: firefox? [12:28:13] Chrome [12:28:25] Is disconnect (disconnect.me) the problem? [12:29:36] hm.. if it's mucking around with cookies, perhaps [12:31:34] Anywho, are you happy using Auto Wiki Browser to bulk delete things, or would you like me to do it? [12:34:25] You can do so [12:34:38] https://en.wikinews.beta.wmflabs.org/wiki/Special:Contributions/Spam_Merge_User [12:34:50] That's a new account I'm merging spam users into [12:34:59] It removed all pages from watchlist [12:35:13] Awesome, are you able to assign sysop? [12:36:08] Yes [12:36:15] Username there is "Samtar" [12:37:23] Granted [12:37:40] Also I gave you flood flag (bot right only) as well as admin [12:37:57] Cool, I'll run AWB now then [13:03:28] 10Tools: Tool "ifttt-testing" loads assets from many sites, mixed http/https - https://phabricator.wikimedia.org/T172609#3611818 (10D3r1ck01) @zhuyifei1999, I have removed some of the external CDNs. Can you help me recheck and see what is left? The output should be like those previous one you posted in the task.... [13:07:57] Thanks Samtar [13:17:29] 10Cloud-Services: Tools.wmflabs small so small is down - https://phabricator.wikimedia.org/T176032#3611835 (10Reedy) [13:30:17] 10Cloud-Services: Tools.wmflabs small so small is down - https://phabricator.wikimedia.org/T176032#3611750 (10valhallasw) For which URL does this happen? [13:33:06] 10Cloud-Services: Tools.wmflabs small so small is down - https://phabricator.wikimedia.org/T176032#3611849 (10Zoranzoki21) >>! In T176032#3611847, @valhallasw wrote: > For which URL does this happen? One of examples is xtools. [13:34:46] 10Cloud-Services: Tools.wmflabs small so small is down - https://phabricator.wikimedia.org/T176032#3611854 (10valhallasw) Xtools is not hosted on tools.wmflabs.org, and the main xtools page https://xtools.wmflabs.org/articleinfo/? seems up for me. Please provide the exact URLs that are not working. [13:37:06] PROBLEM - Puppet errors on tools-exec-1439 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [13:37:27] 10Cloud-Services: Tools.wmflabs small so small is down - https://phabricator.wikimedia.org/T176032#3611857 (10Zoranzoki21) Current work.. But, sometimes more no work.. [13:38:07] 10Cloud-Services: Tools.wmflabs small so small is down - https://phabricator.wikimedia.org/T176032#3611858 (10Zoranzoki21) Examples: tools.wmflabs.org/listeria than commonshelper... [14:07:05] RECOVERY - Puppet errors on tools-exec-1439 is OK: OK: Less than 1.00% above the threshold [0.0] [14:19:26] 10wikitech.wikimedia.org, 10LDAP: Request rename of "Alangi derick" to "Alangi Derick" on wikitech/LDAP/Gerrit - https://phabricator.wikimedia.org/T171417#3611896 (10D3r1ck01) p:05Triage>03Normal [15:09:20] 10Cloud-Services, 10DBA: Prepare and check storage layer for amwikimedia - https://phabricator.wikimedia.org/T176043#3611967 (10Ladsgroup) [15:23:34] 10cloud-services-team (Kanban): Replace kernel and reboot labvirt1015, 1016, 1017, 1018 - https://phabricator.wikimedia.org/T176044#3611990 (10Andrew) [15:23:45] 10cloud-services-team (Kanban): Replace kernel and reboot labvirt1015, 1016, 1017, 1018 - https://phabricator.wikimedia.org/T176044#3612002 (10Andrew) p:05Triage>03High [15:50:28] 10cloud-services-team (Kanban), 10Patch-For-Review: Replace kernel and reboot labvirt1015, 1016, 1017, 1018 - https://phabricator.wikimedia.org/T176044#3612017 (10Andrew) I just upgraded labvirt1015 and 1017 to -93 and rebooted, and both lost network config just like we saw with -83. So something very bad is... [16:27:52] 10cloud-services-team (Kanban), 10Patch-For-Review: Replace kernel and reboot labvirt1015, 1016, 1017, 1018 - https://phabricator.wikimedia.org/T176044#3612063 (10Andrew) I re-imaged labvir1015 and 1017. They're now running 4.4.0-93-generic and rebooting fine. Do I know what just happened here? I do not. [16:39:14] 10Tools: Tool "ifttt-testing" loads assets from many sites, mixed http/https - https://phabricator.wikimedia.org/T172609#3612119 (10zhuyifei1999) Still loaded: ``` https://cdnjs.cloudflare.com/ajax/libs/materialize/0.96.1/css/materialize.min.css http://materialdesignblog.com/Material-Design-Flipper/material/css... [17:10:48] 10Tools: Tool "ifttt-testing" loads assets from many sites, mixed http/https - https://phabricator.wikimedia.org/T172609#3612159 (10D3r1ck01) Thank you @zhuyifei1999 for the recheck. [17:27:56] 10Toolforge: Tools.wmflabs small so small is down - https://phabricator.wikimedia.org/T176032#3612179 (10Framawiki) [17:36:00] 10Toolforge: Tools.wmflabs small so small is down - https://phabricator.wikimedia.org/T176032#3612205 (10Aklapper) I don't know what "small so small tools" are or where to see them. Right now, https://tools.wmflabs.org/listeria/ loads for me. For future reference, bug reports need clear steps to reproduce, ste... [17:36:19] 10Toolforge: Tools.wmflabs small so small sometimes (?) show 503 error - https://phabricator.wikimedia.org/T176032#3612206 (10Aklapper) [17:38:44] 10Toolforge, 10Kubernetes: namespaces "wdq_checker" not found error when trying to start webservice shell - https://phabricator.wikimedia.org/T176027#3612209 (10bd808) According to the Kubernetes source code, a Namespace must comply with the definition of a DNS Label: ``` // DNS_LABEL: This is a string, no mo... [17:39:58] 10Toolforge: Tools.wmflabs small so small sometimes (?) show 503 error - https://phabricator.wikimedia.org/T176032#3612210 (10Zoranzoki21) 05Open>03declined I will reopen this task if it happened again. [17:41:19] 10Toolforge, 10Kubernetes: namespaces "wdq_checker" not found error when trying to start webservice shell - https://phabricator.wikimedia.org/T176027#3612215 (10bd808) @Frettie in the near term, I think you should create a new tool account, possibly `wdq-checker` to run your code. [17:57:30] !Help. Sorry for a pretty trivial question. But I am facing issues while setting gerrit. [17:59:44] !ask | Mridu [17:59:44] PROBLEM - High iowait on tools-webgrid-lighttpd-1413 is CRITICAL: CRITICAL: tools.tools-webgrid-lighttpd-1413.cpu.total.iowait (>10.00%) [18:00:33] Mridu: no questions are trivial :) what's going on [18:09:41] RECOVERY - High iowait on tools-webgrid-lighttpd-1413 is OK: OK: All targets OK [18:22:35] PROBLEM - Puppet errors on tools-exec-1429 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [18:35:22] !help is it necessary to have wikitech and gerrit username and password same? [18:35:23] Sorry, you are not authorized to perform this [18:36:55] Yep [18:36:59] mridu ^^ [18:37:29] though git over http may have a different password in the future if it dosen't now. [18:38:10] I have setup gerrit's username and password as my github username and password :/ [18:38:58] Your username should be the same as wikitech, same for password. [18:39:24] Thanks. Shall figure out how to change. [18:39:55] You carn't [18:40:00] it's hooked up to wikitech [18:40:29] so requires it to be changed on both. [18:41:42] mridu: to clarify, wikitech and the gerrit web interface use the same backend for authentication, so they share usernames and passwords. Wikitech is where you can change the password. [18:43:06] mridu: to make things a bit confusing, there's a seperate (auto-generated) password for git-over-https, which can be found under https://gerrit.wikimedia.org/r/#/settings/http-password [18:43:14] So, all this has nothing to do with my github credentials? [18:43:52] correct, github is completely independent of any of this [18:43:56] I need to have my wikitech and gerrit credentials same.? [18:44:57] As of now I have my github and gerrit credentials same. Which are different from wikitech. How can this be undone? [18:45:21] mridu: if you change your gerrit password, your wikitech password will be changed too [18:45:49] as far as I know, you can't change your password in the Gerrit interface [18:46:46] what about the email id associated with it? Will have to change that as well. I think. [18:46:51] ah yes you may be right [18:47:15] you can add emails to gerrit. [18:47:23] i doint think those will be added to wikitech. [18:47:53] mridu: please use https://wikitech.wikimedia.org/wiki/Special:Preferences to change your password, you can also change email address there if you wish [18:48:02] the same password will be used for gerrit [18:48:25] none of this is tied to github [18:50:10] Thanks. Making the changes. [18:52:03] One more query before I make changes. While I set up username and id for git through command prompt. I will be using wikitech credentials right? [18:53:50] And if I have to push some different file on to github. It won't be working because credentials won't match. [18:54:10] mridu: no, see https://www.mediawiki.org/wiki/Gerrit/Tutorial#Set_Up_SSH_Keys_in_Gerrit [18:55:20] mridu: if you use both github and gerrit, they will be two different 'remotes' in git, and their authentication is completely seperate. [18:56:15] however, to keep everything simple, I would suggest using an SSH key for both -- just register the same public key in Gerrit and on github, and everything should work automatically. [18:57:06] Oh.. okay. Seems confusing. Would do it step by step. [18:57:32] It is confusing :-) [18:57:37] RECOVERY - Puppet errors on tools-exec-1429 is OK: OK: Less than 1.00% above the threshold [0.0] [18:57:59] It makes sense once you're used to it, but it takes some time getting used to the terminology and techno-logic. [19:04:40] Have changed preferences on wikitech. [19:07:18] Now, should I start from configure git step. And do git config --global user.email "example@example.com"(email being my wikitech email). [19:10:46] mridu: note that that email address will be public in the git commits [19:11:25] but yes, that's the correct first step [19:23:54] There is one step ssh -p 29418 @gerrit.wikimedia.org. What's 29418? [19:36:12] mridu: Gerrit's port number [19:37:55] yeah. I have replaced that with the pid I got. But now it shows connection timed out. [19:38:42] ssh -p 11717 Mridubhatnagar@gerrit.wikimedia.org this is what I used. [19:48:42] mridu: you should use 29418 -- it's the same for everyone [19:49:17] that does not work either. [19:49:57] what's the error message? [19:50:39] permission denied(public key) [19:51:09] ok, that's good because it means that the connection itself is working -- we're just dealing with a misconfiguration somewhere. [19:52:04] oh... okay. [19:53:09] what needs to be done next? [19:53:13] mridu: ah, the username should be 'mridub' (your shell name, not your login name) [19:53:49] oh.. no. [19:54:24] hey, thanks a lot. It works :) [19:57:29] cool! [20:08:38] 10Toolforge, 10Outreachy (Round-15): Outreachy - webservice microtask for Mridu_Bhatnagar - https://phabricator.wikimedia.org/T176018#3612320 (10Mridu_Bhatnagar) [20:11:08] 10Toolforge, 10Outreachy (Round-15): Outreachy - webservice microtask for Mridu_Bhatnagar - https://phabricator.wikimedia.org/T176018#3612325 (10Mridu_Bhatnagar) [20:18:17] 10wikitech.wikimedia.org: message error database - https://phabricator.wikimedia.org/T176056#3612334 (10Bnhassin) [20:24:29] 10Data-Services, 10cloud-services-team (Kanban): Designate api error when trying to create an A record in the web.db.svc.eqiad.wmflabs zone - https://phabricator.wikimedia.org/T176057#3612352 (10bd808) [20:44:41] 10wikitech.wikimedia.org: message error database - https://phabricator.wikimedia.org/T176056#3612422 (10Aklapper) Hi @Bnhassin, thanks for taking the time to report this! Unfortunately this report lacks any information. If you have time and can still reproduce the problem: Please [[ https://www.mediawiki.org/wik... [20:50:17] 10Data-Services, 10cloud-services-team (Kanban): Designate api error when trying to create an A record in the web.db.svc.eqiad.wmflabs zone - https://phabricator.wikimedia.org/T176057#3612429 (10bd808) Using the FQDN of `s1.web.db.svc.eqiad.wmflabs.` as the name succeeds. [20:53:08] 10wikitech.wikimedia.org: message error database - https://phabricator.wikimedia.org/T176056#3612432 (10Framawiki) a:05Bnhassin>03None [20:56:19] 10Data-Services, 10cloud-services-team (Kanban), 10Upstream: Designate api error when trying to create an A record in the web.db.svc.eqiad.wmflabs zone - https://phabricator.wikimedia.org/T176057#3612439 (10Krenair) bd808, did you try the FQDN for that record you had trouble creating? i.e... [20:57:01] 10Data-Services, 10cloud-services-team (Kanban), 10Upstream: Designate api error when trying to create an A record in the web.db.svc.eqiad.wmflabs zone - https://phabricator.wikimedia.org/T176057#3612441 (10Krenair) another job well done phabricator [20:58:20] 10Cloud-VPS, 10Upstream: Designate api error when trying to create an A record in the web.db.svc.eqiad.wmflabs zone - https://phabricator.wikimedia.org/T176057#3612442 (10bd808) p:05High>03Low [21:07:20] @bd808 @madhuvishy @Andrew I have completed the initial tasks. The next microtask is to understand what 'scripts/webservice' does... [21:08:16] Before going ahead with this one. Should I first go through kubernetes? How should I proceed. [21:08:51] I think you should just be able to read through the source code [21:09:43] this task is a small test of your Python knowledge and ability to understand code that others have written [21:12:04] 10Tool-Pageviews, 10Hindi-Sites: hi.wikiversity.org is not a valid project error on Pageviews Tool - https://phabricator.wikimedia.org/T176046#3612464 (10Niharika) [21:19:10] Sure. Will work on that. [21:47:10] 10Tool-Pageviews, 10Hindi-Sites: hi.wikiversity.org is not a valid project error on Pageviews Tool - https://phabricator.wikimedia.org/T176046#3612511 (10MusikAnimal) 05Open>03Resolved a:03MusikAnimal Sorry about that, the tool was out of sync with the Analytics whitelist. One day when I grow up I'll mak... [22:03:05] PROBLEM - Puppet errors on tools-exec-1439 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [22:38:03] RECOVERY - Puppet errors on tools-exec-1439 is OK: OK: Less than 1.00% above the threshold [0.0] [23:28:18] !log rcm Xenon: Running update [23:28:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Rcm/SAL [23:53:02] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1413 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]