[00:16:11] PROBLEM - High iowait on tools-exec-1417 is CRITICAL: CRITICAL: tools.tools-exec-1417.cpu.total.iowait (>25.00%) [00:31:12] RECOVERY - High iowait on tools-exec-1417 is OK: OK: All targets OK [00:34:10] PROBLEM - Puppet errors on tools-exec-1436 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [00:37:41] PROBLEM - Puppet errors on tools-exec-1417 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [00:58:45] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [01:09:09] RECOVERY - Puppet errors on tools-exec-1436 is OK: OK: Less than 1.00% above the threshold [0.0] [01:12:40] RECOVERY - Puppet errors on tools-exec-1417 is OK: OK: Less than 1.00% above the threshold [0.0] [01:14:44] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1416 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [01:44:48] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [03:45:08] PROBLEM - Puppet errors on tools-exec-1436 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [04:10:10] RECOVERY - Puppet errors on tools-exec-1436 is OK: OK: Less than 1.00% above the threshold [0.0] [06:34:18] PROBLEM - Puppet errors on tools-exec-1441 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [06:38:00] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1420 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [07:09:13] RECOVERY - Puppet errors on tools-exec-1441 is OK: OK: Less than 1.00% above the threshold [0.0] [07:18:03] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1420 is OK: OK: Less than 1.00% above the threshold [0.0] [07:19:44] 10VPS-project-libraryupgrader, 10Gerrit, 10Repository-Ownership-Requests: Grant libraryupgrader +2 rights for some library bumps - https://phabricator.wikimedia.org/T174760#3574757 (10Legoktm) >>! In T174760#3573778, @demon wrote: > But also....if we're using the API we're having to auth using the actual LDA... [07:40:09] (03PS1) 10Legoktm: Prompt for HTTP password instead of keeping it on disk [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/375500 [07:52:21] 10VPS-project-libraryupgrader, 10Gerrit, 10Repository-Ownership-Requests: Grant libraryupgrader +2 rights for some library bumps - https://phabricator.wikimedia.org/T174760#3574763 (10Paladox) If your using HTTP digest authentication, note that this was removed with gerrit 2.14. [07:55:11] (03PS1) 10Legoktm: Add a script to rotate the Gerrit HTTP password [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/375501 [07:57:14] 10VPS-project-libraryupgrader, 10Gerrit, 10Repository-Ownership-Requests: Grant libraryupgrader +2 rights for some library bumps - https://phabricator.wikimedia.org/T174760#3574764 (10Legoktm) >>! In T174760#3574763, @Paladox wrote: > If your using HTTP digest authentication, note that this was removed with... [08:28:52] (03PS1) 10Legoktm: Remove unused code [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/375503 [08:28:54] (03PS1) 10Legoktm: Set LANG=C.UTF-8 [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/375504 [08:28:56] (03PS1) 10Legoktm: Ignore logs/*.log [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/375505 [08:29:09] (03CR) 10Legoktm: [C: 032] Prompt for HTTP password instead of keeping it on disk [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/375500 (owner: 10Legoktm) [08:29:24] (03CR) 10Legoktm: [C: 032] "Good enough for now, may need to be rewritten after Gerrit 2.14 upgrade" [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/375501 (owner: 10Legoktm) [08:29:27] (03Merged) 10jenkins-bot: Prompt for HTTP password instead of keeping it on disk [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/375500 (owner: 10Legoktm) [08:29:29] (03CR) 10Legoktm: [C: 032] Remove unused code [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/375503 (owner: 10Legoktm) [08:29:32] (03CR) 10Legoktm: [C: 032] Set LANG=C.UTF-8 [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/375504 (owner: 10Legoktm) [08:29:41] (03CR) 10Legoktm: [C: 032] Ignore logs/*.log [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/375505 (owner: 10Legoktm) [08:29:43] (03Merged) 10jenkins-bot: Add a script to rotate the Gerrit HTTP password [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/375501 (owner: 10Legoktm) [08:29:48] (03Merged) 10jenkins-bot: Remove unused code [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/375503 (owner: 10Legoktm) [08:29:50] (03Merged) 10jenkins-bot: Set LANG=C.UTF-8 [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/375504 (owner: 10Legoktm) [08:29:59] (03Merged) 10jenkins-bot: Ignore logs/*.log [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/375505 (owner: 10Legoktm) [08:42:20] 10VPS-project-libraryupgrader, 10Gerrit, 10Repository-Ownership-Requests: Grant libraryupgrader +2 rights for some library bumps - https://phabricator.wikimedia.org/T174760#3574788 (10Paladox) @Legoktm see https://gerrit-review.googlesource.com/Documentation/rest-api.html#authentication (this is from the mas... [08:44:02] 10VPS-project-libraryupgrader, 10Gerrit, 10Repository-Ownership-Requests: Grant libraryupgrader +2 rights for some library bumps - https://phabricator.wikimedia.org/T174760#3574801 (10Legoktm) Ah they switched to HTTP Basic Auth. That'll be trivial to migrate to in the future. [08:46:05] 10VPS-project-libraryupgrader, 10Gerrit, 10Repository-Ownership-Requests: Grant libraryupgrader +2 rights for some library bumps - https://phabricator.wikimedia.org/T174760#3574802 (10Paladox) HTTP basic authentication is recommended now. [08:47:30] (03PS1) 10Legoktm: Escape / in password [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/375506 [08:47:40] (03CR) 10Legoktm: [C: 032] Escape / in password [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/375506 (owner: 10Legoktm) [08:47:58] (03Merged) 10jenkins-bot: Escape / in password [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/375506 (owner: 10Legoktm) [09:20:50] PROBLEM - Puppet errors on tools-exec-1402 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [09:30:38] PROBLEM - Puppet errors on tools-webgrid-generic-1404 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [09:55:50] RECOVERY - Puppet errors on tools-exec-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [10:10:41] RECOVERY - Puppet errors on tools-webgrid-generic-1404 is OK: OK: Less than 1.00% above the threshold [0.0] [10:51:19] !log wikispeech Deploy latest from Git master: 92396b4 (T172520), a9a7816, 2559238 (T165147), 070ad1f [10:51:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikispeech/SAL [10:51:24] T172520: Rename "WikispeechApi" as "ApiWikispeech" - https://phabricator.wikimedia.org/T172520 [10:51:25] T165147: Add a buffer animation to the play button while the sound is loading - https://phabricator.wikimedia.org/T165147 [11:00:16] 10Cloud-Services: Investigate NFS alternatives to the wikistream project - https://phabricator.wikimedia.org/T103148#3574912 (10edsu) I don't know if this is related but I'm unable to ssh from baston.wmflabs.org to ws-web any longer. I notice the service is currently down and wanted to see what was wrong. [11:23:52] 10Cloud-VPS: wikistream.wmflabs.org down - unable to ssh to ws-web - https://phabricator.wikimedia.org/T174850#3574943 (10Paladox) [12:15:21] 10Cloud-VPS: wikistream.wmflabs.org down - unable to ssh to ws-web - https://phabricator.wikimedia.org/T174850#3574927 (10Reedy) Are you using a different SSH key to what you were using previously? Add your current pubkey to https://wikitech.wikimedia.org/wiki/Special:Preferences#mw-prefsection-openstack ? [12:33:18] !log wikispeech Deploy latest from Git master: fe34de0 (T169483) [12:33:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikispeech/SAL [12:33:22] T169483: [Wikispeech] Investigate excluded code sniffs - https://phabricator.wikimedia.org/T169483 [12:47:57] 10Cloud-VPS: wikistream.wmflabs.org down - unable to ssh to ws-web - https://phabricator.wikimedia.org/T174850#3575008 (10edsu) I can see that the ssh-key for my laptop's id_rsa.pub and the id_rsa.pub in my homedir on bastion.wmflabs.org are present in that list. [12:50:51] 10Cloud-VPS: wikistream.wmflabs.org down - unable to ssh to ws-web - https://phabricator.wikimedia.org/T174850#3575010 (10edsu) I don't know if T103148 is related or not. Maybe my homedir on ws-web was somehow modified? [14:01:18] PROBLEM - Puppet errors on tools-redis-1001 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [14:32:23] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1402 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [14:41:20] RECOVERY - Puppet errors on tools-redis-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [15:07:22] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [16:14:49] 10Data-Services, 10cloud-services-team (Kanban), 10User-bd808: Define naming scheme for connecting to new wiki replica cluster - https://phabricator.wikimedia.org/T174860#3575183 (10bd808) [16:16:13] 10Data-Services, 10cloud-services-team (Kanban), 10User-bd808: Promote beta test of new Wiki Replica servers - https://phabricator.wikimedia.org/T172704#3506381 (10bd808) p:05Triage>03Normal [16:18:26] (03PS1) 10Lokal Profil: Investigate harvest of large wikidata dataset [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/375529 (https://phabricator.wikimedia.org/T172691) [16:19:34] (03Abandoned) 10Lokal Profil: Investigate harvest of large wikidata dataset [labs/tools/heritage] (wikidata) - 10https://gerrit.wikimedia.org/r/371744 (https://phabricator.wikimedia.org/T172691) (owner: 10Lokal Profil) [16:20:11] (03CR) 10Lokal Profil: "This is https://gerrit.wikimedia.org/r/#/c/371744/ but rebased onto the master branch and moved of the Wikidata branch" [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/375529 (https://phabricator.wikimedia.org/T172691) (owner: 10Lokal Profil) [18:22:45] PROBLEM - Puppet errors on tools-exec-1437 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [19:02:47] RECOVERY - Puppet errors on tools-exec-1437 is OK: OK: Less than 1.00% above the threshold [0.0] [19:23:44] PROBLEM - Puppet errors on tools-exec-1437 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [19:58:44] RECOVERY - Puppet errors on tools-exec-1437 is OK: OK: Less than 1.00% above the threshold [0.0] [20:02:26] (03PS1) 10Lokal Profil: [WIP] Add monuments_config for Australia [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/375537 [20:05:10] (03PS2) 10Lokal Profil: [WIP] Add monuments_config for Australia [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/375537 (https://phabricator.wikimedia.org/T174333) [20:22:37] (03PS1) 10Lokal Profil: [WIP] Add Peru in Spanish to the database [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/375539 (https://phabricator.wikimedia.org/T174245) [20:26:06] PROBLEM - Puppet errors on tools-exec-1413 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [20:37:59] (03PS2) 10Lokal Profil: Add Peru in Spanish to the database [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/375539 (https://phabricator.wikimedia.org/T174245) [20:39:06] 10Tools: tool extreg-wos not working - https://phabricator.wikimedia.org/T174866#3575374 (10MarcoAurelio) [20:44:31] 10Tools: tool extreg-wos not updating its table - https://phabricator.wikimedia.org/T174866#3575393 (10MarcoAurelio) [20:48:26] 10PAWS: Wikivoyage is missing - https://phabricator.wikimedia.org/T174867#3575394 (10Bugreporter) [21:00:22] 10Tools: tool extreg-wos not updating its table - https://phabricator.wikimedia.org/T174866#3575374 (10SamanthaNguyen) It updates, it's just that this tool is currently hardcoded to detect Maniphest task titles with `Convert {extension} to use extension registration`. See this block (also https://github.com/wiki... [21:01:03] RECOVERY - Puppet errors on tools-exec-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [21:10:19] 10Tools: tool extreg-wos not updating its table - https://phabricator.wikimedia.org/T174866#3575374 (10Krenair) A lot of tasks are like that - see the related objects on T98668 [21:14:28] !help Hi there, I was looking to set up my pywikibot to run on the cloud. So far, PAWS hasn't worked, I tried transferring the necessary files and now all I get is a HTTP 500 error. [21:14:29] kolbert: If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-team [21:19:34] kolbert, can you provide more details? [21:20:37] Sure [21:20:58] I have a list of articles and other parameters for replace.py [21:21:43] I tried uploading them to PAWs, but the page crashed and I haven't been able to reach the PAWs website for around a week [21:21:51] 10Tools: tool extreg-wos not updating its table - https://phabricator.wikimedia.org/T174866#3575425 (10Krenair) I changed the title on them. Let's see if it updates now. T173058 T171819 T174483 T174530 T174529 T174478 T174456 T174663 T174661 T174657 T174483 [21:21:53] while logged into my account, that is [21:21:58] if I'm logged out it works fine [21:22:17] what happens instead? [21:22:34] instead of loading? [21:23:17] yes [21:24:01] It "loads" for around 3-4 minutes, and when it gives up I just get an HTTP 500 message [21:24:06] er, [21:24:07] 504 [21:24:10] time-out [21:27:13] yeah that's gotta be a bug [21:27:33] I'm not familiar with PAWS unfortunately [21:29:42] 10PAWS: Wikivoyage is missing - https://phabricator.wikimedia.org/T174867#3575394 (10Krenair) also wikiboo**t**s instead of wikiboo**k**s? what about wikinews? [21:30:00] kolbert, hm but you have one [21:31:08] what do I have? :P [21:31:15] a task open [21:31:30] unfortunately I think the only maintainer of paws is not around much anymore [21:31:41] oh? [21:31:58] well I opened something and then after that [21:32:04] I wasn't able to access anything [21:32:15] hm? [21:32:55] I was troubleshooting with someone earlier, about a week ago and it was suggested to just paste the list of articles into the terminal [21:33:10] because uploading wasn't really working [21:33:37] after that, I wasn't able to load any pages on PAWs [21:46:07] yes chasemp thanks [22:14:37] (03CR) 10Jean-Frédéric: [C: 032] Add Peru in Spanish to the database [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/375539 (https://phabricator.wikimedia.org/T174245) (owner: 10Lokal Profil) [22:15:30] (03Merged) 10jenkins-bot: Add Peru in Spanish to the database [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/375539 (https://phabricator.wikimedia.org/T174245) (owner: 10Lokal Profil) [22:16:16] (03CR) 10jenkins-bot: Add Peru in Spanish to the database [labs/tools/heritage] - 10https://gerrit.wikimedia.org/r/375539 (https://phabricator.wikimedia.org/T174245) (owner: 10Lokal Profil) [22:19:20] 10Tools: tool extreg-wos not updating its table - https://phabricator.wikimedia.org/T174866#3575492 (10SamanthaNguyen) After an hour, they've been updated (2017-09-02 22:16:00.532121) - now "Patch to review" is shown for these, with the appropriate task linked. Not really sure what else needs to be done here(?)... [22:21:51] 10Tools: tool extreg-wos not updating its table - https://phabricator.wikimedia.org/T174866#3575498 (10SamanthaNguyen) 05Open>03Invalid Well, this task was created originally as a misunderstanding of what was going on, so I'm going to close this as invalid. Cheers 👍 [23:10:47] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1416 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [23:45:44] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0]