[01:20:09] 10Tools: tool extreg-wos not updating its table - https://phabricator.wikimedia.org/T174866#3575565 (10Legoktm) Thanks @Krenair and @SamanthaNguyen :) [01:35:17] (03PS1) 10Legoktm: Blacklist mediawiki/extensions/MediaWikiFarm [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/375546 [01:35:37] (03CR) 10Legoktm: [C: 032] Blacklist mediawiki/extensions/MediaWikiFarm [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/375546 (owner: 10Legoktm) [01:35:56] (03Merged) 10jenkins-bot: Blacklist mediawiki/extensions/MediaWikiFarm [labs/libraryupgrader] - 10https://gerrit.wikimedia.org/r/375546 (owner: 10Legoktm) [01:57:17] PROBLEM - Puppet errors on tools-exec-1420 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [02:22:06] PROBLEM - Puppet errors on tools-exec-1413 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [02:27:29] 10PAWS: Wikivoyage is missing from PAWS system user-config.py - https://phabricator.wikimedia.org/T174867#3575586 (10zhuyifei1999) [02:55:42] 10Data-Services, 10Quarry, 10DBA: CHAR_LENGTH does not return the character count - https://phabricator.wikimedia.org/T174543#3575602 (10zhuyifei1999) [03:02:04] RECOVERY - Puppet errors on tools-exec-1413 is OK: OK: Less than 1.00% above the threshold [0.0] [03:07:19] RECOVERY - Puppet errors on tools-exec-1420 is OK: OK: Less than 1.00% above the threshold [0.0] [03:39:38] PROBLEM - Puppet errors on tools-exec-1411 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [04:11:45] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1416 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [04:34:39] RECOVERY - Puppet errors on tools-exec-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [04:36:03] 10Tools: tool extreg-wos not updating its table - https://phabricator.wikimedia.org/T174866#3575619 (10MarcoAurelio) Thanks all. Maybe the title standards should be mentioned in some place for future reference, or the tool amended to detect that from subtasks of both bugs? Regards. [04:51:45] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [05:03:21] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1402 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [05:25:36] PROBLEM - Puppet errors on tools-exec-1411 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [05:28:31] 10Toolforge, 10Wikisource, 10Bengali-Sites: Update the "tesseract-ben" package on Toolforge for OCR on Bengali Wikisource - https://phabricator.wikimedia.org/T167566#3575708 (10Bodhisattwa) 05Open>03Resolved [05:38:22] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1402 is OK: OK: Less than 1.00% above the threshold [0.0] [06:05:40] RECOVERY - Puppet errors on tools-exec-1411 is OK: OK: Less than 1.00% above the threshold [0.0] [06:30:39] PROBLEM - Puppet errors on tools-exec-1406 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [06:50:24] PROBLEM - Puppet errors on tools-worker-1001 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [07:05:38] RECOVERY - Puppet errors on tools-exec-1406 is OK: OK: Less than 1.00% above the threshold [0.0] [07:30:25] RECOVERY - Puppet errors on tools-worker-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [09:12:43] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1416 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [09:47:45] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [11:10:22] Reedy: did you have any more ideas about https://phabricator.wikimedia.org/T174850 I'm trying to get wikistream.wmflabs.org working again [14:48:45] PROBLEM - Puppet errors on tools-webgrid-lighttpd-1416 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:23:44] RECOVERY - Puppet errors on tools-webgrid-lighttpd-1416 is OK: OK: Less than 1.00% above the threshold [0.0] [17:41:51] edsu: Still around? [18:01:59] 10Cloud-VPS: wikistream.wmflabs.org down - unable to ssh to ws-web - https://phabricator.wikimedia.org/T174850#3574927 (10madhuvishy) Looking at auth.log on ws-web, I saw a bunch of: ``` Sep 2 10:35:20 ws-web sshd[16317]: Connection from 10.68.17.232 port 52570 on 10.68.22.255 port 22 Sep 2 10:35:20 ws-web ss... [18:09:34] 10Cloud-VPS: wikistream.wmflabs.org down - unable to ssh to ws-web - https://phabricator.wikimedia.org/T174850#3574927 (10Krenair) I believe that auth-related service would likely be nslcd/nscd (I forget which). [18:09:58] madhuvishy, in the past I've fixed such problems by ensuring the correct LDAP config is in place and that the correct certificate authority is installed [18:10:05] then restarted the services I mentioned on the ticket [18:10:59] Krenair: yeah I checked ldap. Right nscd/nslcd makes sense [18:11:29] thanks :) [18:11:31] yw [19:23:04] PROBLEM - Puppet errors on tools-exec-1413 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [19:58:03] RECOVERY - Puppet errors on tools-exec-1413 is OK: OK: Less than 1.00% above the threshold [0.0]