[00:02:36] (03CR) 10Legoktm: "Yep" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246171 (https://phabricator.wikimedia.org/T114982) (owner: 10Dereckson) [00:03:34] (03CR) 10Eranroz: [C: 031] Configure default Echo subscriptions user options on he.wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246171 (https://phabricator.wikimedia.org/T114982) (owner: 10Dereckson) [00:03:38] (03CR) 10Catrope: "OK, scheduling for Monday's SWAT" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246171 (https://phabricator.wikimedia.org/T114982) (owner: 10Dereckson) [00:22:48] legoktm: https://gerrit.wikimedia.org/r/#/c/273545/2 I'd like to deploy that one soon too [00:23:34] ok, will look in a few [00:29:16] (03PS1) 10Dzahn: add running puppet to the deployment script [debs/wikistats] - 10https://gerrit.wikimedia.org/r/273552 [00:30:07] (03PS2) 10Dzahn: add running puppet to the deployment script [debs/wikistats] - 10https://gerrit.wikimedia.org/r/273552 [00:30:16] (03CR) 10Dzahn: [C: 032 V: 032] add running puppet to the deployment script [debs/wikistats] - 10https://gerrit.wikimedia.org/r/273552 (owner: 10Dzahn) [00:52:57] !log aaron@tin Synchronized php-1.27.0-wmf.14/includes/filebackend/FileBackendMultiWrite.php: 7ca624b8b93 (duration: 00m 47s) [01:16:45] (03PS1) 10BryanDavis: Camus: specify latest schema for ApiAction [puppet] - 10https://gerrit.wikimedia.org/r/273558 (https://phabricator.wikimedia.org/T108618) [01:18:11] (03PS1) 10BryanDavis: Logging: Add ApiRequest kafka logging [mediawiki-config] - 10https://gerrit.wikimedia.org/r/273559 (https://phabricator.wikimedia.org/T108618) [01:23:12] (03CR) 10BryanDavis: [C: 04-2] "Blocked on Oozie/Camus changes and topic setup." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/273559 (https://phabricator.wikimedia.org/T108618) (owner: 10BryanDavis) [01:50:57] !log updated phabricator/tools to https://phabricator.wikimedia.org/rPHTO1433a20886a4168d41b81b5ecf5e29687678f2e2 (applying fix for T127524 ) [01:50:58] T127524: Phabricator data dump hasn't run automatically since the Feb 17 2016 upgrade. - https://phabricator.wikimedia.org/T127524 [01:54:20] PROBLEM - Eqiad HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [1000.0] [01:56:38] PROBLEM - Text HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [1000.0] [02:14:10] RECOVERY - Text HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] [02:15:29] RECOVERY - Eqiad HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] [02:23:26] !log mwdeploy@tin sync-l10n completed (1.27.0-wmf.14) (duration: 10m 24s) [02:30:28] PROBLEM - puppet last run on mw2136 is CRITICAL: CRITICAL: puppet fail [02:31:10] !log l10nupdate@tin ResourceLoader cache refresh completed at Sat Feb 27 02:31:10 UTC 2016 (duration 7m 44s) [02:35:03] 6Operations, 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-GettingStarted: GettingStarted on Beta Cluster periodically loses its Redis index - https://phabricator.wikimedia.org/T100515#2068542 (10Mattflaschen) 5Resolved>3Open [02:58:19] RECOVERY - puppet last run on mw2136 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:58:48] PROBLEM - Kafka Broker Replica Max Lag on kafka1012 is CRITICAL: CRITICAL: 56.00% of data above the critical threshold [5000000.0] [03:05:20] PROBLEM - Last backup of the others filesystem on labstore1001 is CRITICAL: CRITICAL - Last run result for unit replicate-others was exit-code [03:12:49] RECOVERY - Kafka Broker Replica Max Lag on kafka1012 is OK: OK: Less than 50.00% above the threshold [1000000.0] [03:12:59] RECOVERY - cassandra-a CQL 10.64.32.187:9042 on restbase1008 is OK: TCP OK - 0.001 second response time on port 9042 [03:14:29] !log bootstrap of restbase1008-a.eqiad.wmnet complete; begining `nodetool cleanup' of 1003, 1004-a, and 1008-b : T95253 [03:14:30] T95253: Finish conversion to multiple Cassandra instances per hardware node - https://phabricator.wikimedia.org/T95253 [03:36:54] !log Backfilled missing missing https://wmflabs.org/sal/production data (2016-02-21T14:39 to 2016-02-24T16:53) [03:37:12] bot dead? [03:37:37] worked for me! [03:37:53] maybe i broke it [03:38:00] actually, i guess it didn't [03:38:04] stashbot did, but not the log bot [03:38:34] ya [03:38:53] I'll see if I can restart morebots [03:40:17] jobgrid thinks that it's running. maybe a netsplit? [03:40:21] morebots: you there? [03:40:22] I am a logbot running on tools-exec-1215. [03:40:22] Messages are logged to wikitech.wikimedia.org/wiki/Server_Admin_Log. [03:40:22] To log a message, type !log . [03:41:00] morebots: but we did, and you didn't [03:41:00] I am a logbot running on tools-exec-1215. [03:41:00] Messages are logged to wikitech.wikimedia.org/wiki/Server_Admin_Log. [03:41:00] To log a message, type !log . [03:41:08] something's not right [03:41:37] APIError: (u'internal_api_error_MWException', u"[6f0e2349] Exception Caught: CAS update failed on user_touched for user ID '55' (read from slave); the version of the user to be saved is older than the current version.", None) [03:41:47] blerg [03:42:13] Is ori still around? I think he figured out how to fix this for someone else [03:47:48] !log Tried to fix CAS error for Labslogbot [03:49:07] bd808: i see a problem here [03:50:11] ori's fix was to hack the temporarily source code apparently :/ [03:53:18] !log test [03:53:23] ffs [04:04:20] PROBLEM - Last backup of the maps filesystem on labstore1001 is CRITICAL: CRITICAL - Last run result for unit replicate-maps was exit-code [04:09:20] bd808: hey [04:09:29] hey ori [04:09:58] that CAS bug thing is keeping morebots from writing to wikitech [04:10:07] https://phabricator.wikimedia.org/T128246 [04:10:32] anomie had a theory about how that could be related to my change [04:10:56] I'll look at that. But in the meantime I'll simply comment it out again. [04:11:19] cool beans [04:11:38] I was about to live hack it on wikitech but that seemed yucky [04:12:53] It is a little yucky, but backing it out the change properly would mean a deployment to all app servers, which is riskier. And anomie could have been wrong; _joe_ said he started encountering the bug before I deployed my change. [04:14:17] We could downgrade it back down to a warning (by reverting https://gerrit.wikimedia.org/r/#/c/224030/), but that would be a shame if all it did was mask a bug. [04:14:22] does wikitech do something funky for the CAS stuff? [04:14:40] No idea. [04:15:06] wikitech is a funky wiki still even though we have it riding the train; I know that much [04:15:30] !log On silver: commented out exception from I3eae28a58 to work around T128246 [04:15:31] T128246: Logins failing on wikitech - https://phabricator.wikimedia.org/T128246 [04:15:34] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [04:15:40] bam! [04:15:52] it has a local memcached instead of nutcracker, I think [04:16:22] another thing to track down is why each login tries to save settings I suppose [04:18:53] I guess we should try to track the problem down before the next scap or it will just come back again [04:19:06] E_TOOMANYBUGS [04:38:52] 6Operations, 10MediaWiki-Interface, 10Traffic, 5MW-1.27-release, and 4 others: Incorrect TOC and section edit links rendering in Vector due to ParserCache corruption via ParserOutput::setText( ParserOutput::getText() ) - https://phabricator.wikimedia.org/T124356#2068596 (10Majr) [05:17:09] PROBLEM - puppet last run on mw2105 is CRITICAL: CRITICAL: puppet fail [05:45:00] RECOVERY - puppet last run on mw2105 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [06:29:39] PROBLEM - puppet last run on db1067 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:29] PROBLEM - puppet last run on cp2001 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:49] PROBLEM - puppet last run on mw1158 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:28] PROBLEM - puppet last run on mw2207 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:38] PROBLEM - puppet last run on mw2018 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:49] PROBLEM - puppet last run on cp3048 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:19] PROBLEM - puppet last run on mw1251 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:48] PROBLEM - puppet last run on mw2045 is CRITICAL: CRITICAL: Puppet has 1 failures [06:33:10] PROBLEM - puppet last run on mw1085 is CRITICAL: CRITICAL: Puppet has 1 failures [06:33:18] PROBLEM - puppet last run on wtp2019 is CRITICAL: CRITICAL: Puppet has 1 failures [06:33:39] PROBLEM - puppet last run on hydrogen is CRITICAL: CRITICAL: Puppet has 1 failures [06:33:59] PROBLEM - puppet last run on stat1003 is CRITICAL: CRITICAL: Puppet has 1 failures [06:34:49] PROBLEM - puppet last run on mw1177 is CRITICAL: CRITICAL: Puppet has 1 failures [06:34:59] PROBLEM - puppet last run on mw1073 is CRITICAL: CRITICAL: Puppet has 1 failures [06:35:00] PROBLEM - puppet last run on mw1102 is CRITICAL: CRITICAL: Puppet has 1 failures [06:35:00] PROBLEM - puppet last run on mw1058 is CRITICAL: CRITICAL: Puppet has 1 failures [06:35:09] PROBLEM - puppet last run on mw2052 is CRITICAL: CRITICAL: Puppet has 1 failures [06:35:18] PROBLEM - puppet last run on mw1039 is CRITICAL: CRITICAL: Puppet has 1 failures [06:35:30] PROBLEM - puppet last run on mw2077 is CRITICAL: CRITICAL: Puppet has 1 failures [06:35:39] PROBLEM - puppet last run on mw1157 is CRITICAL: CRITICAL: Puppet has 1 failures [06:56:39] RECOVERY - puppet last run on cp2001 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [06:57:09] RECOVERY - puppet last run on mw2045 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [06:57:18] RECOVERY - puppet last run on mw1177 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [06:57:30] RECOVERY - puppet last run on db1067 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:57:39] RECOVERY - puppet last run on mw2207 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:57:48] RECOVERY - puppet last run on mw1039 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [06:57:49] RECOVERY - puppet last run on mw2018 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [06:57:58] RECOVERY - puppet last run on hydrogen is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [06:58:00] RECOVERY - puppet last run on mw2077 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [06:58:08] RECOVERY - puppet last run on cp3048 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:58:28] RECOVERY - puppet last run on mw1251 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:58:40] RECOVERY - puppet last run on mw1158 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:59:10] RECOVERY - puppet last run on mw1073 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:59:18] RECOVERY - puppet last run on mw1102 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:59:18] RECOVERY - puppet last run on mw1058 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:59:19] RECOVERY - puppet last run on mw1085 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [06:59:20] RECOVERY - puppet last run on wtp2019 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:59:28] RECOVERY - puppet last run on mw2052 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:59:50] RECOVERY - puppet last run on mw1157 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:59:58] RECOVERY - puppet last run on stat1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:07:17] 6Operations, 10Phabricator, 6Project-Admins: Create policy projects and convert people projects to open - https://phabricator.wikimedia.org/T90491#2069137 (10Luke081515) [13:38:46] (03CR) 10Tim Landscheidt: [C: 04-1] "(Depends on the dependencies being deployed.)" [puppet] - 10https://gerrit.wikimedia.org/r/268347 (owner: 10Tim Landscheidt) [13:43:41] (03PS4) 10Tim Landscheidt: shinken: Only regenerate configuration when there are changes [puppet] - 10https://gerrit.wikimedia.org/r/267423 [13:44:29] (03PS3) 10Tim Landscheidt: Tools: Unpuppetize host_aliases [puppet] - 10https://gerrit.wikimedia.org/r/241582 (https://phabricator.wikimedia.org/T109485) [15:23:29] PROBLEM - puppet last run on cp4017 is CRITICAL: CRITICAL: puppet fail [15:49:59] RECOVERY - puppet last run on cp4017 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures [16:00:12] (03CR) 10Tim Landscheidt: "@Mark: You wrote: "[…] I'm just going to merge it", but only +2ed this change, but did not submit it. I assume this was a mistake, so ple" [software] - 10https://gerrit.wikimedia.org/r/141473 (owner: 10ArielGlenn) [16:06:18] (03PS1) 10MarcoAurelio: Maintenance on throttle.php [mediawiki-config] - 10https://gerrit.wikimedia.org/r/273580 [16:21:38] PROBLEM - Kafka Broker Replica Max Lag on kafka1013 is CRITICAL: CRITICAL: 58.33% of data above the critical threshold [5000000.0] [16:32:19] RECOVERY - Kafka Broker Replica Max Lag on kafka1013 is OK: OK: Less than 50.00% above the threshold [1000000.0] [17:21:30] PROBLEM - Disk space on cp3014 is CRITICAL: DISK CRITICAL - free space: /var/lib/nginx 2 MB (0% inode=99%) [17:23:09] 6Operations, 10Traffic, 7HTTPS, 7Security-Other: Server certificate is classified as invalid on government computers - https://phabricator.wikimedia.org/T128182#2069412 (10Florian) I got an answer from one of the people who wrote to us. It seems that the problem is limited (the user asked some other collea... [17:23:14] (03PS12) 10ArielGlenn: data retention audit script for logs, /root and /home dirs [software] - 10https://gerrit.wikimedia.org/r/141473 [17:24:41] (03CR) 10ArielGlenn: [C: 032 V: 032] data retention audit script for logs, /root and /home dirs [software] - 10https://gerrit.wikimedia.org/r/141473 (owner: 10ArielGlenn) [17:25:00] RECOVERY - Disk space on cp3014 is OK: DISK OK [17:34:24] 6Operations, 10Phabricator, 6Project-Admins: Create policy projects and convert people projects to open - https://phabricator.wikimedia.org/T90491#2069430 (10greg) [18:02:50] PROBLEM - puppet last run on mw1119 is CRITICAL: CRITICAL: Puppet has 64 failures [18:07:46] (03PS2) 10ArielGlenn: update code for collection of minions with given grain [software] - 10https://gerrit.wikimedia.org/r/233445 [18:10:40] (03CR) 10ArielGlenn: [C: 032 V: 032] update code for collection of minions with given grain [software] - 10https://gerrit.wikimedia.org/r/233445 (owner: 10ArielGlenn) [18:11:32] (03PS2) 10ArielGlenn: check mysqlconf for log retention and skip explicit logfiles check [software] - 10https://gerrit.wikimedia.org/r/233446 [18:12:14] (03CR) 10ArielGlenn: [C: 032 V: 032] check mysqlconf for log retention and skip explicit logfiles check [software] - 10https://gerrit.wikimedia.org/r/233446 (owner: 10ArielGlenn) [18:12:29] (03PS2) 10ArielGlenn: allow for more than one mysql conf file, audit all of /a [software] - 10https://gerrit.wikimedia.org/r/233447 [18:14:37] (03CR) 10Tim Landscheidt: "@ArielGlenn: Thanks!" [software] - 10https://gerrit.wikimedia.org/r/141473 (owner: 10ArielGlenn) [18:14:55] (03CR) 10ArielGlenn: [C: 032 V: 032] allow for more than one mysql conf file, audit all of /a [software] - 10https://gerrit.wikimedia.org/r/233447 (owner: 10ArielGlenn) [18:15:10] (03PS2) 10ArielGlenn: break some files out into modules, add rulestore util [software] - 10https://gerrit.wikimedia.org/r/233448 [18:15:49] PROBLEM - puppet last run on mw2099 is CRITICAL: CRITICAL: Puppet has 1 failures [18:22:41] (03CR) 10ArielGlenn: [C: 032 V: 032] break some files out into modules, add rulestore util [software] - 10https://gerrit.wikimedia.org/r/233448 (owner: 10ArielGlenn) [18:23:01] (03PS2) 10ArielGlenn: turn retention modules into a package [software] - 10https://gerrit.wikimedia.org/r/233449 [18:24:51] (03CR) 10ArielGlenn: [C: 032 V: 032] turn retention modules into a package [software] - 10https://gerrit.wikimedia.org/r/233449 (owner: 10ArielGlenn) [18:25:35] (03PS2) 10ArielGlenn: remove dup copy of Runner class; read confs from disk, not stdin [software] - 10https://gerrit.wikimedia.org/r/233450 [18:28:49] PROBLEM - Apache HTTP on mw1119 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:28:49] PROBLEM - HHVM rendering on mw1119 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:29:38] PROBLEM - dhclient process on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [18:29:38] PROBLEM - DPKG on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [18:29:58] PROBLEM - HHVM processes on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [18:30:09] PROBLEM - configured eth on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [18:30:09] PROBLEM - salt-minion processes on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [18:30:29] PROBLEM - SSH on mw1119 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:30:29] PROBLEM - nutcracker process on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [18:30:40] PROBLEM - nutcracker port on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [18:30:58] PROBLEM - Check size of conntrack table on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [18:30:59] PROBLEM - Disk space on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [18:31:09] PROBLEM - RAID on mw1119 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [18:31:39] (03CR) 10ArielGlenn: [C: 032 V: 032] remove dup copy of Runner class; read confs from disk, not stdin [software] - 10https://gerrit.wikimedia.org/r/233450 (owner: 10ArielGlenn) [18:31:58] (03PS2) 10ArielGlenn: write config files to subdir in the salt file_root [software] - 10https://gerrit.wikimedia.org/r/233451 [18:32:16] * mafk wonders if there's V: +1 [18:34:36] bleh, constant loss of session data on save [18:35:59] anomie ^ [18:36:00] ? [18:36:24] (03CR) 10ArielGlenn: [C: 032 V: 032] write config files to subdir in the salt file_root [software] - 10https://gerrit.wikimedia.org/r/233451 (owner: 10ArielGlenn) [18:36:36] (03PS2) 10ArielGlenn: use cp.get_file to retrieve configs [software] - 10https://gerrit.wikimedia.org/r/233452 [18:37:22] (03CR) 10ArielGlenn: [C: 032 V: 032] use cp.get_file to retrieve configs [software] - 10https://gerrit.wikimedia.org/r/233452 (owner: 10ArielGlenn) [18:37:37] (03PS2) 10ArielGlenn: turn files auditor into salt module [software] - 10https://gerrit.wikimedia.org/r/233453 [18:40:27] (03CR) 10ArielGlenn: [C: 032 V: 032] turn files auditor into salt module [software] - 10https://gerrit.wikimedia.org/r/233453 (owner: 10ArielGlenn) [18:40:38] RECOVERY - HHVM processes on mw1119 is OK: PROCS OK: 6 processes with command name hhvm [18:40:40] RECOVERY - salt-minion processes on mw1119 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion [18:40:48] RECOVERY - configured eth on mw1119 is OK: OK - interfaces up [18:40:59] RECOVERY - SSH on mw1119 is OK: SSH OK - OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.6 (protocol 2.0) [18:41:08] RECOVERY - nutcracker process on mw1119 is OK: PROCS OK: 1 process with UID = 108 (nutcracker), command name nutcracker [18:41:18] RECOVERY - nutcracker port on mw1119 is OK: TCP OK - 0.000 second response time on port 11212 [18:41:28] RECOVERY - Check size of conntrack table on mw1119 is OK: OK: nf_conntrack is 0 % full [18:41:29] RECOVERY - Disk space on mw1119 is OK: DISK OK [18:41:39] RECOVERY - RAID on mw1119 is OK: OK: no RAID installed [18:42:01] RECOVERY - dhclient process on mw1119 is OK: PROCS OK: 0 processes with command name dhclient [18:42:02] RECOVERY - DPKG on mw1119 is OK: All packages OK [18:42:28] RECOVERY - puppet last run on mw2099 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [18:42:43] mafk: yes, there is [18:43:11] mafk: e.g. 18.34 < grrrit-wm> (CR) Nikerabbit: [C: 2 V: 1] Validate the translation titles against abusefilter in target wiki [extensions/ContentTranslation] - https://gerrit.wikimedia.org/r/269379 (https://phabricator.wikimedia.org/T123909) (owner: Santhosh) [18:43:12] Nemo_bis: I never saw it, just verified-1, or verified+2 [18:43:30] RECOVERY - puppet last run on mw1119 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:43:38] nice :) [18:45:05] (03PS2) 10ArielGlenn: logs audit via salt module [software] - 10https://gerrit.wikimedia.org/r/233454 [18:45:26] (03CR) 10ArielGlenn: [C: 032 V: 032] logs audit via salt module [software] - 10https://gerrit.wikimedia.org/r/233454 (owner: 10ArielGlenn) [18:45:36] (03PS2) 10ArielGlenn: homes audit done via salt module [software] - 10https://gerrit.wikimedia.org/r/233455 [18:45:45] (03CR) 10ArielGlenn: [C: 032 V: 032] homes audit done via salt module [software] - 10https://gerrit.wikimedia.org/r/233455 (owner: 10ArielGlenn) [18:46:00] (03PS2) 10ArielGlenn: remove now unused auditor.py, move static methods into module [software] - 10https://gerrit.wikimedia.org/r/233456 [18:46:16] (03CR) 10ArielGlenn: [C: 032 V: 032] remove now unused auditor.py, move static methods into module [software] - 10https://gerrit.wikimedia.org/r/233456 (owner: 10ArielGlenn) [18:53:09] (03CR) 10ArielGlenn: "Going to push those other yer old patches through today and tommorow, a bunch of them just went now.Thanks for the ping." [software] - 10https://gerrit.wikimedia.org/r/141473 (owner: 10ArielGlenn) [18:58:06] (03PS2) 10ArielGlenn: move rule static methods out to a seperate utils file [software] - 10https://gerrit.wikimedia.org/r/233457 [18:58:16] (03CR) 10ArielGlenn: [C: 032 V: 032] move rule static methods out to a seperate utils file [software] - 10https://gerrit.wikimedia.org/r/233457 (owner: 10ArielGlenn) [18:58:30] (03PS2) 10ArielGlenn: get rid of executor and related vars for most files [software] - 10https://gerrit.wikimedia.org/r/233458 [18:58:37] (03CR) 10ArielGlenn: [C: 032 V: 032] get rid of executor and related vars for most files [software] - 10https://gerrit.wikimedia.org/r/233458 (owner: 10ArielGlenn) [18:58:54] (03PS2) 10ArielGlenn: file examiner: use salt module for remote cmd [software] - 10https://gerrit.wikimedia.org/r/233459 [18:59:03] (03CR) 10ArielGlenn: [C: 032 V: 032] file examiner: use salt module for remote cmd [software] - 10https://gerrit.wikimedia.org/r/233459 (owner: 10ArielGlenn) [18:59:16] (03PS2) 10ArielGlenn: convert dir examiner to use salt module [software] - 10https://gerrit.wikimedia.org/r/233460 [18:59:24] (03CR) 10ArielGlenn: [C: 032 V: 032] convert dir examiner to use salt module [software] - 10https://gerrit.wikimedia.org/r/233460 (owner: 10ArielGlenn) [18:59:36] (03PS2) 10ArielGlenn: last 'executor' code removed, all salt module now [software] - 10https://gerrit.wikimedia.org/r/233461 [18:59:43] (03CR) 10ArielGlenn: [C: 032 V: 032] last 'executor' code removed, all salt module now [software] - 10https://gerrit.wikimedia.org/r/233461 (owner: 10ArielGlenn) [19:32:47] !log ori@tin Synchronized php-1.27.0-wmf.14/includes/user/User.php: Revert I43cde3a48: Prevent duplicate memcached lookups for user record (T128246) (duration: 00m 56s) [19:32:48] T128246: Logins failing on wikitech - https://phabricator.wikimedia.org/T128246 [19:32:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [20:08:11] !log krinkle@tin Synchronized php-1.27.0-wmf.14/includes/mime.types: Iec9459c921: Unbreak ULS fonts via wmfstatic (duration: 00m 41s) [20:08:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [20:19:08] (03PS2) 10ArielGlenn: move cli static methods out to separate file [software] - 10https://gerrit.wikimedia.org/r/233462 [20:19:18] (03CR) 10ArielGlenn: [C: 032 V: 032] move cli static methods out to separate file [software] - 10https://gerrit.wikimedia.org/r/233462 (owner: 10ArielGlenn) [20:19:30] (03PS2) 10ArielGlenn: move readline completion and ignored entry lists out to separate files [software] - 10https://gerrit.wikimedia.org/r/233463 [20:19:37] (03CR) 10ArielGlenn: [C: 032 V: 032] move readline completion and ignored entry lists out to separate files [software] - 10https://gerrit.wikimedia.org/r/233463 (owner: 10ArielGlenn) [20:19:52] (03PS2) 10ArielGlenn: cleanup unused function, unneeded special class instantiations [software] - 10https://gerrit.wikimedia.org/r/233464 [20:20:02] (03CR) 10ArielGlenn: [C: 032 V: 032] cleanup unused function, unneeded special class instantiations [software] - 10https://gerrit.wikimedia.org/r/233464 (owner: 10ArielGlenn) [20:20:17] (03PS2) 10ArielGlenn: clean up ignore list code [software] - 10https://gerrit.wikimedia.org/r/233465 [20:20:25] (03CR) 10ArielGlenn: [C: 032 V: 032] clean up ignore list code [software] - 10https://gerrit.wikimedia.org/r/233465 (owner: 10ArielGlenn) [20:26:26] 6Operations, 10media-storage: Unable to delete, move or upload new versions of files on several wikis ("inconsistent state within the internal storage backends") - https://phabricator.wikimedia.org/T128096#2063784 (10Aklapper) [21:02:39] (03PS2) 10ArielGlenn: use yaml instead of python for global config, ignore list [software] - 10https://gerrit.wikimedia.org/r/233466 [21:02:54] (03CR) 10ArielGlenn: [C: 032 V: 032] use yaml instead of python for global config, ignore list [software] - 10https://gerrit.wikimedia.org/r/233466 (owner: 10ArielGlenn) [21:03:20] (03PS2) 10ArielGlenn: don't modify sys.path, convert remaining execs into local and remote [software] - 10https://gerrit.wikimedia.org/r/233467 [21:03:31] (03CR) 10ArielGlenn: [C: 032 V: 032] don't modify sys.path, convert remaining execs into local and remote [software] - 10https://gerrit.wikimedia.org/r/233467 (owner: 10ArielGlenn) [21:05:12] (03PS2) 10ArielGlenn: clean up names and paths, remove last runpy calls, use more yaml [software] - 10https://gerrit.wikimedia.org/r/233468 [21:05:37] (03CR) 10ArielGlenn: [C: 032 V: 032] clean up names and paths, remove last runpy calls, use more yaml [software] - 10https://gerrit.wikimedia.org/r/233468 (owner: 10ArielGlenn) [21:19:26] !log issuing test repair on cerium (restbase staging), wp_parsoid html keyspace, (-w 5) : T108611 [21:19:26] T108611: perform initial (manual) repair of Cassandra cluster - https://phabricator.wikimedia.org/T108611 [21:19:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [21:22:50] ori, I suspect someone is trying to catch your attention at https://phabricator.wikimedia.org/T87384#2069731 . [21:29:16] 6Operations, 7Easy: server admin log should include year in date (again) - https://phabricator.wikimedia.org/T85803#2069753 (10Aklapper) a:5Elee>3None @Elee: I am resetting the assignee of this task because there have been no signs of progress lately (please correct me if I'm wrong). Resetting the assigne... [22:00:18] (03PS2) 10ArielGlenn: enable directory prompt in cli in a few places in the menu [software] - 10https://gerrit.wikimedia.org/r/233469 [22:00:26] (03CR) 10ArielGlenn: [C: 032 V: 032] enable directory prompt in cli in a few places in the menu [software] - 10https://gerrit.wikimedia.org/r/233469 (owner: 10ArielGlenn) [22:08:11] (03PS2) 10ArielGlenn: some pylint fixes, mainly var names and superfluous imports [software] - 10https://gerrit.wikimedia.org/r/233470 [22:08:11] (03CR) 10ArielGlenn: [C: 032 V: 032] some pylint fixes, mainly var names and superfluous imports [software] - 10https://gerrit.wikimedia.org/r/233470 (owner: 10ArielGlenn) [22:08:11] 6Operations, 7Easy: server admin log should include year in date (again) - https://phabricator.wikimedia.org/T85803#2069842 (10valhallasw) 5Open>3Resolved a:3valhallasw The SAL has been using YYYY-MM-DD for a while now, so I think we can consider this resolved/. [22:08:11] (03PS2) 10ArielGlenn: remove old audit_files.py, no longer needed, remove unused Rule class remove unused timeout from local audit methods remove some dead attrs and args, clean up handling of ignore dicts [software] - 10https://gerrit.wikimedia.org/r/233471 [22:08:12] (03CR) 10ArielGlenn: [C: 032 V: 032] remove old audit_files.py, no longer needed, remove unused Rule class remove unused timeout from local audit methods remove some dead attrs [software] - 10https://gerrit.wikimedia.org/r/233471 (owner: 10ArielGlenn) [22:08:12] (03PS2) 10ArielGlenn: more pylint/pep8 cleanup [software] - 10https://gerrit.wikimedia.org/r/233472 [22:08:12] (03CR) 10ArielGlenn: [C: 032 V: 032] more pylint/pep8 cleanup [software] - 10https://gerrit.wikimedia.org/r/233472 (owner: 10ArielGlenn) [22:09:12] (03CR) 10ArielGlenn: [C: 032 V: 032] more pylint: move dup code for file ignore check into its own method take care of unused imports in borrowed code for 'magic' more pylint an [software] - 10https://gerrit.wikimedia.org/r/233473 (owner: 10ArielGlenn) [22:09:41] (03PS2) 10ArielGlenn: pep8 fixup: remove extra spaces pylint cleanup: work around no method overrides in python [software] - 10https://gerrit.wikimedia.org/r/233474 [22:10:04] (03CR) 10ArielGlenn: [C: 032 V: 032] pep8 fixup: remove extra spaces pylint cleanup: work around no method overrides in python [software] - 10https://gerrit.wikimedia.org/r/233474 (owner: 10ArielGlenn) [22:18:31] Hello, I'm going to sync out some debugging for T124356: https://gerrit.wikimedia.org/r/#/c/273655/ [22:18:31] T124356: Incorrect TOC and section edit links rendering in Vector due to ParserCache corruption via ParserOutput::setText( ParserOutput::getText() ) - https://phabricator.wikimedia.org/T124356 [22:20:59] * legoktm pets stashbot [22:21:20] /me purrs [22:39:46] !log legoktm just deployed https://gerrit.wikimedia.org/r/#/c/273655/ (MF cache pollution debug log) for T124356 [22:39:48] T124356: Incorrect TOC and section edit links rendering in Vector due to ParserCache corruption via ParserOutput::setText( ParserOutput::getText() ) - https://phabricator.wikimedia.org/T124356 [22:39:53] morebots: ? [22:40:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [22:41:06] I am a logbot running on tools-exec-1215. [22:41:06] Messages are logged to wikitech.wikimedia.org/wiki/Server_Admin_Log. [22:41:06] To log a message, type !log . [23:19:52] !log 23:19:43 Synchronized php-1.27.0-wmf.14/extensions/wikihiero/: Revert "Compress PNGs with zopflipng" - https://gerrit.wikimedia.org/r/273661 (duration: 00m 46s) [23:19:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master