[00:45:39] <wikibugs>	 6Operations: Start logmsgbot on neon - https://phabricator.wikimedia.org/T128302#2070005 (10Krenair)
[00:59:15] <ori_>	 !log Restarted logmsgbot on neon; it lost its IRC connection and did not recover it.
[00:59:19] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[00:59:26] <ori_>	 Krenair: ^
[00:59:34] <Krenair>	 ty
[00:59:52] <wikibugs>	 6Operations: Start logmsgbot on neon - https://phabricator.wikimedia.org/T128302#2070033 (10ori) 5Open>3Resolved a:3ori
[01:03:11] <Krenair>	 legoktm, ^
[01:13:26] <legoktm>	 thanks ori and Krenair
[01:21:38] <icinga-wm>	 PROBLEM - puppet last run on mw2030 is CRITICAL: CRITICAL: Puppet has 1 failures
[01:28:59] <icinga-wm>	 PROBLEM - Kafka Broker Replica Max Lag on kafka1022 is CRITICAL: CRITICAL: 57.69% of data above the critical threshold [5000000.0]
[01:39:39] <icinga-wm>	 RECOVERY - Kafka Broker Replica Max Lag on kafka1022 is OK: OK: Less than 50.00% above the threshold [1000000.0]
[01:47:59] <icinga-wm>	 RECOVERY - puppet last run on mw2030 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures
[02:24:18] <logmsgbot>	 !log mwdeploy@tin sync-l10n completed (1.27.0-wmf.14) (duration: 10m 24s)
[02:24:22] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[02:32:00] <logmsgbot>	 !log l10nupdate@tin ResourceLoader cache refresh completed at Sun Feb 28 02:32:00 UTC 2016 (duration 7m 42s)
[02:32:05] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[03:35:49] <icinga-wm>	 PROBLEM - puppet last run on mw1097 is CRITICAL: CRITICAL: Puppet has 1 failures
[03:42:40] <icinga-wm>	 PROBLEM - Kafka Broker Replica Max Lag on kafka1012 is CRITICAL: CRITICAL: 53.57% of data above the critical threshold [5000000.0]
[03:53:11] <icinga-wm>	 RECOVERY - Kafka Broker Replica Max Lag on kafka1012 is OK: OK: Less than 50.00% above the threshold [1000000.0]
[04:02:18] <icinga-wm>	 RECOVERY - puppet last run on mw1097 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[04:42:38] <wikibugs>	 6Operations, 10ops-eqiad, 6Labs: disk failure on labsdb1002 - https://phabricator.wikimedia.org/T126946#2027631 (10liangent) Any plan or progress on this?  Basically I'm using user databases there, and my tools stopped working because they're gone when *wiki.labsdb are switched. I have file dumps of those da...
[04:52:08] <wikibugs>	 6Operations, 10MobileFrontend, 10Traffic, 5MW-1.27-release, and 5 others: Incorrect TOC and section edit links rendering in Vector due to ParserCache corruption via ParserOutput::setText( ParserOutput::getText() ) - https://phabricator.wikimedia.org/T124356#2070083 (10Legoktm) So far in the log... ``` 2016...
[05:58:59] <wikibugs>	 6Operations, 10Ops-Access-Requests: Collaboration With Any Old Programmers\Coders, I am Only An Architect of reverse engineering cross platform DEV. - https://phabricator.wikimedia.org/T128305#2070087 (10M13Kd)
[06:01:16] <legoktm>	 ...what.
[06:26:11] <wikibugs>	 6Operations, 10Ops-Access-Requests: Collaboration With Any Old Programmers\Coders, I am Only An Architect of reverse engineering cross platform DEV. - https://phabricator.wikimedia.org/T128305#2070102 (10EBernhardson) 5Open>3Invalid
[06:30:30] <icinga-wm>	 PROBLEM - puppet last run on mw1226 is CRITICAL: CRITICAL: Puppet has 1 failures
[06:30:39] <icinga-wm>	 PROBLEM - puppet last run on cp2013 is CRITICAL: CRITICAL: Puppet has 1 failures
[06:30:50] <icinga-wm>	 PROBLEM - puppet last run on wtp2017 is CRITICAL: CRITICAL: Puppet has 1 failures
[06:31:09] <icinga-wm>	 PROBLEM - puppet last run on mw1170 is CRITICAL: CRITICAL: Puppet has 1 failures
[06:31:18] <icinga-wm>	 PROBLEM - puppet last run on scb1001 is CRITICAL: CRITICAL: Puppet has 1 failures
[06:31:39] <icinga-wm>	 PROBLEM - puppet last run on holmium is CRITICAL: CRITICAL: Puppet has 1 failures
[06:31:39] <icinga-wm>	 PROBLEM - puppet last run on mw1158 is CRITICAL: CRITICAL: Puppet has 1 failures
[06:31:48] <icinga-wm>	 PROBLEM - puppet last run on subra is CRITICAL: CRITICAL: Puppet has 1 failures
[06:31:49] <icinga-wm>	 PROBLEM - puppet last run on sca1001 is CRITICAL: CRITICAL: Puppet has 1 failures
[06:33:00] <icinga-wm>	 PROBLEM - puppet last run on mw2129 is CRITICAL: CRITICAL: Puppet has 1 failures
[06:33:01] <icinga-wm>	 PROBLEM - puppet last run on mw2021 is CRITICAL: CRITICAL: Puppet has 1 failures
[06:33:13] <wikibugs>	 6Operations, 10Ops-Access-Requests: Collaboration With Any Old Programmers-Coders, I am Only An Architect of reverse engineering cross platform Humanitarianism. - https://phabricator.wikimedia.org/T128305#2070103 (10M13Kd) 5Invalid>3Open p:5Triage>3Unbreak! a:3M13Kd
[06:33:19] <icinga-wm>	 PROBLEM - puppet last run on mw2207 is CRITICAL: CRITICAL: Puppet has 1 failures
[06:33:39] <icinga-wm>	 PROBLEM - puppet last run on mw2073 is CRITICAL: CRITICAL: Puppet has 1 failures
[06:33:59] <icinga-wm>	 PROBLEM - puppet last run on mw2018 is CRITICAL: CRITICAL: Puppet has 1 failures
[06:34:09] <icinga-wm>	 PROBLEM - puppet last run on mw2126 is CRITICAL: CRITICAL: Puppet has 1 failures
[06:50:41] <Matthew_>	 Krenair: Might want to re-close T128305 and block.
[06:50:42] <stashbot>	 T128305: Spam - https://phabricator.wikimedia.org/T128305
[06:51:26] <Krenair>	 They didn't actually give me proper administrator permissions
[06:51:29] <Krenair>	 I can't disable the account
[06:51:44] <Krenair>	 these people can: https://phabricator.wikimedia.org/people/query/zYtnRSCkBXQ./
[06:51:48] <Matthew_>	 That’s disappointing.
[06:51:54] <Krenair>	 (admin = ops, basically)
[06:52:00] <Matthew_>	 Ah.  
[06:52:16] <Krenair>	 so maybe you could bug andrewbogott to get it done :p
[06:53:41] <Krenair>	 heh: https://en.wikipedia.org/w/index.php?title=Exclamation_mark&diff=prev&oldid=668560294
[06:54:31] <Matthew_>	 Why does that not surprise me.
[06:54:35] <Matthew_>	 *?
[06:55:20] <icinga-wm>	 RECOVERY - puppet last run on mw1226 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures
[06:55:48] <icinga-wm>	 RECOVERY - puppet last run on wtp2017 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures
[06:55:59] <icinga-wm>	 RECOVERY - puppet last run on mw1170 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures
[06:55:59] <icinga-wm>	 RECOVERY - puppet last run on mw2021 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures
[06:56:00] <icinga-wm>	 RECOVERY - puppet last run on scb1001 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures
[06:56:19] <icinga-wm>	 RECOVERY - puppet last run on mw2207 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures
[06:56:29] <icinga-wm>	 RECOVERY - puppet last run on holmium is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures
[06:56:38] <icinga-wm>	 RECOVERY - puppet last run on subra is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures
[06:56:39] <icinga-wm>	 RECOVERY - puppet last run on sca1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:56:51] <Matthew_>	 Must step away, apologies.
[06:56:59] <icinga-wm>	 RECOVERY - puppet last run on mw2018 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures
[06:57:00] <icinga-wm>	 RECOVERY - puppet last run on mw2126 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures
[06:57:18] <icinga-wm>	 RECOVERY - puppet last run on cp2013 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures
[06:57:40] <icinga-wm>	 RECOVERY - puppet last run on mw2129 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures
[06:58:18] <icinga-wm>	 RECOVERY - puppet last run on mw1158 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:58:19] <icinga-wm>	 RECOVERY - puppet last run on mw2073 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[07:49:14] <apergos>	 !log restarted logmsbot on neon
[07:49:18] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[08:26:49] <icinga-wm>	 PROBLEM - puppet last run on heze is CRITICAL: CRITICAL: puppet fail
[08:51:10] <grrrit-wm>	 (03PS2) 10ArielGlenn: fix up for install via setup.py, bug fixes [software] - 10https://gerrit.wikimedia.org/r/233475 
[08:51:43] <grrrit-wm>	 (03CR) 10ArielGlenn: [C: 032 V: 032] fix up for install via setup.py, bug fixes [software] - 10https://gerrit.wikimedia.org/r/233475 (owner: 10ArielGlenn)
[08:53:03] <grrrit-wm>	 (03PS2) 10ArielGlenn: bit more pylint [software] - 10https://gerrit.wikimedia.org/r/233476 
[08:53:19] <icinga-wm>	 RECOVERY - puppet last run on heze is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures
[08:54:32] <grrrit-wm>	 (03PS3) 10ArielGlenn: bit more pylint: line lengths, extra imports [software] - 10https://gerrit.wikimedia.org/r/233476 
[08:54:57] <grrrit-wm>	 (03CR) 10ArielGlenn: [C: 032 V: 032] bit more pylint: line lengths, extra imports [software] - 10https://gerrit.wikimedia.org/r/233476 (owner: 10ArielGlenn)
[08:55:30] <grrrit-wm>	 (03PS2) 10ArielGlenn: retention emails fixes: [software] - 10https://gerrit.wikimedia.org/r/233477 
[08:56:04] <grrrit-wm>	 (03CR) 10ArielGlenn: [C: 032 V: 032] retention emails fixes: [software] - 10https://gerrit.wikimedia.org/r/233477 (owner: 10ArielGlenn)
[12:01:34] <grrrit-wm>	 (03PS3) 10Nemo bis: Use full URL in $wgNoticeHideUrls [mediawiki-config] - 10https://gerrit.wikimedia.org/r/250384 (https://phabricator.wikimedia.org/T116609) 
[13:00:53] <Luke081515>	 Can someone bringt https://gerrit.wikimedia.org/r/#/c/273592/ to the masteR? I guess it is not there
[13:00:57] <Luke081515>	 maybe ori?
[13:40:30] <icinga-wm>	 PROBLEM - puppet last run on mw2178 is CRITICAL: CRITICAL: puppet fail
[14:09:20] <icinga-wm>	 RECOVERY - puppet last run on mw2178 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures
[15:44:56] <wikibugs>	 6Operations, 10Phabricator, 6Project-Admins: Create policy projects and convert people projects to open - https://phabricator.wikimedia.org/T90491#2070573 (10greg)
[16:23:48] <grrrit-wm>	 (03CR) 10MarcoAurelio: [C: 031] "All events are now expired. Please sync. at your earliest convenience. Thank you." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/273580 (owner: 10MarcoAurelio)
[17:57:13] <Nemo_bis>	 apergos: operations/software went from 38 to 7 open changesets :P https://www.mediawiki.org/w/index.php?title=Gerrit%2FReports%2FChangesets_by_project&action=historysubmit&type=revision&diff=2063839&oldid=2062821
[17:57:23] <apergos>	 yeah I know
[17:58:04] <apergos>	 and my co-workers' spam rose by the same amount :-P
[17:58:12] <Nemo_bis>	 43 positions in the rank
[17:58:59] <Nemo_bis>	 now about as innocuous as mediawiki/extensions/GlobalUserPage
[18:00:43] <apergos>	 good
[18:15:19] <icinga-wm>	 PROBLEM - puppet last run on mw1159 is CRITICAL: CRITICAL: Puppet has 1 failures
[18:21:36] <grrrit-wm>	 (03PS1) 10Kelson: Fix regex to enable upload from ETHZ Library with the GWT [mediawiki-config] - 10https://gerrit.wikimedia.org/r/273774 
[18:39:58] <icinga-wm>	 RECOVERY - puppet last run on mw1159 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures
[18:44:17] <grrrit-wm>	 (03PS1) 10Luke081515: Correct one Domain at $wgCopyUploadsDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/273776 (https://phabricator.wikimedia.org/T123109) 
[18:46:08] <grrrit-wm>	 (03PS2) 10Luke081515: Correct one Domain at $wgCopyUploadsDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/273776 (https://phabricator.wikimedia.org/T123109) 
[18:53:25] <grrrit-wm>	 (03PS3) 10Luke081515: Correct one Domain at $wgCopyUploadsDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/273776 (https://phabricator.wikimedia.org/T123109) 
[19:18:04] <grrrit-wm>	 (03PS1) 10Nemo bis: [Planet Wikimedia] Multiple additions to English, Spanish, Ukrainian [puppet] - 10https://gerrit.wikimedia.org/r/273777 
[20:26:34] <wikibugs>	 6Operations, 10MobileFrontend, 10Traffic, 5MW-1.27-release, and 5 others: Incorrect TOC and section edit links rendering in Vector due to ParserCache corruption via ParserOutput::setText( ParserOutput::getText() ) - https://phabricator.wikimedia.org/T124356#2070871 (10Legoktm) ``` 2016-02-28 15:32:06 mw107...
[20:54:41] <grrrit-wm>	 (03PS2) 10ArielGlenn: dumps: stash some current dump run config settings in file and reuse [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/266775 
[20:56:05] <grrrit-wm>	 (03PS3) 10ArielGlenn: dumps: stash some current dump run config settings in file and reuse [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/266775 
[20:57:07] <grrrit-wm>	 (03CR) 10ArielGlenn: [C: 032 V: 032] dumps: stash some current dump run config settings in file and reuse [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/266775 (owner: 10ArielGlenn)
[21:11:04] <grrrit-wm>	 (03PS1) 10Luke081515: Enable rollbacker and suppressredirect group at cewiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/273828 (https://phabricator.wikimedia.org/T128205) 
[21:15:12] <grrrit-wm>	 (03CR) 10Luke081515: [C: 04-1] "Needs some reply from the requestor, see task." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/273828 (https://phabricator.wikimedia.org/T128205) (owner: 10Luke081515)
[21:23:36] <grrrit-wm>	 (03PS1) 10Nemo bis: [Planet Wikimedia] Add Albanian and Bulgarian planets [puppet] - 10https://gerrit.wikimedia.org/r/273830 
[21:28:03] <grrrit-wm>	 (03CR) 10Nemo bis: [Planet Wikimedia] Add Albanian and Bulgarian planets (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/273830 (owner: 10Nemo bis)
[21:36:13] <wikibugs>	 6Operations, 6Commons, 10MediaWiki-Uploading, 6Multimedia, and 2 others: Raise max upload limit above 1GB - https://phabricator.wikimedia.org/T76614#2070996 (10matmarex) 5Open>3Resolved
[21:38:29] <icinga-wm>	 PROBLEM - Kafka Broker Replica Max Lag on kafka1020 is CRITICAL: CRITICAL: 58.33% of data above the critical threshold [5000000.0]
[21:40:30] <wikibugs>	 6Operations, 10Dumps-Generation: Migrate dataset1001 and ms1001 to jessie - https://phabricator.wikimedia.org/T123724#2071007 (10ArielGlenn) I'll be doing this over the next two days; the monthly full run completed on Friday.  Ms1001 will be first, with all rsyncs from dataset1001 to it paused.
[21:45:29] <icinga-wm>	 RECOVERY - Kafka Broker Replica Max Lag on kafka1020 is OK: OK: Less than 50.00% above the threshold [1000000.0]
[21:46:58] <grrrit-wm>	 (03PS5) 10ArielGlenn: git deploy cleanup <minion> to toss minion from redis [software/deployment/trebuchet-trigger] - 10https://gerrit.wikimedia.org/r/219852 (https://phabricator.wikimedia.org/T74319) 
[21:51:06] <wikibugs>	 6Operations, 10ops-codfw, 13Patch-For-Review: power off Codfw-Cisco Servers - https://phabricator.wikimedia.org/T115372#1722630 (10Krenair) Is this supposed to still be open?
[21:52:56] <grrrit-wm>	 (03CR) 10ArielGlenn: [C: 032 V: 032] git deploy cleanup <minion> to toss minion from redis [software/deployment/trebuchet-trigger] - 10https://gerrit.wikimedia.org/r/219852 (https://phabricator.wikimedia.org/T74319) (owner: 10ArielGlenn)
[21:55:41] <wikibugs>	 6Operations, 10Salt: Salt minions randomly crashing when the deployment server grain gets changed - https://phabricator.wikimedia.org/T124646#2071029 (10ArielGlenn) No joy so I'll add the above change to our salt packages with logging and update them all.
[22:19:54] <grrrit-wm>	 (03PS2) 10ArielGlenn: report_minions: show minions known about in redis for repo [software/deployment/trebuchet-trigger] - 10https://gerrit.wikimedia.org/r/219845 
[22:27:24] <sjoerddebruin>	 Ehm, I get "Exception encountered, of type "BadMethodCallException"" on https://en.wikipedia.org/wiki/Wolfsberg,_Thuringia
[22:29:03] <legoktm>	 2016-02-28 22:27:33 mw1217 enwiki 1.27.0-wmf.14 exception ERROR: [b158dc6e] /wik
[22:29:03] <legoktm>	 i/Wolfsberg,_Thuringia   BadMethodCallException from line 300 of /srv/mediawiki/
[22:29:03] <legoktm>	 php-1.27.0-wmf.14/extensions/WikidataPageBanner/includes/WikidataPageBanner.func
[22:29:03] <legoktm>	 tions.php: Call to a member function getStatements() on a non-object (NULL) {"ex
[22:29:03] <legoktm>	 ception_id":"b158dc6e"} 
[22:29:23] <sjoerddebruin>	 I've did some Wikidata-edits to the connected item btw
[22:29:27] <hoo>	 WikidataPageBanner wtf
[22:29:31] <sjoerddebruin>	 Somebody messed up the sitelinks.
[22:29:44] <hoo>	 isn't that supposed to be enabled only on the wikivoyages? :P
[22:29:58] <legoktm>	 http://fpaste.org/330906/66985901/raw/ is the full trace
[22:30:18] <legoktm>	 https://en.wikipedia.org/wiki/Special:Version shows it on enwp...
[22:30:28] <legoktm>	 I'll leave you to file a bug about it :)
[22:30:39] <sjoerddebruin>	 https://phabricator.wikimedia.org/rOMWC4fa000ba9255be71bd677019ea55e24a3972ca14
[22:34:54] <logmsgbot>	 !log legoktm@tin Synchronized php-1.27.0-wmf.14/includes/parser/ParserCache.php: Include ParserCache::save() backtrace in MF pollution debug log - T124356 (duration: 00m 50s)
[22:34:55] <stashbot>	 T124356: Incorrect TOC and section edit links rendering in Vector due to ParserCache corruption via ParserOutput::setText( ParserOutput::getText() ) - https://phabricator.wikimedia.org/T124356
[22:34:57] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[22:36:21] <grrrit-wm>	 (03PS1) 10Catrope: Temporarily disable thank-you-edit notifications [mediawiki-config] - 10https://gerrit.wikimedia.org/r/273836 (https://phabricator.wikimedia.org/T128249) 
[22:38:36] <sjoerddebruin>	 hoo: you're on it?
[22:38:53] <hoo>	 sjoerddebruin: Sorry, am not... but will have a look
[22:39:05] <sjoerddebruin>	 Do I need to make a task?
[22:39:36] <hoo>	 Would be nice
[22:41:36] <hoo>	 pretty obvious why that fails
[22:41:48] <hoo>	 we had the same in Wikibase
[22:41:57] <hoo>	 but the extension looks scary to me
[22:42:27] <hoo>	 No tests :(
[22:42:36] <sjoerddebruin>	 Only Wikidata-Page-Banner project or also operations?
[22:42:46] <hoo>	 only the extension
[22:42:49] <sjoerddebruin>	 ok
[22:43:01] <sjoerddebruin>	 See https://phabricator.wikimedia.org/T128342
[22:43:07] <hoo>	 thanks
[22:43:14] <hoo>	 I'll hack up a patch for the extension
[22:43:22] <hoo>	 scary
[22:43:34] <sjoerddebruin>	 "Phabricator project for the experimental WikidataPageBanner extension."
[22:44:06] <hoo>	 "experimental"...
[22:46:20] <sjoerddebruin>	 I'm getting too much "unbreak now"'s on my name.
[22:56:36] <hoo>	 :P
[22:56:42] <hoo>	 That extension really needs some love :/
[22:57:20] <wikibugs>	 6Operations, 10MobileFrontend, 10Traffic, 5MW-1.27-release, and 5 others: Incorrect TOC and section edit links rendering in Vector due to ParserCache corruption via ParserOutput::setText( ParserOutput::getText() ) - https://phabricator.wikimedia.org/T124356#2071102 (10Legoktm) >>! In T124356#2070871, @Lego...