[02:30:30] PROBLEM - puppet last run on wtp1026 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:31:20] PROBLEM - puppet last run on aqs1006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:31:20] PROBLEM - puppet last run on analytics1034 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:31:40] PROBLEM - puppet last run on db1073 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:31:40] PROBLEM - puppet last run on analytics1065 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:31:40] PROBLEM - puppet last run on boron is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:32:31] PROBLEM - puppet last run on mw1238 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:32:41] PROBLEM - puppet last run on mw1262 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:32:41] PROBLEM - puppet last run on cp1047 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:32:41] PROBLEM - puppet last run on rhodium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:33:10] PROBLEM - puppet last run on db1082 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:33:11] PROBLEM - puppet last run on druid1002 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:33:20] PROBLEM - puppet last run on ms-be1018 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:33:30] PROBLEM - puppet last run on cp1062 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:33:30] PROBLEM - puppet last run on elastic1026 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:33:30] PROBLEM - puppet last run on restbase1011 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:33:40] PROBLEM - puppet last run on elastic1048 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:34:04] Looks like puppetdb again [02:34:51] PROBLEM - puppet last run on ms-be1028 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:35:21] PROBLEM - puppet last run on elastic1018 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [02:35:30] I wonder why does puppetdb keeps going off? [02:35:40] Does it need more heep ? [02:58:20] RECOVERY - puppet last run on ms-be1018 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [02:59:51] RECOVERY - puppet last run on ms-be1028 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [03:00:21] RECOVERY - puppet last run on elastic1018 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:00:30] RECOVERY - puppet last run on wtp1026 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:01:20] RECOVERY - puppet last run on aqs1006 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [03:01:20] RECOVERY - puppet last run on analytics1034 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [03:01:40] RECOVERY - puppet last run on db1073 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [03:01:40] RECOVERY - puppet last run on analytics1065 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [03:01:40] RECOVERY - puppet last run on boron is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [03:02:31] RECOVERY - puppet last run on mw1238 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [03:02:41] RECOVERY - puppet last run on rhodium is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [03:02:41] RECOVERY - puppet last run on mw1262 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [03:02:41] RECOVERY - puppet last run on cp1047 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [03:03:10] RECOVERY - puppet last run on db1082 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [03:03:11] RECOVERY - puppet last run on druid1002 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [03:03:30] RECOVERY - puppet last run on cp1062 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures [03:03:30] RECOVERY - puppet last run on elastic1026 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [03:03:30] RECOVERY - puppet last run on restbase1011 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [03:03:41] RECOVERY - puppet last run on elastic1048 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [05:44:48] https://phabricator.wikimedia.org/T45952 [05:45:14] hi, this is important, please have a look ^ [05:45:35] the file can't be undeleted [05:53:20] 10Operations, 10media-storage: Incorrect "non-identical file already exists" error when undeleting file on Commons - https://phabricator.wikimedia.org/T45952#3999476 (10Peachey88) [07:25:57] 10Operations, 10ops-eqiad, 10DBA: Degraded RAID on db1068 - https://phabricator.wikimedia.org/T188187#3999488 (10Marostegui) p:05Triage>03High This is s4 primary master - please replace the disk as soon as you can. Thanks! [07:26:50] RECOVERY - Check systemd state on rhenium is OK: OK - running: The system is fully operational [07:29:50] PROBLEM - Check systemd state on rhenium is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [07:35:52] !log Fix s7 replication on labsdb1010 - T186579 [07:36:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [07:36:09] T186579: labsdb1010 crashed - https://phabricator.wikimedia.org/T186579 [07:45:11] 10Operations, 10ops-eqiad, 10DBA: Degraded RAID on db1068 - https://phabricator.wikimedia.org/T188187#3999509 (10Marostegui) a:03Cmjohnson [08:13:39] (03PS1) 10Umherirrender: Replace wfGetLBFactory [mediawiki-config] - 10https://gerrit.wikimedia.org/r/414310 [08:13:59] (03CR) 10Umherirrender: "Not tested" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/414310 (owner: 10Umherirrender) [09:59:20] PROBLEM - Router interfaces on cr1-eqiad is CRITICAL: CRITICAL: host 208.80.154.196, interfaces up: 232, down: 1, dormant: 0, excluded: 0, unused: 0 [10:08:30] RECOVERY - Router interfaces on cr1-eqiad is OK: OK: host 208.80.154.196, interfaces up: 234, down: 0, dormant: 0, excluded: 0, unused: 0 [10:33:05] (03PS1) 10Zoranzoki21: Add mushroomobserver.org to $wgCopyUploadsDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/414401 (https://phabricator.wikimedia.org/T188203) [10:34:12] (03PS2) 10Zoranzoki21: Add mushroomobserver.org to wgCopyUploadsDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/414401 (https://phabricator.wikimedia.org/T188203) [10:34:15] (03CR) 10jerkins-bot: [V: 04-1] Add mushroomobserver.org to wgCopyUploadsDomains [mediawiki-config] - 10https://gerrit.wikimedia.org/r/414401 (https://phabricator.wikimedia.org/T188203) (owner: 10Zoranzoki21) [10:35:28] (03CR) 10Zoranzoki21: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/414401 (https://phabricator.wikimedia.org/T188203) (owner: 10Zoranzoki21) [10:35:52] (03CR) 10Zoranzoki21: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/414401 (https://phabricator.wikimedia.org/T188203) (owner: 10Zoranzoki21) [11:02:00] 10Operations, 10ops-eqiad, 10DBA: Degraded RAID on db1068 - https://phabricator.wikimedia.org/T188187#3999213 (10jcrespo) @Cmjohnson Please be extra careful here, there are 2 degraded disks here, but we want to change first _only_ the one shown on the list up there. Once we stop being in non-redundant mode,... [11:03:10] RECOVERY - Router interfaces on cr2-eqiad is OK: OK: host 208.80.154.197, interfaces up: 226, down: 0, dormant: 0, excluded: 0, unused: 0 [11:03:20] RECOVERY - Router interfaces on cr1-eqord is OK: OK: host 208.80.154.198, interfaces up: 39, down: 0, dormant: 0, excluded: 0, unused: 0 [11:30:47] 10Operations, 10ops-eqiad, 10DBA: Degraded RAID on db1068 - https://phabricator.wikimedia.org/T188187#3999608 (10Marostegui) I believe only the one marked as failed should be blinking in a different colour. The other one only shows errors as far as the report goes, but yeah, better be careful if there are tw... [12:00:09] 10Operations, 10media-storage: Incorrect "non-identical file already exists" error when undeleting file on Commons - https://phabricator.wikimedia.org/T45952#3999633 (10Aklapper) [12:59:45] (03Draft2) 10Tulsi Bhagat: Add English & Bengali Wikisource as Import sources on pawikisource. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/414451 (https://phabricator.wikimedia.org/T185982) [13:24:01] (03CR) 10Jayprakash12345: [C: 04-1] "As per task add other Indic projects as well. See There is around 10 Indic Project. Add all of them. https://meta.wikimedia.org/wiki/Wikis" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/414451 (https://phabricator.wikimedia.org/T185982) (owner: 10Tulsi Bhagat) [14:14:56] (03PS3) 10Tulsi Bhagat: Add English & other Indic language Wikisource projects as Import sources on pawikisource. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/414451 (https://phabricator.wikimedia.org/T185982) [14:15:04] (03CR) 10jerkins-bot: [V: 04-1] Add English & other Indic language Wikisource projects as Import sources on pawikisource. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/414451 (https://phabricator.wikimedia.org/T185982) (owner: 10Tulsi Bhagat) [14:18:34] (03PS1) 10Jcrespo: tendril: Add memcache to tendril web frontend [puppet] - 10https://gerrit.wikimedia.org/r/414502 (https://phabricator.wikimedia.org/T133906) [14:18:59] (03CR) 10jerkins-bot: [V: 04-1] tendril: Add memcache to tendril web frontend [puppet] - 10https://gerrit.wikimedia.org/r/414502 (https://phabricator.wikimedia.org/T133906) (owner: 10Jcrespo) [14:30:37] (03CR) 10Sau226: [C: 031] "This patch looks good. After the pages using flow are disabled and the "discussion" is over (which hopefully results in the community cons" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/408073 (https://phabricator.wikimedia.org/T186463) (owner: 10Zoranzoki21) [14:38:46] (03PS2) 10Jcrespo: tendril: Add memcache to tendril web frontend [puppet] - 10https://gerrit.wikimedia.org/r/414502 (https://phabricator.wikimedia.org/T133906) [14:39:54] (03PS4) 10Jayprakash12345: Add Import sources on pawikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/414451 (https://phabricator.wikimedia.org/T185982) (owner: 10Tulsi Bhagat) [14:41:28] (03CR) 10jerkins-bot: [V: 04-1] Add Import sources on pawikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/414451 (https://phabricator.wikimedia.org/T185982) (owner: 10Tulsi Bhagat) [14:59:20] (03CR) 10Jayprakash12345: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/414451 (https://phabricator.wikimedia.org/T185982) (owner: 10Tulsi Bhagat) [15:00:49] (03CR) 10jerkins-bot: [V: 04-1] Add Import sources on pawikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/414451 (https://phabricator.wikimedia.org/T185982) (owner: 10Tulsi Bhagat) [15:24:50] PROBLEM - Long running screen/tmux on eventlog1001 is CRITICAL: CRIT: Long running tmux process. (PID: 3873, 1742139s 1728000s). [15:41:16] (03Abandoned) 10Urbanecm: New throttle rule [mediawiki-config] - 10https://gerrit.wikimedia.org/r/413722 (https://phabricator.wikimedia.org/T188091) (owner: 10Urbanecm) [15:59:10] PROBLEM - puppet last run on dataset1001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [16:23:42] (03PS1) 10Jayprakash12345: Enable Quiz Extension at zhwikibooks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/414506 (https://phabricator.wikimedia.org/T188213) [16:24:56] (03CR) 10jerkins-bot: [V: 04-1] Enable Quiz Extension at zhwikibooks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/414506 (https://phabricator.wikimedia.org/T188213) (owner: 10Jayprakash12345) [16:25:31] (03CR) 10Zoranzoki21: [C: 031] "With patch is ok. Failed test is problem with CI which are ill" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/414506 (https://phabricator.wikimedia.org/T188213) (owner: 10Jayprakash12345) [16:29:10] RECOVERY - puppet last run on dataset1001 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [16:37:51] (03CR) 10Tulsi Bhagat: [C: 031] "As per Zoranzoki21" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/414506 (https://phabricator.wikimedia.org/T188213) (owner: 10Jayprakash12345) [17:09:38] (03PS1) 10Sau226: Disable main page deletion on enwiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/414509 [19:10:40] (03CR) 10Jayprakash12345: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/414506 (https://phabricator.wikimedia.org/T188213) (owner: 10Jayprakash12345) [19:10:57] (03CR) 10Jayprakash12345: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/414451 (https://phabricator.wikimedia.org/T185982) (owner: 10Tulsi Bhagat) [19:12:27] (03PS2) 10Zoranzoki21: Disable main page deletion on enwiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/414509 (https://phabricator.wikimedia.org/T184959) (owner: 10Sau226) [19:25:02] (03CR) 10Reedy: [C: 04-1] "Don't just duplicate all the code" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/414509 (https://phabricator.wikimedia.org/T184959) (owner: 10Sau226) [19:44:53] (03PS3) 10Chad: Disable main page deletion on enwiktionary [mediawiki-config] - 10https://gerrit.wikimedia.org/r/414509 (https://phabricator.wikimedia.org/T184959) (owner: 10Sau226) [19:45:11] (03CR) 10Chad: "Got rid of code duplication, but I think this needs further discussion on the task." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/414509 (https://phabricator.wikimedia.org/T184959) (owner: 10Sau226) [19:53:41] (03CR) 10GeoffreyT2000: [C: 04-1] "We also need consensus for disabling the ability to move the Main Page on enwiktionary." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/414509 (https://phabricator.wikimedia.org/T184959) (owner: 10Sau226) [21:24:20] PROBLEM - HHVM jobrunner on mw1303 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 473 bytes in 0.001 second response time [21:25:20] RECOVERY - HHVM jobrunner on mw1303 is OK: HTTP OK: HTTP/1.1 200 OK - 206 bytes in 0.006 second response time [22:42:10] PROBLEM - puppet last run on etcd1006 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [23:01:47] (03CR) 10Jforrester: [C: 031] beta: remove $wgFragmentMode, matches prod now [mediawiki-config] - 10https://gerrit.wikimedia.org/r/414009 (owner: 10MaxSem) [23:01:52] (03CR) 10Jforrester: [C: 031] beta: remove $wgSecureLogin [mediawiki-config] - 10https://gerrit.wikimedia.org/r/414010 (owner: 10MaxSem) [23:02:00] (03CR) 10Jforrester: [C: 031] beta: remove $wgStructuredChangeFiltersShowPreference [mediawiki-config] - 10https://gerrit.wikimedia.org/r/414011 (owner: 10MaxSem) [23:02:08] (03CR) 10Jforrester: [C: 031] beta: remove $wmgUseTimeless [mediawiki-config] - 10https://gerrit.wikimedia.org/r/414012 (owner: 10MaxSem) [23:02:20] (03CR) 10Jforrester: [C: 031] beta: remove $wmgUse3d [mediawiki-config] - 10https://gerrit.wikimedia.org/r/414013 (owner: 10MaxSem) [23:02:32] (03CR) 10Jforrester: [C: 031] "Sorry, I should have cleaned this up." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/414014 (owner: 10MaxSem) [23:02:49] (03CR) 10Jforrester: [C: 031] Clean up $wgEchoPerUserBlacklist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/414015 (owner: 10MaxSem) [23:03:05] (03CR) 10Jforrester: [C: 031] Clean up $wgEchoPerUserBlacklist (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/414015 (owner: 10MaxSem) [23:03:14] (03CR) 10Jforrester: [C: 031] beta: remove $wmgMinervaNeue [mediawiki-config] - 10https://gerrit.wikimedia.org/r/414016 (owner: 10MaxSem) [23:03:23] (03CR) 10Jforrester: [C: 031] beta: remove $wgReadingListsCentralWiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/414017 (owner: 10MaxSem) [23:03:32] (03CR) 10Jforrester: [C: 031] beta: remove $wmgUseReadingLists [mediawiki-config] - 10https://gerrit.wikimedia.org/r/414018 (owner: 10MaxSem) [23:12:10] RECOVERY - puppet last run on etcd1006 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [23:20:50] PROBLEM - puppet last run on mw1227 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [23:30:01] PROBLEM - puppet last run on cerium is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [23:45:51] RECOVERY - puppet last run on mw1227 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [23:53:36] (03PS1) 10Paladox: Add BUILD files to build plugin [software/gerrit/plugins/wikimedia] - 10https://gerrit.wikimedia.org/r/414598 [23:54:49] (03CR) 10Jforrester: "Scheduled for deploy on 2018-02-26T1900Z:" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/413651 (owner: 10Jforrester) [23:55:12] (03CR) 10Jforrester: "Scheduled for deploy on 2018-03-05T1400Z:" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/413652 (owner: 10Jforrester) [23:55:21] (03PS2) 10Paladox: Add BUILD files to build plugin [software/gerrit/plugins/wikimedia] - 10https://gerrit.wikimedia.org/r/414598 [23:55:42] (03PS3) 10Paladox: Add BUILD files to build plugin [software/gerrit/plugins/wikimedia] - 10https://gerrit.wikimedia.org/r/414598 [23:56:07] (03PS4) 10Paladox: Add BUILD files to build plugin [software/gerrit/plugins/wikimedia] - 10https://gerrit.wikimedia.org/r/414598