[00:01:29] <wikibugs>	 (03CR) 10CDanis: [C: 03+2] rsync: provide a hiera default to unbreak cloud [puppet] - 10https://gerrit.wikimedia.org/r/549949 (https://phabricator.wikimedia.org/T237424) (owner: 10CDanis)
[01:06:40] <logmsgbot>	 !log volker-e@deploy1001 Started deploy [design/style-guide@97fb3ee]: Deploy design/style-guide:
[01:06:44] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[01:06:49] <logmsgbot>	 !log volker-e@deploy1001 Finished deploy [design/style-guide@97fb3ee]: Deploy design/style-guide:  (duration: 00m 09s)
[01:06:52] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[01:07:30] <logmsgbot>	 !log volker-e@deploy1001 Started deploy [design/style-guide@ef82b69]: Deploy design/style-guide:
[01:07:33] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[01:07:38] <logmsgbot>	 !log volker-e@deploy1001 Finished deploy [design/style-guide@ef82b69]: Deploy design/style-guide:  (duration: 00m 07s)
[01:07:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[02:14:13] <ebernhardson>	 or at least our modern incarnation
[02:14:19] <ebernhardson>	 ignore
[02:39:49] <logmsgbot>	 !log volker-e@deploy1001 Started deploy [design/style-guide@d2bfc09]: Deploy design/style-guide:
[02:39:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[02:39:56] <logmsgbot>	 !log volker-e@deploy1001 Finished deploy [design/style-guide@d2bfc09]: Deploy design/style-guide:  (duration: 00m 07s)
[02:39:59] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[02:43:21] <wikibugs>	 10Operations, 10Wikimedia-Logstash, 10Privacy: Production logstash should be protected by two-factor auth, at the least - https://phabricator.wikimedia.org/T237630 (10Peachey88)
[02:46:25] <icinga-wm>	 PROBLEM - Check the Netbox report puppetdb for fail status. on netbox1001 is CRITICAL: puppetdb.PuppetDB CRITICAL https://wikitech.wikimedia.org/wiki/Netbox%23Reports
[03:03:47] <wikibugs>	 (03PS6) 10CRusnov: Add script to generate DNS records from Netbox [software/netbox-deploy] - 10https://gerrit.wikimedia.org/r/539013 (https://phabricator.wikimedia.org/T233183)
[03:08:49] <icinga-wm>	 PROBLEM - Check the Netbox report puppetdb for fail status. on netbox1001 is CRITICAL: puppetdb.PuppetDB CRITICAL https://wikitech.wikimedia.org/wiki/Netbox%23Reports
[03:19:57] <icinga-wm>	 PROBLEM - Check the Netbox report puppetdb for fail status. on netbox1001 is CRITICAL: puppetdb.PuppetDB CRITICAL https://wikitech.wikimedia.org/wiki/Netbox%23Reports
[03:29:42] <wikibugs>	 (03PS1) 10CRusnov: netbox: set alert url to report address [puppet] - 10https://gerrit.wikimedia.org/r/549959
[03:40:30] <wikibugs>	 (03PS1) 10Brian Wolff: Back out sandbox rule on doc.wm.org, at least for now. [puppet] - 10https://gerrit.wikimedia.org/r/549960 (https://phabricator.wikimedia.org/T213223)
[03:42:02] <wikibugs>	 (03CR) 10Brian Wolff: "So this is breaking doxygen pretty badly due to the sandbox rule (which only gets actived in enforce mode). I propose backing out that par" [puppet] - 10https://gerrit.wikimedia.org/r/547718 (https://phabricator.wikimedia.org/T213223) (owner: 10Brian Wolff)
[03:46:21] <bawolff>	 Wow, so its not just doxygen, the sandbox property breaks kind of everything
[04:15:53] <icinga-wm>	 PROBLEM - Check the Netbox report puppetdb for fail status. on netbox1001 is CRITICAL: puppetdb.PuppetDB CRITICAL https://wikitech.wikimedia.org/wiki/Netbox%23Reports
[04:38:17] <icinga-wm>	 PROBLEM - Check the Netbox report puppetdb for fail status. on netbox1001 is CRITICAL: puppetdb.PuppetDB CRITICAL https://wikitech.wikimedia.org/wiki/Netbox%23Reports
[05:11:47] <icinga-wm>	 PROBLEM - Check the Netbox report puppetdb for fail status. on netbox1001 is CRITICAL: puppetdb.PuppetDB CRITICAL https://wikitech.wikimedia.org/wiki/Netbox%23Reports
[05:51:48] <wikibugs>	 (03CR) 10Dzahn: [C: 03+2] Back out sandbox rule on doc.wm.org, at least for now. [puppet] - 10https://gerrit.wikimedia.org/r/549960 (https://phabricator.wikimedia.org/T213223) (owner: 10Brian Wolff)
[05:53:55] <wikibugs>	 (03CR) 10Dzahn: "merged the partial revert" [puppet] - 10https://gerrit.wikimedia.org/r/547718 (https://phabricator.wikimedia.org/T213223) (owner: 10Brian Wolff)
[06:13:17] <icinga-wm>	 PROBLEM - Check the Netbox report puppetdb for fail status. on netbox1001 is CRITICAL: puppetdb.PuppetDB CRITICAL https://wikitech.wikimedia.org/wiki/Netbox%23Reports
[06:24:31] <icinga-wm>	 PROBLEM - Check the Netbox report puppetdb for fail status. on netbox1001 is CRITICAL: puppetdb.PuppetDB CRITICAL https://wikitech.wikimedia.org/wiki/Netbox%23Reports
[06:27:15] <icinga-wm>	 PROBLEM - Maps tiles generation on icinga1001 is CRITICAL: CRITICAL: 100.00% of data under the critical threshold [5.0] https://wikitech.wikimedia.org/wiki/Maps/Runbook https://grafana.wikimedia.org/dashboard/db/maps-performances?panelId=8&fullscreen&orgId=1
[06:42:52] <wikibugs>	 10Operations, 10Dumps-Generation, 10SDC General, 10Wikidata: Capacity planning for Commons Structured Data - https://phabricator.wikimedia.org/T226093 (10ArielGlenn) As evidenced by https://graphite.wikimedia.org/S/i we already have 5.5 million images with contents in the MediaInfo slot. Two months to go u...
[07:14:47] <icinga-wm>	 PROBLEM - Check the Netbox report puppetdb for fail status. on netbox1001 is CRITICAL: puppetdb.PuppetDB CRITICAL https://wikitech.wikimedia.org/wiki/Netbox%23Reports
[07:31:45] <icinga-wm>	 PROBLEM - OSPF status on cr2-codfw is CRITICAL: OSPFv2: 4/5 UP : OSPFv3: 4/5 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status
[07:32:13] <icinga-wm>	 PROBLEM - Router interfaces on cr2-eqiad is CRITICAL: CRITICAL: host 208.80.154.197, interfaces up: 240, down: 1, dormant: 0, excluded: 0, unused: 0: https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down
[07:47:29] <icinga-wm>	 RECOVERY - Check systemd state on labtestpuppetmaster2001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[07:52:17] <icinga-wm>	 PROBLEM - Check systemd state on labtestpuppetmaster2001 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[08:10:37] <icinga-wm>	 RECOVERY - Router interfaces on cr2-eqiad is OK: OK: host 208.80.154.197, interfaces up: 242, down: 0, dormant: 0, excluded: 0, unused: 0 https://wikitech.wikimedia.org/wiki/Network_monitoring%23Router_interface_down
[08:11:41] <icinga-wm>	 RECOVERY - OSPF status on cr2-codfw is OK: OSPFv2: 5/5 UP : OSPFv3: 5/5 UP https://wikitech.wikimedia.org/wiki/Network_monitoring%23OSPF_status
[08:16:11] <icinga-wm>	 PROBLEM - Check the Netbox report puppetdb for fail status. on netbox1001 is CRITICAL: puppetdb.PuppetDB CRITICAL https://wikitech.wikimedia.org/wiki/Netbox%23Reports
[08:27:21] <icinga-wm>	 PROBLEM - Check the Netbox report puppetdb for fail status. on netbox1001 is CRITICAL: puppetdb.PuppetDB CRITICAL https://wikitech.wikimedia.org/wiki/Netbox%23Reports
[08:55:15] <icinga-wm>	 PROBLEM - Check the Netbox report puppetdb for fail status. on netbox1001 is CRITICAL: puppetdb.PuppetDB CRITICAL https://wikitech.wikimedia.org/wiki/Netbox%23Reports
[09:12:05] <icinga-wm>	 PROBLEM - Check the Netbox report puppetdb for fail status. on netbox1001 is CRITICAL: puppetdb.PuppetDB CRITICAL https://wikitech.wikimedia.org/wiki/Netbox%23Reports
[10:02:21] <icinga-wm>	 PROBLEM - Check the Netbox report puppetdb for fail status. on netbox1001 is CRITICAL: puppetdb.PuppetDB CRITICAL https://wikitech.wikimedia.org/wiki/Netbox%23Reports
[10:13:35] <icinga-wm>	 PROBLEM - Check the Netbox report puppetdb for fail status. on netbox1001 is CRITICAL: puppetdb.PuppetDB CRITICAL https://wikitech.wikimedia.org/wiki/Netbox%23Reports
[11:26:09] <icinga-wm>	 PROBLEM - Check the Netbox report puppetdb for fail status. on netbox1001 is CRITICAL: puppetdb.PuppetDB CRITICAL https://wikitech.wikimedia.org/wiki/Netbox%23Reports
[11:37:21] <icinga-wm>	 PROBLEM - Check the Netbox report puppetdb for fail status. on netbox1001 is CRITICAL: puppetdb.PuppetDB CRITICAL https://wikitech.wikimedia.org/wiki/Netbox%23Reports
[12:16:25] <icinga-wm>	 PROBLEM - Check the Netbox report puppetdb for fail status. on netbox1001 is CRITICAL: puppetdb.PuppetDB CRITICAL https://wikitech.wikimedia.org/wiki/Netbox%23Reports
[12:19:09] <wikibugs>	 10Operations, 10SRE-tools, 10netbox: Netbox reports Icinga checks timeout - https://phabricator.wikimedia.org/T237803 (10Volans)
[12:19:18] <wikibugs>	 10Operations, 10SRE-tools, 10netbox: Netbox reports Icinga checks timeout - https://phabricator.wikimedia.org/T237803 (10Volans) p:05Triage→03High
[12:30:42] <wikibugs>	 (03CR) 10Faidon Liambotis: [C: 04-1] netbox: set alert url to report address (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/549959 (owner: 10CRusnov)
[12:33:09] <icinga-wm>	 RECOVERY - Check the Netbox report puppetdb for fail status. on netbox1001 is OK: puppetdb.PuppetDB OK https://wikitech.wikimedia.org/wiki/Netbox%23Reports
[15:19:52] <wikibugs>	 (03CR) 10Andrew Bogott: [C: 03+1] "Does this need to be merged in sync with other changes or can it be done whenever?" [puppet] - 10https://gerrit.wikimedia.org/r/547992 (https://phabricator.wikimedia.org/T235218) (owner: 10Alex Monk)
[15:20:46] <wikibugs>	 (03CR) 10Alex Monk: "This can be done whenever." [puppet] - 10https://gerrit.wikimedia.org/r/547992 (https://phabricator.wikimedia.org/T235218) (owner: 10Alex Monk)
[15:21:19] <wikibugs>	 (03PS2) 10Andrew Bogott: cloud-puppetmaster: Prep for new instances [puppet] - 10https://gerrit.wikimedia.org/r/547992 (https://phabricator.wikimedia.org/T235218) (owner: 10Alex Monk)
[15:23:01] <wikibugs>	 (03CR) 10Andrew Bogott: [C: 03+2] cloud-puppetmaster: Prep for new instances [puppet] - 10https://gerrit.wikimedia.org/r/547992 (https://phabricator.wikimedia.org/T235218) (owner: 10Alex Monk)
[20:22:58] <wikibugs>	 (03PS1) 10Reedy: Add cs to langlist-labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/549982 (https://phabricator.wikimedia.org/T237823)
[20:23:21] <wikibugs>	 (03CR) 10Reedy: [C: 03+2] Add cs to langlist-labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/549982 (https://phabricator.wikimedia.org/T237823) (owner: 10Reedy)
[20:24:14] <wikibugs>	 (03Merged) 10jenkins-bot: Add cs to langlist-labs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/549982 (https://phabricator.wikimedia.org/T237823) (owner: 10Reedy)
[20:25:38] <logmsgbot>	 !log reedy@deploy1001 Synchronized langlist-labs: T237823 (duration: 00m 54s)
[20:25:44] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log
[20:25:46] <stashbot>	 T237823: cs.wikipedia is outside main matrix - https://phabricator.wikimedia.org/T237823
[21:03:05] <wikibugs>	 (03PS1) 10Tks4Fish: Add right "abusefilter-log-private" to usergroup "rollbacker" at ptwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/549987 (https://phabricator.wikimedia.org/T237830)
[21:10:29] <wikibugs>	 (03CR) 10Zoranzoki21: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/549987 (https://phabricator.wikimedia.org/T237830) (owner: 10Tks4Fish)
[21:53:20] <wikibugs>	 10Operations, 10Wikimedia-Mailing-lists: Have a conversation about migrating from GNU Mailman 2.1 to GNU Mailman 3.0 - https://phabricator.wikimedia.org/T52864 (10bd808) >>! In T52864#3925248, @Legoktm wrote: > I briefly talked with @herron about this today. I think we are still blocked on the lack of Debian p...