[10:58:18] !log admin icinga downtime for 1h (T227540) cloudvirt100[3-7], cloudvirt1019, cloudvirt1016, cloudvirt1021, cloudvirt1013, cloudnet1004 [10:58:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:58:23] T227540: b4-eqiad pdu refresh (Thursday 10/24 @11am UTC) - https://phabricator.wikimedia.org/T227540 [11:04:54] !log clouddb-services stopped mariadb in clouddb1001 (T227540) [11:04:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Clouddb-services/SAL [11:04:58] T227540: b4-eqiad pdu refresh (Thursday 10/24 @11am UTC) - https://phabricator.wikimedia.org/T227540 [11:09:09] !log clouddb-services stopped postgresl in clouddb1004 (T227540) [11:09:12] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Clouddb-services/SAL [11:10:13] !log admin icinga downtime for 2h (T227540) toolschecker [11:10:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [11:10:17] T227540: b4-eqiad pdu refresh (Thursday 10/24 @11am UTC) - https://phabricator.wikimedia.org/T227540 [11:13:52] !log clouddb-services poweroff VM clouddb1004 (T227540) [11:13:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Clouddb-services/SAL [11:14:59] !log clouddb-services poweroff VM clouddb1001, hypervisor will be powered off (T227540) [11:15:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Clouddb-services/SAL [11:15:40] !log admin poweroff cloudvirt1019 during the PDU operations (T227540) [11:15:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [11:15:45] T227540: b4-eqiad pdu refresh (Thursday 10/24 @11am UTC) - https://phabricator.wikimedia.org/T227540 [11:58:02] !log admin icinga downtime for 2h (T227540) cloudvirt1019 [11:58:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [11:58:06] T227540: b4-eqiad pdu refresh (Thursday 10/24 @11am UTC) - https://phabricator.wikimedia.org/T227540 [12:30:24] !log admin starting cloudvirt1019, PDU operations ended (T227540) [12:30:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [12:30:31] T227540: b4-eqiad pdu refresh (Thursday 10/24 @11am UTC) - https://phabricator.wikimedia.org/T227540 [12:34:17] !log clouddb-services start both clouddb1001 and clouddb1004 (T227540) [12:34:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Clouddb-services/SAL [13:27:47] !log clouddb-services add phamhi as user and projectadmin [13:27:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Clouddb-services/SAL [13:52:20] Hmm … timeout when opening a wiki-website (de.wikipedia.org) via Amsterdam cluster – direct connection to 208.80.154.224 works [14:00:28] Wurgl: #wikimedia-operations or #wikimedia-traffic are probably better places to report connectivity issues to the wikis [14:06:11] bd808: problem is gone … [15:02:42] #wikimedia-cloud Wikimedia Cloud Services (wikitech.wikimedia.org) | Status: Toolsdb unstable after PDU operations | Ask questions here, but please provide links and context. Use "!help" if nobody responds | More details and channel logs at https://wikitech.wikimedia.org/wiki/Help:IRC | Code of Conduct applies: https://www.mediawiki.org/wiki/Code_of_Conduct [15:02:51] oops [16:03:28] !log clouddb-services stopped puppet on clouddb1002 and removed unattended-upgrades for now T236384 [16:03:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Clouddb-services/SAL [16:03:31] T236384: Toolsdb: prevent unattended-upgrades from upgrading mariadb - https://phabricator.wikimedia.org/T236384 [16:32:25] !log tools set the prod rsyslog config for kubernetes to false for Toolforge [16:32:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [16:59:53] !log tools.paws moving db to sqlite per https://wikitech.wikimedia.org/w/index.php?title=PAWS/Tools/Admin#Moving_to_sqlite until toolsdb is stable [16:59:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.paws/SAL [17:50:05] Good move chicocvenancio...sorry about the mess. [18:10:56] !log tools.deadlinks bd808 hacked public_html/api/index.php to stop all db writes. Trying to slow toolsdb high volume writes while we work on fixing a bad software update (see also T236384) [18:11:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.deadlinks/SAL [18:11:02] T236384: Toolsdb: prevent unattended-upgrades from upgrading mariadb - https://phabricator.wikimedia.org/T236384 [18:11:03] bstorm_: ^ one down I hope [18:19:05] !log clouddb-services stopped puppet on clouddb1001 and removed unattended-upgrades for now T236384 [18:19:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Clouddb-services/SAL [18:19:09] T236384: Toolsdb: prevent unattended-upgrades from upgrading mariadb - https://phabricator.wikimedia.org/T236384 [18:34:51] !log clouddb-services downgraded clouddb1001 to 10.1.39 T236420 T236384 [18:34:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Clouddb-services/SAL [18:34:56] T236420: ToolsDB unstable following unplanned software upgrade - https://phabricator.wikimedia.org/T236420 [18:34:57] T236384: Toolsdb: prevent unattended-upgrades from upgrading mariadb - https://phabricator.wikimedia.org/T236384 [20:58:54] !log tools.wd-image-positions deployed b5dd0fdb31 (bugfix) [20:58:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wd-image-positions/SAL [21:53:48] How can I reset my wikitech 2fa? [21:54:03] No scratch codes [22:01:44] matanya: phab task like this one -- https://phabricator.wikimedia.org/T236242 [22:06:51] Thanks [22:09:07] https://phabricator.wikimedia.org/T236441 bd808 [22:09:11] and thank you [22:10:47] matanya: https://phabricator.wikimedia.org/T236441#5604839 -- unless you want to do a video call with me which would also work in this case [22:17:38] bd808 saved you from seeing my face :) [22:20:46] matanya: {{done}] and I'm sure I will see your face again one of these days :) [22:20:49] Thank you for the reset [22:21:07] and you are probably right :D [23:06:56] bd808 seems like I need a reset for horizon as well, same process? [23:09:24] andrewbogott I am planning to attempt to migrate the video project to buster over the weekend, this will require resourcing, as you might recall the transcoding nodes are huge [23:09:59] this is a heads up for you in case you suddenly see a spike in resource utilization on cloud vps [23:10:18] If this plan is problematic in any way, please let me know [23:11:31] (And bd808, never mind, figured out horizon on my own) [23:40:32] matanya: that should be fine, but thanks for the warning. I'm tempted to use this as a test for some questionable hardware but I'll wait and see what Brooke thinks about that tomorrow. [23:40:59] (since I assume that if a video rip fails you can always just start it over) [23:42:01] I am willing to do some experimental tests on it [23:43:03] I also opened a limits increase ticket andrewbogott which is a prerequisite to the work, due to being at limit now [23:43:42] matanya: ok — I'll see if we can get that approved before our standard meeting [23:44:12] Thanks again