[00:01:50] RECOVERY - cp8 Disk Space on cp8 is OK: DISK OK - free space: / 4003 MB (20% inode=93%); [00:13:25] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/Jvrr6 [00:13:27] [02miraheze/puppet] 07paladox 03f5426a9 - varnish: Switch everything to the new backends [00:13:29] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS [00:13:30] [02puppet] 07paladox opened pull request 03#1281: varnish: Switch everything to the new backends - 13https://git.io/Jvrri [00:15:22] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jvrr1 [00:15:23] [02miraheze/dns] 07paladox 0399d14a7 - dns: Switch everything to use the new cache proxy for europe (cp7) [00:15:25] [02dns] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbQXl [00:15:26] [02dns] 07paladox opened pull request 03#139: dns: Switch everything to use the new cache proxy for europe (cp7) - 13https://git.io/JvrrM [00:16:22] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvrrD [00:16:23] [02miraheze/dns] 07paladox 03f813eb4 - Update miraheze.org [00:16:25] [02dns] 07paladox synchronize pull request 03#139: dns: Switch everything to use the new cache proxy for europe (cp7) - 13https://git.io/JvrrM [00:17:13] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jvrry [00:17:14] [02miraheze/dns] 07paladox 0361cb64f - Update nonbinary.wiki [00:17:16] [02dns] 07paladox synchronize pull request 03#139: dns: Switch everything to use the new cache proxy for europe (cp7) - 13https://git.io/JvrrM [00:18:03] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvrrS [00:18:05] [02miraheze/dns] 07paladox 03665187e - Update allthetropes.org [00:18:06] [02dns] 07paladox synchronize pull request 03#139: dns: Switch everything to use the new cache proxy for europe (cp7) - 13https://git.io/JvrrM [00:18:28] [02dns] 07paladox synchronize pull request 03#139: dns: Switch everything to use the new cache proxy for europe (cp7) - 13https://git.io/JvrrM [00:18:29] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvrrH [00:18:31] [02miraheze/dns] 07paladox 036473949 - Update config [00:18:44] [02dns] 07paladox synchronize pull request 03#139: dns: Switch everything to use the new cache proxy for europe (cp7) - 13https://git.io/JvrrM [00:18:45] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvrrQ [00:18:47] [02miraheze/dns] 07paladox 03e6f3de2 - Update miraheze.org [00:20:11] [02miraheze/dns] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvrrF [00:20:12] [02miraheze/dns] 07paladox 0394c6818 - Update nonciclopedia.org [00:20:14] [02dns] 07paladox synchronize pull request 03#139: dns: Switch everything to use the new cache proxy for europe (cp7) - 13https://git.io/JvrrM [00:21:29] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jvrrb [00:21:30] [02miraheze/mw-config] 07paladox 03a721d54 - Set db4 into read only [00:21:32] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [00:21:33] [02mw-config] 07paladox opened pull request 03#2936: Set db4 into read only - 13https://git.io/JvrrN [00:23:42] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jvrrx [00:23:44] [02miraheze/mw-config] 07paladox 035facc10 - Switch default db to db6 [00:23:45] [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3 [00:23:47] [02mw-config] 07paladox opened pull request 03#2937: Switch default db to db6 - 13https://git.io/Jvrrp [00:41:42] [02DataDump] 07paladox synchronize pull request 03#3: Redesign DataDump - 13https://git.io/JvIMk [00:41:44] [02miraheze/DataDump] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jvroq [00:41:45] [02miraheze/DataDump] 07paladox 035bf4b50 - Update DataDump.php [00:47:01] !log delete db5 - not in use anymore [00:47:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [00:47:32] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvroC [00:47:34] [02miraheze/dns] 07paladox 03b041580 - Remove db5 from dns [00:51:32] [02puppet] 07paladox closed pull request 03#1280: bacula: Remove db5 - 13https://git.io/JvrBv [00:51:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/Jvro0 [00:51:35] [02miraheze/puppet] 07paladox 03a5683c4 - bacula: Remove db5 (#1280) * bacula: Remove db5 * Update director.pp * Update nrpe.cfg.erb [00:53:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://git.io/Jvrou [00:53:14] [02miraheze/puppet] 07paladox 03df486fe - Remove db5 [00:53:16] [02puppet] 07paladox created branch 03paladox-patch-11 - 13https://git.io/vbiAS [00:53:17] [02puppet] 07paladox opened pull request 03#1282: Remove db5 - 13https://git.io/Jvroz [00:53:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://git.io/Jvrog [00:54:01] [02miraheze/puppet] 07paladox 032ebc952 - Update groups.conf [00:54:02] [02puppet] 07paladox synchronize pull request 03#1282: Remove db5 - 13https://git.io/Jvroz [00:54:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://git.io/Jvroa [00:54:19] [02miraheze/puppet] 07paladox 038ae2a79 - Update db.pp [00:54:21] [02puppet] 07paladox synchronize pull request 03#1282: Remove db5 - 13https://git.io/Jvroz [00:54:30] [02puppet] 07paladox closed pull request 03#1282: Remove db5 - 13https://git.io/Jvroz [00:54:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JvroV [00:54:33] [02miraheze/puppet] 07paladox 03f605d61 - Remove db5 (#1282) * Remove db5 * Update groups.conf * Update db.pp [00:54:34] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-11 [00:54:36] [02puppet] 07paladox deleted branch 03paladox-patch-11 - 13https://git.io/vbiAS [00:56:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvror [00:56:35] [02miraheze/puppet] 07paladox 034e23e57 - Update ns2.yaml [04:43:10] PROBLEM - cp8 Stunnel Http for test2 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [04:46:39] PROBLEM - cp8 Stunnel Http for test2 on cp8 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 322 bytes in 0.001 second response time [05:01:18] PROBLEM - cp8 Varnish Backends on cp8 is CRITICAL: 2 backends are down. mw4 mw7 [05:01:33] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 2 backends are down. mw4 mw7 [05:03:13] RECOVERY - cp8 Varnish Backends on cp8 is OK: All 12 backends are healthy [05:03:31] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 12 backends are healthy [05:31:56] PROBLEM - wiki.apap04.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.apap04.com' expires in 15 day(s) (Tue 24 Mar 2020 05:29:56 AM GMT +0000). [05:39:14] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvrXK [05:39:16] [02miraheze/ssl] 07MirahezeSSLBot 0365848ff - Bot: Update SSL cert for wiki.apap04.com [05:53:19] RECOVERY - wiki.apap04.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.apap04.com' will expire on Sat 06 Jun 2020 04:39:08 AM GMT +0000. [06:17:26] PROBLEM - oecumene.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'oecumene.org' expires in 15 day(s) (Tue 24 Mar 2020 06:14:27 AM GMT +0000). [06:21:33] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvr1m [06:21:35] [02miraheze/ssl] 07MirahezeSSLBot 0327010f0 - Bot: Update SSL cert for oecumene.org [06:26:51] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3376 MB (14% inode=94%); [06:33:21] RECOVERY - oecumene.org - LetsEncrypt on sslhost is OK: OK - Certificate 'oecumene.org' will expire on Sat 06 Jun 2020 05:21:27 AM GMT +0000. [08:04:40] Reminder: Wikis that have not yet been migrated to the new infra will be migrated today starting 1pm UTC until 11pm UTC. You will not be able to edit affected wikis during this time. Various Sysadmins and myself will be around at points to assist you and keep you up to date. If you have any questions, please ask. [11:52:07] Morning Reception123 [12:02:08] paladox, SPF|Cloud: are we stopping new wiki creations? [12:04:58] Yes [12:06:07] PROBLEM - wiki.ciptamedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ciptamedia.org' expires in 15 day(s) (Tue 24 Mar 2020 12:02:31 PM GMT +0000). [12:07:40] PROBLEM - wiki.gesamtschule-nordkirchen.de - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.gesamtschule-nordkirchen.de' expires in 15 day(s) (Tue 24 Mar 2020 12:05:28 PM GMT +0000). [12:08:22] paladox: should that not already be done? [12:08:54] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvrHK [12:08:56] [02miraheze/mw-config] 07paladox 03dac6534 - metawiki: Disable WC [12:13:21] PROBLEM - adadevelopersacademy.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'adadevelopersacademy.wiki' expires in 15 day(s) (Tue 24 Mar 2020 12:09:54 PM GMT +0000). [12:13:25] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvrH1 [12:13:26] [02miraheze/ssl] 07MirahezeSSLBot 038e29cb0 - Bot: Update SSL cert for wiki.ciptamedia.org [12:20:00] RhinosF1: hi [12:21:11] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvrHF [12:21:13] [02miraheze/ssl] 07MirahezeSSLBot 03120d338 - Bot: Update SSL cert for adadevelopersacademy.wiki [12:22:33] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvrHp [12:22:34] [02miraheze/ssl] 07MirahezeSSLBot 032898558 - Bot: Update SSL cert for wiki.gesamtschule-nordkirchen.de [12:24:10] RECOVERY - wiki.ciptamedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ciptamedia.org' will expire on Sat 06 Jun 2020 11:13:19 AM GMT +0000. [12:33:21] RECOVERY - adadevelopersacademy.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'adadevelopersacademy.wiki' will expire on Sat 06 Jun 2020 11:21:05 AM GMT +0000. [12:33:41] RECOVERY - wiki.gesamtschule-nordkirchen.de - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.gesamtschule-nordkirchen.de' will expire on Sat 06 Jun 2020 11:22:27 AM GMT +0000. [12:35:03] Good luck everyone involved today! [12:39:14] RhinosF1: thanks [12:39:29] no problem [12:57:48] [02miraheze/mw-config] 07Southparkfan pushed 031 commit to 03Southparkfan-patch-1 [+0/-0/±1] 13https://git.io/JvrQF [12:57:49] [02miraheze/mw-config] 07Southparkfan 03b69a390 - Put c1 cluster in read-only [12:57:51] [02mw-config] 07Southparkfan created branch 03Southparkfan-patch-1 - 13https://git.io/vbvb3 [12:57:52] [02mw-config] 07Southparkfan opened pull request 03#2938: Put c1 cluster in read-only - 13https://git.io/JvrQb [13:00:08] Let’s go! - https://youtu.be/LYZ6gXWZfZM [13:00:09] [ YouTube ] - youtu.be [13:00:41] [02mw-config] 07Southparkfan closed pull request 03#2938: Put c1 cluster in read-only - 13https://git.io/JvrQb [13:00:43] [02miraheze/mw-config] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvr7v [13:00:44] [02miraheze/mw-config] 07Southparkfan 0369cdae3 - Put c1 cluster in read-only (#2938) [13:12:42] !log migration started for first batch [13:12:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:14:51] SPF|Cloud: should meta still be allowing edits? No read only message shows [13:15:12] RhinosF1: yes, saw it [13:16:05] SPF|Cloud set default i guess into read only? [13:16:16] since that shouldn't put db6 into read only [13:16:21] based on how i'm reading this. [13:16:28] yep, you're right [13:16:45] !log cancelled migration, dropping databases on db6 [13:16:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:17:32] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jvr70 [13:17:34] [02miraheze/mw-config] 07paladox 034528a92 - Put default cluster into read only [13:17:35] SPF|Cloud ^ [13:17:48] seems correct [13:17:58] * paladox rolls it out [13:18:45] SPF|Cloud your good to go now [13:18:51] okay, dropped the databases now, db6 is pre-migration state now [13:19:26] seen :) [13:19:48] paladox: that’s showing now correct [13:19:53] yup [13:20:11] !log running first batch, screen on db4: 30692.pts-0.db4 [13:20:56] that didn't log, so seems good :P [13:21:55] edits are definately blocked on meta, I checked [13:23:31] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 4 backends are down. mw4 mw5 mw6 mw7 [13:23:53] PROBLEM - cp4 Stunnel Http for mw7 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:24:04] PROBLEM - cp4 Stunnel Http for mw6 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [13:24:16] !log reboot mw4 [13:24:18] PROBLEM - cp8 Varnish Backends on cp8 is CRITICAL: 2 backends are down. mw5 mw7 [13:24:24] paladox: hm? [13:24:46] SPF|Cloud that'll be related to cloud virt i/o, so i'm trying barrier=0 on mw4 [13:25:12] !log reboot mw5 [13:25:14] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw4 [13:25:49] RECOVERY - cp4 Stunnel Http for mw7 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.079 second response time [13:25:58] RECOVERY - cp4 Stunnel Http for mw6 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 15316 bytes in 0.078 second response time [13:26:27] !log reboot mw6 [13:26:44] !log reboot mw7 [13:28:14] RECOVERY - cp8 Varnish Backends on cp8 is OK: All 12 backends are healthy [13:29:12] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 12 backends are healthy [13:29:30] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 10 backends are healthy [13:47:29] !log abort migration, going to switch to plan B [13:51:16] !log MariaDB [(none)]> set global table_definition_cache=20000; - db6 [13:51:31] paladox: created all_wikis_selection_1.5 [13:51:47] that is list 1 minus the wikis that have been migrated already [13:52:09] ah [13:53:28] !log MariaDB [(none)]> set global table_definition_cache=20000; -db4 [13:55:05] !log running plan B script for batch 1.5 [13:56:15] !log MariaDB [(none)]> set global table_open_cache=20000; -db4 & db6 [13:59:15] !log MariaDB [(none)]> set global table_open_cache=30000; - db4 [13:59:24] !log MariaDB [(none)]> set global table_definition_cache=30000; -db4 [14:04:10] !log MariaDB [(none)]> set global table_definition_cache=40000; - db4 [14:04:15] !log MariaDB [(none)]> set global table_open_cache=40000; -db4 [14:07:57] !log MariaDB [(none)]> set global table_open_cache=50000; -db4 [14:08:11] !log first batch import on db6: 21737.pts-0.db6 [14:08:20] (with first batch being batch 1.5' [14:11:33] !log set global table_open_cache=50000; - db6 [14:11:42] !log set global table_definition_cache=40000; -db6 [14:17:18] !log set global table_open_cache=100000; -db6 (temp) [14:17:53] paladox: we're doing 1530 wikis per hour at this rate :-) [14:18:01] SPF|Cloud awesome :) [14:18:21] SPF|Cloud i'm upping the cache (since that *should* help with performance related to opening tables) [14:18:27] sure [14:19:16] paladox: can you fetch a list for me of misc databases that have to be migrated from db4 too? such as centralauth and mhglobal [14:19:34] SPF|Cloud those dbs have not been migrated. [14:19:50] yes, so I need to know which ones we need [14:19:52] I can tell you i've migrated grafana and phabricator_* and piwik_* [14:20:17] icinga/icinga2/icingaweb2 i haven't migrated [14:20:58] SPF|Cloud ^ [14:21:10] *piwik [14:21:10] aha [14:21:53] paladox: cargodb centralauth fileame globaluser2 icinga icingaweb2 incidents mhglobal roundcubemail survey testdatabase test_test [14:22:11] I suspect those names are the only non-wiki databases on db4 [14:22:23] yeh [14:22:50] can you give me a list of databases that need to be migrated, after the wikis have been migrated? [14:22:55] centralauth, and incidents and mhglobal and cargodb would need to be migrated, the rest can be done at the end. [14:23:11] thanks [14:23:42] SPF|Cloud i guess survey needs to be done too, though i think that was used for last year? [14:26:54] I am not sure, but it's not even half a megabyte large, so migrating it is fine [14:28:19] ok [14:31:05] RhinosF1, paladox: batch 1 imported, doing batch 2 now [14:31:11] \o/ [14:32:17] SPF|Cloud: 23 mins [14:32:40] That's ~1304/hr [14:32:55] looks like we'll eventually catch up [14:33:25] (the coming batches are not exactly 500, rather 501, but should not have too much impact on calculations) [14:33:54] batch size https://www.irccloud.com/pastebin/mOWOWgmz/ [14:33:55] [ Snippet | IRCCloud ] - www.irccloud.com [14:33:59] SPF|Cloud: should take 3.3 hours total [14:35:49] SPF|Cloud: that's 17:26 finish for all wikis + the misc/shared dbs [14:35:55] sounds good [14:36:11] very [14:36:48] good :) [14:38:42] just started dumping batch 4 from db4 to db6, batch 2 running in background on db4 [14:38:46] on db6* [14:39:03] db6 has 760 wiki databases now [14:39:36] SPF|Cloud doin't forget to purge the binary logs regulary! [14:39:45] SPF|Cloud: 260 wikis in 8 mins! [14:39:54] paladox: we have 99G free on db4 [14:40:02] i mean on db6 [14:40:10] have 382G free there [14:40:19] oh, heh [14:40:59] 793 wikis on db6 now, 70 wikis were from the previous migration so 723 migrated today [14:41:12] 723-500=223 from this batch so around half way now [14:42:08] SPF|Cloud: that's going excellent [14:45:25] paladox: what's going on with db4's ram usage? [14:45:33] * paladox looks [14:45:52] oh [14:45:58] i bet this is due to the table cache [14:46:24] !log set global table_open_cache=20000; - db4 [14:46:32] !log set global table_definition_cache=20000; -db4 [14:46:51] !log set global table_definition_cache=20000; -db6 [14:46:54] please decrease the cache on db6 too [14:46:55] !log set global table_open_cache=20000; - db6 [14:47:03] SPF|Cloud done [14:47:19] db4's swapping is not helpful for performance and db6 is close to running out of memory [14:48:00] thanks! [14:48:06] SPF|Cloud does 20,000 sound fine to you? [14:50:26] looking at the related graphs in https://grafana.miraheze.org/d/MQWgroiiz/mysql-overview?orgId=1&refresh=1m, I'm not sure [14:50:26] [ Grafana ] - grafana.miraheze.org [14:52:21] SPF|Cloud: how many wikis done? [14:52:32] 983 [14:52:58] fifth batch dumping to db6 now [14:53:08] Hello Entretenimiento! If you have any questions, feel free to ask and someone should answer soon. [14:53:10] ~1340/hr now [14:53:14] Entretenimiento: hi [14:53:58] if this finishes no later than 7 minutes from now on, we're doing fine per the predictions (one batch per hour) [14:54:14] batch 2 imported, doing batch 3 now [14:54:52] \o/ [14:54:56] SPF|Cloud: based on my live calculations. We'll have migrated all wikis by 17:21 [14:55:04] paladox: is that a known ATS? [14:55:05] :p [14:55:13] two batches imported in 47 minutes [14:55:28] I guess so [14:56:36] paladox: guess? I hope so [14:57:13] Either way, that user keeps asking that, so we just kick/ban them. Because we've told them already we're not launching that yet they still ask the same thing. [14:57:37] paladox: ah, I must have missed them using that name [14:57:41] but we really should not be talking about trolls/lta in this channel or in any public venue for that matter, it only encourages them to do this. [14:58:04] RhinosF1 not that name, but the question is the same [14:58:11] always "miraheze toolforge" [14:58:44] paladox: I just remember. You’ve quieted unidentified. [14:59:10] quieted unidentified? [14:59:55] Yeah, I can’t see his message [14:59:59] I was confused [15:00:09] oh [15:00:31] from this batch, 142 wikis done [15:00:48] :) [15:01:07] SPF|Cloud: is that 1142 total then? [15:01:23] 142 + 501 + 500 + 70 [15:01:34] From today I meant [15:01:43] oh, yes, it is [15:01:49] (approximately) [15:01:53] 1143* [15:05:49] dumping batch 6 now [15:07:05] SPF|Cloud: I’m getting 17:32 now by my maths [15:07:11] wow [15:07:13] As at 3pm [15:15:31] batch 3 done, importing abtch 4 [15:16:09] .in 6h Globally block latest LTA ips on-wiki [15:16:09] Zppix: Okay, I will set the reminder for: 2020-03-08 - 16:16:09CDT [15:16:38] SPF|Cloud: is that 1500 today then? [15:16:56] yep [15:18:45] SPF|Cloud: that makes it all wikis by 17:27 [15:19:03] good to know we're not fluctuating too much regarding estimations [15:19:10] all of the seem to be around ~17:30 [15:19:35] Yeah, they’re staying fairly stable [15:23:25] dumping batch 7 [15:34:15] \o/ [15:35:00] dumping batch 9 [15:35:02] 8* [15:37:49] batch 4 done, doing 5 [15:38:49] \o/ [15:39:14] SPF|Cloud: works out as 17:32 again [15:39:41] going much better than the previous migration [15:39:48] By far [15:41:54] paladox: I think we need to restart mariadb on both servers after this batch has been imported [15:42:01] sure [15:42:17] the amount of memory it's using is not making me feel comfortable [15:42:25] SPF|Cloud we could set barrier=0 i guess on db6 too? [15:42:36] it isn't there yet? [15:42:54] well i've put it in fstab, but requires restarting the vm. [15:43:14] server is performing fine, though? [15:43:43] yeh, seems so. I guess we can see how it handles things after we make the flip. [15:45:01] disabling barrier does not seem that safe [15:45:38] SPF|Cloud technically that's already been done (as we've set unsafe cach on the vm in proxmox setting) [15:46:53] I know, but don't barriers still provide some guard of safety despite said setting? [15:47:40] Well no, not really. If the power goes off that vm will be corrupted due to the unsafe cache being on. [15:48:09] so setting either having it or not, dosen't really matter in terms of date integrety. Unless i'm wrong. [15:49:42] SPF|Cloud: how many done? [15:50:37] that's possible if there are writes that were marked as done on system level but not actually written to disks (that's why enterprise servers usually have BBUs in their controllers or expensive SSDs with some sort of power loss protections) [15:51:08] question is whether the presence of barriers have an impact on safety [15:51:26] RhinosF1: 2277 [15:51:29] it would, but it's exactly the same as the unsafe cache. [15:53:05] batch 8 dumped, but skipping 9 for now until mariadb has been restarted [15:53:13] SPF|Cloud: 17:27 still [15:53:20] Back in ~30 mins [15:53:20] :o [15:56:05] SPF|Cloud Should we make the buffer pool smaller? [15:59:08] paladox: ugh, cannot really find the answers for those ext4 questions, whatever [15:59:18] ok [15:59:59] and I don't thinkt the buffer pool is too large, 75% is fine [16:00:14] ok, though doin't we need enough for the open table cache? [16:00:23] I mean i guess we need this at like 30 or even 40,000? [16:01:05] doesn't look like that's needed [16:02:00] ok [16:02:15] but I suggest we follow the tutorial (https://mariadb.com/kb/en/optimizing-table_open_cache/) after the migration has been done [16:02:16] [ Optimizing table_open_cache - MariaDB Knowledge Base ] - mariadb.com [16:02:51] paladox: please announce the reboots [16:03:26] Ok, SPF|Cloud do i say a time? Or just say "Hello all, we are planning to reboot the mysql servers in a bit." [16:05:06] * paladox posts ^ [16:06:04] well, batch 5 just finished [16:06:38] let's announce through site notices and social media that all wikis will go offline at 16:30 UTC for around 15 minutes? [16:07:01] ok [16:07:11] SPF|Cloud i carn't do twitter, but can do fb. [16:07:11] and remind people to save their edits ;) [16:08:42] SPF|Cloud are you doing the site notice? :) [16:08:48] yes [16:09:12] :) [16:11:53] [02miraheze/mw-config] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvrbM [16:11:55] [02miraheze/mw-config] 07Southparkfan 034ff1286 - Add site notice for maintenance [16:14:35] SPF|Cloud: how many done today? [16:14:55] SPF|Cloud posted on fb [16:16:01] and deployed [16:17:54] PROBLEM - cp8 Varnish Backends on cp8 is CRITICAL: 6 backends are down. mw1 mw2 mw3 mw4 mw5 mw6 [16:18:05] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 6 backends are down. mw1 mw2 mw3 mw4 mw5 mw6 [16:18:10] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 6 backends are down. mw1 mw2 mw3 mw4 mw5 mw6 [16:18:12] paladox: ^ [16:18:26] i'm taking that was db4 [16:18:41] but seems back [16:19:34] so: after the reboot has been done, we need to: [16:19:34] - dump batch 9 [16:19:34] - import batches 6-9 [16:20:10] SPF|Cloud so service mysql stop ; reboot [16:20:35] yep [16:24:15] PROBLEM - mw3 MediaWiki Rendering on mw3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4235 bytes in 0.504 second response time [16:24:28] PROBLEM - lizardfs6 MediaWiki Rendering on lizardfs6 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4237 bytes in 0.398 second response time [16:24:38] PROBLEM - test1 MediaWiki Rendering on test1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4237 bytes in 0.391 second response time [16:25:02] SPF|Cloud: have we stopped importing until after the restart? [16:25:23] yes [16:25:34] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is WARNING: WARNING - NGINX Error Rate is 52% [16:25:43] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4235 bytes in 0.542 second response time [16:25:43] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4235 bytes in 0.421 second response time [16:25:44] PROBLEM - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is CRITICAL: CRITICAL - NGINX Error Rate is 73% [16:25:56] what the..? [16:26:10] Hell? [16:26:31] [02miraheze/mw-config] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvrNf [16:26:32] [02miraheze/mw-config] 07Southparkfan 03a26df7d - Fix double declaration [16:27:21] should be fixed now [16:27:34] RECOVERY - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is OK: OK - NGINX Error Rate is 37% [16:27:41] RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 20030 bytes in 0.991 second response time [16:27:43] RECOVERY - mw1 MediaWiki Rendering on mw1 is OK: HTTP OK: HTTP/1.1 200 OK - 20029 bytes in 0.932 second response time [16:27:44] RECOVERY - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is OK: OK - NGINX Error Rate is 2% [16:28:13] RECOVERY - mw3 MediaWiki Rendering on mw3 is OK: HTTP OK: HTTP/1.1 200 OK - 20028 bytes in 0.901 second response time [16:28:21] RECOVERY - lizardfs6 MediaWiki Rendering on lizardfs6 is OK: HTTP OK: HTTP/1.1 200 OK - 20056 bytes in 0.909 second response time [16:28:35] RECOVERY - test1 MediaWiki Rendering on test1 is OK: HTTP OK: HTTP/1.1 200 OK - 20029 bytes in 0.884 second response time [16:31:26] stopping mysql on db4 and db6 [16:33:10] rebooting db4 [16:33:16] SPF|Cloud fyi i have barrier=0 set in fstab on db6 [16:33:25] just in case you wanted me to remove it. [16:33:53] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4235 bytes in 0.416 second response time [16:33:58] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4235 bytes in 0.422 second response time [16:34:00] PROBLEM - cp8 HTTP 4xx/5xx ERROR Rate on cp8 is CRITICAL: CRITICAL - NGINX Error Rate is 83% [16:34:12] !log stopping ircecho temp [16:34:18] PROBLEM - mw3 MediaWiki Rendering on mw3 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Backend fetch failed - 4235 bytes in 0.412 second response time [16:34:50] rebooting db6 [16:39:58] and we're back :) [16:41:30] SPF|Cloud: let me know when restart imports [16:41:44] I will [16:44:06] !log reboot mw4 [16:46:42] [02miraheze/mw-config] 07Southparkfan pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvrNE [16:46:43] [02miraheze/mw-config] 07Southparkfan 037ea1ad2 - Remove site notice [16:50:52] starting dump of batch 9 [16:51:55] !log set open cache to 20000 on db6 again [16:52:12] paladox: why? [16:52:14] RhinosF1: importing batch 6 now [16:52:23] SPF|Cloud that's what i had set before. [16:52:55] SPF|Cloud: should be done for 18:29 [16:53:06] :D [16:53:50] SPF|Cloud: let’s see how many it imports in the next 15-20 mins to see if there’s any speed change [16:55:51] :O 382 kB/s that's terrible, i think barrier=0 is performing worse for us? [16:56:01] i ran `dd if=/dev/zero of=/root/testfile bs=512 count=1000 oflag=dsync ; rm /root/testfile` [16:56:14] oh [16:56:15] 2.3 MB/s [17:00:30] and batch 9 has been dumped [17:00:45] \o/ [17:00:50] Cool [17:01:04] db6's load either goes down or the import goes faster [17:09:26] SPF|Cloud: how many done since reboot? [17:10:26] about 425 wikis [17:13:32] SPF|Cloud: That’s showing it’s going quicker [17:14:27] batch 6 done, importing 7 [17:15:23] Total DISK READ: 2.06 M/s | Total DISK WRITE: 102.15 M/s [17:15:23] Current DISK READ: 2.03 M/s | Current DISK WRITE: 83.31 M/s [17:16:13] SPF|Cloud: 18:11 [17:21:51] it's admirable wikis are still online [17:21:59] Yep [17:22:06]