[00:02:07] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [00:45:10] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 3 minutes ago with 9 failures. Failed resources (up to 3 shown): Service[salt-minion],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666] [00:55:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JedDi [00:55:09] [02miraheze/services] 07MirahezeSSLBot 03eddb062 - BOT: Updating services config for wikis [01:03:10] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:45:51] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 4 minutes ago with 10 failures. Failed resources (up to 3 shown): Service[bacula-fd],Service[salt-minion],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22] [01:53:22] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:26:00] !log /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/maintenance/sql.php /home/paladox/comments.sql (test1) [02:26:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [02:36:33] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 5 minutes ago with 9 failures. Failed resources (up to 3 shown): Service[bacula-fd],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666] [02:58:08] !log rebuild rc and initsitestats for uncyclomirrorwiki after import completion [02:58:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [03:02:02] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [03:15:36] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 4 failures. Last run 4 minutes ago with 4 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [03:20:22] PROBLEM - db4 Disk Space on db4 is CRITICAL: DISK CRITICAL - free space: / 21980 MB (5% inode=95%); [03:23:08] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:28:13] !log /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/maintenance/sql.php /home/paladox/aft.sql (test1) [03:28:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [03:36:34] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ufw-enable] [03:41:46] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 16 seconds ago with 0 failures [04:01:19] !log depool lizardfs6 [04:01:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [04:08:37] !log hacked mw* config (redis) and undone hack [04:14:50] !log hacked mw* config (redis) and undone hack [04:14:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [04:25:45] PROBLEM - cp3 Disk Space on cp3 is CRITICAL: DISK CRITICAL - free space: / 1435 MB (5% inode=94%); [04:28:28] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 1475 MB (6% inode=94%); [04:37:10] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 5 minutes ago with 9 failures. Failed resources (up to 3 shown): Service[salt-minion],Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666] [04:42:26] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [05:04:55] PROBLEM - cp3 Disk Space on cp3 is CRITICAL: DISK CRITICAL - free space: / 1445 MB (5% inode=94%); [05:26:28] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 4 minutes ago with 6 failures. Failed resources (up to 3 shown): Exec[ufw-logging-low],Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [05:52:11] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [06:16:42] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-185.52.1.76-to-any-port-8081] [06:18:11] PROBLEM - mw3 Puppet on mw3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [06:24:23] PROBLEM - db4 Puppet on db4 is CRITICAL: CRITICAL: Puppet has 17 failures. Last run 3 minutes ago with 17 failures. Failed resources (up to 3 shown) [06:24:43] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [06:24:44] PROBLEM - misc4 Puppet on misc4 is CRITICAL: CRITICAL: Puppet has 32 failures. Last run 3 minutes ago with 32 failures. Failed resources (up to 3 shown) [06:24:56] PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Puppet has 430 failures. Last run 3 minutes ago with 430 failures. Failed resources (up to 3 shown) [06:25:07] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CRITICAL: Puppet has 13 failures. Last run 4 minutes ago with 13 failures. Failed resources (up to 3 shown) [06:25:36] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 1994 MB (8% inode=94%); [06:25:37] PROBLEM - misc1 Puppet on misc1 is CRITICAL: CRITICAL: Puppet has 51 failures. Last run 4 minutes ago with 51 failures. Failed resources (up to 3 shown) [06:25:37] PROBLEM - db5 Puppet on db5 is CRITICAL: CRITICAL: Puppet has 15 failures. Last run 4 minutes ago with 15 failures. Failed resources (up to 3 shown) [06:25:49] PROBLEM - misc2 Puppet on misc2 is CRITICAL: CRITICAL: Puppet has 29 failures. Last run 4 minutes ago with 29 failures. Failed resources (up to 3 shown) [06:25:55] PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CRITICAL: Puppet has 14 failures. Last run 4 minutes ago with 14 failures. Failed resources (up to 3 shown) [06:26:06] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [06:26:31] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [06:26:33] PROBLEM - misc3 Puppet on misc3 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [06:26:36] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 239 failures. Last run 5 minutes ago with 239 failures. Failed resources (up to 3 shown) [06:33:16] RECOVERY - db5 Puppet on db5 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [06:33:28] RECOVERY - misc1 Puppet on misc1 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [06:33:29] RECOVERY - misc2 Puppet on misc2 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [06:33:37] RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [06:34:00] RECOVERY - cp4 Puppet on cp4 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [06:34:25] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [06:34:27] RECOVERY - misc3 Puppet on misc3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:34:32] RECOVERY - db4 Puppet on db4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:34:36] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [06:34:37] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:34:52] RECOVERY - misc4 Puppet on misc4 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:34:55] RECOVERY - ns1 Puppet on ns1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [06:35:09] RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:36:27] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [06:37:09] RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [09:08:05] Reception123, SPF|Cloud, PuppyKun, paladox: cp2 is marked as down on icinga [09:09:48] Has been since 21/12/2019 (yesterday) 18:04 strangely [09:09:59] CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [09:10:07] On all services [09:32:53] SantaRhino: This virtual server has been suspended by the administrator. Please contact support [09:48:23] Reception123: bandwidth? [09:48:54] SantaRhino: probably, see PM [10:25:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jed7K [10:25:08] [02miraheze/services] 07MirahezeSSLBot 03d83f7f8 - BOT: Updating services config for wikis [12:00:52] PROBLEM - mw2 MediaWiki Rendering on mw2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:03:50] RECOVERY - mw2 MediaWiki Rendering on mw2 is OK: HTTP OK: HTTP/1.1 200 OK - 19066 bytes in 3.635 second response time [12:31:43] SantaRhino: yes me and John are aware [12:31:44] Reception123: ^ [12:35:53] paladox: okay, doesn’t seem normal to have a server down. Wondered why no one had noticed [12:37:20] Indeed, we are monitoring. Since we have to pay for extra bandwidth. [12:38:00] paladox: I thought it’d be bandwidth, they said that last time right? [12:39:39] Yup [12:40:01] paladox: you’ll have to talk to someone about finances then [12:40:23] From my understanding SPF|Cloud would probably be the go to for this [12:41:42] I spoke with John already.... [12:41:43] He told me to monitor [12:42:18] paladox: by monitor, do you mean speeds? Complaining users? Cause it’s 100% down [12:43:30] !log repool lizardfs6 [12:44:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [12:44:54] [02miraheze/ssl] 07Reception123 pushed 033 commits to 03master [+3/-0/±3] 13https://git.io/JeddW [12:44:55] [02miraheze/ssl] 07Reception123 0315f204f - add wiki.cdntennis.ca cert [12:44:57] [02miraheze/ssl] 07Reception123 03af9e9da - add wiki.iufs.cf cert [12:44:58] [02miraheze/ssl] 07Reception123 03a620e32 - add vault.aics.cf cert [12:46:29] No, bandwith on cp4 [12:46:31] I already spoke to void who said he said the speeds were about normal [12:46:33] SantaRhino: ^ [12:47:28] paladox: ok, so we just hope no other cp does [12:47:30] Does [12:47:32] Dies [12:47:44] yeh [12:48:21] Does it recover on the next billing cycle or? [12:48:38] yes [12:49:19] At least that’s only 9 days then and by then Feran should have done the budgets [12:50:52] 9 days? [12:51:15] Billing cycle is monthly right? [12:51:25] 31st is the end of the month [12:51:29] It’s 22nd [12:51:34] That’s 9 days [12:51:43] no that's not how it works, it's monthly but from the day you got it [12:51:57] paladox: oh, when is that then? [12:52:04] i'm not sure [12:54:52] paladox: Finance shows the charge as being between now and about the 27th anyway [12:54:53] !log depool, repool mw3, restart php7.3-fpm [12:55:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [12:55:13] where does it show cp2 on finance? [12:55:45] paladox: I’m looking at mentions of RamNode [12:56:33] ok [12:57:03] !log depool, repool mw2, restart php7.3-fpm [12:57:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [12:58:48] paladox: can’t tell which is cp2 [12:59:04] Although the payment schedule seems to change towards end of year [12:59:10] ok [13:00:16] And yep it did [13:11:46] !log ulimit -n on misc2 [13:11:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:12:01] !log ulimit -Sn 100000 [13:12:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:17:23] !log 127.0.0.1:6379> CONFIG SET timeout 5 - misc2 [13:17:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:18:20] [02mw-config] 07Reception123 closed pull request 03#2826: add wgPopupsGateway to ManageWiki; fixes T4855 - 13https://git.io/Je5AY [13:18:22] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeddQ [13:18:23] [02miraheze/mw-config] 07The-Voidwalker 0382f52a1 - add wgPopupsGateway to ManageWiki; fixes T4855 (#2826) [13:22:25] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03revert-2826-patch-64 [+0/-0/±1] 13https://git.io/Jedd5 [13:22:26] [02miraheze/mw-config] 07Reception123 031158d8b - Revert "add wgPopupsGateway to ManageWiki; fixes T4855 (#2826)" This reverts commit 82f52a18b2f5f16d40f09741361bed1311468e7f. [13:22:28] [02mw-config] 07Reception123 created branch 03revert-2826-patch-64 - 13https://git.io/vbvb3 [13:22:44] [02mw-config] 07Reception123 opened pull request 03#2827: Revert "add wgPopupsGateway to ManageWiki; fixes T4855" - 13https://git.io/Jeddd [13:22:48] [02mw-config] 07Reception123 closed pull request 03#2827: Revert "add wgPopupsGateway to ManageWiki; fixes T4855" - 13https://git.io/Jeddd [13:22:50] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeddF [13:22:51] [02miraheze/mw-config] 07Reception123 03a9ed5e8 - Revert "add wgPopupsGateway to ManageWiki; fixes T4855 (#2826)" (#2827) This reverts commit 82f52a18b2f5f16d40f09741361bed1311468e7f. [13:23:42] miraheze/mw-config/revert-2826-patch-64/1158d8b - Reception123 The build passed. https://travis-ci.org/miraheze/mw-config/builds/628359568 [13:23:46] Reception123: ? [13:25:52] [02miraheze/ssl] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/Jeddb [13:25:53] [02miraheze/ssl] 07Reception123 03e701ef8 - add mobile for wiki.iufs.cf cert [13:25:55] SantaRhino: it required restbase [13:26:01] but I didn't know since there was no warning on the PR [13:26:58] Reception123: ah, course [13:27:11] * SantaRhino looks at paladox [13:35:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JedFt [13:35:08] [02miraheze/services] 07MirahezeSSLBot 035083da8 - BOT: Updating services config for wikis [13:45:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JedF4 [13:45:08] [02miraheze/services] 07MirahezeSSLBot 03d2eb8ee - BOT: Updating services config for wikis [14:07:03] * hispano76 greetings [14:15:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JedFx [14:15:08] [02miraheze/services] 07MirahezeSSLBot 039500f8d - BOT: Updating services config for wikis [15:44:53] PROBLEM - misc2 Redis Process on misc2 is CRITICAL: PROCS CRITICAL: 2 processes with args 'redis-server' [15:46:36] RECOVERY - misc2 Redis Process on misc2 is OK: PROCS OK: 1 process with args 'redis-server' [15:55:52] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100] [16:03:28] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:43:38] [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JedAL [16:43:39] [02miraheze/ManageWiki] 07JohnFLewis 0300a8a68 - move around how the $ignorePerms if works [16:43:44] paladox [16:43:49] ssh gerrit-te\o/ [16:43:53] \o/ [16:44:11] JohnLewis thanks! [16:44:25] JohnLewis i'm just going to merge in the work from the other branch. [16:46:08] [02miraheze/ManageWiki] 07paladox pushed 037 commits to 03master [+0/-0/±13] 13https://git.io/JedAt [16:46:10] [02miraheze/ManageWiki] 07paladox 03360a7ad - Merge branch 'DBConnRef' * DBConnRef: the -> this in some contexts DBConnRef -> MaintainableDBConnRef (#124) Update ManageWikiFormFactory.php Fix "ManageWikiFormFactory::getFormDescriptor() must be an instance of Wikimedia\Rdbms\Database" DB_SLAVE -> DB_REPLICA Database -> DBConnRef [16:46:36] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JedAq [16:46:38] [02miraheze/mediawiki] 07paladox 0398fc36f - Update ManageWiki [16:46:52] RECOVERY - test1 Puppet on test1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:54:44] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JedAc [16:54:46] [02miraheze/ManageWiki] 07paladox 030378dd3 - Remove unused variable ($remoteWiki) from ::extensions [16:55:33] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JedAC [16:55:34] [02miraheze/mediawiki] 07paladox 03d8b740f - Update ManageWiki [17:04:04] [02miraheze/DataDump] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JedAz [17:04:06] [02miraheze/DataDump] 07paladox 03bcb09a7 - Replace deprecated/removed getTitle function with getPageTitle [17:04:34] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JedA2 [17:04:36] [02miraheze/mediawiki] 07paladox 03bbcbbd1 - Update DataDump [17:06:20] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JedAV [17:06:21] [02miraheze/CreateWiki] 07paladox 03c580bdc - Replace deprecated/removed getTitle function with getPageTitle [17:06:23] [02CreateWiki] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vpJTL [17:06:24] [02CreateWiki] 07paladox opened pull request 03#125: Replace deprecated/removed getTitle function with getPageTitle - 13https://git.io/JedAw [17:07:14] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JedAi [17:07:16] [02miraheze/CreateWiki] 07paladox 03e6ea1b0 - Replace deprecated/removed getTitle function with getPageTitle [17:07:17] [02CreateWiki] 07paladox synchronize pull request 03#125: Replace deprecated/removed getTitle function with getPageTitle - 13https://git.io/JedAw [17:10:34] [02CreateWiki] 07paladox closed pull request 03#125: Replace deprecated/removed getTitle function with getPageTitle - 13https://git.io/JedAw [17:10:36] [02miraheze/CreateWiki] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/JedA7 [17:10:37] [02miraheze/CreateWiki] 07paladox 03629bd79 - Replace deprecated/removed getTitle function with getPageTitle (#125) * Replace deprecated/removed getTitle function with getPageTitle * Replace deprecated/removed getTitle function with getPageTitle [17:11:09] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JedA5 [17:11:11] [02miraheze/mediawiki] 07paladox 03eb921f6 - Update CreateWiki [17:11:21] [02CreateWiki] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vpJTL [17:11:23] [02miraheze/CreateWiki] 07paladox deleted branch 03paladox-patch-1 [17:12:28] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JedAN [17:12:29] [02miraheze/ManageWiki] 07paladox 03081efbd - Replace deprecated/removed getTitle function with getPageTitle [17:13:25] [02miraheze/IncidentReporting] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JedAh [17:13:27] [02miraheze/IncidentReporting] 07paladox 035a3e1a3 - Replace deprecated/removed getTitle function with getPageTitle [17:13:54] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±2] 13https://git.io/JedAj [17:13:56] [02miraheze/mediawiki] 07paladox 03235c4a3 - Update MW and IR [17:16:09] [02miraheze/IncidentReporting] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JedxU [17:16:10] [02miraheze/IncidentReporting] 07paladox 03ad4a868 - DBConnRef -> MaintainableDBConnRef [17:16:12] [02IncidentReporting] 07paladox created branch 03paladox-patch-2 - 13https://git.io/fh5YJ [17:16:13] [02IncidentReporting] 07paladox opened pull request 03#5: DBConnRef -> MaintainableDBConnRef - 13https://git.io/JedxT [17:17:32] [02miraheze/IncidentReporting] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JedxL [17:17:33] [02miraheze/IncidentReporting] 07paladox 03be8b98c - Update SpecialIncidentReports.php [17:17:35] [02IncidentReporting] 07paladox synchronize pull request 03#5: DBConnRef -> MaintainableDBConnRef - 13https://git.io/JedxT [17:19:12] miraheze/IncidentReporting/paladox-patch-2/ad4a868 - paladox The build passed. https://travis-ci.com/miraheze/IncidentReporting/builds/142050995 [17:19:56] [02miraheze/IncidentReporting] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JedxY [17:19:57] [02miraheze/IncidentReporting] 07paladox 0332e7f64 - Revert "Replace deprecated/removed getTitle function with getPageTitle" This reverts commit 5a3e1a30a2051b64c054b12cb156261f98cb0ec4. [17:20:08] [02IncidentReporting] 07paladox closed pull request 03#5: DBConnRef -> MaintainableDBConnRef - 13https://git.io/JedxT [17:20:10] [02miraheze/IncidentReporting] 07paladox pushed 033 commits to 03master [+0/-0/±4] 13https://git.io/Jedxs [17:20:11] [02miraheze/IncidentReporting] 07paladox 039160b7c - Merge pull request #5 from miraheze/paladox-patch-2 DBConnRef -> MaintainableDBConnRef [17:20:13] [02IncidentReporting] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/fh5YJ [17:20:14] [02miraheze/IncidentReporting] 07paladox deleted branch 03paladox-patch-2 [17:20:39] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JedxC [17:20:40] [02miraheze/mediawiki] 07paladox 037c97333 - Update IR [17:21:32] miraheze/IncidentReporting/paladox-patch-2/be8b98c - paladox The build has errored. https://travis-ci.com/miraheze/IncidentReporting/builds/142051048 [17:26:14] [02miraheze/IncidentReporting] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JedxB [17:26:16] [02miraheze/IncidentReporting] 07paladox 03af99c66 - Bump version to 1.1.0 [17:26:17] [02IncidentReporting] 07paladox created branch 03paladox-patch-2 - 13https://git.io/fh5YJ [17:26:19] [02IncidentReporting] 07paladox opened pull request 03#6: Bump version to 1.1.0 - 13https://git.io/JedxR [17:26:40] [02miraheze/IncidentReporting] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JedxE [17:26:42] [02miraheze/IncidentReporting] 07paladox 03ed7973e - Update extension.json [17:26:43] [02IncidentReporting] 07paladox synchronize pull request 03#6: Bump version to 1.1.0 - 13https://git.io/JedxR [17:26:55] [02IncidentReporting] 07paladox closed pull request 03#6: Bump version to 1.1.0 - 13https://git.io/JedxR [17:26:56] [02miraheze/IncidentReporting] 07paladox pushed 033 commits to 03master [+0/-0/±4] 13https://git.io/Jedxz [17:26:58] [02miraheze/IncidentReporting] 07paladox 03171efc3 - Merge pull request #6 from miraheze/paladox-patch-2 Bump version to 1.1.0 [17:27:17] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/Jedxa [17:27:19] [02miraheze/mediawiki] 07paladox 031958267 - Update IR [17:27:27] [02IncidentReporting] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/fh5YJ [17:27:29] [02miraheze/IncidentReporting] 07paladox deleted branch 03paladox-patch-2 [17:28:18] miraheze/IncidentReporting/paladox-patch-2/af99c66 - paladox The build has errored. https://travis-ci.com/miraheze/IncidentReporting/builds/142051280 [17:29:42] miraheze/IncidentReporting/paladox-patch-2/ed7973e - paladox The build has errored. https://travis-ci.com/miraheze/IncidentReporting/builds/142051291 [17:31:28] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jedx6 [17:31:30] [02miraheze/ManageWiki] 07paladox 0311f088a - Revert "Replace deprecated/removed getTitle function with getPageTitle" This reverts commit 081efbd04bb81d9e87a9978f5f6963a52d1870d3. [17:31:57] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/Jedxi [17:31:59] [02miraheze/mediawiki] 07paladox 03efed826 - Update MW [19:16:12] [2acebd3062557773dca051f0] 2019-12-22 19:15:26: Excepción grave de tipo "Wikimedia\Rdbms\DBQueryError" Special:Undelete hispano.miraheze.org [19:18:42] paladox: ^ [19:18:47] i can look [19:18:58] Thanks Zppix [19:20:26] i see nothing in the logs hispano76 [19:21:35] oh i see it nmow [19:21:36] now [19:22:19] paladox: ^ that is a flow talk manager error can you fix it for me please [19:25:01] SPF|Cloud: can you update [[Staff]] to reflect new titles [19:37:54] !log MariaDB [hispanowiki]> update actor set actor_user = 28 where actor_name = 'Flow talk page manager'; [19:37:58] Zppix hispano76 done [19:38:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:38:23] PROBLEM - db4 Disk Space on db4 is WARNING: DISK WARNING - free space: / 32932 MB (8% inode=95%); [19:40:07] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jedhy [19:40:09] [02miraheze/services] 07MirahezeSSLBot 03f8f812d - BOT: Updating services config for wikis [19:42:25] thanks paladox [19:50:21] Zppix: not work [19:50:35] hispano76 not work? [19:50:41] hispano76: whats the error [19:50:47] there's a fix here https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/560327/ [19:50:47] [ Gerrit Code Review ] - gerrit.wikimedia.org [19:55:55] https://hispano.miraheze.org/wiki/Especial:Restaurar/Plantilla:Equivalencias [37426ed776c848e70d5698c0] 2019-12-22 19:55:44: Excepción grave de tipo "Wikimedia\Rdbms\DBQueryError" [19:55:57] [ Error de permisos - Hispano ] - hispano.miraheze.org [19:56:51] hispano76 that'll be due to the issue that this https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/560327/ fixes [19:56:52] [ Gerrit Code Review ] - gerrit.wikimedia.org [19:57:38] !log MariaDB [hispanowiki]> update actor set actor_user = NULL where actor_name = 'Flow talk page manager'; [19:57:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:57:53] Ah, I didn't see it. Okay, thank you both. [20:07:19] np [20:16:24] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±4] 13https://git.io/JedjB [20:16:26] [02miraheze/mediawiki] 07Brad Jorsch 03d4f467a - User: Allow newSystemUser() to create over anonymous actors Various maintenance scripts assume reserved usernames like "MediaWiki default" exist, but since they're reserved User::isUsableName() returns false and therefore the actor migration created them as anonymous actors. Which would then prevent those maintenance scripts from using User::newSystemUser() to [20:16:26] ensure they actually exist. This adjusts User::newSystemUser() to be able to create users for those anonymous actors. This also adjusts uses of "MediaWiki default" in core to create it as a system user. Bug: T236444 Change-Id: I59a646df36ff9343cc43c05aa20b2b69b2ee124a (cherry picked from commit 685b505628099a027ab5c9451502f522b489c109) [20:26:37] Reception123: you’re supposed to respond to my email when doing what it says but thanks :) [20:27:35] SantaRhino: I was planning on responding after I finished with all the renames :D [20:29:42] Reception123: ok [20:30:37] Reception123: thought it would be better an awake person did it [20:30:52] * SantaRhino has had too much juice and chocolate for one night [20:31:21] :D [20:31:38] NDKilla has also updated Discord [20:31:54] * SantaRhino is on event 2 of about 5 of the Xmas season [20:32:26] heh [20:32:42] * Reception123 is hunting down all mentions of Operations [20:35:41] Ja [20:35:44] Ha [20:38:36] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JedjX [20:38:37] [02miraheze/mw-config] 07paladox 03fc53881 - MediaWiki 1.34 site notice [20:38:39] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [20:38:40] [02mw-config] 07paladox opened pull request 03#2828: MediaWiki 1.34 site notice - 13https://git.io/Jedj1 [20:39:09] Hello c^! If you have any questions, feel free to ask and someone should answer soon. [20:39:15] o/ [20:40:08] Hi c^ [20:42:06] hi [20:45:50] [02mw-config] 07paladox closed pull request 03#2828: MediaWiki 1.34 site notice - 13https://git.io/Jedj1 [20:45:51] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jedjy [20:45:53] [02miraheze/mw-config] 07paladox 03f96814d - MediaWiki 1.34 site notice (#2828) [20:45:54] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [20:45:56] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [20:47:07] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JedjH [20:47:09] [02miraheze/mw-config] 07paladox 0393705a0 - Fix typo [20:47:50] RECOVERY - thesimswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.thesimswiki.com' will expire on Fri 14 Feb 2020 08:50:14 AM GMT +0000. [20:49:22] paladox: how specific [20:49:26] But good luck! [20:49:30] ? [20:49:38] paladox: the upgrade time [20:49:47] Can you add an alert in #announcements [20:51:20] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JedjF [20:51:21] [02miraheze/mw-config] 07paladox 03f0ab37b - Update Sitenotice.php [20:51:23] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [20:51:24] [02mw-config] 07paladox opened pull request 03#2829: Update Sitenotice.php - 13https://git.io/Jedjb [20:51:51] [02mw-config] 07paladox synchronize pull request 03#2829: Update Sitenotice.php - 13https://git.io/Jedjb [20:51:52] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/Jedjx [20:51:54] [02miraheze/mw-config] 07paladox 0392341df - Update Sitenotice.php [20:52:08] SantaRhino the upgrade time is in the annoucement? [20:52:21] [02mw-config] 07paladox closed pull request 03#2829: Update Sitenotice.php - 13https://git.io/Jedjb [20:52:23] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jedjp [20:52:24] [02miraheze/mw-config] 07paladox 0323040f0 - Update Sitenotice.php (#2829) * Update Sitenotice.php * Update Sitenotice.php [20:52:26] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [20:52:26] paladox: yes [20:52:27] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [20:53:12] miraheze/mw-config/paladox-patch-1/92341df - paladox The build has errored. https://travis-ci.org/miraheze/mw-config/builds/628471355 [20:57:29] @Stewards can someone PM me for an OS or see the IP I revdel’d [20:58:20] Seen, will handle [20:59:03] Thanks @Void [21:43:31] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://git.io/JeFeX [21:43:33] [02miraheze/mw-config] 07paladox 038018946 - Set all wikis to readonly [21:43:34] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://git.io/vbvb3 [21:43:36] [02mw-config] 07paladox opened pull request 03#2830: Set all wikis to readonly - 13https://git.io/JeFe1 [21:44:54] miraheze/mw-config/paladox-patch-1/8018946 - paladox The build passed. https://travis-ci.org/miraheze/mw-config/builds/628484541 [21:51:24] paladox: see discord [22:01:58] Hi! Here is the list of currently open high priority tasks on Phabricator [22:02:05] No updates for 15 days - https://phabricator.miraheze.org/T4961 - Determine a proper mailerlite access policy - authored by RhinosF1, assigned to Owen [22:02:12] No updates for 31 days - https://phabricator.miraheze.org/T4857 - Thumbnails of recently uploaded images not showing up. Actual image opens up when navigated to. - authored by AngelCakes, assigned to None [22:03:57] paladox: get ready, the upgrade clocked has ticked into place. [22:05:09] [02mw-config] 07paladox closed pull request 03#2830: Set all wikis to readonly - 13https://git.io/JeFe1 [22:05:11] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeFvO [22:05:12] [02miraheze/mw-config] 07paladox 039497b1b - Set all wikis to readonly (#2830) [22:05:14] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [22:05:15] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://git.io/vbvb3 [22:05:46] !log depool lizardfs6 for mw 1.34 upgrade [22:11:49] PROBLEM - lizardfs6 Puppet on lizardfs6 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 9 minutes ago with 0 failures [22:12:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeFvl [22:12:40] [02miraheze/puppet] 07paladox 0346bd583 - lizardfs6: Switch to mw 1.34 [22:14:32] PROBLEM - lizardfs6 Puppet on lizardfs6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:18:38] !log repool lizardfs6 [22:18:44] !log depool mw1 for mw 1.34 upgrade [22:19:13] work edit? [22:19:21] oh, hum [22:19:24] RECOVERY - lizardfs6 Puppet on lizardfs6 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:23:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeFvo [22:23:21] [02miraheze/puppet] 07paladox 03c872555 - mw1: Upgrade to mw 1.34 [22:23:27] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw1 [22:26:01] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 6 backends are healthy [22:27:09] paladox: you can’t !log when the wiki is read only [22:27:26] ok, i know :) [22:28:09] paladox: although keep doing it so I can see what stage you’re at if you want :) [22:30:23] !log repool mw1 [22:30:28] !log depool mw2 for mw 1.34 upgrade [22:31:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeFvH [22:31:58] [02miraheze/puppet] 07paladox 03dc0d38b - mw2: Upgrade to mw 1.34 [22:35:05] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw2 [22:35:26] PROBLEM - mw2 Puppet on mw2 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 12 minutes ago with 0 failures [22:36:54] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 11 failures. Last run 5 minutes ago with 11 failures. Failed resources (up to 3 shown): Service[prometheus-node-exporter],Service[bacula-fd],Service[salt-minion],Exec[ufw-logging-low] [22:37:22] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2 [22:39:54] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy [22:39:56] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 6 backends are healthy [22:40:14] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:41:44] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [22:47:52] PROBLEM - mw2 Puppet on mw2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[cookiewarning_composer] [22:51:02] paladox: why’s mw2 taking longer? [22:51:32] !log repool mw2 [22:51:55] !log depool mw3 for mw 1.34 upgrade [22:53:01] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:53:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeFft [22:53:19] [02miraheze/puppet] 07paladox 03361b470 - mw3: Upgrade to mw 1.34 [22:54:38] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 3 minutes ago with 6 failures. Failed resources (up to 3 shown): Exec[ufw-allow-tcp-from-any-to-any-port-22],Exec[ufw-allow-tcp-from-any-to-any-port-5666],Exec[ufw-allow-tcp-from-185.52.3.121-to-any-port-9100],Service[puppetdb] [22:55:42] PROBLEM - mw3 Puppet on mw3 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [22:58:55] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3 [22:58:58] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw3 [23:00:59] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:01:25] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 6 backends are healthy [23:01:28] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 6 backends are healthy [23:02:27] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [23:07:53] !log repool mw3 [23:09:44] Hello Naleksuh! If you have any questions, feel free to ask and someone should answer soon. [23:10:10] Hi Naleksuh [23:10:50] miraheze started downtime like 2 minutes right before i needed to edit it [23:10:51] sad day [23:11:14] Naleksuh: shouldn’t be much longer [23:11:25] says its gonna be 19 minutes longer [23:12:26] Naleksuh: the sitenotice said 22:30 UTC to finish. We’ve actually ran over [23:12:26] !log root@mw3:/srv# ./foreachwikiindblist /srv/mediawiki/dblist/all.dblist [23:12:33] It’s 23:12 [23:12:45] db going out of read only now [23:12:52] okay cool thanks [23:15:31] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_34 [+0/-0/±1] 13https://git.io/JeFfV [23:15:33] [02miraheze/mediawiki] 07paladox 0330c5529 - Update AR to 9dc02595aed6b780b24637c5e23041b0a1cfe3e0 [23:15:34] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeFfr [23:15:36] [02miraheze/mw-config] 07paladox 0392cb30e - Revert "Set all wikis to readonly (#2830)" This reverts commit 9497b1b18182f5f0ef8d49a4308677fde97607e1. [23:15:50] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[rsyslog] [23:16:05] paladox: sitenotice [23:17:35] rebuild no work [23:17:42] i still can't edit [23:17:49] SantaRhino users wiki is fixed now [23:18:06] Naleksuh: it should be restoring soon [23:18:11] paladox: thanks [23:18:43] Naleksuh now try [23:20:49] yeah i can edit now [23:21:05] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeFfP [23:21:07]