[00:04:19] <icinga-miraheze>	 PROBLEM - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN
[00:07:52] <icinga-miraheze>	 RECOVERY - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is OK: OK - NGINX Error Rate is 1%
[00:40:59] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2648 MB (10% inode=93%);
[01:08:38] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 2001:41d0:800:1056::2/cpweb
[01:12:38] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[01:24:46] <Not-a789>	 [02mw-config] 07paladox closed pull request 03#2951: Migrate 1.6k wikis to dbt1 - 13https://git.io/JvHQN
[01:24:48] <Not-a789>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvHhi
[01:24:49] <Not-a789>	 [02miraheze/mw-config] 07paladox 031036127 - Migrate 1.6k wikis to dbt1 (#2951)
[01:24:51] <Not-a789>	 [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3
[01:24:52] <Not-a789>	 [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2
[01:26:43] <Not-a789>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvHhD
[01:26:44] <Not-a789>	 [02miraheze/mw-config] 07paladox 03d9b021f - Take closed wikis out of read only mode
[01:26:46] <Not-a789>	 [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3
[01:26:47] <Not-a789>	 [02mw-config] 07paladox opened pull request 03#2952: Take closed wikis out of read only mode - 13https://git.io/JvHhy
[01:28:10] <Not-a789>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvHh7
[01:28:12] <Not-a789>	 [02miraheze/mw-config] 07paladox 031820257 - Update Database.php
[01:28:26] <travis-ci>	 miraheze/mw-config/paladox-patch-2/d9b021f - paladox The build passed. https://travis-ci.org/miraheze/mw-config/builds/667946846
[01:31:08] <Not-a789>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvHhA
[01:31:10] <Not-a789>	 [02miraheze/mw-config] 07paladox 03190bfcb - Update Database.php
[01:35:13] <Not-a789>	 [02mw-config] 07paladox closed pull request 03#2952: Take closed wikis out of read only mode - 13https://git.io/JvHhy
[01:35:15] <Not-a789>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvHhh
[01:35:16] <Not-a789>	 [02miraheze/mw-config] 07paladox 03e72e083 - Take closed wikis out of read only mode (#2952)
[01:35:23] <Not-a789>	 [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2
[01:35:24] <Not-a789>	 [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3
[02:07:11] <Not-a789>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvHjK
[02:07:13] <Not-a789>	 [02miraheze/puppet] 07paladox 03d343573 - Backup dbt1
[02:07:14] <Not-a789>	 [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS
[02:07:16] <Not-a789>	 [02puppet] 07paladox opened pull request 03#1298: Backup dbt1 - 13https://git.io/JvHji
[02:07:43] <Not-a789>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvHjP
[02:07:45] <Not-a789>	 [02miraheze/puppet] 07paladox 03d56ace7 - Update director.pp
[02:07:46] <Not-a789>	 [02puppet] 07paladox synchronize pull request 03#1298: Backup dbt1 - 13https://git.io/JvHji
[02:08:12] <Not-a789>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvHj1
[02:08:14] <Not-a789>	 [02miraheze/puppet] 07paladox 033b074a5 - Update nrpe.cfg.erb
[02:08:15] <Not-a789>	 [02puppet] 07paladox synchronize pull request 03#1298: Backup dbt1 - 13https://git.io/JvHji
[02:11:04] <Not-a789>	 [02puppet] 07paladox synchronize pull request 03#1298: Backup dbt1 - 13https://git.io/JvHji
[02:11:06] <Not-a789>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvHj9
[02:11:07] <Not-a789>	 [02miraheze/puppet] 07paladox 0351e3805 - Update bacula-dir.conf
[02:12:19] <Not-a789>	 [02puppet] 07paladox closed pull request 03#1298: Backup dbt1 - 13https://git.io/JvHji
[02:12:21] <Not-a789>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JvHjd
[02:12:22] <Not-a789>	 [02miraheze/puppet] 07paladox 03181e6e7 - Backup dbt1 (#1298) * Backup dbt1 * Update director.pp * Update nrpe.cfg.erb * Update bacula-dir.conf
[02:12:26] <Not-a789>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5
[02:12:27] <Not-a789>	 [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS
[02:14:26] <Not-a789>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvHjA
[02:14:28] <Not-a789>	 [02miraheze/puppet] 07paladox 03df6b5e1 - bacula: Backup puppet2 not puppet1
[02:16:09] <Not-a789>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvHjj
[02:16:10] <Not-a789>	 [02miraheze/puppet] 07paladox 037bda738 - Update nrpe.cfg.erb
[02:26:12] <icinga-miraheze>	 PROBLEM - bacula2 Bacula Databases dbt1 on bacula2 is CRITICAL: CRITICAL: no terminated jobs
[02:35:54] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2400:6180:0:d0::403:f001/cpweb, 2607:5300:205:200::17f6/cpweb
[02:39:04] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[02:44:46] <paladox>	 !log drop 1.6k wikis on db4 (migrated to dbt1)
[02:44:54] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[03:05:39] <icinga-miraheze>	 PROBLEM - cp3 Current Load on cp3 is WARNING: WARNING - load average: 1.88, 1.78, 1.54
[03:08:40] <icinga-miraheze>	 RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 1.38, 1.55, 1.48
[03:20:08] <Not-a789>	 [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvQv6
[03:20:10] <Not-a789>	 [02miraheze/services] 07MirahezeSSLBot 0378b635c - BOT: Updating services config for wikis
[03:30:41] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 51.77.107.210/cpweb
[03:34:16] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[05:00:00] <icinga-miraheze>	 PROBLEM - puppet2 Puppet on puppet2 is CRITICAL: CRITICAL: Puppet has 17 failures. Last run 4 minutes ago with 17 failures. Failed resources (up to 3 shown)
[05:01:36] <icinga-miraheze>	 PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 23 failures. Last run 5 minutes ago with 23 failures. Failed resources (up to 3 shown): File[www.mcpk.wiki],File[www.mcpk.wiki_private],File[rivworld.wiki],File[rivworld.wiki_private]
[05:02:56] <icinga-miraheze>	 RECOVERY - puppet2 Puppet on puppet2 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures
[05:07:36] <icinga-miraheze>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:07:13] <icinga-miraheze>	 PROBLEM - cp3 Current Load on cp3 is WARNING: WARNING - load average: 1.94, 1.98, 1.66
[06:10:11] <icinga-miraheze>	 RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 1.60, 1.70, 1.59
[06:26:26] <icinga-miraheze>	 RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2938 MB (12% inode=93%);
[08:09:54] <icinga-miraheze>	 PROBLEM - cp3 Current Load on cp3 is WARNING: WARNING - load average: 1.46, 1.83, 1.69
[08:13:03] <icinga-miraheze>	 PROBLEM - cp3 Current Load on cp3 is CRITICAL: CRITICAL - load average: 2.78, 2.13, 1.83
[08:16:02] <icinga-miraheze>	 PROBLEM - cp3 Current Load on cp3 is WARNING: WARNING - load average: 1.68, 1.90, 1.79
[08:21:59] <icinga-miraheze>	 PROBLEM - cp3 Current Load on cp3 is CRITICAL: CRITICAL - load average: 2.13, 1.84, 1.77
[08:24:54] <icinga-miraheze>	 PROBLEM - cp3 Current Load on cp3 is WARNING: WARNING - load average: 1.59, 1.71, 1.73
[08:27:50] <icinga-miraheze>	 RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 1.25, 1.61, 1.69
[11:48:06] <icinga-miraheze>	 PROBLEM - cp3 Current Load on cp3 is WARNING: WARNING - load average: 1.51, 1.74, 1.62
[11:53:54] <icinga-miraheze>	 RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 1.40, 1.61, 1.59
[14:05:48] <Not-a789>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvQGU
[14:05:50] <Not-a789>	 [02miraheze/puppet] 07paladox 03122a041 - mediawiki: Increase childs to 10
[14:09:03] <paladox>	 !log restarted php7.3-fpm on mw*
[14:09:12] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[14:19:38] <icinga-miraheze>	 PROBLEM - cp8 Puppet on cp8 is CRITICAL: CRITICAL: Puppet has 33 failures. Last run 3 minutes ago with 33 failures. Failed resources (up to 3 shown)
[14:22:11] <icinga-miraheze>	 PROBLEM - puppet2 Puppet on puppet2 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 6 minutes ago with 10 failures. Failed resources (up to 3 shown)
[14:24:42] <icinga-miraheze>	 PROBLEM - cp3 Puppet on cp3 is CRITICAL: CRITICAL: Puppet has 29 failures. Last run 6 minutes ago with 29 failures. Failed resources (up to 3 shown): File[encyclopediaofastrobiology.org],File[encyclopediaofastrobiology.org_private],File[www.evanswiki.org],File[www.evanswiki.org_private]
[14:25:18] <icinga-miraheze>	 RECOVERY - puppet2 Puppet on puppet2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures
[14:25:43] <icinga-miraheze>	 RECOVERY - cp8 Puppet on cp8 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[14:27:39] <icinga-miraheze>	 RECOVERY - cp3 Puppet on cp3 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[15:00:50] <RhinosF1>	 Afternoon JohnLewis
[15:01:10] <JohnLewis>	 Afternoon
[15:06:01] <icinga-miraheze>	 PROBLEM - cp3 Current Load on cp3 is WARNING: WARNING - load average: 1.98, 1.72, 1.54
[15:08:55] <icinga-miraheze>	 RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 1.62, 1.62, 1.53
[15:13:04] <icinga-miraheze>	 PROBLEM - mw2 Puppet on mw2 is WARNING: WARNING: Puppet last ran 1 hour ago
[15:15:42] <icinga-miraheze>	 PROBLEM - cp3 Current Load on cp3 is WARNING: WARNING - load average: 1.76, 1.70, 1.60
[15:18:32] <icinga-miraheze>	 RECOVERY - cp3 Current Load on cp3 is OK: OK - load average: 1.38, 1.59, 1.57
[15:41:11] <ZppixBot>	 Hello irene_cara! If you have any questions, feel free to ask and someone should answer soon.
[15:41:13] <irene_cara>	 Hello miraheze ! I hope everyone is doing very good during the coronavirus outbreak ! Here are some songs so we can sing along and cheer up! HAPPY EARTH HOUR DAY !!!
[15:41:18] <irene_cara>	 I'm gonna live forever
[15:41:23] <irene_cara>	 I'm gonna learn how to fly
[15:41:28] <irene_cara>	 (High)
[15:41:33] <irene_cara>	 I feel it coming together
[15:41:39] <irene_cara>	 People will see me and cry
[15:41:44] <irene_cara>	 I'm gonna make it to heaven
[15:41:47] <irene_cara>	  Irene_Cara Hello miraheze ! I hope everyone is doing very good during the coronavirus outbreak ! Here are some songs so we can sing along and cheer up! HAPPY EARTH HOUR DAY !!!
[15:41:50] <irene_cara>	 Light up the sky like a flame
[15:41:55] <irene_cara>	 (Fame)
[15:42:01] <irene_cara>	 I'm gonna live forever
[15:42:05] <irene_cara>	 Baby, remember my name
[15:42:12] <irene_cara>	 (Remember, remember, remember, remember)
[15:42:16] <irene_cara>	 (Remember, remember, remember, remember)
[15:42:19] <irene_cara>	 Baby hold me tight, 'cause you can make it right
[15:42:25] <irene_cara>	 You can shoot me straight to the top
[15:42:29] <irene_cara>	 Give me love and take all I've got to give
[15:42:34] <irene_cara>	 Baby I'll be tough, too much is not enough, no
[15:42:38] <irene_cara>	 I'll grab your heart till it breaks
[15:42:42] <irene_cara>	 You got what it…
[15:43:02] <irene_cara>	 Come on staff don't be shy sing with me it's fun and it's Earth day! Zppix Voidwalker Reception123 RhinosF1 !! 
[15:43:32] <irene_cara>	 ................
[15:43:33] <irene_cara>	 What about sunrise
[15:43:37] <irene_cara>	 What about rain
[15:43:48] <irene_cara>	 What about all the things that you said (,Miraheze)
[15:43:52] <irene_cara>	 We were to gain
[15:43:57] <irene_cara>	 What about killing fields
[15:44:04] <irene_cara>	 Is there a time
[15:44:10] <irene_cara>	 What about all the things
[15:44:15] <irene_cara>	 That you said were yours and mine
[15:44:21] <irene_cara>	 Did you ever stop to notice
[15:44:25] <irene_cara>	 All the blood we've shed before
[15:44:29] <irene_cara>	 Did you ever stop to notice
[15:44:35] <irene_cara>	 This crying Earth, these weeping shores
[15:44:42] <irene_cara>	 Come on staff don't be shy sing with me it's fun and it's Earth day! Zppix Voidwalker Reception123 RhinosF1 !! 
[15:44:55] <irene_cara>	 Aah, ooh Aah, ooh Aah, ooh Aah, ooh Aah, ooh Aah, ooh Aah, ooh Aah, ooh
[15:45:03] <irene_cara>	 What have we done to the world Miraheze?
[15:45:08] <irene_cara>	 Look what we've done
[15:45:14] <irene_cara>	 What about all the peace
[15:45:19] <irene_cara>	 That you pledge your only son
[15:45:23] <irene_cara>	 What about flowering fields
[15:45:28] <irene_cara>	 Is there a time
[15:45:32] <irene_cara>	 What about all the dreams
[15:45:37] <irene_cara>	 That you said was yours and mine
[15:45:49] <irene_cara>	 Aah, ooh Aah, ooh Aah, ooh Aah, ooh
[15:46:09] <irene_cara>	 Hello miraheze ! I hope everyone is doing very good during the coronavirus outbreak ! Here are some songs so we can sing along and cheer up! HAPPY EARTH HOUR DAY !!! Reception123 Zppix RhinosF1 Voidwalker
[15:46:19] <irene_cara>	 ............
[15:46:23] <irene_cara>	 We are the world
[15:46:29] <irene_cara>	 We are the children
[15:46:36] <irene_cara>	 We are the ones who make a brighter day, so let's start giving
[15:46:39] <irene_cara>	 There's a choice we're making
[15:46:45] <irene_cara>	 We're saving our own lives
[15:46:49] <irene_cara>	 It's true we'll make a better day, just you and me
[15:46:56] <irene_cara>	 Oh, send them your heart
[15:47:59] <irene_cara>	 So they know that someone cares
[15:48:05] <irene_cara>	 And their lives will be stronger and free
[15:48:12] <irene_cara>	 As God has shown us by turning stones to bread
[15:48:16] <irene_cara>	 And so we all must lend a helping hand
[15:48:22] <irene_cara>	 We can't go on
[15:48:25] <irene_cara>	 We can't go on, Miraheze
[15:48:31] <irene_cara>	 Pretending day-by-day
[15:48:35] <irene_cara>	 That someone, somewhere soon make a change
[15:48:50] <irene_cara>	 We're all a part of God's great big family ! Everyone, you too RhinosF1 Zppix Voidwalker Reception123 !!
[15:48:55] <irene_cara>	 And the truth, you know, love is all we need
[15:48:59] <irene_cara>	 And the truth, you know, love is all we need
[15:49:43] <irene_cara>	 ..............................
[15:49:48] <irene_cara>	 Do you remember
[15:49:54] <irene_cara>	 The twenty first night of September?
[15:50:03] <irene_cara>	 Reception123 RhinosF1 Zppix Voidwalker ?
[15:50:05] <irene_cara>	 Love was changing the minds of pretenders
[15:50:10] <irene_cara>	 While chasing the clouds away
[15:50:15] <irene_cara>	 Our hearts were ringing
[15:50:19] <irene_cara>	 In the key that our souls were singing
[15:50:24] <irene_cara>	 As we danced in the night
[15:50:38] <irene_cara>	 Don't let me sing along Zppix Voidwalker Reception123 RhinosF1 sing with me !!
[15:50:45] <irene_cara>	 Remember how the stars stole the night away
[15:50:50] <irene_cara>	 Ba de ya say do you remember
[15:50:55] <irene_cara>	 Ba de ya dancing in September
[15:51:01] <irene_cara>	 Ba de ya never was a cloudy day
[15:51:08] <irene_cara>	 Badu, badu, badu-du, badu
[15:51:13] <irene_cara>	 Badu, badu, badu-du, badu
[15:51:20] <irene_cara>	 Badu, badu, badu-du, badu
[15:51:30] <irene_cara>	 Hello miraheze ! I hope everyone is doing very good during the coronavirus outbreak ! Here are some songs so we can sing along and cheer up! HAPPY EARTH HOUR DAY !!!
[15:51:36] <irene_cara>	 Hello miraheze ! I hope everyone is doing very good during the coronavirus outbreak ! Here are some songs so we can sing along and cheer up! HAPPY EARTH HOUR DAY !!!
[15:51:51] <irene_cara>	 Ba de ya say do you remember
[15:51:56] <irene_cara>	 Ba de ya dancing in September
[15:52:00] <irene_cara>	 Ba de ya never was a cloudy day
[15:52:22] <irene_cara>	 ..............
[15:52:23] <irene_cara>	 Young man, there's no need to feel down
[15:52:27] <irene_cara>	 I said, young man, pick yourself off the ground
[15:52:31] <irene_cara>	 I said, young man, 'cause you're in a new town
[15:52:38] <irene_cara>	 There's no need to be unhappy
[15:52:43] <irene_cara>	 Young man, there's a place you can go
[15:52:52] <irene_cara>	 I said, young man, when you're short on your dough
[15:52:56] <irene_cara>	 You can stay there, and I'm sure you will find
[15:53:00] <irene_cara>	 Many ways to have a good time
[15:53:07] <irene_cara>	 It's fun to stay at the YMCA
[15:53:08] <irene_cara>	 It's fun to stay at the YMCA
[15:53:14] <irene_cara>	 They have everything for you men to enjoy
[15:53:29] <irene_cara>	 You can hang out with all the boys and the nice Miraheze staffers like Reception123 RhinosF1 Zppix Voidwalker !!!! 
[15:53:35] <irene_cara>	 You can get yourself clean, you can have a good meal
[15:53:39] <irene_cara>	 You can do what ever you feel
[15:53:47] <irene_cara>	 Young man, are you listening to me? Are you listening to me Miraheze ?
[15:53:52] <irene_cara>	 I said, young man, what do you want to be?
[15:53:56] <irene_cara>	 I said, young man, you can make real your dreams
[15:54:00] <irene_cara>	 But you got to know this one thing
[15:54:10] <irene_cara>	 No man does it all by himself ! Yes Miraheze team that's why we're a team!!!
[15:54:16] <irene_cara>	 I said, young man, put your pride on the shelf
[15:54:20] <irene_cara>	 And just go there, to the YMCA
[16:07:16] <RhinosF1>	 grumble: ^
[16:08:05] <paladox>	 RhinosF1 please do that in private.
[16:08:23] <paladox>	 this channel is logged
[16:18:19] <Not-a789>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvQc7
[16:18:20] <Not-a789>	 [02miraheze/mw-config] 07paladox 03c7d4fb1 - Set site notice for all wikis on db4
[16:18:22] <Not-a789>	 [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3
[16:18:23] <Not-a789>	 [02mw-config] 07paladox opened pull request 03#2953: Set site notice for all wikis on db4 - 13https://git.io/JvQc5
[16:30:32] <Not-a789>	 [02mw-config] 07paladox synchronize pull request 03#2953: Set site notice for all wikis on db4 - 13https://git.io/JvQc5
[16:30:33] <Not-a789>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvQC8
[16:30:35] <Not-a789>	 [02miraheze/mw-config] 07paladox 03612bed7 - Update Sitenotice.php
[16:37:31] <icinga-miraheze>	 PROBLEM - mw3 Puppet on mw3 is WARNING: WARNING: Puppet last ran 1 hour ago
[16:51:02] <Not-a789>	 [02mw-config] 07paladox synchronize pull request 03#2953: Set site notice for all wikis on db4 - 13https://git.io/JvQc5
[16:51:04] <Not-a789>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvQWq
[16:51:05] <Not-a789>	 [02miraheze/mw-config] 07paladox 030410caf - Update Sitenotice.php
[17:00:28] <Not-a789>	 [02mw-config] 07paladox closed pull request 03#2953: Set site notice for all wikis on db4 - 13https://git.io/JvQc5
[17:00:29] <Not-a789>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvQW2
[17:00:31] <Not-a789>	 [02miraheze/mw-config] 07paladox 039acd965 - Set site notice for all wikis on db4 (#2953) * Set site notice for all wikis on db4 * Update Sitenotice.php * Update Sitenotice.php
[17:00:32] <Not-a789>	 [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3
[17:00:34] <Not-a789>	 [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2
[17:02:50] <icinga-miraheze>	 RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures
[17:03:16] <Not-a789>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvQWr
[17:03:18] <Not-a789>	 [02miraheze/mw-config] 07paladox 03113c2d1 - fix typo
[17:03:28] <icinga-miraheze>	 RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures
[17:06:42] <Not-a789>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvQW7
[17:06:44] <Not-a789>	 [02miraheze/mw-config] 07paladox 0342c47dc - Disable createwiki
[17:06:45] <Not-a789>	 [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3
[17:06:47] <Not-a789>	 [02mw-config] 07paladox opened pull request 03#2954: Disable createwiki - 13https://git.io/JvQWF
[17:14:32] <RhinosF1>	 paladox: whats the plan
[17:14:50] <paladox>	 its the same plan as before
[17:16:53] <RhinosF1>	 Okay
[17:17:04] <RhinosF1>	 keep us updated
[17:17:49] <Reception123>	 RhinosF1: yes, only we got rid of closed wikis
[17:17:56] <Reception123>	 and we're not doing ones that are scheduled for deletion in a week
[17:18:10] <RhinosF1>	 Hmm
[17:18:33] <Reception123>	 RhinosF1: that way we'll reduce read-only time
[17:18:44] <RhinosF1>	 Yeah
[17:18:51] <Reception123>	 so we're only left with about 1.8k I think
[17:19:12] <Reception123>	 (since due to the deleteWikis script having an error since 1.34, there's a longgg list of wikis waiting to be deleted)
[17:19:21] <Reception123>	 if you see Special:DeletedWikis you'll know what I'm talking about
[17:22:45] <RhinosF1>	 Heh
[17:33:40] <Not-a789>	 [02mw-config] 07paladox closed pull request 03#2954: Disable createwiki - 13https://git.io/JvQWF
[17:33:42] <Not-a789>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvQld
[17:33:43] <Not-a789>	 [02miraheze/mw-config] 07paladox 03e02d584 - Disable createwiki (#2954)
[17:33:45] <Not-a789>	 [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3
[17:33:46] <Not-a789>	 [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2
[17:43:26] <Not-a789>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JvQ8T
[17:43:27] <Not-a789>	 [02miraheze/mw-config] 07paladox 03328db60 - Set all wikis on db4 into read only
[17:43:29] <Not-a789>	 [02mw-config] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbvb3
[17:43:30] <Not-a789>	 [02mw-config] 07paladox opened pull request 03#2955: Set all wikis on db4 into read only - 13https://git.io/JvQ8k
[18:09:14] <icinga-miraheze>	 PROBLEM - mw2 Puppet on mw2 is WARNING: WARNING: Puppet last ran 1 hour ago
[18:15:52] <Not-a789>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://git.io/JvQ4R
[18:15:53] <Not-a789>	 [02miraheze/mw-config] 07paladox 03a44c55b - Set 1.8k wikis to dbt1
[18:15:55] <Not-a789>	 [02mw-config] 07paladox created branch 03paladox-patch-3 - 13https://git.io/vbvb3
[18:15:56] <Not-a789>	 [02mw-config] 07paladox opened pull request 03#2956: Set 1.8k wikis to dbt1 - 13https://git.io/JvQ40
[18:36:02] <Reception123>	 !log reception@mw1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki awesomegameswiki -r "Requested (old fandom accounts) - T5356" /home/reception/agwdel.txt
[18:36:12] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[18:39:00] <icinga-miraheze>	 PROBLEM - mw3 Puppet on mw3 is WARNING: WARNING: Puppet last ran 1 hour ago
[19:20:03] <Reception123>	 !log removed a 2FA from a user
[19:20:26] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master
[19:30:23] <hispano76>	 :)
[19:30:32] * hispano76 greetings
[19:36:39] <Not-a789>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JvQRc
[19:36:40] <Not-a789>	 [02miraheze/mw-config] 07paladox 037da784c - Switch default db connection to dbt1
[19:36:42] <Not-a789>	 [02mw-config] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbvb3
[19:36:43] <Not-a789>	 [02mw-config] 07paladox opened pull request 03#2957: Switch default db connection to dbt1 - 13https://git.io/JvQRC
[20:00:06] <Not-a789>	 [02mw-config] 07paladox closed pull request 03#2955: Set all wikis on db4 into read only - 13https://git.io/JvQ8k
[20:00:08] <Not-a789>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvQ0s
[20:00:09] <Not-a789>	 [02miraheze/mw-config] 07paladox 034fa0d98 - Set all wikis on db4 into read only (#2955)
[20:00:11] <Not-a789>	 [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-2
[20:00:12] <Not-a789>	 [02mw-config] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/vbvb3
[20:02:57] <RhinosF1>	 Good luck!
[20:03:29] <paladox>	 thanks!
[20:03:49] <RhinosF1>	 paladox: isn’t c1 db4 as well?
[20:04:14] <paladox>	 yes, but no wiki is set to c1
[20:04:23] <paladox>	 though i'll just do that too for consitency
[20:04:37] <RhinosF1>	 Good idea
[20:04:51] <Not-a789>	 [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvQ0W
[20:04:52] <Not-a789>	 [02miraheze/mw-config] 07paladox 038879893 - Set all wikis on db4 to read only
[20:06:59] <RhinosF1>	 :)
[20:07:10] <RhinosF1>	 Here if you need anything, just ping!
[20:07:32] <paladox>	 ok
[20:12:36] <Reception123>	 I'll be around for a few hours too :)
[20:13:53] <paladox>	 Starting batch 2 and restoring batch 1
[20:16:27] <RhinosF1>	 Quick!
[20:16:46] <RhinosF1>	 paladox: how many wikis total?
[20:16:55] <paladox>	 1.8k
[20:17:44] <RhinosF1>	 paladox: how many per batch?
[20:17:53] <paladox>	 500
[20:18:22] <RhinosF1>	 So we’ve done 500/1,800 in 5 mins
[20:18:24] <RhinosF1>	 ?
[20:18:55] <paladox>	 no, batch 1 is restoring, i'm currently copying batch 2
[20:19:09] <RhinosF1>	 Ah
[20:19:33] <RhinosF1>	 i see
[20:26:48] <paladox>	 Starting third batch (batch 2 now restoring)
[20:28:24] <RhinosF1>	 paladox: batch 1 - 13 mins then
[20:28:34] <paladox>	 batch 1 hasn't completed though
[20:28:49] <paladox>	 this is copying, batch 1 is still restoring and i started batch 2
[20:29:00] <RhinosF1>	 Are you restoring multiple at same time?
[20:29:10] <RhinosF1>	 I thought this was same plan as last time?
[20:29:32] <paladox>	 yes it is
[20:29:36] <paladox>	 i'm doing batches
[20:30:17] <RhinosF1>	 Each batch was done one by one though
[20:30:32] <RhinosF1>	 This isn’t
[20:30:48] <RhinosF1>	 Let me know when batch one finishes
[20:31:19] <paladox>	 But the essence of the plan is the same, its being done in batches, its using the same commands.
[20:31:38] <paladox>	 just timing is different
[20:33:19] <RhinosF1>	 Yeah but timing is the one bit I care about :)
[21:01:43] <paladox>	 Batches 3 and 4 are copied over now, restoring them now
[21:04:34] <RhinosF1>	 How many are restored
[21:05:09] <paladox>	 I'm not sure how i can tell.
[21:06:23] <RhinosF1>	 Is batch 1 running then?
[21:06:28] <paladox>	 yes
[21:06:47] <RhinosF1>	 Say when it finishes
[21:08:36] <paladox>	 i will :)
[21:15:33] <Not-a789>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/JvQu2
[21:15:34] <Not-a789>	 [02miraheze/puppet] 07paladox 0387e23df - node_exporter: Remove /mnt from ignored_mount_points only on dbt1.miraheze.org
[21:15:36] <Not-a789>	 [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS
[21:15:41] <Not-a789>	 [02puppet] 07paladox opened pull request 03#1299: node_exporter: Remove /mnt from ignored_mount_points only on dbt1.miraheze.org - 13https://git.io/JvQuw
[21:16:03] <Not-a789>	 [02puppet] 07paladox closed pull request 03#1299: node_exporter: Remove /mnt from ignored_mount_points only on dbt1.miraheze.org - 13https://git.io/JvQuw
[21:16:05] <Not-a789>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JvQuo
[21:16:06] <Not-a789>	 [02miraheze/puppet] 07paladox 039554ff4 - node_exporter: Remove /mnt from ignored_mount_points only on dbt1.miraheze.org (#1299)
[21:16:08] <Not-a789>	 [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS
[21:16:09] <Not-a789>	 [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5
[21:17:31] <RhinosF1>	 paladox: don’t forget the clocks change. So 2 am UTC is 3am UK time
[21:17:49] <paladox>	 ok
[21:21:39] <icinga-miraheze>	 PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2650 MB (10% inode=93%);
[21:27:14] <Reception123>	 oh I forgot about that too
[21:30:12] <RhinosF1>	 Reception123: I remembered literally as I was typing
[22:04:56] <RhinosF1>	 paladox: why is Jenkins saying fatal: Could not read from remote repository.
[22:05:04] <RhinosF1>	 https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/584172/
[22:05:04] <ZppixBot>	 [ Gerrit Code Review ] - gerrit.wikimedia.org
[22:05:38] <paladox>	 i guess must be an issue, recheck it and see if that'll resolve it.
[22:06:00] <RhinosF1>	 fired a recheck
[22:09:18] <RhinosF1>	 paladox: worked, silly jenkins
[22:09:32] <paladox>	 ok
[23:05:31] <JohnLewis>	 paladox: what's the update?
[23:06:03] <paladox>	 Its still restoring
[23:06:19] <paladox>	 Usage is at 126G on dbt1
[23:06:25] <JohnLewis>	 I could have guessed that part
[23:06:28] <paladox>	 from around ~50gb when i started
[23:06:48] <paladox>	 JohnLewis what kind of thing are you looking for?
[23:07:11] <JohnLewis>	 which batch? Time estimate?
[23:07:28] <paladox>	 Ohh, i'm doing all batches
[23:07:33] <paladox>	 so 4 batches
[23:07:38] <paladox>	 see what i wrote to RhinosF1 above
[23:08:06] <JohnLewis>	 I know.
[23:08:33] <paladox>	 ok
[23:09:00] <JohnLewis>	 Though it still doesn't answer my question of "which batch[es have done]"
[23:09:26] <paladox>	 oh, none. They are all still restoring.
[23:09:48] <JohnLewis>	 Thank you, took a while to get there but thanks
[23:11:04] <paladox>	 yw :)
[23:18:04] <icinga-miraheze>	 PROBLEM - cp8 Stunnel Http for mw3 on cp8 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[23:18:19] <icinga-miraheze>	 PROBLEM - cp8 Varnish Backends on cp8 is CRITICAL: 1 backends are down. mw3
[23:18:51] <icinga-miraheze>	 PROBLEM - cp6 Stunnel Http for mw3 on cp6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[23:18:53] <icinga-miraheze>	 PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw3
[23:19:22] <icinga-miraheze>	 PROBLEM - cp7 Stunnel Http for mw3 on cp7 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[23:19:33] <icinga-miraheze>	 PROBLEM - cp3 Stunnel Http for mw3 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[23:20:34] <icinga-miraheze>	 PROBLEM - cp7 Varnish Backends on cp7 is CRITICAL: 1 backends are down. mw3
[23:21:20] <icinga-miraheze>	 PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 1 backends are down. mw3
[23:21:34] <paladox>	 First batch finished
[23:21:38] <paladox>	 JohnLewis RhinosF1 ^
[23:24:16] <icinga-miraheze>	 RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures
[23:38:51] <paladox>	 batch 4 complete (batch 2 and 3 still doing)
[23:45:34] <paladox>	 batch 2 complete, batch 3 still doing