[02:27:29] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.74, 6.19, 4.68 [02:29:29] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.17, 5.77, 4.71 [03:04:54] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 0.71, 0.87, 1.86 [03:08:54] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 0.51, 0.89, 1.65 [05:42:45] PROBLEM - bacula1 Disk Space on bacula1 is WARNING: DISK WARNING - free space: / 51885 MB (10% inode=99%); [06:09:13] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.92, 1.65, 1.40 [06:11:07] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 2.40, 1.86, 1.50 [06:13:01] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.64, 1.80, 1.52 [06:14:55] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.42, 1.70, 1.52 [06:25:34] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 3074 MB (12% inode=94%); [06:26:26] I will let you know when I see JohnLewis and I will deliver that message to them [06:26:26] @notify JohnLewis Can you please look at some of the extension review backlog? [09:14:45] PROBLEM - bacula1 Disk Space on bacula1 is CRITICAL: DISK CRITICAL - free space: / 28578 MB (5% inode=99%); [12:10:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeYdY [12:10:11] [02miraheze/services] 07MirahezeSSLBot 035f0fd47 - BOT: Updating services config for wikis [12:18:15] !log rhinos@test1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki colchaguawiki (following import) [12:18:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [12:28:54] PROBLEM - test1 Current Load on test1 is CRITICAL: CRITICAL - load average: 6.11, 2.94, 1.86 [12:30:34] ^ I'll move off [12:31:52] !log moved to mw1 (high load on test1) [12:31:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [12:40:54] PROBLEM - test1 Current Load on test1 is WARNING: WARNING - load average: 1.49, 1.75, 1.99 [12:46:54] RECOVERY - test1 Current Load on test1 is OK: OK - load average: 1.07, 1.27, 1.69 [12:47:12] !log rhinos@mw1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --update --active --wiki colchaguawiki (following import - observed dodgy stats) [12:47:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:15:20] Incase anyone was wondering I just took care of 99.9% of AWS ips [13:21:58] Zppix: huh [13:22:23] RhinosF1: https://meta.miraheze.org/wiki/Special:Log/Zppix [13:22:24] [ All public logs - Miraheze Meta ] - meta.miraheze.org [13:22:59] RhinosF1: around 3k ips [13:27:30] Zppix: good [15:13:34] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2644 MB (10% inode=94%); [15:15:33] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2671 MB (11% inode=94%); [15:23:34] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2649 MB (10% inode=94%); [15:25:34] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 2670 MB (11% inode=94%); [15:53:28] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 2649 MB (10% inode=94%); [16:04:21] !log reboot test1 [16:04:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:05:14] PROBLEM - test1 Puppet on test1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:06:50] PROBLEM - test1 php-fpm on test1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:07:18] PROBLEM - test1 Current Load on test1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:07:32] PROBLEM - test1 Disk Space on test1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:07:33] PROBLEM - cp3 Stunnel Http for test1 on cp3 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:07:34] PROBLEM - cp4 Stunnel Http for test1 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:07:59] PROBLEM - Host test1 is DOWN: PING CRITICAL - Packet loss = 100% [16:08:26] PROBLEM - cp2 Stunnel Http for test1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:09:59] RECOVERY - Host test1 is UP: PING OK - Packet loss = 0%, RTA = 0.37 ms [16:10:03] PROBLEM - test1 HTTPS on test1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:10:41] RECOVERY - test1 php-fpm on test1 is OK: PROCS OK: 3 processes with command name 'php-fpm7.3' [16:11:34] RECOVERY - cp4 Stunnel Http for test1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24499 bytes in 0.012 second response time [16:11:42] RECOVERY - cp3 Stunnel Http for test1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24499 bytes in 0.806 second response time [16:12:23] RECOVERY - test1 HTTPS on test1 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 444 bytes in 0.008 second response time [16:12:27] RECOVERY - cp2 Stunnel Http for test1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24499 bytes in 0.499 second response time [16:13:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeYNx [16:13:31] [02miraheze/puppet] 07paladox 0388cd456 - mediawiki: Set gluster client mount to log at warning level not info [16:17:31] PROBLEM - test1 Puppet on test1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:18:42] PROBLEM - cp2 Stunnel Http for test1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:18:49] PROBLEM - test1 HTTPS on test1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:18:50] PROBLEM - test1 php-fpm on test1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:19:31] PROBLEM - test1 Puppet on test1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 1 day ago with 1 failures [16:19:45] PROBLEM - cp4 Stunnel Http for test1 on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [16:20:04] PROBLEM - cp3 Stunnel Http for test1 on cp3 is CRITICAL: HTTP CRITICAL - No data received from host [16:20:37] RECOVERY - cp2 Stunnel Http for test1 on cp2 is OK: HTTP OK: HTTP/1.1 200 OK - 24499 bytes in 0.484 second response time [16:20:44] RECOVERY - test1 php-fpm on test1 is OK: PROCS OK: 3 processes with command name 'php-fpm7.3' [16:20:48] RECOVERY - test1 HTTPS on test1 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 444 bytes in 0.019 second response time [16:21:40] RECOVERY - cp4 Stunnel Http for test1 on cp4 is OK: HTTP OK: HTTP/1.1 200 OK - 24499 bytes in 0.030 second response time [16:22:05] RECOVERY - cp3 Stunnel Http for test1 on cp3 is OK: HTTP OK: HTTP/1.1 200 OK - 24499 bytes in 0.864 second response time [17:20:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeYAg [17:20:11] [02miraheze/services] 07MirahezeSSLBot 036ae25d6 - BOT: Updating services config for wikis [17:25:09] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeYAr [17:25:11] [02miraheze/services] 07MirahezeSSLBot 03be7a6c2 - BOT: Updating services config for wikis [17:30:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeYAX [17:30:12] [02miraheze/services] 07MirahezeSSLBot 03c9e4a7b - BOT: Updating services config for wikis [17:36:19] such update [17:40:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeYAy [17:40:12] [02miraheze/services] 07MirahezeSSLBot 03b1a4790 - BOT: Updating services config for wikis [17:42:17] Zppix: huh? [17:44:29] RhinosF1: the github bot [17:45:05] Zppix: what's unusual though? [17:46:55] RhinosF1: nothing :P [17:47:23] Okay.... Random comment then [18:10:11] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeYxZ [18:10:12] [02miraheze/services] 07MirahezeSSLBot 031159cf3 - BOT: Updating services config for wikis [18:16:34] !log depool mw1 [18:16:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:20:20] !log repool mw1 temporarily [18:20:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:30:38] !log depool mw1 temporarily [18:30:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:33:23] PROBLEM - mw3 Current Load on mw3 is CRITICAL: CRITICAL - load average: 8.39, 6.27, 5.39 [18:34:08] !log repool mw1 [18:34:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [18:34:22] PROBLEM - mw2 Current Load on mw2 is CRITICAL: CRITICAL - load average: 8.07, 5.67, 3.76 [18:35:22] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 5.97, 6.48, 5.60 [18:36:22] RECOVERY - mw2 Current Load on mw2 is OK: OK - load average: 4.73, 5.00, 3.73 [18:36:27] PROBLEM - mw1 Puppet on mw1 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 3 minutes ago with 0 failures [18:39:20] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.20, 6.81, 5.90 [18:41:19] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.87, 6.20, 5.79 [19:01:32] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [19:01:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JeYpJ [19:01:35] [02miraheze/puppet] 07paladox 03ab1d699 - Remove mw[456] [19:01:36] [02puppet] 07paladox opened pull request 03#1084: Remove mw[456] - 13https://git.io/JeYpU [19:02:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JeYpT [19:02:15] [02miraheze/puppet] 07paladox 03a8933ff - Update stunnel.conf [19:02:16] [02puppet] 07paladox synchronize pull request 03#1084: Remove mw[456] - 13https://git.io/JeYpU [19:02:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JeYpk [19:02:36] [02miraheze/puppet] 07paladox 03f96b51f - Update config.yaml [19:02:37] [02puppet] 07paladox synchronize pull request 03#1084: Remove mw[456] - 13https://git.io/JeYpU [19:02:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JeYpI [19:02:46] [02miraheze/puppet] 07paladox 03d0c3df3 - Update storage_firewall.yaml [19:02:48] [02puppet] 07paladox synchronize pull request 03#1084: Remove mw[456] - 13https://git.io/JeYpU [19:03:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JeYpt [19:03:06] [02miraheze/puppet] 07paladox 03013515d - Update site.pp [19:03:07] [02puppet] 07paladox synchronize pull request 03#1084: Remove mw[456] - 13https://git.io/JeYpU [19:03:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JeYpm [19:03:38] [02miraheze/puppet] 07paladox 03e08ae6b - Update irc.pp [19:03:39] [02puppet] 07paladox synchronize pull request 03#1084: Remove mw[456] - 13https://git.io/JeYpU [19:04:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JeYpY [19:04:04] [02miraheze/puppet] 07paladox 03be29e47 - Update varnish.pp [19:04:05] [02puppet] 07paladox synchronize pull request 03#1084: Remove mw[456] - 13https://git.io/JeYpU [19:04:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JeYpO [19:04:26] [02miraheze/puppet] 07paladox 03f340020 - Update services.pp [19:04:27] [02puppet] 07paladox synchronize pull request 03#1084: Remove mw[456] - 13https://git.io/JeYpU [19:04:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JeYp3 [19:04:48] [02miraheze/puppet] 07paladox 0355b75e6 - Update db.pp [19:04:50] [02puppet] 07paladox synchronize pull request 03#1084: Remove mw[456] - 13https://git.io/JeYpU [19:05:36] !log revoke mw[456] cert on puppet1 [19:05:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:05:59] [02puppet] 07paladox closed pull request 03#1084: Remove mw[456] - 13https://git.io/JeYpU [19:06:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±9] 13https://git.io/JeYpZ [19:06:02] [02miraheze/puppet] 07paladox 03e1bd34e - Remove mw[456] (#1084) * Remove mw[456] * Update stunnel.conf * Update config.yaml * Update storage_firewall.yaml * Update site.pp * Update irc.pp * Update varnish.pp * Update services.pp * Update db.pp [19:09:10] paladox: https://perfectrobloxgames.miraheze.org/wiki/Special:Upload lock file error again (per SN) [19:09:11] [ Login required - Perfect Roblox Games Wiki ] - perfectrobloxgames.miraheze.org [19:10:32] RhinosF1 fixed [19:10:38] !log root@mw2:/mnt/mediawiki-static# mkdir -p perfectrobloxgameswiki [19:10:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:11:09] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 2 backends are down. mw4 mw5 [19:11:13] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 2 backends are down. mw4 mw5 [19:11:31] known ^ [19:12:26] !log revoking firewalls on all servers for mw[456] [19:12:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:13:09] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [19:13:10] paladox: setting up new mw* ? [19:13:13] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [19:13:19] nope, more like removing them [19:13:30] didnt we just get them? [19:13:40] Yeh, they were an experiment. [19:13:45] PROBLEM - mw4 Puppet on mw4 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:13:46] ^ like this order more :P [19:14:04] PROBLEM - mw6 Puppet on mw6 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:14:05] SPF|Cloud: looks good [19:14:13] PROBLEM - mw5 Puppet on mw5 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:14:31] SPF|Cloud: no no its all wrong scrap the entire topic and rewrite from scratch :P jk [19:15:52] Zppix: Welcome to Miraheze! you know who we are https://meta.miraheze.org | Status: not down (yet?) | You can haz logs? http://wm-bot.wmflabs.org/browser/index.php?display=%23miraheze | By participating in this channel, you agree to abide by our Code of Conduct: https://meta.miraheze.org/m/Np [19:15:54] better? [19:15:56] [ Meta ] - meta.miraheze.org [19:15:57] [ Wikimedia IRC logs browser ] - wm-bot.wmflabs.org [19:15:58] [ Code of Conduct - Miraheze Meta ] - meta.miraheze.org [19:16:31] SPF|Cloud: perfect except s/you agree to abibe by our code of conduct/follow the code of conduct or be hung [19:16:41] haha :P [19:17:05] SPF|Cloud: or is my correction to passive aggressive? [19:17:13] I don't know [19:17:48] also dont try to use my bot's weather cmd [19:18:01] if you do, well then youll figure out why [19:26:53] !log delete mw[456] [19:26:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [19:28:13] Zppix: you make me want to try using it [19:28:25] RhinosF1: i mean go ahead it wont crash it [19:28:34] but it gets interesting [19:28:37] .help weather [19:28:39] RhinosF1: .weather location - Show the weather at the given location. [19:28:40] RhinosF1: e.g. .weather w7174408 [19:28:46] .weather Leeds [19:28:48] Leeds, GB: 13°C (55°F), Rain, Humidity: 81%, Light breeze 2.6m/s (→) [19:28:49] Leeds, GB: 13°C (55°F), Rain, Humidity: 81%, Light breeze 2.6m/s (→) [19:28:50] Leeds, GB: 13°C (55°F), Rain, Humidity: 81%, Light breeze 2.6m/s (→) [19:28:51] Leeds, GB: 13°C (55°F), Rain, Humidity: 81%, Light breeze 2.6m/s (→) [19:28:52] Leeds, GB: 13°C (55°F), Rain, Humidity: 81%, Light breeze 2.6m/s (→) [19:28:54] ... [19:28:55] ... [19:28:56] ... [19:29:03] well it did more then last i used it [19:29:15] it actually sent something besides an error [19:29:26] RhinosF1: it auto kills the cmd if it loops x amount of times [19:29:31] PROBLEM - mw6 SSH on mw6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:29:38] Zppix: good [19:29:45] I was about to shut it up [19:29:54] PROBLEM - mw5 HTTPS on mw5 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:29:55] RhinosF1: thats not nice, poor ZppixBot [19:30:01] PROBLEM - mw6 php-fpm on mw6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:30:05] PROBLEM - mw6 Current Load on mw6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:30:06] PROBLEM - mw6 HTTPS on mw6 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:30:06] PROBLEM - mw4 Disk Space on mw4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:30:13] PROBLEM - mw5 Disk Space on mw5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:30:15] PROBLEM - mw5 Current Load on mw5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:30:19] Zppix: if the loop didn't stop [19:30:21] PROBLEM - mw4 SSH on mw4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:30:21] PROBLEM - mw6 Disk Space on mw6 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:30:23] PROBLEM - mw4 HTTPS on mw4 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:30:32] PROBLEM - mw4 php-fpm on mw4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:30:53] PROBLEM - mw4 Current Load on mw4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:30:53] paladox: ^ that u? [19:30:57] .cry [19:30:57] ( p′︵‵。) [19:31:01] Yes [19:31:04] K [19:31:06] please read the log :) [19:31:08] ZppixBot: aww [19:31:10] PROBLEM - mw5 SSH on mw5 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:31:23] Ok cool [19:31:27] PROBLEM - mw5 php-fpm on mw5 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [19:31:48] PROBLEM - Host mw4 is DOWN: PING CRITICAL - Packet loss = 100% [19:32:06] PROBLEM - Host mw6 is DOWN: PING CRITICAL - Packet loss = 100% [19:32:11] PROBLEM - Host mw5 is DOWN: PING CRITICAL - Packet loss = 100% [19:44:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeYp1 [19:44:15] [02miraheze/puppet] 07paladox 032899ed5 - mediawiki: Set reader-thread-count to 20 for glusterfs [19:45:50] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.95, 6.92, 6.05 [19:46:27] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [19:47:49] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 4.22, 6.04, 5.84 [19:59:02] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 128.199.139.216/cpweb [20:02:48] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 2 datacenters are down: 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [20:03:41] PROBLEM - mw3 Current Load on mw3 is WARNING: WARNING - load average: 7.75, 6.66, 6.06 [20:04:43] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [20:05:01] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:05:40] RECOVERY - mw3 Current Load on mw3 is OK: OK - load average: 6.20, 6.39, 6.03 [20:08:05] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw1 [20:09:00] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 128.199.139.216/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [20:10:01] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [20:11:00] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:25:13] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw3 [20:25:27] [02mediawiki] 07RhinosF1 opened pull request 03#125: Add AdvancedSearch and CommonsMetadata - 13https://git.io/JeYhY [20:27:13] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [20:30:17] [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-1 - 13https://git.io/vbvb3 [20:30:19] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JeYhZ [20:30:20] [02miraheze/mw-config] 07RhinosF1 038899edb - add 3 extensions to local extensions [20:30:36] [02mw-config] 07RhinosF1 opened pull request 03#2758: Install 3 extensions to test1 - 13https://git.io/JeYhc [20:31:22] 503 [20:31:38] * RhinosF1 isn't touching anything [20:31:56] paladox: ^ [20:32:36] * paladox looks [20:32:47] it's back up for me [20:32:47] not the current issue, but someone should clear db4 soon [20:33:09] RhinosF1: Voidwalker lol https://meta.miraheze.org/wiki/Meta:Requests_for_permissions#Sysop [20:33:14] i see: [20:33:15] Sep 15 20:30:16 mw1 mfsmount[30589]: read file error, inode: 1406487, index: 0, chunk: 1079044, version: 1 - Chunkserver communication timed out: 81.4.122.238:9422 (try counter: 1) [20:33:21] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 4 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [20:33:35] Voidwalker done, thanks! [20:33:56] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [20:34:00] [ Meta:Requests for permissions - Miraheze Meta ] - meta.miraheze.org [20:34:04] what is going on... jesus [20:34:34] It's LizardFS [20:34:44] i'm currently working on a replacement [20:34:52] but that's going to take time to copy data [20:35:03] * Voidwalker doesn't like LFS [20:35:04] paladox: it cant decide if it wants to 503 or not thats what is annoying [20:35:15] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [20:35:20] Zppix that's because it goes up or down :P [20:35:27] Voidwalker i feel that way now :P [20:35:55] Voidwalker: Honestly i want to just close that request for sysop i just linked but i honestly want to see it play out just for shits and gigles [20:35:56] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [20:35:57] giggles* [20:36:09] also need a more permanent solution to db4 [20:36:48] Zppix, leave it open, it's not going anywhere >:D [20:37:00] Voidwalker: i am thats what i just said lol [20:37:32] Voidwalker: we should create a fake userright named admin and give it like no perms and assign it to him and close the request as success :P [20:38:38] if the request goes anywhere, I'd absolutely bring up the fact that he's blocked on three of his top five most edited wikis [20:38:52] and one of them is for abusing admin privileges [20:39:07] Voidwalker: if it goes anywhere I will legit lock my own account [20:39:20] lol [20:39:53] Voidwalker: i mean he wanted steward back in augst [20:39:55] august* [20:40:48] his MO seems to be to ask for adminship/advanced rights on wikis [20:41:06] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JeYhE [20:41:07] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 4 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [20:41:07] [02miraheze/mw-config] 07RhinosF1 030378d1e - Update LocalSettings.php [20:41:07] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw3 [20:41:09] [02mw-config] 07RhinosF1 synchronize pull request 03#2758: Install 3 extensions to test1 - 13https://git.io/JeYhc [20:41:09] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw3 [20:41:22] I tend to be lenient on hat collecting, but that's hat collecting [20:41:51] Zppix: I'm going to ignore it because it just isn't worth it - it will fail [20:42:06] RhinosF1: Meh free edit count :P [20:42:35] mw3 mfsmount[22991]: read file error, inode: 424242, index: 0, chunk: 4116886, version: 1 - Chunkserver communication timed out: 81.4.122.238:9422 (try counter: 1) [20:42:37] Zppix: meh, I spend most of my time doing logged actions rather than edits or blowing the servers up [20:42:56] RhinosF1: wait.. blowing servers up is my job [20:43:01] miraheze/mw-config/RhinosF1-patch-1/0378d1e - RhinosF1 The build was broken. https://travis-ci.org/miraheze/mw-config/builds/585303072 [20:43:03] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [20:43:07] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [20:43:08] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeYha [20:43:09] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [20:43:10] [02miraheze/puppet] 07paladox 0367ecbaa - Install telnet as a global package [20:43:15] * Voidwalker hides the logic bomb [20:43:35] Voidwalker: its cute that you think I use logic in anything [20:43:49] https://github.com/lizardfs/lizardfs/issues/377 [20:43:50] [ Chunkserver communication timed out · Issue #377 · lizardfs/lizardfs · GitHub ] - github.com [20:43:53] :P [20:44:12] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JeYhw [20:44:14] [02miraheze/mw-config] 07RhinosF1 03f387dd3 - Update LocalSettings.php [20:44:15] [02mw-config] 07RhinosF1 synchronize pull request 03#2758: Install 3 extensions to test1 - 13https://git.io/JeYhc [20:44:29] what is Lizardfs anyway? [20:44:55] * RhinosF1 has fixed travis [20:45:34] Zppix it's a network file storage solution. [20:45:34] https://lizardfs.com [20:45:38] [ LizardFS - Get your storage up and running in 28 minutes ] - lizardfs.com [20:45:40] i'm trying glusterfs now :P [20:46:15] "up and running" more like down and failing [20:46:21] miraheze/mw-config/RhinosF1-patch-1/0378d1e - RhinosF1 The build was broken. https://travis-ci.org/miraheze/mw-config/builds/585303072 [20:46:51] shut up travis [20:50:22] :P [20:54:15] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JeYhX [20:54:17] [02miraheze/mw-config] 07RhinosF1 039804374 - +3 [20:54:18] [02mw-config] 07RhinosF1 synchronize pull request 03#2758: Install 3 extensions to test1 - 13https://git.io/JeYhc [20:56:06] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JeYhS [20:56:07] [02miraheze/mw-config] 07RhinosF1 03670929a - +2 [20:56:09] [02mw-config] 07RhinosF1 synchronize pull request 03#2758: Install 3 extensions to test1 - 13https://git.io/JeYhc [20:58:11] [02mediawiki] 07RhinosF1 closed pull request 03#125: Add AdvancedSearch and CommonsMetadata - 13https://git.io/JeYhY [20:58:13] [02miraheze/mediawiki] 07RhinosF1 pushed 031 commit to 03REL1_33 [+2/-0/±1] 13https://git.io/JeYhQ [20:58:14] [02miraheze/mediawiki] 07RhinosF1 037a21d1c - Add AdvancedSearch and CommonsMetadata (#125) [20:59:20] [02mw-config] 07RhinosF1 closed pull request 03#2758: Install 3 extensions to test1 - 13https://git.io/JeYhc [20:59:22] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±4] 13https://git.io/JeYh7 [20:59:23] [02miraheze/mw-config] 07RhinosF1 033f53a9d - Install 3 extensions to test1 (#2758) [21:03:52] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 6 datacenters are down: 107.191.126.23/cpweb, 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2400:6180:0:d0::403:f001/cpweb, 81.4.109.133/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [21:04:27] !log sudo -u www-data php /srv/mediawiki/w/maintenance/mergeMessageFileList.php --output /srv/mediawiki/config/ExtensionMessageFiles.php --wiki loginwiki [21:04:53] !log sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki (both on test1) [21:05:07] PROBLEM - misc1 GDNSD Datacenters on misc1 is CRITICAL: CRITICAL - 3 datacenters are down: 2604:180:0:33b::2/cpweb, 128.199.139.216/cpweb, 2a00:d880:5:8ea::ebc7/cpweb [21:05:09] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw3 [21:05:13] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw2 [21:05:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:06:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:07:25] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeYjU [21:07:26] [02miraheze/mw-config] 07RhinosF1 03dbcd78e - Update extension-list [21:09:37] !log rerun both due to error in commit [21:10:08] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JeYjm [21:10:09] [02miraheze/services] 07MirahezeSSLBot 03eba340b - BOT: Updating services config for wikis [21:10:19] paladox: 503 [21:10:20] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 3 backends are down. mw1 mw2 mw3 [21:10:31] that'll be... lizardfs again [21:10:36] :( [21:10:52] paladox: pls stop scaring me as I try to test [21:11:11] Sep 15 21:10:06 mw1 mfsmount[30589]: read file error, inode: 4347786, index: 0, chunk: 5366309, version: 1 - Chunkserver communication timed out: 81.4.122.238:9422 (try counter: 1) [21:12:20] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [21:13:07] RECOVERY - misc1 GDNSD Datacenters on misc1 is OK: OK - all datacenters are online [21:13:13] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [21:13:50] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [21:15:09] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [21:17:01] paladox: can you get commons meta data working on test1 [21:17:08] https://www.mediawiki.org/wiki/Extension:CommonsMetadata [21:17:08] [ Extension:CommonsMetadata - MediaWiki ] - www.mediawiki.org [21:17:54] Sorry i'm busy with gluster :( [21:18:03] * RhinosF1 is busy fixing i18n [21:19:02] !log sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki test1wiki [21:20:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:29:00] paladox: I need uploads on test1 for like 2 minutes to test [21:29:17] You'll need to test using gluster [21:29:31] paladox: Special:Upload isn't working [21:29:36] It should [21:29:47] Could not open lock file for "mwstore://local-backend/local-public/8/80/White.png". Make sure your upload directory is configured correctly and your web server has permission to write to that directory. See https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:$wgUploadDirectory for more information. [21:29:47] [ Manual:$wgUploadDirectory - MediaWiki ] - www.mediawiki.org [21:29:50] paladox: ^ [21:30:54] hmm [21:32:33] RhinosF1 try now [21:33:52] paladox: nope [21:35:01] RhinosF1 try now [21:36:59] paladox: file uploaded but image page is 404ing - https://test1.miraheze.org/wiki/File:White.png [21:37:00] [ File:White.png - Test1 ] - test1.miraheze.org [21:37:08] https://static.miraheze.org/test1wiki/8/80/White.png [21:37:09] [ 404 Not Found ] - static.miraheze.org [21:37:28] wierd... [21:37:34] root@test1:/mnt/mediawiki-static/test1wiki# ls 8/80/White.png [21:37:34] 8/80/White.png [21:38:09] oh! [21:38:11] static [21:39:17] paladox: what? [21:39:23] RhinosF1 now it works [21:39:28] static goes through mw* [21:39:39] so it wouldn't pick up test1 gluster change [21:40:14] * RhinosF1 signs off commons meta data [21:40:22] anyone any issues before I merge [21:40:28] to mw* [21:42:38] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-2 [+0/-0/±1] 13https://git.io/JeYj6 [21:42:39] [02miraheze/mw-config] 07RhinosF1 03f82ab55 - move 3 extensions to prod [21:42:41] [02mw-config] 07RhinosF1 created branch 03RhinosF1-patch-2 - 13https://git.io/vbvb3 [21:42:50] [02mw-config] 07RhinosF1 opened pull request 03#2759: move 3 extensions to prod - 13https://git.io/JeYji [21:43:55] miraheze/mw-config/RhinosF1-patch-2/f82ab55 - RhinosF1 The build was broken. https://travis-ci.org/miraheze/mw-config/builds/585318290 [21:45:39] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-2 [+0/-0/±1] 13https://git.io/JeYjy [21:45:41] [02miraheze/mw-config] 07RhinosF1 03d63bb41 - turn off 3 extensions on test1 and change CommonsMetadata setting to true (was true only on test1) [21:45:42] [02mw-config] 07RhinosF1 synchronize pull request 03#2759: move 3 extensions to prod - 13https://git.io/JeYji [21:47:10] miraheze/mw-config/RhinosF1-patch-2/d63bb41 - RhinosF1 The build is still failing. https://travis-ci.org/miraheze/mw-config/builds/585318944 [21:47:31] paladox: ^ I haven't touched that area of the code [21:48:11] I'm not sure what your asking? Me to review? [21:48:14] [02mw-config] 07paladox reviewed pull request 03#2759 commit - 13https://git.io/JeYj9 [21:48:33] [02mw-config] 07paladox reviewed pull request 03#2759 commit - 13https://git.io/JeYjH [21:49:39] [02mw-config] 07RhinosF1 reviewed pull request 03#2759 commit - 13https://git.io/JeYj7 [21:50:18] paladox: responded [21:51:22] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-2 [+0/-0/±1] 13https://git.io/JeYjb [21:51:23] [02miraheze/mw-config] 07RhinosF1 034864df9 - Update ManageWikiExtensions.php [21:51:25] [02mw-config] 07RhinosF1 synchronize pull request 03#2759: move 3 extensions to prod - 13https://git.io/JeYji [21:53:00] [02mw-config] 07paladox reviewed pull request 03#2759 commit - 13https://git.io/JeYjA [21:54:07] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-2 [+0/-0/±1] 13https://git.io/JeYjx [21:54:08] [02miraheze/mw-config] 07RhinosF1 03470b79a - per @paladox [21:54:10] [02mw-config] 07RhinosF1 synchronize pull request 03#2759: move 3 extensions to prod - 13https://git.io/JeYji [21:55:01] RhinosF1 uh [21:55:02] https://github.com/miraheze/mediawiki/pull/125/files [21:55:03] [ Add AdvancedSearch and CommonsMetadata by RhinosF1 · Pull Request #125 · miraheze/mediawiki · GitHub ] - github.com [21:55:37] revertoh [21:55:38] oh [21:55:55] paladox: huh, all I did was sync our repo with my fork and then add the submodules, pls revert yes [21:56:34] ok [21:57:03]