[00:05:31] RECOVERY - db4 Current Load on db4 is OK: OK - load average: 6.54, 6.78, 6.72 [00:09:30] PROBLEM - db4 Current Load on db4 is WARNING: WARNING - load average: 7.79, 7.58, 7.10 [00:19:30] RECOVERY - db4 Current Load on db4 is OK: OK - load average: 5.55, 6.50, 6.77 [00:21:34] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjOQL [00:21:36] [02miraheze/ManageWiki] 07paladox 0306a0f5c - Fix ManageWikiCDB::delete [00:22:11] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fjOQt [00:22:12] [02miraheze/mediawiki] 07paladox 03441b4f1 - Update MW [00:23:30] PROBLEM - db4 Current Load on db4 is WARNING: WARNING - load average: 7.24, 7.47, 7.14 [00:27:30] PROBLEM - db4 Current Load on db4 is CRITICAL: CRITICAL - load average: 8.76, 7.47, 7.15 [00:31:30] PROBLEM - db4 Current Load on db4 is WARNING: WARNING - load average: 7.67, 7.88, 7.43 [00:43:30] PROBLEM - db4 Current Load on db4 is CRITICAL: CRITICAL - load average: 9.06, 7.57, 7.39 [00:45:30] PROBLEM - db4 Current Load on db4 is WARNING: WARNING - load average: 7.03, 7.35, 7.33 [00:47:22] [02ManageWiki] 07The-Voidwalker opened pull request 03#86: fix if else execution - 13https://git.io/fjOQ4 [00:47:30] PROBLEM - db4 Current Load on db4 is CRITICAL: CRITICAL - load average: 8.71, 7.53, 7.38 [00:48:00] [02ManageWiki] 07paladox commented on pull request 03#86: fix if else execution - 13https://git.io/fjOQB [00:48:04] [02ManageWiki] 07paladox closed pull request 03#86: fix if else execution - 13https://git.io/fjOQ4 [00:48:06] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjOQR [00:48:08] [02miraheze/ManageWiki] 07The-Voidwalker 033cfcfc8 - fix if else execution (#86) null == false, so all versions get set to 0, and the cache is never set properly [00:48:40] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fjOQ0 [00:48:42] [02miraheze/mediawiki] 07paladox 03ceae954 - Update MW [00:49:09] PROBLEM - cp3 Disk Space on cp3 is WARNING: DISK WARNING - free space: / 5060 MB (20% inode=95%); [00:49:30] PROBLEM - db4 Current Load on db4 is WARNING: WARNING - load average: 5.07, 6.68, 7.09 [00:51:30] RECOVERY - db4 Current Load on db4 is OK: OK - load average: 3.08, 5.49, 6.61 [01:09:43] !log update mw_permissions set perm_addgroups = '[]' where perm_addgroups like 'null%'; on db4 [01:09:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:09:55] !log update mw_permissions set perm_removegroups = '[]' where perm_removegroups like 'null%' on db4 [01:09:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [01:17:16] PROBLEM - bacula1 Disk Space on bacula1 is WARNING: DISK WARNING - free space: / 98978 MB (20% inode=99%); [06:12:43] PROBLEM - mw3 JobQueue on mw3 is CRITICAL: JOBQUEUE CRITICAL - job queue greater than 300 jobs. Current queue: 597 [06:22:43] RECOVERY - mw3 JobQueue on mw3 is OK: JOBQUEUE OK - job queue below 300 jobs [06:27:09] RECOVERY - cp3 Disk Space on cp3 is OK: DISK OK - free space: / 5511 MB (22% inode=95%); [10:11:36] PROBLEM - bacula1 Puppet on bacula1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [10:13:11] PROBLEM - bacula1 Current Load on bacula1 is CRITICAL: CRITICAL - load average: 2.52, 1.84, 0.82 [10:13:36] RECOVERY - bacula1 Puppet on bacula1 is OK: OK: Puppet is currently enabled, last run 11 minutes ago with 0 failures [10:15:11] RECOVERY - bacula1 Current Load on bacula1 is OK: OK - load average: 0.57, 1.35, 0.77 [13:20:21] !log installing gdb on mw2 [13:20:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [13:21:44] !log running gdb for php (due to a segmentation fault) [13:21:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:08:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fjOxc [14:08:22] [02miraheze/puppet] 07paladox 036337d58 - Tweek php-fpm params some more Change-Id: I6abe28199ccdce4dfffbd0e37ce2ddafcc85f361 [14:10:06] PROBLEM - mw3 Puppet on mw3 is WARNING: WARNING: Puppet is currently disabled, message: php tweeks incomming - paladox, last run 7 minutes ago with 0 failures [14:10:21] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjOxl [14:10:22] [02miraheze/puppet] 07paladox 031a64e88 - php: Remove duplicate mysql configs Change-Id: Ic636e5b905c9df9ea11e3872c5649417221d8cf5 [14:10:38] PROBLEM - mw2 Puppet on mw2 is WARNING: WARNING: Puppet is currently disabled, message: php tweeks incomming - paladox, last run 7 minutes ago with 0 failures [14:14:38] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [14:16:06] RECOVERY - mw3 Puppet on mw3 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures [14:16:32] !log upgraded some packages on mw3 (php-fpm, nodejs, nginx, systemd, wget and a few more) [14:16:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:17:17] !log upgraded some packages on mw1 (php-fpm, nodejs, nginx, systemd, wget and a few more) [14:17:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:18:44] !log upgraded some packages on mw2 (php-fpm, nodejs, nginx, systemd, wget and a few more) [14:18:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [14:37:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±8] 13https://git.io/fjOx9 [14:37:18] [02miraheze/puppet] 07paladox 030d579f0 - tweek nginx confs a bit Change-Id: Ic4660bb6722b0dc886d9f35fbd3a776e4c60ab9e [14:40:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjOxd [14:40:49] [02miraheze/puppet] 07paladox 03f62931c - Fix tweeks Change-Id: Iceb1341d9b215f93ca079f326224de5bc243d707 [15:13:25] [02miraheze/ManageWiki] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjOp0 [15:13:27] [02miraheze/ManageWiki] 07paladox 03dcfa9db - Revert "Fix wgAddGroups and wgRemoveGroups json encoding null when they should be doing arrays (#85)" This reverts commit e5e829ddf4d034a867e24dfeaf0ee3f1c666a9ab. [15:13:51] [02mw-config] 07ucronistaw opened pull request 03#2676: add "olb" on wgExtraInterlanguageLinkPrefixe - 13https://git.io/fjOpu [15:21:03] [02mw-config] 07paladox closed pull request 03#2676: add "olb" on wgExtraInterlanguageLinkPrefixe - 13https://git.io/fjOpu [15:21:04] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjOp6 [15:21:06] [02miraheze/mw-config] 07ucronistaw 0356b8b77 - add "olb" on wgExtraInterlanguageLinkPrefixe (#2676) [15:28:07] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjOpQ [15:28:08] [02miraheze/puppet] 07paladox 0389b1517 - matomo: Incrase max-child for php-fpm Change-Id: I1b5a1ded1ec7262cdbf54569bbdde149e3d17bed [15:34:27] [02miraheze/ManageWiki] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjOpF [15:34:29] [02miraheze/ManageWiki] 07JohnFLewis 037b2322d - fix null for groups [15:34:38] [02miraheze/mw-config] 07paladox pushed 032 commits to 03master [+0/-0/±3] 13https://git.io/fjOpb [15:34:40] [02miraheze/mw-config] 07paladox 03af8847b - Migrate wgSVGMetadataCutoff to ManageWiki Change-Id: I39cacf2ac919eedb636468f8a2d3212a5f70eb14 [15:34:41] [02miraheze/mw-config] 07paladox 0366e47f0 - Merge https://github.com/miraheze/mw-config [15:34:42] Title: [ GitHub - miraheze/mw-config: Production MediaWiki configuration ] - github.com [15:35:38] [02miraheze/mediawiki] 07JohnFLewis pushed 033 commits to 03REL1_32 [+0/-0/±5] 13https://git.io/fjOpA [15:35:40] [02miraheze/mediawiki] 07JohnFLewis 036af7da8 - update MW [15:35:41] [02miraheze/mediawiki] 07JohnFLewis 03f7ffb39 - Merge branch 'REL1_32' of www.github.com:miraheze/mediawiki into REL1_32 [15:35:43] [02miraheze/mediawiki] 07JohnFLewis 033e3694d - update MW [15:38:26] !log UPDATE mw_permissions SET perm_addgroups = '[]' WHERE perm_addgroups = 'null'; [15:38:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [15:38:36] !log UPDATE mw_permissions SET perm_removegroups = '[]' WHERE perm_removegroups = 'null'; [15:38:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:39:41] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjOje [16:39:42] [02miraheze/MirahezeMagic] 07paladox 03088d3e8 - Add's "settings-localisation", "settings-restricted" and "settings-styling" i18n [16:42:08] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fjOjt [16:42:10] [02miraheze/MirahezeMagic] 07paladox 03e11ef35 - Add's "settings-localisation", "settings-restricted", "settings-styling" and "extensions-skins" to qqq i18n [16:42:40] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fjOjq [16:42:42] [02miraheze/mediawiki] 07paladox 0309fe903 - Update MM [16:44:30] !log running lc on mw* [16:44:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [16:45:34] [02miraheze/mw-config] 07paladox pushed 033 commits to 03master [+0/-0/±3] 13https://git.io/fjOjG [16:45:35] [02miraheze/mw-config] 07paladox 03bb33e1b - Add wgEnableScaryTranscluding to ManageWiki Change-Id: Ic8abdfa734d4036308ca7b03b04a0810886d15f1 [16:45:37] [02miraheze/mw-config] 07paladox 0323859e6 - Break up ManageWiki#other Change-Id: Ia493ed4370548a53806b5bbe2e315181a838450b [16:45:38] [02miraheze/mw-config] 07paladox 037013342 - Merge changes Ia493ed43,Ic8abdfa7 * changes: Break up ManageWiki#other Add wgEnableScaryTranscluding to ManageWiki [16:52:25] [02miraheze/mw-config] 07paladox pushed 032 commits to 03master [+0/-0/±2] 13https://git.io/fjOjB [16:52:26] [02miraheze/mw-config] 07paladox 038e2ee58 - Fix missing section for some configs Change-Id: I5a8456de71d7e1eed09835607cefc377872393f7 [16:52:28] [02miraheze/mw-config] 07paladox 038901639 - Merge "Fix missing section for some configs" [19:02:27] Hello eduaddad! If you have any questions feel free to ask and someone should answer soon. [21:29:31] [02MirahezeMagic] 07paladox created branch 03paladox-patch-2 - 13https://git.io/fQRGX [21:29:33] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fj3JX [21:29:34] [02miraheze/MirahezeMagic] 07paladox 03552e554 - Add "settings-maps" i18n [21:29:36] [02MirahezeMagic] 07paladox opened pull request 03#75: Add "settings-maps" i18n - 13https://git.io/fj3J1 [21:29:57] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fj3JM [21:29:59] [02miraheze/MirahezeMagic] 07paladox 03cdaa907 - Update qqq.json [21:30:00] [02MirahezeMagic] 07paladox synchronize pull request 03#75: Add "settings-maps" i18n - 13https://git.io/fj3J1 [21:30:05] [02MirahezeMagic] 07paladox closed pull request 03#75: Add "settings-maps" i18n - 13https://git.io/fj3J1 [21:30:07] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fj3JD [21:30:08] [02miraheze/MirahezeMagic] 07paladox 035d50ab2 - Add "settings-maps" i18n (#75) * Add "settings-maps" i18n * Update qqq.json [21:30:10] [02MirahezeMagic] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/fQRGX [21:30:11] [02miraheze/MirahezeMagic] 07paladox deleted branch 03paladox-patch-2 [21:30:34] miraheze/MirahezeMagic/paladox-patch-2/552e554 - paladox The build is still failing. https://travis-ci.com/miraheze/MirahezeMagic/builds/109091250 [21:31:36] miraheze/MirahezeMagic/paladox-patch-2/cdaa907 - paladox The build is still failing. https://travis-ci.com/miraheze/MirahezeMagic/builds/109091264 [21:32:02] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fj3J9 [21:32:03] [02miraheze/mediawiki] 07paladox 03c5a0a1e - Update MM [21:35:26] !log depooled mw2 to reinstall php [21:35:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:37:47] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw2 [21:38:23] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw2 [21:38:30] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw2 [21:39:51] !log repooled mw2 [21:39:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log, Master [21:40:15] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fj3Jd [21:40:16] [02miraheze/services] 07MirahezeSSLBot 03e824486 - BOT: Updating services config for wikis [21:40:44] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [21:42:38] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [21:45:12] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fj3Jp [21:45:14] [02miraheze/puppet] 07paladox 03fd7d703 - php: Fix variable Change-Id: If643d3cd2fcc398fc429e2f7ba1b768bf69b8505 [21:46:30] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 5 backends are healthy [21:47:47] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 5 backends are healthy [21:48:23] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 5 backends are healthy [21:54:52] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fj3Uv [21:54:54] [02miraheze/mw-config] 07paladox 03fe884fa - Add 'wgKartographerWikivoyageMode' and 'wgKartographerUseMarkerStyle' to ManageWiki Change-Id: I7ffbac623511f45ab0f3dd0d8062ea66ba27abf7 [21:58:45] paladox: can you take a look at https://phabricator.miraheze.org/T4299 this seems to be another instance of special pages not updating [21:58:46] Title: [ ⚓ T4299 Update wiki statistics ] - phabricator.miraheze.org [21:59:02] That was me when i ran the script [21:59:07] So the script did that. [21:59:33] paladox: but if you ran the script it should update? [21:59:46] paladox: regardless shouldnt special pages update automatically? [21:59:56] Yup, but the statistic showing wrong there was caused by the script [22:01:17] Zppix the script that did that was updateArticleCount.php [22:01:35] (i later ran updateSpecialPages.php which fixed one of there issue) [22:01:38] paladox: it also appears the url for that wiki is a 502 [22:01:46] 502? [22:02:23] paladox: nevermind cache issue [22:02:30] ok [22:02:59] i doin't think that was cache [22:03:10] you should not be served a 5xxx from the cache. [22:11:56] paladox: regardless it was just my end [22:13:34] paladox: i closed the task as LGTM, and told them to reopen if it comes back [22:13:39] ok [22:14:01] Zppix https://phabricator.miraheze.org/T4299#82130 [22:14:02] Title: [ ⚓ T4299 Update wiki statistics ] - phabricator.miraheze.org [22:28:56] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fj3Uu [22:28:57] [02miraheze/mw-config] 07paladox 03a4303cb - Update wgEnableEmail to state mw exempt Change-Id: I3879f57542acb4636519ecbf80cee3c5c1815904 [22:39:18] [02MirahezeMagic] 07paladox created branch 03paladox-patch-2 - 13https://git.io/fQRGX [22:39:19] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fj3Uo [22:39:21] [02miraheze/MirahezeMagic] 07paladox 03f08300a - Add "settings-chat" to i18n [22:39:22] [02MirahezeMagic] 07paladox opened pull request 03#76: Add "settings-chat" to i18n - 13https://git.io/fj3UK [22:39:45] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/fj3U6 [22:39:47] [02miraheze/MirahezeMagic] 07paladox 0347f2782 - Update qqq.json [22:39:48] [02MirahezeMagic] 07paladox synchronize pull request 03#76: Add "settings-chat" to i18n - 13https://git.io/fj3UK [22:40:07] [02MirahezeMagic] 07paladox closed pull request 03#76: Add "settings-chat" to i18n - 13https://git.io/fj3UK [22:40:09] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fj3Ui [22:40:10] [02miraheze/MirahezeMagic] 07paladox 03a52f251 - Add "settings-chat" to i18n (#76) * Add "settings-chat" to i18n * Update qqq.json [22:40:37] miraheze/MirahezeMagic/paladox-patch-2/f08300a - paladox The build was fixed. https://travis-ci.com/miraheze/MirahezeMagic/builds/109093262 [22:40:45] [02MirahezeMagic] 07paladox deleted branch 03paladox-patch-2 - 13https://git.io/fQRGX [22:40:47] [02miraheze/MirahezeMagic] 07paladox deleted branch 03paladox-patch-2 [22:41:22] [02miraheze/mediawiki] 07paladox pushed 031 commit to 03REL1_32 [+0/-0/±1] 13https://git.io/fj3UX [22:41:23] [02miraheze/mediawiki] 07paladox 03f2656a5 - Update MM [22:41:25] miraheze/MirahezeMagic/paladox-patch-2/47f2782 - paladox The build is still failing. https://travis-ci.com/miraheze/MirahezeMagic/builds/109093273 [22:48:35] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fj3U1 [22:48:37] [02miraheze/mw-config] 07paladox 032f54acc - Add wgChatLinkUsernames and wgChatMeCommand to ManageWiki Change-Id: I1c9e2c73247f25d9c3a91f18f2b41635826d139f [23:05:10] [02miraheze/services] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/fj3Tf [23:05:12] [02miraheze/services] 07MirahezeSSLBot 032b34b22 - BOT: Updating services config for wikis [23:10:39] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fj3TL [23:10:40] [02miraheze/mw-config] 07paladox 034e0f82f - Add "Metrolook" config's to ManageWiki Change-Id: I06d8f82a9228df29ed343a7ba7b2cc6ced77c3ca [23:34:26] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/fj3TM [23:34:28] [02miraheze/mw-config] 07paladox 0371de41c - Add a few misc configs to ManageWiki Change-Id: I0e69d2aa95c89e1a1b82d1eb7050d9b3545aeb4c