[00:13:41] RECOVERY - puppet last run on ms-fe2004 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [00:17:46] (03PS14) 10Paladox: Gerrit: Support having phab commits as links [puppet] - 10https://gerrit.wikimedia.org/r/302229 (https://phabricator.wikimedia.org/T76459) [00:55:22] 06Operations, 06Commons, 10media-storage: Storage backend errors on commons when deleting/restoring pages - https://phabricator.wikimedia.org/T141704#2530920 (10Josve05a) [01:02:20] Same issue with deleted pages showing up in RC [01:02:27] On Simple Wikipedia... [01:02:50] https://phabricator.wikimedia.org/T137280 [01:02:52] this ^ [01:03:48] It still exists, Glaisher [01:06:39] Oh, nevermind. It fixed itself. [01:33:40] PROBLEM - puppet last run on mw2177 is CRITICAL: CRITICAL: puppet fail [02:00:00] RECOVERY - puppet last run on mw2177 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [02:11:01] RECOVERY - Improperly owned -0:0- files in /srv/mediawiki-staging on mira is OK: Files ownership is ok. [02:19:05] !log mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.13) (duration: 08m 55s) [02:19:10] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:24:55] !log l10nupdate@tin ResourceLoader cache refresh completed at Sun Aug 7 02:24:55 UTC 2016 (duration 5m 51s) [02:25:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:33:00] PROBLEM - Improperly owned -0:0- files in /srv/mediawiki-staging on mira is CRITICAL: Improperly owned (0:0) files in /srv/mediawiki-staging [03:35:33] PROBLEM - puppet last run on mw1216 is CRITICAL: CRITICAL: Puppet has 1 failures [03:36:41] PROBLEM - puppet last run on db2051 is CRITICAL: CRITICAL: Puppet has 1 failures [03:37:02] PROBLEM - puppet last run on analytics1042 is CRITICAL: CRITICAL: Puppet has 1 failures [03:37:11] PROBLEM - puppet last run on mw1248 is CRITICAL: CRITICAL: Puppet has 2 failures [03:54:21] PROBLEM - puppet last run on uranium is CRITICAL: CRITICAL: puppet fail [03:55:31] PROBLEM - puppet last run on mw2193 is CRITICAL: CRITICAL: puppet fail [04:02:21] RECOVERY - puppet last run on db2051 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [04:02:41] RECOVERY - puppet last run on analytics1042 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [04:02:51] RECOVERY - puppet last run on mw1248 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [04:03:11] RECOVERY - puppet last run on mw1216 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:22:01] RECOVERY - puppet last run on uranium is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [04:23:11] RECOVERY - puppet last run on mw2193 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [04:34:07] PROBLEM - Juniper alarms on asw-d-eqiad.mgmt.eqiad.wmnet is CRITICAL: JNX_ALARMS CRITICAL - No response from remote host 10.65.0.24 [04:35:47] RECOVERY - Juniper alarms on asw-d-eqiad.mgmt.eqiad.wmnet is OK: JNX_ALARMS OK - 0 red alarms, 0 yellow alarms [04:47:37] PROBLEM - puppet last run on analytics1033 is CRITICAL: CRITICAL: Puppet has 1 failures [05:14:38] RECOVERY - puppet last run on analytics1033 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [05:26:15] (03PS1) 10Yuvipanda: tools: Don't include etcd in k8s master [puppet] - 10https://gerrit.wikimedia.org/r/303405 [05:53:48] PROBLEM - puppet last run on ocg1002 is CRITICAL: CRITICAL: Puppet has 1 failures [06:08:27] PROBLEM - MD RAID on relforge1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [06:10:17] RECOVERY - MD RAID on relforge1001 is OK: OK: Active: 8, Working: 8, Failed: 0, Spare: 0 [06:19:27] RECOVERY - puppet last run on ocg1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:35:04] PROBLEM - puppet last run on mw2198 is CRITICAL: CRITICAL: Puppet has 1 failures [06:53:27] 06Operations, 06Commons, 06Multimedia, 10media-storage: Storage backend errors on commons when deleting/restoring pages - https://phabricator.wikimedia.org/T141704#2531142 (10aaron) a:05aaron>03None [07:00:34] RECOVERY - puppet last run on mw2198 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:04:28] PROBLEM - Juniper alarms on asw-d-eqiad.mgmt.eqiad.wmnet is CRITICAL: JNX_ALARMS CRITICAL - Received genError(5) error-status at error-index 1 [07:06:18] RECOVERY - Juniper alarms on asw-d-eqiad.mgmt.eqiad.wmnet is OK: JNX_ALARMS OK - 0 red alarms, 0 yellow alarms [07:08:44] 06Operations, 06Commons, 10MediaWiki-Page-deletion, 10media-storage, and 3 others: Unable to delete file pages on commons: MWException/LocalFileLockError: "Could not acquire lock" - https://phabricator.wikimedia.org/T132921#2531161 (10Steinsplitter) 05Resolved>03Open Unfortunately, they seems back. Cit... [07:12:41] 06Operations, 06Commons, 10MediaWiki-Page-deletion, 10media-storage, and 3 others: Unable to delete file pages on commons: MWException/LocalFileLockError: "Could not acquire lock" - https://phabricator.wikimedia.org/T132921#2531165 (10Steinsplitter) 05Open>03stalled [07:13:28] 06Operations, 06Commons, 10MediaWiki-Page-deletion, 10media-storage, and 3 others: Unable to delete file pages on commons: MWException/LocalFileLockError: "Could not acquire lock" - https://phabricator.wikimedia.org/T132921#2218103 (10Steinsplitter) T141704 is something else? [07:13:37] PROBLEM - MD RAID on relforge1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:14:33] 06Operations, 06Commons, 10MediaWiki-Page-deletion, 10media-storage, and 3 others: Unable to delete file pages on commons: MWException/LocalFileLockError: "Could not acquire lock" - https://phabricator.wikimedia.org/T132921#2531169 (10aaron) a:05aaron>03None [07:17:27] RECOVERY - MD RAID on relforge1001 is OK: OK: Active: 8, Working: 8, Failed: 0, Spare: 0 [07:44:59] PROBLEM - MD RAID on relforge1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [07:46:57] RECOVERY - MD RAID on relforge1001 is OK: OK: Active: 8, Working: 8, Failed: 0, Spare: 0 [08:12:32] PROBLEM - MD RAID on relforge1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [08:14:22] RECOVERY - MD RAID on relforge1001 is OK: OK: Active: 8, Working: 8, Failed: 0, Spare: 0 [08:30:12] PROBLEM - MD RAID on relforge1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [08:32:01] RECOVERY - MD RAID on relforge1001 is OK: OK: Active: 8, Working: 8, Failed: 0, Spare: 0 [08:57:21] PROBLEM - MD RAID on relforge1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [08:59:21] RECOVERY - MD RAID on relforge1001 is OK: OK: Active: 8, Working: 8, Failed: 0, Spare: 0 [09:04:12] PROBLEM - MD RAID on relforge1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [09:06:01] RECOVERY - MD RAID on relforge1001 is OK: OK: Active: 8, Working: 8, Failed: 0, Spare: 0 [09:27:22] PROBLEM - MD RAID on relforge1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [09:27:32] PROBLEM - MediaWiki exceptions and fatals per minute on graphite1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [50.0] [09:33:32] RECOVERY - MediaWiki exceptions and fatals per minute on graphite1001 is OK: OK: Less than 1.00% above the threshold [25.0] [09:35:12] RECOVERY - MD RAID on relforge1001 is OK: OK: Active: 8, Working: 8, Failed: 0, Spare: 0 [09:41:12] PROBLEM - MD RAID on relforge1001 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [09:41:21] PROBLEM - MediaWiki exceptions and fatals per minute on graphite1001 is CRITICAL: CRITICAL: 80.00% of data above the critical threshold [50.0] [09:45:02] RECOVERY - MD RAID on relforge1001 is OK: OK: Active: 8, Working: 8, Failed: 0, Spare: 0 [09:49:49] "he database is currently locked to new entries and other modifications, probably for routine database maintenance, after which it will be back to normal. " [09:49:53] The* [09:50:10] Mmk [09:50:13] "The system administrator who locked it offered this explanation: The database has been automatically locked while the slave database servers catch up to the master. " [09:52:52] PROBLEM - MediaWiki exceptions and fatals per minute on graphite1001 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [50.0] [09:54:51] RECOVERY - MediaWiki exceptions and fatals per minute on graphite1001 is OK: OK: Less than 1.00% above the threshold [25.0] [09:56:31] * Bsadowski1 pokes lots of people [09:56:39] * Bsadowski1 pokes Reedy [09:56:43] * Bsadowski1 pokes apergos [09:57:15] Bsadowski1: Is it still happening? [09:57:18] Yes [09:57:21] Just now [09:57:27] Possibly related to the above [09:57:30] Doing what? [09:57:47] I was trying to block an IP. [09:57:56] And before I was trying to rollback an edit. [09:58:00] This is on Meta. [09:58:11] Which wiki? [09:58:20] Meta [09:58:21] ... [09:58:27] Checking [09:58:29] meta.wikimedia.org [09:58:54] Hmm, tried to do the same action and same problem. [09:58:57] yep still locked [09:59:14] DB slave lag errors generally gets fixed within a few seconds [09:59:19] but apparently this is persisting [10:00:22] hmm "Due to high database server lag, changes newer than 34 seconds may not be shown in this list." [10:00:38] looks like jynus is not here atm [10:00:42] PROBLEM - MediaWiki exceptions and fatals per minute on graphite1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [50.0] [10:01:13] hmm i wonder if TimStarling is around, since its 8pm on a weekend but not marked as |away as per normal [10:02:53] looks like there's lag on s2 as well [10:02:57] but it's worse on s7 [10:03:10] (meta is on s7, iirc) [10:03:17] https://dbtree.wikimedia.org/ [10:04:20] looks fixed now [10:07:08] There were two global renames around that time.. maybe it was causing that again?.. [10:14:21] Okay. [10:14:27] Not sure. [10:14:30] Hold on.. [10:16:01] Doesn't appear to be, Glaisher. One had 7000 edits [10:16:21] PROBLEM - MediaWiki exceptions and fatals per minute on graphite1001 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [50.0] [10:16:45] ~7000 can also cause it if there is no proper waits during the updates [10:17:33] https://meta.wikimedia.org/wiki/Special:CentralAuth/Derakhshan fawiki is also on s7 [10:17:47] Oh [10:18:06] I'll report it on the task. [10:18:07] Weird. Thought it would be higher than that. [10:19:12] Is it still going on? [10:20:15] Hey Reedy :) [10:20:22] Doesn't appear to be. [10:21:40] Oh wait, scratch that. [10:21:48] It's still happening. [10:22:08] PROBLEM - MariaDB Slave Lag: s7 on db1028 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 315.58 seconds [10:22:12] ^^^ [10:22:20] ugh [10:22:47] you will need jynus [10:22:55] but he is not around [10:23:13] https://phabricator.wikimedia.org/T116425#2531281 [10:26:47] something something nice things something something [10:27:01] Bsadowski1 and Glaisher i guess create a task about the PROBLEM - MariaDB Slave Lag: s7 on db1028 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 315.58 and add jynus to it. [10:27:12] It only seems to be one really lagged slave... [10:27:25] I already reported it on T116425 [10:27:26] T116425: Rename user creates lag on enwiki - https://phabricator.wikimedia.org/T116425 [10:27:58] Oh but that task is closed as resolved [10:28:06] I mean create another one [10:28:13] so jynus can invastigate. [10:29:00] He's cc'd to that task so I think he will take a look and decide whether it's another issue [10:34:25] If there's a global rename going on, chances of it being that seem very high [10:35:24] If you read the bug, it was causing massive lag on all slaves in a cluster [10:35:29] This just seems to be 1 slave lagging [10:36:32] That one host seems a lot more loaded in the last hour or so [10:38:02] PROBLEM - MediaWiki exceptions and fatals per minute on graphite1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [50.0] [10:39:10] srslu [10:39:30] mmmm [10:39:40] so I should feel sorry for the latest renaming? :P [10:39:55] * Reedy blocks revi [10:39:59] :O [10:41:13] * Bsadowski1 locks revi [10:41:18] Better solution. [10:41:20] :P [10:41:21] * revi cries [10:41:29] I'm kidding :) [10:41:36] I know ofc :D [10:41:38] :P [10:41:43] Hehe [10:41:52] actually this https://meta.wikimedia.org/wiki/Special:GlobalRenameProgress/Supykun was my only rename today [10:42:03] I don't see a load of exceptions spewing [10:42:11] 6 renames in 10 mins.... heh [10:42:18] Oh, now I do [10:42:21] 6 renames on (on local)* [10:42:30] readonly errors [10:43:12] s2 is a little lagged [10:43:33] hmm, db1028 is vslow and dump [10:43:33] some people are saying that run into the db lock error too much [10:43:43] Is everything alright? [10:43:47] PROBLEM - MariaDB Slave Lag: s7 on db1028 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 313.33 seconds [10:43:58] Amir1: some global renames upsetting a few things it seems [10:45:23] so [10:45:30] time to issue kinda no-renaming warning again? [10:45:36] (lol) [10:45:58] well, until things catch up, most certainly [10:46:01] okay, thanks, informed users [10:46:28] I hope you don't mind pasting this to global-renamers@lists.wm.o [10:46:39] (well I'll send the mail after my dinner) [10:47:38] RECOVERY - MariaDB Slave Lag: s7 on db1028 is OK: OK slave_sql_lag Replication lag: 0.70 seconds [10:47:52] PROBLEM - MediaWiki exceptions and fatals per minute on graphite1001 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [50.0] [10:48:01] oh look, it caught up [10:48:23] See if that fixes up the fatal stuff [10:50:02] seems to have stopped spamming the exception logs [10:51:19] nope [10:52:49] morning jynus [10:53:02] hi there [10:53:22] seems a global rename caused db1028 to get excessively lagged [10:53:29] icinga reckons it has recovered now [10:53:43] good ol' icinga [10:54:20] seems to be a fair spam of connection already closed/dropped in the exception logs from MW [10:54:24] which is ok because we set those to not ceck lag [10:54:58] we just now need to create dynamic icinga checks [10:58:16] it is 3 things [10:58:35] the combination of exports; renames and slow queries on the same host [10:59:36] if there wasn't slow queries running on it too.. You could almost make MW ignore the lag [10:59:50] I did [11:00:11] I just cannot tell nagios easily; as it depends on mediawiki code [11:00:17] heh [11:00:49] we need to set puppet-master as a mediawiki host; sync its config then reporgram the nagios script based on mediawiki-config [11:01:21] which some people will object because there is a better option; but not a faster one [11:03:32] Hi nodepool seems to be down in zuul [11:03:38] https://integration.wikimedia.org/zuul/ [11:08:02] (03PS1) 10Merlijn van Deen: toollabs: install opencv-data (trusty, jessie) [puppet] - 10https://gerrit.wikimedia.org/r/303416 (https://phabricator.wikimedia.org/T142321) [11:15:32] RECOVERY - MediaWiki exceptions and fatals per minute on graphite1001 is OK: OK: Less than 1.00% above the threshold [25.0] [11:16:01] hmm, no progress on be-tarask for about 20?30? mins :P [11:16:09] s/on/after [11:22:02] PROBLEM - Router interfaces on cr1-esams is CRITICAL: CRITICAL: host 91.198.174.245, interfaces up: 73, down: 1, dormant: 0, excluded: 0, unused: 0BRxe-1/3/0: down - Transit: FiberRing (AS38930, donated) {#SMF01412} [10Gbps]BR [11:25:12] PROBLEM - MediaWiki exceptions and fatals per minute on graphite1001 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [50.0] [11:25:53] RECOVERY - Router interfaces on cr1-esams is OK: OK: host 91.198.174.245, interfaces up: 75, down: 0, dormant: 0, excluded: 0, unused: 0 [11:47:00] PROBLEM - puppet last run on mw2227 is CRITICAL: CRITICAL: puppet fail [11:59:21] PROBLEM - puppet last run on mw1219 is CRITICAL: CRITICAL: Puppet has 1 failures [12:10:50] !log CI stuck spawning instances via Nodepool apparently due to : Quota exceeded for instances: Requested 1, but already used 10 of 10 instances (HTTP 403) --- Though there is only 8 instances ... [12:10:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [12:12:15] !log [12:12:31] PROBLEM - puppet last run on gallium is CRITICAL: CRITICAL: puppet fail [12:12:56] !log CI stuck spawning instances via Nodepool apparently due to : Quota exceeded for instances: Requested 1, but already used 10 of 10 instances (HTTP 403) --- Though there is only 8 instances ... [12:13:21] RECOVERY - puppet last run on mw2227 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:14:15] hashar it seems log bot is not working [12:14:20] !log [12:24:51] RECOVERY - puppet last run on mw1219 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [12:27:52] RECOVERY - puppet last run on gallium is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures [12:28:33] RECOVERY - MediaWiki exceptions and fatals per minute on graphite1001 is OK: OK: Less than 1.00% above the threshold [25.0] [12:34:32] PROBLEM - MediaWiki exceptions and fatals per minute on graphite1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [50.0] [12:40:22] RECOVERY - MediaWiki exceptions and fatals per minute on graphite1001 is OK: OK: Less than 1.00% above the threshold [25.0] [12:41:01] !log CI fully back. Root cause was Jenkins that could not properly create slaves config due to : Could not create rootDir /var/lib/jenkins/config-history/xxxx . Deleting via find /var/lib/jenkins/config-history/nodes/ -path '*_deleted_*' -delete [12:41:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [12:47:31] !log root cause of CI outage is T126552 [12:47:33] T126552: Jenkins files under /var/lib/jenkins/config-history/config need to be garbage collected - https://phabricator.wikimedia.org/T126552 [12:47:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [12:55:52] PROBLEM - MediaWiki exceptions and fatals per minute on graphite1001 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [50.0] [12:59:44] RECOVERY - MediaWiki exceptions and fatals per minute on graphite1001 is OK: OK: Less than 1.00% above the threshold [25.0] [13:05:51] PROBLEM - MediaWiki exceptions and fatals per minute on graphite1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [50.0] [13:13:32] RECOVERY - MediaWiki exceptions and fatals per minute on graphite1001 is OK: OK: Less than 1.00% above the threshold [25.0] [13:46:51] 06Operations, 06Release-Engineering-Team, 15User-greg, 07Wikimedia-Incident: Institute a weekly review of all UBN! tasks - https://phabricator.wikimedia.org/T141130#2531402 (10Aklapper) >>! In T141130#2515799, @greg wrote: > Should we re-instate a recurring weekly meeting for this? Something public (which... [14:07:02] PROBLEM - MediaWiki exceptions and fatals per minute on graphite1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [50.0] [14:09:01] RECOVERY - MediaWiki exceptions and fatals per minute on graphite1001 is OK: OK: Less than 1.00% above the threshold [25.0] [16:02:33] (03PS1) 10Alexandros Kosiaris: puppetmaster: Fully depool palladium as a puppetmaster [puppet] - 10https://gerrit.wikimedia.org/r/303433 [16:03:12] (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] puppetmaster: Fully depool palladium as a puppetmaster [puppet] - 10https://gerrit.wikimedia.org/r/303433 (owner: 10Alexandros Kosiaris) [16:03:46] (03PS1) 10Andrew Bogott: Nova: update api-paste.ini.erb to conform with Liberty defaults [puppet] - 10https://gerrit.wikimedia.org/r/303434 [16:11:55] PROBLEM - puppet last run on lvs1006 is CRITICAL: CRITICAL: puppet fail [16:12:06] PROBLEM - puppet last run on cp2020 is CRITICAL: CRITICAL: Puppet has 5 failures [16:12:07] PROBLEM - puppet last run on aqs1006 is CRITICAL: CRITICAL: puppet fail [16:12:16] PROBLEM - puppet last run on mw2119 is CRITICAL: CRITICAL: Puppet has 10 failures [16:12:16] PROBLEM - puppet last run on mw2165 is CRITICAL: CRITICAL: puppet fail [16:12:17] PROBLEM - puppet last run on db1019 is CRITICAL: CRITICAL: Puppet has 19 failures [16:12:36] PROBLEM - puppet last run on mw2190 is CRITICAL: CRITICAL: Puppet has 10 failures [16:12:45] PROBLEM - puppet last run on mw2222 is CRITICAL: CRITICAL: Puppet has 13 failures [16:12:46] PROBLEM - puppet last run on mw1162 is CRITICAL: CRITICAL: Puppet has 9 failures [16:12:48] PROBLEM - puppet last run on cp3020 is CRITICAL: CRITICAL: Puppet has 4 failures [16:12:55] PROBLEM - puppet last run on db1048 is CRITICAL: CRITICAL: puppet fail [16:12:55] PROBLEM - puppet last run on analtyics1046 is CRITICAL: CRITICAL: puppet fail [16:12:56] PROBLEM - puppet last run on db2043 is CRITICAL: CRITICAL: puppet fail [16:12:57] PROBLEM - puppet last run on mw1237 is CRITICAL: CRITICAL: Puppet has 11 failures [16:13:06] PROBLEM - puppet last run on mw2178 is CRITICAL: CRITICAL: Puppet has 11 failures [16:13:07] PROBLEM - puppet last run on analytics1046 is CRITICAL: CRITICAL: puppet fail [16:13:15] PROBLEM - puppet last run on elastic1024 is CRITICAL: CRITICAL: puppet fail [16:13:26] PROBLEM - puppet last run on cp3033 is CRITICAL: CRITICAL: puppet fail [16:13:26] PROBLEM - puppet last run on db1036 is CRITICAL: CRITICAL: Puppet has 10 failures [16:13:53] !log restarted apache2 on palladium for full depool to take place [16:13:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [16:38:07] RECOVERY - puppet last run on mw1162 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [16:38:25] RECOVERY - puppet last run on db1048 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [16:38:26] RECOVERY - puppet last run on db2043 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [16:38:26] RECOVERY - puppet last run on mw1237 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:38:36] RECOVERY - puppet last run on mw2178 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [16:38:36] RECOVERY - puppet last run on elastic1024 is OK: OK: Puppet is currently enabled, last run 50 seconds ago with 0 failures [16:38:56] RECOVERY - puppet last run on db1036 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [16:38:56] RECOVERY - puppet last run on cp3033 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [16:39:07] RECOVERY - puppet last run on db2042 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:39:07] RECOVERY - puppet last run on mw1017 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [16:39:16] RECOVERY - puppet last run on lvs1006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:39:26] RECOVERY - puppet last run on mw2192 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [16:39:26] RECOVERY - puppet last run on cp3005 is OK: OK: Puppet is currently enabled, last run 49 seconds ago with 0 failures [16:39:27] RECOVERY - puppet last run on cp2020 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:39:35] RECOVERY - puppet last run on aqs1006 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:39:36] RECOVERY - puppet last run on mw2119 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [16:39:45] RECOVERY - puppet last run on mw2086 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:39:45] RECOVERY - puppet last run on db1019 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:39:46] RECOVERY - puppet last run on mw2165 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [16:40:05] RECOVERY - puppet last run on mw2190 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:40:07] RECOVERY - puppet last run on mw2222 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:40:16] RECOVERY - puppet last run on cp3020 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:40:16] RECOVERY - puppet last run on analtyics1046 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:40:27] RECOVERY - puppet last run on analytics1046 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:41:56] RECOVERY - puppet last run on mw1281 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [16:42:16] RECOVERY - puppet last run on wtp2002 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [16:42:55] RECOVERY - puppet last run on mw1197 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [16:42:56] RECOVERY - puppet last run on elastic1017 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [16:42:56] RECOVERY - puppet last run on analytics1001 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [16:43:05] RECOVERY - puppet last run on mw2227 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [16:43:06] RECOVERY - puppet last run on mw2153 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [16:43:06] RECOVERY - puppet last run on mw1225 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [16:43:06] RECOVERY - puppet last run on mw2065 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:43:07] RECOVERY - puppet last run on mw1263 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [16:43:15] RECOVERY - puppet last run on mw2220 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [16:43:26] RECOVERY - puppet last run on cp1049 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [16:43:26] RECOVERY - puppet last run on mw2164 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [16:43:26] RECOVERY - puppet last run on ms-be2002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:43:27] RECOVERY - puppet last run on db1088 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:43:36] RECOVERY - puppet last run on mw1299 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:43:36] RECOVERY - puppet last run on mw2221 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:43:45] RECOVERY - puppet last run on mw2243 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [16:43:47] RECOVERY - puppet last run on mw2170 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:43:47] RECOVERY - puppet last run on mw2139 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:43:56] RECOVERY - puppet last run on planet1001 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [16:43:57] RECOVERY - puppet last run on mw2183 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:43:57] RECOVERY - puppet last run on mw1266 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:44:05] RECOVERY - puppet last run on mw2161 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [16:44:07] RECOVERY - puppet last run on helium is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [16:44:17] RECOVERY - puppet last run on mw2100 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [16:44:17] RECOVERY - puppet last run on mw2112 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:44:17] RECOVERY - puppet last run on db2039 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:44:25] RECOVERY - puppet last run on mw2064 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [16:44:35] RECOVERY - puppet last run on mw2188 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [16:44:36] RECOVERY - puppet last run on restbase1011 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:44:36] RECOVERY - puppet last run on analytics1033 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:44:37] RECOVERY - puppet last run on analytics1043 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:44:46] RECOVERY - puppet last run on mw2116 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:44:46] RECOVERY - puppet last run on mw2199 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:44:55] RECOVERY - puppet last run on ms-be2003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:44:56] RECOVERY - puppet last run on mw1201 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:44:56] RECOVERY - puppet last run on mw2229 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [16:45:35] RECOVERY - puppet last run on mw2105 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [16:46:06] RECOVERY - puppet last run on mw2075 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:42:10] (03Draft3) 10Paladox: Gerrit: Support labs https [puppet] - 10https://gerrit.wikimedia.org/r/303435 [17:42:14] (03Draft2) 10Paladox: Gerrit: Support labs https [puppet] - 10https://gerrit.wikimedia.org/r/303435 [17:42:17] (03Draft1) 10Paladox: Gerrit: Support labs https [puppet] - 10https://gerrit.wikimedia.org/r/303435 [17:43:30] (03CR) 10jenkins-bot: [V: 04-1] Gerrit: Support labs https [puppet] - 10https://gerrit.wikimedia.org/r/303435 (owner: 10Paladox) [17:44:37] (03PS4) 10Paladox: Gerrit: Support labs https [puppet] - 10https://gerrit.wikimedia.org/r/303435 [18:14:41] (03PS5) 10Paladox: Gerrit: Support labs https [puppet] - 10https://gerrit.wikimedia.org/r/303435 [18:16:56] (03PS6) 10Paladox: Gerrit: Support labs https [puppet] - 10https://gerrit.wikimedia.org/r/303435 (https://phabricator.wikimedia.org/T141803) [18:35:23] (03CR) 10Paladox: "I have managed to support https in labs in patch https://gerrit.wikimedia.org/r/#/c/303435/" [puppet] - 10https://gerrit.wikimedia.org/r/303146 (https://phabricator.wikimedia.org/T141803) (owner: 10Paladox) [18:38:56] PROBLEM - puppet last run on mw2148 is CRITICAL: CRITICAL: Puppet has 1 failures [19:04:35] RECOVERY - puppet last run on mw2148 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [20:28:46] 06Operations, 10Wikimedia-Language-setup, 10Wikimedia-Site-requests, 05MW-1.28-release-notes, and 4 others: Create Wikipedia Tulu - https://phabricator.wikimedia.org/T140898#2480108 (10Amire80) [20:29:48] (03CR) 10QChris: [C: 031] Gerrit: Support footer prefix Task: for phabricator [puppet] - 10https://gerrit.wikimedia.org/r/302482 (https://phabricator.wikimedia.org/T91001) (owner: 10Paladox) [20:36:15] (03CR) 10Legoktm: [C: 04-1] "No, we've standardized on "Bug:" as our footer. Our guidelines and bots use "Bug:", we're not going to add another header and be inconsist" [puppet] - 10https://gerrit.wikimedia.org/r/302482 (https://phabricator.wikimedia.org/T91001) (owner: 10Paladox) [20:37:30] legoktm: https://phabricator.wikimedia.org/T91001 [20:37:45] yes, commenting there too [20:37:49] thanks :-) [20:39:03] legoktm: is it possible to have deplooyed now to our wikis a quick fix for globalblocking without having to wait for the next mediawiki train? [20:39:12] valhallasw`cloud and legoktm hi what about this https://gerrit.wikimedia.org/r/#/c/302229/ [20:39:13] mafk: link? [20:39:18] legoktm: sure [20:39:51] paladox: ? [20:40:17] legoktm -1 the other one [20:40:28] i am wondering about https://gerrit.wikimedia.org/r/#/c/302229/ [20:40:31] that one [20:41:06] legoktm: https://gerrit.wikimedia.org/r/303380 [20:41:09] and [20:41:15] that one's not about Task and Bug as far as I can see? [20:41:18] https://phabricator.wikimedia.org/T142306#2531500 [20:42:22] nope [20:42:53] paladox: so why are you asking me about it? [20:43:19] becuase you were saying about the bots needing updates to support the links [20:43:27] i am wondering if it is the same to that [20:43:48] It seems to be unlikly a bot needs a reconfigure for that. [20:44:48] paladox: so, rephrased, your question is: 'Do other bots need to be (re)configured to allow for comment links such as Bug: T12345#abcd' ? [20:44:48] T12345: Create "annotation" namespace on Hebrew Wikisource - https://phabricator.wikimedia.org/T12345 [20:44:58] yep [20:45:05] uh [20:45:08] let's not allow that either [20:45:12] But it seems to be unlikly since it wont link to phabricator anyways [20:45:29] IE the bots wont be able to publish on phabricator [20:45:51] this is to support plain T1#1 links for example [20:45:52] T1: Get puppet runs into logstash - https://phabricator.wikimedia.org/T1 [20:45:59] bots typically don't depend on gerrit's parsing [20:46:06] oh [20:46:39] but yeah, what legoktm said -- supporting 'Txxx#yyy' inline makes sense, but not so much for 'Bug: ' lines [20:48:27] yep [20:49:30] !log legoktm@tin Synchronized php-1.28.0-wmf.13/extensions/GlobalBlocking/extension.json: Adding globalblock-exempt grant for OAuth - T142306 (duration: 00m 57s) [20:49:32] T142306: Adding globalblock-exempt to Grants in GlobalBlocking extension for OAuth use - https://phabricator.wikimedia.org/T142306 [20:49:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [20:51:34] mafk: ^ [20:51:50] legoktm: thank you so much [20:53:36] legoktm: it's live on mw1099 or everywhere? [20:53:41] everywhere [20:53:43] yay [20:54:11] https://meta.wikimedia.org/wiki/Special:ListGrants <-- listed there as well [20:56:14] (03CR) 10QChris: "From the T91001 at the time I did my CR+1, it looked like there was" [puppet] - 10https://gerrit.wikimedia.org/r/302482 (https://phabricator.wikimedia.org/T91001) (owner: 10Paladox) [20:58:36] PROBLEM - MediaWiki exceptions and fatals per minute on graphite1001 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [50.0] [21:00:53] eh [21:02:45] RECOVERY - MediaWiki exceptions and fatals per minute on graphite1001 is OK: OK: Less than 1.00% above the threshold [25.0] [21:30:01] jouncebot: next [21:30:01] In 17 hour(s) and 29 minute(s): Morning SWAT(Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20160808T1500) [21:30:08] meh [21:30:24] eu swat pls :) [21:30:56] PROBLEM - puppet last run on wtp2017 is CRITICAL: CRITICAL: Puppet has 1 failures [21:56:17] RECOVERY - puppet last run on wtp2017 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:01:47] PROBLEM - MediaWiki exceptions and fatals per minute on graphite1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [50.0] [22:05:46] RECOVERY - MediaWiki exceptions and fatals per minute on graphite1001 is OK: OK: Less than 1.00% above the threshold [25.0] [22:13:04] (03CR) 10BryanDavis: "> Code looks OK to me, but why do we need an autopatrolled group" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/303183 (owner: 10BryanDavis) [22:26:15] (03CR) 10Reedy: [C: 032] Switch TrustedXff to use php rather than cdb [mediawiki-config] - 10https://gerrit.wikimedia.org/r/303390 (https://phabricator.wikimedia.org/T141125) (owner: 10Reedy) [22:26:42] (03Merged) 10jenkins-bot: Switch TrustedXff to use php rather than cdb [mediawiki-config] - 10https://gerrit.wikimedia.org/r/303390 (https://phabricator.wikimedia.org/T141125) (owner: 10Reedy) [22:28:59] !log reedy@tin Synchronized wmf-config/: Swap trusted-xff from cdb to php (duration: 00m 51s) [22:29:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [22:29:59] (03PS2) 10Reedy: Update Image Area to 100MP [mediawiki-config] - 10https://gerrit.wikimedia.org/r/255822 [22:30:06] !log reedy@tin Synchronized docroot: trusted-xff symlink updates (duration: 00m 50s) [22:30:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [22:32:26] (03PS1) 10Reedy: Remove old symlinks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/303496 [22:33:19] (03PS1) 10Reedy: Remove CDB files from noc conf page (none left!) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/303497 [22:33:31] (03CR) 10Reedy: [C: 032] Remove old symlinks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/303496 (owner: 10Reedy) [22:33:57] (03Merged) 10jenkins-bot: Remove old symlinks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/303496 (owner: 10Reedy) [22:34:03] (03CR) 10Reedy: [C: 032] Remove CDB files from noc conf page (none left!) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/303497 (owner: 10Reedy) [22:34:31] (03Merged) 10jenkins-bot: Remove CDB files from noc conf page (none left!) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/303497 (owner: 10Reedy) [22:35:07] PROBLEM - Apache HTTP on mw1267 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:35:35] RECOVERY - Improperly owned -0:0- files in /srv/mediawiki-staging on tin is OK: Files ownership is ok. [22:35:37] !log reedy@tin Synchronized docroot/noc/conf: Cleanup! (duration: 00m 50s) [22:35:42] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [22:36:35] RECOVERY - Improperly owned -0:0- files in /srv/mediawiki-staging on mira is OK: Files ownership is ok. [22:40:17] (03Abandoned) 10Reedy: [Timeline] Update path to extension [mediawiki-config] - 10https://gerrit.wikimedia.org/r/291673 (owner: 10Paladox) [22:42:46] RECOVERY - Apache HTTP on mw1267 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 612 bytes in 0.048 second response time [22:42:53] (03PS4) 10Reedy: extension-list: Move GlobalBlocking and MoodBar to extension.json [mediawiki-config] - 10https://gerrit.wikimedia.org/r/302943 (https://phabricator.wikimedia.org/T139800) [22:42:59] (03CR) 10Reedy: [C: 032] extension-list: Move GlobalBlocking and MoodBar to extension.json [mediawiki-config] - 10https://gerrit.wikimedia.org/r/302943 (https://phabricator.wikimedia.org/T139800) (owner: 10Reedy) [22:43:28] (03Merged) 10jenkins-bot: extension-list: Move GlobalBlocking and MoodBar to extension.json [mediawiki-config] - 10https://gerrit.wikimedia.org/r/302943 (https://phabricator.wikimedia.org/T139800) (owner: 10Reedy) [22:44:45] !log reedy@tin Synchronized wmf-config/extension-list: 2 more to extension.json (duration: 00m 48s) [22:44:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [22:48:18] (03CR) 10Reedy: [C: 032] Update Image Area to 100MP [mediawiki-config] - 10https://gerrit.wikimedia.org/r/255822 (owner: 10Reedy) [22:52:48] (03PS3) 10Reedy: Update Image Area to 100MP [mediawiki-config] - 10https://gerrit.wikimedia.org/r/255822 [22:52:56] (03CR) 10Reedy: Update Image Area to 100MP [mediawiki-config] - 10https://gerrit.wikimedia.org/r/255822 (owner: 10Reedy) [22:52:58] (03CR) 10Reedy: [C: 032] Update Image Area to 100MP [mediawiki-config] - 10https://gerrit.wikimedia.org/r/255822 (owner: 10Reedy) [22:53:25] (03Merged) 10jenkins-bot: Update Image Area to 100MP [mediawiki-config] - 10https://gerrit.wikimedia.org/r/255822 (owner: 10Reedy) [22:54:23] (03PS6) 10Reedy: Load RestBaseUpdateJobs via wfLoadExtension() [mediawiki-config] - 10https://gerrit.wikimedia.org/r/298095 (https://phabricator.wikimedia.org/T139800) [22:55:01] !log reedy@tin Synchronized wmf-config/CommonSettings.php: Image Area to 100MP (duration: 00m 48s) [22:55:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [22:55:21] (03CR) 10Reedy: [C: 032] Load RestBaseUpdateJobs via wfLoadExtension() [mediawiki-config] - 10https://gerrit.wikimedia.org/r/298095 (https://phabricator.wikimedia.org/T139800) (owner: 10Reedy) [22:55:48] (03Merged) 10jenkins-bot: Load RestBaseUpdateJobs via wfLoadExtension() [mediawiki-config] - 10https://gerrit.wikimedia.org/r/298095 (https://phabricator.wikimedia.org/T139800) (owner: 10Reedy) [22:56:41] This is the creepiest channel to just sit and watch. Very skynetish. The machines are going to take over :/ [22:57:05] !log reedy@tin Synchronized wmf-config: RestbaseUpdateJobs to wfLoadExtension (duration: 00m 51s) [22:57:09] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [22:59:15] Cameron11598, a lot of it is just bots relaying user activity in other places [23:00:10] granted logmsgbot uses the !log command on morebots :) [23:00:31] (logmsgbot runs in production and morebots runs in labs) [23:01:58] (03PS2) 10Dereckson: Use extension registration for FundraisingTranslateWorkflow [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281243 (https://phabricator.wikimedia.org/T119117) [23:02:00] (03PS3) 10Reedy: Use extension registration for FundraisingTranslateWorkflow [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281243 (https://phabricator.wikimedia.org/T119117) (owner: 10Dereckson) [23:02:35] and of course there is shinken-wm monitoring the status of hundreds of other machines :) [23:03:15] (03PS2) 10Dereckson: Use extension registration for SpamBlacklist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/292312 (https://phabricator.wikimedia.org/T119117) [23:03:19] (03PS3) 10Reedy: Use extension registration for SpamBlacklist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/292312 (https://phabricator.wikimedia.org/T119117) (owner: 10Dereckson) [23:04:42] (03PS2) 10Dereckson: Use extension registration for ProofreadPage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281976 (https://phabricator.wikimedia.org/T119117) [23:04:45] (03PS3) 10Reedy: Use extension registration for ProofreadPage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281976 (https://phabricator.wikimedia.org/T119117) (owner: 10Dereckson) [23:05:14] (03CR) 10Reedy: [C: 032] Use extension registration for FundraisingTranslateWorkflow [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281243 (https://phabricator.wikimedia.org/T119117) (owner: 10Dereckson) [23:05:42] (03Merged) 10jenkins-bot: Use extension registration for FundraisingTranslateWorkflow [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281243 (https://phabricator.wikimedia.org/T119117) (owner: 10Dereckson) [23:05:54] (03CR) 10Reedy: [C: 032] Use extension registration for SpamBlacklist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/292312 (https://phabricator.wikimedia.org/T119117) (owner: 10Dereckson) [23:06:40] (03PS4) 10Reedy: Use extension registration for SpamBlacklist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/292312 (https://phabricator.wikimedia.org/T119117) (owner: 10Dereckson) [23:06:42] (03CR) 10Reedy: [V: 032] Use extension registration for SpamBlacklist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/292312 (https://phabricator.wikimedia.org/T119117) (owner: 10Dereckson) [23:06:46] (03CR) 10Reedy: Use extension registration for SpamBlacklist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/292312 (https://phabricator.wikimedia.org/T119117) (owner: 10Dereckson) [23:06:48] (03CR) 10Reedy: [C: 032] Use extension registration for SpamBlacklist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/292312 (https://phabricator.wikimedia.org/T119117) (owner: 10Dereckson) [23:07:14] (03Merged) 10jenkins-bot: Use extension registration for SpamBlacklist [mediawiki-config] - 10https://gerrit.wikimedia.org/r/292312 (https://phabricator.wikimedia.org/T119117) (owner: 10Dereckson) [23:07:38] (03PS2) 10Dereckson: Use extension registration for ExtensionDistributor [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281303 (https://phabricator.wikimedia.org/T119117) [23:07:49] (03PS3) 10Reedy: Use extension registration for ExtensionDistributor [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281303 (https://phabricator.wikimedia.org/T119117) (owner: 10Dereckson) [23:07:52] (03CR) 10Reedy: [C: 032] Use extension registration for ExtensionDistributor [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281303 (https://phabricator.wikimedia.org/T119117) (owner: 10Dereckson) [23:08:20] (03Merged) 10jenkins-bot: Use extension registration for ExtensionDistributor [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281303 (https://phabricator.wikimedia.org/T119117) (owner: 10Dereckson) [23:08:22] (03PS2) 10Dereckson: Use extension registration for MwEmbedSupport [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281246 (https://phabricator.wikimedia.org/T119117) [23:08:46] (03PS2) 10Dereckson: Use extension registration for Quiz [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281242 (https://phabricator.wikimedia.org/T119117) [23:08:53] (03PS3) 10Reedy: Use extension registration for MwEmbedSupport [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281246 (https://phabricator.wikimedia.org/T119117) (owner: 10Dereckson) [23:08:55] (03CR) 10Reedy: [C: 032] Use extension registration for MwEmbedSupport [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281246 (https://phabricator.wikimedia.org/T119117) (owner: 10Dereckson) [23:09:22] (03Merged) 10jenkins-bot: Use extension registration for MwEmbedSupport [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281246 (https://phabricator.wikimedia.org/T119117) (owner: 10Dereckson) [23:09:37] PROBLEM - puppet last run on ms-fe2001 is CRITICAL: CRITICAL: Puppet has 1 failures [23:09:40] (03PS3) 10Reedy: Use extension registration for Quiz [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281242 (https://phabricator.wikimedia.org/T119117) (owner: 10Dereckson) [23:09:43] (03CR) 10Reedy: [C: 032] Use extension registration for Quiz [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281242 (https://phabricator.wikimedia.org/T119117) (owner: 10Dereckson) [23:09:50] Cameron11598, much more activity than usual though :) [23:10:11] (03Merged) 10jenkins-bot: Use extension registration for Quiz [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281242 (https://phabricator.wikimedia.org/T119117) (owner: 10Dereckson) [23:10:16] (03PS4) 10Reedy: Use extension registration for ProofreadPage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281976 (https://phabricator.wikimedia.org/T119117) (owner: 10Dereckson) [23:10:18] (03CR) 10Reedy: [C: 032] Use extension registration for ProofreadPage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281976 (https://phabricator.wikimedia.org/T119117) (owner: 10Dereckson) [23:10:46] (03Merged) 10jenkins-bot: Use extension registration for ProofreadPage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281976 (https://phabricator.wikimedia.org/T119117) (owner: 10Dereckson) [23:12:10] !log reedy@tin Synchronized wmf-config: Handful more extensions to wfLoadExtension (duration: 00m 49s) [23:12:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:14:33] (03PS2) 10Dereckson: Use extension registration for GlobalBlocking [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281311 (https://phabricator.wikimedia.org/T119117) [23:14:42] (03CR) 10jenkins-bot: [V: 04-1] Use extension registration for GlobalBlocking [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281311 (https://phabricator.wikimedia.org/T119117) (owner: 10Dereckson) [23:14:50] (03PS2) 10Dereckson: Use extension registration for Gadgets [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281244 (https://phabricator.wikimedia.org/T119117) [23:15:05] (03CR) 10jenkins-bot: [V: 04-1] Use extension registration for Gadgets [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281244 (https://phabricator.wikimedia.org/T119117) (owner: 10Dereckson) [23:17:21] mutante or robh around? [23:18:38] (03PS3) 10Reedy: Use extension registration for GlobalBlocking [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281311 (https://phabricator.wikimedia.org/T119117) (owner: 10Dereckson) [23:18:47] (03CR) 10jenkins-bot: [V: 04-1] Use extension registration for GlobalBlocking [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281311 (https://phabricator.wikimedia.org/T119117) (owner: 10Dereckson) [23:21:48] (03PS3) 10Reedy: Use extension registration for Gadgets [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281244 (https://phabricator.wikimedia.org/T119117) (owner: 10Dereckson) [23:22:04] damn it [23:22:18] (03CR) 10jenkins-bot: [V: 04-1] Use extension registration for Gadgets [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281244 (https://phabricator.wikimedia.org/T119117) (owner: 10Dereckson) [23:22:20] Hi. Let me rebase GlobalBlocking. [23:22:33] (03PS4) 10Reedy: Use extension registration for Gadgets [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281244 (https://phabricator.wikimedia.org/T119117) (owner: 10Dereckson) [23:22:58] DAMN ITY [23:23:01] (03PS4) 10Dereckson: Use extension registration for GlobalBlocking [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281311 (https://phabricator.wikimedia.org/T119117) [23:23:04] (03CR) 10jenkins-bot: [V: 04-1] Use extension registration for Gadgets [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281244 (https://phabricator.wikimedia.org/T119117) (owner: 10Dereckson) [23:23:10] (03CR) 10jenkins-bot: [V: 04-1] Use extension registration for GlobalBlocking [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281311 (https://phabricator.wikimedia.org/T119117) (owner: 10Dereckson) [23:23:20] arf [23:23:25] (03PS5) 10Reedy: Use extension registration for Gadgets [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281244 (https://phabricator.wikimedia.org/T119117) (owner: 10Dereckson) [23:23:52] okay so for globalblocking [23:23:58] it has a dependency against https://gerrit.wikimedia.org/r/#/c/281309/ [23:24:19] oh [23:24:32] makes sense [23:24:45] (03CR) 10Reedy: [C: 032] Use extension registration for Gadgets [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281244 (https://phabricator.wikimedia.org/T119117) (owner: 10Dereckson) [23:25:00] (03PS5) 10Dereckson: Use extension registration for GlobalBlocking [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281311 (https://phabricator.wikimedia.org/T119117) [23:25:11] (03Merged) 10jenkins-bot: Use extension registration for Gadgets [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281244 (https://phabricator.wikimedia.org/T119117) (owner: 10Dereckson) [23:25:48] (03PS6) 10Reedy: Use extension registration for GlobalBlocking [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281311 (https://phabricator.wikimedia.org/T119117) (owner: 10Dereckson) [23:25:50] (03CR) 10Reedy: [C: 032] Use extension registration for GlobalBlocking [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281311 (https://phabricator.wikimedia.org/T119117) (owner: 10Dereckson) [23:26:17] Dereckson: we've backported a patch on extension.json for globalblocking, please don't remove that ;) [23:37:04] (03CR) 10Reedy: [C: 032] Use extension registration for GlobalBlocking [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281311 (https://phabricator.wikimedia.org/T119117) (owner: 10Dereckson) [23:37:16] RECOVERY - puppet last run on ms-fe2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:37:30] (03Merged) 10jenkins-bot: Use extension registration for GlobalBlocking [mediawiki-config] - 10https://gerrit.wikimedia.org/r/281311 (https://phabricator.wikimedia.org/T119117) (owner: 10Dereckson) [23:38:05] "The database has been locked for maintenance, so you will not be able to save your edits right now. You may wish to copy-and-paste the text into a text file and save it for later." @ commons [23:38:24] More global renames? :P [23:39:02] db1056 lag 15 [23:39:03] !log reedy@tin Synchronized wmf-config: last 2 to wfLoadExtension (duration: 00m 59s) [23:39:07] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:39:36] Reedy: not that I'm aware of [23:39:39] 'db1056' => 0, # 2.8TB 96GB, watchlist, recentchanges, contributions, logpager [23:40:24] blame the users [23:42:37] * mafk blocks them all for fun :P [23:43:56] thing running when I looked was Title::invalidateCache was updating page_touched of File:P�re-Lachaise_-_Division_7_-_Hayem_07.jpg [23:44:02] seems to be okay now though. mafk? [23:44:21] Krenair: yep, I could save my edit already, thanks :) [23:44:41] * Krenair did nothing