[00:01:23] I think there was a task about adding a ops log for that script, but I can't find it atm [00:03:12] 06Operations, 10MediaWiki-Cache, 10Traffic: Cached outdated revisions served to logged-out users - https://phabricator.wikimedia.org/T141687#2508995 (10Gestrid) Also, I was talking to someone in IRC (specifically the wikipedia-en channel), and they seemed to remember something similar from a couple of years... [00:08:41] 06Operations, 10MediaWiki-Cache, 10Traffic: Cached outdated revisions served to logged-out users - https://phabricator.wikimedia.org/T141687#2507607 (10Perhelion) I'm not sure it is related but on Commons are also some cache problems (since 2 days) reported (unrelated logged in or not). [00:16:36] 06Operations, 10MediaWiki-Cache, 10Traffic: Cached outdated revisions served to logged-out users - https://phabricator.wikimedia.org/T141687#2509033 (10Gestrid) >>! In T141687#2508997, @Perhelion wrote: > I'm not sure it is related but on Commons are also some cache problems (since 2 days) reported (unrelate... [00:25:30] I have this too ^ [00:27:00] 06Operations, 10MediaWiki-Cache, 10Traffic: Cached outdated revisions served to logged-out users - https://phabricator.wikimedia.org/T141687#2507607 (10Matanya) I faced this too, logged in or not, after editing https://he.wikipedia.org/wiki/%D7%A4%D7%95%D7%A8%D7%98%D7%9C:%D7%A2%D7%A8%D7%9B%D7%99%D7%9D_%D7%9E... [00:33:26] 06Operations, 10MediaWiki-Cache, 10Traffic: Cached outdated revisions served to logged-out users - https://phabricator.wikimedia.org/T141687#2507607 (10AlexMonk-WMF) I just briefly looked through the 1.28.0-wmf.12 change log, only thing obviously from this area of core functionality was https://gerrit.wikime... [01:52:46] equest from 90.180.83.194 via cp3008 cp3008, Varnish XID 770378229 [01:52:48] Error: 503, Backend fetch failed at Sun, 31 Jul 2016 01:52:27 GMT [02:22:19] !log mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.12) (duration: 08m 07s) [02:22:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:27:53] !log l10nupdate@tin ResourceLoader cache refresh completed at Sun Jul 31 02:27:53 UTC 2016 (duration 5m 34s) [02:27:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [03:25:00] 06Operations, 10Traffic: Support TLS chacha20-poly1305 AEAD ciphers - https://phabricator.wikimedia.org/T131908#2509064 (10BBlack) I've built (but not yet tested) the wmf3 commit above with the cloudflare patch. It required some follow-on patching to even make it compile. The two issues were due to debian's... [03:35:28] PROBLEM - puppet last run on analytics1039 is CRITICAL: CRITICAL: Puppet has 1 failures [04:03:00] RECOVERY - puppet last run on analytics1039 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:52:59] 06Operations, 10media-storage: Storage backend errors on commons when deleting/restoring pages - https://phabricator.wikimedia.org/T141704#2509100 (10Peachey88) [06:31:39] PROBLEM - puppet last run on nobelium is CRITICAL: CRITICAL: Puppet has 2 failures [06:31:57] PROBLEM - puppet last run on elastic2007 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:57] PROBLEM - puppet last run on analytics1047 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:58] PROBLEM - puppet last run on mw2129 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:58] PROBLEM - puppet last run on mw2073 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:28] PROBLEM - puppet last run on db1056 is CRITICAL: CRITICAL: Puppet has 2 failures [06:32:47] PROBLEM - puppet last run on mw2145 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:57] PROBLEM - puppet last run on ms-be2022 is CRITICAL: CRITICAL: Puppet has 1 failures [06:33:19] PROBLEM - puppet last run on mw2207 is CRITICAL: CRITICAL: Puppet has 1 failures [06:33:38] PROBLEM - puppet last run on mw2126 is CRITICAL: CRITICAL: Puppet has 1 failures [06:33:39] PROBLEM - puppet last run on planet2001 is CRITICAL: CRITICAL: Puppet has 1 failures [06:35:09] PROBLEM - puppet last run on ms-be2011 is CRITICAL: CRITICAL: Puppet has 1 failures [06:35:18] PROBLEM - puppet last run on mw2166 is CRITICAL: CRITICAL: Puppet has 1 failures [06:49:18] PROBLEM - tools homepage -admin tool- on tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 531 bytes in 0.046 second response time [06:52:31] 06Operations, 10media-storage: Storage backend errors on commons when deleting/restoring pages - https://phabricator.wikimedia.org/T141704#2509075 (10Nemo_bis) p:05Triage>03High a:03aaron [06:53:09] RECOVERY - tools homepage -admin tool- on tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 3670 bytes in 0.057 second response time [06:55:47] RECOVERY - puppet last run on db1056 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [06:56:08] RECOVERY - puppet last run on mw2145 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [06:56:58] RECOVERY - puppet last run on mw2126 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [06:56:59] RECOVERY - puppet last run on nobelium is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [06:57:17] RECOVERY - puppet last run on analytics1047 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:57:18] RECOVERY - puppet last run on elastic2007 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [06:57:19] RECOVERY - puppet last run on mw2129 is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures [06:57:19] RECOVERY - puppet last run on mw2073 is OK: OK: Puppet is currently enabled, last run 20 seconds ago with 0 failures [06:57:59] PROBLEM - puppet last run on elastic2020 is CRITICAL: CRITICAL: Puppet has 1 failures [06:58:08] RECOVERY - puppet last run on ms-be2022 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:58:38] RECOVERY - puppet last run on ms-be2011 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:58:39] RECOVERY - puppet last run on mw2166 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [06:58:39] RECOVERY - puppet last run on mw2207 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:58:59] RECOVERY - puppet last run on planet2001 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [07:23:38] RECOVERY - puppet last run on elastic2020 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:17:53] Help? Now that I am in the right place? [08:21:25] 06Operations, 10media-storage: Storage backend errors on commons when deleting/restoring pages - https://phabricator.wikimedia.org/T141704#2509075 (10Revent) "Error undeleting file: The file "mwstore://local-multiwrite/local-public/b/b8/Kit_body_FCF2016a.png" is in an inconsistent state within the internal sto... [08:21:44] (03PS1) 10Ema: cache_maps: remove varnish3 VCL compat [puppet] - 10https://gerrit.wikimedia.org/r/302074 (https://phabricator.wikimedia.org/T122880) [08:23:58] 06Operations, 10media-storage: Storage backend errors on commons when deleting/restoring pages - https://phabricator.wikimedia.org/T141704#2509181 (10Nemo_bis) [08:24:57] 06Operations, 10media-storage, 07Documentation: Document how to handle 'inconsistent state within the internal storage backends' issues - https://phabricator.wikimedia.org/T135318#2509183 (10Peachey88) >>! In T135318#2296824, @Joe wrote: > I don't think this has anything to do with operations, but if I'm mis... [09:02:08] (03CR) 10MarcoAurelio: [C: 04-1] allow sysops on hewikt to remove autopatroller and patroller rights (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/302054 (owner: 10Matanya) [09:10:36] (03PS2) 10Matanya: Allow sysops on he.wiktionary to remove autopatroller and patroller user rights. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/302054 (https://phabricator.wikimedia.org/T140563) [09:19:08] (03CR) 10MarcoAurelio: MoodBar: Disable on all wikis except nlwiki (033 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/301893 (https://phabricator.wikimedia.org/T131340) (owner: 10Jforrester) [09:35:37] 06Operations, 10MediaWiki-Cache, 10Traffic: Cached outdated revisions served to logged-out users - https://phabricator.wikimedia.org/T141687#2507607 (10Tbayer) Probably related: I have twice (now and one or two days ago) seen version histories that lacked the most recent revision(s) when viewed as logged-out... [09:48:41] (03CR) 10Nemo bis: Initial configuration for tcy.wikipedia (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/300182 (https://phabricator.wikimedia.org/T140898) (owner: 10Paladox) [10:12:50] 06Operations, 10MediaWiki-Cache, 10Traffic: Cached outdated revisions served to logged-out users - https://phabricator.wikimedia.org/T141687#2509250 (10Glaisher) p:05High>03Unbreak! This issue seems more widespread than initially thought and apparently it's not just enwiki so raising priority to UBN. A u... [10:27:32] ori: you here? [10:29:52] Unlikely, it'll be... 3.30 am for him? [10:32:35] Reedy: oh, i thought he is located in UTC +2 [10:34:16] Greet your best friend, /wii [10:35:59] 06Operations, 10MediaWiki-Cache, 10Traffic: Cached outdated revisions served to logged-out users - https://phabricator.wikimedia.org/T141687#2507607 (10Nemo_bis) I noticed this two days ago while visiting a user talk page on the Italian Wikipedia, where a message posted over 10 minutes before was not visible. [10:36:55] 06Operations, 10MediaWiki-Cache, 10Traffic, 07Performance: Cached outdated revisions served to logged-out users - https://phabricator.wikimedia.org/T141687#2509266 (10Nemo_bis) [10:47:58] (03CR) 10Dereckson: "Logic looks good to me." (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/302054 (https://phabricator.wikimedia.org/T140563) (owner: 10Matanya) [10:50:27] (03CR) 10MarcoAurelio: [C: 04-1] Initial configuration for tcy.wikipedia (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/300182 (https://phabricator.wikimedia.org/T140898) (owner: 10Paladox) [10:57:03] (03CR) 10MarcoAurelio: Allow sysops on he.wiktionary to remove autopatroller and patroller user rights. (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/302054 (https://phabricator.wikimedia.org/T140563) (owner: 10Matanya) [10:58:56] (03PS3) 10Matanya: Allow sysops on he.wiktionary to remove autopatroller and patroller user rights. [mediawiki-config] - 10https://gerrit.wikimedia.org/r/302054 (https://phabricator.wikimedia.org/T140563) [11:02:01] (03PS11) 10Paladox: Initial configuration for tcy.wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/300182 (https://phabricator.wikimedia.org/T140898) [11:02:24] (03CR) 10Paladox: Initial configuration for tcy.wikipedia (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/300182 (https://phabricator.wikimedia.org/T140898) (owner: 10Paladox) [11:34:58] (03CR) 10MarcoAurelio: "> Uploaded patch set 11." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/300182 (https://phabricator.wikimedia.org/T140898) (owner: 10Paladox) [11:36:00] (03CR) 10Paladox: "Oh yes I rebased." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/300182 (https://phabricator.wikimedia.org/T140898) (owner: 10Paladox) [11:37:46] (03CR) 10Paladox: "According to https://phabricator.wikimedia.org/rOMWC1d5bd037082b1453365c6e180e37f87ee3f25af4#57537e71 I didn't touch or remove any other c" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/300182 (https://phabricator.wikimedia.org/T140898) (owner: 10Paladox) [11:40:19] (03CR) 10Paladox: "Also looking at the file, there is mswiki in there." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/300182 (https://phabricator.wikimedia.org/T140898) (owner: 10Paladox) [11:40:59] (03CR) 10MarcoAurelio: "> Also looking at the file, there is mswiki in there." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/300182 (https://phabricator.wikimedia.org/T140898) (owner: 10Paladox) [11:41:44] (03CR) 10Paladox: "Oh, ok." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/300182 (https://phabricator.wikimedia.org/T140898) (owner: 10Paladox) [11:42:53] (03CR) 10MarcoAurelio: "> Oh, ok." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/300182 (https://phabricator.wikimedia.org/T140898) (owner: 10Paladox) [11:43:52] (03CR) 10Paladox: "That's because patchset 10 is based on a prevous version of master and patchset 11 is based on the current master." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/300182 (https://phabricator.wikimedia.org/T140898) (owner: 10Paladox) [11:51:28] paladox: that explains, sorry for bothering :( [11:51:43] Oh ok [11:51:47] yeh it is a new feature [11:51:49] in gerrit [11:51:58] and not everyone will be used to it yet. [11:52:07] I'm still getting used, sorry. [11:52:25] Ok [11:52:33] You can edit in the browser now [11:52:45] Meaning you can edit files in the browser [11:53:54] mafk ^^ [11:53:56] :) [11:55:27] oh yeah, I've already used that paladox and I like it [11:55:38] Oh [11:55:38] instead of git review -d xxxx etc. [11:55:39] :) [11:55:43] yeh much easy [12:37:07] 06Operations, 10Phabricator, 10Traffic: Phabricator needs to expose notification daemon (websocket) - https://phabricator.wikimedia.org/T112765#2509448 (10Danny_B) What is the status, please? Having {T765} (this task is its blocker) finally resolved would help a lot to prevent increasing number of actions b... [12:42:54] (03CR) 10MarcoAurelio: [C: 031] Initial configuration for tcy.wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/300182 (https://phabricator.wikimedia.org/T140898) (owner: 10Paladox) [12:43:22] (03PS1) 10Addshore: Debug logging for T138987 [mediawiki-config] - 10https://gerrit.wikimedia.org/r/302085 (https://phabricator.wikimedia.org/T138987) [12:43:48] PROBLEM - MediaWiki exceptions and fatals per minute on graphite1001 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [50.0] [12:45:48] RECOVERY - MediaWiki exceptions and fatals per minute on graphite1001 is OK: OK: Less than 1.00% above the threshold [25.0] [13:10:25] 06Operations, 10MediaWiki-Cache, 10Traffic, 07Performance: Cached outdated revisions served to logged-out users - https://phabricator.wikimedia.org/T141687#2509511 (10BBlack) >>! In T141687#2509250, @Glaisher wrote: > This issue seems more widespread than initially thought and apparently it's not just enwi... [13:19:01] 06Operations, 10Phabricator, 10Traffic: Phabricator needs to expose notification daemon (websocket) - https://phabricator.wikimedia.org/T112765#2509512 (10BBlack) There's a little bit of refactoring work (already in-progress) to do on the Varnish side to support it "correctly", but even if that weren't ready... [13:51:40] 06Operations, 10MediaWiki-Cache, 10Traffic, 07Performance: Cached outdated revisions served to logged-out users - https://phabricator.wikimedia.org/T141687#2509527 (10BBlack) I've dug through live PURGE logs to see if I can notice any odd patterns or malformed requests, but so far everything looks relative... [13:54:09] 06Operations, 10MediaWiki-Cache, 10Traffic, 07Performance: Cached outdated revisions served to logged-out users - https://phabricator.wikimedia.org/T141687#2509530 (10BBlack) Also, can someone better explain what's going on in https://gerrit.wikimedia.org/r/#/c/299859/1/includes/page/WikiPage.php (same com... [13:58:05] 06Operations, 10Phabricator, 10Traffic: Phabricator needs to expose notification daemon (websocket) - https://phabricator.wikimedia.org/T112765#2509532 (10Danny_B) Thanks for the update @BBlack. To be honest, not knowing all the context, I am a bit confused, so I would appreciate any ETA (in a week, in a mon... [14:22:24] 06Operations, 10Phabricator, 10Traffic: Phabricator needs to expose notification daemon (websocket) - https://phabricator.wikimedia.org/T112765#2509603 (10BBlack) I wouldn't be the one working on turning up the service on iridium, and I'm not sure who would, so I can't really answer that. [15:04:28] PROBLEM - Juniper alarms on asw-d-eqiad.mgmt.eqiad.wmnet is CRITICAL: JNX_ALARMS CRITICAL - Received genError(5) error-status at error-index 1 [15:06:27] RECOVERY - Juniper alarms on asw-d-eqiad.mgmt.eqiad.wmnet is OK: JNX_ALARMS OK - 0 red alarms, 0 yellow alarms [15:26:51] 06Operations, 10MediaWiki-Cache, 10Traffic, 07Performance: Cached outdated revisions served to logged-out users - https://phabricator.wikimedia.org/T141687#2509686 (10Nemo_bis) >>! In T141687#2509530, @BBlack wrote: > varnish purge might pick up not-yet-invalidated parsercache output when it refreshes from... [16:06:53] 06Operations, 10MediaWiki-Cache, 10Traffic, 07Performance: Cached outdated revisions served to logged-out users - https://phabricator.wikimedia.org/T141687#2509697 (10AlexMonk-WMF) >>! In T141687#2509530, @BBlack wrote: > Also, can someone better explain what's going on in https://gerrit.wikimedia.org/r/#/... [17:03:22] (03PS1) 10Addshore: Remove T107711 debug log [mediawiki-config] - 10https://gerrit.wikimedia.org/r/302113 (https://phabricator.wikimedia.org/T107711) [17:08:00] (03PS1) 10Addshore: Remove T124356 debug logging [mediawiki-config] - 10https://gerrit.wikimedia.org/r/302114 (https://phabricator.wikimedia.org/T124356) [17:08:21] 06Operations, 10MediaWiki-Cache, 10Traffic, 07Performance: Cached outdated revisions served to logged-out users - https://phabricator.wikimedia.org/T141687#2509737 (10Glaisher) >>! In T141687#2509511, @BBlack wrote: > I think this is a dubious argument if it's the basis of raising it to UBN. It's just as... [17:09:14] (03PS2) 10Addshore: Remove T107711 debug logging [mediawiki-config] - 10https://gerrit.wikimedia.org/r/302113 (https://phabricator.wikimedia.org/T107711) [17:21:24] 06Operations, 10MediaWiki-Cache, 10Traffic, 07Performance: Cached outdated revisions served to logged-out users - https://phabricator.wikimedia.org/T141687#2509761 (10Glaisher) I think https://gerrit.wikimedia.org/r/#/c/300338/3/includes/page/WikiPage.php is what happened here. If that is true, purges done... [17:40:40] (03PS1) 10Addshore: WIP Add simple-json-datasource plugin to labs grafana [puppet] - 10https://gerrit.wikimedia.org/r/302119 (https://phabricator.wikimedia.org/T141636) [17:43:29] PROBLEM - puppet last run on mw2192 is CRITICAL: CRITICAL: Puppet has 1 failures [17:46:15] (03PS2) 10Addshore: WIP Add simple-json-datasource plugin to labs grafana [puppet] - 10https://gerrit.wikimedia.org/r/302119 (https://phabricator.wikimedia.org/T141636) [17:49:43] !log aaron@tin Synchronized php-1.28.0-wmf.12/includes/page/WikiPage.php: 23330222a2af3fb (duration: 00m 34s) [17:49:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [17:52:30] (03CR) 10Zppix: [C: 031] "Looks like a painless addition to me, no errors on my own either when attempting to test" [puppet] - 10https://gerrit.wikimedia.org/r/302119 (https://phabricator.wikimedia.org/T141636) (owner: 10Addshore) [17:57:02] Glaisher: https://gerrit.wikimedia.org/r/#/c/302122/ too [17:59:44] oh [18:02:35] Glaisher: you don't have +2? [18:02:58] yeah, I asked Krenair to +2 it. [18:03:32] * AaronSchulz should fix that [18:04:32] (03CR) 10Paladox: Avoid breaking full phabricator URLs (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/242237 (owner: 10Daniel Kinzler) [18:04:53] (03CR) 10Paladox: [C: 04-1] "Breaks using just plain T1 in comment box." [puppet] - 10https://gerrit.wikimedia.org/r/242237 (owner: 10Daniel Kinzler) [18:07:32] !log aaron@tin Synchronized php-1.28.0-wmf.12/includes/page/WikiPage.php: 061737668d729ba1a76 (duration: 00m 24s) [18:07:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [18:09:18] RECOVERY - puppet last run on mw2192 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:10:35] AaronSchulz, you think we should give Glaisher +2 across mediawiki? [18:10:39] {{support}} --~~~~ [18:11:45] 06Operations, 10MediaWiki-Cache, 10Traffic, 05MW-1.28-release-notes, and 3 others: Cached outdated revisions served to logged-out users - https://phabricator.wikimedia.org/T141687#2507607 (10aaron) I suspected missing PRESEND looking at this before, bug @Glaiser beat me to it. [18:12:15] (03PS1) 10Paladox: Gerrit: Avoid breaking full phabricator URLs [puppet] - 10https://gerrit.wikimedia.org/r/302129 (https://phabricator.wikimedia.org/T75997) [18:12:29] Krenair: +1 on that [18:13:03] AaronSchulz, Krenair: {{support}} as well. --~~~~ [18:14:42] (03CR) 10Paladox: "I have done all the testing." [puppet] - 10https://gerrit.wikimedia.org/r/302129 (https://phabricator.wikimedia.org/T75997) (owner: 10Paladox) [18:15:03] AaronSchulz: would you please do review at https://gerrit.wikimedia.org/r/#/c/300816/ ? - thank you :) [18:15:31] (03CR) 10Zppix: [C: 031] Gerrit: Avoid breaking full phabricator URLs [puppet] - 10https://gerrit.wikimedia.org/r/302129 (https://phabricator.wikimedia.org/T75997) (owner: 10Paladox) [18:16:02] mafk, okay but, you're not an existing mediawiki +2er :/ [18:16:23] Krenair: so {{support|Moral support}} :) [18:16:32] okay :) [18:16:37] I finally found a fix for broken phab links in gerrit [18:16:38] https://gerrit.wikimedia.org/r/#/c/302129/ [18:16:40] yay [18:20:12] Dosent break any other links either [18:20:31] The regex is from somebody else who i said who did it in the comment [18:20:49] but i did testing and finally we can fix the links to work properly [18:20:56] when we use https://phabricator.wikimedia.org/T76459 in the comment box [18:21:19] (03PS1) 10Addshore: Add dbslave to statistics::wmde config [puppet] - 10https://gerrit.wikimedia.org/r/302130 [18:23:18] mafk, Krenair : I put up https://phabricator.wikimedia.org/T141730 [18:23:28] (03CR) 10Luke081515: [C: 031] Gerrit: Avoid breaking full phabricator URLs [puppet] - 10https://gerrit.wikimedia.org/r/302129 (https://phabricator.wikimedia.org/T75997) (owner: 10Paladox) [18:23:32] (03CR) 10Paladox: "Note that where I says this is what it will look like, it is broken since the patch has not been applied on gerrit.wikimedia.org yet." [puppet] - 10https://gerrit.wikimedia.org/r/302129 (https://phabricator.wikimedia.org/T75997) (owner: 10Paladox) [18:23:34] oh right we do this on phab instead of on-wiki now, ok [18:23:53] (03CR) 10Paladox: "Thanks." [puppet] - 10https://gerrit.wikimedia.org/r/302129 (https://phabricator.wikimedia.org/T75997) (owner: 10Paladox) [18:29:09] (03CR) 10Platonides: [C: 031] Gerrit: Avoid breaking full phabricator URLs [puppet] - 10https://gerrit.wikimedia.org/r/302129 (https://phabricator.wikimedia.org/T75997) (owner: 10Paladox) [18:29:38] (03CR) 10Paladox: "Thanks." [puppet] - 10https://gerrit.wikimedia.org/r/302129 (https://phabricator.wikimedia.org/T75997) (owner: 10Paladox) [18:49:24] (03PS1) 10Andrew Bogott: Update-grub in the new debian image [puppet] - 10https://gerrit.wikimedia.org/r/302133 (https://phabricator.wikimedia.org/T134944) [18:51:44] 06Operations, 10MediaWiki-Cache, 10Traffic, 05MW-1.28-release-notes, and 3 others: Cached outdated revisions served to logged-out users - https://phabricator.wikimedia.org/T141687#2509924 (10aaron) 05Open>03Resolved a:03aaron Closing unless new edits have this happen again. [18:59:46] 06Operations, 10MediaWiki-Cache, 10Traffic, 05MW-1.28-release-notes, and 3 others: Cached outdated revisions served to logged-out users - https://phabricator.wikimedia.org/T141687#2507607 (10Tgr) >>! In T141687#2509033, @Gestrid wrote: >>>! In T141687#2508997, @Perhelion wrote: >> I'm not sure it is relate... [19:09:06] (03CR) 10Andrew Bogott: [C: 032] Update-grub in the new debian image [puppet] - 10https://gerrit.wikimedia.org/r/302133 (https://phabricator.wikimedia.org/T134944) (owner: 10Andrew Bogott) [19:32:49] PROBLEM - Redis status tcp_6479 on rdb2006 is CRITICAL: CRITICAL: replication_delay is 629 600 - REDIS on 10.192.48.44:6479 has 1 databases (db0) with 4914257 keys - replication_delay is 629 [19:34:49] RECOVERY - Redis status tcp_6479 on rdb2006 is OK: OK: REDIS on 10.192.48.44:6479 has 1 databases (db0) with 4850896 keys - replication_delay is 0 [20:38:47] PROBLEM - Redis status tcp_6479 on rdb2006 is CRITICAL: CRITICAL: replication_delay is 668 600 - REDIS on 10.192.48.44:6479 has 1 databases (db0) with 4854728 keys - replication_delay is 668 [20:40:40] RECOVERY - Redis status tcp_6479 on rdb2006 is OK: OK: REDIS on 10.192.48.44:6479 has 1 databases (db0) with 4848331 keys - replication_delay is 0 [22:01:28] (03PS5) 10MarcoAurelio: Closing wikimania2015wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/298772 (https://phabricator.wikimedia.org/T139032) [22:13:49] (03CR) 10Zppix: [C: 031] Closing wikimania2015wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/298772 (https://phabricator.wikimedia.org/T139032) (owner: 10MarcoAurelio)