[00:21:08] PROBLEM - puppet last run on analytics1032 is CRITICAL: CRITICAL: Puppet last ran 6 hours ago [02:20:50] !log l10nupdate@tin Synchronized php-1.27.0-wmf.5/cache/l10n: l10nupdate for 1.27.0-wmf.5 (duration: 06m 21s) [02:20:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:21:06] PROBLEM - Host mw2027 is DOWN: PING CRITICAL - Packet loss = 100% [02:21:46] RECOVERY - Host mw2027 is UP: PING OK - Packet loss = 0%, RTA = 34.22 ms [03:35:46] PROBLEM - puppet last run on analytics1042 is CRITICAL: CRITICAL: Puppet has 1 failures [03:36:07] PROBLEM - puppet last run on mw2214 is CRITICAL: CRITICAL: Puppet has 1 failures [03:36:27] PROBLEM - puppet last run on elastic1011 is CRITICAL: CRITICAL: Puppet has 1 failures [03:36:37] PROBLEM - puppet last run on mw1212 is CRITICAL: CRITICAL: Puppet has 1 failures [03:36:46] PROBLEM - puppet last run on mw1163 is CRITICAL: CRITICAL: Puppet has 2 failures [03:40:28] PROBLEM - puppet last run on mw1164 is CRITICAL: CRITICAL: Puppet has 1 failures [03:50:18] PROBLEM - puppet last run on mw1066 is CRITICAL: CRITICAL: Puppet has 1 failures [03:59:38] PROBLEM - puppet last run on db1015 is CRITICAL: CRITICAL: Puppet has 1 failures [04:01:57] RECOVERY - puppet last run on mw2214 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [04:02:26] RECOVERY - puppet last run on elastic1011 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:02:36] RECOVERY - puppet last run on mw1212 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [04:02:38] RECOVERY - puppet last run on mw1163 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [04:03:28] RECOVERY - puppet last run on analytics1042 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:06:18] RECOVERY - puppet last run on mw1164 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:16:17] RECOVERY - puppet last run on mw1066 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [04:25:36] RECOVERY - puppet last run on db1015 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [05:16:56] PROBLEM - puppet last run on wtp2002 is CRITICAL: CRITICAL: Puppet has 1 failures [05:19:19] (03PS1) 10Ori.livneh: coal-web: make the default period 'day' instead of 'hour' [puppet] - 10https://gerrit.wikimedia.org/r/251799 [05:19:21] (03PS1) 10Ori.livneh: redis: prohibit commands CONFIG, SLAVEOF and DEBUG by default [puppet] - 10https://gerrit.wikimedia.org/r/251800 [05:19:46] (03CR) 10Ori.livneh: [C: 032 V: 032] coal-web: make the default period 'day' instead of 'hour' [puppet] - 10https://gerrit.wikimedia.org/r/251799 (owner: 10Ori.livneh) [05:31:38] PROBLEM - DPKG on mw1041 is CRITICAL: DPKG CRITICAL dpkg reports broken packages [05:33:37] RECOVERY - DPKG on mw1041 is OK: All packages OK [05:42:56] RECOVERY - puppet last run on wtp2002 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [05:57:56] PROBLEM - puppet last run on cp3016 is CRITICAL: CRITICAL: puppet fail [06:00:47] PROBLEM - Kafka Broker Replica Max Lag on kafka1014 is CRITICAL: CRITICAL: 80.00% of data above the critical threshold [5000000.0] [06:10:06] RECOVERY - Kafka Broker Replica Max Lag on kafka1014 is OK: OK: Less than 1.00% above the threshold [1000000.0] [06:25:37] RECOVERY - puppet last run on cp3016 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:30:28] PROBLEM - puppet last run on mw1260 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:56] PROBLEM - puppet last run on mw1203 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:16] PROBLEM - puppet last run on eventlog2001 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:27] PROBLEM - puppet last run on cp1068 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:16] PROBLEM - puppet last run on mw1170 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:17] PROBLEM - puppet last run on mw2045 is CRITICAL: CRITICAL: Puppet has 2 failures [06:33:06] PROBLEM - puppet last run on mw2207 is CRITICAL: CRITICAL: Puppet has 2 failures [06:33:08] PROBLEM - puppet last run on mw2126 is CRITICAL: CRITICAL: Puppet has 1 failures [06:33:17] PROBLEM - puppet last run on mw2073 is CRITICAL: CRITICAL: Puppet has 1 failures [06:33:47] PROBLEM - puppet last run on mw2050 is CRITICAL: CRITICAL: Puppet has 2 failures [06:44:36] PROBLEM - Apache HTTP on mw1083 is CRITICAL: Connection refused [06:50:17] RECOVERY - Apache HTTP on mw1083 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 440 bytes in 0.511 second response time [06:56:17] RECOVERY - puppet last run on mw1170 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [06:56:36] RECOVERY - puppet last run on mw1260 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:56:57] RECOVERY - puppet last run on mw1203 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:57:16] RECOVERY - puppet last run on mw2207 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [06:57:17] RECOVERY - puppet last run on mw2126 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [06:57:26] RECOVERY - puppet last run on eventlog2001 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [06:57:28] RECOVERY - puppet last run on cp1068 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:57:57] RECOVERY - puppet last run on mw2050 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [06:58:18] RECOVERY - puppet last run on mw2045 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:59:18] RECOVERY - puppet last run on mw2073 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:56:17] PROBLEM - puppet last run on mw2144 is CRITICAL: CRITICAL: puppet fail [09:06:56] PROBLEM - HTTP 5xx req/min on graphite1001 is CRITICAL: CRITICAL: 8.33% of data above the critical threshold [500.0] [09:16:07] RECOVERY - HTTP 5xx req/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] [09:18:26] PROBLEM - puppet last run on mw2199 is CRITICAL: CRITICAL: Puppet has 1 failures [09:24:07] RECOVERY - puppet last run on mw2144 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:44:14] (03CR) 10Jcrespo: [C: 031] Use a more useful error message when DB connection fails [software/dbtree] - 10https://gerrit.wikimedia.org/r/251791 (owner: 10Alex Monk) [09:44:27] RECOVERY - puppet last run on mw2199 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [11:40:22] I fixed the dbstore1002 issue [11:40:28] but no icinga bot? [12:42:57] (03PS3) 10Paladox: Re enable git.enableGitServlet [puppet] - 10https://gerrit.wikimedia.org/r/250450 [12:47:29] (03CR) 10Paladox: "@Krinkle would this fix the png files not showing. Since the message shows that it clones and push to gitblit." [puppet] - 10https://gerrit.wikimedia.org/r/250450 (owner: 10Paladox) [12:56:29] (03PS1) 10Jcrespo: Depooling db1060 for restart- it has started creating bad plans [mediawiki-config] - 10https://gerrit.wikimedia.org/r/251807 [12:58:28] (03PS2) 10Jcrespo: Depooling db1060 for restart- it has started creating bad plans [mediawiki-config] - 10https://gerrit.wikimedia.org/r/251807 [12:59:37] (03CR) 10Jcrespo: [C: 032] Depooling db1060 for restart- it has started creating bad plans [mediawiki-config] - 10https://gerrit.wikimedia.org/r/251807 (owner: 10Jcrespo) [13:02:37] !log jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1060 (duration: 01m 29s) [13:02:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [13:19:22] (03PS1) 10Jcrespo: Back to previous state (and repool db1060) after maintenance [mediawiki-config] - 10https://gerrit.wikimedia.org/r/251810 [13:28:44] (03CR) 10Jcrespo: [C: 032] Back to previous state (and repool db1060) after maintenance [mediawiki-config] - 10https://gerrit.wikimedia.org/r/251810 (owner: 10Jcrespo) [13:32:31] !log jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1060 (duration: 00m 34s) [13:32:37] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [13:34:10] is there a ducumentation about the jobqueue? [13:34:13] how it works etc. [13:34:24] * Steinsplitter like to write a helppage on commons [13:34:50] Steinsplitter: https://www.mediawiki.org/wiki/Manual:Job_queue [13:34:52] and if i am right it is only possible to get the nrs of running jobs via api action=query&meta=siteinfo&siprop=statistics&format=json' [13:35:21] valhallasw`cloud: oh, i know that pay yet. cool. thanks. so no doc on wikitech :) [13:35:48] https://grafana.wikimedia.org/dashboard/db/job-queue-health [13:35:53] https://grafana.wikimedia.org/dashboard/db/job-queue-rate [13:36:01] There might be one on wikitech about how the job queue is implemented for WMF specifically [13:36:33] thanks both of you. [13:37:14] Steinsplitter: if possible, please try to improve the mw.org page rather than creating a seperate one for commons [13:37:50] having documentation on even more places makes life hard for everyone, because they always get out of sync [13:41:51] valhallasw`cloud: that makes sense. but i plan to write a script which is posting the count or running jobs online. So users knowing if there is a lag or not O_O [13:48:27] PROBLEM - Kafka Broker Replica Max Lag on kafka1022 is CRITICAL: CRITICAL: 87.50% of data above the critical threshold [5000000.0] [13:53:23] <_joe_> Steinsplitter: number of running jobs is available in grafana [13:53:56] <_joe_> as jynus pointed out, sorry reading the backlog in reverse order [13:54:07] PROBLEM - Kafka Broker Replica Max Lag on kafka1022 is CRITICAL: CRITICAL: 14.29% of data above the critical threshold [5000000.0] [13:54:41] yesterday issue happened again, I've put a patch, will look at it later [13:54:57] <_joe_> jynus: I didn't get paged this time though [13:55:04] <_joe_> jynus: :( [13:55:18] I had some things to make it less impacting [13:55:27] _joe_ thanks aniway. also thanks jynus vand valhalle. was helpful for me to know :) [13:55:57] RECOVERY - Kafka Broker Replica Max Lag on kafka1022 is OK: OK: Less than 1.00% above the threshold [1000000.0] [13:56:09] _joe_, compare: https://logstash.wikimedia.org/#dashboard/temp/AVDnYcwLptxhN1XaQLCa [13:57:13] <_joe_> yup, there is some differences [13:57:19] <_joe_> *are [13:57:39] the load is excessive, I would say [13:57:47] but we should be able to handle [13:57:53] we can do it [14:11:27] PROBLEM - puppet last run on gallium is CRITICAL: CRITICAL: puppet fail [14:14:29] (03CR) 10Chmarkine: "An alternative is to block old browsers like IE 8 (by user-agent) at the application layer only for those security-sensitive services (lik" [puppet] - 10https://gerrit.wikimedia.org/r/251704 (owner: 10BBlack) [14:28:17] RECOVERY - puppet last run on gallium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:04:16] PROBLEM - puppet last run on rdb1001 is CRITICAL: CRITICAL: Puppet has 1 failures [15:07:36] PROBLEM - Kafka Broker Replica Max Lag on kafka1022 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [5000000.0] [15:07:46] PROBLEM - Kafka Broker Replica Max Lag on kafka1012 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [5000000.0] [15:22:27] RECOVERY - Kafka Broker Replica Max Lag on kafka1022 is OK: OK: Less than 1.00% above the threshold [1000000.0] [15:22:37] RECOVERY - Kafka Broker Replica Max Lag on kafka1012 is OK: OK: Less than 1.00% above the threshold [1000000.0] [15:30:17] RECOVERY - puppet last run on rdb1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:02:11] Greetings everyone -- I am one of the authors of IPFS -- https://ipfs.io -- https://en.wikipedia.org/wiki/IPFS -- we would like to help you (a) save storage space and bandwidth, (b) make offline wikimedia caches much easier to use, (c) make it so users can contribute bandwidth to serving wikipedia, and help replicate the data to safeguard it. Who should I [16:02:11] discuss with? Would like to start with a pilot test set of a bunch of wikimedia images. :) [16:08:10] jbenet: I guess you need to be more specific... so you want to offer the Wikimedia foundation to host "a bunch of" images? [16:29:52] 6operations, 6Release-Engineering-Team: Manage Appveyor account - https://phabricator.wikimedia.org/T104306#1792057 (10greg) What/who uses this appveyor account? [16:33:46] PROBLEM - puppet last run on mw1259 is CRITICAL: CRITICAL: puppet fail [16:43:39] 6operations, 6Release-Engineering-Team: Manage Appveyor account - https://phabricator.wikimedia.org/T104306#1792073 (10valhallasw) Pywikibot does, for running tests on Windows: https://ci.appveyor.com/project/wikimedia/pywikibot-core [16:44:57] PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL: CRITICAL: Anomaly detected: 11 data above and 9 below the confidence bounds [16:59:57] RECOVERY - puppet last run on mw1259 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:04:38] PROBLEM - very high load average likely xfs on ms-be1009 is CRITICAL: CRITICAL - load average: 235.63, 146.41, 71.05 [17:08:52] <_joe_> I'm looking at ms-be1009 [17:09:38] PROBLEM - Host ms-be1009 is DOWN: PING CRITICAL - Packet loss = 100% [17:10:25] <_joe_> !log powercycled ms-be1009, unresponsive to ping, console unconneptable [17:10:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [17:12:57] RECOVERY - Host ms-be1009 is UP: PING OK - Packet loss = 0%, RTA = 1.42 ms [17:13:58] RECOVERY - very high load average likely xfs on ms-be1009 is OK: OK - load average: 23.81, 7.88, 2.78 [17:15:18] PROBLEM - YARN NodeManager Node-State on analytics1032 is CRITICAL: CHECK_NRPE: Socket timeout after 10 seconds. [17:18:57] RECOVERY - YARN NodeManager Node-State on analytics1032 is OK: OK: YARN NodeManager analytics1032.eqiad.wmnet:8041 Node-State: RUNNING [17:24:26] RECOVERY - HTTP error ratio anomaly detection on graphite1001 is OK: OK: No anomaly detected [17:35:13] (03PS1) 10Ori.livneh: Omit request hostname and URI from Varnish error page [puppet] - 10https://gerrit.wikimedia.org/r/251832 (https://phabricator.wikimedia.org/T118129) [17:35:27] (03CR) 10Ori.livneh: [C: 032 V: 032] Omit request hostname and URI from Varnish error page [puppet] - 10https://gerrit.wikimedia.org/r/251832 (https://phabricator.wikimedia.org/T118129) (owner: 10Ori.livneh) [17:48:13] (03PS1) 10Ori.livneh: Follow-up to Iaa3efabc8e: add missing '}' [puppet] - 10https://gerrit.wikimedia.org/r/251833 [17:48:28] (03CR) 10Ori.livneh: [C: 032 V: 032] Follow-up to Iaa3efabc8e: add missing '}' [puppet] - 10https://gerrit.wikimedia.org/r/251833 (owner: 10Ori.livneh) [17:59:08] (03PS1) 10Paladox: Update gitblit.properties file with new configs [puppet] - 10https://gerrit.wikimedia.org/r/251836 [18:05:41] I think logging table missing some entires [18:07:27] eranroz: What entries are you missing? Please be aware that they could have been deleted or even oversighted [18:07:36] Try not to expose hidden data [18:08:16] hoo: it seems that some revisions were undeleted (they appear in site), and there is no logging entires either for their deletions nor to the undeletion [18:08:41] I will explain the details in phab as a possible security issue [18:08:42] Link? Maybe they weren't deleted in the first place [18:08:46] Ok [18:08:48] https://he.wikipedia.org/w/index.php?title=%D7%90%D7%A0%D7%93%D7%A8_%D7%90%D7%A8%D7%9E%D7%95%D7%A8&action=history [18:09:07] but dont trust the site and logging table - compare it to /public/dumps/public/hewiki/20151002/hewiki-20151002-pages-logging.xml.gz [18:10:01] <_joe_> I would trust the logging table way more than the dumps [18:11:45] indeed [18:13:59] eranroz: Can you tell me the log ids you are looking for? [18:14:08] I can only find patrol logs for that page in the dump [18:14:10] and those are there [18:14:49] I actully didn't look into the specific entires but did zcat /public/dumps/public/hewiki/20151002/hewiki-20151002-pages-logging.xml.gz | grep "אנדר ארמור" [18:15:08] I am running $ zgrep -A 3 -B 9 'אנדר ארמור' /public/dumps/public/hewiki/20151002/hewiki-20151002-pages-logging.xml.gz [18:15:13] and all entries I see are there [18:15:47] https://he.wikipedia.org/w/index.php?title=%D7%9E%D7%99%D7%95%D7%97%D7%93:%D7%99%D7%95%D7%9E%D7%A0%D7%99%D7%9D&page=%D7%90%D7%A0%D7%93%D7%A8+%D7%90%D7%A8%D7%9E%D7%95%D7%A8&hide_patrol_log=0 [18:17:42] hoo: its not just about a one-off, have you taken a look at what IPFS is and how it works? if not, it would be useful to start there. https://ipfs.io -- https://github.com/ipfs/archives/blob/master/README.md (useful quick summary) and https://www.youtube.com/watch?v=HUVmypx9HGI (long talk about the project) -- [18:17:42] https://ipfs.io/ipfs/QmR7GSQM93Cx5eAg6a6yRzNde1FQv7uL6X1o4k7zrJa3LX/ipfs.draft3.pdf (paper if you're so inclined, but note it's somewhat outdated at this point -- github docs more up to date) [18:18:14] hoo: yes it looks OK. so I will check it more carefully [18:19:20] jbenet: I read the Wikipedia article quickly [18:20:12] hoo: OK I found were it was missing - there was a different page that was merged to this one [18:20:29] thanks for the help [18:21:30] jbenet: I'm still not sure what you want to offer... most of Wikimedia's content can be taken (as long as you stick to the licenses) and be shared or archived or whatever [18:21:46] the internet archive eg. already has all(?) or our dumps replicated [18:21:58] but we don't have full public image dumps anymore :S [18:24:06] hoo: yeah that's fine and we'll do that (archive all the things), and that's a good place to start. the _more_ interesting part is to setup a live mirror of wikipedia on ipfs and test browsing it that way. will make it easy to distribute changes to replicas wherever bandwidth is very expensive (and even for offline use cases) [18:25:38] mh... doing that in a distributed manner might not be trivial. [18:26:07] We don't have full html dumps, thus you need either a proxy or a MediaWiki instance, but both can't really be distributed well [18:26:13] at least not as part of your project [18:30:57] hoo: can be done with a program receiving mediawiki updates, adding them to ipfs, and updating the head [18:31:26] hoo: is wikipedia just stock mediawiki? or is it very patched? [18:31:57] hoo: i.e. if we get it working with any mediawiki instance, will it work with the main wikipedia instance too [18:33:18] <_joe_> jbenet: there are several ways to follow wikimedia changes - i.e. https://wikitech.wikimedia.org/wiki/RCStream?redirect=no [18:33:32] Well, Wikipedia is MediaWiki with many extensions (but all open source and stuff) and updates are public via the link _joe_ posted [18:33:43] (also other APIs, but that is probably the most convenient) [18:33:52] But getting that all together will be anything but trivial [18:34:04] You will also need to mirror Wikidata [18:34:19] great! that looks pretty good. [18:34:22] hoo: indeed. [18:35:05] hoo: I'll write up a plan with incremental steps. [18:35:49] you could also just take full html captures and store these, I guess [18:35:54] hoo _joe_: what's the best doc explaining all the data structures and relations? (e.g. articles, users, versions, media files, ... ) [18:35:55] and invalidate whenever an edit happens [18:36:13] There's none covering that all [18:36:28] maybe on a very high level [18:36:41] high level + links/search is ok. [18:37:31] You will probably need to search your way around https://www.mediawiki.org/ [18:39:01] ok will do. [18:41:53] away for a bit [19:31:57] PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL: CRITICAL: Anomaly detected: 10 data above and 1 below the confidence bounds [19:41:07] PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL: CRITICAL: Anomaly detected: 10 data above and 4 below the confidence bounds [20:09:07] PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL: CRITICAL: Anomaly detected: 10 data above and 6 below the confidence bounds [20:14:34] !log ori@tin Synchronized php-1.27.0-wmf.5/resources/src: I24d9b16ed: Rework the Preferences to prevent FOUC (T115692) (duration: 00m 34s) [20:14:40] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [20:16:56] Hm.. getting MWException from load.php on mediawiki.org [20:17:21] uhm [20:17:23] which page(s)? [20:17:31] Special:Preferences on eswiki broken [20:17:47] yeah, give it a couple of minutes, it's the cache [20:17:52] I24d9b16ed changes more than just src/ also Resources.php and Preferences.php [20:18:09] i synced all, but silenced the previous ones so that it's one log entry [20:18:23] https://www.mediawiki.org/wiki/Special:Preferences [20:18:32] style file not found: "/srv/mediawiki/php-1.27.0-wmf.5/resources/src/mediawiki.special/mediawiki.special.preferences.css" [20:19:13] 2015-11-08 20:18:13 mw1071 mediawikiwiki exception ERROR: [a0972ca1] /w/load.php?debug=false&lang=en&modules=mediawiki.special.preferences&skin=vector&version=b23be595d971 MWException from line 876 of /srv/mediawiki/php-1.27.0-wmf.5/includes/resourceloader/ResourceLoaderFileModule.php: ResourceLoaderFileModule::readStyleFile: style file not found: [20:19:13] "/srv/mediawiki/php-1.27.0-wmf.5/resources/src/mediawiki.special/mediawiki.special.preferences.css" {"exception_id":"a0972ca1"} [20:19:41] k, sec [20:19:59] That's the old file, so presumably sync of Resources.php failed [20:21:22] !log ori@tin Synchronized php-1.27.0-wmf.5/resources: (no message) (duration: 00m 34s) [20:21:23] synced it again(?) [20:21:26] looks fine to me [20:21:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [20:21:54] yeah, it's fine again [20:22:04] better than fine, no fouc [20:22:15] CSS renders fine now [20:22:18] but still broken [20:22:19] JS is missing [20:22:24] tab switching doesn't work [20:22:31] WFM [20:22:37] WFM too [20:22:51] OK. Works now :) [20:23:39] Although something else did regress [20:23:43] the green "Your preferences have been saved." now shows up post-save [20:23:49] in addition to the mw notify bubble [20:23:53] I thought it was supposed to hide that [20:24:48] we should kill one of those (probably the mw.notify one) instead of having two and hiding one [20:26:28] I merged the change that introduced mw.notify for that [20:26:31] which worked fine [20:26:35] but this commit broke it [20:26:40] it was always flashinf for me [20:26:42] https://gerrit.wikimedia.org/r/#/c/250287/14/resources/src/mediawiki.special/mediawiki.special.preferences.css [20:26:54] That css loads it ahead of time so it is never shown [20:27:07] in JS mode [20:27:25] will check monday if nobody beats me to it [20:27:26] o/ [20:27:58] bye [20:32:57] (03CR) 10Krinkle: [C: 04-1] "This is not related to the rendering bug." [puppet] - 10https://gerrit.wikimedia.org/r/250450 (owner: 10Paladox) [20:37:07] PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL: CRITICAL: Anomaly detected: 12 data above and 5 below the confidence bounds [20:42:37] PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL: CRITICAL: Anomaly detected: 12 data above and 5 below the confidence bounds [20:43:54] !log krenair@tin Synchronized php-1.27.0-wmf.5/extensions/ConfirmEdit/SimpleCaptcha/Captcha.php: https://gerrit.wikimedia.org/r/#/c/251673/ (duration: 00m 34s) [20:43:59] quiddity, ^ can you try now [20:43:59] ? [20:44:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [20:44:34] will do.. [20:45:24] thanks [20:46:18] Krenair, that works now. [20:46:32] ty! [20:47:20] np [20:53:11] (03CR) 10Alex Monk: [C: 031] Add patroller group to sawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/250454 (https://phabricator.wikimedia.org/T117314) (owner: 10Luke081515) [21:03:50] (03CR) 10Alex Monk: [C: 031] Add an apple-app-site-association file used to support iOS deep-linking [mediawiki-config] - 10https://gerrit.wikimedia.org/r/250897 (https://phabricator.wikimedia.org/T111829) (owner: 10Bgerstle) [21:05:00] (03CR) 10Alex Monk: [C: 031] Enable VisualEditor for draft namespace in fawiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/251674 (https://phabricator.wikimedia.org/T118060) (owner: 10Ladsgroup) [21:10:28] RECOVERY - HTTP error ratio anomaly detection on graphite1001 is OK: OK: No anomaly detected [21:11:39] (03CR) 10Alex Monk: [C: 04-1] "-1, unaddressed comment" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/247093 (https://phabricator.wikimedia.org/T115812) (owner: 10Luke081515) [21:12:48] (03CR) 10Alex Monk: [C: 04-1] "Contains part of Iaecd06f4 as well as the enwikiversity change?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/250709 (https://phabricator.wikimedia.org/T115938) (owner: 10Dereckson) [21:14:05] (03CR) 10Alex Monk: [C: 031] Set $wgCategoryCollation for bs.wikipedia.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/250862 (https://phabricator.wikimedia.org/T116527) (owner: 10Dereckson) [21:14:16] Krenair: yes, it does. Fixing that. [21:14:30] k [21:15:35] Catching up on the mediawiki-config backlog :) [21:18:08] (03PS2) 10Dereckson: Set import sources on en.wikiversity [mediawiki-config] - 10https://gerrit.wikimedia.org/r/250709 (https://phabricator.wikimedia.org/T115938) [21:19:44] (03CR) 10Alex Monk: [C: 04-1] Allow import from any Labs/Beta Cluster project to any other (033 comments) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/157338 (https://phabricator.wikimedia.org/T17583) (owner: 10TTO) [21:22:23] (03CR) 10Alex Monk: [C: 031] Set import sources on en.wikiversity [mediawiki-config] - 10https://gerrit.wikimedia.org/r/250709 (https://phabricator.wikimedia.org/T115938) (owner: 10Dereckson) [21:23:56] PROBLEM - Kafka Broker Replica Max Lag on kafka1014 is CRITICAL: CRITICAL: 28.57% of data above the critical threshold [5000000.0] [21:38:51] (03CR) 10Alex Monk: [C: 04-1] "unaddressed comments" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/246171 (https://phabricator.wikimedia.org/T114982) (owner: 10Dereckson) [21:38:56] RECOVERY - Kafka Broker Replica Max Lag on kafka1014 is OK: OK: Less than 1.00% above the threshold [1000000.0] [21:46:57] PROBLEM - puppet last run on wtp2005 is CRITICAL: CRITICAL: puppet fail [21:58:37] PROBLEM - HTTP error ratio anomaly detection on graphite1001 is CRITICAL: CRITICAL: Anomaly detected: 10 data above and 2 below the confidence bounds [22:09:49] (03CR) 10Alex Monk: Tidy robots.txt (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/240065 (https://phabricator.wikimedia.org/T104251) (owner: 10Mdann52) [22:14:47] RECOVERY - puppet last run on wtp2005 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:30:56] (03PS1) 10Yuvipanda: dynamicproxy: Send log line metrics to graphite [puppet] - 10https://gerrit.wikimedia.org/r/251899 [22:32:59] (03PS2) 10Yuvipanda: dynamicproxy: Send log line metrics to graphite [puppet] - 10https://gerrit.wikimedia.org/r/251899 [22:33:10] (03CR) 10Yuvipanda: [C: 032 V: 032] dynamicproxy: Send log line metrics to graphite [puppet] - 10https://gerrit.wikimedia.org/r/251899 (owner: 10Yuvipanda) [22:35:29] Dereckson, are you planning to put those two commits up for swat? [22:35:56] I can do that tomorrow, yes. [22:41:51] (03PS1) 10Yuvipanda: dynamicproxy: Fix typo [puppet] - 10https://gerrit.wikimedia.org/r/251900 [22:42:57] (03CR) 10Yuvipanda: [C: 032] dynamicproxy: Fix typo [puppet] - 10https://gerrit.wikimedia.org/r/251900 (owner: 10Yuvipanda) [23:29:46] RECOVERY - HTTP error ratio anomaly detection on graphite1001 is OK: OK: No anomaly detected [23:35:43] Krenair: Hi! There are some small follow-on patches that fix issues with the update that I found at the last minute... I was hoping someone would kick them through over the weekend. Sorry about that!! [23:36:04] NEway that's why I didn't put a commit there... [23:36:18] you have multiple patches to add? [23:36:19] Seems likely we'd just put it off instead [23:36:45] Krenair: it would have just been submodule bump [23:36:53] ok [23:36:56] To point to the update of the wmf_deploy branch [23:37:09] But I think likely we'll hold off...