[03:20:42] (03PS1) 10Tulsi Bhagat: Enable Quiz extension on ru.wikibooks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/481512 [03:25:57] (03PS2) 10Tulsi Bhagat: Enable Quiz extension on ru.wikibooks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/481512 (https://phabricator.wikimedia.org/T212622) [03:31:52] (03PS1) 10Tulsi Bhagat: Add 'suppressredirect' user right to editor user group at pl.wikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/481513 [03:37:51] (03PS2) 10Tulsi Bhagat: Add 'suppressredirect' user right to editor user group at pl.wikisource [mediawiki-config] - 10https://gerrit.wikimedia.org/r/481513 (https://phabricator.wikimedia.org/T212655) [03:39:59] PROBLEM - MariaDB Slave Lag: s1 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 1024.13 seconds [04:05:43] RECOVERY - MariaDB Slave Lag: s1 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 210.46 seconds [04:35:47] PROBLEM - HTTP availability for Nginx -SSL terminators- at eqiad on icinga1001 is CRITICAL: cluster=cache_text site=eqiad https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4fullscreenrefresh=1morgId=1 [04:39:27] RECOVERY - HTTP availability for Nginx -SSL terminators- at eqiad on icinga1001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4fullscreenrefresh=1morgId=1 [06:28:37] PROBLEM - netbox HTTPS on netmon1002 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 547 bytes in 0.008 second response time [06:29:09] PROBLEM - Check systemd state on netmon1002 is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [06:30:59] PROBLEM - puppet last run on db1092 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/local/bin/puppet-enabled] [06:31:21] PROBLEM - puppet last run on mw1307 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 5 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/usr/local/bin/cgroup-mediawiki-clean],File[/usr/local/bin/hhvmadm] [06:32:53] PROBLEM - puppet last run on db1079 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 7 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/lib/nagios/plugins/check_conntrack] [06:37:07] RECOVERY - netbox HTTPS on netmon1002 is OK: HTTP OK: HTTP/1.1 302 Found - 348 bytes in 0.550 second response time [06:37:39] RECOVERY - Check systemd state on netmon1002 is OK: OK - running: The system is fully operational [06:57:03] RECOVERY - puppet last run on db1092 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [06:57:25] RECOVERY - puppet last run on mw1307 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:58:55] RECOVERY - puppet last run on db1079 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [08:10:35] PROBLEM - Citoid LVS eqiad on citoid.svc.eqiad.wmnet is CRITICAL: /api (Scrapes sample page) timed out before a response was received [08:11:41] RECOVERY - Citoid LVS eqiad on citoid.svc.eqiad.wmnet is OK: All endpoints are healthy [11:24:47] 10Operations, 10Developer-Advocacy, 10Discourse: Discourse migration from wmflabs to production - https://phabricator.wikimedia.org/T184461 (10Elitre) (It'd be nice to have https://phabricator.wikimedia.org/T184461#4287321 erased as that's likely spam. Sorry for the noise.) [11:49:46] (03PS1) 10MarcoAurelio: Define $wgMetaNamespace for be.wikibooks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/481520 (https://phabricator.wikimedia.org/T212665) [12:05:10] (03CR) 10MarcoAurelio: "Requires `namespaceDupes.php --wiki=bewikibooks --fix` after deployment." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/481520 (https://phabricator.wikimedia.org/T212665) (owner: 10MarcoAurelio) [12:10:31] 10Operations, 10MediaWiki-Cache, 10MW-1.33-notes (1.33.0-wmf.12; 2019-01-08), 10Patch-For-Review, and 3 others: Mcrouter periodically reports soft TKOs for mc1022 (was mc1035) leading to MW Memcached exceptions - https://phabricator.wikimedia.org/T203786 (10elukey) I was curious and I checked some memcache... [13:11:29] PROBLEM - HTTP availability for Nginx -SSL terminators- at eqiad on icinga1001 is CRITICAL: cluster=cache_text site=eqiad https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4fullscreenrefresh=1morgId=1 [13:13:41] PROBLEM - Eqiad HTTP 5xx reqs/min on graphite1004 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [1000.0] https://grafana.wikimedia.org/dashboard/file/varnish-aggregate-client-status-codes?panelId=3fullscreenorgId=1var-site=eqiadvar-cache_type=Allvar-status_type=5 [13:15:07] RECOVERY - HTTP availability for Nginx -SSL terminators- at eqiad on icinga1001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4fullscreenrefresh=1morgId=1 [13:19:43] RECOVERY - Eqiad HTTP 5xx reqs/min on graphite1004 is OK: OK: Less than 1.00% above the threshold [250.0] https://grafana.wikimedia.org/dashboard/file/varnish-aggregate-client-status-codes?panelId=3fullscreenorgId=1var-site=eqiadvar-cache_type=Allvar-status_type=5 [13:20:42] link not rendered with & afaics.. --^ [13:25:44] does it need escaping? [13:26:10] (checked the 50x, recurrent issue, not a big deal afaics) [13:30:16] mhh interesting, it used to work [13:30:30] godog: o/ [13:30:37] ciao elukey ! [13:31:13] interesting, even the HTTP availability for nginx is missing & [13:31:54] and also the last logstash one [13:35:02] the ampersands are in icinga's config but not in the logs that icinga-wm reads btw [13:35:11] ah!! [13:52:10] looks like it might have to do with the icinga upgrade [13:55:18] ping to whomever is on duty [13:55:51] robh: around? [13:56:00] * sDrewth looks up and down for hints [13:56:21] sDrewth: I'm not on clinic duty, I can help for emergencies tho [13:56:32] we have feral spambots [13:56:41] godog: we need to block the spambot [13:56:58] 1700 account creations attempts in 15-20 minutes [13:57:11] godog: preferebly directly on network level so it won't hit mediawiki servers at all [13:57:17] and have had similar elsewhere, and different IP{s [13:57:25] IPs [13:57:51] beyond steward controls, have written gross filter that is holding, but not ideal [13:58:44] ack, is there a task ? [13:58:50] also there is obviously some bug in api, as it allows more account creations than throttle says [14:00:41] spambots: ignored by WMF for so long [14:01:46] * sDrewth deletes that unhelpful comment for this time [14:05:30] godog: https://www.mediawiki.org/w/index.php?title=Special:AbuseLog&wpSearchFilter=61 or https://meta.wikimedia.org/w/index.php?title=Special:AbuseLog&wpSearchFilter=195 [14:05:44] I have turned off MWO [14:05:53] so use meta /195 [14:06:15] Public 4,664 hits [14:06:37] 30 minutes [14:06:40] faaahk [14:07:09] thanks Danny_B [14:07:10] can we just simply turn of account creation via api? [14:07:19] at least for a bit [14:08:15] I wouldn't know how to do that, I'm looking through the logs tho [14:08:59] though the throttle failing to throttle seems like a legitimate bug, please file a task for that in the meantime [14:11:32] throttle is 6=> as default [14:11:50] can we pull the global throttle back to 2 as a temporary measure? [14:12:10] or do i misunderstand [14:12:26] https://noc.wikimedia.org/conf/highlight.php?file=InitialiseSettings.php [14:12:26] 'wgAccountCreationThrottle' => [ [14:12:27] 'default' => 6, // previously 10 [14:19:52] sDrewth: for mediawikiwiki ? we could though please file a task so there's a trail [14:20:59] godog, it was at koWP earlier, and that created 1600 accounts [14:21:15] they had a really crude filter that stops all creations [14:21:29] but can do for MWO [14:22:19] not sure what you mean with MWO, mediawiki.org ? [14:24:22] mediawikiwiki yes [14:24:36] godog, rough task https://phabricator.wikimedia.org/T212667 [14:24:48] going back to add more detail [14:26:18] thanks sDrewth [14:27:44] Usually the argument against turning off API account creation, is its trivial to do it not via the API [14:28:04] So at best it would be a temporary roadblock for spambots until they rewrite their script [14:28:20] actually it seems that it was not via api [14:28:20] bawolff, nearly up to 10k accounts [14:28:22] 10000 [14:28:29] they solved captcha [14:28:31] in ah hour [14:28:35] an [14:28:38] https://grafana.wikimedia.org/d/000000004/authentication-metrics?orgId=1&from=now-6h&to=now&var-entrypoint=* [14:28:40] out captcha is crap [14:28:43] *our [14:28:50] (courtesy of Nemo_bis) [14:29:08] meh, see my commentary in phabricator for years [14:29:26] there was a task where the captchas were reset [14:29:40] cannot remember the detail [14:29:45] That account creation graph is interesting [14:29:54] spike in *failed* account creations? [14:30:21] Krenair: abuse filter blocks it [14:30:42] Krenair, see meta stewards' noticeboard [14:31:07] oh this https://www.mediawiki.org/wiki/Special:AbuseFilter/61 [14:31:14] Why is "Enable this filter" not ticked? [14:31:14] btw who's on duty when shdubsh is not online? [14:31:29] Krenair: now handled via global af [14:31:37] ok [14:31:48] Krenair: https://meta.wikimedia.org/w/index.php?title=Special:AbuseLog&wpSearchFilter=195 [14:32:18] Krenair, I turned it off, as meta /195 is the same [14:32:29] I was seeing if blocking would do anything, NADA [14:33:18] so we're blocking all account creations by people who aren't sysops where the account name is 12 alphanumeric characters and there is at least one digit, including auto-creations [14:34:24] yes [14:34:48] Krenair: it has some false positives but like 1 in thousand or so [14:35:02] yeah I don't think this is something that should remain long-term [14:35:14] alright enough as a temporary measure I guess [14:35:21] maybe a time to switch to defcon 3 and bring jimbo the suitcase with big red button "turn wikipedia off"? [14:36:55] bawolff, do you agree with dropping wgAccountCreationThrottle down to 2 or 3? [14:36:58] Krenair, at least as a test to see if we are getting 1 or 5 per IP address [14:37:11] I'm fine with doing it as an emergency measure [14:37:12] sDrewth, can't you get that info from CU? [14:37:19] <- not a CU [14:37:20] btw: don't we have any other captchas such as count 2 + 3 and write the result? [14:37:24] I was just trying to look at logstash to see what the IP pattern is [14:37:37] if the spambot now solves current captcha, we may configure mwo to use different one [14:37:51] I'm wondering what's "1" "1_2" "1_2_3" in https://grafana.wikimedia.org/d/000000004/authentication-metrics?orgId=1&from=1545984578486&to=1546007678680&var-entrypoint=*&panelId=12&fullscreen&edit there has been a spike of "1_2" a little while ago [14:38:19] Danny_B, there is an extension for that but we don't use it [14:38:26] bawolff, Krenair, Danny_B here is the earlier phabricator task with commentary https://phabricator.wikimedia.org/T125132 [14:38:36] You do not have permission to view this object. [14:39:08] Krenair: which object? [14:39:16] the one sDrewth linked to [14:39:19] an emergency switch to simplecaptcha could be a reasonable response for the very short term [14:39:38] you wanna deploy simplecaptcha to prod on 28th December...? [14:39:39] Krenair, wait one [14:39:47] unless this bot is already trained for multiple MediaWiki farms and has a solver for that already in store :) [14:40:17] sDrewth: i also don't have access to that task [14:40:29] I'll add you [14:40:38] godog: those should just be the abusefilters created in response, no? [14:40:39] done already bawolff [14:41:24] sDrewth: Actually using @username to subscribe people doesn't work on security tasks [14:41:25] godog, what is your phabricator nick? [14:41:34] you have to actually set a subscriber [14:41:43] oh, f [14:41:43] sDrewth: fgiunchedi, I've added myself [14:42:24] Wow, login logs are super spammed with GrantsBot. Its logging in multiple times a second [14:42:52] it's not getting throttled back? [14:42:53] Nemo_bis: could be! no idea what those spikes are, the graphite query is groupByNode(MediaWiki.authmanager.accountcreation.$entrypoint.failure.*.sum, 5, 'sum') [14:45:11] Krenair: might just be the time period i'm looking at, but no, as the login is succesful [14:46:07] going to change the filter [14:46:17] https://meta.wikimedia.org/w/index.php?title=Special:AbuseLog&wpSearchFilter=195 NOT INCREMENTING :-) [14:46:22] it switched to longer [14:46:58] working :-) [14:47:42] i changed [14:47:45] nope, back again [14:47:54] it hits the af [14:48:01] three minute hiatus only [14:48:52] that is imho because of slower response when account is created [14:49:14] also on the bot's side it may have consequences when successfull creation [14:49:23] okay, can see one attempt to meta, same methodology [14:49:49] stopped now on mwo [14:49:59] they are obviously changing to another pattern ;-) [14:50:20] Danny_B, it could be that some IPs haven't hit throttle, and they are the newer [14:50:26] nah, continuing [14:50:30] again [14:51:08] Danny_B, account name length different [14:51:10] can af distinguish between creation and autocreation? [14:51:26] sDrewth: where? [14:51:39] User:Qz1p0N4mK2wFz2Av [14:51:51] not 12 chARACTERS [14:51:57] af now blocks 12 OR 16 chars [14:52:08] okay, missed your change [14:52:27] 15:47:41 < Danny_B> i changed [14:52:32] (UTC+1) [14:52:40] https://logstash.wikimedia.org/goto/3c1db80c02746ed632b82ebeb7eb7ea3 [14:52:48] sure, but you have to reload it! [14:52:53] Looks like IP addresses are not repeating more than 3 times [14:53:23] we may modify the filter to block only creations not autocreations - that would lower down the false positives when user just visits different wiki [14:53:26] we could temporarily turn off creations at mediawikiwiki [14:54:10] yes, but preferebly with autocreation available [14:54:55] i'll try to take the autocreateoff the filter to see [14:55:19] we could change it on login.wmf [14:55:48] stops multiples for a while [14:57:29] sh*t, new pattern [14:57:30] Are we sure they are succesfully filling out the captcha? [14:57:51] bawolff: unless there is any bug to bypass it [14:58:07] graphana says captcha successfull [14:58:31] right, all account creations need captcha [14:58:42] sorry, I was thinking it was like login [14:59:05] * Danny_B changed the filter [14:59:15] to match new length of 14 chars [14:59:17] Its still a little early in the morning for me [14:59:26] and took down the autocreation, let's see [15:00:02] sigh, it starts to vary the pattern [15:01:15] ok, modified the filter again [15:02:06] Ok, looking at the captcha log, I'm seeing lots of failures too as well as successes [15:02:23] so I guess they are bruteforcing the captcha [15:02:29] so can we turn off the account creation on mwo to see if the bot will stop or move to other wiki? [15:04:21] bawolff: well, if we have limited dictionary of words available for captcha then it is not so hard to bruteforce it [15:04:48] Not really bruteforcing, more like taking 5ish guesses [15:05:37] So maybe we should limit how many guesses each IP is allowed for a captcha [15:05:48] although i don't know if they are using a different IP for every guess [15:06:10] well so what happened to my suggestion to block this spambot already on network level? is that difficult? [15:07:01] We don't neccesarily have a list of IPs its coming from. Each request seems to mostly come from a different IP [15:07:45] alas, the confirmedit log doesn't include ips, so i don't know if captcha guesses are all coming from different IPs [15:08:39] so what are the possibilities? [15:08:58] 1. have junked log of af/gaf [15:09:08] 2. turn off account creation ond mwo [15:09:16] 3 deploy different captcha [15:09:29] 4. limit somehow traffic on network level [15:09:39] fun when the power goes off [15:24:13] I am hoping that silence is a good thing [15:25:22] sDrewth: well the filter filters, but that's not sustainable and scalable solution [15:26:18] aside of flooding of logs (speaking of - could we after the attack is over, simply clear the log?) [15:28:05] !log bawolff@deploy1001 Synchronized private/PrivateSettings.php: Attempt to adjust captcha settings for T212667 (duration: 00m 46s) [15:28:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [15:28:09] T212667: Emergency measure: Set wgAccountCreationThrottle => 2 - https://phabricator.wikimedia.org/T212667 [15:28:21] I tried to adjust the captcha settings to be more strict [15:28:28] I'm really not sure if that will do anything [15:28:34] But i thought it was worth a shot [15:29:25] 10Operations, 10monitoring: icinga doesn't log ampersand in notes_url links - https://phabricator.wikimedia.org/T212669 (10fgiunchedi) [15:33:17] let|s see [15:34:14] speed seems a bit slower [15:35:01] login in Error rates graph at https://grafana.wikimedia.org/d/000000004/authentication-metrics?orgId=1&from=now-6h&to=now&var-entrypoint=* shows steep drop [15:37:02] account creation is still higher than 10am by larger margin, though way less than 1400 [15:38:31] and the throttle seems to have significant more impact than filters, which is good [15:38:43] might be a coincidence [15:39:03] bawolff: did you set the captcha for mwo or for all servers? [15:39:09] might be, though the times align with actions [15:39:10] all servers [15:39:45] well, my irc client no longer chokes, so the filter hit is obviously slower [15:40:54] bawolff: you can do something on network level or not? i mean if you have the rights / access [15:41:07] Danny_B, grafana is showing 18-28 AF per minute [15:41:15] I'd need someone from operations to do something at the network level [15:41:20] 50+ for throttle [15:41:21] I can do things at the mediawiki level [15:41:41] The captcha thing so far doesn't seem to do much, the only hit for it thus far is when i was testing it myself [15:42:19] Danny_B: So for example, I could disable account creation given a list of IPs (or ranges of IPs) [15:43:03] Bigger issue, is there is no obvious pattern to the IPs being used [15:43:21] It looks like its a quite large botnet [15:47:00] yip. pee. [15:52:41] okay, it being 3am, and taking daughter out at 9am, time to head to bed [15:57:31] it would be helpful if we could log ip's of unsuccessful account creations. what legal issues it can have? [15:59:10] Its probably ok. Its not that different from other things we log [16:03:32] then it could help to do the mass block [16:05:53] i would actually incline to have that filter we use now automatically blocking those ip's for account creation for some period of time with message that you can write an email to if you are affected with false positive [16:07:26] what consequences it would have? as far as i can see now in the log, there are no false positives for the long time [16:08:39] so then the number of log lines would slowly decrease because the number of ip's used by the botnet which are available would decrease too by that blocking [16:08:56] aside of, we would have a nice log of those ips [16:22:17] PROBLEM - Citoid LVS eqiad on citoid.svc.eqiad.wmnet is CRITICAL: /api (Scrapes sample page) timed out before a response was received [16:23:23] RECOVERY - Citoid LVS eqiad on citoid.svc.eqiad.wmnet is OK: All endpoints are healthy [16:39:22] so i turned on the blocking, let's see... at least we'll have some list of affected ip's [16:40:06] paradoxically so far just two ip's [16:47:24] hmm, seems we should have done that earlier and it would have been quiet already... [18:02:04] !log bawolff@deploy1001 Synchronized private/PrivateSettings.php: T212667 - adjust account creation (duration: 00m 47s) [18:02:07] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:02:08] T212667: Emergency measure: Set wgAccountCreationThrottle => 2 - https://phabricator.wikimedia.org/T212667 [18:05:15] (03PS1) 10MarcoAurelio: Temporary remove AbuseFilter autoshutoff for mediawikiwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/481528 (https://phabricator.wikimedia.org/T212667) [18:07:05] (03CR) 10Rxy: [C: 03+1] "LGTM" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/481528 (https://phabricator.wikimedia.org/T212667) (owner: 10MarcoAurelio) [18:08:11] (03CR) 10Brian Wolff: [C: 03+2] Temporary remove AbuseFilter autoshutoff for mediawikiwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/481528 (https://phabricator.wikimedia.org/T212667) (owner: 10MarcoAurelio) [18:09:16] (03Merged) 10jenkins-bot: Temporary remove AbuseFilter autoshutoff for mediawikiwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/481528 (https://phabricator.wikimedia.org/T212667) (owner: 10MarcoAurelio) [18:14:29] PROBLEM - HHVM jobrunner on mw1309 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 473 bytes in 9.427 second response time [18:14:50] !log bawolff@deploy1001 Synchronized wmf-config/InitialiseSettings.php: 97446843a27 T212667 - Temp increase abusefilter emergency cutoff on mw.org to deal with spam attack (duration: 00m 46s) [18:14:53] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:14:54] T212667: Emergency measure: Set wgAccountCreationThrottle => 2 - https://phabricator.wikimedia.org/T212667 [18:15:33] RECOVERY - HHVM jobrunner on mw1309 is OK: HTTP OK: HTTP/1.1 200 OK - 206 bytes in 0.002 second response time [18:20:52] (03CR) 10jenkins-bot: Temporary remove AbuseFilter autoshutoff for mediawikiwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/481528 (https://phabricator.wikimedia.org/T212667) (owner: 10MarcoAurelio) [18:22:33] (03PS1) 10MarcoAurelio: Revert "Temporary remove AbuseFilter autoshutoff for mediawikiwiki" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/481529 [19:36:45] (03PS3) 10Reedy: langlist: Add hyw [dns] - 10https://gerrit.wikimedia.org/r/481335 (https://phabricator.wikimedia.org/T212597) (owner: 10MarcoAurelio) [19:39:17] PROBLEM - pdfrender on scb1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:40:31] RECOVERY - pdfrender on scb1004 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 8.867 second response time [19:45:25] PROBLEM - pdfrender on scb1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:16:09] PROBLEM - Citoid LVS codfw on citoid.svc.codfw.wmnet is CRITICAL: /api (Ensure Zotero is working) timed out before a response was received [20:17:17] RECOVERY - Citoid LVS codfw on citoid.svc.codfw.wmnet is OK: All endpoints are healthy [20:29:21] PROBLEM - HTTP availability for Nginx -SSL terminators- at eqiad on icinga1001 is CRITICAL: cluster=cache_text site=eqiad https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4fullscreenrefresh=1morgId=1 [20:31:47] RECOVERY - HTTP availability for Nginx -SSL terminators- at eqiad on icinga1001 is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4fullscreenrefresh=1morgId=1 [20:54:11] PROBLEM - Citoid LVS codfw on citoid.svc.codfw.wmnet is CRITICAL: /api (Scrapes sample page) timed out before a response was received [20:55:19] RECOVERY - Citoid LVS codfw on citoid.svc.codfw.wmnet is OK: All endpoints are healthy [21:03:36] (03PS1) 10Fomafix: Add 'lzh' as alias for 'zh-classical' [dns] - 10https://gerrit.wikimedia.org/r/481532 (https://phabricator.wikimedia.org/T167513) [21:03:59] (03PS1) 10Fomafix: Add redirects from lzh to zh-classical [puppet] - 10https://gerrit.wikimedia.org/r/481533 (https://phabricator.wikimedia.org/T167513) [21:45:42] {{LOCALTIMEOFDAYGREETING}} [21:46:06] Ops, am I right to turn off that username creation abusefilter? [21:46:19] whatever it was has stopped [21:46:33] <- just shaking out morning cobwebs [22:05:54] 10Operations, 10Wikimedia-Mailing-lists: Update Wikimedia logo on Mailman web pages from colored version to black and white version - https://phabricator.wikimedia.org/T212674 (10Aklapper) p:05Triage→03Lowest [22:11:03] RECOVERY - pdfrender on scb1004 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 3.241 second response time [22:14:47] PROBLEM - pdfrender on scb1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:17:05] RECOVERY - pdfrender on scb1004 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 1.361 second response time [22:20:53] PROBLEM - pdfrender on scb1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [22:58:49] RECOVERY - pdfrender on scb1004 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 3.463 second response time [23:02:37] PROBLEM - pdfrender on scb1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds