[00:35:48] PROBLEM - Host maps-test2003 is DOWN: PING CRITICAL - Packet loss = 100% [00:48:38] PROBLEM - Memory correctable errors -EDAC- on cp1053 is CRITICAL: 223 ge 4 https://grafana.wikimedia.org/dashboard/db/host-overview?orgId=1&var-server=cp1053&var-datasource=eqiad%2520prometheus%252Fops [01:02:56] 10Operations, 10Analytics, 10CheckUser, 10Gamepress, and 10 others: 3aaaaaaaaa - https://phabricator.wikimedia.org/T198152 (10Vvjjkkii) p:05Normal>03High a:05JAllemandou>03None [01:03:05] 10Operations, 10CheckUser, 10Discovery, 10Gamepress, and 12 others: 5daaaaaaaa - https://phabricator.wikimedia.org/T198042 (10Vvjjkkii) p:05Triage>03High [01:03:27] 10Operations, 10ops-codfw, 10CheckUser, 10Gamepress, and 10 others: 6daaaaaaaa - https://phabricator.wikimedia.org/T198041 (10Vvjjkkii) p:05Low>03High [01:03:30] 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install authdns2001.wikimedia.org - https://phabricator.wikimedia.org/T196664 (10Vvjjkkii) [01:03:35] uh [01:03:37] spam [01:03:59] 10Operations, 10ops-codfw, 10CheckUser, 10Gamepress, and 9 others: zdaaaaaaaa - https://phabricator.wikimedia.org/T198048 (10Vvjjkkii) p:05Normal>03High a:05Papaul>03None [01:04:03] any admins around? [01:04:04] Reedy ^^ [01:04:18] PROBLEM - etcd request latencies on neon is CRITICAL: instance=10.64.0.40:6443 operation=compareAndSwap https://grafana.wikimedia.org/dashboard/db/kubernetes-api [01:04:19] PROBLEM - etcd request latencies on argon is CRITICAL: instance=10.64.32.133:6443 operation=compareAndSwap https://grafana.wikimedia.org/dashboard/db/kubernetes-api [01:04:28] PROBLEM - Request latencies on argon is CRITICAL: instance=10.64.32.133:6443 verb=PATCH https://grafana.wikimedia.org/dashboard/db/kubernetes-api [01:04:56] 10Operations, 10ops-eqiad, 10CheckUser, 10Gamepress, and 9 others: yaaaaaaaaa - https://phabricator.wikimedia.org/T198157 (10Vvjjkkii) 05Invalid>03Open p:05Triage>03High [01:05:19] PROBLEM - Request latencies on neon is CRITICAL: instance=10.64.0.40:6443 verb=PUT https://grafana.wikimedia.org/dashboard/db/kubernetes-api [01:06:26] twentyafterfour ^^ [01:08:39] Reedy [01:08:42] legoktm [01:09:04] greg-g [01:09:12] chasemp [01:11:30] 10Operations, 10CheckUser, 10Discovery, 10Gamepress, and 13 others: sdaaaaaaaa - https://phabricator.wikimedia.org/T198055 (10Vvjjkkii) 05Resolved>03Open p:05Normal>03High a:05Gehel>03None [01:12:06] 10Operations, 10CheckUser, 10Discovery-Search, 10Gamepress, and 10 others: maaaaaaaaa - https://phabricator.wikimedia.org/T198169 (10Vvjjkkii) p:05Triage>03High [01:12:08] 10Operations, 10CheckUser, 10Gamepress, 10Hashtags, and 8 others: pdaaaaaaaa - https://phabricator.wikimedia.org/T198058 (10Vvjjkkii) p:05Triage>03High a:05MoritzMuehlenhoff>03None [01:12:11] 10Operations, 10CheckUser, 10Discovery, 10Gamepress, and 13 others: wdaaaaaaaa - https://phabricator.wikimedia.org/T198051 (10Vvjjkkii) p:05Normal>03High [01:13:48] 10Operations, 10Electron-PDFs, 10Proton, 10Readers-Web-Backlog, and 4 others: New service request: chromium-render/deploy - https://phabricator.wikimedia.org/T186748 (10Vvjjkkii) [01:14:03] 10Operations, 10Ops-Access-Reviews, 10Research, 10Research-collaborations, and 3 others: Request access to data for Wikimedia Donation Patterns research - https://phabricator.wikimedia.org/T188945 (10Vvjjkkii) [01:14:14] 10Operations, 10ops-codfw, 10CheckUser, 10Gamepress, and 10 others: glbaaaaaaa - https://phabricator.wikimedia.org/T196483 (10Vvjjkkii) [01:14:17] 10Operations, 10Analytics, 10Analytics-Kanban, 10CheckUser, and 11 others: rcaaaaaaaa - https://phabricator.wikimedia.org/T198092 (10Vvjjkkii) a:05elukey>03None [01:14:37] 10Operations, 10CheckUser, 10Gamepress, 10Hashtags, and 13 others: 8haaaaaaaa - https://phabricator.wikimedia.org/T197895 (10Vvjjkkii) 05Resolved>03Open p:05Normal>03High a:05herron>03None [01:15:26] 10Operations, 10CheckUser, 10Gamepress, 10Hashtags, and 10 others: 5iaaaaaaaa - https://phabricator.wikimedia.org/T197862 (10Vvjjkkii) p:05Low>03High [01:18:54] 10Operations, 10CheckUser, 10Gamepress, 10Hashtags, and 9 others: rmaaaaaaaa - https://phabricator.wikimedia.org/T197732 (10Vvjjkkii) 05Resolved>03Open p:05Normal>03High a:05herron>03None [01:18:56] 10Operations, 10CheckUser, 10Gamepress, 10Hashtags, and 9 others: ydaaaaaaaa - https://phabricator.wikimedia.org/T198049 (10Vvjjkkii) p:05Triage>03High [01:19:10] 10Operations, 10CheckUser, 10Gamepress, 10Hashtags, and 9 others: 6maaaaaaaa - https://phabricator.wikimedia.org/T197717 (10Vvjjkkii) 05Invalid>03Open p:05Triage>03High [01:19:29] 10Operations, 10CheckUser, 10DNS, 10Gamepress, and 10 others: 6kbaaaaaaa - https://phabricator.wikimedia.org/T196493 (10Vvjjkkii) [01:20:01] 10Operations, 10ops-codfw, 10CheckUser, 10DNS, and 12 others: qnaaaaaaaa - https://phabricator.wikimedia.org/T197697 (10Vvjjkkii) 05Resolved>03Open p:05Normal>03High a:05ayounsi>03None [01:20:12] 10Operations, 10ops-eqiad, 10CheckUser, 10Gamepress, and 9 others: hnaaaaaaaa - https://phabricator.wikimedia.org/T197706 (10Vvjjkkii) 05Resolved>03Open p:05Triage>03High a:05Marostegui>03None [01:21:09] 10Operations, 10Electron-PDFs, 10Proton, 10Readers-Web-Backlog, and 4 others: New service request: chromium-render/deploy - https://phabricator.wikimedia.org/T186748 (10Vvjjkkii) [01:21:35] 10Operations, 10ops-eqiad, 10Analytics, 10CheckUser, and 11 others: gnaaaaaaaa - https://phabricator.wikimedia.org/T197707 (10Vvjjkkii) 05Resolved>03Open a:05Cmjohnson>03None [01:22:24] 10Operations, 10ops-eqiad, 10CheckUser, 10Gamepress, and 9 others: lpaaaaaaaa - https://phabricator.wikimedia.org/T197630 (10Vvjjkkii) p:05Normal>03High [01:23:25] 10Operations, 10ops-codfw, 10CheckUser, 10Gamepress, and 9 others: 9paaaaaaaa - https://phabricator.wikimedia.org/T197606 (10Vvjjkkii) 05duplicate>03Open p:05Normal>03High [01:24:04] 10Operations, 10TemplateStyles, 10Traffic, 10Wikimedia-Extension-setup, and 4 others: Deploy TemplateStyles to WMF production - https://phabricator.wikimedia.org/T133410 (10Vvjjkkii) [01:24:25] 10Operations, 10CheckUser, 10Gamepress, 10Hashtags, and 9 others: rpaaaaaaaa - https://phabricator.wikimedia.org/T197624 (10Vvjjkkii) p:05Normal>03High [01:24:45] 10Operations, 10CheckUser, 10Gamepress, 10Hashtags, and 8 others: fraaaaaaaa - https://phabricator.wikimedia.org/T197564 (10Vvjjkkii) p:05Low>03High a:05Joe>03None [01:24:53] yeah I'm gonna need to do something about wikibugs [01:25:09] don't want freenode to ban it [01:34:36] Disabled mxn [01:36:11] mxn? [01:42:11] Hi, can someone please reverse vandal edits in phabricator https://phabricator.wikimedia.org/T198040 [01:42:17] and block the wanker [01:43:37] sDrewth, blocking was already handled hence the little dot next to their name [01:43:49] reversing edits on this scale is difficult [01:44:00] okay, thx Krenair, that I didn't know [01:44:26] no revert button for admins :-( [01:44:51] sDrewth, yeah. ever met phabricator upstream? [01:45:30] I will revert it manually [01:49:11] ouch, they did run rampant [01:49:20] yeah [01:50:05] I have fixed one anyway [01:51:16] thx Krenair, hope you are well. now I will shuffle away [02:05:03] phab could definitely use a "revert all" button [02:07:18] RECOVERY - Request latencies on neon is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-api [02:07:19] RECOVERY - etcd request latencies on neon is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-api [02:07:28] RECOVERY - etcd request latencies on argon is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-api [02:07:38] RECOVERY - Request latencies on argon is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/kubernetes-api [02:08:16] Lith, it could, but careful what you wish for when dealing with phabricator upstream [02:08:35] they might make the revert button itself accessible to vandals [02:11:03] lol [02:11:13] tho it looks like we already have a task going about it [02:11:14] https://secure.phabricator.com/T11254 [02:11:30] also https://secure.phabricator.com/T10215 [02:12:17] 10Operations, 10CheckUser, 10Gamepress, 10Hashtags, and 11 others: wecaaaaaaa - https://phabricator.wikimedia.org/T195423 (10Vvjjkkii) p:05Normal>03High [02:12:47] Great, another one. [02:13:12] Nevermind, same guy. Just not cleaned up yet. [02:14:48] 10Operations, 10CheckUser, 10Gamepress, 10Hashtags, and 11 others: rfcaaaaaaa - https://phabricator.wikimedia.org/T195392 (10Vvjjkkii) [02:16:52] yep, its going to take a bit to get cleaned up [02:17:22] Probably should get wikibugs turned off for a bit so it stops joinparting [02:18:23] looks like they put this on every task if someone wants to see if it means anything [02:18:29] ```26570726f6475636520796f757220627567207573696e67206120726563656e742076657273696f6e206f662074686520736f6674776172652c20746f2068652077696b6920636f6e74656e74206c616e67756167652e0a0a5468616e6b20796f752e0a546167730a436865636b557365720ad70a436f6e6e65637465642d4f70656e2d48657269746167652d42617463682d75706c6f61647320285241c42d4b4d425f315f323031372d3032290ad70a54616d696c2d53697465730ad70a47616d6570726573730ad70a48617368746167730ad70a4a4144450ad7 [02:18:29] 0a4b6172746f456469746f720ad70a4c616e67756167652d323031382d4170722d4a756e650ad70a4e65772d456469746f722d457870657269656e6365730ad70a4d61696c0ad70a5443422d5465616d0ad70a53756273637269626572730a4465736372697074696f6e20507265766965770a436f6e74656e77a6f6e652073657474696e6720696e20796f75722070726f66696c652c20636c69636b20746f207265636f6e63696c652e``` [02:19:43] Am I misreading or are they still active despite the account saying “disabled”? [02:19:55] The job queue is still catching up [02:20:03] ^ [02:20:12] They made so many changes so fast Phab slowed down [02:20:22] Ah :( [02:20:28] dammit people [02:20:33] stop asking the same thing [02:21:38] Lith, Hex -> ASCII decoder spits out a bunch of random text, then "zone setting in your profile, click to reconcile" [02:21:56] did you remove the line break? [02:22:08] Yeah, that's copied out of a task [02:22:11] ah [02:37:08] PROBLEM - Host labservices1001 is DOWN: CRITICAL - Host Unreachable (208.80.155.117) [02:37:29] what the **** [02:38:17] I'm pretty sure that box is labs authoritative dns [02:39:16] ; <<>> DiG 9.10.3-P4-Ubuntu <<>> wmflabs.org @labs-ns0.wikimedia.org [02:39:16] ;; global options: +cmd [02:39:16] ;; connection timed out; no servers could be reached [02:39:43] labs-ns1 is fine [02:42:08] https://grafana.wikimedia.org/dashboard/file/server-board.json?refresh=1m&orgId=1&var-server=labservices1001&var-network=eth0 [02:42:20] Last I checked those graphs aren't supposed to stop [02:43:03] well if a server goes off the network that'll cause the graphs to stop [02:43:17] True [02:43:18] likely needs a root to go in and look at it over serial [02:44:39] !log forcing reboot of labservices1001 via mgmt [02:44:48] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:47:29] PROBLEM - toolschecker: Start a job and verify on Trusty on checker.tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 504 Gateway Time-out - string OK not found on http://checker.tools.wmflabs.org:80/grid/start/trusty - 356 bytes in 60.007 second response time [02:47:38] RECOVERY - Host labservices1001 is UP: PING OK - Packet loss = 0%, RTA = 0.21 ms [02:51:39] RECOVERY - toolschecker: Start a job and verify on Trusty on checker.tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 166 bytes in 1.403 second response time [02:53:29] !log forcing reboot of labservices1001 via mgmt [02:53:30] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:54:39] !log forcing reboot of cp3033 via mgmt [02:54:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:55:45] ? [02:55:53] that's not a real opsen [03:13:36] !log forcing reboot of labservices1001 via mgmt !log forcing reboot of labservices1001 via mgmt !log forcing reboot of labservices1001 via mgmt !log forcing reboot of labservices1001 via mgmt !log forcing reboot of labservices1001 via mgmt !log forcing reboot of labservices1001 via mgmt !log forcing reboot of labservices1001 via mgmt !log forcing reboot of labservices1001 via mgmt !log forcing reboot of labservices1001 v [03:13:38] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:13:38] !log forcing reboot of labservices1001 via mgmt [03:13:39] !log forcing reboot of labservices1001 via mgmt [03:13:39] !log forcing reboot of labservices1001 via mgmt [03:13:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:13:40] !log forcing reboot of labservices1001 via mgmt [03:13:40] !log forcing reboot of labservices1001 via mgmt [03:13:40] !log forcing reboot of labservices1001 via mgmt [03:13:41] !log forcing reboot of labservices1001 via mgmt [03:13:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:13:41] !log forcing reboot of labservices1001 via mgmt [03:13:42] !log forcing reboot of labservices1001 via mgmt [03:13:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:13:43] AlexZ: ^ [03:13:44] !log forcing reboot of labservices1001 via mgmt [03:13:45] !log forcing reboot of labservices1001 via mgmt [03:13:46] !log forcing reboot of labservices1001 via mgmt [03:13:46] !log forcing reboot of labservices1001 via mgmt [03:13:46] !log forcing reboot of labservices1001 via mgmt [03:13:47] !log forcing reboot of labservices1001 via mgmt [03:13:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:13:47] !log forcing reboot of labservices1001 via mgmt [03:13:48] !log forcing reboot of labservices1001 via mgmt [03:13:49] !log forcing reboot of labservices1001 via mgmt [03:13:49] !log forcing reboot of labservices1001 via mgmt [03:13:49] !log forcing reboot of labservices1001 via mgmt [03:13:50] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:13:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:13:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:14:01] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:14:03] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:14:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:14:06] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:14:06] bye ^^ [03:14:08] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:14:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:14:13] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:14:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:14:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:14:19] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:14:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:14:22] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:14:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:14:34] why no whitelist [03:14:50] ffs [03:15:08] At least they've got a ratelimit on this one [03:15:21] AlexZ [03:15:40] Lith, luckily that bot isn't too dangerous [03:15:51] it prepends stuff to a page on wikitech [03:15:57] can just revert [03:16:09] It still should probably be looking for a WM cloak [03:16:09] oh and it puts stuff on twitter, which I doubt anyone actually reads [03:16:12] and inserts into a elastic index [03:16:23] AntiComposite, some of the roots have cloaks from other projects [03:16:52] WMF cloak + manual whitelist for people that don't have a WMF cloak? [03:17:23] I know it was thought of before in another spam attack, not sure why they decided not too (I assume just dind't want to deal with it) [03:18:28] Yeah I know it got brought up previously.. as being a bad idea [03:18:32] I just filed a task for it [03:18:53] for some reason wikibugs hasn't come back [03:21:00] 10Operations, 10Phabricator, 10Release-Engineering-Team: Spam on phabricator - https://phabricator.wikimedia.org/T198547 (10JJMC89) @bd808 disabled the account [03:21:03] Krenair: but manually whitelisting those users or affiliated projects could solve that issue or adding a command to white list then and granting certain users to add others to the white list would probably work [03:21:06] 10Operations, 10Phabricator, 10Release-Engineering-Team: Spam on phabricator - https://phabricator.wikimedia.org/T198547 (10Paladox) 05Open>03Resolved a:03bd808 Thanks @bd808! [03:22:07] yeah [03:22:56] ugh. I had tried to fix spamming that in stashbot, but apparently I have it wrong [03:27:56] 10Operations, 10ops-codfw, 10DC-Ops: Replace disk on wasat - https://phabricator.wikimedia.org/T197562 (10JJMC89) p:05High>03Normal [03:35:28] PROBLEM - puppet last run on analytics1053 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 4 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/usr/share/GeoIP/GeoIP2-City.mmdb.gz],File[/usr/share/GeoIP/GeoIP2-City.mmdb.test] [03:35:50] 10Operations, 10TemplateStyles, 10Traffic, 10Wikimedia-Extension-setup, and 4 others: Deploy TemplateStyles to WMF production - https://phabricator.wikimedia.org/T133410 (10JJMC89) [03:50:55] 10Operations, 10monitoring, 10Patch-For-Review, 10User-herron: Reduce false positive icinga alerts during host reimages - https://phabricator.wikimedia.org/T195423 (10AntiCompositeNumber) p:05High>03Normal [03:52:25] 10Operations, 10Traffic, 10Patch-For-Review: Create and deploy a centralized letsencrypt service - https://phabricator.wikimedia.org/T194962 (10Krenair) p:05High>03Normal a:03Krenair [03:52:49] Cleanup's coming though now [03:52:52] 10Operations: Update prometheus-varnish-exporter on debian to 1.4 - https://phabricator.wikimedia.org/T195252 (10TerraCodes) p:05High>03Normal [03:53:52] any phabricator admins in here? [03:54:01] Let me guess, there's spam? [03:54:12] Coming from Vvjjkkii? [03:54:17] yea #wikimedia-dev [03:54:27] yes, Vvjjkkii [03:54:44] not so much spam, just vandalizing a bunch of tickets [03:54:50] They've been disabled already, job queue is still catching up, cleanup is manual. [03:55:00] thanks [03:55:10] Complaints department is upstream. [03:55:35] I wasn't so much "complaining" - just wasn't sure where to go to report [03:55:56] 10Operations, 10Traffic, 10Patch-For-Review: Create and deploy a centralized letsencrypt service - https://phabricator.wikimedia.org/T194962 (10Krenair) Per @Vgutierrez I have started developing this in a separate repository, operations/software/certcentral.git [03:56:47] xaosflux: thanks for taking the time to point it out. [03:58:01] gnight [03:58:02] part [04:00:18] 10Operations: test - https://phabricator.wikimedia.org/T198548 (10Krenair) [04:00:58] RECOVERY - puppet last run on analytics1053 is OK: OK: Puppet is currently enabled, last run 15 seconds ago with 0 failures [04:06:23] 10Operations, 10Analytics, 10EventBus, 10GlobalRename, and 5 others: Global renames get stuck at metawiki - https://phabricator.wikimedia.org/T193254 (101997kB) 05Open>03Resolved a:03mobrovac [04:08:39] PROBLEM - exim queue on mx1001 is CRITICAL: CRITICAL: 9437 mails in exim queue. [04:10:56] 10Operations, 10ops-eqiad, 10cloud-services-team: Labservices1001 crashed - https://phabricator.wikimedia.org/T196252 (10Krenair) p:05High>03Normal [04:11:12] 10Operations, 10ops-eqiad, 10cloud-services-team: Labservices1001 crashed - https://phabricator.wikimedia.org/T196252 (10Krenair) This just happened again [04:11:19] 10Operations: test - https://phabricator.wikimedia.org/T198548 (10Krenair) 05Open>03Invalid yeah clearly this didn't work [04:12:53] 10Operations, 10Phabricator, 10Release-Engineering-Team: Spam on phabricator - https://phabricator.wikimedia.org/T198547 (10Wong128hk) [04:14:00] 10Operations, 10Maps-Sprint, 10Patch-For-Review: reimage maps-test2004 to stretch and cassandra 2.2 - https://phabricator.wikimedia.org/T195741 (10WhitePhosphorus) 05Open>03Resolved p:05High>03Normal a:03Gehel [04:14:36] 10Operations, 10Cassandra, 10Discovery, 10Maps, 10Patch-For-Review: cassandra 2.2.6-wmf4 is not compatible with python 2.7.13 (debian stretch) - https://phabricator.wikimedia.org/T196044 (10WhitePhosphorus) 05Open>03Resolved p:05High>03Normal a:03Gehel [04:15:11] 10Operations, 10ops-eqiad, 10cloud-services-team: Labservices1001 crashed - https://phabricator.wikimedia.org/T196252 (10Andrew) ``` Jul 1 02:32:51 labservices1001 pdns[2129]: Domain 'deployment-prep.wmflabs.org' is fresh (not presigned, no RRSIG check) Jul 1 02:32:51 labservices1001 pdns[2129]: Domain 'de... [04:16:09] 10Operations, 10User-fgiunchedi: mw1230 sdb "Raw_Read_Error_Rate" SMART - https://phabricator.wikimedia.org/T194036 (10Wong128hk) p:05High>03Normal [04:16:20] 10Operations, 10User-fgiunchedi: mw1230 sdb "Raw_Read_Error_Rate" SMART - https://phabricator.wikimedia.org/T194036 (10Wong128hk) [04:23:54] 10Operations, 10Analytics, 10Traffic: Size of headers processed by varnish? - https://phabricator.wikimedia.org/T198152 (10JJMC89) a:03JAllemandou [04:24:02] 10Operations, 10Analytics, 10Traffic: Size of headers processed by varnish? - https://phabricator.wikimedia.org/T198152 (10JJMC89) p:05High>03Normal [04:26:19] 10Operations, 10Discovery, 10Wikidata, 10Wikidata-Query-Service, 10Discovery-Wikidata-Query-Service-Sprint: WDQS timeout on the public eqiad cluster - https://phabricator.wikimedia.org/T198042 (10JJMC89) 05Open>03Resolved p:05High>03Triage [04:26:44] 10Operations, 10ops-codfw, 10monitoring: graphite2001 crashed - https://phabricator.wikimedia.org/T198041 (10JJMC89) p:05High>03Low [04:26:51] 10Operations, 10ops-codfw, 10netops: Swith port information for authdns2001 - https://phabricator.wikimedia.org/T198126 (10JJMC89) 05Open>03Resolved p:05High>03Normal a:03Papaul [04:26:57] 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install authdns2001.wikimedia.org - https://phabricator.wikimedia.org/T196664 (10JJMC89) [04:27:20] 10Operations, 10ops-codfw, 10Patch-For-Review: rack/setup/install authdns2001.wikimedia.org - https://phabricator.wikimedia.org/T196664 (10JJMC89) p:05High>03Normal a:03Papaul [04:27:50] 10Operations, 10ops-codfw: db2056: disk with predictive failure - https://phabricator.wikimedia.org/T198048 (10JJMC89) p:05High>03Normal a:03Papaul [04:29:31] 10Operations, 10ops-eqiad: Degraded RAID on db1054 - https://phabricator.wikimedia.org/T198157 (10JJMC89) 05Open>03Invalid p:05High>03Triage [04:33:04] is anyone else seeing all these phab tasks gettng changed to goobledgook? [04:33:14] yes, its been dealt with [04:33:19] great - thanks [04:33:26] phab just has a jobqueue that is catching up [04:34:06] oh - I can ignore then [04:55:24] p858snake|L: Dealt with? It needs a manual cleanup, I believe? [04:55:54] its been progressively done, but the emails are delayed due to the jobqueue [04:56:18] 10Operations, 10TemplateStyles, 10Traffic, 10Wikimedia-Extension-setup, and 4 others: Deploy TemplateStyles to WMF production - https://phabricator.wikimedia.org/T133410 (10AfroThundr3007730) [05:14:29] 10Operations, 10DBA, 10Patch-For-Review: Rack and setup db1116 - db1123 - https://phabricator.wikimedia.org/T191792 (10Marostegui) [05:17:47] 10Operations, 10TemplateStyles, 10Traffic, 10Wikimedia-Extension-setup, and 4 others: Deploy TemplateStyles to WMF production - https://phabricator.wikimedia.org/T133410 (10AfroThundr3007730) [05:52:00] 10Operations, 10MediaWiki-Platform-Team, 10Performance-Team, 10MW-1.27-release-notes, and 3 others: php-memcached 3.0 (PHP 7) incompatible with BagOStuff - https://phabricator.wikimedia.org/T196125 (10Joe) p:05High>03Normal a:03aaron [05:58:38] 10Operations, 10Puppet: puppetmaster puppet.conf refers to noexistent files - https://phabricator.wikimedia.org/T192848 (10Wong128hk) p:05High>03Triage [05:59:16] 10Operations, 10Puppet: puppetmaster puppet.conf refers to noexistent files - https://phabricator.wikimedia.org/T192848 (10Wong128hk) [05:59:20] 10Operations, 10Puppet, 10Patch-For-Review, 10User-Joe: puppetmaster hostcert and hostprivkey point to nonexistent files - https://phabricator.wikimedia.org/T179099 (10Wong128hk) [06:00:40] (03CR) 1020after4: [C: 031] "We've just seen more vandalism from this IP range" [puppet] - 10https://gerrit.wikimedia.org/r/440510 (owner: 10Aklapper) [06:05:43] <_joe_> twentyafterfour: should we merge that? [06:06:09] <_joe_> since people now get frustrated for not having patches merged which have no one with +2 on them as a reviewer... [06:08:12] _joe_: I think the frustration is that Andre has to deal with the vandalism and hasn't felt very supported [06:08:40] I'm working on tools to help but that's going to take some time [06:09:09] PROBLEM - exim queue on mx1001 is CRITICAL: CRITICAL: 25033 mails in exim queue. [06:09:26] <_joe_> twentyafterfour: of course it will take time [06:10:02] <_joe_> also I think we should just accept it's going to be approvals of new accounts for quite some time [06:10:31] should we be looking at blocking them from creating on wikitech before they can even get to phab? [06:10:41] _joe_: https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/441525/ might help as well [06:10:50] that adds connection rate limiting [06:11:06] which was inadvertantly removed due to changes upstream [06:11:18] <_joe_> oh sigh, yes [06:11:28] <_joe_> p858snake|L: maybe [06:11:31] why do we have our block list public? [06:11:37] 10Operations, 10Analytics, 10Analytics-Kanban, 10EventBus, and 2 others: Kafka API negotiation errors on kafka main brokers - https://phabricator.wikimedia.org/T193238 (10AfroThundr3007730) 05Open>03Resolved a:03Imarlier [06:11:38] <_joe_> twentyafterfour: let's do it [06:12:05] <_joe_> Lith: we could have it private, I don't think it grants a lot of protection FWIW [06:12:12] _joe_: that patch isn't well tested because I don't have a test environment, so if it merges we might have to roll back [06:12:14] <_joe_> a vpn goes 1 dollar/month right now [06:12:19] can we remove a lot of the rights from standard accounts, which is what we did in BZ days, they could create but didn't have much in the way of editing task settings [06:12:25] tru, [06:12:32] <_joe_> twentyafterfour: ok, lemme downtime phab though then [06:12:33] ? [06:12:54] <_joe_> p858snake|L: that would be great, but I don't think phab supports that [06:13:02] p858snake|L: that's been proposed and I'm looking into it [06:13:21] <_joe_> twentyafterfour: ok so, let's merge that patch and see how it goes? [06:13:29] _joe_: ok cool [06:14:39] (03PS5) 10Giuseppe Lavagetto: Fix phabricator rate limiting [puppet] - 10https://gerrit.wikimedia.org/r/441525 (https://phabricator.wikimedia.org/T197922) (owner: 1020after4) [06:15:48] 10Operations, 10Mail, 10monitoring, 10User-herron, 10Wikimedia-Incident: Improve outbound mail service alerting - https://phabricator.wikimedia.org/T197172 (10ArielGlenn) [06:15:59] (03CR) 10Giuseppe Lavagetto: [C: 032] Fix phabricator rate limiting [puppet] - 10https://gerrit.wikimedia.org/r/441525 (https://phabricator.wikimedia.org/T197922) (owner: 1020after4) [06:17:05] <_joe_> twentyafterfour: running puppet on phab1001 [06:17:19] <_joe_> let's see if phab keeps working :P [06:17:25] 10Operations, 10Mail, 10monitoring, 10User-herron, 10Wikimedia-Incident: Graph outbound mail volume on per-service or hostgroup level - https://phabricator.wikimedia.org/T197171 (10ArielGlenn) p:05High>03Normal [06:17:36] <_joe_> so there is an error, clearly [06:17:38] <_joe_> sigh [06:17:41] 10Operations, 10CheckUser, 10Gamepress, 10Hashtags, and 16 others: Multiple projects reporting Cannot access the database: No working replica DB server - https://phabricator.wikimedia.org/T195520 (101339861mzb) [06:19:01] 10Operations, 10CheckUser, 10Gamepress, 10Hashtags, and 16 others: Multiple projects reporting Cannot access the database: No working replica DB server - https://phabricator.wikimedia.org/T195520 (101339861mzb) [06:19:47] 10Operations, 10ops-eqiad, 10decommission, 10User-ArielGlenn: decommission snapshot1001 - https://phabricator.wikimedia.org/T197021 (10ArielGlenn) p:05High>03Normal [06:19:49] <_joe_> twentyafterfour: I did set the values you wanted by hand on phab1001 [06:19:57] <_joe_> I'm going to fix the pathc in the meanwhile [06:21:15] 10Operations, 10CheckUser, 10Gamepress, 10Hashtags, and 16 others: Multiple projects reporting Cannot access the database: No working replica DB server - https://phabricator.wikimedia.org/T195520 (101339861mzb) 05Open>03Resolved [06:22:25] 10Operations, 10Analytics, 10DC-Ops, 10procurement: Analytics hosts missing in Inventory/Refresh - https://phabricator.wikimedia.org/T196072 (10Community_Tech_bot) [06:22:28] 10Operations, 10Analytics, 10DC-Ops, 10procurement: Analytics hosts missing in Inventory/Refresh - https://phabricator.wikimedia.org/T196072 (10Community_Tech_bot) a:03elukey [06:22:31] 10Operations, 10Analytics, 10DC-Ops, 10procurement: Analytics hosts missing in Inventory/Refresh - https://phabricator.wikimedia.org/T196072 (10Community_Tech_bot) 05Open>03Resolved [06:22:34] 10Operations, 10Analytics, 10DC-Ops, 10procurement: Analytics hosts missing in Inventory/Refresh - https://phabricator.wikimedia.org/T196072 (10Community_Tech_bot) [06:22:37] 10Operations, 10Analytics, 10DC-Ops, 10procurement: Analytics hosts missing in Inventory/Refresh - https://phabricator.wikimedia.org/T196072 (10Community_Tech_bot) [06:22:40] 10Operations, 10Patch-For-Review, 10User-herron, 10Wikimedia-Incident: Add email queueing/failover to services currently using mail_smarthost[0] - https://phabricator.wikimedia.org/T196920 (10ArielGlenn) [06:23:42] _joe_: ok [06:24:04] hah [06:24:13] <_joe_> why is preamble included in phabricator::redirector??? [06:24:24] * _joe_ scratches his head [06:24:57] 10Operations, 10Mail, 10Phabricator, 10Release-Engineering-Team, and 3 others: Phabricator outbound email seems to have a SPOF of mx1001 - https://phabricator.wikimedia.org/T196916 (10ArielGlenn) [06:28:14] is it just me, or did wikibugs just join the channel despite already being in it? [06:28:58] Lith: wikibugs (tools.wiki@wikimedia/bot/puwikibugs) has quit (Excess Flood) [06:29:06] huh [06:29:19] _joe_: it's in redirector because that was the original use for the preamble [06:29:30] we only used it to redirect bugzilla urls [06:29:45] but now throttling goes there according to upstream tasks [06:30:09] they removed throttling from core and replaced it with a hook that you can configure with the preamble.php [06:30:28] because I don't see a leave https://usercontent.irccloud-cdn.com/file/DiImOrRR/image.png [06:31:48] (03PS1) 10Giuseppe Lavagetto: phabricator: fixup for If4fd73de [puppet] - 10https://gerrit.wikimedia.org/r/443300 [06:32:00] <_joe_> twentyafterfour: yeah we ought to move it [06:32:39] PROBLEM - puppet last run on cp1065 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/local/lib/nagios/plugins/check_strongswan] [06:32:46] Lith: Blame your IRC client... I see a leave between "ok" and "hah" [06:32:50] (03CR) 1020after4: [C: 031] "heh, well lets merge either this or the other one:" [puppet] - 10https://gerrit.wikimedia.org/r/443045 (owner: 10Alexandros Kosiaris) [06:33:41] huh, weird it didn't catch it [06:34:08] (03CR) 10Giuseppe Lavagetto: [C: 032] "https://puppet-compiler.wmflabs.org/compiler02/11623/phab1001.eqiad.wmnet/" [puppet] - 10https://gerrit.wikimedia.org/r/443300 (owner: 10Giuseppe Lavagetto) [06:34:16] (03CR) 1020after4: [C: 031] phabricator: Use the mysql native driver (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/443045 (owner: 10Alexandros Kosiaris) [06:34:39] (03CR) 10MusikAnimal: "this hit my rollback bot https://phabricator.wikimedia.org/p/Community_Tech_bot/" [puppet] - 10https://gerrit.wikimedia.org/r/441525 (https://phabricator.wikimedia.org/T197922) (owner: 1020after4) [06:35:50] _joe_: yeah that would be a good idea [06:36:15] <_joe_> twentyafterfour: done, now going for coffeee [06:36:19] <_joe_> ping me if needed again [06:39:14] twentyafterfour: do you know about how https://gerrit.wikimedia.org/r/c/operations/puppet/+/441525/ works? anyway you can whitelist my IP (I will share in private?) [06:39:29] ok thanks joe [06:40:28] musikanimal: ok but your IP will be public if I whitelist it in the code [06:40:57] hmm right, I guess I could put it on Toolforge [06:41:33] actually we can whitelist accounts [06:41:37] instead of ips [06:42:05] that would be fantastic, could you whitelist Community_Tech_bot and MusikAnimal [06:42:14] I can't access phab either :( [06:45:16] musikanimal: the rate limiting will expire after 5 minutes [06:45:40] Does your bot use conduit apis or is it directly accessing phabricator? [06:45:49] API [06:46:09] If we create a bot user that will only have access to APIs then it will bypass the rate limiting [06:46:37] sounds good to me [06:58:09] RECOVERY - puppet last run on cp1065 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [07:18:00] 10Operations, 10Mail, 10cloud-services-team, 10monitoring: Prometheus vs. CPU usage vs. hyperthreading - https://phabricator.wikimedia.org/T193272 (10Samwilson) p:05High>03Normal [07:27:32] twentyafterfour: Morn! FYI, there's another issue created by all this mentioned in https://phabricator.wikimedia.org/T198552#4362456 (cannot access some Phab project pages anymore) [07:39:59] @andre__ If I understood your message there correctly, you guys will revert all vandalism and we don't have to worry about it? Thanks for spending your Sunday morning with this crap, by the way [08:04:01] tim_WMDE, I haven't done anything so far; the thanks goes to many other people :) [08:05:41] 10Operations, 10Traffic, 10Goal, 10Patch-For-Review: Begin execution of non-forward-secret ciphers deprecation - https://phabricator.wikimedia.org/T192555 (10Vgutierrez) [08:24:01] twentyafterfour: i wonder if we can use projects as white lists too? [08:24:09] Ie the trusted project [08:32:58] RECOVERY - Router interfaces on cr2-ulsfo is OK: OK: host 198.35.26.193, interfaces up: 77, down: 0, dormant: 0, excluded: 0, unused: 0 [08:33:28] RECOVERY - Router interfaces on cr1-codfw is OK: OK: host 208.80.153.192, interfaces up: 126, down: 0, dormant: 0, excluded: 0, unused: 0 [08:38:13] 10Operations, 10Patch-For-Review: Merge one-line puppet fix - https://phabricator.wikimedia.org/T193660 (10CommunityTechBot) a:03RobH [08:38:15] 10Operations, 10Patch-For-Review: Merge one-line puppet fix - https://phabricator.wikimedia.org/T193660 (10CommunityTechBot) [08:38:17] 10Operations, 10Patch-For-Review: Merge one-line puppet fix - https://phabricator.wikimedia.org/T193660 (10CommunityTechBot) 05Open>03Resolved [08:38:19] 10Operations, 10Patch-For-Review: Merge one-line puppet fix - https://phabricator.wikimedia.org/T193660 (10CommunityTechBot) [08:38:23] 10Operations, 10Patch-For-Review: Merge one-line puppet fix - https://phabricator.wikimedia.org/T193660 (10CommunityTechBot) [08:39:48] PROBLEM - exim queue on mx1001 is CRITICAL: CRITICAL: 26657 mails in exim queue. [09:03:58] paladox: what's the usecase? [09:11:03] andre__: to prevent some trusted user to trigger the rate limiter :) [09:12:22] 10Operations, 10Wikimedia-Mailing-lists: delete "wmfproduct" list - https://phabricator.wikimedia.org/T193093 (10Mainframe98) 05Open>03Resolved p:05High>03Triage a:03Dzahn [09:38:58] PROBLEM - HHVM jobrunner on mw1299 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 473 bytes in 0.001 second response time [09:39:59] RECOVERY - HHVM jobrunner on mw1299 is OK: HTTP OK: HTTP/1.1 200 OK - 206 bytes in 0.002 second response time [09:52:05] Morning [09:52:09] I have a concern [09:52:19] I've appraently found a glitch in phabricator [09:52:34] Or at the very least a user that was meaning to edit something else [09:52:34] what would the glitch be? [09:52:42] https://phabricator.wikimedia.org/T197712 [09:52:58] Some editor's replaced an old ticket with rubbish [09:53:03] There are some others [09:53:23] https://phabricator.wikimedia.org/transactions/detail/PHID-XACT-TASK-3h2qzrffl2yltwv/ [09:53:36] https://phabricator.wikimedia.org/transactions/detail/PHID-XACT-TASK-c33kqiutkjvqm7v/ [09:53:46] https://phabricator.wikimedia.org/transactions/detail/PHID-XACT-TASK-pchyaohv4iton2k/ [09:54:07] <_joe_> ShakespeareFan00: that's vandalism [09:54:16] https://phabricator.wikimedia.org/transactions/detail/PHID-XACT-TASK-pchyaohv4iton2k/ [09:54:28] _joe_: Can it it be undone and the user concerned warned? [09:54:30] <_joe_> ShakespeareFan00: revert whichever ticket you stumble upon if you have time to its natural state [09:54:48] I'm not sure how [09:54:51] BTW - https://phabricator.wikimedia.org/p/Vvjjkkii/ [09:54:55] Is rather informative [09:54:56] <_joe_> ShakespeareFan00: the user has vandalized thousands of tickets, and as far as i understand it can only be done manually [09:55:10] <_joe_> so you go and reenter the old content in the ticket [09:55:18] Tiresome [09:55:31] Filing a ticket about this on phabricator ;) [09:55:31] <_joe_> indeed [09:55:38] <_joe_> there are several I think [09:55:45] <_joe_> andre__: might know better [09:56:52] ShakespeareFan00: its known [09:56:57] no need to create a dup [10:00:36] I thought a bot was reverting the shit (see wikitech) [10:01:34] Thanks [10:01:40] Can someone link me to the ticket on this? [10:02:03] (Apologies - Warm weather and chep router hubs don't make for a reliable connection) [10:02:18] revi: yes it is [10:02:24] but there is a lot [10:03:09] yeah gonna wait [10:03:19] Okay [10:03:34] So, Apologies for an unreliable connection [10:03:52] Someone was mentioning the phabricator vandal had already been reported? [10:04:02] Which ticket ? [10:04:08] (So I don't duplicate) [10:04:10] we know who they are [10:04:25] I don’t know which ticket it is [10:04:52] ShakespeareFan00: https://phabricator.wikimedia.org/T198552 [10:09:00] revi: Thanks [10:09:33] Obviously you can't share specfics, but I assume a report will be passed to WMF legal as well? [10:10:17] The vandalisim of more than a few tickets isn't in the same leauge as the sort of casual wiki-test edits that get handled [10:14:47] https://phabricator.wikimedia.org/T197944 this one is still bad [10:16:55] massive vandalism from https://phabricator.wikimedia.org/p/Vvjjkkii/ [10:17:56] yannf: https://phabricator.wikimedia.org/T198552 ? [10:20:02] yannf: it is being worked on [10:20:39] ok, thanks [10:21:51] A 48 to 72 cleanup effort per something in the ticket :( [10:22:03] I hope the vandal concerned goes to jail. [10:26:00] goes to jail on what charges? [10:31:46] Lith: "Disruption of an electronic communications service" [10:47:31] 10Operations, 10Proton, 10Services (doing): Increase the CPU count for proton[12]00[12] - https://phabricator.wikimedia.org/T197862 (10mobrovac) p:05High>03Low [10:59:03] I reverted all the ones I am subscribed [11:14:43] 19:31:47 Lith: "Disruption of an electronic communications service" [11:14:51] well I think it varies by jurisdictions [11:14:58] oh sorry li-th unintended ping [11:18:46] lol its fine [11:23:18] removed the point value for this task. -> what do I do? what was the old state? [11:28:01] maybe leave it and someone who knows the original value will set it? [11:29:00] if you scroll up it should be there [11:32:21] ah ok, but then how to change the point value? [11:32:52] Edit Task should have it [11:32:54] when you go to edit the task, theres a field at the bottom of the page [11:33:06] is it "Story Points [11:33:06] "? [11:33:38] Lith, ^ [11:34:20] yep [11:37:30] ok, thanks [11:46:19] np [12:14:07] someone with wikitech admin? [12:15:07] https://phabricator.wikimedia.org/T193328 [12:15:08] Vvjjkkii reopened this task as Open. [12:15:16] https://wikitech.wikimedia.org/wiki/Special:Contributions/Liza_Veniza need a block [12:15:22] but I can't find the previous state [12:15:25] akosiaris [12:15:29] Lith, ^ [12:15:40] arturo [12:15:52] bd808 [12:15:59] and I'm sure you'll want to rangeblock that IP, since that is known VPS abuser [12:16:05] (the IP behind the account) [12:16:09] How can you do that while disabled? [12:16:21] Or did it happen before? [12:16:24] :P [12:16:27]