[00:07:06] robh, what I mean is I think you can set the email field to whatever you like can't you? [00:07:35] Krenair: afaik yes [00:07:43] On what? [00:07:51] ldap/wikitech [00:08:01] If its valid i believe so [00:08:07] but if person Y asks for person X to have wmf flag [00:08:17] and person x has a wikitech account tied to their personal address [00:08:26] then i likely need to ask person X to confirm its their account [00:08:46] in this case, person X used a @wikiemdia.org account, and the wikitech account was valid and working, so its legit [00:09:13] otherwise i'd ask that person to confirm that was their account, and the one they want to use, and i would do the work. [00:09:52] I don't particularly agree with that part, but there are accounts on wikitech that are NOT tied to @wikimedia.org accounts that have the wmf flag [00:09:55] so its not my call ;] [03:27:26] PROBLEM - MariaDB Slave Lag: s1 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 660.24 seconds [03:40:36] PROBLEM - Host cp4024 is DOWN: PING CRITICAL - Packet loss = 100% [03:45:56] PROBLEM - IPsec on cp2024 is CRITICAL: Strongswan CRITICAL - ok: 80 not-conn: cp4024_v4, cp4024_v6 [03:46:05] PROBLEM - IPsec on cp1063 is CRITICAL: Strongswan CRITICAL - ok: 66 not-conn: cp4024_v4, cp4024_v6 [03:46:06] PROBLEM - IPsec on cp1062 is CRITICAL: Strongswan CRITICAL - ok: 66 not-conn: cp4024_v4, cp4024_v6 [03:46:15] PROBLEM - IPsec on cp2008 is CRITICAL: Strongswan CRITICAL - ok: 80 not-conn: cp4024_v4, cp4024_v6 [03:46:16] PROBLEM - IPsec on cp2017 is CRITICAL: Strongswan CRITICAL - ok: 80 not-conn: cp4024_v4, cp4024_v6 [03:46:16] PROBLEM - IPsec on cp2014 is CRITICAL: Strongswan CRITICAL - ok: 80 not-conn: cp4024_v4, cp4024_v6 [03:46:25] PROBLEM - IPsec on cp2022 is CRITICAL: Strongswan CRITICAL - ok: 80 not-conn: cp4024_v4, cp4024_v6 [03:46:26] PROBLEM - IPsec on cp1073 is CRITICAL: Strongswan CRITICAL - ok: 66 not-conn: cp4024_v4, cp4024_v6 [03:46:35] PROBLEM - IPsec on cp1071 is CRITICAL: Strongswan CRITICAL - ok: 66 not-conn: cp4024_v4, cp4024_v6 [03:46:35] PROBLEM - IPsec on cp1049 is CRITICAL: Strongswan CRITICAL - ok: 66 not-conn: cp4024_v4, cp4024_v6 [03:46:36] PROBLEM - IPsec on cp2011 is CRITICAL: Strongswan CRITICAL - ok: 80 not-conn: cp4024_v4, cp4024_v6 [03:46:36] PROBLEM - IPsec on cp1099 is CRITICAL: Strongswan CRITICAL - ok: 66 not-conn: cp4024_v4, cp4024_v6 [03:46:55] PROBLEM - IPsec on cp1050 is CRITICAL: Strongswan CRITICAL - ok: 66 not-conn: cp4024_v4, cp4024_v6 [03:46:55] PROBLEM - IPsec on cp2020 is CRITICAL: Strongswan CRITICAL - ok: 80 not-conn: cp4024_v4, cp4024_v6 [03:46:56] PROBLEM - IPsec on cp2005 is CRITICAL: Strongswan CRITICAL - ok: 80 not-conn: cp4024_v4, cp4024_v6 [03:46:56] PROBLEM - IPsec on cp1072 is CRITICAL: Strongswan CRITICAL - ok: 66 not-conn: cp4024_v4, cp4024_v6 [03:46:56] PROBLEM - IPsec on cp1074 is CRITICAL: Strongswan CRITICAL - ok: 66 not-conn: cp4024_v4, cp4024_v6 [03:47:05] PROBLEM - IPsec on cp1048 is CRITICAL: Strongswan CRITICAL - ok: 66 not-conn: cp4024_v4, cp4024_v6 [03:47:05] PROBLEM - IPsec on cp2002 is CRITICAL: Strongswan CRITICAL - ok: 80 not-conn: cp4024_v4, cp4024_v6 [03:47:06] PROBLEM - IPsec on cp2026 is CRITICAL: Strongswan CRITICAL - ok: 80 not-conn: cp4024_v4, cp4024_v6 [03:47:15] PROBLEM - IPsec on cp1064 is CRITICAL: Strongswan CRITICAL - ok: 66 not-conn: cp4024_v4, cp4024_v6 [04:08:05] RECOVERY - MariaDB Slave Lag: s1 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 213.32 seconds [04:21:36] PROBLEM - Host mw2256 is DOWN: PING CRITICAL - Packet loss = 100% [04:55:43] (03PS3) 10Jayprakash12345: Enable ArticlePlaceholder on sqwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/375015 (https://phabricator.wikimedia.org/T174335) [05:26:06] PROBLEM - CirrusSearch eqiad 95th percentile latency on graphite1001 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [1000.0] [05:39:16] RECOVERY - CirrusSearch eqiad 95th percentile latency on graphite1001 is OK: OK: Less than 20.00% above the threshold [500.0] [06:51:45] ACKNOWLEDGEMENT - Host mw2256 is DOWN: PING CRITICAL - Packet loss = 100% Elukey T163346 [06:52:19] 10Operations, 10ops-codfw, 10Patch-For-Review: mw2256 - hardware issue - https://phabricator.wikimedia.org/T163346#3574730 (10elukey) @Papaul the host froze again, all yours :) [07:48:55] (03PS12) 10MarcoAurelio: Initial configuration for hi.wikivoyage [mediawiki-config] - 10https://gerrit.wikimedia.org/r/371109 (https://phabricator.wikimedia.org/T173013) [09:08:32] 10Operations, 10ops-eqiad, 10DBA, 10hardware-requests: Decommission db1045 - https://phabricator.wikimedia.org/T174806#3574827 (10Peachey88) [10:34:10] 10Operations, 10Commons, 10MediaWiki-extensions-Scribunto, 10Patch-For-Review, 10Wikimedia-log-errors: Some Commons pages transcluding Template:Countries_of_Europe HTTP 500/503 when accessed from non-English languages specified in the template - https://phabricator.wikimedia.org/T171392#3574891 (10Johnuni... [14:38:19] so I can get to the UI for logstash at ogstash.wikimedia.org but how do you get to the API? different port? [14:38:48] I found this page https://wikitech.wikimedia.org/wiki/Logstash but nothing on accessing the API [14:42:42] PROBLEM - HHVM rendering on mw1284 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 1308 bytes in 0.001 second response time [14:42:42] PROBLEM - Nginx local proxy to apache on mw1227 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 1308 bytes in 0.010 second response time [14:43:42] RECOVERY - HHVM rendering on mw1284 is OK: HTTP OK: HTTP/1.1 200 OK - 74623 bytes in 0.201 second response time [14:43:42] RECOVERY - Nginx local proxy to apache on mw1227 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 613 bytes in 0.041 second response time [15:13:38] bd808 so I can get to the UI for logstash at ogstash.wikimedia.org but how do you get to the API? different port? [15:51:31] PROBLEM - Check systemd state on chlorine is CRITICAL: CRITICAL - degraded: The system is operational but one or more units failed. [15:51:32] PROBLEM - puppet last run on chlorine is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 13 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[kube-apiserver] [15:55:10] 10Operations, 10Commons, 10Multimedia, 10Traffic, and 2 others: Disable serving unpatrolled new files to Wikipedia Zero users - https://phabricator.wikimedia.org/T167400#3575133 (10Dispenser) A [[https://gerrit.wikimedia.org/r/363264|media block was implemented]] into Phabricator to combat T168142 WP0 abus... [15:55:51] PROBLEM - MegaRAID on db1059 is CRITICAL: CRITICAL: 1 failed LD(s) (Degraded) [15:55:52] ACKNOWLEDGEMENT - MegaRAID on db1059 is CRITICAL: CRITICAL: 1 failed LD(s) (Degraded) nagiosadmin RAID handler auto-ack: https://phabricator.wikimedia.org/T174857 [15:55:56] 10Operations, 10ops-eqiad: Degraded RAID on db1059 - https://phabricator.wikimedia.org/T174857#3575136 (10ops-monitoring-bot) [16:11:11] 10Operations, 10ops-eqiad, 10DBA: Degraded RAID on db1059 - https://phabricator.wikimedia.org/T174857#3575177 (10Volans) p:05Triage>03Normal [19:42:14] (03PS24) 10MarcoAurelio: [WIP DNM] Create computed list of wikis that can use SecurePoll [mediawiki-config] - 10https://gerrit.wikimedia.org/r/371926 (https://phabricator.wikimedia.org/T174398) [19:44:28] (03PS7) 10MarcoAurelio: Cloud VPS configuration for hi.wikivoyage [puppet] - 10https://gerrit.wikimedia.org/r/371096 (https://phabricator.wikimedia.org/T173013) [19:48:01] 10Operations, 10ops-eqiad, 10DBA: Degraded RAID on db1059 - https://phabricator.wikimedia.org/T174857#3575287 (10Marostegui) @Cmjohnson please go ahead and replace the disk when you can [20:04:21] (03CR) 10MarcoAurelio: "recheck" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/375015 (https://phabricator.wikimedia.org/T174335) (owner: 10Jayprakash12345) [20:06:20] (03PS1) 10Joal: Update pivot systemd start command [puppet] - 10https://gerrit.wikimedia.org/r/375538 [20:07:06] (03CR) 10Joal: "@Elukey: Can we try that manually before applying?" [puppet] - 10https://gerrit.wikimedia.org/r/375538 (owner: 10Joal) [20:07:08] (03CR) 10MarcoAurelio: [C: 031] Enable ArticlePlaceholder on sqwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/375015 (https://phabricator.wikimedia.org/T174335) (owner: 10Jayprakash12345) [21:23:08] (03PS1) 10Eranroz: Enable usage aspect C on elwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/375544 (https://phabricator.wikimedia.org/T151717) [21:55:18] 10Operations, 10Commons: Error: 500, Internal Server Error on many Commons categories. - https://phabricator.wikimedia.org/T174868#3575445 (10Multichill) [22:07:18] who's around? [22:07:45] would be good to have someone check the error logs for that task that multichill just filed to see how widespread the problem is [22:11:33] 10Operations, 10Commons: Error: 500, Internal Server Error on many Commons categories. - https://phabricator.wikimedia.org/T174868#3575476 (10zhuyifei1999) [22:11:40] 10Operations, 10Commons, 10MediaWiki-extensions-Scribunto, 10Patch-For-Review, 10Wikimedia-log-errors: Some Commons pages transcluding Template:Countries_of_Europe HTTP 500/503 when accessed from non-English languages specified in the template - https://phabricator.wikimedia.org/T171392#3575479 (10zhuyife... [22:26:16] 10Operations, 10Commons: Error: 500, Internal Server Error on many Commons categories. - https://phabricator.wikimedia.org/T174868#3575445 (10aude) hhvm error logs have "Fatal error: unknown exception" could this be similar to T70196 ?