[00:22:58] https://meta.wikimedia.org/wiki/User:Aldnonymous error occured.. [00:23:46] Probably babel related [00:24:01] Yeah, almost certainly [00:24:11] k, thanks for check it [00:24:31] https://phabricator.wikimedia.org/T199941 [02:06:18] 10Operations, 10WikiApiary, 10Wikimedia-Mailing-lists: WikiApiary: subscribe button on https://lists.wikimedia.org/mailman/listinfo/wikiapiary gives 403 Forbidden - https://phabricator.wikimedia.org/T200116 (10Peachey88) [02:07:10] 10Operations, 10Wikimedia-Mailing-lists: WikiApiary: subscribe button on https://lists.wikimedia.org/mailman/listinfo/wikiapiary gives 403 Forbidden - https://phabricator.wikimedia.org/T200116 (10JJMC89) [02:11:27] 10Operations, 10Wikimedia-Mailing-lists: WikiApiary: subscribe button on https://lists.wikimedia.org/mailman/listinfo/wikiapiary gives 403 Forbidden - https://phabricator.wikimedia.org/T200116 (10edwardspec) Duplicate of T195750 [02:12:16] 10Operations, 10Wikimedia-Mailing-lists: WikiApiary: subscribe button on https://lists.wikimedia.org/mailman/listinfo/wikiapiary gives 403 Forbidden - https://phabricator.wikimedia.org/T200116 (10edwardspec) 05Open>03Resolved [02:12:54] 10Operations, 10Wikimedia-Mailing-lists: WikiApiary: subscribe button on https://lists.wikimedia.org/mailman/listinfo/wikiapiary gives 403 Forbidden - https://phabricator.wikimedia.org/T200116 (10Peachey88) [02:13:07] 10Operations, 10Wikimedia-Mailing-lists, 10Patch-For-Review: Mailman issues a "403 Forbidden" error when subscribing to a list - https://phabricator.wikimedia.org/T195750 (10Peachey88) [03:26:43] PROBLEM - MariaDB Slave Lag: s1 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 927.99 seconds [03:43:21] 10Operations, 10Wikimedia-Apache-configuration, 10Chinese-Sites, 10Patch-For-Review, 10User-Urbanecm: All "zh-my" variant page views get 404 Not Found on zh.wikipedia.org - https://phabricator.wikimedia.org/T198371 (101233thehongkonger) @RazeSoldier , should be problem within the Operations tem? [03:52:54] RECOVERY - MariaDB Slave Lag: s1 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 284.75 seconds [04:56:23] PROBLEM - Nginx local proxy to apache on mw1285 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 1308 bytes in 0.007 second response time [04:56:34] PROBLEM - HHVM rendering on mw1285 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - 1308 bytes in 0.002 second response time [04:57:33] RECOVERY - Nginx local proxy to apache on mw1285 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 617 bytes in 0.046 second response time [04:57:44] RECOVERY - HHVM rendering on mw1285 is OK: HTTP OK: HTTP/1.1 200 OK - 75156 bytes in 0.303 second response time [05:40:26] 10Operations, 10Wikimedia-Apache-configuration, 10Chinese-Sites, 10Patch-For-Review, 10User-Urbanecm: All "zh-my" variant page views get 404 Not Found on zh.wikipedia.org - https://phabricator.wikimedia.org/T198371 (10RazeSoldier) >>! In T198371#4442589, @1233thehongkonger wrote: > @RazeSoldier , should... [05:46:54] PROBLEM - HTTP availability for Varnish at esams on einsteinium is CRITICAL: job=varnish-upload site=esams https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1 [05:47:54] PROBLEM - HTTP availability for Nginx -SSL terminators- at esams on einsteinium is CRITICAL: cluster=cache_upload site=esams https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1 [05:53:53] RECOVERY - HTTP availability for Varnish at esams on einsteinium is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1 [05:54:53] RECOVERY - HTTP availability for Nginx -SSL terminators- at esams on einsteinium is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1 [06:16:54] PROBLEM - HTTP availability for Varnish at esams on einsteinium is CRITICAL: job=varnish-upload site=esams https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1 [06:17:54] PROBLEM - HTTP availability for Nginx -SSL terminators- at esams on einsteinium is CRITICAL: cluster=cache_upload site=esams https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1 [06:19:14] PROBLEM - Upload HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [1000.0] https://grafana.wikimedia.org/dashboard/file/varnish-aggregate-client-status-codes.json?panelId=3&fullscreen&orgId=1&var-site=All&var-cache_type=upload&var-status_type=5 [06:19:53] PROBLEM - Esams HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [1000.0] https://grafana.wikimedia.org/dashboard/file/varnish-aggregate-client-status-codes.json?panelId=3&fullscreen&orgId=1&var-site=esams&var-cache_type=All&var-status_type=5 [06:28:23] PROBLEM - puppet last run on pollux is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/local/lib/nagios/plugins/check_long_procs] [06:29:13] RECOVERY - HTTP availability for Nginx -SSL terminators- at esams on einsteinium is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=4&fullscreen&refresh=1m&orgId=1 [06:29:23] RECOVERY - HTTP availability for Varnish at esams on einsteinium is OK: All metrics within thresholds. https://grafana.wikimedia.org/dashboard/db/frontend-traffic?panelId=3&fullscreen&refresh=1m&orgId=1 [06:29:53] PROBLEM - puppet last run on ms-be1030 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/bin/swift-drive-audit] [06:34:13] RECOVERY - Esams HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] https://grafana.wikimedia.org/dashboard/file/varnish-aggregate-client-status-codes.json?panelId=3&fullscreen&orgId=1&var-site=esams&var-cache_type=All&var-status_type=5 [06:34:43] RECOVERY - Upload HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] https://grafana.wikimedia.org/dashboard/file/varnish-aggregate-client-status-codes.json?panelId=3&fullscreen&orgId=1&var-site=All&var-cache_type=upload&var-status_type=5 [06:59:04] RECOVERY - puppet last run on pollux is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [07:00:23] RECOVERY - puppet last run on ms-be1030 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [07:09:22] 10Operations, 10ops-eqiad, 10DBA: db1069 bad disk - https://phabricator.wikimedia.org/T199056 (10Marostegui) Hey @Cmjohnson can we try to get this disk swapped soon? It is x1's primary master [08:29:24] 10Operations, 10WMF-Legal, 10Privacy, 10WMF-Microsites (Policy site): Consider moving policy.wikimedia.org away from WordPress.com - https://phabricator.wikimedia.org/T132104 (10Krinkle) [08:46:17] 10Operations, 10Traffic, 10WMF-Microsites (TransparencyReport): move transparency report from zirconium to bromine - https://phabricator.wikimedia.org/T104937 (10Peachey88) [08:46:21] 10Operations, 10WMF-Microsites (TransparencyReport): Staging area for the next version of the transparency report - https://phabricator.wikimedia.org/T138197 (10Peachey88) [08:46:36] 10Operations, 10Gerrit, 10WMF-Microsites (TransparencyReport): TransparencyReport repository master in Gerrit silently made private - https://phabricator.wikimedia.org/T89640 (10Peachey88) [08:46:38] 10Operations, 10WMF-Microsites (TransparencyReport): TransparencyReport-private is not auto deploying - https://phabricator.wikimedia.org/T188224 (10Peachey88) [08:46:42] 10Operations, 10DNS, 10Traffic, 10WMF-Microsites (TransparencyReport): Move "transparency.wikimedia.org/private" to "transparency-private.wikimedia.org" - https://phabricator.wikimedia.org/T188362 (10Peachey88) [08:53:33] 10Operations, 10WMF-Legal, 10WMF-Microsites (Policy site): Set up new URL policy.wikimedia.org - https://phabricator.wikimedia.org/T97329 (10Peachey88) [08:53:58] 10Operations, 10WMF-Microsites (Policy site): migrate policy.wikimedia.org from WMF cluster to Wordpress - https://phabricator.wikimedia.org/T110203 (10Peachey88) [08:54:36] 10Operations, 10Traffic, 10WMF-Microsites (Policy site): SSL certificate for policy.wikimedia.org - https://phabricator.wikimedia.org/T110197 (10Peachey88) [08:55:26] 10Operations, 10Traffic, 10WMF-Microsites (Policy site): SSL cert for policy.wm.org expiring Aug 27 - https://phabricator.wikimedia.org/T141564 (10Peachey88) [09:04:12] 10Operations, 10WMF-Microsites: create endowment.wm.org microsite - https://phabricator.wikimedia.org/T136735 (10Peachey88) [09:05:47] 10Operations, 10WMF-Legal, 10Privacy: Consider moving policy.wikimedia.org away from WordPress.com - https://phabricator.wikimedia.org/T132104 (10Krinkle) [09:22:00] [W1L6XgpAMEkAAFEWSvoAAABD] 2018-07-21 09:18:27: Fatal exception of type "InvalidArgumentException" [09:22:16] while undeleting a file on Commons [09:47:22] https://phabricator.wikimedia.org/T200121 [09:47:30] this is quite annoying ^ [11:01:32] *newbie alert*: I'd like to learn wiki syntax using personal wiki software on my windows 10, that is easy to setup and run. Can anyone recommend software for me? [11:29:09] https://phabricator.wikimedia.org/T200121 [11:29:23] this is quite worrying ^ [11:30:07] issue is confirmed by another admin [11:30:47] (03CR) 10Sau226: "Well I guess this needs SWATing and a rebase. Ping just to remind concerned parties that this might need to be ready when the relevant com" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/441839 (https://phabricator.wikimedia.org/T198056) (owner: 10Urbanecm) [11:35:04] ACKNOWLEDGEMENT - Device not healthy -SMART- on db1069 is CRITICAL: cluster=mysql device=megaraid,0 instance=db1069:9100 job=node site=eqiad Marostegui T199056 - The acknowledgement expires at: 2018-07-24 11:34:47. https://grafana.wikimedia.org/dashboard/db/host-overview?var-server=db1069&var-datasource=eqiad%2520prometheus%252Fops [11:41:13] yannf: according to https://wikitech.wikimedia.org/wiki/SRE_Team_requests the #operations project could (should?) maybe added to the phabricator task so it appears on the relevant dashboards [11:50:52] 10Operations, 10Commons, 10Wikimedia-log-errors: Fatal exception of type "InvalidArgumentException" while undeleting a file on Commons - https://phabricator.wikimedia.org/T200121 (10Yann) [11:50:58] hoffie, ok, done [11:52:10] I don't see how it's a problem for the SRE team [11:54:46] then i'm sorry for providing wrong suggestions :/ [14:56:28] (03PS4) 10ArielGlenn: generate and email a few dump-related stats each month [puppet] - 10https://gerrit.wikimedia.org/r/303707 (https://phabricator.wikimedia.org/T142435) [14:58:13] PROBLEM - Disk space on elastic1025 is CRITICAL: DISK CRITICAL - free space: /srv 52344 MB (10% inode=99%) [15:06:04] RECOVERY - Disk space on elastic1025 is OK: DISK OK [15:45:03] PROBLEM - Disk space on elastic1032 is CRITICAL: DISK CRITICAL - free space: /srv 72158 MB (10% inode=99%) [16:12:13] RECOVERY - Disk space on elastic1032 is OK: DISK OK [16:39:44] (03CR) 10MarcoAurelio: "Would this change make unable to login those [unlikely] accounts not meeting the new password guidelines after implementing? If so, an ann" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/440834 (https://phabricator.wikimedia.org/T197577) (owner: 10MarcoAurelio) [16:39:52] (03PS5) 10MarcoAurelio: Increase password policies for 'steward' to max [mediawiki-config] - 10https://gerrit.wikimedia.org/r/440834 (https://phabricator.wikimedia.org/T197577) [17:04:18] (03CR) 10Reedy: "MinimumPasswordLengthToLogin is still 1... Shouldn't be an issue?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/440834 (https://phabricator.wikimedia.org/T197577) (owner: 10MarcoAurelio) [17:35:13] !log rolling restart of eventstreams on scb2* nodes to reduce the memory pressure before the weekend (still waiting for a permanent fix) [17:35:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [17:35:45] Cc: mobrovac --^ [17:51:58] (03PS5) 10ArielGlenn: generate and email a few dump-related stats each month [puppet] - 10https://gerrit.wikimedia.org/r/303707 (https://phabricator.wikimedia.org/T142435) [20:03:13] PROBLEM - https://grafana.wikimedia.org/dashboard/db/varnish-http-requests grafana alert on einsteinium is CRITICAL: CRITICAL: https://grafana.wikimedia.org/dashboard/db/varnish-http-requests is alerting: 70% GET drop in 30min alert. [20:04:14] RECOVERY - https://grafana.wikimedia.org/dashboard/db/varnish-http-requests grafana alert on einsteinium is OK: OK: https://grafana.wikimedia.org/dashboard/db/varnish-http-requests is not alerting. [20:24:32] hello. https://meta.wikimedia.org/wiki/Special:Log/block <- does not work for change a log type drop down ... This bug already created a phab task? [20:38:51] I filed the task: https://phabricator.wikimedia.org/T200136 [20:46:02] (03PS6) 10ArielGlenn: generate and email a few dump-related stats each month [puppet] - 10https://gerrit.wikimedia.org/r/303707 (https://phabricator.wikimedia.org/T142435) [20:51:23] rxy: Thanks! [20:54:10] Krinkle: Thanks for confirm that. :) [21:45:28] 10Operations: Staging area for the next version of the transparency report - https://phabricator.wikimedia.org/T138197 (10Krinkle) [21:46:06] 10Operations, 10WMF-Legal: Set up new URL policy.wikimedia.org - https://phabricator.wikimedia.org/T97329 (10Krinkle) [21:52:26] 10Operations, 10Annual-Report: create endowment.wm.org microsite - https://phabricator.wikimedia.org/T136735 (10Krinkle) >>! In T136735#2348027, @MZMcBride wrote: > > We should at least keep a list of these micro-sites somewhere so that we can thoroughly kill them all at a later date. From browsing around Pha... [21:53:52] 10Operations, 10Annual-Report: create endowment.wm.org microsite - https://phabricator.wikimedia.org/T136735 (10Krenair) Does the bugzilla archive really count as a microsite? It's huge... [22:15:45] Getting an exception on wikitech when moving a page [22:15:48] ChangeTags::updateTags 10.64.16.79 1054 Unknown column 'ct_tag_id' in 'field list' (10.64.16.79) INSERT IGNORE INTO `change_tag` (ct_tag,ct_rev_id,ct_tag_id) VALUES ('mw-new-redirect','1797804','23') [22:15:54] [ccde5a6338d6939b949b6a8c] 2018-07-21 22:14:59: Fatal exception of type "Wikimedia\Rdbms\DBQueryError" [22:16:04] Got to go, but maybe someone can investigate or file a task [22:17:15] wouldn't surprise me if it's a schema update someone forgot to apply to labswiki [22:18:35] -> https://phabricator.wikimedia.org/T200139 [22:26:36] Looks very much like that [22:29:38] !log Added ct_tag_id to labswiki and labtestwiki T200139 [22:29:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:29:42] T200139: Report of exception while moving pages on wikitech - https://phabricator.wikimedia.org/T200139 [22:46:23] Reedy, lol, good luck with https://phabricator.wikimedia.org/T200140 [22:48:48] :D [22:48:57] I mean in comparison to other WMF wikis [22:49:00] Not to tables.sql ;) [22:49:10] isn't there a task for that already? [22:49:11] Mah tables! [22:49:53] * Reedy pets SQL [22:50:03] p858snake: Generically, or for labswiki? [22:50:29] i have vague memories for labswiki, or might have been back in the day when wikitech was seperate [22:50:37] but either way, i'm too lazy to look [23:06:54] Reedy, no I know [23:06:55] I mean [23:07:20] That's something I was talking about years ago [23:07:27] differences between labswiki and other wmf wikis [23:07:32] at the DB schema level [23:08:46] p858snake, wikitech is still separate in some ways [23:08:56] not as much as it used to be [23:09:02] but still