[00:02:00] Krenair: snapshot host applies cleanly on trusty! \o/ [00:02:06] s/host/role/ [00:02:10] yay [00:03:24] (03PS1) 10MarcoAurelio: Enable Education Program extension at srwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/236231 (https://phabricator.wikimedia.org/T110619) [00:03:39] RECOVERY - puppet last run on osmium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:05:47] 7Blocked-on-Operations, 6operations, 10Datasets-General-or-Unknown: Snapshot hosts need to be manually added to dataset1001's exports - https://phabricator.wikimedia.org/T111586#1611081 (10ori) 3NEW [00:06:31] 7Blocked-on-Operations, 6operations, 10Datasets-General-or-Unknown: Snapshot hosts need to be manually added to dataset1001's exports - https://phabricator.wikimedia.org/T111586#1611089 (10ori) [00:06:32] 6operations, 10Datasets-General-or-Unknown, 7HHVM: Convert snapshot hosts to use HHVM and trusty - https://phabricator.wikimedia.org/T94277#1611088 (10ori) [00:08:50] PROBLEM - Unmerged changes on repository mediawiki_config on tin is CRITICAL: There are 2 unmerged changes in mediawiki_config (dir /srv/mediawiki-staging/). [00:09:52] (03PS1) 10Ori.livneh: Document bug T111586 in a comment in site.pp [puppet] - 10https://gerrit.wikimedia.org/r/236233 [00:10:48] (03CR) 10Ori.livneh: [C: 032 V: 032] Document bug T111586 in a comment in site.pp [puppet] - 10https://gerrit.wikimedia.org/r/236233 (owner: 10Ori.livneh) [00:11:52] Krenair: re. access-requests - seem no one knows with anything decided [00:12:12] mar.k says it's not worth the hassle of importing it so... that's one stance clear [00:12:35] so what are they doing to preserve the history then? [00:13:08] 6operations, 10Datasets-General-or-Unknown, 7HHVM: Convert snapshot hosts to use HHVM and trusty - https://phabricator.wikimedia.org/T94277#1611114 (10ori) To nudge this along, I applied the roles and classes currently applied to the snapshot hosts to osmium, which is running Trusty. With the exception of th... [00:14:54] magic to my knowledge :) [00:16:13] (03PS1) 10Ori.livneh: Remove snapshot roles and classes from osmium [puppet] - 10https://gerrit.wikimedia.org/r/236234 [00:16:35] (03CR) 10Ori.livneh: [C: 032 V: 032] "Test is done; roles applied successfully. Reverting the host to its prior state now." [puppet] - 10https://gerrit.wikimedia.org/r/236234 (owner: 10Ori.livneh) [00:42:53] 7Blocked-on-Operations, 6operations, 10Datasets-General-or-Unknown: Snapshot hosts need to be manually added to dataset1001's exports - https://phabricator.wikimedia.org/T111586#1611174 (10Krenair) [01:01:59] (03PS1) 10Faidon Liambotis: Switch Mexico to codfw [dns] - 10https://gerrit.wikimedia.org/r/236235 [01:02:01] (03PS1) 10Faidon Liambotis: Switch US states AR,LA,NM,OK to codfw [dns] - 10https://gerrit.wikimedia.org/r/236236 [01:07:43] (03PS25) 10Ori.livneh: Basic role for Sentry [puppet] - 10https://gerrit.wikimedia.org/r/199598 (https://phabricator.wikimedia.org/T84956) (owner: 10Gilles) [01:08:06] (03CR) 10Ori.livneh: [C: 031] "Made a few cosmetic fixes. LGTM." [puppet] - 10https://gerrit.wikimedia.org/r/199598 (https://phabricator.wikimedia.org/T84956) (owner: 10Gilles) [01:41:27] (03PS1) 10Dzahn: wikistats: crons for db backup (WIP) [puppet] - 10https://gerrit.wikimedia.org/r/236238 [01:42:11] (03CR) 10jenkins-bot: [V: 04-1] wikistats: crons for db backup (WIP) [puppet] - 10https://gerrit.wikimedia.org/r/236238 (owner: 10Dzahn) [02:20:20] 6operations: Do not apply spam headers on email assessed NOT to be spam - https://phabricator.wikimedia.org/T111595#1611313 (10JKrauska) 3NEW [02:27:12] !log l10nupdate@tin Synchronized php-1.26wmf21/cache/l10n: l10nupdate for 1.26wmf21 (duration: 05m 53s) [02:27:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:30:06] !log l10nupdate@tin LocalisationUpdate completed (1.26wmf21) at 2015-09-05 02:30:06+00:00 [02:30:11] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [02:31:38] (03PS1) 10Dduvall: Rename and simplify some git deploy functions [tools/scap] - 10https://gerrit.wikimedia.org/r/236241 (https://phabricator.wikimedia.org/T109514) [02:38:58] (03PS1) 10RobH: robh on vacation [puppet] - 10https://gerrit.wikimedia.org/r/236242 [02:39:21] reverting this patchset later is always sad. [02:39:44] (cuz who wants to be paged? ;) [02:40:00] (03PS2) 10RobH: robh on vacation [puppet] - 10https://gerrit.wikimedia.org/r/236242 [02:40:48] (03CR) 10RobH: [C: 032] robh on vacation [puppet] - 10https://gerrit.wikimedia.org/r/236242 (owner: 10RobH) [02:41:13] mutante: ^ yer on yer own now in PDT ;] [02:59:16] PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet). [03:09:32] (03PS26) 10Gergő Tisza: Basic role for Sentry [puppet] - 10https://gerrit.wikimedia.org/r/199598 (https://phabricator.wikimedia.org/T84956) (owner: 10Gilles) [03:11:05] (03PS27) 10Gergő Tisza: Basic role for Sentry [puppet] - 10https://gerrit.wikimedia.org/r/199598 (https://phabricator.wikimedia.org/T84956) (owner: 10Gilles) [03:12:23] (03CR) 10Gergő Tisza: [C: 04-1] "DB creation fails on first run but works on second; there is an undeclared dependency somewhere." [puppet] - 10https://gerrit.wikimedia.org/r/199598 (https://phabricator.wikimedia.org/T84956) (owner: 10Gilles) [03:32:11] PROBLEM - puppet last run on mw2018 is CRITICAL: CRITICAL: Puppet has 1 failures [03:57:49] RECOVERY - puppet last run on mw2018 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:31:34] !log l10nupdate@tin ResourceLoader cache refresh completed at Sat Sep 5 04:31:34 UTC 2015 (duration 31m 33s) [04:31:39] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [06:30:39] PROBLEM - puppet last run on cp2001 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:21] PROBLEM - puppet last run on holmium is CRITICAL: CRITICAL: Puppet has 2 failures [06:31:39] PROBLEM - puppet last run on mw1060 is CRITICAL: CRITICAL: Puppet has 2 failures [06:31:50] PROBLEM - puppet last run on mw2023 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:59] PROBLEM - puppet last run on mw1158 is CRITICAL: CRITICAL: Puppet has 2 failures [06:32:00] PROBLEM - puppet last run on db2056 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:10] PROBLEM - puppet last run on mc2015 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:19] PROBLEM - puppet last run on mw2158 is CRITICAL: CRITICAL: Puppet has 1 failures [06:32:19] PROBLEM - puppet last run on mw2073 is CRITICAL: CRITICAL: Puppet has 2 failures [06:33:10] PROBLEM - puppet last run on mw2207 is CRITICAL: CRITICAL: Puppet has 1 failures [06:33:29] PROBLEM - puppet last run on mw2045 is CRITICAL: CRITICAL: Puppet has 1 failures [06:55:57] RECOVERY - puppet last run on cp2001 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [06:56:36] RECOVERY - puppet last run on mc2015 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [06:56:38] RECOVERY - puppet last run on mw1060 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:56:57] RECOVERY - puppet last run on mw2023 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:58:06] RECOVERY - puppet last run on mw1158 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:58:17] RECOVERY - puppet last run on db2056 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:58:37] RECOVERY - puppet last run on holmium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:58:47] RECOVERY - puppet last run on mw2158 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:58:47] RECOVERY - puppet last run on mw2207 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [06:58:47] RECOVERY - puppet last run on mw2073 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:59:17] RECOVERY - puppet last run on mw2045 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:23:49] PROBLEM - puppet last run on cp3009 is CRITICAL: CRITICAL: puppet fail [07:50:00] RECOVERY - puppet last run on cp3009 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [08:07:06] 6operations, 10Datasets-Archiving: Import Wikimania 2015 Videos - https://phabricator.wikimedia.org/T106565#1611407 (10Nemo_bis) https://lists.wikimedia.org/pipermail/wikimania-l/2015-September/007021.html [08:11:55] PROBLEM - puppet last run on mw1042 is CRITICAL: CRITICAL: Puppet has 1 failures [08:37:46] RECOVERY - puppet last run on mw1042 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [08:49:16] PROBLEM - HTTP 5xx req/min on graphite1001 is CRITICAL: CRITICAL: 7.69% of data above the critical threshold [500.0] [08:55:16] RECOVERY - HTTP 5xx req/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] [09:19:15] PROBLEM - HTTP 5xx req/min on graphite1001 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [500.0] [09:25:07] RECOVERY - HTTP 5xx req/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] [12:42:15] PROBLEM - Router interfaces on cr1-eqdfw is CRITICAL: CRITICAL: host 208.80.153.198, interfaces up: 33, down: 1, dormant: 0, excluded: 0, unused: 0BRxe-0/0/1: down - Transit: ! NTT (service ID 253065) {#11401} [10Gbps]BR [15:21:14] (03PS2) 10Amire80: Configure $wgBabelCategoryNames for the Ladino Wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/236042 [16:06:24] (03CR) 10Nikerabbit: [C: 031] Configure $wgBabelCategoryNames for the Hebrew Wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/236025 (owner: 10Amire80) [16:06:37] (03CR) 10Nikerabbit: [C: 031] Configure $wgBabelCategoryNames for the Ladino Wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/236042 (owner: 10Amire80) [16:12:56] (03CR) 10Amire80: "Scheduled for September 9: https://wikitech.wikimedia.org/wiki/Deployments#Wednesday.2C.C2.A0September.C2.A009" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/236025 (owner: 10Amire80) [16:13:08] (03CR) 10Amire80: "Scheduled for September 9: https://wikitech.wikimedia.org/wiki/Deployments#Wednesday.2C.C2.A0September.C2.A009" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/236042 (owner: 10Amire80) [17:03:42] 6operations, 10OTRS, 10vm-requests, 5Patch-For-Review: EQIAD: 1 VM request for OTRS - https://phabricator.wikimedia.org/T111532#1611693 (10akosiaris) Hello, This is for https://www.mediawiki.org/wiki/Wikimedia_Engineering/2015-16_Q1_Goals#Core_Ops_team, specifically * Investigate OTRS upgrade (from 3.2.1... [17:21:30] (03PS5) 10Ori.livneh: Disallow indexing for /api/ [mediawiki-config] - 10https://gerrit.wikimedia.org/r/236200 (https://phabricator.wikimedia.org/T109023) (owner: 10GWicke) [18:35:25] PROBLEM - puppet last run on mw2155 is CRITICAL: CRITICAL: puppet fail [19:03:16] RECOVERY - puppet last run on mw2155 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [20:34:56] RECOVERY - Router interfaces on cr1-eqdfw is OK: OK: host 208.80.153.198, interfaces up: 35, down: 0, dormant: 0, excluded: 0, unused: 0 [20:42:55] PROBLEM - Router interfaces on cr1-eqdfw is CRITICAL: CRITICAL: host 208.80.153.198, interfaces up: 33, down: 1, dormant: 0, excluded: 0, unused: 0BRxe-0/0/1: down - Transit: ! NTT (service ID 253065) {#11401} [10Gbps]BR [21:02:55] RECOVERY - Router interfaces on cr1-eqdfw is OK: OK: host 208.80.153.198, interfaces up: 35, down: 0, dormant: 0, excluded: 0, unused: 0 [22:37:21] PROBLEM - puppet last run on mw2148 is CRITICAL: CRITICAL: puppet fail [22:44:55] 6operations: ircecho should support nickserv registration - https://phabricator.wikimedia.org/T48254#1611983 (10Krenair) [22:46:51] 6operations: morebots missing from #wikimedia-operations - https://phabricator.wikimedia.org/T45897#1611986 (10Krenair) [22:47:53] 6operations: morebots need restart - https://phabricator.wikimedia.org/T28782#1611989 (10Krenair) [22:49:38] 6operations: IPv6 support for WMF-sponsored freenode server - https://phabricator.wikimedia.org/T59129#1612001 (10Krenair) [22:50:59] 6operations: gerrit-wm no more notify on comment or merge - https://phabricator.wikimedia.org/T41797#1612008 (10Krenair) [22:57:15] 6operations, 7Icinga: register a nickserv account for icinga-wm - https://phabricator.wikimedia.org/T22771#1612023 (10Krenair) [23:05:30] RECOVERY - puppet last run on mw2148 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [23:35:11] PROBLEM - HTTP 5xx req/min on graphite1001 is CRITICAL: CRITICAL: 23.08% of data above the critical threshold [500.0] [23:37:09] !log mwscript deleteEqualMessages.php --wiki fywiktionary [23:37:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master [23:46:53] Is ircecho master known to be broken? :/ [23:47:11] RECOVERY - HTTP 5xx req/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] [23:50:17] It can't possibly work unless you use the supposedly-optional 'infile' [23:51:26] (03PS1) 10Alex Monk: Unbreak non-infile mode [debs/ircecho] - 10https://gerrit.wikimedia.org/r/236379