[00:00:39] <logmsgbot>	 !log rmoen Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 11s)
[00:00:43] <rmoen>	 Mjbmr: Done
[00:00:46] <morebots>	 Logged the message, Master
[00:00:49] <Mjbmr>	 Thanks!
[00:00:56] <rmoen>	 Mjbmr: thank you
[00:01:03] <Mjbmr>	 np
[00:01:04] <rmoen>	 My first schema change
[00:01:22] <rmoen>	 RoanKattouw: cheers
[00:01:39] <RoanKattouw>	 rmoen: Did you run the population script as well?
[00:01:43] <rmoen>	 uh
[00:01:44] <rmoen>	 oops
[00:01:52] <rmoen>	 no
[00:02:15] <rmoen>	 RoanKattouw: where to run populateShortUrlTable.php  from ? 
[00:02:29] <RoanKattouw>	 [16:58]	RoanKattouw	Then do mwscript extensions/ShortUrl/populateShortUrlTable.php --wiki=eswikibooks
[00:02:32] <rmoen>	 oh
[00:02:33] <RoanKattouw>	 You can run that command from anywhere
[00:02:40] <Mjbmr>	 lol
[00:02:46] <rmoen>	 Done
[00:02:50] <RoanKattouw>	 Awesome
[00:02:52] * RoanKattouw high-fives rmoen
[00:03:26] <wikibugs>	 6operations, 6WMF-Legal, 6WMF-NDA-Requests: Add multichill to WMF-NDA group - https://phabricator.wikimedia.org/T87097#1271100 (10Dzahn) 5Open>3Resolved Thank you! It's done then. You are also in the NDA LDAP group which lets you login on icinga, graphite, etc.
[00:04:01] <Mjbmr>	 rmoen: see RoanKattouw's comment: https://wikitech.wikimedia.org/wiki/How_to_do_a_schema_change
[00:05:44] <RoanKattouw>	 haha that's old
[00:05:49] <rmoen>	 Mjbmr: I've read this just have never done it until now. 
[00:05:54] <RoanKattouw>	 You can tell from the comment that was before I lived in SF
[00:05:57] <RoanKattouw>	 I've lived here for 3 years now
[00:06:12] <rmoen>	 Before you became a native english speaker
[00:06:14] <rmoen>	 :)
[00:41:52] <wikibugs>	 6operations, 6Labs, 10Tool-Labs, 7Monitoring: Add catchall tests for toollabs to catchpoint - https://phabricator.wikimedia.org/T97321#1271185 (10yuvipanda) 5Open>3declined a:3yuvipanda Superseeded by T97748 and friends. We're having fine grained tests there rather than catchall ones.
[00:56:15] <grrrit-wm>	 (03CR) 10BryanDavis: "Over 24 hours with groups 0+1 logging to logstash and everything looks great. Monday morning seems like a good time to start logging every" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209172 (https://phabricator.wikimedia.org/T88732) (owner: 10BryanDavis)
[02:16:36] <icinga-wm>	 PROBLEM - puppet last run on mw2157 is CRITICAL puppet fail
[02:24:25] <logmsgbot>	 !log l10nupdate Synchronized php-1.26wmf4/cache/l10n: (no message) (duration: 06m 06s)
[02:24:36] <morebots>	 Logged the message, Master
[02:27:14] <jamesofur>	 hmm, can anyone help get a new project-logo uploaded? It looks like in ori's change today the wikimania2015 one went missing
[02:27:25] <jamesofur>	 when it was just added a day or two ago and they are trying to launch Registration
[02:27:36] <jamesofur>	 so getting it uploaded asap would be really really nice :)
[02:27:41] <jamesofur>	 since they just emailed me freaked out
[02:29:10] <logmsgbot>	 !log LocalisationUpdate completed (1.26wmf4) at 2015-05-08 02:28:07+00:00
[02:29:22] <morebots>	 Logged the message, Master
[02:34:35] <icinga-wm>	 RECOVERY - puppet last run on mw2157 is OK Puppet is currently enabled, last run 24 seconds ago with 0 failures
[02:44:59] <logmsgbot>	 !log l10nupdate Synchronized php-1.26wmf5/cache/l10n: (no message) (duration: 05m 47s)
[02:45:13] <morebots>	 Logged the message, Master
[02:49:18] <logmsgbot>	 !log LocalisationUpdate completed (1.26wmf5) at 2015-05-08 02:48:15+00:00
[02:49:28] <morebots>	 Logged the message, Master
[02:51:00] <wikibugs>	 6operations, 6Labs: Investigate ways of getting off raid6 for labs store - https://phabricator.wikimedia.org/T96063#1271404 (10coren) >>! In T96063#1268232, @mark wrote: > With the current stability and performance problems of NFS with RAID6, this is definitely not a "nice to have" but something that needs to...
[02:59:26] <icinga-wm>	 PROBLEM - puppet last run on eeden is CRITICAL puppet fail
[03:17:26] <icinga-wm>	 RECOVERY - puppet last run on eeden is OK Puppet is currently enabled, last run 1 minute ago with 0 failures
[03:24:27] <legoktm>	 jamesofur: still need help?
[03:24:33] <jamesofur>	 aye
[03:25:36] <legoktm>	 jamesofur: what project and what should the logo be?
[03:25:56] <jamesofur>	 https://wikimania2015.wikimedia.org/wiki/Main_Page and https://wikimania2015.wikimedia.org/wiki/File:Wiki.svg
[03:29:37] <legoktm>	 hmm
[03:30:51] <legoktm>	 oh heh
[03:30:52] <legoktm>	 I see 
[03:31:38] <grrrit-wm>	 (03PS1) 10Legoktm: Use png for wikimania2015wiki logo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209675 
[03:32:03] <grrrit-wm>	 (03CR) 10Legoktm: [C: 032] Use png for wikimania2015wiki logo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209675 (owner: 10Legoktm)
[03:32:08] <grrrit-wm>	 (03Merged) 10jenkins-bot: Use png for wikimania2015wiki logo [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209675 (owner: 10Legoktm)
[03:33:06] <logmsgbot>	 !log legoktm Synchronized w/static/images/project-logos/wikimania2015wiki.png: Use png for wikimania2015wiki logo (duration: 00m 12s)
[03:33:15] <morebots>	 Logged the message, Master
[03:33:21] * jamesofur sees it
[03:33:26] <legoktm>	 woot
[04:03:46] <ori>	 legoktm, Jamesofur|cloud: thank you
[04:14:15] <grrrit-wm>	 (03PS1) 10KartikMistry: Fix wikiname: roa-rupwiki -> roa_rupwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209676 
[04:14:37] <kart_>	 Can anyone quickly fix ^^
[04:15:24] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 032] Fix wikiname: roa-rupwiki -> roa_rupwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209676 (owner: 10KartikMistry)
[04:15:30] <grrrit-wm>	 (03Merged) 10jenkins-bot: Fix wikiname: roa-rupwiki -> roa_rupwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209676 (owner: 10KartikMistry)
[04:15:59] <kart_>	 ori: thanks!
[04:16:39] <logmsgbot>	 !log ori Synchronized wmf-config/InitialiseSettings.php: I4c70ce4d0: Fix wikiname: roa-rupwiki -> roa_rupwiki (duration: 00m 12s)
[04:16:50] <morebots>	 Logged the message, Master
[04:17:48] <ori>	 kart_: your turn! https://github.com/wikimedia/jquery.uls/pull/186
[04:17:49] <ori>	 ;)
[04:17:54] <ori>	 (i'm kidding, it's not urgent)
[04:18:05] <kart_>	 :)
[04:30:16] <grrrit-wm>	 (03PS4) 10MZMcBride: Rsyncing slow-parse logs from fluorine to dumps.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/49678 (https://phabricator.wikimedia.org/T98563) (owner: 10Ottomata)
[04:30:20] <grrrit-wm>	 (03PS5) 10MZMcBride: Rsyncing slow-parse logs from fluorine to dumps.wikimedia.org [puppet] - 10https://gerrit.wikimedia.org/r/49678 (https://phabricator.wikimedia.org/T98563) (owner: 10Ottomata)
[04:32:24] <grrrit-wm>	 (03CR) 10MZMcBride: "Huh, I got some "database connection wasn't allocated" error message from Gerrit after adding https://phabricator.wikimedia.org/T98563 to " [puppet] - 10https://gerrit.wikimedia.org/r/49678 (https://phabricator.wikimedia.org/T98563) (owner: 10Ottomata)
[04:37:08] <grrrit-wm>	 (03Abandoned) 10KartikMistry: Use dblist for contenttranslation [mediawiki-config] - 10https://gerrit.wikimedia.org/r/199576 (owner: 10KartikMistry)
[04:38:08] <grrrit-wm>	 (03PS7) 10KartikMistry: Add initial Debian packaging for apertium-dan-nor [debs/contenttranslation/apertium-dan-nor] - 10https://gerrit.wikimedia.org/r/195905 (https://phabricator.wikimedia.org/T91493) 
[04:41:40] <grrrit-wm>	 (03PS1) 10BryanDavis: beta: Remove custom apache2 log config [puppet] - 10https://gerrit.wikimedia.org/r/209680 (https://phabricator.wikimedia.org/T98289) 
[04:41:46] <bd808>	 yuvipanda: ^
[04:42:00] <yuvipanda>	 looking
[04:42:17] <yuvipanda>	 bd808: are you cherry picking or should I just merge?
[04:42:21] <yuvipanda>	 it’s clearly beta only so I don’t mind
[04:42:41] <bd808>	 hmm... maybe I should cherry-pick first
[04:42:50] <bd808>	 easier to undo if needed
[04:43:02] <yuvipanda>	 bd808: the only other commit I made today required two follow up commits to fix, so maybe yeah do cherry pick :)
[04:43:39] <bd808>	 git to keep that commit count high! I'm only the #17 committer to ops/puppet
[04:43:47] <bd808>	 according to github anyway
[04:44:56] <yuvipanda>	 :P
[04:54:59] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032] beta: Remove custom apache2 log config [puppet] - 10https://gerrit.wikimedia.org/r/209680 (https://phabricator.wikimedia.org/T98289) (owner: 10BryanDavis)
[04:55:44] <yuvipanda>	 bd808: done
[04:56:13] <bd808>	 cool
[05:04:41] <grrrit-wm>	 (03PS1) 10Andrew Bogott: Rename CONF.libvirt_images_type to CONF.libvirt.images_type [puppet] - 10https://gerrit.wikimedia.org/r/209681 
[05:05:31] <grrrit-wm>	 (03PS1) 10BryanDavis: beta: fix symlinks in docroot/bits/static/master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209682 
[05:05:46] <grrrit-wm>	 (03CR) 10Andrew Bogott: [C: 032] Rename CONF.libvirt_images_type to CONF.libvirt.images_type [puppet] - 10https://gerrit.wikimedia.org/r/209681 (owner: 10Andrew Bogott)
[05:09:18] <grrrit-wm>	 (03PS1) 10OliverKeyes: Add fluorine rsync connector [puppet] - 10https://gerrit.wikimedia.org/r/209684 
[05:09:56] <icinga-wm>	 PROBLEM - puppet last run on palladium is CRITICAL puppet fail
[05:10:37] <grrrit-wm>	 (03PS2) 10OliverKeyes: Add fluorine rsync connector [puppet] - 10https://gerrit.wikimedia.org/r/209684 
[05:14:27] <logmsgbot>	 !log LocalisationUpdate ResourceLoader cache refresh completed at Fri May  8 05:13:23 UTC 2015 (duration 13m 22s)
[05:14:40] <morebots>	 Logged the message, Master
[05:16:26] <icinga-wm>	 RECOVERY - puppet last run on palladium is OK Puppet is currently enabled, last run 9 seconds ago with 0 failures
[06:03:12] <grrrit-wm>	 (03PS6) 10KartikMistry: CX: Use RESTBase API for page fetch [puppet] - 10https://gerrit.wikimedia.org/r/207378 
[06:13:52] <grrrit-wm>	 (03PS8) 10Yuvipanda: [WIP]mesos: Add simple mesos module [puppet] - 10https://gerrit.wikimedia.org/r/208483 
[06:21:09] <grrrit-wm>	 (03PS9) 10Yuvipanda: [WIP]mesos: Add simple mesos module [puppet] - 10https://gerrit.wikimedia.org/r/208483 
[06:28:35] <grrrit-wm>	 (03PS10) 10Yuvipanda: [WIP]mesos: Add simple mesos module [puppet] - 10https://gerrit.wikimedia.org/r/208483 
[06:29:07] <icinga-wm>	 PROBLEM - puppet last run on db1034 is CRITICAL puppet fail
[06:29:14] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] [WIP]mesos: Add simple mesos module [puppet] - 10https://gerrit.wikimedia.org/r/208483 (owner: 10Yuvipanda)
[06:32:15] <icinga-wm>	 PROBLEM - puppet last run on ms-fe2001 is CRITICAL Puppet has 1 failures
[06:32:16] <icinga-wm>	 PROBLEM - puppet last run on mw2134 is CRITICAL Puppet has 1 failures
[06:32:25] <icinga-wm>	 PROBLEM - puppet last run on mw2016 is CRITICAL Puppet has 1 failures
[06:32:26] <icinga-wm>	 PROBLEM - puppet last run on cp4014 is CRITICAL Puppet has 1 failures
[06:33:36] <icinga-wm>	 PROBLEM - puppet last run on mw1170 is CRITICAL Puppet has 1 failures
[06:33:55] <icinga-wm>	 PROBLEM - puppet last run on mw2212 is CRITICAL Puppet has 1 failures
[06:33:55] <icinga-wm>	 PROBLEM - puppet last run on mw2184 is CRITICAL Puppet has 1 failures
[06:34:05] <icinga-wm>	 PROBLEM - puppet last run on mw2045 is CRITICAL Puppet has 1 failures
[06:34:06] <icinga-wm>	 PROBLEM - puppet last run on mw1251 is CRITICAL Puppet has 2 failures
[06:34:45] <icinga-wm>	 PROBLEM - puppet last run on mw2003 is CRITICAL Puppet has 1 failures
[06:35:06] <icinga-wm>	 PROBLEM - puppet last run on mw2136 is CRITICAL Puppet has 1 failures
[06:42:05] <wikibugs>	 6operations, 5Patch-For-Review: Scale up and out our puppetmaster infrastructure - https://phabricator.wikimedia.org/T98128#1271725 (10Joe) @chasemp we do exported resources, which don't really work in a masterless config
[06:46:45] <icinga-wm>	 RECOVERY - puppet last run on mw1170 is OK Puppet is currently enabled, last run 33 seconds ago with 0 failures
[06:46:56] <icinga-wm>	 RECOVERY - puppet last run on ms-fe2001 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:46:56] <icinga-wm>	 RECOVERY - puppet last run on mw2212 is OK Puppet is currently enabled, last run 32 seconds ago with 0 failures
[06:46:57] <icinga-wm>	 RECOVERY - puppet last run on mw2134 is OK Puppet is currently enabled, last run 21 seconds ago with 0 failures
[06:47:05] <icinga-wm>	 RECOVERY - puppet last run on db1034 is OK Puppet is currently enabled, last run 48 seconds ago with 0 failures
[06:47:06] <icinga-wm>	 RECOVERY - puppet last run on mw2016 is OK Puppet is currently enabled, last run 38 seconds ago with 0 failures
[06:47:06] <icinga-wm>	 RECOVERY - puppet last run on mw2045 is OK Puppet is currently enabled, last run 8 seconds ago with 0 failures
[06:47:06] <icinga-wm>	 RECOVERY - puppet last run on mw1251 is OK Puppet is currently enabled, last run 16 seconds ago with 0 failures
[06:47:06] <icinga-wm>	 RECOVERY - puppet last run on cp4014 is OK Puppet is currently enabled, last run 20 seconds ago with 0 failures
[06:47:46] <icinga-wm>	 RECOVERY - puppet last run on mw2003 is OK Puppet is currently enabled, last run 46 seconds ago with 0 failures
[06:48:15] <icinga-wm>	 RECOVERY - puppet last run on mw2136 is OK Puppet is currently enabled, last run 56 seconds ago with 0 failures
[06:48:35] <icinga-wm>	 RECOVERY - puppet last run on mw2184 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures
[06:52:09] <grrrit-wm>	 (03PS11) 10Yuvipanda: [WIP]mesos: Add simple mesos module [puppet] - 10https://gerrit.wikimedia.org/r/208483 
[07:22:51] <wikibugs>	 6operations, 6Release-Engineering, 10Wikimedia-Language-setup, 10Wikimedia-Site-requests: nb subdomain redirects - https://phabricator.wikimedia.org/T86924#1271748 (10jayvdb)
[07:26:07] <icinga-wm>	 ACKNOWLEDGEMENT - Unmerged changes on repository puppet on rhodium is CRITICAL: There are 48 unmerged changes in puppet (dir /var/lib/git/operations/puppet). alexandros kosiaris still being onlined. Ignore
[07:26:07] <icinga-wm>	 ACKNOWLEDGEMENT - puppetmaster backend https on rhodium is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 8141: HTTP/1.1 500 Internal Server Error alexandros kosiaris still being onlined. Ignore
[07:36:05] <icinga-wm>	 PROBLEM - puppet last run on einsteinium is CRITICAL Puppet last ran 4 hours ago
[07:42:49] <wikibugs>	 6operations: Upgrade salt to 2014.7 (investigating) - https://phabricator.wikimedia.org/T88971#1271786 (10ArielGlenn)
[07:44:47] <wikibugs>	 6operations, 6Labs: upgrade salt in labs - https://phabricator.wikimedia.org/T98578#1271787 (10ArielGlenn) 3NEW a:3ArielGlenn
[07:52:06] <icinga-wm>	 PROBLEM - DPKG on sca1002 is CRITICAL: DPKG CRITICAL dpkg reports broken packages
[07:53:45] <icinga-wm>	 RECOVERY - DPKG on sca1002 is OK: All packages OK
[08:04:09] <wikibugs>	 6operations, 6Labs: upgrade salt in labs - https://phabricator.wikimedia.org/T98578#1271817 (10ArielGlenn)
[08:04:11] <wikibugs>	 6operations: Upgrade salt to 2014.7 (investigating) - https://phabricator.wikimedia.org/T88971#1271816 (10ArielGlenn)
[08:05:02] <wikibugs>	 6operations: Upgrade salt to 2014.7 (investigating) - https://phabricator.wikimedia.org/T88971#1024437 (10ArielGlenn)
[08:05:03] <wikibugs>	 6operations, 6Labs: upgrade salt in labs - https://phabricator.wikimedia.org/T98578#1271819 (10ArielGlenn) 5Open>3Resolved This has been done.  Some instances were skipped, a full list follows.  1) Instances that were shut off at the time:  Instance: i-000000fd Status: SHUTOFF hostname: mwreview-merl Insta...
[08:05:56] <wikibugs>	 6operations, 6Labs: upgrade salt in labs - https://phabricator.wikimedia.org/T98578#1271822 (10ArielGlenn)
[08:08:00] <wikibugs>	 6operations: Upgrade salt to 2014.7 (investigating) - https://phabricator.wikimedia.org/T88971#1271835 (10ArielGlenn)
[08:13:18] <wikibugs>	 6operations: fix trebuchet-trigger (git deploy) publish.runner arguments - https://phabricator.wikimedia.org/T98581#1271844 (10ArielGlenn) 3NEW
[08:14:02] <wikibugs>	 6operations: fix trebuchet-trigger (git deploy) publish.runner arguments - https://phabricator.wikimedia.org/T98581#1271851 (10ArielGlenn) a:3ArielGlenn
[08:14:12] <wikibugs>	 6operations: fix trebuchet-trigger (git deploy) publish.runner arguments - https://phabricator.wikimedia.org/T98581#1271844 (10ArielGlenn)
[08:16:49] <wikibugs>	 6operations: Upgrade salt to 2014.7 (investigating) - https://phabricator.wikimedia.org/T88971#1271855 (10ArielGlenn) Close to done. See the blocking tasks.
[08:30:37] <wikibugs>	 6operations, 6Labs, 10Labs-Infrastructure: Investigate ways of getting off raid6 for labs store - https://phabricator.wikimedia.org/T96063#1271872 (10mark)
[08:32:07] <wikibugs>	 6operations, 6Labs, 10Labs-Infrastructure: Investigate ways of getting off raid6 for labs store - https://phabricator.wikimedia.org/T96063#1207452 (10mark) @Coren: Where can I see the mapping of raid array (md125 etc) to shelf? Is this documented?
[08:33:35] <icinga-wm>	 PROBLEM - Disk space on analytics1017 is CRITICAL: DISK CRITICAL - free space: /var/lib/hadoop/data/b 81128 MB (4% inode=99%): /var/lib/hadoop/data/c 81178 MB (4% inode=99%): /var/lib/hadoop/data/d 83091 MB (4% inode=99%): /var/lib/hadoop/data/e 74765 MB (3% inode=99%): /var/lib/hadoop/data/f 83114 MB (4% inode=99%): /var/lib/hadoop/data/g 83387 MB (4% inode=99%): /var/lib/hadoop/data/h 82980 MB (4% inode=99%): /var/lib/hadoop/data/i 
[08:34:04] <wikibugs>	 6operations, 6Labs, 10Labs-Infrastructure: Migrate Labs NFS storage from RAID6 to RAID10 - https://phabricator.wikimedia.org/T96063#1271874 (10mark)
[08:37:29] <icinga-wm>	 PROBLEM - Disk space on analytics1014 is CRITICAL: DISK CRITICAL - free space: /var/lib/hadoop/data/g 78636 MB (4% inode=99%): /var/lib/hadoop/data/c 82689 MB (4% inode=99%): /var/lib/hadoop/data/e 79088 MB (4% inode=99%): /var/lib/hadoop/data/f 82575 MB (4% inode=99%): /var/lib/hadoop/data/h 80635 MB (4% inode=99%): /var/lib/hadoop/data/l 83500 MB (4% inode=99%): /var/lib/hadoop/data/b 77944 MB (4% inode=99%): /var/lib/hadoop/data/k 
[08:46:18] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] puppetmaster: DRY on inclusion of hiera configuration [puppet] - 10https://gerrit.wikimedia.org/r/209272 (owner: 10Alexandros Kosiaris)
[08:46:39] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] puppetmaster: Move backups to the role class [puppet] - 10https://gerrit.wikimedia.org/r/209271 (owner: 10Alexandros Kosiaris)
[08:46:54] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] puppetmaster::logstash. Avoid out of module dependencies [puppet] - 10https://gerrit.wikimedia.org/r/209270 (owner: 10Alexandros Kosiaris)
[08:47:23] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] puppetmaster: latest to present [puppet] - 10https://gerrit.wikimedia.org/r/209269 (owner: 10Alexandros Kosiaris)
[08:47:53] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] puppetmaster::gitpuppet lint cleanups [puppet] - 10https://gerrit.wikimedia.org/r/209268 (owner: 10Alexandros Kosiaris)
[08:52:05] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] puppetmaster::config. Minor lints [puppet] - 10https://gerrit.wikimedia.org/r/209266 (owner: 10Alexandros Kosiaris)
[08:54:44] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] Add initial Debian packaging for apertium-dan-nor [debs/contenttranslation/apertium-dan-nor] - 10https://gerrit.wikimedia.org/r/195905 (https://phabricator.wikimedia.org/T91493) (owner: 10KartikMistry)
[08:55:57] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032 V: 032] Added initial Debian package for apertium-kaz [debs/contenttranslation/apertium-kaz] - 10https://gerrit.wikimedia.org/r/209463 (https://phabricator.wikimedia.org/T95876) (owner: 10KartikMistry)
[09:11:53] <wikibugs>	 6operations, 10Datasets-General-or-Unknown, 10Wikidata, 3Wikidata-Sprint-2015-04-07, and 2 others: Wikidata dumps contain old-style serialization. - https://phabricator.wikimedia.org/T74348#1271902 (10daniel) The double-check didn't turn anything up either. The dump seems to be clean.
[09:14:39] <wikibugs>	 6operations, 10Datasets-General-or-Unknown, 10Wikidata, 3Wikidata-Sprint-2015-04-07, and 2 others: Wikidata dumps contain old-style serialization. - https://phabricator.wikimedia.org/T74348#1271909 (10ArielGlenn) Great news!
[09:18:06] <wikibugs>	 6operations: snaphot1004 running dumps very slowly, investigate - https://phabricator.wikimedia.org/T98585#1271915 (10ArielGlenn) 3NEW a:3ArielGlenn
[09:20:30] <wikibugs>	 6operations: snaphot1004 running dumps very slowly, investigate - https://phabricator.wikimedia.org/T98585#1271929 (10ArielGlenn) Suspect memory leak in dumpBackup.php and/or all the stuff it uses from mw core. Running the command by hand instead of from the wrapper script shows that it is indeed somewhere in th...
[09:26:46] <icinga-wm>	 PROBLEM - puppet last run on dbstore2001 is CRITICAL puppet fail
[09:39:27] <akosiaris>	 !log uploaded to apt.wikimedia.org jessie-wikimedia: apertium-dan-nor_1.0.0~r48173-1
[09:39:27] <akosiaris>	 !log uploaded to apt.wikimedia.org jessie-wikimedia: apertium-kaz_0.1.0~r60155-1
[09:39:36] <morebots>	 Logged the message, Master
[09:39:45] <morebots>	 Logged the message, Master
[09:43:05] <icinga-wm>	 RECOVERY - puppet last run on dbstore2001 is OK Puppet is currently enabled, last run 4 seconds ago with 0 failures
[09:47:36] <grrrit-wm>	 (03PS2) 10Dereckson: Alphabetical order for groupOverrides [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209286 
[09:47:43] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Alphabetical order for groupOverrides [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209286 (owner: 10Dereckson)
[09:51:13] <grrrit-wm>	 (03PS3) 10Dereckson: Alphabetical order for groupOverrides [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209286 
[09:51:52] <grrrit-wm>	 (03CR) 10Dereckson: [C: 031] "PS3: rebased." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209286 (owner: 10Dereckson)
[10:06:39] <grrrit-wm>	 (03PS1) 10Dereckson: Add flood user group on ca.wikinews [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209699 (https://phabricator.wikimedia.org/T98576) 
[10:26:16] <icinga-wm>	 PROBLEM - HTTP 5xx req/min on graphite1001 is CRITICAL 35.71% of data above the critical threshold [500.0]
[10:40:56] <icinga-wm>	 RECOVERY - HTTP 5xx req/min on graphite1001 is OK Less than 1.00% above the threshold [250.0]
[10:42:39] <grrrit-wm>	 (03PS12) 10Giuseppe Lavagetto: etcd: create puppet module [puppet] - 10https://gerrit.wikimedia.org/r/208928 (https://phabricator.wikimedia.org/T97973) 
[10:48:19] <grrrit-wm>	 (03PS13) 10Giuseppe Lavagetto: etcd: create puppet module [puppet] - 10https://gerrit.wikimedia.org/r/208928 (https://phabricator.wikimedia.org/T97973) 
[10:52:25] <icinga-wm>	 PROBLEM - Varnishkafka Delivery Errors per minute on cp4017 is CRITICAL 11.11% of data above the critical threshold [20000.0]
[10:54:37] <T13|mobile>	 Wow.. 21.3 million this morning.
[10:55:36] <icinga-wm>	 RECOVERY - Varnishkafka Delivery Errors per minute on cp4017 is OK Less than 1.00% above the threshold [0.0]
[11:06:01] <grrrit-wm>	 (03PS2) 10Dereckson: Enable NewUserMessage on bh.wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209146 (https://phabricator.wikimedia.org/T97920) 
[11:06:12] <grrrit-wm>	 (03CR) 10Dereckson: "PS2: wmgNewUserMessageOnAutoCreate, rebased" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209146 (https://phabricator.wikimedia.org/T97920) (owner: 10Dereckson)
[11:09:16] <icinga-wm>	 PROBLEM - puppet last run on cp4014 is CRITICAL puppet fail
[11:12:11] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] puppetmaster::config Avoid out of module dependencies [puppet] - 10https://gerrit.wikimedia.org/r/209267 (owner: 10Alexandros Kosiaris)
[11:17:07] <wikibugs>	 6operations, 6Labs, 10Labs-Infrastructure: Migrate Labs NFS storage from RAID6 to RAID10 - https://phabricator.wikimedia.org/T96063#1272083 (10coren) @mark: It's in the slides (https://commons.wikimedia.org/wiki/File:WMF_Labs_storage_presentation.pdf) but also ridiculously straightforward: shelves are mapped...
[11:19:57] <wikibugs>	 6operations, 6Labs, 10Labs-Infrastructure: Migrate Labs NFS storage from RAID6 to RAID10 - https://phabricator.wikimedia.org/T96063#1272087 (10coren) A note: while it will probably increase the amount of necessary juggling, the entire setup would be //immensely// improved with raid10 if - rather than one she...
[11:22:57] <wikibugs>	 6operations, 6Analytics-Kanban: Event Logging data is not showing up in Graphite anymore since last week - https://phabricator.wikimedia.org/T98380#1272092 (10fgiunchedi) @milimetric no problem -- I'll give a bit more context :) we've switched statsd implementation from txstatsd to statsite for performance/eff...
[11:25:11] <grrrit-wm>	 (03PS1) 10KartikMistry: CX: Hide 'crh' [puppet] - 10https://gerrit.wikimedia.org/r/209704 
[11:25:53] <kart_>	 akosiaris: when you've time, https://gerrit.wikimedia.org/r/#/c/209704/
[11:26:20] <kart_>	 akosiaris: and kaz-tat :)
[11:27:17] <icinga-wm>	 RECOVERY - puppet last run on cp4014 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures
[11:27:59] <grrrit-wm>	 (03CR) 10Filippo Giunchedi: [C: 032 V: 032] sentry3: add Sentry Smart CDU detection [software/librenms] - 10https://gerrit.wikimedia.org/r/209443 (https://phabricator.wikimedia.org/T84416) (owner: 10Filippo Giunchedi)
[11:29:04] <JohannesK_WMDE>	 hey all
[11:30:05] <JohannesK_WMDE>	 one of my catgraph instances, i-00000184.eqiad.wmflabs (sylvester) is in SHUTOFF state. i tried rebooting it from the labs console, waited 10 minutes and reloaded the page no change.
[11:31:25] <JohannesK_WMDE>	 now, when i click "reboot" i get a little popup status message "failed to reboot instance sylvester". "get console output" shows a similar thing "Failed to get console output for instance sylvester."
[11:31:43] <JohannesK_WMDE>	 is there some maintenance going on or something? 
[11:31:59] <JohannesK_WMDE>	 Coren: maybe you know something?
[11:32:59] <JohannesK_WMDE>	 yesterday the instance was up, today it's down for no apparent reason.
[11:45:43] <grrrit-wm>	 (03CR) 10Mjbmr: [C: 031] Enable NewUserMessage on bh.wikipedia [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209146 (https://phabricator.wikimedia.org/T97920) (owner: 10Dereckson)
[11:49:32] <godog>	 !log deploy librenms 2fa805ff
[11:49:41] <morebots>	 Logged the message, Master
[11:51:55] <paravoid>	 <audio id="mwe_player_0" style="width:220px" poster="//127.0.0.1/static/1.26wmf4/resources/assets/file-type-icons/fileicon-ogg.png"
[11:51:58] <paravoid>	 wth
[11:53:05] <godog>	 ! where is it from?
[11:53:11] <paravoid>	 commons' homepage
[11:53:57] <wikibugs>	 6operations, 6Labs, 10Labs-Infrastructure: Migrate Labs NFS storage from RAID6 to RAID10 - https://phabricator.wikimedia.org/T96063#1272119 (10mark) Can you work out a plan, and list all the individual steps (ideally with command line invocations) on this ticket?
[11:55:22] <paravoid>	 gilles: ping?
[11:56:53] <paravoid>	                         $url = "$wgResourceBasePath/resources/assets/file-type-icons/fileicon-ogg.png";
[11:57:27] <paravoid>	 wmf-config/CommonSettings.php:$wgResourceBasePath = "//{$wmgAssetsHost}/static/$wmfVersionNumber";
[11:58:00] <paravoid>	 wmf-config/CommonSettings.php:$wmgAssetsHost = $_SERVER['SERVER_NAME'];
[12:06:16] <icinga-wm>	 PROBLEM - puppet last run on es2005 is CRITICAL puppet fail
[12:06:53] <paravoid>	 so why would SERVER_NAME be 127.0.0.1
[12:07:07] <bblack>	 yeah that's an interesting question
[12:07:23] <_joe_>	 paravoid: because it's fastcgi?
[12:07:23] <paravoid>	 but only for this particular instance, the rest of the page is fine
[12:07:40] <bblack>	 _joe_: it's generally correct, we've used that variable elsewhere too
[12:07:43] <_joe_>	 is it used in other places?
[12:07:47] <paravoid>	 yeah
[12:07:57] <bblack>	 hmmmm
[12:08:01] <wikibugs>	 6operations, 10ops-codfw, 5Patch-For-Review: Set up missing PDUs in codfw and eqiad - https://phabricator.wikimedia.org/T84416#1272124 (10fgiunchedi) looks like it is working, upstream PR at https://github.com/librenms/librenms/pull/938 I've rediscovered all ps in codfw and eqiad in row c/d, not sure what's...
[12:08:29] <_joe_>	 does that part of the page come from a specific request?
[12:08:42] <bblack>	 wait commons homepage plays audio?
[12:09:37] <bblack>	 how can I get a page with that in it?
[12:09:51] <paravoid>	 commons' main_page :)
[12:10:00] <bblack>	 not for me!
[12:10:21] <hoo>	 SERVER_NAME is probably not set if you parse that page from CLI
[12:10:26] <hoo>	 easy to test, if you want
[12:10:52] <hoo>	 (assuming that's part of the ParserOutput)
[12:10:57] <_joe_>	 hoo: it won't even be 127.0.0.1
[12:11:04] <paravoid>	 curl -H 'Host: commons.wikimedia.org' -v 'http://appservers.svc.eqiad.wmnet/wiki/Main_Page'  shows it
[12:11:33] <_joe_>	 paravoid: we don't use server name in a lot of places
[12:11:39] <_joe_>	 we use it in the 404 page
[12:11:52] <_joe_>	 and in MWVersion
[12:12:28] <_joe_>	 lemme see a 404
[12:12:45] <bblack>	 heh I do get this for the audio stuff: https://phabricator.wikimedia.org/P626
[12:12:49] <bblack>	 in a browser I mean
[12:12:56] <_joe_>	 $serv = strlen( $_SERVER['HTTP_HOST'] )                                                                                                                                                                           
[12:13:00] <_joe_>	  ? $_SERVER['HTTP_HOST']                                                                                                                                                                                        : $_SERVER['SERVER_NAME'];       
[12:13:04] <_joe_>	 this is correct
[12:13:24] <_joe_>	 $_SERVER['SERVER_NAME'] in apache would have been "mwXXXX" btw
[12:14:38] <_joe_>	 and well, let me see with MWVersion, this could be a problem
[12:14:45] <bblack>	 ....
[12:15:15] <bblack>	 in any case, at one point during this transform, we used $_SERVER['SERVER_NAME'] indirectly through wmgAssetHost for a lot of things, and still do
[12:15:22] <bblack>	 so in the general case, I don't think it's broken in practice
[12:16:20] <bblack>	 on the other hand, the whole reason for using that variable was to support transition.  IMHO, we could dump it completely at this point and just use host-relative URIs instead'
[12:17:32] <bblack>	 paravoid: where does that url line come from? an extension?
[12:17:34] <_joe_>	 no, it should work in HHVM
[12:17:37] <_joe_>	 just tested that
[12:17:39] <paravoid>	 yes, TMH
[12:17:59] <bblack>	 I wonder if the extension somehow evaluates that in a different way/order than the rest?
[12:18:32] <_joe_>	 maybe some http request to localhost, for some mysterious reason
[12:18:43] <bblack>	 or that, I guess
[12:19:20] <bblack>	 before it would've been an explicit string through a variable
[12:19:20] <icinga-wm>	 PROBLEM - Kafka Broker Messages In on analytics1021 is CRITICAL: kafka.server.BrokerTopicMetrics.AllTopicsMessagesInPerSec.FifteenMinuteRate CRITICAL: 901.898332679
[12:26:32] <bblack>	 paravoid: did you see it in a browser?
[12:26:32] <paravoid>	 yes
[12:26:32] <paravoid>	 catchpoint does too
[12:26:33] <paravoid>	 huh, now I don't see it with curl
[12:26:33] <paravoid>	 what the hell
[12:26:33] <paravoid>	 I even tried mw1256, which definitely gave me 127.0.0.1 before
[12:26:33] <icinga-wm>	 RECOVERY - puppet last run on es2005 is OK Puppet is currently enabled, last run 26 seconds ago with 0 failures
[12:26:33] <bblack>	 I have yet to see it anywhere but irc, and I've been trying
[12:26:33] <grrrit-wm>	 (03PS2) 10Filippo Giunchedi: role::cache: decommission statsite [puppet] - 10https://gerrit.wikimedia.org/r/209188 (https://phabricator.wikimedia.org/T95687) 
[12:26:33] <bblack>	 cachebust vs not, through frontends only though, not direct to applayer
[12:26:33] <paravoid>	 I got it via curl to an appserver before
[12:26:33] <paravoid>	 multiple times, in fact
[12:26:33] <grrrit-wm>	 (03CR) 10Filippo Giunchedi: [C: 032 V: 032] role::cache: decommission statsite [puppet] - 10https://gerrit.wikimedia.org/r/209188 (https://phabricator.wikimedia.org/T95687) (owner: 10Filippo Giunchedi)
[12:26:33] <bblack>	 well catchpoint gets it from the public end of things, somehow, right?
[12:26:33] <bblack>	 there must be something different about the reqs that go through varnish, I would say, except for the catchpoint bit
[12:26:33] <paravoid>	 yes
[12:26:33] <paravoid>	 I got it in a browser first, then curl'ed appservers directly and got it there as well
[12:26:34] <icinga-wm>	 RECOVERY - Kafka Broker Messages In on analytics1021 is OK: kafka.server.BrokerTopicMetrics.AllTopicsMessagesInPerSec.FifteenMinuteRate OKAY: 2895.68394135
[12:26:42] <paravoid>	 and the browser window which I still have open says
[12:26:48] <paravoid>	 mw.config.set({"wgBackendResponseTime":220,"wgHostname":"mw1256"});
[12:26:56] <icinga-wm>	 PROBLEM - Persistent high iowait on labstore1001 is CRITICAL 88.89% of data above the critical threshold [35.0]
[12:27:33] <bblack>	 hmmmm
[12:27:41] <hoo>	 paravoid: curl -L -d 'action=purge' -H 'Host: commons.wikimedia.org' 'http://appservers.svc.eqiad.wmnet/wiki/Main_Page' | grep -c 127.0.0.1
[12:27:45] <hoo>	 That's probably what you want to do
[12:27:55] <icinga-wm>	 PROBLEM - High load average on labstore1001 is CRITICAL 100.00% of data above the critical threshold [24.0]
[12:27:57] <hoo>	 If you need to purge
[12:28:08] <hoo>	 -If
[12:28:34] <paravoid>	 hmm, parser cache you think?
[12:28:44] <hoo>	 yeah, I guess
[12:28:44] <paravoid>	 Parsed by mw1008
[12:28:49] <paravoid>	 <!-- Saved in parser cache with key commonswiki:pcache:idhash:1-0!*!0!*!en!4!* and timestamp 20150508113828 and revision id 159445437
[12:28:52] <paravoid>	 was the broken one
[12:29:09] <paravoid>	 aha! mw1008 is a jobrunner
[12:29:17] <bblack>	 bblack@palladium:~$ curl -H 'Host: commons.wikimedia.org' -v 'http://appservers.svc.eqiad.wmnet/wiki/Main_Page' 2>&1 |grep 127.0.0.1
[12:29:19] <paravoid>	 and SERVER_NAME wouldn't be set in a job
[12:29:49] <paravoid>	 makes complete sense now, doesn't it
[12:29:49] <bblack>	 I've done ^ a bunch of times with no result
[12:29:59] <hoo>	 Yeah, job runners aren't running cli mode anymore
[12:30:39] <hoo>	 Is $wgServer propagated at that point?
[12:30:42] <hoo>	 We have $wgConf, so that should be accessible at least
[12:31:54] <bblack>	 yeah I'm sure that $wgResourceBasePath is set
[12:31:54] <bblack>	 it's just that it's set from the mediawiki-config stuff based on SERVER_NAME
[12:32:38] <hoo>	 Maybe wgServer could be used instead of the super global
[12:32:38] <paravoid>	 lots of stuff are set relative to $wmgAssetsHost, which is set to SERVER_NAME, which is set to 127.0.0.1 on jobrunners
[12:32:38] <bblack>	 in any case, at this point couldn't we switch the config to: $wgResourceBasePath = "/static/$wmfVersionNumber"; ?
[12:32:38] <paravoid>	 that's essentially the problem
[12:32:50] <paravoid>	 bblack: yeah, I was wondering the same but I'm not all that familiar with how these are being used
[12:33:11] <bblack>	 I think the reason it was still variable, at one point, was only some wikis were switched off of bits, and others had the bits hostname still in there
[12:41:33] <bblack>	 but that point passed by a couple days ago
[12:41:34] <hoo>	 Is that ending up in JavaScript at some point? I know that various things in MediaWiki's JS will blow up with relative URLs
[12:41:34] <hoo>	 been there at some point...
[12:41:34] <paravoid>	 yeah, almost certainly
[12:41:34] <icinga-wm>	 RECOVERY - Persistent high iowait on labstore1001 is OK Less than 50.00% above the threshold [25.0]
[12:41:34] <_joe_>	 paravoid: the jobrunner calls localhost via HTTP
[12:41:34] <paravoid>	 I know
[12:41:34] <bblack>	 I'm still doing some digging on past versions of CommonSettings, etc...
[12:41:34] <paravoid>	 and with a virtualhost with no ServerName
[12:41:35] <bblack>	 So, https://gerrit.wikimedia.org/r/#/c/207886/ introduced the $wgAssetsHost variable, which later became $wmgAssetsHost
[12:41:35] <bblack>	 the point of making sure of that is, that variable itself doesn't need to keep living or having a useful value.  it was just part of this hack.
[12:41:35] <paravoid>	 this isn't related to that
[12:41:35] <bblack>	 it is
[12:41:35] <paravoid>	 this breakage is a combination of HHVM on the jobrunners + moving away from bits
[12:42:11] <bblack>	 does the SERVER_NAME thing for HHVM on the jobrunners matter, other than problems introduced through the 'move away from bits' changes?
[12:43:05] <bblack>	 I guess what I'm saying is, we can eliminate all of the SERVER_NAME dependency introduced via the bits change stuff, that was the point of ensuring wmgAssetsHost could die
[12:43:25] <icinga-wm>	 PROBLEM - Persistent high iowait on labstore1001 is CRITICAL 100.00% of data above the critical threshold [35.0]
[12:43:32] <paravoid>	 switch to relative URLs you mean?
[12:43:57] <wikibugs>	 6operations, 5Patch-For-Review: Scale up and out our puppetmaster infrastructure - https://phabricator.wikimedia.org/T98128#1272155 (10chasemp) Sure but they are lackluster at best so if I am wondering if that is the only thing holding us back.
[12:45:04] <paravoid>	 what's the status of bits? bits still exists, right?
[12:45:17] <paravoid>	 can I just revert this before we get more 127.0.0.1 in our caches?
[12:46:04] <bblack>	 paravoid: yes, but what exactly are you going to revert? it's spread over several commits to mediawiki-config over the past 8-ish days
[12:46:16] <paravoid>	 yeah :/
[12:46:40] <paravoid>	 I could revert b0ab2380 + set wmgUseBits's default to true
[12:46:51] <paravoid>	 kinda sucks
[12:47:09] <paravoid>	 hoo: any smarter ideas on how MW could handle this case?
[12:47:26] <grrrit-wm>	 (03PS1) 10BBlack: remove $wmgAssetsHost [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209713 
[12:47:37] <bblack>	 and yes, I mean something like ^
[12:49:07] <paravoid>	 I admit I'm not too familiar with e.g. the jobrunners' context at the time of the request
[12:49:07] <paravoid>	 bblack: see above regarding JavaScript
[12:49:07] <bblack>	 it's not relative, it's absolute
[12:49:07] <paravoid>	 huh?
[12:49:07] <hoo>	 paravoid: $wgServer is maybe populated at that point, would need to check that on beta/ mw1017... if not you could pull it from $wgConf which is populated at that time
[12:49:07] <bblack>	 we'd be switching from proto-relative to host-relative, but it's not path-relative
[12:49:14] <bblack>	 they all still start with /
[12:49:46] <icinga-wm>	 RECOVERY - Persistent high iowait on labstore1001 is OK Less than 50.00% above the threshold [25.0]
[12:49:56] <bblack>	 is host-relative a problem for JS where proto-relative is not?
[12:50:02] <hoo>	 bblack: Eg. mw.loader.load can't take these
[12:50:13] <bblack>	 they were never absolute...
[12:51:14] <hoo>	 True
[12:51:20] <hoo>	 but still eg. that doesn't work
[12:51:26] <hoo>	 it might not be a problem, but it could be
[12:51:38] <bblack>	 uhm, ok?
[12:51:51] <hoo>	 mw.loader.load with a URL is mostly called from user code, so *that* might not break
[12:51:55] <hoo>	 not sure about other things
[12:52:19] <hoo>	 mw.loader.load( '//commons.wikimedia.org/whatever.js' ); << Fires a request
[12:52:25] <hoo>	 mw.loader.load( '/Foo.js' ); << wont
[12:52:39] <bblack>	 why?
[12:52:43] <hoo>	 Could be tested in beta I think
[12:52:53] <bblack>	 I mean, that seems like a nonsensical thing, I just don't get it
[12:54:08] <bblack>	 paravoid: did you make a revert patch already? :)
[12:54:27] <paravoid>	 doing so
[12:55:12] <hoo>	 bblack: That specific mw.loader thing has been declined, btw https://phabricator.wikimedia.org/T36036
[12:55:45] <hoo>	 That's just an example out of possibly many
[12:56:15] <bblack>	 I was too heh
[12:56:44] <grrrit-wm>	 (03PS1) 10Faidon Liambotis: Switch assets back to bits.wikimedia.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209716 
[12:57:22] <paravoid>	 bblack: ^
[12:57:34] <grrrit-wm>	 (03CR) 10BBlack: [C: 031] Switch assets back to bits.wikimedia.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209716 (owner: 10Faidon Liambotis)
[12:57:47] <hoo>	 wgLoadScript = "//{$wmgAssetsHost}/{$_SERVER['SERVER_NAME']}/load.php";
[12:58:05] <paravoid>	 I know, this was there before though
[12:58:11] <hoo>	 Ok, weird
[12:58:17] <paravoid>	 probably jobrunners never generate a page that reference a LoadScript
[12:58:27] <hoo>	 Very well possible
[12:58:28] <paravoid>	 kinda makes sense, LoadScript is usually in the <head> or something
[12:58:34] <bblack>	 right
[12:58:34] <paravoid>	 while jobrunners generate content
[12:58:35] <hoo>	 Those aren't parser cached
[12:58:41] <paravoid>	 yup
[12:59:36] <hoo>	 Can we set wmgUse bits to false for test or test2 and check whether wgServer is set there?
[12:59:53] <hoo>	 wmgUseBits has been false for these for a long time
[12:59:57] <hoo>	 these = test wikis
[12:59:57] <bblack>	 hoo: that ticket is about actual relative paths
[13:00:08] <bblack>	 a long time has only been a couple of days
[13:00:12] <bblack>	 in the general case
[13:00:22] <bblack>	 oh I guess a week for testwikis
[13:00:34] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 032] Switch assets back to bits.wikimedia.org [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209716 (owner: 10Faidon Liambotis)
[13:00:56] <hoo>	 bblack: No, test wikis haven't used bits in quite a while to ease debugging of scripts
[13:00:58] <bblack>	 hoo: re that ticket, it still seems to be about actual relative URLs, not absolute from root path?
[13:01:02] <hoo>	 at least months
[13:01:19] <hoo>	 bblack: I think it's absolute from root path
[13:02:33] <bblack>	 well
[13:02:47] <bblack>	 the only real "absolute" is a URL that starts with a protocol
[13:02:56] <bblack>	 we're talking about a diff between proto-rel and host-rel, nothing here is path-rel
[13:02:58] <logmsgbot>	 !log faidon Synchronized wmf-config/InitialiseSettings.php: Switch assets back to bits.wikimedia.org (duration: 00m 14s)
[13:03:02] <bblack>	 the ticket seems to be about path-rel
[13:03:04] <morebots>	 Logged the message, Master
[13:03:21] <logmsgbot>	 !log faidon Synchronized wmf-config/CommonSettings.php: Switch assets back to bits.wikimedia.org (duration: 00m 15s)
[13:03:22] <hoo>	 bblack: Do, sorry, misphrased... I mean even rel to root path doesn't work
[13:03:25] <hoo>	 * Doh
[13:03:28] <morebots>	 Logged the message, Master
[13:03:42] <paravoid>	 <link rel="stylesheet" href="//bits.wikimedia.org/commons.wikimedia.org/load.php?debug=false&amp;lang=en&amp;modules=ext.gadget.AjaxQuickDelete%2CCollapsibleTemplates%2CLanguageSelect%2CLong-Image-Names-in-Categories%2CPictureOfTheYearEnhancements%2CSlideshow%2CStockphoto%2CUploadWizard%2CWatchlistNotice%2CWikiMiniAtlas%2CZoomViewer%2Cfastcci%7Cext.uls.nojs%7Cext.visualEditor.viewPageTarget.
[13:03:48] <paravoid>	 noscript%7Cext.wikihiero%2CwikimediaBadges%7Cmediawiki.legacy.commonPrint%2Cshared%7Cmediawiki.sectionAnchor%7Cmediawiki.skinning.interface%7Cmediawiki.ui.button%7Cmw.PopUpMediaTransform%7Cskins.vector.styles%7Cwikibase.client.init&amp;only=styles&amp;skin=vector&amp;*" />
[13:03:54] <paravoid>	 so, bits are live again
[13:04:49] <paravoid>	 bits traffic ramping up
[13:05:10] <bblack>	 yeah
[13:05:54] <paravoid>	 you might be right and we could ditch the proto-relative URL altogether, I just wasn't feeling very hot with trying it in prod
[13:06:17] <paravoid>	 and since this was causing cache pollution, essentially
[13:07:11] <hoo>	 We could try it on beta for a bit (probably at least the weekend, though)
[13:07:55] <bblack>	 no, hoo is correct
[13:08:06] <icinga-wm>	 PROBLEM - puppet last run on db2059 is CRITICAL puppet fail
[13:08:21] <bblack>	 mediawiki.js tests for whether the load() argument is a URL or some other kind of string via:
[13:08:24] <bblack>	 if ( /^(https?:)?\/\//.test( modules ) ) {
[13:08:34] <bblack>	 so, no leading '//', not a URL, as far as it's concerned
[13:08:34] <paravoid>	 lol
[13:09:00] <paravoid>	 good catch hoo ;-)
[13:09:31] <bblack>	 @param {string|Array} modules Either the name of a module, array of modules, *  or a URL of an external script or style
[13:09:53] <paravoid>	 you know
[13:09:59] <bblack>	 I guess in theory, that regex could be updated to support '/foo.js', assuming that the name of a module can't start with a slash
[13:09:59] <paravoid>	 this may have been because of bits
[13:10:16] <hoo>	 paravoid: Uh, we took down logos
[13:10:20] <hoo>	 revert revert?
[13:10:20] <paravoid>	 uh oh
[13:10:28] <paravoid>	 why the hell
[13:10:37] <hoo>	 https://bits.wikimedia.org/static/images/project-logos/wikidata.png
[13:10:38] <hoo>	 404
[13:10:50] <hoo>	 Not sure which sites are affected
[13:10:54] <hoo>	 Wikidata at least
[13:10:58] <paravoid>	 all apparently
[13:11:07] <bblack>	 it's the /w/
[13:11:12] <bblack>	 in the revert...
[13:11:37] <paravoid>	 a62a097f8e6a39a903783f48e8c81812f915670e
[13:11:44] <bblack>	 yes
[13:11:47] <paravoid>	 grumble
[13:11:53] <bblack>	 we can go further down the rabbit hole and strip the /w/
[13:12:29] <grrrit-wm>	 (03PS1) 10Faidon Liambotis: Revert "Switch assets back to bits.wikimedia.org" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209718 
[13:12:37] <grrrit-wm>	 (03PS1) 10Nemo bis: Increase jobrunner::runners_basic [puppet] - 10https://gerrit.wikimedia.org/r/209719 
[13:12:39] <grrrit-wm>	 (03CR) 10Faidon Liambotis: [C: 032] Revert "Switch assets back to bits.wikimedia.org" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209718 (owner: 10Faidon Liambotis)
[13:12:46] <grrrit-wm>	 (03Merged) 10jenkins-bot: Revert "Switch assets back to bits.wikimedia.org" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209718 (owner: 10Faidon Liambotis)
[13:13:12] <grrrit-wm>	 (03PS2) 10Nemo bis: Increase jobrunner::runners_basic [puppet] - 10https://gerrit.wikimedia.org/r/209719 
[13:13:21] <bblack>	 oh
[13:13:32] <bblack>	 or that...
[13:13:53] <bblack>	 I think the fix on top of the revert was just:
[13:13:56] <bblack>	 -       $wgExtensionAssetsPath = "//{$wmgAssetsHost}/w/static/$wmfVersionNumber/extensions";
[13:13:57] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Increase jobrunner::runners_basic [puppet] - 10https://gerrit.wikimedia.org/r/209719 (owner: 10Nemo bis)
[13:13:59] <bblack>	 -       $wgStyleSheetPath = "//{$wmgAssetsHost}/w/static/$wmfVersionNumber/skins";
[13:14:02] <bblack>	 -       $wgResourceBasePath = "//{$wmgAssetsHost}/w/static/$wmfVersionNumber";
[13:14:05] <bblack>	 +       $wgExtensionAssetsPath = "//{$wmgAssetsHost}/static/$wmfVersionNumber/extensions";
[13:14:08] <bblack>	 +       $wgStyleSheetPath = "//{$wmgAssetsHost}/static/$wmfVersionNumber/skins";
[13:14:11] <bblack>	 +       $wgResourceBasePath = "//{$wmgAssetsHost}/static/$wmfVersionNumber";
[13:14:20] <logmsgbot>	 !log faidon Synchronized wmf-config/CommonSettings.php: revert bits.wm.org change (duration: 00m 12s)
[13:14:21] <grrrit-wm>	 (03CR) 10Phuedx: Import lists for the Browse experiment on beta (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209242 (https://phabricator.wikimedia.org/T95446) (owner: 10Phuedx)
[13:14:27] <bblack>	 the location of /static/ changed since the change you're reverting, basically
[13:14:45] <morebots>	 Logged the message, Master
[13:14:50] <logmsgbot>	 !log faidon Synchronized wmf-config/InitialiseSettings.php: revert bits.wm.org change (duration: 00m 12s)
[13:14:55] <morebots>	 Logged the message, Master
[13:15:28] <bblack>	 ugh, this is getting silly
[13:15:34] <paravoid>	 it is
[13:16:13] <paravoid>	 I haven't been following those changes at all
[13:16:30] <paravoid>	 I'm just trying to get us back to where we were and failing, clearly
[13:16:38] <bblack>	 we could just wake up ori, he knows these changes best
[13:16:46] <grrrit-wm>	 (03PS3) 10Nemo bis: Increase jobrunner::runners_basic [puppet] - 10https://gerrit.wikimedia.org/r/209719 
[13:16:56] <icinga-wm>	 RECOVERY - High load average on labstore1001 is OK Less than 50.00% above the threshold [16.0]
[13:17:25] <wikibugs>	 6operations, 5Patch-For-Review: Scale up and out our puppetmaster infrastructure - https://phabricator.wikimedia.org/T98128#1272248 (10Joe) @chasemp That and the fact we don't want to spam our (pretty big) public + **private** repo on every host I guess.
[13:17:28] <logmsgbot>	 !log faidon Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 14s)
[13:17:30] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Increase jobrunner::runners_basic [puppet] - 10https://gerrit.wikimedia.org/r/209719 (owner: 10Nemo bis)
[13:17:40] <morebots>	 Logged the message, Master
[13:17:43] <bblack>	 honestly, there's been a lot of complexity in it, with how the paths changed in the meantime, etc
[13:17:53] <logmsgbot>	 !log faidon Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 12s)
[13:17:55] <jzerebecki>	 yay! logo is back
[13:18:00] <morebots>	 Logged the message, Master
[13:19:05] <sjoerddebruin>	 Hm, isn't this the old logo of MediaWiki? https://www.mediawiki.org/wiki/MediaWiki
[13:19:15] <sjoerddebruin>	 I thought it was different...
[13:19:32] <wikibugs>	 6operations, 5Patch-For-Review: Scale up and out our puppetmaster infrastructure - https://phabricator.wikimedia.org/T98128#1272249 (10chasemp) If then public repo is too big I guess OK, I am not sure what problem that represents really though.  The private repo is a failed idea entirely I was under the impres...
[13:19:48] <bblack>	 I'm gonna text ori
[13:19:54] <paravoid>	 I'm calling
[13:19:55] <grrrit-wm>	 (03PS4) 10Nemo bis: Increase jobrunner::runners_basic [puppet] - 10https://gerrit.wikimedia.org/r/209719 
[13:20:11] <bblack>	 ok
[13:20:12] <hoo>	 sjoerddebruin: Could be, ori pushed them to git, thus they're no longer coming from upload.wm.o
[13:20:18] <paravoid>	 no answer
[13:20:20] <sjoerddebruin>	 ok
[13:20:30] <grrrit-wm>	 (03CR) 10Nemo bis: "Can someone find the change which switched the processes from "nice" to "user"? https://ganglia.wikimedia.org/latest/?c=Jobrunners%20eqiad" [puppet] - 10https://gerrit.wikimedia.org/r/209719 (owner: 10Nemo bis)
[13:22:20] <bblack>	 I don't think my /w/ path fix was the best way forward on top of that revert anyways
[13:22:35] <bblack>	 logos were a special case in the changes - was anything other than logos broken with the revert, that we saw?
[13:24:26] <icinga-wm>	 RECOVERY - puppet last run on db2059 is OK Puppet is currently enabled, last run 12 seconds ago with 0 failures
[13:25:16] <icinga-wm>	 PROBLEM - High load average on labstore1001 is CRITICAL 87.50% of data above the critical threshold [24.0]
[13:25:21] <bblack>	 logos got moved from upload to a new path, *after* the useBits we were reverting
[13:25:37] <paravoid>	 yup
[13:25:56] <paravoid>	 in two commits too
[13:26:02] <paravoid>	 I saw that before I re-reverted
[13:26:05] <bblack>	 so we could revert those changes too, if logos was the only brakage
[13:26:10] <bblack>	 *beakage
[13:26:19] <bblack>	 heh
[13:26:26] <paravoid>	 I wasn't looking forward to reverting those commits in the middle of a "logos outage" :P
[13:26:34] <bblack>	 were other resources loading?
[13:26:46] <paravoid>	 yes, that we saw of
[13:27:48] <grrrit-wm>	 (03CR) 10Nemo bis: "The fact was on January 8. There were no puppet changes around that date, so the cause is probably just the reimaging and the new HHVM pac" [puppet] - 10https://gerrit.wikimedia.org/r/209719 (owner: 10Nemo bis)
[13:29:50] <grrrit-wm>	 (03PS1) 10BBlack: Reverts 3x changes (logo paths + removal of wmgUseBits), sets wmgUseBits = true [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209722 
[13:30:16] <paravoid>	 so this has been broken for 3 days now, right?
[13:30:23] <paravoid>	 and the only impact that we saw was TMH?
[13:31:12] <bblack>	 well, it's been broken for $various days depending on the wiki
[13:31:17] <bblack>	 commons, only since the final set..
[13:32:03] <bblack>	 https://gerrit.wikimedia.org/r/#/c/208990/ <- May 5
[13:32:23] <bblack>	 before that, it was live on several language wikis for a period of a few days, though
[13:32:52] <bblack>	 so yes, TMH breakage vs risk of another bad-revert, I guess
[13:33:49] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] CX: Hide 'crh' [puppet] - 10https://gerrit.wikimedia.org/r/209704 (owner: 10KartikMistry)
[13:34:13] <bblack>	 we could also take another angle, and leave all the other bits changes in place, but just fix $wgResourceBasePath to hardcode a bits hostname again
[13:34:58] <wikibugs>	 6operations, 5Patch-For-Review: Scale up and out our puppetmaster infrastructure - https://phabricator.wikimedia.org/T98128#1272267 (10chasemp) My thinking is we need to change how private things are stored and distributed, and exported resources are not a great solution, and I have a solution for distributing...
[13:36:36] <icinga-wm>	 RECOVERY - High load average on labstore1001 is OK Less than 50.00% above the threshold [16.0]
[13:37:26] <bblack>	 as near as I can tell checking how URLs would look/work in a browser, the 3x revert change would DTRT
[13:38:02] <wikibugs>	 6operations, 5Patch-For-Review: Scale up and out our puppetmaster infrastructure - https://phabricator.wikimedia.org/T98128#1272277 (10Joe) @chasemp I'm not against this in principle, to be clear.  But I think that scaling the puppetmasters won't be needed if we make our puppet run on ruby 2.1 and upgrade to j...
[13:40:26] <grrrit-wm>	 (03PS1) 10BBlack: revert state of wgResourceBasePath to before bits-related changes [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209724 
[13:40:42] <bblack>	 ^ that's the simpler "other angle" approach, which just attacks the variable that TMH ends up using, directly.
[13:41:38] <paravoid>	 I don't feel confident enough for either
[13:41:40] <paravoid>	 if you do, go for it
[13:42:06] <paravoid>	 what about the '/w/' bit?
[13:42:33] <paravoid>	 I guess that's redundant, since the else block won't be reached
[13:42:39] <paravoid>	 for now at least
[13:43:17] <icinga-wm>	 PROBLEM - High load average on labstore1001 is CRITICAL 50.00% of data above the critical threshold [24.0]
[13:43:26] <_joe_>	 uhm high load again?
[13:43:28] <bblack>	 the /w/ part shouldn't be an issue
[13:43:41] <mark>	 coren and I are working on it
[13:43:49] <bblack>	 it's only /w/ when it's on a $domain, not when it's on bits
[13:43:53] <bblack>	 for static stuff
[13:43:59] <_joe_>	 mark: ok :)
[13:44:16] <bblack>	 e.g.: http://bits.wikimedia.org/static-1.26wmf4/resources/assets/file-type-icons/fileicon-ogg.png
[13:44:33] <mark>	 preparing for moving away from raid6
[13:44:42] <bblack>	 (is a URL in the form that 209724 would put them into)
[13:44:47] <bblack>	 (which works)
[13:46:35] <bblack>	 I think 209724 is probably the right thing, if we want to unbreak TMH
[13:47:06] <bblack>	 I haven't deployed a mw-config in a while, what's the step after +2 -> merge, or is there one anymore?
[13:49:32] <bblack>	 ?
[13:51:14] <hoo>	 bblack: sync-file ... on tine
[13:51:15] <hoo>	 * tin
[13:51:29] <_joe_>	 you pull the changes to tin, and do sync-file
[13:51:37] <_joe_>	 wikitech is accurate, usually
[13:51:48] <bblack>	 if one can search up which page on wikitech :P
[13:51:49] <hoo>	 https://wikitech.wikimedia.org/wiki/How_to_deploy_code
[13:52:01] <hoo>	 One of the best docs on wikitech, very accurate :)
[13:52:17] <bblack>	 very huge!
[13:52:32] <hoo>	 You only need a small fraction of that
[13:52:48] <bblack>	 paravoid: objections to 209724 other than general unease?
[13:55:02] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032 V: 04-1] "gbp:error: upstream/0.2.1_r57554 is not a valid treeish" [debs/contenttranslation/apertium-kaz-tat] - 10https://gerrit.wikimedia.org/r/209468 (https://phabricator.wikimedia.org/T95876) (owner: 10KartikMistry)
[13:59:45] <grrrit-wm>	 (03PS2) 10BBlack: revert state of wgResourceBasePath to before bits-related changes [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209724 
[13:59:53] <bblack>	 ^ added explanatory commitmsg
[14:00:03] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 032] puppetmaster: use require_packages [puppet] - 10https://gerrit.wikimedia.org/r/209257 (owner: 10Alexandros Kosiaris)
[14:04:59] <bblack>	 ok, well
[14:05:14] <bblack>	 any other objectors to try 209724 to fix the TMH thing?
[14:09:25] <_joe_>	 *crickets*
[14:10:31] <paravoid>	 I have no objections if you feel that's ok
[14:11:45] <icinga-wm>	 PROBLEM - Persistent high iowait on labstore1001 is CRITICAL 55.56% of data above the critical threshold [35.0]
[14:13:06] <grrrit-wm>	 (03CR) 10BBlack: [C: 032] revert state of wgResourceBasePath to before bits-related changes [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209724 (owner: 10BBlack)
[14:13:36] <logmsgbot>	 !log bblack Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 13s)
[14:13:48] <morebots>	 Logged the message, Master
[14:14:56] <icinga-wm>	 RECOVERY - Persistent high iowait on labstore1001 is OK Less than 50.00% above the threshold [25.0]
[14:15:50] <bblack>	 how do I force rendering the page through jobrunner somehow, to see the effect in the failing case?
[14:16:13] <bblack>	 (I've still never gotten anything but a correct URL through appservers.svc.eqiad.wmnet)
[14:17:12] <_joe_>	 bblack: mh you go to mw1001 and request said url to 127.0.0.1:$something
[14:17:25] <_joe_>	 I don't remember if apache is running on port 80 there
[14:17:29] <bblack>	 ok
[14:17:41] <_joe_>	 bblack: mh not sure that would work btw
[14:17:53] <_joe_>	 it may be redered only via submitting a job to that
[14:18:11] <_joe_>	 aaron would be the source of authority on this
[14:18:15] <grrrit-wm>	 (03PS1) 10coren: Add codfw to ganglia's labs nfs cluster [puppet] - 10https://gerrit.wikimedia.org/r/209730 
[14:18:34] <_joe_>	 (sorry, I'm in the middle of a test for a complex bug report)
[14:21:22] <grrrit-wm>	 (03CR) 10Mark Bergsma: [C: 031] Add codfw to ganglia's labs nfs cluster [puppet] - 10https://gerrit.wikimedia.org/r/209730 (owner: 10coren)
[14:21:38] <grrrit-wm>	 (03CR) 10coren: [C: 032] Add codfw to ganglia's labs nfs cluster [puppet] - 10https://gerrit.wikimedia.org/r/209730 (owner: 10coren)
[14:23:29] <paravoid>	 no that won't work, you need to submit a job
[14:24:10] * bblack google's "submit a job"
[14:24:14] <bblack>	 :)
[14:24:22] <paravoid>	 lol
[14:24:37] <paravoid>	 yeah I'm unsure how you would do that except editing a template
[14:24:59] <bblack>	 in any case, nobody's screaming
[14:25:06] <bblack>	 so I think this didn't break anything
[14:25:19] <bblack>	 it should send some traffic back to bits, too, but not all
[14:26:22] <bblack>	 so, do we have some complex things to deal with on trying to grep + purge broken cache entries now?
[14:27:01] <paravoid>	 nope :)
[14:27:28] <bblack>	 I thought we had the 127.0.0.1 stuff in a parser cache somewhere?
[14:27:28] <_joe_>	 paravoid: well there is a specific HTTP call to do :)
[14:27:28] <paravoid>	 yes :)
[14:27:37] <paravoid>	 yes there is ?action=purge
[14:27:43] <_joe_>	 and given time to think about it, I may be able to figure that out
[14:27:49] <paravoid>	 but it would be a needle in a haystack
[14:27:51] <bblack>	 yeah but I assume this wasn't *just* commonswiki Main_Page
[14:28:04] <bblack>	 it's anywhere TMH was, that got run through a jobrunner
[14:28:18] <_joe_>	 bblack: grep 127.0.0.1 ibdata.0
[14:28:21] <_joe_>	 :P
[14:28:23] <bblack>	 lol
[14:28:36] <_joe_>	 yeah I'm joking
[14:28:40] <_joe_>	 it's surely myisam
[14:29:24] <bblack>	 anyways, why was the bad cached page only coming up through certain mw* appservers but not others?
[14:29:41] <paravoid>	 why do you say so?
[14:29:58] <paravoid>	 at point the page got purged, not sure if we or some edit(or) did that
[14:30:02] <bblack>	 because I hit appservers.svc.eqiad.wmnet several times, back before all the revert attempts, and was never able to get the bad page
[14:30:18] <paravoid>	 yeah, the page got purged and regenerated in a webserver properly
[14:30:22] <bblack>	 I've still never seen it from any fetch of my own heh
[14:30:26] <bblack>	 hmmm ok
[14:31:37] <bblack>	 anyways, clearly the wmf-config side of the bits excursion has wandered off into dangerous territory.  When ori gets in we can chat about documenting and planning the further changes better...
[14:31:58] <bblack>	 sorry for the mess :(
[14:32:24] <paravoid>	 not your mess :)
[14:32:30] <paravoid>	 I should be sorry for the mess of my revert :)
[14:32:43] <bblack>	 well, it's half my mess at least, I've been encouraging/pushing on this stuff
[14:33:19] <bblack>	 (and helping over on our side w/ VCL)
[14:35:07] <wikibugs>	 6operations, 10Mathoid, 6Services, 5Patch-For-Review: Standardise Mathoid's deployment - https://phabricator.wikimedia.org/T97124#1272382 (10mobrovac) @akosiaris, yup, I am aware of that. The deploy repo has been created (yet need to structure it and import the src and modules). As for   >>! In T97124#1265...
[14:36:04] <wikibugs>	 6operations, 10ops-codfw, 5Patch-For-Review: Set up missing PDUs in codfw and eqiad - https://phabricator.wikimedia.org/T84416#1272383 (10fgiunchedi) all psu are reporting metrics, however codfw D2 through D8 are reporting voltage but no current, nothing's attached there I guess?
[14:38:39] <grrrit-wm>	 (03CR) 10Mobrovac: [C: 031] "Great, thnx @GWicke. LGTM" [puppet] - 10https://gerrit.wikimedia.org/r/208193 (https://phabricator.wikimedia.org/T97840) (owner: 10GWicke)
[14:47:18] <grrrit-wm>	 (03CR) 10KartikMistry: "Oops. Sorry. Pushed tag now." [debs/contenttranslation/apertium-kaz-tat] - 10https://gerrit.wikimedia.org/r/209468 (https://phabricator.wikimedia.org/T95876) (owner: 10KartikMistry)
[14:48:03] <godog>	 bd808: no problem! not sure how it ever worked tbh before https://gerrit.wikimedia.org/r/#/c/209462/
[14:50:35] <kart_>	 akosiaris: once you push kaz-tat, can you restart apertium-apy on sca1001/sca1002?
[14:50:38] <kart_>	 Thanks.
[14:51:20] <akosiaris>	 kart_: push kaz-kat ? why is restarting apertium-apy dependent on kaz-kat ? 
[14:54:11] <kart_>	 akosiaris: no. it is for all packages uploaded.
[14:55:15] <kart_>	 akosiaris: well, wait. I have to make sure they are installed first :)
[14:55:23] <akosiaris>	 kart_: :-)
[14:55:46] <kart_>	 will submit patch after kaz-tat is done.
[14:56:29] <grrrit-wm>	 (03CR) 10Alex Monk: "Probably also solves this problem:" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209713 (owner: 10BBlack)
[14:57:19] <grrrit-wm>	 (03PS1) 10Ottomata: Add ability to configure spark to run in standalone mode [puppet/cdh] - 10https://gerrit.wikimedia.org/r/209734 
[14:58:19] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032] Add ability to configure spark to run in standalone mode [puppet/cdh] - 10https://gerrit.wikimedia.org/r/209734 (owner: 10Ottomata)
[15:05:25] <icinga-wm>	 PROBLEM - Disk space on analytics1016 is CRITICAL: DISK CRITICAL - free space: /var/lib/hadoop/data/f 76223 MB (4% inode=99%): /var/lib/hadoop/data/c 82967 MB (4% inode=99%): /var/lib/hadoop/data/e 79602 MB (4% inode=99%): /var/lib/hadoop/data/i 84093 MB (4% inode=99%): /var/lib/hadoop/data/h 83767 MB (4% inode=99%): /var/lib/hadoop/data/k 81191 MB (4% inode=99%): /var/lib/hadoop/data/d 82086 MB (4% inode=99%): /var/lib/hadoop/data/g 
[15:05:50] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [V: 032] "LGTM, merging" [debs/contenttranslation/apertium-kaz-tat] - 10https://gerrit.wikimedia.org/r/209468 (https://phabricator.wikimedia.org/T95876) (owner: 10KartikMistry)
[15:05:51] <ottomata>	 whaa wow
[15:06:59] <wikibugs>	 6operations: upgrade salt on production cluster - https://phabricator.wikimedia.org/T98580#1272427 (10JohnLewis)
[15:10:31] <wikibugs>	 6operations, 10ops-codfw, 5Patch-For-Review: Set up missing PDUs in codfw and eqiad - https://phabricator.wikimedia.org/T84416#1272440 (10Papaul) In D1 we have the 2 mx80 and in D6 we have the DB servers connected the other racks have nothing but just the asw's and msw's
[15:11:43] <wikibugs>	 6operations: snaphot1004 running dumps very slowly, investigate - https://phabricator.wikimedia.org/T98585#1272442 (10greg) snapshot1004 has also been causing us tons of grief during deploys:  ``` May 6th: 20:14 twentyafterfour: ignore all rumors of scap failures, the scaps were successful, with the exception of...
[15:11:49] <wikibugs>	 6operations: snaphot1004 running dumps very slowly, investigate - https://phabricator.wikimedia.org/T98585#1272443 (10greg)
[15:14:33] <grrrit-wm>	 (03CR) 10GWicke: "We could write a wrapper script that does the compression too, but otoh it seems that you plan to manage original dumps, workspaces etc in" [dumps] (ariel) - 10https://gerrit.wikimedia.org/r/206849 (owner: 10ArielGlenn)
[15:15:06] <icinga-wm>	 RECOVERY - Disk space on analytics1017 is OK: DISK OK
[15:15:16] <icinga-wm>	 RECOVERY - Disk space on analytics1016 is OK: DISK OK
[15:15:55] <icinga-wm>	 RECOVERY - Disk space on analytics1014 is OK: DISK OK
[15:18:32] <grrrit-wm>	 (03CR) 10BBlack: [C: 04-2] "Sorry, this was one of several options for fixing some related breakage we saw earlier, and I ended up merging a different option. If the" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209713 (owner: 10BBlack)
[15:19:27] <grrrit-wm>	 (03CR) 10BBlack: [C: 04-2] "Went with a different option, but holding this here in case we need it..." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209722 (owner: 10BBlack)
[15:21:21] <greg-g>	 anyone else getting loss of session data errors on wikitech lately?
[15:21:26] <greg-g>	 happened yesterday and today
[15:21:45] <bblack>	 I think I recall someone mentioning that...
[15:21:53] <bblack>	 and some sort of explanation, but I don't know what it was
[15:22:59] <greg-g>	 helpful :P
[15:23:30] <greg-g>	 alright...
[15:23:32] <greg-g>	 happened again
[15:23:41] <grrrit-wm>	 (03Abandoned) 10John F. Lewis: move scap::master to hieradata [puppet] - 10https://gerrit.wikimedia.org/r/208801 (owner: 10John F. Lewis)
[15:23:41] <greg-g>	 I got to save one edit, and now it's doing it to me again
[15:23:56] <bblack>	 I think it was robh maybe? that was having saving problems there
[15:24:06] <grrrit-wm>	 (03PS5) 10John F. Lewis: Deploy mira as codfw deployment server [puppet] - 10https://gerrit.wikimedia.org/r/208723 (https://phabricator.wikimedia.org/T95436) 
[15:24:11] <greg-g>	 I sure do love entering in 2fa tokens :)
[15:24:13] <bd808>	 greg-g: I've hit that on and off for the last 2-3 days.
[15:24:16] <grrrit-wm>	 (03PS6) 10John F. Lewis: Deploy mira as codfw deployment server [puppet] - 10https://gerrit.wikimedia.org/r/208723 (https://phabricator.wikimedia.org/T95436) 
[15:24:24] <akosiaris>	 !log uploaded to apt.wikimedia.org jessie-wikimedia: apertium-kaz-tat_0.2.1~r57554-1
[15:24:38] <bd808>	 fwiw the last time I just kept hitting save and it eventually worked :/
[15:25:04] <JohnLewis>	 greg-g: always :p
[15:25:06] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Deploy mira as codfw deployment server [puppet] - 10https://gerrit.wikimedia.org/r/208723 (https://phabricator.wikimedia.org/T95436) (owner: 10John F. Lewis)
[15:25:07] <greg-g>	 now it's telling me I have cookies disabled (I do, but I whitelisted *.wikimedia.org etc)
[15:25:35] <JohnLewis>	 bblack: last I remember was Andrew saying it a memcached or keystone issue or along that line of tools?
[15:25:43] <greg-g>	 there, logged in (with no changes to my cookies settings)
[15:26:06] <akosiaris>	 last time it was keystone killing the performance of the mysql db 
[15:26:14] <greg-g>	 !log multiple independent reports of wikitech wiki having session data errors
[15:26:15] <akosiaris>	 but I think that was fixed
[15:26:22] <morebots>	 Logged the message, Master
[15:26:32] <greg-g>	 suuuure, morebots can make edits there easily ;)
[15:26:53] <akosiaris>	 it took it 2 minutes...
[15:26:54] <JohnLewis>	 akosiaris: since you're here, who is on ops duty next week? :)
[15:27:05] <grrrit-wm>	 (03PS14) 10Giuseppe Lavagetto: etcd: create puppet module [puppet] - 10https://gerrit.wikimedia.org/r/208928 (https://phabricator.wikimedia.org/T97973) 
[15:27:13] <JohnLewis>	 (since you can search it up accurately while I doubt my access has anything up to date :) )
[15:27:16] <_joe_>	 godog: I think ^^ is ready
[15:27:17] <bblack>	 wtf is all this on silver that runs within MW in some way?
[15:27:18] <bblack>	 www-data 21498 21496  0 14:55 pts/4    00:00:00 sh -c /bin/bash '/srv/mediawiki/php-1.26wmf4/includes/limit.sh' '/bin/bash -c '\''read -er -p '\''\'\'''\''> '\''\'\'''\'' && echo "$REPLY"'\''' 'MW_INCLUDE_STDERR=;MW_CPU_LIMIT=50; MW_CGROUP='\''/sys/fs/cgroup/memory/mediawiki/job'\''; MW_MEM_LIMIT=307200; MW_FILE_SIZE_LIMIT=102400; MW_WALL_CLOCK_LIMIT=0; MW_USE_LOG_PIPE=yes'
[15:27:20] <_joe_>	 (it's already tested)
[15:27:24] <bblack>	 www-data 21499 21498  0 14:55 pts/4    00:00:00 /bin/bash /srv/mediawiki/php-1.26wmf4/includes/limit.sh /bin/bash -c 'read -er -p '\''> '\'' && echo "$REPLY"' MW_INCLUDE_STDERR=;MW_CPU_LIMIT=50; MW_CGROUP='/sys/fs/cgroup/memory/mediawiki/job'; MW_MEM_LIMIT=307200; MW_FILE_SIZE_LIMIT=102400; MW_WALL_CLOCK_LIMIT=0; MW_USE_LOG_PIPE=yes
[15:27:28] <greg-g>	 MUTHER SH!T FACK
[15:27:31] <grrrit-wm>	 (03PS1) 10Ottomata: Set default spark standalone parameters to undef to use spark's builtin defaults [puppet/cdh] - 10https://gerrit.wikimedia.org/r/209738 
[15:27:32] <bblack>	 we have long-running jobs in wikitech somehow?
[15:27:36] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Set default spark standalone parameters to undef to use spark's builtin defaults [puppet/cdh] - 10https://gerrit.wikimedia.org/r/209738 (owner: 10Ottomata)
[15:27:42] <_joe_>	 bblack: probably OSM
[15:27:52] <bblack>	 on wikitech?
[15:28:05] <_joe_>	 OpenStackManager
[15:28:12] <grrrit-wm>	 (03PS2) 10Ottomata: Set default spark standalone parameters to undef to use spark's builtin defaults [puppet/cdh] - 10https://gerrit.wikimedia.org/r/209738 
[15:28:16] <akosiaris>	 JohnLewis: I think apergos 
[15:28:22] <bblack>	 oh
[15:28:25] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032] Set default spark standalone parameters to undef to use spark's builtin defaults [puppet/cdh] - 10https://gerrit.wikimedia.org/r/209738 (owner: 10Ottomata)
[15:28:25] <apergos>	 eh?
[15:28:32] <JohnLewis>	 akosiaris: okay thanks
[15:28:33] <akosiaris>	 apergos: clinic duty next week
[15:28:34] <bblack>	 anyways, whatever that is has been running for half an hour
[15:28:35] <greg-g>	 !log wikitech's session data errors are transient, hitting save multiple times will eventually work
[15:28:36] <apergos>	 yes.
[15:28:37] <bblack>	 is that normal?
[15:28:43] <morebots>	 Logged the message, Master
[15:28:55] <apergos>	 ah I saw one of those earlier today in fact
[15:29:00] <apergos>	 session data error
[15:29:04] <bblack>	 krenair  pts/4    hooft.esams.wiki 14:55   33:05   0.21s  0.00s /bin/bash /usr/local/bin/mwscript eval.php labswiki
[15:29:07] <bblack>	 Krenair: ^
[15:29:10] <_joe_>	 bblack: no idea, limits.sh is the standard way to launch scripts from mediawiki
[15:29:17] <bblack>	 it was launched from CLI
[15:30:19] <grrrit-wm>	 (03PS1) 10KartikMistry: Install new Apertium packages for ContentTranslation [puppet] - 10https://gerrit.wikimedia.org/r/209739 
[15:31:28] <bd808>	 greg-g: I've got a mediawiki-config change to beta cluster that I found last night -- https://gerrit.wikimedia.org/r/#/c/209682/ -- I'd like to deploy it
[15:31:53] <godog>	 _joe_: ok! will take a look
[15:32:19] <greg-g>	 bd808: OK!
[15:32:30] <grrrit-wm>	 (03PS2) 10BryanDavis: beta: fix symlinks in docroot/bits/static/master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209682 
[15:32:41] <grrrit-wm>	 (03CR) 10BryanDavis: [C: 032] beta: fix symlinks in docroot/bits/static/master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209682 (owner: 10BryanDavis)
[15:32:44] <grrrit-wm>	 (03CR) 10John F. Lewis: "recheck" [puppet] - 10https://gerrit.wikimedia.org/r/208723 (https://phabricator.wikimedia.org/T95436) (owner: 10John F. Lewis)
[15:32:46] <grrrit-wm>	 (03Merged) 10jenkins-bot: beta: fix symlinks in docroot/bits/static/master [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209682 (owner: 10BryanDavis)
[15:34:04] <logmsgbot>	 !log bd808 Synchronized docroot/bits/static/master: Fix symlinks in docroot/bits/static/master (duration: 00m 12s)
[15:35:55] <Krenair>	 bblack, okay, yes I ran that command
[15:35:59] <Krenair>	 what's your point?
[15:36:13] <bblack>	 just curious if it's related, or if you knew anything about wikitech ongoing issues
[15:36:17] <Krenair>	 nope
[15:36:24] <Krenair>	 Session issues?
[15:36:26] <Krenair>	 Has no one dealt with it?
[15:36:36] <bblack>	 I don't think so
[15:37:12] <wikibugs>	 10Ops-Access-Requests, 6operations: Grant access to stat1002 and stat1003 - https://phabricator.wikimedia.org/T98536#1272502 (10coren) I need approval language from your manager, but I'm no longer sure who that is.  @jhernandez?
[15:37:16] <Krenair>	 !log restarted apache on silver -again- to deal with reports of session errors
[15:37:16] <grrrit-wm>	 (03CR) 10KartikMistry: [C: 04-1] "Still working on this. Not to merge until MT for beta is added." [puppet] - 10https://gerrit.wikimedia.org/r/209739 (owner: 10KartikMistry)
[15:37:22] <morebots>	 Logged the message, Master
[15:37:25] <Krenair>	 greg-g, try now
[15:37:44] <bblack>	 do we have any idea what an underlying cause could be?
[15:37:52] <bblack>	 I haven't a clue really
[15:37:55] <Krenair>	 certainly works for me
[15:37:59] <wikibugs>	 10Ops-Access-Requests, 6operations: Grant access to stat1002 and stat1003 - https://phabricator.wikimedia.org/T98536#1272505 (10Jdouglas) That'll be @Tfinc.
[15:38:56] <wikibugs>	 10Ops-Access-Requests, 6operations: Grant access to stat1002 and stat1003 - https://phabricator.wikimedia.org/T98536#1272506 (10coren) Ah, no, @tfinc.  Sorry.
[15:39:06] <wikibugs>	 10Ops-Access-Requests, 6operations, 10Analytics: Access to stat1003 for jdouglas - https://phabricator.wikimedia.org/T98209#1272507 (10coren) @tfinc, I need manager approval language, if you please.
[15:54:14] <wikibugs>	 6operations, 10Wikimedia-General-or-Unknown, 7Wikimedia-log-errors: eval.php on labswiki shows SERVER_NAME undefined index errors - https://phabricator.wikimedia.org/T98615#1272519 (10Krenair) 3NEW
[15:56:30] <wikibugs>	 6operations, 10wikitech.wikimedia.org: transient failures of wiki page saves - https://phabricator.wikimedia.org/T98084#1272527 (10Krenair) This came up again today, from SAL: ```15:37 Krenair: restarted apache on silver -again- to deal with reports of session errors 15:28 greg-g: wikitech's session data error...
[15:56:41] <wikibugs>	 10Ops-Access-Requests, 6operations, 6Release-Engineering, 5Patch-For-Review: Grant access for aklapper to phab-admins - https://phabricator.wikimedia.org/T97642#1272529 (10coren) @aklapper: once you return, we need and ssh key and a signature on L3 then we're all set to set you up.
[16:01:24] <grrrit-wm>	 (03PS1) 10Paladox: Adding task support instead of using Bug: which was for bugzilla [puppet] - 10https://gerrit.wikimedia.org/r/209741 
[16:01:38] <grrrit-wm>	 (03PS2) 10Paladox: Adding task support instead of using Bug: which was for bugzilla [puppet] - 10https://gerrit.wikimedia.org/r/209741 
[16:07:47] <grrrit-wm>	 (03PS3) 10Paladox: Adding task support instead of using Bug: which was for bugzilla [puppet] - 10https://gerrit.wikimedia.org/r/209741 
[16:08:16] <grrrit-wm>	 (03PS3) 10coren: admin: add gpaumier to ana-priv-data and bastion [puppet] - 10https://gerrit.wikimedia.org/r/209001 (https://phabricator.wikimedia.org/T98077) (owner: 10Dzahn)
[16:09:55] <grrrit-wm>	 (03PS4) 10Paladox: Adding task support instead of using Bug: which was for bugzilla [puppet] - 10https://gerrit.wikimedia.org/r/209741 
[16:14:36] <icinga-wm>	 RECOVERY - High load average on labstore1001 is OK Less than 50.00% above the threshold [16.0]
[16:18:37] <grrrit-wm>	 (03PS1) 10Alex Monk: Re-enable OAuth on wikitech [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209744 (https://phabricator.wikimedia.org/T98567) 
[16:23:05] <icinga-wm>	 PROBLEM - High load average on labstore1001 is CRITICAL 87.50% of data above the critical threshold [24.0]
[16:23:14] <grrrit-wm>	 (03PS1) 10Ottomata: Puppetize standalone cluster on test Ciscos [puppet] - 10https://gerrit.wikimedia.org/r/209745 
[16:24:11] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Puppetize standalone cluster on test Ciscos [puppet] - 10https://gerrit.wikimedia.org/r/209745 (owner: 10Ottomata)
[16:24:50] <Krenair>	 why does `sql labswiki` appear to fail on silver?
[16:24:54] <grrrit-wm>	 (03PS2) 10Ottomata: Puppetize standalone cluster on test Ciscos [puppet] - 10https://gerrit.wikimedia.org/r/209745 
[16:26:43] <grrrit-wm>	 (03PS3) 10Ottomata: Puppetize standalone cluster on test Ciscos [puppet] - 10https://gerrit.wikimedia.org/r/209745 
[16:27:27] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Puppetize standalone cluster on test Ciscos [puppet] - 10https://gerrit.wikimedia.org/r/209745 (owner: 10Ottomata)
[16:28:06] <grrrit-wm>	 (03PS4) 10Ottomata: Puppetize standalone cluster on test Ciscos [puppet] - 10https://gerrit.wikimedia.org/r/209745 
[16:29:00] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032] Puppetize standalone cluster on test Ciscos [puppet] - 10https://gerrit.wikimedia.org/r/209745 (owner: 10Ottomata)
[16:32:06] <Krenair>	 I think it's broken due to the mwscript eval.php thing I noticed earlier
[16:33:20] <Krenair>	 ooohhh
[16:33:35] <Krenair>	 perhaps it's an issue on php machines but not hhvm?
[16:34:04] <gilles>	 paravoid: pong?
[16:34:07] <Krenair>	 no that makes no sense because it still works ok on tin
[16:34:07] <grrrit-wm>	 (03PS1) 10Ottomata: Need Hadoop and Hive clients to run spark [puppet] - 10https://gerrit.wikimedia.org/r/209746 
[16:34:16] <paravoid>	 gilles: too late ;)
[16:34:57] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032] Need Hadoop and Hive clients to run spark [puppet] - 10https://gerrit.wikimedia.org/r/209746 (owner: 10Ottomata)
[16:35:00] <wikibugs>	 6operations, 10Wikimedia-General-or-Unknown, 7Wikimedia-log-errors: eval.php on labswiki shows SERVER_NAME undefined index errors - https://phabricator.wikimedia.org/T98615#1272618 (10Krenair) This breaks the `sql` command on silver (due to $host containing these notices, as it just executes mwscript to get...
[16:35:25] <icinga-wm>	 PROBLEM - puppet last run on analytics1003 is CRITICAL puppet fail
[16:35:40] <wikibugs>	 6operations, 10Wikimedia-General-or-Unknown, 7Wikimedia-log-errors: eval.php on silver shows SERVER_NAME undefined index errors - https://phabricator.wikimedia.org/T98615#1272619 (10Krenair)
[16:38:45] <icinga-wm>	 RECOVERY - puppet last run on analytics1003 is OK Puppet is currently enabled, last run 31 seconds ago with 0 failures
[16:51:45] <grrrit-wm>	 (03PS1) 10BryanDavis: Add missing labswiki.png [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209749 
[16:52:41] <grrrit-wm>	 (03CR) 10BryanDavis: [C: 032] Add missing labswiki.png [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209749 (owner: 10BryanDavis)
[16:52:47] <grrrit-wm>	 (03Merged) 10jenkins-bot: Add missing labswiki.png [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209749 (owner: 10BryanDavis)
[16:52:56] <grrrit-wm>	 (03PS1) 10Ottomata: Fix typo in site.pp for analytics1003 and 1010 [puppet] - 10https://gerrit.wikimedia.org/r/209750 
[16:53:36] <logmsgbot>	 !log bd808 Synchronized w/static/images/project-logos/labswiki.png: Add missing labswiki.png (duration: 00m 13s)
[16:53:42] <morebots>	 Logged the message, Master
[16:53:53] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032] Fix typo in site.pp for analytics1003 and 1010 [puppet] - 10https://gerrit.wikimedia.org/r/209750 (owner: 10Ottomata)
[16:54:32] <bd808>	 we can haz unicorn again :)
[16:54:52] <Krenair>	 \o/
[16:55:47] <grrrit-wm>	 (03PS5) 10Paladox: Adding task support instead of using Bug: which was for bugzilla [puppet] - 10https://gerrit.wikimedia.org/r/209741 
[16:56:43] <wikibugs>	 10Ops-Access-Requests, 6operations, 6Release-Engineering, 5Patch-For-Review: Grant access for aklapper to phab-admins - https://phabricator.wikimedia.org/T97642#1272648 (10Dzahn) a:3Aklapper
[17:00:49] <grrrit-wm>	 (03PS1) 10Ottomata: analytics1004 is a spark worker [puppet] - 10https://gerrit.wikimedia.org/r/209751 
[17:01:11] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032 V: 032] analytics1004 is a spark worker [puppet] - 10https://gerrit.wikimedia.org/r/209751 (owner: 10Ottomata)
[17:12:15] <grrrit-wm>	 (03PS6) 10Paladox: Adding task support instead of using Bug: which was for bugzilla [puppet] - 10https://gerrit.wikimedia.org/r/209741 
[17:13:46] <grrrit-wm>	 (03CR) 10coren: [C: 032] "Time's up!" [puppet] - 10https://gerrit.wikimedia.org/r/209001 (https://phabricator.wikimedia.org/T98077) (owner: 10Dzahn)
[17:14:58] <wikibugs>	 10Ops-Access-Requests, 6operations, 6Editing-Department, 5Patch-For-Review: Requesting access to analytics-privatedata-users for Guillaume Paumier - https://phabricator.wikimedia.org/T98077#1272669 (10coren) 5Open>3Resolved a:3coren Done.
[17:15:02] <wikibugs>	 6operations, 6Release-Engineering, 10Wikimedia-Language-setup, 10Wikimedia-Site-requests: nb subdomain redirects - https://phabricator.wikimedia.org/T86924#1272673 (10Dzahn) >>! In T86924#985115, @Aklapper wrote: > Not sure either if this is still site-requests level (shell) or already lower on DNS level (...
[17:16:16] <Betacommand>	 Are ops aware of the enwiki job queue issue?
[17:16:56] <JohnLewis>	 Coren ^
[17:17:00] <grrrit-wm>	 (03PS7) 10Paladox: Adding task support instead of using Bug: which was for bugzilla [puppet] - 10https://gerrit.wikimedia.org/r/209741 
[17:17:35] <Glaisher>	 I think Krenair was talking about that some days ago?
[17:18:00] <Coren>	 Betacommand: I wasn't, personally.  Lemme check if there is someone on it otherwise I'll see if I can do something.
[17:18:18] <Betacommand>	 Coren: its up to about 21m
[17:18:31] <Krenair>	 Yes
[17:18:34] <Krenair>	 I was talking about it
[17:18:38] <Krenair>	 On 2 or 3 separate occasions
[17:18:44] <Krenair>	 T13|mobile was talking about it
[17:18:48] <Krenair>	 Other people have all mentioned it
[17:18:52] <Krenair>	 No one seems to care
[17:19:45] <Glaisher>	 wtf..
[17:19:45] <Glaisher>	  "jobs": 21894746,
[17:19:59] <Betacommand>	 Ok make that 22m
[17:20:01] * Coren looks at it.
[17:20:08] <Krenair>	 alex@alex-laptop:~$ ssh tin.eqiad.wmnet -t mwscript showJobs.php enwiki --group
[17:20:09] <Krenair>	 refreshLinks: 21792952 queued; 136 claimed (1 active, 135 abandoned); 0 delayed
[17:20:17] <Coren>	 Lots of people seemed to have talked about it, did anyone mention a ticket?
[17:20:38] <Krenair>	 the other job types seem relatively low
[17:20:44] <Krenair>	 I don't think so
[17:21:23] <wikibugs>	 6operations: Degraded RAID-1 arrays on new logstash hosts: [UU__] - https://phabricator.wikimedia.org/T98620#1272683 (10Gage) 3NEW
[17:21:26] <icinga-wm>	 PROBLEM - High load average on labstore1001 is CRITICAL 66.67% of data above the critical threshold [24.0]
[17:21:37] <grrrit-wm>	 (03CR) 10Kaldari: [C: 031] Import lists for the Browse experiment on beta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209242 (https://phabricator.wikimedia.org/T95446) (owner: 10Phuedx)
[17:21:49] <T13|mobile>	 Coren: I'd have made a ticket, but am unsure what is considered "high" backlog for it
[17:22:17] <Betacommand>	 T13|mobile: 2-3m is high normally
[17:22:32] <T13|mobile>	 Was going to put one in later today if I hadn't gotten an answer
[17:22:47] <Krenair>	 This has been going on since April 30th at least
[17:22:52] <Coren>	 T13|mobile: I'm not all that certain what is high for that queue, but the simple fact that it is not improving is reason enough.
[17:23:23] <T13|mobile>	 I know a bunch of people have given up on the jq and are null editing the heck out of everything trying to "help".
[17:23:50] <wikibugs>	 6operations, 10MediaWiki-JobQueue: enwiki's job is about 22m and increasing - https://phabricator.wikimedia.org/T98621#1272696 (10Glaisher) 3NEW
[17:24:01] <T13|mobile>	 Last I knew legoktm was looking inti it at least.
[17:24:02] <Coren>	 T13|mobile: "help"?
[17:24:10] <T13|mobile>	 lol yeah..
[17:24:11] <Coren>	 As in, WP:POINT?
[17:24:39] <wikibugs>	 6operations, 10MediaWiki-JobQueue, 10MediaWiki-JobRunner: enwiki's job is about 22m atm and increasing - https://phabricator.wikimedia.org/T98621#1272712 (10Glaisher) p:5Triage>3High
[17:24:48] <wikibugs>	 6operations, 10MediaWiki-JobQueue, 10MediaWiki-JobRunner: enwiki's job is about 22m atm and increasing - https://phabricator.wikimedia.org/T98621#1272719 (10Krenair) ```Apr 30 13:50:29 <Krenair> Someone just pointed out in tech that enwiki has a ridiculously large job queue at the moment Apr 30 16:44:53 <leg...
[17:25:16] <T13|mobile>	 Does having a bunch of people WP:NULLEDITing everything have any effect on the queue? I'm under the impression it doesn't really.
[17:25:19] <mutante>	 Coren: Krenair: i will be at office in like an hour, i can see if Aaron is around 
[17:25:23] <Betacommand>	 Coren: No, as in screw it, the JQ is broken so there is no since in waiting for it
[17:25:32] <Betacommand>	 T13|mobile: it does
[17:25:34] <wikibugs>	 6operations, 10MediaWiki-JobQueue, 10MediaWiki-JobRunner: enwiki's job is about 22m atm and increasing - https://phabricator.wikimedia.org/T98621#1272721 (10Glaisher) <Krenair> This has been going on since April 30th at least
[17:26:24] <Coren>	 mutante: That could be helpful; my knowledge of the job queue is more in the 'kill everything' category and basic diagnostic.
[17:29:03] <Coren>	 afaict, the jobrunners are... well, running jobs.
[17:29:11] <Krenair>	 yes, they are
[17:29:36] <T13|mobile>	 Do they need more resources to catch up?
[17:30:03] <Coren>	 T13|mobile: The boxes don't seem to be straining.
[17:30:05] <valhallasw`nuage>	 lock enwiki for the night! [/jk]
[17:30:17] <Nemo_bis>	 s/night/month/
[17:30:19] <Krenair>	 the other job types seem healthy
[17:30:28] <JohnLewis>	 valhallasw`nuage: drop database enwiki;!
[17:30:29] <Coren>	 Very busy hhvms.
[17:30:30] <Krenair>	 it's refreshLinks that's completely nuts
[17:30:42] <Coren>	 Krenair: That may be a db issue.
[17:31:16] <mutante>	 refreshlinks? there was some change to the cronjobs not that long ago
[17:31:24] <Nemo_bis>	 IMHO it's not particularly useful to chase ghosts; someone should whip all those lazy jobrunners. https://gerrit.wikimedia.org/r/209719
[17:32:02] <grrrit-wm>	 (03PS1) 10Ottomata: Run hdfs balancer every day [puppet] - 10https://gerrit.wikimedia.org/r/209756 
[17:32:13] <Coren>	 Nemo_bis: It doesn't look like resource starvation on the jobrunners themselves.
[17:32:43] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032 V: 032] Run hdfs balancer every day [puppet] - 10https://gerrit.wikimedia.org/r/209756 (owner: 10Ottomata)
[17:33:07] <wikibugs>	 6operations, 10MediaWiki-JobQueue, 10MediaWiki-JobRunner: enwiki's job is about 22m atm and increasing - https://phabricator.wikimedia.org/T98621#1272733 (10coren) I did a quick look around the job runners, and they seem to be running fine without being starved of resources.  A point of note is that it appar...
[17:33:34] <grrrit-wm>	 (03PS1) 10Ottomata: Use role to include hive, hadoop, and spark classes on analytics ciscos [puppet] - 10https://gerrit.wikimedia.org/r/209757 
[17:33:55] <grrrit-wm>	 (03PS2) 10Ottomata: Use role to include hive, hadoop, and spark classes on analytics ciscos [puppet] - 10https://gerrit.wikimedia.org/r/209757 
[17:35:19] <grrrit-wm>	 (03PS3) 10Ottomata: Use role to include hive, hadoop, and spark classes on analytics ciscos [puppet] - 10https://gerrit.wikimedia.org/r/209757 
[17:35:26] <mutante>	 so you say refreshlinks, right
[17:35:30] <mutante>	 https://phabricator.wikimedia.org/T97926#1255734
[17:35:52] <mutante>	 and enwiki , but s1 doesnt have those cron jobs 
[17:36:16] <mutante>	 did somebody manually try to run them for s1 maybe?
[17:36:25] <mutante>	 or is this unrelated 
[17:37:08] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032] Use role to include hive, hadoop, and spark classes on analytics ciscos [puppet] - 10https://gerrit.wikimedia.org/r/209757 (owner: 10Ottomata)
[17:37:47] <wikibugs>	 6operations, 10Wikimedia-Site-requests: Run "refreshLinks.php --dfn-only" on all wikis periodically - https://phabricator.wikimedia.org/T18112#1272738 (10Nemo_bis) 5Resolved>3Open
[17:37:57] <wikibugs>	 6operations, 10Wikimedia-Site-requests: refreshLinks.php --dfn-only cron jobs do not seem to be running - https://phabricator.wikimedia.org/T97926#1272744 (10Nemo_bis)
[17:38:00] <wikibugs>	 6operations, 10Wikimedia-Site-requests: Run "refreshLinks.php --dfn-only" on all wikis periodically - https://phabricator.wikimedia.org/T18112#200641 (10Nemo_bis)
[17:38:33] <wikibugs>	 6operations, 10Wikimedia-Site-requests: refreshLinks.php --dfn-only cron jobs do not seem to be running - https://phabricator.wikimedia.org/T97926#1255273 (10Nemo_bis) >>! In T97926#1255734, @Reedy wrote: > https://github.com/wikimedia/operations-puppet/blob/98d7853ce884edd5478d31a72369255b8c7b4f6f/manifests/m...
[17:41:02] <grrrit-wm>	 (03CR) 10BryanDavis: "Posted for SWAT on 2015-05-11T15:00Z" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209172 (https://phabricator.wikimedia.org/T88732) (owner: 10BryanDavis)
[17:42:52] <grrrit-wm>	 (03PS12) 10Yuvipanda: mesos: Add simple mesos module [puppet] - 10https://gerrit.wikimedia.org/r/208483 
[17:42:57] <Nemo_bis>	 Are those QPS values (10k, 25k) normal for s1? https://dbtree.wikimedia.org/
[17:43:07] <yuvipanda>	 _joe_: I’m going to merge https://gerrit.wikimedia.org/r/#/c/208483/ (mesos patch)
[17:43:43] <_joe_>	 yuvipanda: I had no way to review it
[17:43:56] <_joe_>	 blame etcd's broken peer TLS for that
[17:43:59] <yuvipanda>	 _joe_: aaah
[17:44:02] <_joe_>	 yuvipanda: +1 anyway
[17:44:04] <yuvipanda>	 _joe_: I shall blame it
[17:44:05] <yuvipanda>	 _joe_: thanks
[17:44:17] <grrrit-wm>	 (03PS13) 10Yuvipanda: mesos: Add simple mesos module [puppet] - 10https://gerrit.wikimedia.org/r/208483 
[17:44:20] <_joe_>	 it's not like it's going to prod tomorrow
[17:44:24] <hoo>	 Nemo_bis: QPS is not a good indicator... and for a strong slave that sounds ok
[17:44:42] <hoo>	 strong as in large enough buffer
[17:44:52] <yuvipanda>	 _joe_: true
[17:45:06] <Nemo_bis>	 hoo: if abnormally high it would be an indicator of something hitting it :)
[17:45:10] <grrrit-wm>	 (03PS1) 10Ottomata: Be explicit about spark worker memory in hiera [puppet] - 10https://gerrit.wikimedia.org/r/209759 
[17:45:28] <hoo>	 Nemo_bis: Well, that *could*...
[17:45:43] <hoo>	 Look at ganglia, we should have stats there that could give a hint
[17:45:54] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032] mesos: Add simple mesos module [puppet] - 10https://gerrit.wikimedia.org/r/208483 (owner: 10Yuvipanda)
[17:46:01] <hoo>	 (select a slave with high weigth)
[17:46:07] <Nemo_bis>	 Too lazy to check ganglia since it's no longer link from dbtree
[17:47:12] <grrrit-wm>	 (03PS2) 10Ottomata: Be explicit about spark worker memory in hiera [puppet] - 10https://gerrit.wikimedia.org/r/209759 
[17:47:19] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032 V: 032] Be explicit about spark worker memory in hiera [puppet] - 10https://gerrit.wikimedia.org/r/209759 (owner: 10Ottomata)
[17:50:37] <grrrit-wm>	 (03PS1) 10Ottomata: Need unit on spark worker memory value [puppet] - 10https://gerrit.wikimedia.org/r/209763 
[17:50:41] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Need unit on spark worker memory value [puppet] - 10https://gerrit.wikimedia.org/r/209763 (owner: 10Ottomata)
[17:50:53] <grrrit-wm>	 (03PS2) 10Ottomata: Need unit on spark worker memory value [puppet] - 10https://gerrit.wikimedia.org/r/209763 
[17:51:08] <grrrit-wm>	 (03CR) 10Ottomata: [C: 032 V: 032] Need unit on spark worker memory value [puppet] - 10https://gerrit.wikimedia.org/r/209763 (owner: 10Ottomata)
[17:51:31] <wikibugs>	 6operations: Degraded RAID-1 arrays on new logstash hosts: [UU__] - https://phabricator.wikimedia.org/T98620#1272790 (10Gage)
[17:51:36] <icinga-wm>	 PROBLEM - puppet last run on analytics1010 is CRITICAL Puppet has 1 failures
[17:53:07] <icinga-wm>	 RECOVERY - puppet last run on analytics1010 is OK Puppet is currently enabled, last run 40 seconds ago with 0 failures
[17:54:03] <^d>	 _joe_: I know it's almost $weekend for you; any chance you could have a look at my 'dsh-cleanup' topic for puppet sometime next week? The current situation blocks staging, complicates beta and makes installing scap in $random_labs_project for testing basically impossible.
[17:57:20] <wikibugs>	 10Ops-Access-Requests, 6operations: Grant ssh access on deployment-elastic0{5-8}.eqiad.wmflabs - https://phabricator.wikimedia.org/T98624#1272824 (10Jdouglas) 3NEW a:3Jdouglas
[17:58:01] <grrrit-wm>	 (03PS1) 10Rush: phab: phd is not lsb compliant and we work around it [puppet] - 10https://gerrit.wikimedia.org/r/209767 
[17:58:48] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] phab: phd is not lsb compliant and we work around it [puppet] - 10https://gerrit.wikimedia.org/r/209767 (owner: 10Rush)
[17:59:55] <grrrit-wm>	 (03PS2) 10Rush: phab: phd is not lsb compliant and we work around it [puppet] - 10https://gerrit.wikimedia.org/r/209767 
[18:00:04] <wikibugs>	 10Ops-Access-Requests, 6operations: Grant ssh access on deployment-elastic0{5-8}.eqiad.wmflabs - https://phabricator.wikimedia.org/T98624#1272843 (10JohnLewis) You seem to be a member of the deployment-prep project already. You should be able to ssh to the instances from the main labs bastion.
[18:00:41] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] phab: phd is not lsb compliant and we work around it [puppet] - 10https://gerrit.wikimedia.org/r/209767 (owner: 10Rush)
[18:01:56] <grrrit-wm>	 (03PS3) 10Rush: phab: phd is not lsb compliant and we work around it [puppet] - 10https://gerrit.wikimedia.org/r/209767 
[18:02:56] <wikibugs>	 10Ops-Access-Requests, 6operations: Grant ssh access on deployment-elastic0{5-8}.eqiad.wmflabs - https://phabricator.wikimedia.org/T98624#1272859 (10Krenair) A link to T97408 would've been more helpful
[18:02:57] <grrrit-wm>	 (03CR) 10Rush: [C: 032] phab: phd is not lsb compliant and we work around it [puppet] - 10https://gerrit.wikimedia.org/r/209767 (owner: 10Rush)
[18:03:15] <grrrit-wm>	 (03PS1) 10BryanDavis: beta: hiera config for deployment-fluorine [puppet] - 10https://gerrit.wikimedia.org/r/209769 
[18:04:10] <ori>	 bblack, paravoid: hey, i saw the bits issue in the backscroll on my phone on my commute, just getting caught up now.
[18:04:22] <grrrit-wm>	 (03CR) 10Chad: "Why host-specific?" [puppet] - 10https://gerrit.wikimedia.org/r/209769 (owner: 10BryanDavis)
[18:05:27] <grrrit-wm>	 (03CR) 10BryanDavis: "So it doesn't apply to deployment-bastion while that host has duplicate roles" [puppet] - 10https://gerrit.wikimedia.org/r/209769 (owner: 10BryanDavis)
[18:05:37] <icinga-wm>	 RECOVERY - puppet last run on iridium is OK Puppet is currently enabled, last run 1 minute ago with 0 failures
[18:06:36] <icinga-wm>	 PROBLEM - Persistent high iowait on labstore1001 is CRITICAL 62.50% of data above the critical threshold [35.0]
[18:12:11] <wikibugs>	 10Ops-Access-Requests, 6operations: Grant ssh access on deployment-elastic0{5-8}.eqiad.wmflabs - https://phabricator.wikimedia.org/T98624#1272898 (10Jdouglas) Awesome, thanks!
[18:12:19] <wikibugs>	 10Ops-Access-Requests, 6operations: Grant ssh access on deployment-elastic0{5-8}.eqiad.wmflabs - https://phabricator.wikimedia.org/T98624#1272899 (10Jdouglas) 5Open>3Resolved
[18:14:22] <logmsgbot>	 !log yurik Synchronized php-1.26wmf5/extensions/Graph/: Bumping graph (duration: 00m 14s)
[18:14:29] <morebots>	 Logged the message, Master
[18:14:36] <icinga-wm>	 PROBLEM - Persistent high iowait on labstore1001 is CRITICAL 50.00% of data above the critical threshold [35.0]
[18:14:47] <logmsgbot>	 !log yurik Synchronized php-1.26wmf4/extensions/Graph/: Bumping graph (duration: 00m 14s)
[18:14:53] <morebots>	 Logged the message, Master
[18:17:01] <grrrit-wm>	 (03PS1) 10ArielGlenn: update multiple args format to publish.runner [software/deployment/trebuchet-trigger] - 10https://gerrit.wikimedia.org/r/209775 
[18:17:22] <grrrit-wm>	 (03PS1) 10Mattflaschen: Have Beta Cluster use same settings as prod for LQT [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209776 
[18:17:47] <icinga-wm>	 RECOVERY - Persistent high iowait on labstore1001 is OK Less than 50.00% above the threshold [25.0]
[18:18:28] <wikibugs>	 6operations: fix trebuchet-trigger (git deploy) publish.runner arguments - https://phabricator.wikimedia.org/T98581#1272911 (10ArielGlenn) Chad kindly created a local repo based on upstream and here's the patch: https://gerrit.wikimedia.org/r/#/c/209775/
[18:18:56] <icinga-wm>	 RECOVERY - High load average on labstore1001 is OK Less than 50.00% above the threshold [16.0]
[18:19:49] <grrrit-wm>	 (03CR) 10ArielGlenn: "I've tested this on a precise deployment server running 2014.1.10 with the rest of the hosts running 2014.7.5 and on a cluster where all h" [software/deployment/trebuchet-trigger] - 10https://gerrit.wikimedia.org/r/209775 (owner: 10ArielGlenn)
[18:19:52] <wikibugs>	 6operations, 10Beta-Cluster: Can't apply ::role::logging::mediawiki on a trusty host - https://phabricator.wikimedia.org/T98627#1272912 (10bd808) 3NEW
[18:23:02] <grrrit-wm>	 (03PS8) 10Paladox: Adding task support instead of using Bug: which was for bugzilla [puppet] - 10https://gerrit.wikimedia.org/r/209741 
[18:25:36] <icinga-wm>	 PROBLEM - High load average on labstore1001 is CRITICAL 87.50% of data above the critical threshold [24.0]
[18:26:57] <grrrit-wm>	 (03CR) 10Catrope: [C: 031] Flow should use VE by default [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209042 (https://phabricator.wikimedia.org/T98168) (owner: 10Mattflaschen)
[18:27:10] <yuvipanda>	 bd808: this is why I suggested precise :D Since it’s precise in prod, our options are either to make it work on both or move prod to trusty as well
[18:27:41] <bd808>	 yuvipanda: *nod* I will be rebuilding as old stuff
[18:27:47] <yuvipanda>	 cool
[18:29:48] <wikibugs>	 10Ops-Access-Requests, 6operations: Grant ssh access on deployment-elastic0{5-8}.eqiad.wmflabs - https://phabricator.wikimedia.org/T98624#1272956 (10Krenair) 5Resolved>3Invalid
[18:30:04] <wikibugs>	 10Ops-Access-Requests, 6operations: Grant ssh access on deployment-elastic0{5-8}.eqiad.wmflabs - https://phabricator.wikimedia.org/T98624#1272824 (10Krenair) Also, wmflabs access is still not an #ops-access-requests thing
[18:30:50] <wikibugs>	 6operations, 10Beta-Cluster: Can't apply ::role::logging::mediawiki on a trusty host - https://phabricator.wikimedia.org/T98627#1272966 (10bd808) I'm going to rebuild the instance I was working on as a precise host. Someday we can redo the beta cluster version after it has been sorted out for prod as either tr...
[18:31:25] <ottomata>	 akosiaris: yt?
[18:33:20] <greg-g>	 is there a "convert fleet away from precise" task?
[18:33:35] * greg-g looks more deeply
[18:34:13] <grrrit-wm>	 (03CR) 10Chad: [C: 031] "makes sense :)" [puppet] - 10https://gerrit.wikimedia.org/r/209769 (owner: 10BryanDavis)
[18:34:17] <^d>	 bd808: ^
[18:34:35] <bd808>	 ^d: I figured that would sit ok with you
[18:34:57] <bd808>	 Are you filled in on what I'm hacking up over in beta cluster land?
[18:35:08] <bd808>	 just trying to get logs off of nfs
[18:35:10] <^d>	 I am not
[18:35:28] <bd808>	 I did apache2 logs last night
[18:35:44] <bd808>	 next step is udp2log stuff
[18:36:01] <bd808>	 then last bit will be central syslog
[18:36:56] <icinga-wm>	 RECOVERY - High load average on labstore1001 is OK Less than 50.00% above the threshold [16.0]
[18:37:50] <Krenair>	 greg-g, we're still waiting for the hhvm migration to be finished...
[18:38:21] <manybubbles>	 hi ops! am I right in assuming that you are working on some kind of plan for to make deploying services simpler?
[18:42:19] <csteipp>	 !log deployed patch for T98313 for wmf4/5
[18:42:30] <morebots>	 Logged the message, Master
[18:43:19] <legoktm>	 T13|mobile: fix your client
[18:43:21] <legoktm>	 gah
[18:58:26] <grrrit-wm>	 (03PS1) 10Ori.livneh: Use $wgServer to construct static-asset URLs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209786 
[18:59:45] <hoo>	 ori: Did you check that $wgServer is defined at that point?
[18:59:47] <hoo>	 I suggested tha tmove earlier, but wasn't sure
[19:00:10] <grrrit-wm>	 (03CR) 10Alex Monk: "Will this fix T98615?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209786 (owner: 10Ori.livneh)
[19:00:36] <icinga-wm>	 PROBLEM - puppet last run on mw1194 is CRITICAL Puppet has 1 failures
[19:04:44] <ori>	 hoo|away: yes, it is
[19:04:56] <icinga-wm>	 PROBLEM - High load average on labstore1001 is CRITICAL 55.56% of data above the critical threshold [24.0]
[19:05:05] <grrrit-wm>	 (03PS2) 10Ori.livneh: Use $wgServer to construct static-asset URLs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209786 
[19:06:40] <grrrit-wm>	 (03CR) 10Ori.livneh: "@krenair: yep" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209786 (owner: 10Ori.livneh)
[19:07:39] <grrrit-wm>	 (03PS3) 10Alex Monk: Use $wgServer to construct static-asset URLs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209786 (https://phabricator.wikimedia.org/T98615) (owner: 10Ori.livneh)
[19:07:52] <wikibugs>	 6operations, 10Wikimedia-General-or-Unknown, 5Patch-For-Review, 7Wikimedia-log-errors: eval.php on silver shows SERVER_NAME undefined index errors - https://phabricator.wikimedia.org/T98615#1273139 (10Krenair) a:3ori
[19:09:34] <ori>	 Krenair: want to review?
[19:09:58] <Krenair>	 ori, ok...
[19:12:02] <logmsgbot>	 !log yurik Synchronized php-1.26wmf5/extensions/CentralAuth: Bumping CentralAuth (duration: 00m 12s)
[19:12:09] <morebots>	 Logged the message, Master
[19:15:45] <Krenair>	 ori, yeah seems ok
[19:16:02] <grrrit-wm>	 (03CR) 10Alex Monk: [C: 031] Use $wgServer to construct static-asset URLs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209786 (https://phabricator.wikimedia.org/T98615) (owner: 10Ori.livneh)
[19:16:11] <ori>	 I tested it on mw1017 as well
[19:16:13] <ori>	 thanks!
[19:16:21] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 032] Use $wgServer to construct static-asset URLs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209786 (https://phabricator.wikimedia.org/T98615) (owner: 10Ori.livneh)
[19:16:57] <grrrit-wm>	 (03Merged) 10jenkins-bot: Use $wgServer to construct static-asset URLs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209786 (https://phabricator.wikimedia.org/T98615) (owner: 10Ori.livneh)
[19:18:23] <logmsgbot>	 !log ori Synchronized wmf-config/CommonSettings.php: I6236f5e2c: Use $wgServer to construct static-asset URLs (duration: 00m 12s)
[19:18:26] <icinga-wm>	 RECOVERY - puppet last run on mw1194 is OK Puppet is currently enabled, last run 1 minute ago with 0 failures
[19:18:30] <morebots>	 Logged the message, Master
[19:18:39] <logmsgbot>	 !log yurik Synchronized php-1.26wmf4/extensions/CentralAuth: Bumping CentralAuth (duration: 00m 13s)
[19:18:43] <morebots>	 Logged the message, Master
[19:19:58] <wikibugs>	 6operations, 10Wikimedia-General-or-Unknown, 5Patch-For-Review, 7Wikimedia-log-errors: eval.php on silver shows SERVER_NAME undefined index errors - https://phabricator.wikimedia.org/T98615#1273190 (10Krenair) 5Open>3Resolved
[19:20:14] <grrrit-wm>	 (03Abandoned) 10Ori.livneh: remove $wmgAssetsHost [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209713 (owner: 10BBlack)
[19:36:31] <ori>	 Coren: there are two things I see on fluorine:
[19:36:35] <ori>	 2015-05-08 19:31:29 mw1013 enwiki JobQueueRedis INFO: Could not acknowledge refreshLinksPrioritized job. {"private":false}
[19:36:55] <ori>	 there's one of those every 30 seconds or so in /a/mw-log/JobQueueRedis.log
[19:37:33] <ori>	 additionally, there are failures to connect every few seconds in redis.log
[19:37:36] <ori>	 (same host/directory)
[19:38:25] <Coren>	 Hm
[19:40:11] <ori>	 the hosts are all in the mw11* range
[19:40:15] <ori>	 not sure if that is significant, yet.
[19:40:30] <ori>	 https://dpaste.de/QRhz/raw
[19:40:43] <Coren>	 Hm
[19:41:00] <Coren>	 Are the jobs dispatched to certain hosts by tipe?
[19:41:40] <Coren>	 type*
[19:42:11] <ori>	 i'm not sure. those are the hosts doing the dispatching; the redis hosts they are failing to connect to are broken down in this paste:
[19:42:12] <ori>	 https://dpaste.de/bFne/raw
[19:42:37] <icinga-wm>	 RECOVERY - High load average on labstore1001 is OK Less than 50.00% above the threshold [16.0]
[19:42:52] <ori>	 it's interesting that 10.64.0.x are all at the top
[19:43:17] <Coren>	 mc1005 has the top spot.
[19:43:50] <ori>	 could you check if anything about that host looks odd in ganglia? I'll look at the job runner log files.
[19:44:09] <Coren>	 I'm looking at its redis logs and it's *definitely* odd:
[19:44:16] <Coren>	 [1546] 08 May 19:43:50.665 * Connecting to MASTER...
[19:44:16] <Coren>	 [1546] 08 May 19:43:50.666 # Unable to connect to MASTER: Success
[19:44:23] <Coren>	 Basically every second.
[19:44:41] <ori>	 weird
[19:44:44] <Coren>	 I loves me "XXX faied: Success" error messages.
[19:45:04] <Coren>	 Means whatever failed did not set errno; which makes debugging extra crispy.
[19:45:28] * Coren straces.
[19:45:59] <Coren>	 Aha, the plot thickens.
[19:48:33] <ori>	 Coren: ?
[19:48:49] <Coren>	 I'm examining what redis is trying to connect /to/
[19:51:31] <grrrit-wm>	 (03PS1) 10Yurik: Disable Graph security header on private wikis [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209800 
[19:51:49] <yurik>	 csteipp, could you +2 ^
[19:51:52] <yurik>	 greg-g, ^
[19:53:01] <Coren>	 ori: At this point, I'm thinking that the redis server on the box is unable to sync up; and since its set for 'slave-serve-stale-data yes' then it still sees an (empty) queue
[19:54:40] <ori>	 are those redises even used by the job queue?
[19:55:36] <icinga-wm>	 PROBLEM - High load average on labstore1001 is CRITICAL 62.50% of data above the critical threshold [24.0]
[19:55:50] <Coren>	 Hm.  I suppose the two error messages may be unrelated, but it's suspect.
[19:56:40] <grrrit-wm>	 (03PS1) 10BryanDavis: beta: use deployment-fluorine as udp2log aggregator [puppet] - 10https://gerrit.wikimedia.org/r/209801 
[19:59:22] <legoktm>	 mwscript showJobs.php --wiki=enwiki --list=refreshLinks is still broken
[19:59:44] <Coren>	 (Shouldn't that pulralize 'redes' because third declension?)
[19:59:52] <Coren>	 :-)
[20:02:07] <Coren>	 Oh duh, it's the verb so 'redetis'.
[20:02:19] <ori>	 legoktm: it's interesting that `mwscript extensions/WikimediaMaintenance/getJobQueueLengths.php --grouponly` works
[20:02:47] <Coren>	 ori: That connection error is mysterious, but doesn't seem directly related.
[20:03:06] <grrrit-wm>	 (03CR) 10Alex Monk: [C: 04-2] "suppression, CU, etc. data? I don't think private wikis are any different here" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209800 (owner: 10Yurik)
[20:03:29] <legoktm>	 ori: I think it's the --list part that breaks, showJobs.php --group works fine
[20:05:19] <grrrit-wm>	 (03CR) 10Nemo bis: "Guillom, will you notify the wikis in question? Thanks." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209616 (owner: 10Ori.livneh)
[20:05:43] <yurik>	 Krenair, i suspect zero must have it, otherwise whenever the graph is accessing API, it will be treated as an anonymous request, which means that they won't be able to access anything
[20:05:52] <yurik>	 csteipp is ok with that
[20:06:09] <Krenair>	 CSteipp is OK with this?
[20:06:19] <Krenair>	 He hasn't reviewed the change.
[20:06:37] <Krenair>	 We shouldn't grant zerowiki another bizarre exception here.
[20:06:53] <Krenair>	 There needs to be one decision for all private wikis and I suspect the proposed one is wrong.
[20:08:26] <icinga-wm>	 PROBLEM - HTTP 5xx req/min on graphite1001 is CRITICAL 23.08% of data above the critical threshold [500.0]
[20:08:47] <ori>	 legoktm: did SULF trigger linksupdate jobs?
[20:09:17] <legoktm>	 do we trigger them during page moves?
[20:09:23] * legoktm looks
[20:10:26] <icinga-wm>	 RECOVERY - High load average on labstore1001 is OK Less than 50.00% above the threshold [16.0]
[20:12:07] <guillom>	 ori: I'm looking at https://gerrit.wikimedia.org/r/#/c/209616/ ; Just to make sure I understand: We took the localized logos for all those wikis, and put them in a static place to improve performance, and the downside is that local changes to Wiki.png now won't change the logo?
[20:12:37] <guillom>	 (I'm trying to prase it correctly for the tech newsletter.)
[20:13:15] <guillom>	 phrase, too
[20:13:30] <ori>	 guillom: correct, but the way logo changes have typically been done is by community request, since the new logo is not usually uploaded on top of the old one
[20:13:47] <legoktm>	 ori: testing locally I don't see any jobs queued when I move pages
[20:14:09] <guillom>	 ori: OK; my recollection may be tainted by how it was done in Ye Olde Days. Thanks!
[20:14:26] <ori>	 guillom: np, thank you
[20:14:53] <Krenair>	 Oh wow you just uploaded that and self-merged it the next day?
[20:14:53] <ori>	 legoktm: kk. what about https://gerrit.wikimedia.org/r/#/c/205648/ ? it coincides with the job spike. was it a reaction to it or a cause?
[20:15:17] <ori>	 Krenair: same day
[20:15:42] <Krenair>	 oh wow yes, the next day timestamp I was looking at was caused by Nemo's comment the next day
[20:15:42] <ori>	 Krenair: what's the issue?
[20:15:57] <Krenair>	 you uploaded it and self-merged it under 10 minutes later
[20:16:03] * legoktm looks at his IRC logs
[20:16:41] <Krenair>	 You removed the references to local Wiki.png/commons?
[20:16:41] <ori>	 Krenair: go on..
[20:16:57] <grrrit-wm>	 (03PS1) 10EBernhardson: Enable CirrusSearch-PerUser PoolCounter on group0 only [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209802 (https://phabricator.wikimedia.org/T76497) 
[20:17:19] <guillom>	 ori: Some wikis like to use temporary festive logos, and they've historically used the local logo file for that purpose, so they may be unhappy that they can't do that any longer. (Although personally, I'm quite happy that we won't see those horrible things any more.)
[20:17:45] <ori>	 guillom: they can still do that via Common.css
[20:17:59] <guillom>	 ori: Ok :) Thanks for the clarification.
[20:18:03] <Krenair>	 I'm sure this is going to make you popular...
[20:18:24] <ori>	 Krenair, guillom: did I misjudge the community impact of this change? Why are you so shocked, Krenair?
[20:18:38] <ori>	 It'd help if you explain matters practically rather than simply express shock / dismay, Krenair.
[20:18:59] <Krenair>	 Wikis tend to appreciate local control of things
[20:19:03] <valhallasw`nuage>	 ori: taking autonomy from wikis is a good way to get people annoyed
[20:19:05] <valhallasw`nuage>	 ^ that.
[20:19:06] <guillom>	 ori: If people can use another mechanism to customize the logo temporarily, I think you're fine.
[20:19:30] <valhallasw`nuage>	 so this kind of change needs a) a way to override it (which there is, good!) and b) good communication to the communities about the change
[20:19:46] <ori>	 if there is a way to override it, is it actually taking autonomy away from wikis?
[20:20:00] <grrrit-wm>	 (03PS2) 10EBernhardson: Enable CirrusSearch-PerUser PoolCounter on group0 only [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209802 (https://phabricator.wikimedia.org/T76497) 
[20:20:02] <grrrit-wm>	 (03CR) 10Nemo bis: "I'm downloading this change to check what ended up being uploaded here. There are probably several temporary logos." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209615 (owner: 10Ori.livneh)
[20:20:14] <guillom>	 ori: We just need to make sure that they know about it and how to do it; I'll explain it in tech news.
[20:20:19] <valhallasw`nuage>	 it doesn't matter if there is a way to override it; if it /seems/ like there is no way to override it, it will be seen as a grab for power by the WMF, etc.
[20:21:06] <valhallasw`nuage>	 but if you put in tech news 'because of performance issues, the way to override the local logo changed etc' it's clear what's happening, why, and how to work around it
[20:21:18] <guillom>	 yup
[20:21:42] <ori>	 OK, I'm glad you're clarifying, because this was not only not my intent, but not even a consequence I anticipated. The task numbers referenced in the comments attached to particular logos contained many examples of community requests to change the logo, so I had assumed that that is how it was commonly done, rather than by clobbering the old logo by uploading a new revision.
[20:21:45] <Krenair>	 Nemo has a point that you might have just made some temporary logos more permanent. Are these going to be cached heavily?
[20:21:45] <grrrit-wm>	 (03PS3) 10EBernhardson: Enable CirrusSearch-PerUser PoolCounter on group0 only [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209802 (https://phabricator.wikimedia.org/T76497) 
[20:21:52] <grrrit-wm>	 (03PS4) 10EBernhardson: Enable CirrusSearch-PerUser PoolCounter on group0 only [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209802 (https://phabricator.wikimedia.org/T76497) 
[20:22:10] <ori>	 Krenair: yes, but we can bump the file name
[20:22:12] <grrrit-wm>	 (03CR) 10Manybubbles: "Thanks for doing this on the group0 first. I'm unreasonably afraid that something will blow up here. Probably because I wrote the pool cou" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209802 (https://phabricator.wikimedia.org/T76497) (owner: 10EBernhardson)
[20:22:47] <guillom>	 ori: Example: https://fr.wikipedia.org/wiki/Fichier:Wiki.png#filehistory (it hasn't changed in a few years on fr.wp but other wikis have continued with the tradition.)
[20:22:49] <ori>	 guillom: hang on a moment, if you don't mind, so we can figure out if we need to re-think this
[20:22:53] <grrrit-wm>	 (03CR) 10Manybubbles: [C: 031] "Want to schedule this for SWAT next week?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209802 (https://phabricator.wikimedia.org/T76497) (owner: 10EBernhardson)
[20:23:07] <icinga-wm>	 RECOVERY - HTTP 5xx req/min on graphite1001 is OK Less than 1.00% above the threshold [250.0]
[20:23:17] <guillom>	 ori: No problem; Just lets me know today so I can finish the newsletter before my week-end :)
[20:23:27] <icinga-wm>	 PROBLEM - High load average on labstore1001 is CRITICAL 62.50% of data above the critical threshold [24.0]
[20:23:49] <ori>	 losing the reference to the commons image seems unfortunate, independently of issues of control / autonomy
[20:23:53] <legoktm>	 ori: I believe that change was related to increased db slave lag we were seeing on s3 at the time, not necessarily job queue lengths. I was only watching/thinking about the SULF jobs, so I don't know whether it was a reaction or cause
[20:23:57] <ori>	 so one possible solution that i'd like to propose is as follows:
[20:24:33] <ori>	 1) we maintain the mapping of dbname => commons URL in a file in wmf-config
[20:25:02] <ori>	 2) we add a Makefile that retrieves each of those, applies maximum compression, etc.
[20:25:23] <ori>	 3) We run it nightly via cron?
[20:25:44] <ori>	 it's #3 i'm unsure of. it seems like it would only be useful for community logo hacks, but i think common.css is the better way to do that, honestly.
[20:26:08] <guillom>	 Krenair: I'm unsure about something now. I *know* that we used to use File:Wiki.png, including for local customization, but I also remember that when we rolled out the new logo a few years ago, there was a change in behavior. Are we still actually loading Wiki.png?
[20:26:58] <legoktm>	 I think some wikis were using File:Wiki.png and others were straight from commons...
[20:27:01] <valhallasw`nuage>	 guillom: yes. Not all logos came from commons
[20:27:04] <Krenair>	 Well not since ori's change
[20:27:07] <valhallasw`nuage>	 e.g. 'cswikiquote' => '//upload.wikimedia.org/wikiquote/cs/b/bc/Wiki.png',
[20:27:13] <guillom>	 Ok, thanks
[20:27:25] <Krenair>	 Before Ori's change, however...
[20:27:26] <ori>	 Krenair: I only migrated $wgLogos that were defined in the file
[20:27:32] <ori>	 and they were all //upload.wikimedia.org previously
[20:27:42] <Krenair>	 We still assumed all wikis defaulted to local Wiki.png
[20:28:02] <valhallasw`nuage>	 ori: //upload.wikimedia.org also hosts local uploads ;-)
[20:28:02] <Krenair>	 Some wikis used this, some went to commons
[20:28:26] <icinga-wm>	 RECOVERY - High load average on labstore1001 is OK Less than 50.00% above the threshold [16.0]
[20:28:35] * guillom --> meeting, bbl.
[20:29:12] <Krenair>	 ori, are you sure everything was set in the file?
[20:29:26] <ori>	 Krenair: whatever was not set in the file was not changed
[20:30:01] <Krenair>	 So some groups of wikis still default to �stdlogo
[20:30:03] <Krenair>	 $stdlogo*
[20:30:32] <ori>	 guillom: the specific Common.css hack is to add: #p-logo a { background-image: url("/image/url.png") !important; }, but please discourage doing that and ask that people request changes via phab instead.
[20:31:01] <guillom>	 ori: yeah, i don't want to encourage it either. I think just saying to use CSS is enough.
[20:31:18] <Krenair>	 There is https://meta.wikimedia.org/wiki/Requesting_wiki_configuration_changes
[20:31:22] <Krenair>	 But we will require community consensus
[20:31:41] <ori>	 guillom: *nod* Thanks very much.
[20:31:43] <Krenair>	 Local files only required a local admin to change
[20:31:51] <valhallasw`nuage>	 Krenair: should we? we can also just require sysop rights, to mirror what's needed for local files
[20:31:52] <Krenair>	 and commons a commons admin
[20:32:19] <valhallasw`nuage>	 and it was even less for some wikis, where the files were uploaded locally/on commons and maybe not protected?
[20:32:39] <ori>	 there were cases in which the logo image was not protected, yes
[20:32:49] <Krenair>	 I thought those were all cleaned up?
[20:32:51] <Krenair>	 sigh.
[20:33:06] <ori>	 there were in the past, I mean -- I didn't check, yesterday.
[20:33:07] <icinga-wm>	 PROBLEM - High load average on labstore1001 is CRITICAL 62.50% of data above the critical threshold [24.0]
[20:33:23] <Krenair>	 valhallasw`nuage, well it's not exactly my decision
[20:33:39] <ori>	 it's not exactly not your decision, either.
[20:33:59] <valhallasw`nuage>	 it's anyone who has +2's decision :-p
[20:34:22] <ori>	 well, let me explain for a second why the historic approach is horrible from a perf perspective
[20:34:22] <Krenair>	 True.
[20:34:30] <ori>	 and you guys tell me what you think is sensible
[20:34:54] <Krenair>	 They're hugely-used files with the cache expiries of normal files, I'm guessing?
[20:35:00] <csteipp>	 Krenair / yurik: I'm ok with it on for wikis where editing is highly restricted. So no, definitely not private wikis. This is along the lines of allowing raw html.
[20:35:15] <ori>	 Krenair: cache expiries of normal files == no cache expiry
[20:35:40] <Krenair>	 csteipp, so basically we can allow it on foundationwiki, and... the other wikis where we allow raw html?
[20:35:53] <Krenair>	 ori, heh, ok
[20:35:54] <ori>	 uploading a new version of a file keeps the URL in tact, so we can't set far-future expires
[20:36:07] <ori>	 basically the browser has to check on each page view if there is a new logo
[20:36:54] <ori>	 if the files and the references are both managed in operations/mediawiki-config, whenever a logo is changed, we can simply increment a version counter in the filename (enwiki.v1.png, enwiki.v2.png, etc.)
[20:37:03] <ori>	 this allows us to set a max-age of one year
[20:37:07] <Krenair>	 I think the change makes sense
[20:37:43] <ori>	 thanks for explaining the community impact, it was not obvious to me
[20:37:52] <Krenair>	 But there was no announcement about it
[20:37:58] <Krenair>	 No discussion about the consequences with respect to policies
[20:38:02] <valhallasw`nuage>	 ori: so, it's not really using common.css file that's the issue, it's using commons specifically. So if we allow the community to upload logos for static without extensive approval, they can still do the switch in common.css
[20:38:04] <Krenair>	 No checks for logos that should be temporary
[20:38:23] <Krenair>	 There appears to be a gap for the wikinews/wikiversity defaults
[20:38:58] <ori>	 valhallasw`nuage: I'm not sure I follow
[20:39:22] <ori>	 Krenair: I didn't know about these practices (local temp changes). The example guillom provided is from 2010. Is this still commonly done?
[20:39:24] <valhallasw`nuage>	 ori: a commons.css override is fine, as long as the file it refers to is on static and not on upload
[20:39:49] <Krenair>	 I know we've done a temp logo change or two through config changes
[20:39:56] <Krenair>	 I wouldn't say it's commonly done
[20:40:08] <Krenair>	 But I wouldn't say most wikis have a regular stream of config requests, so...
[20:40:09] <valhallasw`nuage>	 so that means we don't even have to require community consensus, we can just upload the file to static and let the community do the switching themselves
[20:40:10] <ori>	 temp logo changes via config is still doable like before
[20:40:23] <yurik>	 csteipp, i was under the impression that private wikis are highly restricted. will make it case by case than
[20:40:27] <grrrit-wm>	 (03CR) 10Nemo bis: "Quick look: bawiki.png is wrong, celebrative logo. Needs to be amended here and then restored on the local wiki via MediaWiki:Common.css u" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209615 (owner: 10Ori.livneh)
[20:41:02] <grrrit-wm>	 (03CR) 10Nemo bis: "Also anwiki" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209615 (owner: 10Ori.livneh)
[20:41:12] <Krenair>	 	'donatewiki' => true,
[20:41:13] <Krenair>	 	'foundationwiki' => true,
[20:41:13] <Krenair>	 	'internalwiki' => true,
[20:41:13] <Krenair>	 	'collabwiki' => true, // Bug T31269
[20:41:13] <Krenair>	 	'zerowiki' => true,
[20:41:20] <Krenair>	 These wikis allow raw HTML, yurik 
[20:41:54] <yurik>	 Krenair, csteipp , i will include just these than
[20:41:59] <csteipp>	 yurik: No, when an editor can modify a graph that a more privileged user will see, then this is not safe
[20:42:13] <Krenair>	 csteipp, well to be fair we allow admins to modify JS
[20:42:25] <csteipp>	 On zerowiki (and foundationwiki), those who can edit are the most trusted users.
[20:42:26] <Krenair>	 And then send that JS to CU/OS/stewards/staff/sysadmins/whatever
[20:42:32] <ori>	 Nemo_bis, Krenair, valhallasw`nuage, guillom: I'll file a task to look into these use-cases and make sure there is a coherent process in place for carrying them out.
[20:42:39] <valhallasw`nuage>	 sounds good
[20:43:09] <csteipp>	 Krenair: Yes... although I'd like to do less of that, not more.
[20:43:14] <Krenair>	 But I suppose we should try to avoid introducing more things like that
[20:43:15] <Krenair>	 yeah
[20:43:43] <Krenair>	 thanks ori
[20:43:57] <ori>	 thanks for the explanations
[20:44:28] <grrrit-wm>	 (03CR) 10CSteipp: [C: 031] "I'm ok with removing the headers on wikis with highly restricted editing-- like zerowiki or foundationwiki. In general, not on any wikis w" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209800 (owner: 10Yurik)
[20:47:00] <grrrit-wm>	 (03CR) 10Alex Monk: [C: 04-1] "Shouldn't we just set $wgGraphIsTrusted = $wgRawHtml; then?" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209800 (owner: 10Yurik)
[20:48:24] <yurik>	 Krenair, trusted graph functionality will be needed whenever the data accessed by the graph, e.g. with action=raw, is not visible otherwise
[20:49:27] <icinga-wm>	 PROBLEM - High load average on labstore1001 is CRITICAL 50.00% of data above the critical threshold [24.0]
[20:52:38] <yurik>	 does labstore1001 keep /srv data in it?
[20:52:52] <yurik>	 i might be causing it if it is
[20:55:05] <grrrit-wm>	 (03CR) 10Catrope: [C: 032] Have Beta Cluster use same settings as prod for LQT [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209776 (owner: 10Mattflaschen)
[20:55:11] <grrrit-wm>	 (03Merged) 10jenkins-bot: Have Beta Cluster use same settings as prod for LQT [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209776 (owner: 10Mattflaschen)
[20:57:23] <grrrit-wm>	 (03PS2) 10Yurik: Disable Graph security header on zero wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209800 
[20:59:57] <grrrit-wm>	 (03CR) 10Alex Monk: [C: 04-1] Disable Graph security header on zero wiki (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209800 (owner: 10Yurik)
[21:00:33] <grrrit-wm>	 (03CR) 10Alex Monk: "Do we really want to grant more strange exceptions to this wiki? :/" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209800 (owner: 10Yurik)
[21:02:27] <grrrit-wm>	 (03CR) 10Yurik: "Alex, this is the new feature - zero portal has been without any special headers since graphs were introduced (specifically developed for " [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209800 (owner: 10Yurik)
[21:02:29] <logmsgbot>	 !log mattflaschen Synchronized wmf-config/InitialiseSettings-labs.php: Sync out change that only affects Beta Cluster (duration: 00m 11s)
[21:02:38] <morebots>	 Logged the message, Master
[21:03:17] <yurik>	 Krenair, csteipp, greg-g, i need to deploy this - zero portal is broken
[21:04:12] <Krenair>	 Sigh, ok, if it's needed to unbreak existing functionality.
[21:04:37] <greg-g>	 Krenair: you're now my deputy? :P
[21:04:47] <Krenair>	 Deal with the -1 (just the comment needs updating) and I won't block it
[21:05:04] <Krenair>	 greg-g, no, I just not going to -1 it
[21:05:07] <Krenair>	 I'm*
[21:05:29] <ori>	 yurik: can you explain this change to me?
[21:05:33] <ori>	 the commit message is unclear
[21:05:38] <Krenair>	 I'm not approving of it or anything
[21:05:46] <icinga-wm>	 PROBLEM - High load average on labstore1001 is CRITICAL 50.00% of data above the critical threshold [24.0]
[21:05:53] <yurik>	 ori, https://phabricator.wikimedia.org/T98313
[21:05:58] <ori>	 "Zero wiki portal can access any graph data they need, including access to the API." <--- is that the problem you are trying to solve, or how things are intended to work?
[21:06:43] <yurik>	 ori, basically this restores zerowiki to the state it has always been in
[21:06:47] <ori>	 bawolff is such a badass
[21:06:56] <yurik>	 oh yeah :)
[21:08:49] <grrrit-wm>	 (03PS3) 10Yurik: Disable Graph security header on zero wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209800 
[21:08:50] <yurik>	 Krenair, ^
[21:09:17] <Krenair>	 No objections from me...
[21:09:34] <yurik>	 Krenair, +1 ;)
[21:10:01] <Krenair>	 I guess you could explain what untrusted wiki means in this context, in that comment
[21:10:10] <yurik>	 greg-g, deploying ^
[21:10:14] <Krenair>	 but I won't -1 it
[21:10:31] <yurik>	 let me get the wiki in the working order first
[21:10:42] <grrrit-wm>	 (03CR) 10Yurik: [C: 032] Disable Graph security header on zero wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209800 (owner: 10Yurik)
[21:12:29] <grrrit-wm>	 (03Merged) 10jenkins-bot: Disable Graph security header on zero wiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209800 (owner: 10Yurik)
[21:14:41] <logmsgbot>	 !log yurik Synchronized wmf-config/InitialiseSettings.php: Disable security header for Graphs on zerowiki (duration: 00m 12s)
[21:14:46] <morebots>	 Logged the message, Master
[21:15:16] <icinga-wm>	 PROBLEM - check_disk on lutetium is CRITICAL: DISK CRITICAL - free space: / 3797 MB (10% inode=92%): /dev 32200 MB (99% inode=99%): /run 6403 MB (99% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 32209 MB (100% inode=99%): /srv 352765 MB (24% inode=99%)
[21:15:17] <icinga-wm>	 PROBLEM - High load average on labstore1001 is CRITICAL 50.00% of data above the critical threshold [24.0]
[21:16:56] <icinga-wm>	 RECOVERY - High load average on labstore1001 is OK Less than 50.00% above the threshold [16.0]
[21:17:15] <logmsgbot>	 !log yurik Synchronized wmf-config/CommonSettings.php: Disable security header for Graphs on zerowiki (duration: 00m 12s)
[21:17:23] <morebots>	 Logged the message, Master
[21:20:17] <icinga-wm>	 PROBLEM - check_disk on lutetium is CRITICAL: DISK CRITICAL - free space: / 3314 MB (9% inode=92%): /dev 32200 MB (99% inode=99%): /run 6403 MB (99% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 32209 MB (100% inode=99%): /srv 351885 MB (24% inode=99%)
[21:20:44] <K4-713>	 Bah, not again. 
[21:21:58] <icinga-wm>	 PROBLEM - High load average on labstore1001 is CRITICAL 75.00% of data above the critical threshold [24.0]
[21:25:16] <icinga-wm>	 PROBLEM - check_disk on lutetium is CRITICAL: DISK CRITICAL - free space: / 2848 MB (7% inode=92%): /dev 32200 MB (99% inode=99%): /run 6403 MB (99% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 32209 MB (100% inode=99%): /srv 351057 MB (24% inode=99%)
[21:25:30] <Krenair>	 ori, also, what are we doing about all of those commons/local wiki file pages?
[21:25:35] <Krenair>	 That are now no longer used?
[21:25:46] <Krenair>	 Doesn't commons mark those things as part of the interface, and now they need to be removed, unprotected etc.?
[21:30:16] <icinga-wm>	 PROBLEM - check_disk on lutetium is CRITICAL: DISK CRITICAL - free space: / 2555 MB (7% inode=92%): /dev 32200 MB (99% inode=99%): /run 6403 MB (99% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 32209 MB (100% inode=99%): /srv 350500 MB (24% inode=99%)
[21:33:07] <icinga-wm>	 RECOVERY - High load average on labstore1001 is OK Less than 50.00% above the threshold [16.0]
[21:35:16] <icinga-wm>	 PROBLEM - check_disk on lutetium is CRITICAL: DISK CRITICAL - free space: / 2555 MB (7% inode=92%): /dev 32200 MB (99% inode=99%): /run 6403 MB (99% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 32209 MB (100% inode=99%): /srv 350500 MB (24% inode=99%)
[21:40:16] <icinga-wm>	 PROBLEM - check_disk on lutetium is CRITICAL: DISK CRITICAL - free space: / 2554 MB (7% inode=92%): /dev 32200 MB (99% inode=99%): /run 6403 MB (99% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 32209 MB (100% inode=99%): /srv 360856 MB (24% inode=99%)
[21:45:16] <icinga-wm>	 PROBLEM - check_disk on lutetium is CRITICAL: DISK CRITICAL - free space: / 2554 MB (7% inode=92%): /dev 32200 MB (99% inode=99%): /run 6403 MB (99% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 32209 MB (100% inode=99%): /srv 360856 MB (24% inode=99%)
[21:50:16] <icinga-wm>	 PROBLEM - check_disk on lutetium is CRITICAL: DISK CRITICAL - free space: / 2554 MB (7% inode=92%): /dev 32200 MB (99% inode=99%): /run 6403 MB (99% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 32209 MB (100% inode=99%): /srv 360856 MB (24% inode=99%)
[21:51:42] <mutante>	 ^ eh, that's because it of the /run/ part and it's fundraising
[21:52:58] <bblack>	 I don't have access, so don't ask me!
[21:53:09] <mutante>	 and /srv are fine. but should it even check /run ?
[21:53:19] <mutante>	  "/ and /srv are fine?
[21:54:06] <icinga-wm>	 PROBLEM - High load average on labstore1001 is CRITICAL 75.00% of data above the critical threshold [24.0]
[21:54:14] <bblack>	 it looks like it's only complaining about the root fs
[21:54:27] <bblack>	 7% space free left
[21:54:32] <mutante>	 bblack: it looks to me like only /run is the issue
[21:54:47] <mutante>	 because that is 99%
[21:54:53] <bblack>	 99% free, right?
[21:55:10] <bblack>	 the /run/ stuff is probably all tmpfs/shmfs type things anyways
[21:55:14] <mutante>	 arr, right
[21:55:16] <icinga-wm>	 PROBLEM - check_disk on lutetium is CRITICAL: DISK CRITICAL - free space: / 2553 MB (7% inode=92%): /dev 32200 MB (99% inode=99%): /run 6403 MB (99% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 32209 MB (100% inode=99%): /srv 360855 MB (24% inode=99%)
[21:55:35] <mutante>	 so it is probably /home again, i'm checking
[21:55:45] <mutante>	 there are users on it generating large files
[21:56:14] <greg-g>	 mutante: yeah, that's what K4-713 just said in person
[21:56:22] <mutante>	 yea 2 home dirs with over 5G each
[21:56:32] <mutante>	 there is also /srv/ which has way more space
[21:56:52] <mutante>	 so if they could use that instead
[22:00:16] <icinga-wm>	 PROBLEM - check_disk on lutetium is CRITICAL: DISK CRITICAL - free space: / 2553 MB (7% inode=92%): /dev 32200 MB (99% inode=99%): /run 6403 MB (99% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 32209 MB (100% inode=99%): /srv 360855 MB (24% inode=99%)
[22:02:17] <icinga-wm>	 RECOVERY - High load average on labstore1001 is OK Less than 50.00% above the threshold [16.0]
[22:05:16] <icinga-wm>	 PROBLEM - check_disk on lutetium is CRITICAL: DISK CRITICAL - free space: / 3516 MB (9% inode=92%): /dev 32200 MB (99% inode=99%): /run 6403 MB (99% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 32209 MB (100% inode=99%): /srv 360855 MB (24% inode=99%)
[22:10:16] <icinga-wm>	 PROBLEM - check_disk on lutetium is CRITICAL: DISK CRITICAL - free space: / 3516 MB (9% inode=92%): /dev 32200 MB (99% inode=99%): /run 6403 MB (99% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 32209 MB (100% inode=99%): /srv 360854 MB (24% inode=99%)
[22:11:04] <mutante>	 !log gzipping some user data on lutetium 
[22:11:13] <morebots>	 Logged the message, Master
[22:20:16] <icinga-wm>	 RECOVERY - check_disk on lutetium is OK: DISK OK - free space: / 9191 MB (25% inode=92%): /dev 32200 MB (99% inode=99%): /run 6403 MB (99% inode=99%): /run/lock 5 MB (100% inode=99%): /run/shm 32209 MB (100% inode=99%): /srv 360854 MB (24% inode=99%)
[22:36:57] <grrrit-wm>	 (03PS1) 10coren: Tool Labs: add whois to exec nodes [puppet] - 10https://gerrit.wikimedia.org/r/209818 (https://phabricator.wikimedia.org/T98555) 
[22:37:36] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Tool Labs: add whois to exec nodes [puppet] - 10https://gerrit.wikimedia.org/r/209818 (https://phabricator.wikimedia.org/T98555) (owner: 10coren)
[22:39:23] <Negative24>	 jouncebot: next
[22:39:23] <jouncebot>	 In 62 hour(s) and 20 minute(s): Wikidata (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20150511T1300)
[22:39:33] <Coren>	 Hm.  Jenkins looks broked.  Truncated files smell like out of disk space.
[22:40:04] <mutante>	 Coren: try to put "recheck" once 
[22:40:13] <mutante>	 i had a similar one and it just worked the second time
[22:40:57] <mutante>	 duh, grrrit-wm
[22:41:11] <mutante>	 just when i wanted it to tell us
[22:41:33] <mutante>	 Coren: fixed 
[22:41:48] <legoktm>	 Coren: it's not disk space, /mnt has plenty of room left
[22:41:59] <mutante>	 it verified it now
[22:42:32] <Coren>	 legoktm: Yeah; I saw after that.  But empty files are a common symptom so it looked like a good place to check.  :-)
[22:42:38] <Coren>	 ty mutante
[22:42:52] <grrrit-wm>	 (03CR) 10coren: [C: 032] "Trivial package addition" [puppet] - 10https://gerrit.wikimedia.org/r/209818 (https://phabricator.wikimedia.org/T98555) (owner: 10coren)
[23:09:35] <grrrit-wm>	 (03PS1) 10BryanDavis: Fix logster tracking for CirrusSearch-slow.log [puppet] - 10https://gerrit.wikimedia.org/r/209821 
[23:10:17] <icinga-wm>	 PROBLEM - High load average on labstore1001 is CRITICAL 50.00% of data above the critical threshold [24.0]
[23:11:47] <icinga-wm>	 RECOVERY - High load average on labstore1001 is OK Less than 50.00% above the threshold [16.0]
[23:15:33] <wikibugs>	 6operations: snaphot1004 running dumps very slowly, investigate - https://phabricator.wikimedia.org/T98585#1273764 (10Tbayer)
[23:16:14] <grrrit-wm>	 (03CR) 10BryanDavis: "applied via cherry-pick; works" [puppet] - 10https://gerrit.wikimedia.org/r/209769 (owner: 10BryanDavis)
[23:16:32] <grrrit-wm>	 (03CR) 10BryanDavis: "applied via cherry-pick; works" [puppet] - 10https://gerrit.wikimedia.org/r/209801 (owner: 10BryanDavis)
[23:17:24] <grrrit-wm>	 (03PS2) 10Yuvipanda: beta: hiera config for deployment-fluorine [puppet] - 10https://gerrit.wikimedia.org/r/209769 (owner: 10BryanDavis)
[23:18:15] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032] beta: hiera config for deployment-fluorine [puppet] - 10https://gerrit.wikimedia.org/r/209769 (owner: 10BryanDavis)
[23:19:09] <grrrit-wm>	 (03PS2) 10Yuvipanda: beta: use deployment-fluorine as udp2log aggregator [puppet] - 10https://gerrit.wikimedia.org/r/209801 (owner: 10BryanDavis)
[23:19:27] <bd808>	 yuvipanda: you should link that one to the phab bug
[23:19:32] <bd808>	 I just noticed I didn't 
[23:20:17] <grrrit-wm>	 (03PS3) 10Yuvipanda: beta: use deployment-fluorine as udp2log aggregator [puppet] - 10https://gerrit.wikimedia.org/r/209801 (https://phabricator.wikimedia.org/T98289) (owner: 10BryanDavis)
[23:20:51] <grrrit-wm>	 (03CR) 10Yuvipanda: [C: 032 V: 032] beta: use deployment-fluorine as udp2log aggregator [puppet] - 10https://gerrit.wikimedia.org/r/209801 (https://phabricator.wikimedia.org/T98289) (owner: 10BryanDavis)
[23:21:16] <yuvipanda>	 bd808: done
[23:28:58] <grrrit-wm>	 (03PS1) 10BryanDavis: beta: switch $wmfUdp2logDest to deployment-fluorine.eqiad.wmflabs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209825 (https://phabricator.wikimedia.org/T98289) 
[23:35:07] <icinga-wm>	 RECOVERY - puppet last run on einsteinium is OK Puppet is currently enabled, last run 45 seconds ago with 0 failures
[23:36:32] <bd808>	 can I get a +1 from someone on https://gerrit.wikimedia.org/r/#/c/209825 just as a sanity check. beta udp2log logging only change
[23:37:02] * bd808 is trying to help keep nfs in labs from paging folks over the weekend
[23:38:07] <bd808>	 Reedy: ^^
[23:38:45] <grrrit-wm>	 (03CR) 10Jforrester: Re-enable OAuth on wikitech (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209744 (https://phabricator.wikimedia.org/T98567) (owner: 10Alex Monk)
[23:39:31] <grrrit-wm>	 (03CR) 10Alex Monk: Re-enable OAuth on wikitech (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209744 (https://phabricator.wikimedia.org/T98567) (owner: 10Alex Monk)
[23:40:05] * Reedy attemps to login to gerrit
[23:41:01] <Bsadowski1>	 Reedy
[23:41:18] <grrrit-wm>	 (03CR) 10Jforrester: Re-enable OAuth on wikitech (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209744 (https://phabricator.wikimedia.org/T98567) (owner: 10Alex Monk)
[23:42:08] <Reedy>	 Can't even get logged into my password manager
[23:42:13] <Reedy>	 This is going well
[23:42:57] <bd808>	 meh. I can jfdi if you are stuck
[23:44:44] <grrrit-wm>	 (03CR) 10BryanDavis: [C: 032] ""Reedy made me do it" (or other Friday self-merge excuse)" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209825 (https://phabricator.wikimedia.org/T98289) (owner: 10BryanDavis)
[23:44:50] <grrrit-wm>	 (03Merged) 10jenkins-bot: beta: switch $wmfUdp2logDest to deployment-fluorine.eqiad.wmflabs [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209825 (https://phabricator.wikimedia.org/T98289) (owner: 10BryanDavis)
[23:45:25] <grrrit-wm>	 (03CR) 10Alex Monk: Re-enable OAuth on wikitech (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/209744 (https://phabricator.wikimedia.org/T98567) (owner: 10Alex Monk)
[23:45:49] <logmsgbot>	 !log bd808 Synchronized wmf-config/CommonSettings.php: beta: switch $wmfUdp2logDest to deployment-fluorine.eqiad.wmflabs (duration: 00m 12s)
[23:45:56] <morebots>	 Logged the message, Master
[23:49:10] <bd808>	 yuvipanda: I'm not seeing an logs written to /data/project/logs now and I am seeing them on deployment-fluorine
[23:49:23] <yuvipanda>	 bd808: \o/
[23:49:36] <bd808>	 I think I still need a scap config change too to get the last of them
[23:50:06] <yuvipanda>	 we shouldn’t delete the logs today, or do anything to them (the ones on NFS) until the backup is done, I reckon
[23:50:15] <bd808>	 agreed
[23:50:32] <bd808>	 just getting the write pressure off should help you out
[23:50:48] <yuvipanda>	 yeah +1
[23:51:00] <bd808>	 wfDebug.log is 129M already in ~5 minutes
[23:51:09] <yuvipanda>	 haha wow
[23:51:10] <yuvipanda>	 yeah
[23:51:38] <Reedy>	 much logging
[23:51:46] <bd808>	 such noise
[23:51:48] <bd808>	 wow
[23:52:03] <yuvipanda>	 heh
[23:58:09] <wikibugs>	 6operations, 10Deployment-Systems, 7Graphite, 5Patch-For-Review: [scap] Deploy events aren't showing up in graphite/gdash - https://phabricator.wikimedia.org/T64667#1273893 (10greg) Thanks @fgiunchedi !