[00:22:26] PROBLEM - Disk space on palladium is CRITICAL: DISK CRITICAL - free space: / 4074 MB (3% inode=95%): [00:28:06] RECOVERY - Disk space on palladium is OK: DISK OK [02:21:03] !log l10nupdate Synchronized php-1.25wmf21/cache/l10n: (no message) (duration: 04m 51s) [02:21:21] Logged the message, Master [02:21:43] FYI: Deploying https://gerrit.wikimedia.org/r/198671 [02:24:16] !log krinkle Synchronized php-1.25wmf22/includes/TemplateParser.php: Ie90074e4885de7340e (duration: 00m 06s) [02:24:20] Logged the message, Master [02:24:29] !log LocalisationUpdate completed (1.25wmf21) at 2015-03-23 02:23:26+00:00 [02:24:32] Logged the message, Master [02:24:35] Krenair: watch 'tail -n400 /a/mw-log/hhvm.log | grep "TemplateParser.php"' [02:25:09] !log l10nupdate Synchronized php-1.25wmf22/cache/l10n: (no message) (duration: 00m 03s) [02:25:16] Logged the message, Master [02:26:11] Meh, I guess using fatalmonitor works fine too [02:26:16] !log LocalisationUpdate completed (1.25wmf22) at 2015-03-23 02:25:13+00:00 [02:26:18] it does drop off [02:26:21] Logged the message, Master [02:26:26] IIRC it takes a while to drop from fatalmonitor [02:28:09] `tail -f /a/mw-log/hhvm.log` is not too fast to see what's going on [03:05:32] Deploying https://gerrit.wikimedia.org/r/198672 [03:07:21] !log krinkle Synchronized php-1.25wmf21/includes/TemplateParser.php: Ie90074e4885de7340e (duration: 00m 06s) [03:07:27] Logged the message, Master [03:27:16] PROBLEM - puppet last run on cp3016 is CRITICAL: CRITICAL: puppet fail [03:30:19] (03CR) 10KartikMistry: "Eh. Not merged yet? :)" [debs/contenttranslation/apertium-hbs-mkd] - 10https://gerrit.wikimedia.org/r/195264 (https://phabricator.wikimedia.org/T89936) (owner: 10KartikMistry) [03:45:57] RECOVERY - puppet last run on cp3016 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [03:48:09] 6operations, 10Parsoid, 6Services: Move Parsoid config into ops/puppet - https://phabricator.wikimedia.org/T92636#1139870 (10faidon) 5Open>3declined a:3faidon I've suggested alternatives but these don't seem to be addressed/commented on? I also don't see the Beta people pushing for this (if anything,... [04:15:55] PROBLEM - puppet last run on mw2091 is CRITICAL: CRITICAL: puppet fail [04:18:46] PROBLEM - puppet last run on mw2196 is CRITICAL: CRITICAL: puppet fail [04:34:35] RECOVERY - puppet last run on mw2091 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:37:26] RECOVERY - puppet last run on mw2196 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:42:35] (03PS1) 10Gage: IPsec: use aes128gcm instead of aes256gcm [puppet] - 10https://gerrit.wikimedia.org/r/198685 [04:45:52] 6operations, 3codfw-appserver-setup, 7database, 3wikis-in-codfw: Grant access to the databases to codfw appserver networks - https://phabricator.wikimedia.org/T93211#1139884 (10Springle) a:3Springle [04:55:08] 6operations: Retire Torrus - https://phabricator.wikimedia.org/T87840#1139901 (10Gage) Not sure, as @faidon declined my ticket to monitor the Netapp stats with LibreNMS. So those stats are still only being collected by Torrus. Faidon, what would you like to do? [04:57:21] 6operations: Retire Torrus - https://phabricator.wikimedia.org/T87840#1139902 (10faidon) p:5Normal>3Low I don't use Torrus so I wouldn't mind it if it was gone. I know @mark uses it/likes it, though, so he will probably be against killing it entirely. [04:57:50] (03PS1) 10Springle: Grant access to the databases from codfw appserver networks [puppet] - 10https://gerrit.wikimedia.org/r/198686 (https://phabricator.wikimedia.org/T93211) [04:59:23] (03CR) 10Springle: [C: 032] Grant access to the databases from codfw appserver networks [puppet] - 10https://gerrit.wikimedia.org/r/198686 (https://phabricator.wikimedia.org/T93211) (owner: 10Springle) [05:03:57] (03CR) 10Faidon Liambotis: "That doesn't make any sense -- what is exactly being installed by default?" [puppet] - 10https://gerrit.wikimedia.org/r/198227 (https://phabricator.wikimedia.org/T90922) (owner: 10Filippo Giunchedi) [05:04:38] (03CR) 10Faidon Liambotis: [C: 031] "I don't see a particular need for code review for those changes :)" [software/swift-ring] - 10https://gerrit.wikimedia.org/r/198256 (https://phabricator.wikimedia.org/T1268) (owner: 10Filippo Giunchedi) [05:16:13] 6operations, 5Patch-For-Review, 3codfw-appserver-setup, 7database, 3wikis-in-codfw: Grant access to the databases to codfw appserver networks - https://phabricator.wikimedia.org/T93211#1139909 (10Springle) Grants for 'wikiuser'@'10.192.%' now exist on shards s[1-7], es[1-3], x1. [05:30:17] (03PS3) 10Springle: Replicate centralauth tables in dbstores [puppet] - 10https://gerrit.wikimedia.org/r/198292 (owner: 10Hoo man) [05:31:11] (03CR) 10Springle: [C: 032] Replicate centralauth tables in dbstores [puppet] - 10https://gerrit.wikimedia.org/r/198292 (owner: 10Hoo man) [06:03:21] !log LocalisationUpdate ResourceLoader cache refresh completed at Mon Mar 23 06:02:14 UTC 2015 (duration 2m 13s) [06:03:25] Logged the message, Master [06:29:26] PROBLEM - puppet last run on cp1056 is CRITICAL: CRITICAL: Puppet has 1 failures [06:29:46] PROBLEM - puppet last run on mw1009 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:16] PROBLEM - puppet last run on mw1123 is CRITICAL: CRITICAL: Puppet has 1 failures [06:30:16] PROBLEM - puppet last run on db1051 is CRITICAL: CRITICAL: Puppet has 1 failures [06:31:05] PROBLEM - puppet last run on cp4014 is CRITICAL: CRITICAL: Puppet has 1 failures [06:33:57] PROBLEM - puppet last run on mw1144 is CRITICAL: CRITICAL: Puppet has 1 failures [06:34:16] PROBLEM - puppet last run on mw1065 is CRITICAL: CRITICAL: Puppet has 1 failures [06:35:56] PROBLEM - puppet last run on mw2143 is CRITICAL: CRITICAL: Puppet has 1 failures [06:36:15] PROBLEM - puppet last run on mw2022 is CRITICAL: CRITICAL: Puppet has 1 failures [06:36:15] PROBLEM - puppet last run on mw2013 is CRITICAL: CRITICAL: Puppet has 1 failures [06:45:35] RECOVERY - puppet last run on mw1009 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [06:46:05] RECOVERY - puppet last run on db1051 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [06:46:57] RECOVERY - puppet last run on mw1144 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [06:47:06] RECOVERY - puppet last run on mw1065 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [06:47:26] RECOVERY - puppet last run on mw2143 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [06:47:47] RECOVERY - puppet last run on mw2022 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [06:47:47] RECOVERY - puppet last run on mw2013 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:48:06] RECOVERY - puppet last run on cp1056 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [06:48:16] RECOVERY - puppet last run on cp4014 is OK: OK: Puppet is currently enabled, last run 48 seconds ago with 0 failures [06:48:55] RECOVERY - puppet last run on mw1123 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:05:30] (03PS1) 10Nemo bis: Restore unregistered editing on mobile sites (staggered) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/198691 (https://phabricator.wikimedia.org/T93210) [07:23:44] (03PS1) 10KartikMistry: Beta: Add missing 'af' and 'az' [puppet] - 10https://gerrit.wikimedia.org/r/198692 [07:37:17] (03PS1) 10KartikMistry: Apertium: Install needed language pairs for CX [puppet] - 10https://gerrit.wikimedia.org/r/198693 [08:26:06] PROBLEM - puppet last run on mw1089 is CRITICAL: CRITICAL: Puppet has 1 failures [08:43:16] RECOVERY - puppet last run on mw1089 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [09:20:45] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] eqiad-prod: add ms-be101[678] [software/swift-ring] - 10https://gerrit.wikimedia.org/r/198256 (https://phabricator.wikimedia.org/T1268) (owner: 10Filippo Giunchedi) [09:22:35] !log deploy new swift ring including ms-be101[678] [09:22:41] Logged the message, Master [09:35:16] PROBLEM - puppet last run on ms-be1018 is CRITICAL: CRITICAL: Puppet has 2 failures [09:38:26] PROBLEM - puppet last run on es2010 is CRITICAL: CRITICAL: puppet fail [09:42:26] PROBLEM - puppet last run on ms-be1016 is CRITICAL: CRITICAL: Puppet has 2 failures [09:44:36] PROBLEM - puppet last run on ms-be1017 is CRITICAL: CRITICAL: Puppet has 2 failures [09:52:16] PROBLEM - puppet last run on mw1055 is CRITICAL: CRITICAL: Puppet has 1 failures [09:55:46] RECOVERY - puppet last run on es2010 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [10:05:10] (03CR) 10Yuvipanda: "*bump*" [puppet] - 10https://gerrit.wikimedia.org/r/192335 (owner: 10coren) [10:08:15] RECOVERY - puppet last run on mw1055 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [10:09:16] (03PS1) 10Filippo Giunchedi: swift: increase rsync server max_connections [puppet] - 10https://gerrit.wikimedia.org/r/198697 (https://phabricator.wikimedia.org/T1268) [10:12:11] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] swift: increase rsync server max_connections [puppet] - 10https://gerrit.wikimedia.org/r/198697 (https://phabricator.wikimedia.org/T1268) (owner: 10Filippo Giunchedi) [10:36:43] (03CR) 10Filippo Giunchedi: "afaict trusty installs with linux-image 3.13.0-24-generic (from main) however installer boots with linux-image 3.13.0-32-generic which acc" [puppet] - 10https://gerrit.wikimedia.org/r/198227 (https://phabricator.wikimedia.org/T90922) (owner: 10Filippo Giunchedi) [10:38:01] (03PS3) 10Filippo Giunchedi: install-server: upgrade kernel on swift HP machines [puppet] - 10https://gerrit.wikimedia.org/r/198227 (https://phabricator.wikimedia.org/T90922) [10:39:02] (03PS4) 10Filippo Giunchedi: install-server: upgrade kernel on swift HP machines [puppet] - 10https://gerrit.wikimedia.org/r/198227 (https://phabricator.wikimedia.org/T90922) [10:48:47] !log Manually attached frwiki:Otets to the global account Otets [10:48:56] Logged the message, Master [10:50:01] !log downgrade rsync to 3.0.9-1ubuntu1 on ms-be101[678] (precise's version) problems when senders are on 3.0.9 but receivers 3.1 [10:50:04] Logged the message, Master [10:59:39] <_joe_> godog: ouch! [11:01:54] heh, I've downgraded for now but I'll need to figure it out [11:02:30] (03CR) 10Mobrovac: [C: 031] "@GWicke re deploy, I think the best way to proceed would be:" [puppet] - 10https://gerrit.wikimedia.org/r/198433 (https://phabricator.wikimedia.org/T93452) (owner: 10GWicke) [11:23:24] ping hashar [11:27:32] 7Puppet, 6Multimedia, 6Release-Engineering, 6Scrum-of-Scrums, and 2 others: Create basic puppet role for Sentry - https://phabricator.wikimedia.org/T84956#1140290 (10Gilles) I've turned what I had done manually into a shell script that will automatically package the latest version of these django extension... [11:28:46] (03CR) 10Filippo Giunchedi: "agreed on the coordinated deploy/merge, wouldn't be easier to rollout in chunks instead of all together? that'd give us control on how fas" [puppet] - 10https://gerrit.wikimedia.org/r/198433 (https://phabricator.wikimedia.org/T93452) (owner: 10GWicke) [11:30:41] (03CR) 10Filippo Giunchedi: [C: 031] add ferm service for poolcounterd [puppet] - 10https://gerrit.wikimedia.org/r/198442 (https://phabricator.wikimedia.org/T93261) (owner: 10Dzahn) [11:31:17] (03CR) 10Filippo Giunchedi: [C: 031] have base::firewall on codfw poolcounters [puppet] - 10https://gerrit.wikimedia.org/r/198440 (https://phabricator.wikimedia.org/T93261) (owner: 10Dzahn) [11:35:21] godog: you aren’t in the sekrit channel? :) [11:43:33] (03CR) 10Mobrovac: "This patch creates 6 new keyspaces in Cassandra. The good thing about the manual approach I suggested is that these are created sequential" [puppet] - 10https://gerrit.wikimedia.org/r/198433 (https://phabricator.wikimedia.org/T93452) (owner: 10GWicke) [11:48:13] (03CR) 10Filippo Giunchedi: [C: 031] Tools: Don't let user names mask system aliases [puppet] - 10https://gerrit.wikimedia.org/r/198563 (owner: 10Tim Landscheidt) [11:48:34] (03CR) 10Filippo Giunchedi: [C: 031] Tools: Make "admin" and "administrator" system aliases [puppet] - 10https://gerrit.wikimedia.org/r/198571 (owner: 10Tim Landscheidt) [11:49:40] !log restart txstatsd on graphite1001 to drop old diamond metrics [11:49:46] Logged the message, Master [11:57:56] godog: heard there’s a txstatsd replacement coming :D [12:03:21] YuviPanda: yeah! this guy https://gerrit.wikimedia.org/r/#/c/193095/ as soon I'm done with the three new swift machines [12:03:34] godog: nice :D [12:03:58] godog: might be required for next quarter’s labs goals, if we end up putting ‘uptime metrics’ on graphite... [12:04:16] although that does make me crawl a bit [12:04:36] YuviPanda: sure, let me know either way! [12:05:41] godog: ty :) [12:16:50] when https://phabricator.wikimedia.org/T91504 will be fixed? [12:17:31] (03PS5) 10KartikMistry: Added initial Debian packaging [debs/contenttranslation/apertium-fr-es] - 10https://gerrit.wikimedia.org/r/195577 (https://phabricator.wikimedia.org/T92252) [12:19:02] (03PS6) 10KartikMistry: Added initial Debian packaging [debs/contenttranslation/apertium-fr-es] - 10https://gerrit.wikimedia.org/r/195577 (https://phabricator.wikimedia.org/T92252) [12:22:04] * Steinsplitter looks to the Panda [12:29:20] (03CR) 10Filippo Giunchedi: "I see what you are saying now, another way to go about this would be to expose "create required keyspaces and columnfamilies" functionalit" [puppet] - 10https://gerrit.wikimedia.org/r/198433 (https://phabricator.wikimedia.org/T93452) (owner: 10GWicke) [12:31:07] 6operations: Provide dh-virtualenv 0.9 package on apt.wikimedia.org Precise and Trusty distributions - https://phabricator.wikimedia.org/T91631#1140408 (10hashar) While doing git-buildpackage I just found: W: dh_python2:479: Please add dh-python package to Build-Depends I have added it. Note that on Jess... [12:32:03] (03PS7) 10KartikMistry: Added initial Debian packaging [debs/contenttranslation/apertium-fr-es] - 10https://gerrit.wikimedia.org/r/195577 (https://phabricator.wikimedia.org/T92252) [12:32:33] (03PS4) 10KartikMistry: Added initial Debian packaging [debs/contenttranslation/apertium-dan] - 10https://gerrit.wikimedia.org/r/195897 (https://phabricator.wikimedia.org/T91493) [12:32:40] (03CR) 10coren: [C: 031] "This is backwards from the "usual" setup where more specific matches are expected to override later, "system" accounts but in this context" [puppet] - 10https://gerrit.wikimedia.org/r/198563 (owner: 10Tim Landscheidt) [12:36:05] (03CR) 10coren: [C: 032] Tools: Make "admin" and "administrator" system aliases [puppet] - 10https://gerrit.wikimedia.org/r/198571 (owner: 10Tim Landscheidt) [12:36:48] (03PS2) 10coren: Tools: Don't let user names mask system aliases [puppet] - 10https://gerrit.wikimedia.org/r/198563 (owner: 10Tim Landscheidt) [12:37:26] (03CR) 10coren: [C: 032] Tools: Don't let user names mask system aliases [puppet] - 10https://gerrit.wikimedia.org/r/198563 (owner: 10Tim Landscheidt) [12:39:01] Did tools-login's RSA key change? [12:39:38] hoo: It did, per the email on labs-l. The new key is on wikitech to double check it. [12:42:14] 6operations, 10ops-eqiad, 5Patch-For-Review: Rack and set up ms-be1016-1018 - https://phabricator.wikimedia.org/T90922#1140417 (10fgiunchedi) machines are in service so this is complete, modulo remaining gerrit patches. the implemented solution was what Faidon proposed, use sda/sdb instead of sdm/sdn for SS... [12:42:31] Ah, good to know, nice :) [12:42:36] * hoo goes to compare [12:44:39] 6operations, 10ops-ulsfo: cp4009 hardware fault - https://phabricator.wikimedia.org/T92476#1140419 (10Cmjohnson) a:5Cmjohnson>3RobH had this sent and assigning to Rob [12:45:08] 6operations: Provide dh-virtualenv 0.9 package on apt.wikimedia.org Precise and Trusty distributions - https://phabricator.wikimedia.org/T91631#1140421 (10hashar) I have sent a pull request to upstream: https://github.com/spotify/dh-virtualenv/pull/83 [12:48:28] Meh... it's using curve25519-sha256@libssh.org as KEX now [12:52:59] 7Puppet, 6Multimedia, 6Release-Engineering, 6Scrum-of-Scrums, and 2 others: Create basic puppet role for Sentry - https://phabricator.wikimedia.org/T84956#1140424 (10Gilles) I've updated P421 and it now works with all missing packages except MySQL-python. I haven't tried the "sentry" package itself yet. [12:56:36] PROBLEM - check if wikidata.org dispatch lag is higher than 2 minutes on wikidata is CRITICAL: HTTP CRITICAL: HTTP/1.1 200 OK - pattern not found - 1481 bytes in 0.292 second response time [13:14:59] mlitn: Regarding SWAT this morning, can you prepare the patches against mediawiki/core that update your extension(s) and update the Deployments page? [13:17:21] anomie: sure thing [13:21:26] PROBLEM - puppet last run on mw1137 is CRITICAL: CRITICAL: Puppet has 1 failures [13:22:37] (03CR) 10Rush: "It seems like this is what the majority of hosts are doing using exim.minimal?" [puppet] - 10https://gerrit.wikimedia.org/r/198114 (owner: 10Rush) [13:26:56] RECOVERY - check if wikidata.org dispatch lag is higher than 2 minutes on wikidata is OK: HTTP OK: HTTP/1.1 200 OK - 1475 bytes in 0.270 second response time [13:29:35] (03PS1) 10Matanya: otrs: Enable strict transport security [puppet] - 10https://gerrit.wikimedia.org/r/198714 [13:37:06] RECOVERY - puppet last run on mw1137 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [13:39:50] (03CR) 10Faidon Liambotis: "They're not mail relays, though, they are just using a smarthost." [puppet] - 10https://gerrit.wikimedia.org/r/198114 (owner: 10Rush) [13:40:24] 6operations, 10OTRS, 6Security, 7HTTPS, 5Patch-For-Review: SSL-config of the OTRS is outdated - https://phabricator.wikimedia.org/T91504#1140477 (10Matanya) [13:41:58] (03CR) 10Steinsplitter: [C: 031] otrs: Enable strict transport security [puppet] - 10https://gerrit.wikimedia.org/r/198714 (owner: 10Matanya) [13:42:17] matanya :-) [13:42:28] at your service :) [13:57:28] (03PS4) 10Chad: Hiera-ize the Elasticsearch config [puppet] - 10https://gerrit.wikimedia.org/r/197533 [13:57:54] <_joe_> ^d: should I review it ^^ ? [13:58:20] <^d> If you've got time, it'd be nice to land this week and get off my todo list [13:58:55] <_joe_> ^d: I'll take a look today [13:59:24] <^d> Thanks! [14:04:35] 6operations, 10OTRS, 6Security, 7HTTPS, 5Patch-For-Review: SSL-config of the OTRS is outdated - https://phabricator.wikimedia.org/T91504#1140486 (10DaBPunkt) >>! In T91504#1140476, @Matanya wrote: > SSL config supports PFS Sure it does, but the webserver for our OTRS doesn’t use it. HSTS is a nice idea,... [14:12:08] (03CR) 10Mobrovac: "> I see what you are saying now, another way to go about this would be to expose "create required keyspaces and columnfamilies" functional" [puppet] - 10https://gerrit.wikimedia.org/r/198433 (https://phabricator.wikimedia.org/T93452) (owner: 10GWicke) [14:28:08] 6operations, 10Continuous-Integration, 3Continuous-Integration-Isolation, 7Upstream: Create a Debian package for NodePool - https://phabricator.wikimedia.org/T89142#1140534 (10hashar) [14:28:23] 6operations, 10Continuous-Integration, 3Continuous-Integration-Isolation, 7Upstream: Create a Debian package for NodePool - https://phabricator.wikimedia.org/T89142#1028174 (10hashar) I have filled a Debian intent to package: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=781027 [14:35:06] (03PS2) 10Giuseppe Lavagetto: ganglia: DRY, use hiera [puppet] - 10https://gerrit.wikimedia.org/r/198566 [14:35:08] (03PS1) 10Giuseppe Lavagetto: ganglia: remove unused configs from ganglia::collector::config [puppet] - 10https://gerrit.wikimedia.org/r/198720 [14:35:10] (03PS1) 10Giuseppe Lavagetto: ganglia: autogenerate datasources from the list of clusters [puppet] - 10https://gerrit.wikimedia.org/r/198721 [14:36:52] (03CR) 10GWicke: [C: 04-1] "Lets hold off on enabling more wikis until we have reduced the load generated by template updates." [puppet] - 10https://gerrit.wikimedia.org/r/198433 (https://phabricator.wikimedia.org/T93452) (owner: 10GWicke) [14:40:42] hashar: you realize ITPs are only needed for uploading to Debian proper, not Wikimedia, right? [14:40:47] !log restarted cassandra nodes to stop repair [14:40:52] Logged the message, Master [14:41:17] hashar: you have three packages in Debian now; two won't make it for jessie, because statsd has an unresolved RC bug for > 300 days [14:41:42] paravoid: hello :) [14:41:42] (python-statsd, that is) [14:42:38] I feel the ITP is a nice central point to communicate about packaging intent. Upstream seems interested in having Debian packages for their softs (zuul/nodepool) [14:42:52] there's an "RFP" status as well [14:43:00] so since I am working on packaging them for us, I guess we can as well upload them back to Debianproject [14:43:02] "request for", rather than "intent to" [14:43:11] "intent to" means that you'll package and maintain it [14:43:14] yeah [14:43:16] that is the intent [14:43:25] well you're not maintaining your existing packages properly [14:43:32] I do [14:43:35] but have troubles uploading them [14:43:47] https://qa.debian.org/excuses.php?package=python-gear [14:43:54] 311 days old (needed 10 days) [14:43:58] for python statsd, I got a RC / critical bug mentioning I messed up with the description and upstream source [14:44:17] I did the change in svn , poked a few folks to get the new version uploaded to debian and eventually forgot about it [14:44:26] 6operations: disable contacts.wikimedia.org? - https://phabricator.wikimedia.org/T84158#1140569 (10Dzahn) a:3Dzahn [14:44:50] fast forward 300 days later, Filippo needed version 3.0 and pushed a version bump that included my fix [14:44:53] ok [14:44:56] meanwhile, the opackage got removed from testing :( [14:45:02] you need to poke more :) [14:45:08] me or filippo [14:45:08] yeah [14:45:11] or urandom ;) [14:45:12] newbie mistake ! [14:45:36] at least [14:45:51] ok [14:45:56] you should talk with zigo about nodepool [14:46:01] I managed to update the zuul packages all by myself for our Precise/Trusty distributions [14:46:07] he's the debian openstack person [14:46:10] and french :) [14:46:14] but in China :) [14:46:19] I think he moved back [14:46:27] 6operations, 6MediaWiki-Core-Team: Figure out a replication strategy for Swift - https://phabricator.wikimedia.org/T91869#1140576 (10fgiunchedi) no affinity features at the moment, we run swiftrepl in an ad-hoc fashion, I looked into container replication and setup an initial trial but got stuck with some cont... [14:46:38] I have met in Paris a few months ago, he mostly working on the core of openstack. Apparently not willing so much to work on side tools [14:46:43] but I should poke him again. You are right [14:46:52] ah ok, I didn't know [14:47:40] I still think that nodepool is a huge overkill and waste of effort for wikimedia btw [14:47:52] but it's not my effort :) [14:47:53] and since I have dropped water on my Macbook, I am now working on a Jessie machine. That makes it ten times easier to set a local build env [14:49:18] hashar: welcome to the real machine :) [14:50:23] yeah hmm [14:50:27] s/real/hardcore/ [14:51:14] manybubbles, ^d, thcipriani (no marktraceur again?): Who wants to SWAT this morning? [14:51:39] <^d> I can [14:51:41] anomie: not it this morning if that is ok - actually consentrating reasonably effectively this morning [14:51:44] k. good [14:51:48] ^d: ok! [14:52:08] mlitn, James_F: Ping for SWAT (^d will be doing it). James_F, I'm guessing you're working on patches for your first two bullets? [14:52:24] anomie: Just finished them, yes. [14:53:12] I'm here! [14:53:29] But thanks ^d [14:53:48] marktraceur: You're not listed in the Deployer column again for today [14:54:35] check [14:57:22] Oh, huh. [14:57:26] nodepool_0.0.1-90-ga34aae2.dsc !! :) [14:58:35] <^d> Ok, I'm going to start the jenkins dance for everyone's patches [14:59:51] Isn't it fun? [14:59:55] (03CR) 10Giuseppe Lavagetto: [C: 04-1] "Very well done, a small comment on placement of config data, but looks good. I'll review more thoroughly after you amended this detail." (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/197533 (owner: 10Chad) [15:00:04] manybubbles, anomie, ^d, thcipriani, superm401, mlitn: Dear anthropoid, the time has come. Please deploy Morning SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20150323T1500). [15:09:20] (03PS2) 10Tim Landscheidt: Tools: Make "admin" and "administrator" system aliases [puppet] - 10https://gerrit.wikimedia.org/r/198571 [15:11:50] (03PS1) 10coren: Labs: upgrade codfw labstores to Jessie [puppet] - 10https://gerrit.wikimedia.org/r/198728 [15:16:59] !log demon Synchronized php-1.25wmf22/extensions/Echo: (no message) (duration: 00m 07s) [15:17:04] Logged the message, Master [15:17:11] !log demon Synchronized php-1.25wmf21/extensions/Echo: (no message) (duration: 00m 06s) [15:17:14] Logged the message, Master [15:17:26] !log demon Synchronized php-1.25wmf21/extensions/Flow: (no message) (duration: 00m 08s) [15:17:26] <^d> mlitn: ^^^ you're all live [15:17:29] Logged the message, Master [15:17:31] <^d> and ^ [15:18:25] alright thanks! [15:18:34] works fine [15:18:38] <^d> James_F: Yours is going in a scap [15:18:42] <^d> because i18n. [15:19:34] * James_F nods. [15:19:39] Fine by me. [15:20:16] !log demon Started scap: VE + wikieditor + new msg for core [15:20:19] Logged the message, Master [15:21:06] PROBLEM - HHVM rendering on mw1193 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:21:36] PROBLEM - Apache HTTP on mw1193 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:24:05] PROBLEM - HHVM queue size on mw1193 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [80.0] [15:24:05] PROBLEM - HHVM busy threads on mw1193 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [115.2] [15:25:53] 6operations, 7Swift: incompatible rsync transfers between rsync 3.0.9 and 3.1 (precise vs trusty) - https://phabricator.wikimedia.org/T93587#1140625 (10fgiunchedi) 3NEW a:3fgiunchedi [15:25:56] PROBLEM - Unmerged changes on repository puppet on palladium is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet). [15:25:56] PROBLEM - Unmerged changes on repository puppet on strontium is CRITICAL: There is one unmerged change in puppet (dir /var/lib/git/operations/puppet). [15:29:27] PROBLEM - HTTPS on cp1008 is CRITICAL: Return code of 255 is out of bounds [15:29:57] ugh cp1008 is supposed to be in perma-downtime :P [15:33:55] RECOVERY - HTTPS on cp1008 is OK: SSLXNN OK - 36 OK [15:35:39] mw1193 seems borked? [15:36:55] RECOVERY - HHVM rendering on mw1193 is OK: HTTP OK: HTTP/1.1 200 OK - 71775 bytes in 0.313 second response time [15:37:13] !log Eventlogging deployment & restart "28a0bf667a3869e95af0997c90af28dd329f6485" [15:37:16] Logged the message, Master [15:37:16] RECOVERY - Apache HTTP on mw1193 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 440 bytes in 0.137 second response time [15:37:26] 6operations, 10ops-eqiad: install 4 * 3TB disks in francium - https://phabricator.wikimedia.org/T93114#1140676 (10Cmjohnson) 5Open>3Resolved Received the disks and installed them. Dell Serial ATA AHCI BIOS Version 1.0.2 Copyright (c) 1988-2014 Dell Inc. Port A: ST3000DM001-1ER166 Port B: ST3000DM001-1ER... [15:37:27] 6operations: deploy francium for html/zim dumps - https://phabricator.wikimedia.org/T93113#1140678 (10Cmjohnson) [15:38:33] !log restarted hhvm on mw1193 -- done this for this particular host a few times now? [15:38:36] Logged the message, Master [15:42:51] !log demon Finished scap: VE + wikieditor + new msg for core (duration: 22m 34s) [15:42:56] Logged the message, Master [15:43:40] <^d> James_F: ^ [15:44:14] * James_F tests. [15:44:20] ^d: Was that just wmf22? [15:44:25] <^d> Both [15:44:33] Kk. [15:45:46] RECOVERY - HHVM queue size on mw1193 is OK: OK: Less than 30.00% above the threshold [10.0] [15:45:47] RECOVERY - HHVM busy threads on mw1193 is OK: OK: Less than 30.00% above the threshold [76.8] [15:45:49] ^d: Looks good to me. [15:45:56] <^d> \o/ [15:47:57] (03CR) 10Dzahn: [C: 04-2] "this is a duplicate. we are already doing this. manifests/role/otrs.pp sets $ssl_settings = ssl_ciphersuite('apache-2.2', 'compat', '365')" [puppet] - 10https://gerrit.wikimedia.org/r/198714 (owner: 10Matanya) [15:48:13] <^d> Ok, swat done for the morning. Thanks guys [15:48:22] (03Abandoned) 10Matanya: otrs: Enable strict transport security [puppet] - 10https://gerrit.wikimedia.org/r/198714 (owner: 10Matanya) [15:48:41] mutante: thanks! duh [15:50:28] thanks ^d :) [15:52:47] 10Ops-Access-Requests, 6operations, 6Phabricator, 6Release-Engineering, 5Patch-For-Review: Chad H. needs access to iridium (Phabricator host) to manage repos - https://phabricator.wikimedia.org/T92564#1140824 (10chasemp) [15:58:54] (03CR) 10Nuria: ">We can avoid having to maintain more inline C code by using the cookie vmod" [puppet] - 10https://gerrit.wikimedia.org/r/196009 (https://phabricator.wikimedia.org/T88813) (owner: 10Nuria) [15:59:56] PROBLEM - puppet last run on cp4010 is CRITICAL: CRITICAL: puppet fail [16:03:17] PROBLEM - Disk space on mw2056 is CRITICAL: Connection refused by host [16:03:18] PROBLEM - Disk space on mw2057 is CRITICAL: Connection refused by host [16:03:18] PROBLEM - Disk space on mw2064 is CRITICAL: Connection refused by host [16:03:18] PROBLEM - Disk space on mw2063 is CRITICAL: Connection refused by host [16:03:18] PROBLEM - Disk space on mw2058 is CRITICAL: Connection refused by host [16:03:18] PROBLEM - Disk space on mw2062 is CRITICAL: Connection refused by host [16:03:18] PROBLEM - Disk space on mw2061 is CRITICAL: Connection refused by host [16:04:17] PROBLEM - RAID on mw2064 is CRITICAL: Connection refused by host [16:04:18] PROBLEM - RAID on mw2056 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:04:18] PROBLEM - RAID on mw2057 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:04:18] PROBLEM - RAID on mw2063 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:04:18] PROBLEM - RAID on mw2061 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:04:18] PROBLEM - RAID on mw2062 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:04:18] PROBLEM - RAID on mw2058 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:04:38] PROBLEM - configured eth on mw2056 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:04:38] PROBLEM - configured eth on mw2058 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:04:38] PROBLEM - configured eth on mw2057 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:04:38] PROBLEM - configured eth on mw2064 is CRITICAL: Connection refused by host [16:04:38] PROBLEM - configured eth on mw2062 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:04:38] PROBLEM - configured eth on mw2061 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:04:38] PROBLEM - configured eth on mw2063 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:04:48] PROBLEM - dhclient process on mw2064 is CRITICAL: Connection refused by host [16:04:48] PROBLEM - dhclient process on mw2057 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:04:48] PROBLEM - dhclient process on mw2056 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:04:48] PROBLEM - dhclient process on mw2061 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:04:48] PROBLEM - dhclient process on mw2063 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:04:49] PROBLEM - dhclient process on mw2058 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:04:49] PROBLEM - dhclient process on mw2062 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:05:07] PROBLEM - nutcracker port on mw2058 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:05:07] PROBLEM - nutcracker port on mw2063 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:05:07] PROBLEM - nutcracker port on mw2056 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:05:07] PROBLEM - nutcracker port on mw2061 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:05:07] PROBLEM - nutcracker port on mw2057 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:05:08] PROBLEM - nutcracker port on mw2064 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:05:08] PROBLEM - nutcracker port on mw2062 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:05:17] PROBLEM - nutcracker process on mw2057 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:05:18] PROBLEM - nutcracker process on mw2062 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:05:18] PROBLEM - nutcracker process on mw2056 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:05:18] PROBLEM - nutcracker process on mw2063 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:05:18] PROBLEM - nutcracker process on mw2064 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:05:18] PROBLEM - nutcracker process on mw2058 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:05:18] PROBLEM - nutcracker process on mw2061 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:05:28] PROBLEM - puppet last run on mw2064 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:05:28] PROBLEM - puppet last run on mw2062 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:05:28] PROBLEM - puppet last run on mw2056 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:05:28] PROBLEM - puppet last run on mw2061 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:05:28] PROBLEM - puppet last run on mw2058 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:05:29] PROBLEM - puppet last run on mw2063 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:05:29] PROBLEM - puppet last run on mw2057 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:05:47] uhhh bblack or _joe_ is this a known thing?^ [16:05:48] PROBLEM - salt-minion processes on mw2057 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:05:48] PROBLEM - salt-minion processes on mw2056 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:05:48] PROBLEM - salt-minion processes on mw2061 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:05:48] PROBLEM - salt-minion processes on mw2058 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:05:48] PROBLEM - salt-minion processes on mw2062 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:05:48] PROBLEM - salt-minion processes on mw2063 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:05:48] PROBLEM - salt-minion processes on mw2064 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:05:58] PROBLEM - DPKG on mw2056 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:05:58] PROBLEM - DPKG on mw2058 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:05:58] PROBLEM - DPKG on mw2057 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:05:58] PROBLEM - DPKG on mw2062 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:05:58] PROBLEM - DPKG on mw2063 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:05:58] PROBLEM - DPKG on mw2061 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:05:58] PROBLEM - DPKG on mw2064 is CRITICAL: CHECK_NRPE: Error - Could not complete SSL handshake. [16:07:31] (03PS1) 10Rush: phab set daemon management scripts as phd user [puppet] - 10https://gerrit.wikimedia.org/r/198743 [16:09:07] (03PS2) 10Dzahn: scholarships: use HTTPS by default [puppet] - 10https://gerrit.wikimedia.org/r/198567 (owner: 10John F. Lewis) [16:09:35] <_joe_> chasemp: it is, sorry [16:09:58] (03PS1) 10Andrew Bogott: Rename and move quota settings. [puppet] - 10https://gerrit.wikimedia.org/r/198744 [16:09:59] _joe_: ok thanks [16:10:08] (03CR) 10BBlack: Adding a Last-Access cookie to text and mobile requests (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/196009 (https://phabricator.wikimedia.org/T88813) (owner: 10Nuria) [16:11:25] (03CR) 10Rush: [C: 032] phab set daemon management scripts as phd user [puppet] - 10https://gerrit.wikimedia.org/r/198743 (owner: 10Rush) [16:11:57] RECOVERY - configured eth on mw2056 is OK: NRPE: Unable to read output [16:11:57] RECOVERY - configured eth on mw2058 is OK: NRPE: Unable to read output [16:11:57] RECOVERY - configured eth on mw2061 is OK: NRPE: Unable to read output [16:11:57] RECOVERY - configured eth on mw2063 is OK: NRPE: Unable to read output [16:11:57] RECOVERY - configured eth on mw2062 is OK: NRPE: Unable to read output [16:11:58] RECOVERY - Disk space on mw2061 is OK: DISK OK [16:11:58] RECOVERY - Disk space on mw2063 is OK: DISK OK [16:11:59] RECOVERY - Disk space on mw2057 is OK: DISK OK [16:11:59] RECOVERY - Disk space on mw2056 is OK: DISK OK [16:12:00] RECOVERY - Disk space on mw2062 is OK: DISK OK [16:12:00] RECOVERY - Disk space on mw2058 is OK: DISK OK [16:12:08] RECOVERY - dhclient process on mw2057 is OK: PROCS OK: 0 processes with command name dhclient [16:12:08] RECOVERY - dhclient process on mw2062 is OK: PROCS OK: 0 processes with command name dhclient [16:12:08] RECOVERY - dhclient process on mw2061 is OK: PROCS OK: 0 processes with command name dhclient [16:12:08] RECOVERY - dhclient process on mw2063 is OK: PROCS OK: 0 processes with command name dhclient [16:12:08] RECOVERY - dhclient process on mw2056 is OK: PROCS OK: 0 processes with command name dhclient [16:12:08] RECOVERY - dhclient process on mw2058 is OK: PROCS OK: 0 processes with command name dhclient [16:12:19] RECOVERY - nutcracker port on mw2062 is OK: TCP OK - 0.000 second response time on port 11212 [16:12:19] RECOVERY - nutcracker port on mw2063 is OK: TCP OK - 0.000 second response time on port 11212 [16:12:19] RECOVERY - nutcracker port on mw2061 is OK: TCP OK - 0.000 second response time on port 11212 [16:12:19] RECOVERY - nutcracker port on mw2056 is OK: TCP OK - 0.000 second response time on port 11212 [16:12:19] RECOVERY - nutcracker port on mw2057 is OK: TCP OK - 0.000 second response time on port 11212 [16:12:19] RECOVERY - nutcracker port on mw2058 is OK: TCP OK - 0.000 second response time on port 11212 [16:12:37] RECOVERY - nutcracker process on mw2057 is OK: PROCS OK: 1 process with UID = 108 (nutcracker), command name nutcracker [16:12:37] RECOVERY - nutcracker process on mw2062 is OK: PROCS OK: 1 process with UID = 108 (nutcracker), command name nutcracker [16:12:37] RECOVERY - nutcracker process on mw2056 is OK: PROCS OK: 1 process with UID = 108 (nutcracker), command name nutcracker [16:12:37] RECOVERY - nutcracker process on mw2058 is OK: PROCS OK: 1 process with UID = 108 (nutcracker), command name nutcracker [16:12:37] RECOVERY - nutcracker process on mw2063 is OK: PROCS OK: 1 process with UID = 108 (nutcracker), command name nutcracker [16:12:38] RECOVERY - nutcracker process on mw2061 is OK: PROCS OK: 1 process with UID = 108 (nutcracker), command name nutcracker [16:12:51] (03CR) 10Andrew Bogott: [C: 032] Rename and move quota settings. [puppet] - 10https://gerrit.wikimedia.org/r/198744 (owner: 10Andrew Bogott) [16:12:58] RECOVERY - RAID on mw2061 is OK: OK: no RAID installed [16:12:58] RECOVERY - RAID on mw2057 is OK: OK: no RAID installed [16:12:58] RECOVERY - RAID on mw2063 is OK: OK: no RAID installed [16:12:58] RECOVERY - RAID on mw2062 is OK: OK: no RAID installed [16:12:58] RECOVERY - RAID on mw2056 is OK: OK: no RAID installed [16:12:59] RECOVERY - RAID on mw2058 is OK: OK: no RAID installed [16:12:59] RECOVERY - salt-minion processes on mw2057 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion [16:13:00] RECOVERY - salt-minion processes on mw2056 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion [16:13:00] RECOVERY - salt-minion processes on mw2061 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion [16:13:01] RECOVERY - salt-minion processes on mw2058 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion [16:13:07] RECOVERY - salt-minion processes on mw2062 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion [16:13:08] RECOVERY - salt-minion processes on mw2063 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion [16:13:18] RECOVERY - DPKG on mw2062 is OK: All packages OK [16:13:18] RECOVERY - DPKG on mw2063 is OK: All packages OK [16:13:18] RECOVERY - DPKG on mw2057 is OK: All packages OK [16:13:18] RECOVERY - DPKG on mw2056 is OK: All packages OK [16:13:18] RECOVERY - DPKG on mw2061 is OK: All packages OK [16:13:19] RECOVERY - DPKG on mw2058 is OK: All packages OK [16:13:19] RECOVERY - configured eth on mw2057 is OK: NRPE: Unable to read output [16:13:47] RECOVERY - nutcracker port on mw2064 is OK: TCP OK - 0.000 second response time on port 11212 [16:13:58] RECOVERY - Unmerged changes on repository puppet on palladium is OK: No changes to merge. [16:13:58] RECOVERY - Unmerged changes on repository puppet on strontium is OK: No changes to merge. [16:13:59] RECOVERY - nutcracker process on mw2064 is OK: PROCS OK: 1 process with UID = 108 (nutcracker), command name nutcracker [16:14:28] RECOVERY - RAID on mw2064 is OK: OK: no RAID installed [16:14:28] RECOVERY - salt-minion processes on mw2064 is OK: PROCS OK: 1 process with regex args ^/usr/bin/python /usr/bin/salt-minion [16:14:38] RECOVERY - DPKG on mw2064 is OK: All packages OK [16:14:48] RECOVERY - configured eth on mw2064 is OK: NRPE: Unable to read output [16:14:57] RECOVERY - Disk space on mw2064 is OK: DISK OK [16:14:58] RECOVERY - dhclient process on mw2064 is OK: PROCS OK: 0 processes with command name dhclient [16:15:45] (03CR) 10Dzahn: [C: 032] scholarships: use HTTPS by default [puppet] - 10https://gerrit.wikimedia.org/r/198567 (owner: 10John F. Lewis) [16:15:51] 6operations, 6Phabricator: Phabricator's phd can't sudo to user phd - https://phabricator.wikimedia.org/T93477#1141383 (10chasemp) 5Open>3Resolved a:3chasemp should be fixed https://gerrit.wikimedia.org/r/#/c/198743/ [16:18:08] RECOVERY - puppet last run on cp4010 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:18:54] (03CR) 10Dzahn: "curl -vvv http://scholarships.wikimedia.org 2>/dev/null| grep moved" [puppet] - 10https://gerrit.wikimedia.org/r/198567 (owner: 10John F. Lewis) [16:20:05] (03CR) 10Dzahn: [C: 031] RT - Enable HSTS max-age=7 days [puppet] - 10https://gerrit.wikimedia.org/r/198455 (https://phabricator.wikimedia.org/T40516) (owner: 10Chmarkine) [16:21:31] (03CR) 10Dzahn: [C: 031] ishmael - Enable HSTS max-age=7 days [puppet] - 10https://gerrit.wikimedia.org/r/198457 (https://phabricator.wikimedia.org/T40516) (owner: 10Chmarkine) [16:23:36] (03CR) 10Dzahn: "_joe_ asked on https://gerrit.wikimedia.org/r/#/c/198564/ "Did we change the host on which check_graphite operates to use a direct connect" [puppet] - 10https://gerrit.wikimedia.org/r/98003 (owner: 10Ori.livneh) [16:24:34] (03CR) 10Dzahn: "+ori" [puppet] - 10https://gerrit.wikimedia.org/r/198564 (owner: 10John F. Lewis) [16:25:38] (03CR) 10Dzahn: [C: 031] gdash - Enable HSTS max-age=7 days [puppet] - 10https://gerrit.wikimedia.org/r/198469 (https://phabricator.wikimedia.org/T40516) (owner: 10Chmarkine) [16:26:27] (03CR) 10Dzahn: [C: 031] integration - Enable HSTS max-age=7 days [puppet] - 10https://gerrit.wikimedia.org/r/198458 (https://phabricator.wikimedia.org/T40516) (owner: 10Chmarkine) [16:27:29] 6operations: Delete gadolinium:/a/log/fundraising/ - https://phabricator.wikimedia.org/T92336#1141452 (10Jgreen) Ah, I forgot that we'd moved collection to erbium. It's ok to unmount the netapp and delete the dirs here. [16:28:05] Krenair: I haven't had a chance to look at (and make sense of) https://phabricator.wikimedia.org/P405 yet, and I'm in a 3 hour long budget meeting (and mostly busy rest of day): is there anything actionable you want from me from that? [16:28:12] Krenair: if so, what specifically? :) [16:28:57] 6operations: disable contacts.wikimedia.org? - https://phabricator.wikimedia.org/T84158#1141461 (10Dzahn) merging this into T90679 , should have technically used this one to disable it but now it's the same thing, disabling an unpuppetized service [16:30:03] 6operations: disable contacts.wikimedia.org? - https://phabricator.wikimedia.org/T84158#1141463 (10Dzahn) [16:30:04] 6operations, 5Patch-For-Review: contacts.wikimedia.org drupal unpuppetized - https://phabricator.wikimedia.org/T90679#1064781 (10Dzahn) [16:30:19] 6operations, 6Phabricator, 7domains: enable email for tickets in domains project? - https://phabricator.wikimedia.org/T88842#1141467 (10chasemp) 5Open>3Resolved a:3chasemp Closing as inactive and AFAIK it looks like what was wanted is accomplished. @dzahn, please reopen if I'm mistaken. [16:30:28] RECOVERY - puppet last run on mw2062 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [16:31:20] 6operations, 5Patch-For-Review: contacts.wikimedia.org drupal unpuppetized / retire contacts - https://phabricator.wikimedia.org/T90679#1141471 (10Dzahn) [16:31:58] RECOVERY - puppet last run on mw2056 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:31:58] RECOVERY - puppet last run on mw2063 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:31:58] RECOVERY - puppet last run on mw2061 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:31:58] RECOVERY - puppet last run on mw2058 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:31:58] RECOVERY - puppet last run on mw2064 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:31:58] RECOVERY - puppet last run on mw2057 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:34:12] (03CR) 10Glaisher: "Isn't there some actual technical blockers for this?" (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/198691 (https://phabricator.wikimedia.org/T93210) (owner: 10Nemo bis) [16:35:49] 6operations, 6Phabricator, 7domains: enable email for tickets in domains project? - https://phabricator.wikimedia.org/T88842#1141480 (10Dzahn) It should be resolved, it just wasn't technically tested. the difference to other queues where it works and has been tested is that it specifies domain names instead... [16:36:00] 6operations, 6Phabricator, 6Project-Creators: Create policy projects and convert people projects to open - https://phabricator.wikimedia.org/T90491#1141481 (10chasemp) p:5Normal>3High legit high I think [16:42:04] PROBLEM - Host mw2088 is DOWN: PING CRITICAL - Packet loss = 100% [16:44:58] (03CR) 10Dzahn: [C: 04-1] fix puppet error due to missing parent directory (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/198461 (owner: 1020after4) [16:45:54] RECOVERY - Host mw2088 is UP: PING OK - Packet loss = 0%, RTA = 43.18 ms [16:46:31] (03PS2) 10Yuvipanda: tools: Add CORS header to tools-static [puppet] - 10https://gerrit.wikimedia.org/r/198474 (https://phabricator.wikimedia.org/T93466) [16:46:49] (03CR) 10Yuvipanda: [C: 032 V: 032] tools: Add CORS header to tools-static [puppet] - 10https://gerrit.wikimedia.org/r/198474 (https://phabricator.wikimedia.org/T93466) (owner: 10Yuvipanda) [16:48:58] (03PS1) 10Gerardduenas: Create and modify groups in eswikibooks [mediawiki-config] - 10https://gerrit.wikimedia.org/r/198749 (https://phabricator.wikimedia.org/T93371) [16:49:34] PROBLEM - Host mw2088 is DOWN: PING CRITICAL - Packet loss = 100% [16:50:08] (03CR) 10Rush: "Just a note for mukunda :)" [puppet] - 10https://gerrit.wikimedia.org/r/198461 (owner: 1020after4) [16:50:27] (03PS2) 10Nemo bis: Restore unregistered editing on mobile sites (staggered) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/198691 (https://phabricator.wikimedia.org/T93210) [16:50:54] (03CR) 10Nemo bis: "Glaisher, no longer, as far as we know. If you know of any blocker, please report them!" (031 comment) [mediawiki-config] - 10https://gerrit.wikimedia.org/r/198691 (https://phabricator.wikimedia.org/T93210) (owner: 10Nemo bis) [16:51:10] (03CR) 10Dzahn: "i'll amend with some "Require" syntax" [puppet] - 10https://gerrit.wikimedia.org/r/198461 (owner: 1020after4) [16:51:57] godog: +1 or CR when you have time for https://gerrit.wikimedia.org/r/#/c/195081/ [16:52:49] (03CR) 10Glaisher: [C: 031] "Ah, okay. I haven't really been following this issue. Fine with enabling this then." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/198691 (https://phabricator.wikimedia.org/T93210) (owner: 10Nemo bis) [16:52:55] 6operations, 10hardware-requests: Upgrade eqiad LVS to 10G - https://phabricator.wikimedia.org/T89120#1141537 (10RobH) Discussion on RT ticket & price of upgrade to old, out of warranty systems is not worth it. These are critical, and instead we'll look at quotes to replace them outright. [16:53:57] (03PS2) 10Dzahn: fix puppet error due to missing parent directory [puppet] - 10https://gerrit.wikimedia.org/r/198461 (owner: 1020after4) [16:54:44] RECOVERY - Host mw2088 is UP: PING OK - Packet loss = 0%, RTA = 42.95 ms [16:55:42] 6operations, 10ops-codfw: mw2088 has a faulty RAM - https://phabricator.wikimedia.org/T93370#1141544 (10Papaul) Removed the memory and ran the install, the install complete with no problem. The system is running now at 56GB. I will call Dell for them to send me a replacement memory stick. [16:57:53] PROBLEM - dhclient process on mw2088 is CRITICAL: Connection refused by host [16:57:53] PROBLEM - Disk space on mw2088 is CRITICAL: Connection refused by host [16:58:05] PROBLEM - nutcracker port on mw2088 is CRITICAL: Connection refused by host [16:58:23] PROBLEM - nutcracker process on mw2088 is CRITICAL: Connection refused by host [16:58:34] PROBLEM - puppet last run on mw2088 is CRITICAL: Connection refused by host [16:58:44] PROBLEM - RAID on mw2088 is CRITICAL: Connection refused by host [16:58:45] PROBLEM - salt-minion processes on mw2088 is CRITICAL: Connection refused by host [16:59:03] PROBLEM - DPKG on mw2088 is CRITICAL: Connection refused by host [16:59:04] PROBLEM - configured eth on mw2088 is CRITICAL: Connection refused by host [17:00:44] (03PS1) 10Dzahn: phab: small lint fixes phd.pp [puppet] - 10https://gerrit.wikimedia.org/r/198750 [17:00:49] (03CR) 10Rush: [C: 04-1] fix puppet error due to missing parent directory (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/198461 (owner: 1020after4) [17:02:22] 6operations, 10hardware-requests: Upgrade eqiad LVS to 10G - https://phabricator.wikimedia.org/T89120#1141586 (10RobH) The quote request for 6 new LVS systems for EQIAD: https://rt.wikimedia.org/Ticket/Display.html?id=9278 [17:02:42] 6operations, 6Phabricator: Phabricator's phd can't sudo to user phd - https://phabricator.wikimedia.org/T93477#1141589 (10Negative24) >>! In T93477#1141383, @chasemp wrote: > should be fixed https://gerrit.wikimedia.org/r/#/c/198743/ Thanks. (Is @gerritbot working?) [17:03:30] (03PS1) 10Glaisher: Add import sources and set wgImportTargetNamespace at ptwikinews [mediawiki-config] - 10https://gerrit.wikimedia.org/r/198751 (https://phabricator.wikimedia.org/T93218) [17:05:07] (03CR) 10Florianschmidtwelzow: [C: 04-1] "see comment on phabricator task" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/198691 (https://phabricator.wikimedia.org/T93210) (owner: 10Nemo bis) [17:05:09] (03PS2) 10Dzahn: phab: small lint fixes phd.pp [puppet] - 10https://gerrit.wikimedia.org/r/198750 [17:06:18] (03CR) 10Rush: "I think this is fixed already if you rebase?" [puppet] - 10https://gerrit.wikimedia.org/r/198750 (owner: 10Dzahn) [17:06:26] (03CR) 10Chad: "Fixing 2 nits inline." (032 comments) [puppet] - 10https://gerrit.wikimedia.org/r/197533 (owner: 10Chad) [17:06:42] (03PS5) 10Chad: Hiera-ize the Elasticsearch config [puppet] - 10https://gerrit.wikimedia.org/r/197533 [17:06:44] (03PS3) 10Dzahn: phab: small lint fixes phd.pp [puppet] - 10https://gerrit.wikimedia.org/r/198750 [17:08:35] <^d> _joe_: Amended [17:10:07] (03PS4) 10Dzahn: phab: small lint fixes phd.pp,init.pp [puppet] - 10https://gerrit.wikimedia.org/r/198750 [17:10:39] (03PS5) 10Dzahn: phab: small lint fixes phd.pp,init.pp [puppet] - 10https://gerrit.wikimedia.org/r/198750 [17:12:14] YuviPanda: ack [17:12:37] (03CR) 10BBlack: "re: the cookie/lua vmods, I don't know that they're worth it in this case at first glance. The saved C code is pretty short and trivial i" [puppet] - 10https://gerrit.wikimedia.org/r/196009 (https://phabricator.wikimedia.org/T88813) (owner: 10Nuria) [17:13:02] (03PS1) 10Glaisher: Let dawiki bureaucrats add/remove accountcreator group [mediawiki-config] - 10https://gerrit.wikimedia.org/r/198753 (https://phabricator.wikimedia.org/T93260) [17:16:01] (03CR) 10Dzahn: [C: 04-1] nova: lint compute.pp (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/195535 (owner: 10Matanya) [17:18:50] 6operations, 6Phabricator: Phabricator's phd can't sudo to user phd - https://phabricator.wikimedia.org/T93477#1141664 (10jeremyb) >>! In T93477#1141589, @Negative24 wrote: > Thanks. (Is @gerritbot working?) See https://www.mediawiki.org/wiki/Gerrit/Commit_message_guidelines#Cross-references It's not enough... [17:20:47] 6operations, 6Phabricator: Phabricator's phd can't sudo to user phd - https://phabricator.wikimedia.org/T93477#1141685 (10Negative24) Ah. Okay then. [17:21:54] 6operations, 10hardware-requests: Upgrade eqiad LVS to 10G - https://phabricator.wikimedia.org/T89120#1141695 (10RobH) The new LVS systems we use replace the 1G copper NICs with 10G fiber. @faidon or @mark: Do I need to support both 1G copper and 10G fiber on the new eqiad lvs systems? If so, we'll have to... [17:21:56] (03CR) 10Filippo Giunchedi: Ensure that apt preferences are named *.pref (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/195081 (https://phabricator.wikimedia.org/T60681) (owner: 10Tim Landscheidt) [17:23:53] PROBLEM - NTP on mw2088 is CRITICAL: NTP CRITICAL: No response from NTP server [17:25:18] (03CR) 10Gerardduenas: [C: 031] Add import sources and set wgImportTargetNamespace at ptwikinews [mediawiki-config] - 10https://gerrit.wikimedia.org/r/198751 (https://phabricator.wikimedia.org/T93218) (owner: 10Glaisher) [17:27:19] (03Abandoned) 10Gerardduenas: Enable UploadWizard on idwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/190744 (https://phabricator.wikimedia.org/T88918) (owner: 10Gerardduenas) [17:29:01] <_joe_> ^d: ok thanks [17:30:55] (03CR) 10Dzahn: Wikidata builder (034 comments) [puppet] - 10https://gerrit.wikimedia.org/r/195567 (https://phabricator.wikimedia.org/T90567) (owner: 10JanZerebecki) [17:36:39] !log ms-be101[678] weight to 1000 [17:36:46] Logged the message, Master [17:37:00] (03PS1) 10Filippo Giunchedi: eqiad-prod: ms-be101[567] object weight to 1000 [software/swift-ring] - 10https://gerrit.wikimedia.org/r/198756 (https://phabricator.wikimedia.org/T1268) [17:37:14] (03CR) 10Filippo Giunchedi: [C: 032 V: 032] eqiad-prod: ms-be101[567] object weight to 1000 [software/swift-ring] - 10https://gerrit.wikimedia.org/r/198756 (https://phabricator.wikimedia.org/T1268) (owner: 10Filippo Giunchedi) [17:39:14] (03PS3) 10Dzahn: wikistats: lint [puppet] - 10https://gerrit.wikimedia.org/r/195866 (https://phabricator.wikimedia.org/T91908) (owner: 10Matanya) [17:40:24] someone reported hitting the poolcounter limit ("Too many users are trying to view this page. Please wait a while before you try to access this page again.") viewing [[Lee Kuan Yew]] on zhwiki [17:45:40] 6operations, 10OTRS, 6Security, 7HTTPS: SSL-config of the OTRS is outdated - https://phabricator.wikimedia.org/T91504#1141910 (10MC8) [17:46:35] 6operations, 10ops-codfw: audit and update all codfw server's racktables info - https://phabricator.wikimedia.org/T84891#1141916 (10Papaul) @Chris did you have the chance to complete this task when you was here with Tony or is there anything else i need to do on it? Thanks. [17:53:53] twentyafterfour: Would I need shell access to iridium to see production phabricator's configs? [17:54:13] hey - mw1135 is having problems with shelling out from php, can someone take a look, please? [17:55:45] mmm, YuviPanda - you're on duty? ^^^ :) [17:55:54] oh, I was on duty last week... [17:55:59] not sure who is this week? [17:56:06] * YuviPanda finds out [17:56:23] 7Puppet, 6operations, 7Swift: puppet failure "invalid byte sequence in utf-8" while copying swift ring builder files - https://phabricator.wikimedia.org/T93614#1142002 (10fgiunchedi) 3NEW [17:56:34] MaxSem: https://wikitech.wikimedia.org/wiki/Ops_Clinic_Duty says jgage [17:56:48] 7Puppet, 6operations, 7Swift: puppet failure "invalid byte sequence in utf-8" while copying swift ring builder files - https://phabricator.wikimedia.org/T93614#1142009 (10fgiunchedi) p:5Triage>3Normal setting to normal since for this particular case it doesn't impact swift functionality [17:59:15] 6operations, 6WMF-Legal, 7domains: enwikipedia.org not hosted by WMF - https://phabricator.wikimedia.org/T93523#1142032 (10Slaporte) legal-tm-vio@wikimedia.org is the correct way to report TM violations, and I'll investigate both of these. [18:00:46] uh, and enwikipedia.org is asking you to download "Flash".... [18:01:56] * greg-g choses not to open that url [18:02:03] just two redirects... [18:02:10] (03PS3) 10BBlack: add generic nrpe script check-fresh-files-in-dir.py [puppet] - 10https://gerrit.wikimedia.org/r/198387 [18:02:12] (03PS4) 10BBlack: test OCSP Stapling on cp1008 [puppet] - 10https://gerrit.wikimedia.org/r/198388 [18:02:14] (03PS18) 10BBlack: protoproxy/sslcert/cache: nginx ssl_stapling_file support [puppet] - 10https://gerrit.wikimedia.org/r/198110 [18:04:25] andre__, that's cause you have lunux so completely useless for them;) [18:05:52] heyyyy opsen, so can someone investigate mw1135? :) [18:06:26] MaxSem: ‘investigate’? [18:06:50] andrewbogott, hey - mw1135 is having problems with shelling out from php, can someone take a look, please? [18:08:21] namely, Warning: Not a valid stream resource in /srv/mediawiki/php-1.25wmf21/includes/GlobalFunctions.php on line 3196 and similar erors indicating that it can't spawn a child process [18:09:12] <_joe_> MaxSem: look for "ops on duty" [18:09:16] <_joe_> jgage: ^^ [18:09:44] mutante: is the .puppet-lint.rc outdated? because it disables the checks you noticed failing in my patch. [18:09:51] <_joe_> jgage: I don't have time now, but if you depool that server and create a phab ticket and assign it to me [18:10:05] <_joe_> I can take a look tomorrow [18:10:39] mutante: even giving it an empty config file doesn't help, i need to execute puppet-lint in a different directory :-O [18:11:09] (03PS6) 10Nuria: Adding a Last-Access cookie to text and mobile requests [puppet] - 10https://gerrit.wikimedia.org/r/196009 (https://phabricator.wikimedia.org/T88813) [18:11:38] (03PS7) 10Nuria: Adding a Last-Access cookie to text and mobile requests [puppet] - 10https://gerrit.wikimedia.org/r/196009 (https://phabricator.wikimedia.org/T88813) [18:12:56] (03CR) 10Nuria: Adding a Last-Access cookie to text and mobile requests (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/196009 (https://phabricator.wikimedia.org/T88813) (owner: 10Nuria) [18:14:13] 6operations, 10ops-eqiad: Replace PEM that was taken from spare ex4500 wmf5738 - https://phabricator.wikimedia.org/T93621#1142122 (10Cmjohnson) 3NEW a:3Cmjohnson [18:15:05] 6operations, 10ops-eqiad: Replace PEM that was taken from spare ex4500 wmf5738 - https://phabricator.wikimedia.org/T93621#1142133 (10Cmjohnson) Dear Juniper Networks Customer, Thank you for contacting the Juniper Networks Global Support. We have opened Service Request number 2015-0323-0581 to track this probl... [18:17:00] 6operations, 10ops-codfw: Set up pdu's - https://phabricator.wikimedia.org/T84416#1142135 (10Cmjohnson) I am not able to get to their web portal while on the eqiad data center network. [18:20:25] jzerebecki: i ran puppet lint manually on my laptop. note there are 2 puppetlint checks run by jenkins, lenient and strict. i expected that kind of error to show up in the "strict" version which isn't voting so far [18:20:52] mutante: not it's not, as it is fully disabled in the .rc [18:21:22] jzerebecki: then i think i disagree with it being disabled [18:21:31] at least strict should have it [18:21:34] good thx [18:23:48] (03PS1) 10Negative24: Configure labs Phabricators with default local repo store [puppet] - 10https://gerrit.wikimedia.org/r/198769 (https://phabricator.wikimedia.org/T93615) [18:25:42] (03CR) 10Dzahn: [C: 032] wikistats: lint [puppet] - 10https://gerrit.wikimedia.org/r/195866 (https://phabricator.wikimedia.org/T91908) (owner: 10Matanya) [18:26:10] !log depooled mw1135 (eqiad api) [18:26:14] Logged the message, Master [18:32:12] 6operations, 10MediaWiki-API: mw1135 has errors, depooled - https://phabricator.wikimedia.org/T93626#1142247 (10BBlack) 3NEW a:3Joe [18:32:24] 6operations: Force https for archiva.wikimedia.org - https://phabricator.wikimedia.org/T88139#1142255 (10Ottomata) [18:33:15] question: on friday, a change was merged to disabled a cron job on the restbase nodes (https://gerrit.wikimedia.org/r/#/c/198297), but that job continued to run [18:33:33] anyone know why? [18:34:03] (03PS1) 10Filippo Giunchedi: check_graphite: accept additional date specification [puppet] - 10https://gerrit.wikimedia.org/r/198775 [18:34:04] where does puppet add cron jobs? a grep -r on /etc/cron* doesn't turn it up. [18:34:30] urandom: hehe it uses the users' crontab in /var/spool [18:34:41] !? [18:35:07] which user? [18:35:16] /var/spool/cron/crontabs/cassandra:45 4 * * * /usr/bin/nodetool repair -par -inc > /var/log/cassandra/repair.log 2>&1 [18:35:30] oh, right [18:36:09] what am I missing, why is that still there? [18:36:34] puppet is good at adding stuff, but does not necessarily remove it automatically [18:36:59] is it enough to just remove it now? [18:37:17] I think so, yes [18:37:32] (03PS1) 10Hoo man: Deploy Capiunto on beta [mediawiki-config] - 10https://gerrit.wikimedia.org/r/198776 (https://phabricator.wikimedia.org/T93418) [18:38:00] urandom: yep, it is [18:38:11] cool, thanks [18:38:19] 6operations: Force https for archiva.wikimedia.org - https://phabricator.wikimedia.org/T88139#1142279 (10RobH) a:5Ottomata>3RobH [18:45:17] (03PS1) 10Yuvipanda: puppetmaster: Use cleaner syntax for shelling out [puppet] - 10https://gerrit.wikimedia.org/r/198778 [18:45:20] andrewbogott: ^ untested [18:49:44] (03PS1) 10Nuria: Adding template for apache to serve static content [puppet] - 10https://gerrit.wikimedia.org/r/198780 (https://phabricator.wikimedia.org/T89255) [18:49:59] (03PS2) 10Yuvipanda: puppetmaster: Use cleaner syntax for shelling out [puppet] - 10https://gerrit.wikimedia.org/r/198778 [18:50:08] (03Abandoned) 10Nuria: Adding template for apache to serve static content [puppet] - 10https://gerrit.wikimedia.org/r/198780 (https://phabricator.wikimedia.org/T89255) (owner: 10Nuria) [18:51:48] (03PS3) 10Andrew Bogott: puppetmaster: Use cleaner syntax for shelling out [puppet] - 10https://gerrit.wikimedia.org/r/198778 (owner: 10Yuvipanda) [18:51:51] (03PS1) 10Eevans: increased compaction concurrency and throughput [puppet] - 10https://gerrit.wikimedia.org/r/198781 (https://phabricator.wikimedia.org/T93140) [18:53:08] (03PS1) 10Nuria: Adding template for apache to serve static content [puppet] - 10https://gerrit.wikimedia.org/r/198782 (https://phabricator.wikimedia.org/T89255) [18:53:19] * urandom nods [18:53:27] * urandom sighs [18:54:31] (03CR) 10Andrew Bogott: [C: 032] puppetmaster: Use cleaner syntax for shelling out [puppet] - 10https://gerrit.wikimedia.org/r/198778 (owner: 10Yuvipanda) [18:55:52] apergos: ping [18:55:54] (03PS1) 10Chad: Better support checking out MediaWiki & extension masters [mediawiki-config] - 10https://gerrit.wikimedia.org/r/198783 [18:59:24] (03PS2) 10Negative24: Configure Labs Phabricators with default local repo store [puppet] - 10https://gerrit.wikimedia.org/r/198769 (https://phabricator.wikimedia.org/T93615) [19:00:21] (03CR) 10Dzahn: [C: 032] add ferm service for poolcounterd [puppet] - 10https://gerrit.wikimedia.org/r/198442 (https://phabricator.wikimedia.org/T93261) (owner: 10Dzahn) [19:00:22] PROBLEM - check_load on db1025 is CRITICAL: CRITICAL - load average: 34.05, 22.01, 11.43 [19:01:36] apergos: In case you're around, please have a look at https://phabricator.wikimedia.org/T72385#1142368 [19:01:47] (03CR) 10Dzahn: "noop for eqiad, but preparing for codfw" [puppet] - 10https://gerrit.wikimedia.org/r/198442 (https://phabricator.wikimedia.org/T93261) (owner: 10Dzahn) [19:04:18] (03PS1) 10Gerardduenas: Add import sources for cawikinews [mediawiki-config] - 10https://gerrit.wikimedia.org/r/198786 (https://phabricator.wikimedia.org/T93203) [19:05:13] PROBLEM - check_load on db1025 is CRITICAL: CRITICAL - load average: 44.43, 38.40, 21.67 [19:08:07] (03CR) 10Dzahn: [C: 032] have base::firewall on codfw poolcounters [puppet] - 10https://gerrit.wikimedia.org/r/198440 (https://phabricator.wikimedia.org/T93261) (owner: 10Dzahn) [19:08:09] (03PS1) 10Yuvipanda: puppetmaster: Cleanup autosigner script some more [puppet] - 10https://gerrit.wikimedia.org/r/198790 [19:08:20] (03CR) 10jenkins-bot: [V: 04-1] puppetmaster: Cleanup autosigner script some more [puppet] - 10https://gerrit.wikimedia.org/r/198790 (owner: 10Yuvipanda) [19:10:22] PROBLEM - check_load on db1025 is CRITICAL: CRITICAL - load average: 46.56, 41.36, 27.36 [19:12:20] (03CR) 10Dzahn: "works. before: 7531/tcp open after: 7531/tcp open from iron but now you need to nmap -P0 because we blocking ping probes and all the defa" [puppet] - 10https://gerrit.wikimedia.org/r/198440 (https://phabricator.wikimedia.org/T93261) (owner: 10Dzahn) [19:13:37] (03PS2) 10Andrew Bogott: puppetmaster: Cleanup autosigner script some more [puppet] - 10https://gerrit.wikimedia.org/r/198790 (owner: 10Yuvipanda) [19:14:30] (03CR) 10jenkins-bot: [V: 04-1] puppetmaster: Cleanup autosigner script some more [puppet] - 10https://gerrit.wikimedia.org/r/198790 (owner: 10Yuvipanda) [19:15:12] PROBLEM - check_load on db1025 is CRITICAL: CRITICAL - load average: 42.15, 41.45, 31.28 [19:15:59] (03CR) 10Dzahn: [C: 032] check_graphite: accept additional date specification [puppet] - 10https://gerrit.wikimedia.org/r/198775 (owner: 10Filippo Giunchedi) [19:20:01] (03PS1) 10Alexandros Kosiaris: Ganeti module/role introduced [puppet] - 10https://gerrit.wikimedia.org/r/198794 [19:20:12] PROBLEM - check_load on db1025 is CRITICAL: CRITICAL - load average: 25.40, 33.60, 30.90 [19:23:31] (03PS1) 10GWicke: Enable trickle_fsync by default [puppet/cassandra] - 10https://gerrit.wikimedia.org/r/198796 [19:25:13] PROBLEM - check_load on db1025 is CRITICAL: CRITICAL - load average: 6.09, 19.33, 25.79 [19:27:11] (03PS4) 10Yuvipanda: add roles for staging redis ( *-rdb\d\d? ) [puppet] - 10https://gerrit.wikimedia.org/r/197348 (owner: 1020after4) [19:27:34] (03CR) 10Yuvipanda: [C: 032 V: 032] add roles for staging redis ( *-rdb\d\d? ) [puppet] - 10https://gerrit.wikimedia.org/r/197348 (owner: 1020after4) [19:29:25] (03PS29) 10JanZerebecki: Wikidata builder [puppet] - 10https://gerrit.wikimedia.org/r/195567 (https://phabricator.wikimedia.org/T90567) [19:31:14] * jzerebecki rages about ****ing style not being checked by computers [19:32:17] robh: would it be easy for us to give holmium an internal IP as well as the public one? It would be nice to have a back door for labs instances to query it. [19:32:28] * jzerebecki and non-****ing-voting checks [19:32:40] andrewbogott: you'll need mark to say thats ok to have a host intentionally bridge two vlans in the same host [19:32:43] we dont normally do that [19:32:53] robh: ok. I’ll make a ticket [19:32:54] (03CR) 10JanZerebecki: Wikidata builder (034 comments) [puppet] - 10https://gerrit.wikimedia.org/r/195567 (https://phabricator.wikimedia.org/T90567) (owner: 10JanZerebecki) [19:32:59] and then we'd add a second connection to it [19:33:06] since most systems have two nic ports, its usually fine [19:33:14] (well, we tend to do this for bonding, not two vlans) [19:34:48] (03PS3) 10Yuvipanda: puppetmaster: Cleanup autosigner script some more [puppet] - 10https://gerrit.wikimedia.org/r/198790 [19:35:04] (03CR) 10jenkins-bot: [V: 04-1] puppetmaster: Cleanup autosigner script some more [puppet] - 10https://gerrit.wikimedia.org/r/198790 (owner: 10Yuvipanda) [19:36:17] jesus jenkins [19:36:30] (03PS4) 10Yuvipanda: puppetmaster: Cleanup autosigner script some more [puppet] - 10https://gerrit.wikimedia.org/r/198790 [19:37:22] (03CR) 10jenkins-bot: [V: 04-1] puppetmaster: Cleanup autosigner script some more [puppet] - 10https://gerrit.wikimedia.org/r/198790 (owner: 10Yuvipanda) [19:38:31] (03PS5) 10Yuvipanda: puppetmaster: Cleanup autosigner script some more [puppet] - 10https://gerrit.wikimedia.org/r/198790 [19:38:58] 6operations: re-deploy cp4009 - https://phabricator.wikimedia.org/T93640#1142517 (10RobH) 3NEW [19:39:12] 6operations, 10ops-ulsfo: cp4009 hardware fault - https://phabricator.wikimedia.org/T92476#1142525 (10RobH) [19:39:13] 6operations: re-deploy cp4009 - https://phabricator.wikimedia.org/T93640#1142524 (10RobH) [19:39:48] !log Manually created the following global accounts (name@homewiki), per Keegan: Lugal@enwiki, Aoe@enwiki, and Moonkey@eswiki [19:39:52] Logged the message, Master [19:40:31] 6operations, 10ops-ulsfo: cp4009 hardware fault - https://phabricator.wikimedia.org/T92476#1142529 (10RobH) 5Open>3Resolved The mainboard has been replaced, and I'm taking the defective mainboard past a FedEx location to drop off for shipping. - updated service tag on mainboard to match - updated idrac li... [19:40:32] 6operations: re-deploy cp4009 - https://phabricator.wikimedia.org/T93640#1142517 (10RobH) [19:41:44] (03PS1) 10Dr0ptp4kt: Do not fragment cache with provenance parameter [puppet] - 10https://gerrit.wikimedia.org/r/198805 [19:43:48] bblack, tgr|away (and mr. liambotis if you're around and available) ^ would you please review? [19:44:38] 6operations, 10Analytics, 6Scrum-of-Scrums, 10Wikipedia-App-Android-App, and 3 others: Avoid cache fragmenting URLs for Share a Fact shares - https://phabricator.wikimedia.org/T90606#1063059 (10dr0ptp4kt) https://gerrit.wikimedia.org/r/#/c/198805/ under review. [19:46:33] dr0ptp4kt: is there a ticket ref for the overall URL scheme and such? [19:46:54] the email linked in the commit seems to be outdated compared to current [19:47:26] bblack: it rolled over to march. https://lists.wikimedia.org/pipermail/analytics/2015-March/003588.html [19:47:35] I mean, this sounds like a new feature effort that would be a phab task, etc [19:47:39] bblack: actually, that reminds me, i'm going to make a page on mediawiki.org real quick [19:47:45] (the issuing of these wprov things) [19:48:12] surely our process for adding new query arg metaparameters is not ML discussion -> commit+merge [19:49:35] (03CR) 10Steinsplitter: [C: 031] "yes, fine." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/194913 (https://phabricator.wikimedia.org/T91630) (owner: 10Odder) [19:50:08] the varnish part is mostly mechanical. if we're adding these arguments, then yes we need the varnish bit to avoid frag. I'm just wondering where the rest of the process/doc was on adding the param in the first place. [19:50:25] (03CR) 10Steinsplitter: "Please merge in the next days, needed for a edit-a-thon on thursday, March 26. Thanks" [mediawiki-config] - 10https://gerrit.wikimedia.org/r/198242 (https://phabricator.wikimedia.org/T93104) (owner: 10Steinsplitter) [19:51:37] bblack: the ticket is https://phabricator.wikimedia.org/T90606 [19:53:09] 6operations: align puppet-lint config with coding style - https://phabricator.wikimedia.org/T93645#1142580 (10JanZerebecki) 3NEW [19:53:38] 6operations: align puppet-lint config with coding style - https://phabricator.wikimedia.org/T93645#1142588 (10JanZerebecki) [19:54:41] Currently, the apps share URLs by adding a query to the URL: [19:54:41] ?source=app [19:54:42] 6operations: align puppet-lint config with coding style - https://phabricator.wikimedia.org/T93645#1142580 (10JanZerebecki) [19:54:51] dr0ptp4kt: ^ so we're already fragmenting today? :) [19:55:13] RECOVERY - check_load on db1025 is OK: OK - load average: 0.77, 1.08, 4.63 [19:56:03] I wonder how many things like these are causing our low hitrates :P [19:58:13] bblack: it's not in the stable channel [19:58:34] bblack: ...and yeah, me too :P [19:59:52] (03PS1) 10Ottomata: Log X-Cache header as new field in webrequest logs via varnishkafka [puppet] - 10https://gerrit.wikimedia.org/r/198809 (https://phabricator.wikimedia.org/T91749) [20:00:05] gwicke, cscott, arlolra, subbu: Dear anthropoid, the time has come. Please deploy Services – Parsoid / OCG / Citoid / … (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20150323T2000). [20:00:22] RECOVERY - Host mw2147 is UP: PING WARNING - Packet loss = 80%, RTA = 44.66 ms [20:01:01] (03PS1) 10JanZerebecki: Enable ensure_not_symlink_target puppet-lint check [puppet] - 10https://gerrit.wikimedia.org/r/198810 (https://phabricator.wikimedia.org/T93645) [20:01:34] 6operations: iridium "standard" conflict with exim in role - https://phabricator.wikimedia.org/T92879#1142636 (10yuvipanda) a:5yuvipanda>3None [20:03:21] (03CR) 10OliverKeyes: Do not fragment cache with provenance parameter (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/198805 (owner: 10Dr0ptp4kt) [20:05:10] (03CR) 10Ottomata: [C: 032] Log X-Cache header as new field in webrequest logs via varnishkafka [puppet] - 10https://gerrit.wikimedia.org/r/198809 (https://phabricator.wikimedia.org/T91749) (owner: 10Ottomata) [20:06:26] (03CR) 10Yuvipanda: Do not fragment cache with provenance parameter (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/198805 (owner: 10Dr0ptp4kt) [20:06:42] (03CR) 10Yuvipanda: "(however, I"ve no idea what the word 'provenance' means)" [puppet] - 10https://gerrit.wikimedia.org/r/198805 (owner: 10Dr0ptp4kt) [20:10:35] (03PS2) 10Dr0ptp4kt: Do not fragment cache with provenance parameter [puppet] - 10https://gerrit.wikimedia.org/r/198805 [20:11:04] starting parsoid/ocg deploy [20:11:15] nobody's in the middle of anything, are they? [20:11:41] everyone's in the middle of everything, always. hopefully not in ways related to what you're doing, though :) [20:11:45] :) [20:12:29] just double-checking since my irc client had left this room somehow, so i wouldn't know if i was jumping in in the middle of some #ops emergency [20:13:03] (03PS1) 10Yuvipanda: toollabs: Add mosh to bastions [puppet] - 10https://gerrit.wikimedia.org/r/198812 [20:13:04] cscott: Can usually check the SAL too. [20:13:13] marktraceur: that's good advice [20:13:49] (03PS2) 10Yuvipanda: toollabs: Add mosh to bastions [puppet] - 10https://gerrit.wikimedia.org/r/198812 [20:13:51] (03PS2) 10Chad: Better support checking out MediaWiki & extension masters [mediawiki-config] - 10https://gerrit.wikimedia.org/r/198783 [20:14:01] (03PS3) 10Yuvipanda: toollabs: Add mosh to bastions [puppet] - 10https://gerrit.wikimedia.org/r/198812 [20:14:09] (03CR) 10Yuvipanda: [C: 032 V: 032] toollabs: Add mosh to bastions [puppet] - 10https://gerrit.wikimedia.org/r/198812 (owner: 10Yuvipanda) [20:14:13] (03CR) 10jenkins-bot: [V: 04-1] Better support checking out MediaWiki & extension masters [mediawiki-config] - 10https://gerrit.wikimedia.org/r/198783 (owner: 10Chad) [20:14:28] 6operations, 5Patch-For-Review: align puppet-lint config with coding style - https://phabricator.wikimedia.org/T93645#1142674 (10JanZerebecki) [20:18:08] (03PS1) 10Chad: Add a couple of missing extensions from the entry list [mediawiki-config] - 10https://gerrit.wikimedia.org/r/198813 [20:19:03] (03PS1) 10Dzahn: puppet-lint.rc - do not disable indentation checks [puppet] - 10https://gerrit.wikimedia.org/r/198814 [20:19:32] (03PS2) 10Dzahn: puppet-lint.rc - do not disable indentation checks [puppet] - 10https://gerrit.wikimedia.org/r/198814 [20:20:07] (03PS3) 10Dzahn: puppet-lint.rc - do not disable indentation checks [puppet] - 10https://gerrit.wikimedia.org/r/198814 (https://phabricator.wikimedia.org/T93645) [20:20:21] 6operations, 5Patch-For-Review: align puppet-lint config with coding style - https://phabricator.wikimedia.org/T93645#1142717 (10Dzahn) https://gerrit.wikimedia.org/r/#/c/198814/ [20:20:29] (03CR) 10Faidon Liambotis: [C: 04-1] "This is pretty cool. Minor nitpicks, one of which is me wondering loudly :)" (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/198110 (owner: 10BBlack) [20:21:00] (03CR) 10jenkins-bot: [V: 04-1] puppet-lint.rc - do not disable indentation checks [puppet] - 10https://gerrit.wikimedia.org/r/198814 (https://phabricator.wikimedia.org/T93645) (owner: 10Dzahn) [20:23:50] (03CR) 10Dzahn: [C: 031] Enable ensure_not_symlink_target puppet-lint check [puppet] - 10https://gerrit.wikimedia.org/r/198810 (https://phabricator.wikimedia.org/T93645) (owner: 10JanZerebecki) [20:24:56] !log updated Parsoid to version a5d7483f [20:25:02] (03CR) 10JanZerebecki: "The failures are where leading spaces are not divisible by 2." [puppet] - 10https://gerrit.wikimedia.org/r/198814 (https://phabricator.wikimedia.org/T93645) (owner: 10Dzahn) [20:25:02] Logged the message, Master [20:26:18] (03PS3) 10Chad: Better support checking out MediaWiki & extension masters [mediawiki-config] - 10https://gerrit.wikimedia.org/r/198783 [20:34:09] (03CR) 10Dr0ptp4kt: "@Yuvipanda :) I remembered I needed to create the following:" [puppet] - 10https://gerrit.wikimedia.org/r/198805 (owner: 10Dr0ptp4kt) [20:34:58] (03PS1) 10Chmarkine: doc - Enable HSTS max-age=7 days [puppet] - 10https://gerrit.wikimedia.org/r/198819 (https://phabricator.wikimedia.org/T40516) [20:36:20] (03CR) 10JanZerebecki: [C: 031] doc - Enable HSTS max-age=7 days [puppet] - 10https://gerrit.wikimedia.org/r/198819 (https://phabricator.wikimedia.org/T40516) (owner: 10Chmarkine) [20:40:26] (03PS1) 10Andrew Bogott: Allow the labs dns server to recurse when hit from a labs instance. [puppet] - 10https://gerrit.wikimedia.org/r/198820 [20:40:32] 6operations, 10Analytics, 6Scrum-of-Scrums, 10Wikipedia-App-Android-App, and 3 others: Avoid cache fragmenting URLs for Share a Fact shares - https://phabricator.wikimedia.org/T90606#1142810 (10dr0ptp4kt) Reserved codes now listed at https://www.mediawiki.org/wiki/Provenance [20:41:45] (03CR) 10Tim Landscheidt: "On tools-login and tools-dev, mosh was installed because these hosts included role::labs::bastion (in addition to role::labs::tools::basti" [puppet] - 10https://gerrit.wikimedia.org/r/198812 (owner: 10Yuvipanda) [20:42:23] (03PS1) 10Dereckson: New user groups on fr.wikinews [mediawiki-config] - 10https://gerrit.wikimedia.org/r/198822 (https://phabricator.wikimedia.org/T90979) [20:42:37] <^d> aude: You know why we don't have $IP/Wikidata/Wikidata.php in extension-list? [20:42:53] (03PS2) 10Andrew Bogott: Allow the labs dns server to recurse when hit from a labs instance. [puppet] - 10https://gerrit.wikimedia.org/r/198820 [20:44:56] (03CR) 10Tim Landscheidt: "And I assume this will lead to Puppet errors on tools-login and tools-dev then?" [puppet] - 10https://gerrit.wikimedia.org/r/198812 (owner: 10Yuvipanda) [20:45:10] ^d: I think it adds to the global directly. [20:45:23] ^d: $wgExtensionEntryPointListFiles[] = "$IP/extensions/Wikidata/extension-list-wikidata"; [20:45:26] https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-config/CommonSettings.php#L2762 [20:46:08] <^d> Bleh ok [20:46:09] (03CR) 10Andrew Bogott: [C: 032] Allow the labs dns server to recurse when hit from a labs instance. [puppet] - 10https://gerrit.wikimedia.org/r/198820 (owner: 10Andrew Bogott) [20:46:10] i think that was to make things work when we were switching to have the build [20:46:15] might not be needed that way now [20:46:34] it's basically $IP/extensions/Wikidata/Wikidata.localisation.php [20:47:49] <^d> Does that file even exist? [20:47:49] <^d> :p [20:47:52] (03PS1) 10Dzahn: puppet-lint: fix all 2sp_soft_tabs errors [puppet] - 10https://gerrit.wikimedia.org/r/198854 [20:48:00] https://github.com/wikimedia/operations-mediawiki-config/commit/11ef274bf0da770430a440bfc384d1412b0d543f [20:48:04] ^d: in our build, yes [20:48:17] !log updated OCG to version 11f096b6e45ef183826721f5c6b0f933a387b1bb [20:48:22] Logged the message, Master [20:49:05] <^d> Oh dur, helps to look in the right place [20:49:37] <^d> aude: Reason I ask is https://gerrit.wikimedia.org/r/#/c/198783/ and https://gerrit.wikimedia.org/r/#/c/198813/ for staging [20:49:51] (03CR) 10Dzahn: "this should be all that needs to be fixed:" [puppet] - 10https://gerrit.wikimedia.org/r/198814 (https://phabricator.wikimedia.org/T93645) (owner: 10Dzahn) [20:51:32] ...and that does it for the parsoid/ocg deploy. [20:52:01] ^d: looking [20:53:36] (03CR) 10Dzahn: [C: 031] doc - Enable HSTS max-age=7 days [puppet] - 10https://gerrit.wikimedia.org/r/198819 (https://phabricator.wikimedia.org/T40516) (owner: 10Chmarkine) [20:54:24] (03CR) 10Dzahn: "meeting has been moved to March 24" [puppet] - 10https://gerrit.wikimedia.org/r/197798 (https://phabricator.wikimedia.org/T93151) (owner: 10Dzahn) [20:55:05] ^d: somewhat concerned and need to look in more detail on how to make it work [20:55:08] but see if ( $dstVersionNum == 'master' ) { [20:55:11] (03PS1) 10Ori.livneh: Move misc. utilities to utils/; remove typos [puppet] - 10https://gerrit.wikimedia.org/r/198923 [20:55:37] _joe_: ^ [20:55:40] so it might not apply to our stuff at this point [20:55:42] (03CR) 10BBlack: protoproxy/sslcert/cache: nginx ssl_stapling_file support (033 comments) [puppet] - 10https://gerrit.wikimedia.org/r/198110 (owner: 10BBlack) [20:55:52] <^d> aude: Yeah, I'm finding that :) [20:56:00] <^d> I'll keep poking [20:56:04] * ^d takes coffee break [20:56:07] <_joe_> ori: on my radar, but not right now :) [20:56:12] we still have the issue though of Special:Version saying wikidata was updated last May [20:56:49] i think it might need to be fixed in this script, instead or in addition to the makeWmfBranch script [20:56:51] _joe_: it just moves some files around. np [20:57:01] (03PS4) 10BBlack: add generic nrpe script check-fresh-files-in-dir.py [puppet] - 10https://gerrit.wikimedia.org/r/198387 [20:57:03] (03PS5) 10BBlack: test OCSP Stapling on cp1008 [puppet] - 10https://gerrit.wikimedia.org/r/198388 [20:57:05] <_joe_> ori: yes, I was about to give a +1 [20:57:05] (03PS19) 10BBlack: protoproxy/sslcert/cache: nginx ssl_stapling_file support [puppet] - 10https://gerrit.wikimedia.org/r/198110 [20:57:55] * aude heads home [20:58:13] (03CR) 10Giuseppe Lavagetto: [C: 031] "+1 for me but I think this is a beautiful occasion for an epic bikeshedding, let's not waste it merging before 10 people have argued" [puppet] - 10https://gerrit.wikimedia.org/r/198923 (owner: 10Ori.livneh) [20:59:36] Hi all [21:00:12] Is there anybody who has experience deploying MediaWiki with Fastly in relation to SquidPurgeClient to send HTTP PURGE requests even with SSL? [21:00:24] ori: do you mind reviewing some Cassandra config changes? [21:00:48] https://gerrit.wikimedia.org/r/#/c/198796/ and https://gerrit.wikimedia.org/r/#/c/198781/ [21:00:58] (03PS1) 10Andrew Bogott: Fix the fixed ip range for labs [puppet] - 10https://gerrit.wikimedia.org/r/198983 [21:01:28] bd808? You know somebody who have experience with SSL and Fastly in relation of PURGE? [21:02:00] renoirb: you? :) [21:02:10] Trying to [21:02:26] (03CR) 10Andrew Bogott: [C: 032] Fix the fixed ip range for labs [puppet] - 10https://gerrit.wikimedia.org/r/198983 (owner: 10Andrew Bogott) [21:02:53] I’ve been sending some debug output from the classes, they always sends me a 301. Even though I configure Varnish to NOT do redirect to SSL when its a PURGE [21:05:10] oh! it works now. [21:05:58] hasharDinner: where (on which host) is /mnt/jenkins-workspace/ ? since gallium only has /mnt/jenkins-tmp [21:06:38] 6operations, 10ops-ulsfo: cp4009 hardware fault - https://phabricator.wikimedia.org/T92476#1142889 (10RobH) {F103186} dropped off mainboard at fedex, proof of return. [21:06:39] the path shows up in console output pages on integration.wm [21:07:36] ah, i think i found it. integration-slave1004 .. right [21:07:37] bd808, I think that the fact I made $wgVaryOnXFP = true; made a difference. [21:11:55] (03CR) 10Dzahn: [C: 032] "confirmed on integration-slave1004" [puppet] - 10https://gerrit.wikimedia.org/r/198810 (https://phabricator.wikimedia.org/T93645) (owner: 10JanZerebecki) [21:12:02] (03PS1) 10Chmarkine: annual - Enable HSTS max-age=7 days [puppet] - 10https://gerrit.wikimedia.org/r/199087 (https://phabricator.wikimedia.org/T599) [21:12:14] (03PS1) 10Rush: exim4.conf.SMTP_IMAP_MM.erb local mail can cause loops [puppet] - 10https://gerrit.wikimedia.org/r/199090 [21:13:02] (03Abandoned) 10Rush: exim should send unaliased local mail to root@wikimedia [puppet] - 10https://gerrit.wikimedia.org/r/198114 (owner: 10Rush) [21:13:10] mutante: the labs instances we are using [21:13:35] mutante: nowadays gallium is barely running any jobs :) [21:14:34] hasharDinner: right, i confirmed on integration-slave1004, thanks [21:17:26] mutante: we had to that to preserve the instance / or it would be overfilled :) [21:19:12] RECOVERY - Host cp4009 is UP: PING OK - Packet loss = 0%, RTA = 77.86 ms [21:21:05] 6operations, 7HTTPS, 3HTTPS-by-default: Expand HTTP frontend clusters with new hardware - https://phabricator.wikimedia.org/T86663#1142954 (10BBlack) FYI, we've regressed on the minor capacity bump for eqiad-upload. We had to depool the new machines from service over row network issues. Now blocking on T92... [21:21:18] 6operations, 7HTTPS, 3HTTPS-by-default: Expand HTTP frontend clusters with new hardware - https://phabricator.wikimedia.org/T86663#1142957 (10BBlack) [21:21:18] 6operations, 10ops-eqiad: Increase asw-d-eqiad uplink capacity - https://phabricator.wikimedia.org/T92914#1123769 (10BBlack) [21:22:13] 7Puppet, 6Phabricator, 5Patch-For-Review: Phabricator labs instance isn't configured with a local repo storage - https://phabricator.wikimedia.org/T93615#1142966 (10Negative24) [21:30:40] (03CR) 10Dzahn: "compare to https://integration.wikimedia.org/ci/job/operations-puppet-puppetlint-lenient/16845/console" [puppet] - 10https://gerrit.wikimedia.org/r/198854 (owner: 10Dzahn) [21:30:59] (03CR) 10JanZerebecki: [C: 031] "Checked that it only contains space changes that are no-ops." [puppet] - 10https://gerrit.wikimedia.org/r/198854 (owner: 10Dzahn) [21:32:29] (03CR) 10Dzahn: [C: 032] puppet-lint: fix all 2sp_soft_tabs errors [puppet] - 10https://gerrit.wikimedia.org/r/198854 (owner: 10Dzahn) [21:33:22] (03PS4) 10Dzahn: puppet-lint.rc - do not disable indentation checks [puppet] - 10https://gerrit.wikimedia.org/r/198814 (https://phabricator.wikimedia.org/T93645) [21:34:55] (03CR) 10jenkins-bot: [V: 04-1] puppet-lint.rc - do not disable indentation checks [puppet] - 10https://gerrit.wikimedia.org/r/198814 (https://phabricator.wikimedia.org/T93645) (owner: 10Dzahn) [21:36:41] PROBLEM - Disk space on lanthanum is CRITICAL: DISK CRITICAL - free space: /srv/ssd 5454 MB (3% inode=86%): [21:39:19] greg-g: FYI: https://wikitech.wikimedia.org/wiki/Deployments#Tuesday.2C.C2.A0March.C2.A024 [21:41:34] hoo: cool [21:42:12] 6operations: This video gets you laid? - https://phabricator.wikimedia.org/T93667#1143088 (10emailbot) [21:42:30] :) [21:42:57] how do we deal with spam in phab again? [21:43:20] Close the task, remove all projects, add the spam project [21:43:22] I think [21:44:38] <^d> I closed T93667 [21:44:45] Thanks [21:46:22] (03PS1) 10Chmarkine: servermon - Enable HSTS max-age=7 days [puppet] - 10https://gerrit.wikimedia.org/r/199134 (https://phabricator.wikimedia.org/T40516) [21:48:43] (03PS1) 10BBlack: repool cp4009 T92476 [puppet] - 10https://gerrit.wikimedia.org/r/199135 [21:49:20] (03CR) 10Dzahn: [C: 031] servermon - Enable HSTS max-age=7 days [puppet] - 10https://gerrit.wikimedia.org/r/199134 (https://phabricator.wikimedia.org/T40516) (owner: 10Chmarkine) [21:49:25] 6operations, 10Analytics, 6Scrum-of-Scrums, 10Wikipedia-App-Android-App, and 3 others: Avoid cache fragmenting URLs for Share a Fact shares - https://phabricator.wikimedia.org/T90606#1143136 (10dr0ptp4kt) The iOS code update was https://gerrit.wikimedia.org/r/#/c/196243/, by the way. [21:49:28] (03CR) 10BBlack: [C: 032 V: 032] repool cp4009 T92476 [puppet] - 10https://gerrit.wikimedia.org/r/199135 (owner: 10BBlack) [21:50:11] 6operations, 10ops-ulsfo: cp4009 hardware fault - https://phabricator.wikimedia.org/T92476#1143143 (10BBlack) [21:50:44] (03PS1) 10JanZerebecki: puppet-lint: fix all 2sp_soft_tabs errors [puppet] - 10https://gerrit.wikimedia.org/r/199136 [21:50:44] 6operations, 10ops-ulsfo: cp4009 hardware fault - https://phabricator.wikimedia.org/T92476#1112175 (10BBlack) server is reinstalled and repooled, seems to be functional! leaving this open in case others not done tracking hw-related bits w/ Dell. [21:53:46] (03CR) 10Dzahn: [C: 032] puppet-lint: fix all 2sp_soft_tabs errors [puppet] - 10https://gerrit.wikimedia.org/r/199136 (owner: 10JanZerebecki) [21:54:11] (03PS5) 10Dzahn: puppet-lint.rc - do not disable indentation checks [puppet] - 10https://gerrit.wikimedia.org/r/198814 (https://phabricator.wikimedia.org/T93645) [21:54:57] (03CR) 10JanZerebecki: [C: 031] servermon - Enable HSTS max-age=7 days [puppet] - 10https://gerrit.wikimedia.org/r/199134 (https://phabricator.wikimedia.org/T40516) (owner: 10Chmarkine) [21:55:38] (03CR) 10Dzahn: "operations-puppet-puppetlint-lenient SUCCESS in 42s" [puppet] - 10https://gerrit.wikimedia.org/r/198814 (https://phabricator.wikimedia.org/T93645) (owner: 10Dzahn) [21:55:52] (03CR) 10JanZerebecki: [C: 031] puppet-lint.rc - do not disable indentation checks [puppet] - 10https://gerrit.wikimedia.org/r/198814 (https://phabricator.wikimedia.org/T93645) (owner: 10Dzahn) [21:57:21] (03CR) 10Dzahn: [C: 031] annual - Enable HSTS max-age=7 days [puppet] - 10https://gerrit.wikimedia.org/r/199087 (https://phabricator.wikimedia.org/T599) (owner: 10Chmarkine) [21:58:39] (03CR) 10JanZerebecki: [C: 031] annual - Enable HSTS max-age=7 days [puppet] - 10https://gerrit.wikimedia.org/r/199087 (https://phabricator.wikimedia.org/T599) (owner: 10Chmarkine) [21:58:46] 6operations, 10Continuous-Integration: fix failures of jenkins job operations-puppet-puppetlint-strict - https://phabricator.wikimedia.org/T93642#1143178 (10Krenair) [21:59:00] (03PS2) 10Ori.livneh: Move misc. utilities to utils/; remove typos [puppet] - 10https://gerrit.wikimedia.org/r/198923 [21:59:13] (03CR) 10Ori.livneh: [C: 032 V: 032] "Better approach: see if anyone complains." [puppet] - 10https://gerrit.wikimedia.org/r/198923 (owner: 10Ori.livneh) [22:00:06] (03PS1) 10Chmarkine: dbtree - Enable HSTS max-age=7 days [puppet] - 10https://gerrit.wikimedia.org/r/199139 (https://phabricator.wikimedia.org/T40516) [22:01:25] Negative24: no, the phab configs are in puppet [22:02:29] twentyafterfour: I can see that and that is what I've been going off of but seeing the actual files (mysql in this case) would be helpful. Probably not going to happen though. [22:03:45] Negative24: what do you mean specifically? the mysql login credentials? [22:04:38] twentyafterfour: nah. Those are in the private puppet repo. I'm trying to see if the production phab has the mysql configs that phab is asking me to config on labs [22:04:58] I'm not trying to break the server :) [22:05:10] production works pretty much the same as labs via the puppet roles [22:05:13] just different data [22:05:33] chasemp: I think he's referring to the custom mysql settings that phab recently started suggesting [22:05:57] Negative24: I think we implemented most of the recommendations [22:06:02] chasemp: If that is the case then I'll file a ticket about it [22:06:18] ah here is a difference, prod uses mariadb and labs is set to mysql [22:06:34] someone should fix that :) and put in the setup suggestions I imagine [22:06:37] Jenkins seems stuck [22:06:51] gate-and-submit [22:06:53] i suspect ori's recent change [22:06:59] twentyafterfour: But if that is the case then I would like to know what you guys did so that I can (maybe) implement something similar in labs [22:07:13] or maybe I'm just worrying about trivial things (again) [22:08:34] it's a good point but the setup config for prod db's will be separated logistically in puppet as it is on another host [22:08:42] whereas labs is obvously self contained [22:08:43] (03CR) 10Dzahn: "ok, i will. the typos were likely and we now have reports that jenkins is stuck" [puppet] - 10https://gerrit.wikimedia.org/r/198923 (owner: 10Ori.livneh) [22:08:49] worth a ticket probably to clean up [22:11:42] (03CR) 10Dzahn: [C: 032] puppet-lint.rc - do not disable indentation checks [puppet] - 10https://gerrit.wikimedia.org/r/198814 (https://phabricator.wikimedia.org/T93645) (owner: 10Dzahn) [22:13:46] PROBLEM - Disk space on gallium is CRITICAL: DISK CRITICAL - free space: /srv/ssd 5843 MB (3% inode=84%): [22:13:55] chasemp: I don't know exactly what you mean so could you submit one (CC me on it)? [22:14:19] Negative24: I assume you installed phab and see some setup errors / guidance? [22:14:25] and you are wondering if they ocurred in prod? [22:14:37] chasemp: yup [22:15:00] so if you take that, screenshot and make an issue saying default labs setup has issues [22:15:08] we can start figuring out how to lab-ify it [22:15:25] yea that is exactly what I want [22:15:55] my point is you have the details so best if you make the issue :) [22:16:47] 6operations, 10Continuous-Integration: fix failures of jenkins job operations-puppet-puppetlint-strict - https://phabricator.wikimedia.org/T93642#1143262 (10hashar) [22:16:48] 7Puppet, 6operations, 5Patch-For-Review: Make Puppet repository pass lenient and strict lint checks - https://phabricator.wikimedia.org/T87132#1143263 (10hashar) [22:17:55] 6operations, 5Patch-For-Review: align puppet-lint config with coding style - https://phabricator.wikimedia.org/T93645#1143276 (10hashar) Please see also {T87132} [22:18:05] 7Puppet, 6operations, 5Patch-For-Review: Make Puppet repository pass lenient and strict lint checks - https://phabricator.wikimedia.org/T87132#1143278 (10Dzahn) we fixed all remaining tab errors and enabled checks for it https://gerrit.wikimedia.org/r/#/c/198814/ [22:19:38] chasemp twentyafterfour: https://phabricator.wikimedia.org/T93677 [22:19:54] (03PS1) 10Chmarkine: iegreview - Enable HSTS max-age=7 days [puppet] - 10https://gerrit.wikimedia.org/r/199142 (https://phabricator.wikimedia.org/T40516) [22:38:02] (03CR) 10Dzahn: [C: 031] dbtree - Enable HSTS max-age=7 days [puppet] - 10https://gerrit.wikimedia.org/r/199139 (https://phabricator.wikimedia.org/T40516) (owner: 10Chmarkine) [22:38:17] (03CR) 10Dzahn: [C: 031] iegreview - Enable HSTS max-age=7 days [puppet] - 10https://gerrit.wikimedia.org/r/199142 (https://phabricator.wikimedia.org/T40516) (owner: 10Chmarkine) [22:42:34] chasemp: Do we have a rescheduled date for the phab update on the 18th [22:43:03] we never scheduled teh any for the 18th so sort of yes and sort of no [22:43:08] no next upgrade is scheduled [22:43:11] yet [22:43:39] so the HOLD: Window to update phabricator.wikimedia.org doesn't actually mean anything? [22:43:57] well, it's there to use it if its needed [22:44:21] ah, just a reservation [22:46:06] Its so great that the update windows are also in MDT kudos to the person who does that :) [22:47:33] it's in your browser's local time and UTC [22:48:57] oh [22:49:12] well that's even more clever [22:50:23] are there API problems happening? queries to the liststudents API taking *forever* [22:58:50] I think I should have access to x1-analytics-slave.eqiad.wmnet. I do have access to stat1003, but when I try to connect to the DB it doesn't work. [22:59:54] I get: [22:59:59] 6operations, 10Fundraising Dash: Create sandbox site for Dash - https://phabricator.wikimedia.org/T87809#1143447 (10atgo) Ping @jgreen - do you have an idea for when this might be doable? [23:00:05] RoanKattouw, ^d, Krenair: Dear anthropoid, the time has come. Please deploy Evening SWAT (Max 8 patches) (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20150323T2300). [23:00:06] RECOVERY - Disk space on lanthanum is OK: DISK OK [23:00:12] ERROR 1045 (28000): Access denied for user 'research'@ (using password: YES) [23:00:44] RoanKattouw is _away [23:00:50] you doing it ^d? [23:00:58] hi [23:01:01] I have 2 patches for swat [23:01:14] superm401: the research password has been updated at some point, but you should always have access to the current one [23:01:23] a little flustered since everything just blew up [23:01:32] superm401: puppet puts it into a file you can read on shell [23:01:34] mutante, yeah, and I do since I can access the file in /etc directly. [23:02:05] superm401: good, that was kind of the point of it, so when it's changed the researchers can get the current one without having to mail [23:02:14] mutante, right, but it doesn't actually work. [23:02:17] mysql -h x1-analytics-slave.eqiad.wmnet [23:02:30] superm401: oh, then i think it's a question for springle [23:02:40] (03PS1) 10Negative24: puppet-lint: Disable URL modules/ check [puppet] - 10https://gerrit.wikimedia.org/r/199154 [23:02:45] just gives the above error message. It doesn't seem that I need a home-directory config (since it picks up the username at least apparently from /etc), but copying it to ~/.my.cnf doesn't help either. [23:03:09] Okay, I'll do it then. [23:03:13] kaldari, ping [23:03:22] howdy [23:03:40] (03PS2) 10Alex Monk: Removing old author WikiGrok campaign [mediawiki-config] - 10https://gerrit.wikimedia.org/r/198195 (owner: 10Kaldari) [23:04:06] (03CR) 10Alex Monk: [C: 032] Removing old author WikiGrok campaign [mediawiki-config] - 10https://gerrit.wikimedia.org/r/198195 (owner: 10Kaldari) [23:04:22] Krenair: added my two patches [23:04:24] <^d> Krenair: I hadn't planned on it :p [23:04:48] legoktm, can we expect jenkins to be functional? [23:04:58] Krenair: ...I wouldn't count on it [23:04:59] or is it force approve all the patches deployment fun time? [23:05:00] (03CR) 10jenkins-bot: [V: 04-1] puppet-lint: Disable URL modules/ check [puppet] - 10https://gerrit.wikimedia.org/r/199154 (owner: 10Negative24) [23:05:02] (03CR) 10jenkins-bot: [V: 04-1] Removing old author WikiGrok campaign [mediawiki-config] - 10https://gerrit.wikimedia.org/r/198195 (owner: 10Kaldari) [23:05:22] Yeah that's broken. [23:05:23] Okay. [23:05:42] (03CR) 10Alex Monk: [V: 032] "Jenkins is broken." [mediawiki-config] - 10https://gerrit.wikimedia.org/r/198195 (owner: 10Kaldari) [23:06:40] !log krenair Synchronized wmf-config/mobile.php: https://gerrit.wikimedia.org/r/#/c/198195/ (duration: 00m 06s) [23:06:43] kaldari, ^ [23:06:47] looking [23:06:48] Logged the message, Master [23:07:20] gwicke, hey. your patches look... [23:07:26] slightly odd. [23:08:05] Krenair: what is odd about them? [23:08:13] You're updating the entire extension to master, OK... [23:08:48] it's a small extension [23:09:37] yeah [23:09:37] Oh that's whats going on. [23:09:46] Jenkins fault not mine :) [23:09:55] there's a big discussion about it in -releng [23:10:01] I think this patch includes i18n changes [23:10:06] Krenair: jenkins should be back theoretically [23:11:11] legoktm, does it matter if I just sync-dir an extension that has i18n changes? [23:11:16] or does it need a full scap? [23:11:18] Krenair: yeah, no reason to cut the i18n changes out [23:11:22] bd808, ^ [23:11:36] Krenair: looks good! [23:11:40] we don't really care about the i18n changes for this I don't think [23:11:47] Krenair: the i18n messages will use the old ones, and the next time scap happens it'll pick up the message changes [23:11:52] Krenair: it's not critical at all to get the i18n stuff out right now [23:12:04] (03PS2) 10Negative24: puppet-lint: Disable URL modules check [puppet] - 10https://gerrit.wikimedia.org/r/199154 [23:12:12] gwicke, right but they would end up on tin but not fully synced. [23:12:14] it's only used in Special:Version [23:13:01] Krenair: you could sync the dir [23:13:12] l10nupdate will run in ~4 hours and push them then to the l10n cache [23:13:14] That's what I'm planning to do [23:13:17] that won't do the full i18n update of course [23:13:21] but imho that's okay [23:13:44] Heh, Jenkins seems to be working now. [23:13:46] bd808: *nod* [23:13:57] This does lose the commit that changes the git-review module thing, of course [23:14:23] (this: https://github.com/wikimedia/mediawiki-extensions-RestBaseUpdateJobs/commit/e6b57f1ff18cce764c209d67fae57cfc85a2826c ) [23:14:32] but that doesn't matter much, I don't thin [23:14:34] think* [23:15:05] yeah, that won't break anything [23:15:15] we don't use the branches really [23:15:23] it will only effect git-review activity from a clone [23:17:19] Krenair: do you want me to do the CA submodule updates? [23:17:32] yes please [23:18:40] !log Stopping Jenkins for an upgrade [23:18:45] Logged the message, Master [23:19:06] well then. [23:19:36] ... [23:19:38] hasharDinner [23:19:51] This is the evening SWAT window. [23:20:35] Krenair: I think it should be back soon if things go to plan [23:20:50] I'm just going to force approve stuff until it's back. [23:21:27] Krenair: https://gerrit.wikimedia.org/r/199157 and https://gerrit.wikimedia.org/r/199158 [23:21:37] ty [23:23:38] sigh. [23:24:14] (03CR) 10Gage: [C: 032] Enable trickle_fsync by default [puppet/cassandra] - 10https://gerrit.wikimedia.org/r/198796 (owner: 10GWicke) [23:24:16] ok [23:24:17] gwicke [23:24:22] wmf22 syncing [23:24:25] !log krenair Synchronized php-1.25wmf22/extensions/RestBaseUpdateJobs: https://gerrit.wikimedia.org/r/#/c/199137/ (duration: 00m 10s) [23:24:28] Logged the message, Master [23:24:36] (03PS2) 10Gage: increased compaction concurrency and throughput [puppet] - 10https://gerrit.wikimedia.org/r/198781 (https://phabricator.wikimedia.org/T93140) (owner: 10Eevans) [23:25:02] please check [23:25:39] (03CR) 10Gage: [V: 032] Enable trickle_fsync by default [puppet/cassandra] - 10https://gerrit.wikimedia.org/r/198796 (owner: 10GWicke) [23:26:08] Krenair: yeah we had an outage on Jenkins due to disk space being full [23:26:14] Yeah I saw [23:26:22] Krenair: and I really had to unexpectingly upgrade Jenkins :( [23:26:26] sorry for the SWAT people! [23:26:32] It then started working again, and processing our jobs [23:26:43] after they removed some thigns [23:27:11] does it look OK, gwicke? [23:29:23] gwicke? [23:29:24] Krenair: looking at logstash [23:29:26] ok [23:30:05] Krenair: looks okay [23:32:29] !log krenair Synchronized php-1.25wmf21/extensions/RestBaseUpdateJobs: https://gerrit.wikimedia.org/r/#/c/199138/ (duration: 00m 07s) [23:32:29] gwicke, wmf21 syncing [23:32:32] Logged the message, Master [23:33:06] (03PS3) 10Dzahn: include firewall on uranium [puppet] - 10https://gerrit.wikimedia.org/r/172434 (owner: 10John F. Lewis) [23:33:30] ebernhardson, hey, could you do submodule updates please? [23:33:40] Krenair: sure [23:33:44] ty [23:33:55] (03PS4) 10Dzahn: include firewall on uranium [puppet] - 10https://gerrit.wikimedia.org/r/172434 (owner: 10John F. Lewis) [23:35:11] (03CR) 10Dzahn: [C: 032] include firewall on uranium [puppet] - 10https://gerrit.wikimedia.org/r/172434 (owner: 10John F. Lewis) [23:36:51] Krenair: still looking all good [23:37:13] ok, moving on to legoktm [23:37:24] (03PS1) 10GWicke: Update the cassandra submodule [puppet] - 10https://gerrit.wikimedia.org/r/199166 [23:37:27] (03CR) 10Aaron Schulz: [C: 031] Add dedicated runner for MessageIndexRebuildJob [puppet] - 10https://gerrit.wikimedia.org/r/197919 (https://phabricator.wikimedia.org/T90704) (owner: 10Nikerabbit) [23:37:28] o/ [23:38:13] (03CR) 10Aaron Schulz: [C: 031] $wgTranslateDelayedMessageIndexRebuild = true; [mediawiki-config] - 10https://gerrit.wikimedia.org/r/197920 (https://phabricator.wikimedia.org/T90704) (owner: 10Nikerabbit) [23:39:40] Krenair: https://gerrit.wikimedia.org/r/199168 and https://gerrit.wikimedia.org/r/199167 [23:40:17] the jobs are very far behind in the queue :/ [23:40:59] (03CR) 10Gage: [C: 032] increased compaction concurrency and throughput [puppet] - 10https://gerrit.wikimedia.org/r/198781 (https://phabricator.wikimedia.org/T93140) (owner: 10Eevans) [23:44:11] !log krenair Synchronized php-1.25wmf21/extensions/CentralAuth/includes/specials/SpecialGlobalRenameQueue.php: https://gerrit.wikimedia.org/r/#/c/199158/1 (duration: 00m 05s) [23:44:12] legoktm [23:44:16] Logged the message, Master [23:50:11] !log krenair Synchronized php-1.25wmf22/extensions/CentralAuth/includes/specials/SpecialGlobalRenameQueue.php: https://gerrit.wikimedia.org/r/#/c/199157/1 (duration: 00m 07s) [23:50:14] Logged the message, Master [23:50:40] (just checking these myself per pm) [23:51:46] greg-g: ping [23:51:46] gwicke: You sent me a contentless ping. This is a contentless pong. Please provide a bit of information about what you want and I will respond when I am around. [23:52:07] greg-g: I think you'll find that is far form contentless :P [23:52:36] JohnLewis: yeah, it was the default message from a non-native english speaker, and I haven't modified it yet [23:52:39] gwicke: pong [23:52:50] greg-g: praveena just asks about outages last week [23:53:00] anything that comes to mind apart from eventlogging? [23:53:01] greg-g: content is the same in any language though [23:53:01] don't think so? [23:53:24] ok, she's happy with that answer ;) [23:53:25] JohnLewis: how it's phrased, I wouldn't say the bot's response is contentless [23:53:32] gwicke: :) [23:53:44] greg-g: I still disagree but agree to disagree :) [23:53:59] Yep, looks good [23:54:07] ok, ebernhardson [23:54:11] JohnLewis: which part do you think is contentless? I might have misunderstood :) [23:54:37] greg-g: any content is content even if you're saying it is contentless. content! [23:55:07] I see I see. It did pass the Shannon definition of information [23:57:24] ebernhardson, syncing wmf22 [23:57:26] !log krenair Synchronized php-1.25wmf22/extensions/Flow/container.php: https://gerrit.wikimedia.org/r/#/c/199167/1 (duration: 00m 07s) [23:57:31] Logged the message, Master [23:57:36] RECOVERY - Graphite Carbon on graphite2001 is OK: OK: All defined Carbon jobs are runnning. [23:57:40] please check [23:58:57] PROBLEM - Disk space on palladium is CRITICAL: DISK CRITICAL - free space: / 4076 MB (3% inode=93%): [23:59:44] have wmf21 ready to go as soon as you are