[01:13:11] <icinga-wm>	 PROBLEM - Disk space on scb1002 is CRITICAL: DISK CRITICAL - free space: / 350 MB (3% inode=87%)
[01:50:02] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps-test2002 is CRITICAL: CRITICAL - Rep Delay is: 1804.189202 Seconds
[01:52:01] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps-test2002 is OK: OK - Rep Delay is: 73.058234 Seconds
[02:21:58] <logmsgbot>	 !log mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.15) (duration: 10m 05s)
[02:22:03] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[02:27:39] <logmsgbot>	 !log l10nupdate@tin ResourceLoader cache refresh completed at Sun Aug 21 02:27:39 UTC 2016 (duration 5m 41s)
[02:27:45] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[02:29:13] <grrrit-wm>	 (03CR) 10BryanDavis: [C: 031] mw-log-cleanup: remove wfDebug files in deployment-prep every week [puppet] - 10https://gerrit.wikimedia.org/r/305768 (owner: 10Alex Monk)
[02:29:54] <grrrit-wm>	 (03CR) 10BryanDavis: [C: 031] deployment-prep: remove old deployment-fluorine hieradata [puppet] - 10https://gerrit.wikimedia.org/r/305765 (owner: 10Alex Monk)
[02:31:37] <grrrit-wm>	 (03CR) 10BryanDavis: [C: 031] Remove the hard-coded /a/mw-log references scattered around everywhere [puppet] - 10https://gerrit.wikimedia.org/r/305767 (owner: 10Alex Monk)
[02:32:22] <grrrit-wm>	 (03CR) 10BryanDavis: [C: 031] udp2log::instance: require psmisc package for use of killall command [puppet] - 10https://gerrit.wikimedia.org/r/305766 (owner: 10Alex Monk)
[03:20:32] <icinga-wm>	 RECOVERY - MediaWiki exceptions and fatals per minute on graphite1001 is OK: OK: Less than 1.00% above the threshold [25.0]
[03:24:16] <grrrit-wm>	 (03PS5) 10Yuvipanda: [WIP] Introduce 'clush' module and toollabs role [puppet] - 10https://gerrit.wikimedia.org/r/305804 
[03:25:35] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] [WIP] Introduce 'clush' module and toollabs role [puppet] - 10https://gerrit.wikimedia.org/r/305804 (owner: 10Yuvipanda)
[03:26:38] <grrrit-wm>	 (03PS6) 10Yuvipanda: [WIP] Introduce 'clush' module and toollabs role [puppet] - 10https://gerrit.wikimedia.org/r/305804 
[03:28:03] <grrrit-wm>	 (03PS7) 10Yuvipanda: [WIP] Introduce 'clush' module and toollabs role [puppet] - 10https://gerrit.wikimedia.org/r/305804 
[03:30:25] <grrrit-wm>	 (03PS8) 10Yuvipanda: [WIP] Introduce 'clush' module and toollabs role [puppet] - 10https://gerrit.wikimedia.org/r/305804 
[03:34:34] <wikibugs>	 06Operations, 06MediaWiki-Stakeholders-Group, 10Traffic, 07Developer-notice, and 2 others: Get rid of geoiplookup service - https://phabricator.wikimedia.org/T100902#2570041 (10MZMcBride) >>! In T100902#2566841, @faidon wrote: > We cannot and will not be a general purpose GeoIP provider, for various and di...
[03:35:11] <grrrit-wm>	 (03PS9) 10Yuvipanda: [WIP] Introduce 'clush' module and toollabs role [puppet] - 10https://gerrit.wikimedia.org/r/305804 
[03:40:54] <grrrit-wm>	 (03PS10) 10Yuvipanda: [WIP] Introduce 'clush' module and toollabs role [puppet] - 10https://gerrit.wikimedia.org/r/305804 
[03:47:05] <grrrit-wm>	 (03PS11) 10Yuvipanda: [WIP] Introduce 'clush' module and toollabs role [puppet] - 10https://gerrit.wikimedia.org/r/305804 
[05:08:13] <icinga-wm>	 PROBLEM - puppet last run on mw2185 is CRITICAL: CRITICAL: puppet fail
[05:34:02] <icinga-wm>	 RECOVERY - puppet last run on mw2185 is OK: OK: Puppet is currently enabled, last run 5 seconds ago with 0 failures
[05:47:53] <icinga-wm>	 PROBLEM - Postgres Replication Lag on maps2002 is CRITICAL: CRITICAL - Rep Delay is: 1808.826049 Seconds
[05:49:52] <icinga-wm>	 RECOVERY - Postgres Replication Lag on maps2002 is OK: OK - Rep Delay is: 0.0 Seconds
[05:54:55] <fhisjfkasda>	 我账号被盗了、
[05:54:59] <fhisjfkasda>	 账号被盗了
[05:55:03] <fhisjfkasda>	 维基百科的账号
[06:01:32] <Cameron11598>	 fhisjfkasda: Try reporting it in #Wikimedia-stewards
[06:21:21] <icinga-wm>	 PROBLEM - puppet last run on mw2130 is CRITICAL: CRITICAL: puppet fail
[06:27:52] <icinga-wm>	 RECOVERY - Disk space on scb1002 is OK: DISK OK
[06:49:01] <icinga-wm>	 RECOVERY - puppet last run on mw2130 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures
[09:41:51] <icinga-wm>	 PROBLEM - Disk space on scb1001 is CRITICAL: DISK CRITICAL - free space: / 350 MB (3% inode=84%)
[10:25:14] <andalousie>	 hello
[10:25:46] <andalousie>	 my friends' account on wikipedia was stolen
[10:26:40] <andalousie>	 can anyone help recover it?
[11:11:55] <grrrit-wm>	 (03PS2) 10Addshore: Add Collection render note for articles & rdf2latex [mediawiki-config] - 10https://gerrit.wikimedia.org/r/305012 (https://phabricator.wikimedia.org/T135613) 
[11:54:38] <andalousie>	 hello, any system administrator online?
[11:55:43] <Luke081515>	 andalousie: I'm not, but what's the exact problem?
[11:56:04] <Luke081515>	 IF you can't recover that acc, you may ask a steward for locking that account at #wikimedia-stewards
[11:56:16] <andalousie>	 my friend's account was hijacked
[11:56:41] <andalousie>	 locking?
[11:57:08] <andalousie>	 no way of recovering?
[12:00:11] <andalousie>	 oh, still thanks
[13:17:32] <icinga-wm>	 PROBLEM - MariaDB Slave Lag: s2 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 1177.45 seconds
[13:23:54] <paladox>	 Hi it seems phabricator phd daemons are not running
[13:24:00] <paladox>	 https://phabricator.wikimedia.org/diffusion/ESHG/manage/status/
[13:24:16] <paladox>	 Could someone start them please so that repos and other things can update please>
[13:24:19] <paladox>	 > = ?
[13:24:28] <paladox>	 twentyafterfour ^^
[13:36:17] <icinga-wm>	 ACKNOWLEDGEMENT - MariaDB Slave Lag: s2 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 2270.29 seconds Jcrespo ongoing s2 schema change, lag only affects tokudb servers (not critical)
[14:26:36] <xiaocao2015>	 Anyone here？
[14:26:39] <xiaocao2015>	 Anyone here?
[14:26:47] <andalousie>	 Shanmugamp7
[14:26:47] <andalousie>	 Shanmugamp7
[14:26:48] <andalousie>	 好了，没人回的
[14:27:02] <xiaocao2015>	 ？？？
[14:27:15] <Shanmugamp7>	 Yes andalousie 
[14:27:19] <xiaocao2015>	 @Shanmugamp7 Are you here?
[14:27:27] <andalousie>	 oh, hi
[14:27:32] <xiaocao2015>	 Any sysadmin here?
[14:27:37] <andalousie>	 I am just a friend of xiaocao2015
[14:27:53] <andalousie>	 and tried to find if any sysadmins are here
[14:27:53] <xiaocao2015>	 My ALL account was hacked.
[14:28:07] <xiaocao2015>	 My password was been publiced in a website.
[14:28:17] <andalousie>	 thank you very much for helping xiaocao2015
[14:28:25] <xiaocao2015>	 I changed all password and format my computer.
[14:28:32] <xiaocao2015>	 But It is too late.
[14:28:35] <andalousie>	 but we are not very familiar with Phabricator
[14:29:59] <xiaocao2015>	 I change the password of all account 
[14:30:04] <xiaocao2015>	 But too late,
[14:30:17] <xiaocao2015>	 The Wikipedia E-mail was be changed.
[14:30:26] <xiaocao2015>	 My E-mail was be abused,too.
[14:30:28] <Shanmugamp7>	 andalousie , xiaocao2015:  Seems no sysadmin is online, please email proof of your account to stewards@wikimedia.org  , i will help you raise a ticket
[14:30:45] <andalousie>	 thank you so much
[14:31:00] <xiaocao2015>	 Thank you
[14:36:47] <xiaocao2015>	 Email sent.
[14:43:42] <icinga-wm>	 PROBLEM - puppet last run on mw2135 is CRITICAL: CRITICAL: puppet fail
[14:45:52] <Shanmugamp7>	 xiaocao2015:okay,   i didn't receive the email yet, once i receive the email i will raise the ticket
[14:47:12] <xiaocao2015>	 But I send it
[14:47:35] <andalousie>	 thanks, I know mailing lists are sometimes slow
[14:47:40] <xiaocao2015>	 May I send it again?
[14:48:01] <andalousie>	 小曹，不用的
[14:48:21] <xiaocao2015>	 stewards@wikimedia.org Yse?
[14:48:26] <xiaocao2015>	 stewards@wikimedia.org Yes?
[14:48:39] <Shanmugamp7>	 andalousie:  xiaocao2015  yeah, it  is OTRS queue  
[14:48:49] <xiaocao2015>	 OK
[15:01:16] <Dereckson>	 !log Run deleteEqualMessages maintenance script on urwiki (T45917)
[15:01:17] <stashbot>	 T45917: Delete all redundant "MediaWiki" pages for system messages - https://phabricator.wikimedia.org/T45917
[15:01:21] <morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log, Master
[15:05:33] <icinga-wm>	 PROBLEM - puppet last run on cp2006 is CRITICAL: CRITICAL: Puppet has 1 failures
[15:10:02] <icinga-wm>	 RECOVERY - puppet last run on mw2135 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures
[15:20:51] <Dereckson>	 xiaocao2015: Shanmugamp7 andalousie: we can't do anything without approval of trust & safety department
[15:21:30] <Dereckson>	 You want to explain the situation to Jamesofur.
[15:21:53] <andalousie>	 even Email can't confirm identity?
[15:22:25] <Dereckson>	 We technically can change e-mail to allow to recover an account. But to avoid any social engineering, and protect user accounts, Trust & Safety must allow that.
[15:23:11] <andalousie>	 oh~
[15:23:53] <andalousie>	 then we should first explain the situation to  trust & safety department then create a ticket?
[15:24:05] <Dereckson>	 you can do the same in parallel
[15:24:11] <Dereckson>	 the two in parellel
[15:24:28] <andalousie>	 OK
[15:24:44] <Dereckson>	 There is a procedure offered at https://wikitech.wikimedia.org/wiki/Password_reset
[15:25:08] <Dereckson>	 For the trust & safety, part this is https://wikitech.wikimedia.org/wiki/Password_reset/Confirming_identities
[15:25:51] <andalousie>	 OK
[15:26:15] <andalousie>	 I'll tell xiaocao2015
[15:27:11] <xiaocao2015>	 ???
[15:30:12] <icinga-wm>	 PROBLEM - puppet last run on mw2076 is CRITICAL: CRITICAL: Puppet has 1 failures
[15:31:53] <icinga-wm>	 RECOVERY - puppet last run on cp2006 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures
[15:56:13] <icinga-wm>	 RECOVERY - puppet last run on mw2076 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[16:07:03] <grrrit-wm>	 (03CR) 10Alex Monk: "After applying I25699df4 this seems good" [puppet] - 10https://gerrit.wikimedia.org/r/247587 (https://phabricator.wikimedia.org/T50501) (owner: 10Alex Monk)
[16:18:21] <grrrit-wm>	 (03PS1) 10Kaldari: Adding AssessmentSaveJob (from PageAssessments) to low-priority jobs [puppet] - 10https://gerrit.wikimedia.org/r/305871 
[16:22:07] <xiaocao2015>	 I will go to sleep. Good bye.
[17:19:28] <grrrit-wm>	 (03PS1) 10ArielGlenn: enable runs of missing prereq steps for dumps at later step [puppet] - 10https://gerrit.wikimedia.org/r/305872 
[17:20:44] <grrrit-wm>	 (03CR) 10ArielGlenn: [C: 032] enable runs of missing prereq steps for dumps at later step [puppet] - 10https://gerrit.wikimedia.org/r/305872 (owner: 10ArielGlenn)
[20:28:20] <grrrit-wm>	 (03PS1) 10ArielGlenn: fix up prereq option in bash wrapper script for dumps [dumps] - 10https://gerrit.wikimedia.org/r/305893 
[20:29:45] <Platonides>	 I guess I can +2 my own patch backported by another user?
[20:30:00] <Krenair>	 to which branch?
[20:30:09] <Platonides>	 wmf/1.28.0-wmf.15
[20:30:11] <Platonides>	 oh, wait
[20:30:19] <Platonides>	 I don't think I have +2 to those branches
[20:30:20] <Krenair>	 I don't think you have the technical ability to do that
[20:30:23] <Krenair>	 yeah
[20:30:33] <grrrit-wm>	 (03CR) 10ArielGlenn: [C: 032] fix up prereq option in bash wrapper script for dumps [dumps] - 10https://gerrit.wikimedia.org/r/305893 (owner: 10ArielGlenn)
[20:30:45] <Platonides>	 nope, only +1
[20:30:59] <Krenair>	 you're not in the gerrit group for deploying to the wmf cluster so your +2 permissions don't extend to the wmf/* branches
[20:31:21] <bd808>	 a backport to a release branch typically would only be merged during a SWAT release
[20:31:33] <Platonides>	 I didn't realise that  wmf/* acl was different until you asked
[20:33:07] <Krenair>	 yeah the wmf/* branches are covered by wmf-deployment instead of mediawiki
[20:49:58] <grrrit-wm>	 (03CR) 10Mobrovac: [C: 04-1] "One minor documentation nit and we're good to go" (031 comment) [puppet] - 10https://gerrit.wikimedia.org/r/305414 (owner: 10Ppchelko)
[21:57:01] <grrrit-wm>	 (03CR) 10Jcrespo: "Sorry, I do not have a context for this, can you provide the specific use case that you run into? In particular, you mention mariadb::conf" [puppet] - 10https://gerrit.wikimedia.org/r/305737 (owner: 10Dduvall)
[22:19:19] <grrrit-wm>	 (03CR) 10Jcrespo: [C: 04-1] "Probably this is the cause for 305737 (see my comment there why I think this is a bad idea, I read that first). I do not feel that much st" [puppet] - 10https://gerrit.wikimedia.org/r/305668 (https://phabricator.wikimedia.org/T138778) (owner: 10Dduvall)
[22:20:05] <vieu>	 Hello
[22:20:09] <vieu>	 What's up
[22:21:06] <vieu>	 Kick you ass
[22:21:34] <vieu>	 fuck you
[22:21:55] <vieu>	 You retard
[22:22:29] <vieu>	 serious stuff?
[22:22:32] <grrrit-wm>	 (03CR) 10Alex Monk: "Think that just means we'd use the existing role::labs::lvm::srv" [puppet] - 10https://gerrit.wikimedia.org/r/305668 (https://phabricator.wikimedia.org/T138778) (owner: 10Dduvall)
[22:22:33] <vieu>	 my ass
[22:23:33] <vieu>	 Wikimedia Platform operations, serious stuff | Status: up | Log: https://bit.ly/wikitech | Channel logs: http://ur1.ca/edq22 | Ops Clinic Duty: robh
[22:26:21] <Krenair>	 ty. for some reason I lack chop rights in these channels
[23:39:22] <wikibugs>	 06Operations, 06Commons, 10Wikimedia-SVG-rendering, 07User-notice: SVG files larger than 10 MB cannot be thumbnailed - https://phabricator.wikimedia.org/T111815#2570791 (10Josve05a) Umm, looking at the thumbnails under the section "File history" on https://commons.wikimedia.org/wiki/File:Schwachhauser_Heer...