[00:31:29] <bd808>	 !log tools.squirrelnestbot Stopped grid job running tools.unblockbot/unblockbot.sh. Script is in an infinite crash loop because it does not handle https properly.
[00:31:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.squirrelnestbot/SAL
[00:31:57] <bd808>	 !log tools.squirrelnestbot Also, why is tools.squirrelnestbot running a job for tools.unblockbot?
[00:31:58] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.squirrelnestbot/SAL
[00:39:29] <bd808>	 !log tools Compressed /var/log/account/pacct.0 ahead of rotation schedule to free some space on the root partition
[00:39:32] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[00:39:53] <bd808>	 !log tools Ugh. Prior SAL message was about tools-sgeexec-0940
[00:39:55] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[07:46:14] <RF1dle>	 !log tools.zppixbot add notice for T254046 to wiki index about
[07:46:16] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL
[07:46:17] <stashbot>	 T254046: Clean ZppixBot's filesystem up - https://phabricator.wikimedia.org/T254046
[08:17:03] <RhinosF1>	 !log upload starter-new.sh and switched sopelbot.yaml foor T254046
[08:17:04] <stashbot>	 RhinosF1: Unknown project "upload"
[08:17:04] <stashbot>	 T254046: Clean ZppixBot's filesystem up - https://phabricator.wikimedia.org/T254046
[08:17:12] <RhinosF1>	 !log tools.zppixbot upload starter-new.sh and switched sopelbot.yaml foor T254046
[08:17:14] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL
[08:51:20] * RhinosF1 sorry if he gets spammy
[09:01:31] <RhinosF1>	 !log tools.zppixbot deleted sopel.bot deployment and stopped webservice - START T254046
[09:01:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL
[09:01:34] <stashbot>	 T254046: Clean ZppixBot's filesystem up - https://phabricator.wikimedia.org/T254046
[09:01:40] <RhinosF1>	 Reception123: we begin
[09:02:00] <Reception123>	 k
[09:26:27] <RhinosF1>	 !log tools.zppixbot dropped channels noot watched from seen database
[09:26:30] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL
[09:27:07] <RhinosF1>	 !log tools.zppixbot cleaned up channel_values
[09:27:08] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL
[09:31:16] <RhinosF1>	 !log tools.zppixbot cleaned known_users
[09:31:17] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL
[09:34:36] <RhinosF1>	 !log tar+gzip sopel logs, empty non sopel rubbish
[09:34:37] <stashbot>	 RhinosF1: Unknown project "tar+gzip"
[09:34:48] <RhinosF1>	 !log tools.zppixbot tar+gzip sopel logs, empty non sopel rubbish
[09:34:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL
[09:45:00] <RhinosF1>	 !log tools.zppixbot empty pycache and switch to new deployment config
[09:45:02] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL
[09:55:46] <RhinosF1>	 !log tools.zppixbot revert sitenotice for T254046
[09:55:48] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL
[09:55:48] <stashbot>	 T254046: Clean ZppixBot's filesystem up - https://phabricator.wikimedia.org/T254046
[09:56:57] <RhinosF1>	 !log tools.zppixbot webservice --backend=kubernetes php7.2 start --canonical for T254046
[09:56:59] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL
[10:00:40] <RhinosF1>	 !log sopel.bot re-created for T254046
[10:00:41] <stashbot>	 RhinosF1: Unknown project "sopel.bot"
[10:00:50] <RhinosF1>	 !log tools.zppixbot sopel.bot re-created for T254046
[10:00:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL
[10:00:53] <stashbot>	 T254046: Clean ZppixBot's filesystem up - https://phabricator.wikimedia.org/T254046
[10:08:36] <RhinosF1>	 !log tools.zppixbot END MAINT -- T254046
[10:08:38] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL
[10:08:39] <stashbot>	 T254046: Clean ZppixBot's filesystem up - https://phabricator.wikimedia.org/T254046
[10:09:58] * RhinosF1 done
[10:19:59] <RhinosF1>	 can anyone tell me how big the zppixbot instance now is?
[10:26:42] <hauskatze>	 !help Can an admin please sudo to the `meta` toolforge tool and run webservice restart?
[10:26:42] <wm-bot>	 If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-kanban
[10:27:02] <hauskatze>	 I am a maintainer but currently not able to ssh to the instance
[10:27:29] <Reedy>	 `become meta` just feels odd
[10:27:34] <Reedy>	 become meta relevant
[10:27:58] <Reedy>	 !log meta webservice restart
[10:27:59] <stashbot>	 Reedy: Unknown project "meta"
[10:27:59] <stashbot>	 Reedy: Did you mean to say "tools.meta" instead?
[10:28:02] <Reedy>	 !log tools.meta webservice restart
[10:28:03] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.meta/SAL
[10:28:04] <hauskatze>	 Just to clarify I cannot ssh because I don't have my ssh keys with me right now
[10:28:29] <hauskatze>	 thanks Reedy 
[10:28:37] <hauskatze>	 that fixed the issue with magicredirect
[10:47:44] <arturo>	 thanks Reedy 
[11:08:56] <hauskatze>	 morning arturo
[11:11:39] * arturo waves to hauskatze 
[11:12:04] * RhinosF1 waves
[11:12:54] * Majavah waves to all of you
[11:14:14] <RhinosF1>	 Hi Majavah
[11:15:24] <Nikerabbit>	 Hi Majavah
[11:15:33] <Majavah>	 o/
[11:18:22] * RhinosF1 spent tidying up his tool instance
[12:40:29] <RhinosF1>	 !log tools.zppixbot deleting 54 orphan accounts that don't have any attached values
[12:40:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL
[13:30:13] <RhinosF1>	 !log tools.zppixbot ran select nick_id from nick_values where value < 1546300800 and key = "seen_timestamp" on default.db to generate P11346 for T254154
[13:30:16] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL
[13:30:16] <stashbot>	 T254154: Establish data retention guidelines - https://phabricator.wikimedia.org/T254154
[18:51:28] <bstorm_>	 !log clouddb-services restarting mariadb on clouddb1002 T253738
[18:51:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Clouddb-services/SAL
[18:51:31] <stashbot>	 T253738: ToolsDB: master crashed, replica having consistency issues - https://phabricator.wikimedia.org/T253738
[19:25:52] <jem>	 Hi everyone, after the recent high replag episode, in detabase replicas it seems there is some data missing, at least for eswiki.. any help about that?
[19:27:25] <bd808>	 jem: you are going to need to be a lot more specific. Are you reporting corrupt data or just replication lag?
[19:28:07] <AntiComposite>	 an example query and the server you ran it on would also be helpful
[19:28:16] <bd808>	 https://replag.toolforge.org/ looks like it thinks things are mostly caught up
[19:28:34] <jem>	 bd808: Missing data
[19:28:53] <jem>	 Yes, I have been checking and today it all seemed normal, but then...
[19:29:23] <jem>	 I can paste the query if it's ok
[19:29:42] <jem>	 One moment
[19:30:01] <bd808>	 use a pastebin please. https://phabricator.wikimedia.org/paste/ is an easy one
[19:30:11] <jem>	 Ok
[19:34:07] <jem>	 bd808 and AntiComposite https://phabricator.wikimedia.org/P11348
[19:34:13] <jem>	 And thanks in advance
[19:35:14] <jem>	 The page was moved but anyway the edits should be in one namespace or the other
[19:36:39] <jem>	 (Added, just in case: the database is eswiki)
[19:47:55] <jem>	 I guess you are looking into it...
[19:48:21] <AntiComposite>	 I can replicate it, https://quarry.wmflabs.org/query/45487 vs https://quarry.wmflabs.org/query/45488
[19:49:25] <jem>	 Ok, much clearer that way
[19:57:40] <AntiComposite>	 https://quarry.wmflabs.org/query/45489 should show every revision from 2020-05-30 10:00:00 to 2020-05-30 19:59:59 without a corresponding recentchanges entry.
[19:58:15] <AntiComposite>	 there's 10,685 rows there, or 1000 mismatches per hour
[20:09:06] <jem>	 Ok then, so what should be done next? Should I create a task, or maybe is there already one?
[20:11:12] <AntiComposite>	 hmm...looking at 2020-05-10 also has 11,569 missing https://quarry.wmflabs.org/query/45493
[20:17:37] <AntiComposite>	 it's not just eswiki too, https://quarry.wmflabs.org/query/45492 https://quarry.wmflabs.org/query/45491
[20:17:45] <AntiComposite>	 I don't see any reason not to file a phab task
[20:18:42] <jem>	 Now I'm surprised that nobody noted this if it affects more projects
[20:24:49] <jem>	 bd808 ?
[20:28:08] <bd808>	 jem: a phabricator task is always a good idea.
[20:28:23] * bd808 is in other meetings right now
[20:28:34] <jem>	 Ok then
[20:29:18] <jem>	 I'll leave it open about the tables and projects involved
[20:41:45] <jem>	 https://phabricator.wikimedia.org/T254193
[20:42:08] <jem>	 Thanks, AntiComposite and please check if it's ok
[20:42:39] <jem>	 I think it's ok because there will be more bots and statistics affected
[20:42:53] <jem>	 it's ok to mark it high priority*
[20:44:40] <AntiComposite>	 unless you're planning to work on it yourself, no
[20:45:09] <AntiComposite>	 I've set it back to needs triage and tagged Data-Services, which handles the wiki replicas
[20:45:17] <jem>	 Ok, thanks
[20:45:42] <AntiComposite>	 (see https://www.mediawiki.org/wiki/Phabricator/Project_management#Setting_task_priorities for why)
[20:47:51] <jem>	 Very reasonable, sorry for my ignorance, now I know
[20:50:54] <jem>	 Getting afk now, thanks to bd808 also, let's hope it's solved soon
[21:10:29] <RhinosF1>	 !log tools.zppixbot add *.db to sopel/.gitignore
[21:10:32] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL
[21:11:43] <RhinosF1>	 !log tools.zppixbot-test add *.db to sopel/.gitignore
[21:11:44] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL
[21:13:37] <RhinosF1>	 !log tools.zppixbot-test .sopel/welcome.db>CREATE TABLE chanops ( id INTEGER PRIMARY key autoincrement, nick VARCHAR(20), chan VARCHAR(20) ); for T254166
[21:13:39] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL
[21:13:39] <stashbot>	 T254166: Create channelmgnt.db and tables - https://phabricator.wikimedia.org/T254166
[21:13:40] <RhinosF1>	 Earwig: 
[21:13:47] <RhinosF1>	 Texas: ^
[21:14:13] <RhinosF1>	 Texas: Is that schema good? https://usercontent.irccloud-cdn.com/file/aqGJdKP0/Screenshot%202020-06-01%20at%2022.13.59.png
[21:14:39] <Texas>	 RhinosF1: Yes that looks fine to me
[21:15:18] <RhinosF1>	 !log tools.zppixbot .sopel/welcome.db>CREATE TABLE chanops ( id INTEGER PRIMARY key autoincrement, nick VARCHAR(20), chan VARCHAR(20) ); for T254166
[21:15:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot/SAL
[21:15:36] <RhinosF1>	 Texas: databases created, have you got the cleanup script?
[21:18:54] <RhinosF1>	 Texas: should they be NOT NULL ?
[21:19:48] <Texas>	 RhinosF1: We can make the bot enforce that
[21:20:25] <RhinosF1>	 Texas: the bot should follow it but it seems a bad idea to not have the table set?
[21:20:41] <Texas>	 I guess you can
[21:23:27] <RhinosF1>	 Texas: it only works for id so {{not done}}
[21:23:43] <Texas>	 oh okay
[21:25:37] <RhinosF1>	 Texas: anything for me to test while logged in?
[21:26:13] <Texas>	 RhinosF1: For now no. I am only half awake right now.
[21:26:22] * Texas did not get much sleep last night
[21:26:41] <RhinosF1>	 Texas: ah k, I guess we just got the data purge to work on
[21:27:25] <Texas>	 yep
[21:28:02] <RhinosF1>	 Texas: do you know what tables were touching or shall I get proper deletes now?
[21:28:16] <Texas>	 ??
[21:28:36] <RhinosF1>	 Texas: do you know what we run to delete the data now in raw sql?
[21:28:49] <Texas>	 no
[21:29:12] <RhinosF1>	 Texas: One sec
[21:29:29] * RhinosF1 is going to droop ek_mobile
[21:29:42] <RhinosF1>	 https://www.irccloud.com/pastebin/2BfVYNV2/sqltorestoreekmobile
[21:33:06] <RhinosF1>	 Texas: Replace is 34; with the number from the file it reads https://www.irccloud.com/pastebin/hfjuh6gG/deleteusersql
[21:33:29] <RhinosF1>	 !log tools.zppixbot-test run https://www.irccloud.com/pastebin/hfjuh6gG/deleteusersql
[21:33:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL
[21:35:05] <Texas>	 RhinosF1: That looks okay to me
[21:35:11] <Texas>	 did it work?
[21:35:16] <RhinosF1>	 Texas: yep
[21:35:25] <Texas>	 yayy
[21:35:32] <Texas>	 so we can make a script for it
[21:36:23] <RhinosF1>	 Texas: yep
[21:36:54] <RhinosF1>	 !log tools.zppixbot. ran https://www.irccloud.com/pastebin/2BfVYNV2/sqltorestoreekmobile to add ek back
[21:36:54] <stashbot>	 RhinosF1: Unknown project "tools.zppixbot."
[21:37:06] <RhinosF1>	 !log tools.zppixbot-test ran https://www.irccloud.com/pastebin/2BfVYNV2/sqltorestoreekmobile to add ek back
[21:37:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.zppixbot-test/SAL
[21:37:17] <Texas>	 yayyy
[21:37:37] <RhinosF1>	 Texas: as long as we can open a connection to the database, run that for each user and commit it
[21:37:55] <Texas>	 yeah
[21:38:22] <Texas>	 so we can do like for every line in the file, run your deleteusersql
[21:38:30] <RhinosF1>	 Texas: yep
[21:38:35] <Texas>	 RhinosF1: Why are we bothering -cloud with this though?
[21:39:22] <RhinosF1>	 Texas: so I could log the sqls
[21:39:31] <Texas>	 oh
[23:51:57] <bstorm_>	 !log tools refreshed certs for the custom webhook controllers on the k8s cluster T250874
[23:52:00] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[23:52:01] <stashbot>	 T250874: Refresh external certs for the toolforge k8s cluster after the upgrade - https://phabricator.wikimedia.org/T250874