[02:50:07] <wikibugs>	 3Wikimedia Labs / 3Infrastructure: Replica MySQL:  Views completely missing from some wiki's - 10https://bugzilla.wikimedia.org/71041#c6 (10Sean Pringle) It means only that labsd1003 is in the same state it has been for the last year[1]: three MariaDB 5.5 instances replicating S3, S6, and S7 on ports 3306, 3...
[03:03:57] <sveta>	 tools.gpy@tools-login:~$ mysql --defaults-file="${HOME}"/replica.my.cnf -h enwiki.labsdb enwiki_p
[03:03:58] <sveta>	 ERROR 2003 (HY000): Can't connect to MySQL server on 'enwiki.labsdb' (111)
[03:04:09] <sveta>	 yes? what can I do please? :)
[03:06:28] <sveta>	 worked after a few more tries, weird
[03:06:59] <icinga-wm>	 PROBLEM - ToolLabs: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL: tools.tools-master.diskspace._var.byte_avail.value (10.00%)  
[03:23:19] <icinga-wm>	 RECOVERY - ToolLabs: Low disk space on /var on labmon1001 is OK: OK: All targets OK  
[03:27:24] <wikibugs>	 3Wikimedia Labs / 3tools: Can't send email from tools-exec-14 or -15 - 10https://bugzilla.wikimedia.org/71097 (10Brad Jorsch) 3NEW p:3Unprio s:3normal a:3Marc A. Pelletier I have a daily cronjob that pipes a status report to /usr/sbin/exim -v -odf -i. This evening, I noticed that the job was submitte...
[05:11:52] <wikibugs>	 3Wikimedia Labs / 3Infrastructure: Replica MySQL:  Views completely missing from some wiki's - 10https://bugzilla.wikimedia.org/71041#c7 (10metatron) @Sean: Thanks for info & clarification.
[06:02:52] <sveta>	 I'm reading about replicas access, it involves a "-h enwiki.labsdb" param to mysql; in the man page -h is documented as a 'hostname' param so I'd like to know how 'enwiki.labsdb' is a hostname :)
[06:03:26] <sveta>	 'host enwiki.labsdb' returns nothing and I'm not sure where else to look to understand that
[06:29:52] <ebraminio>	 hey guys, seems `sql fawiki` has serious problem, `echo "SELECT COUNT(*) FROM page;" | sql fawiki` returns "ERROR 1146 (42S02) at line 1: Table 'fawiki_p.page' doesn't exist" but dewiki or enwiki is okay?
[06:31:24] <hedonil>	 Svetlana: it's a little bit confusing sometimes. "enwiki" is used as hostname (in combination with .labsdb" as well as database name (without appendix)
[06:31:53] <hedonil>	 Svetlana: you can see all hostnames and aliases with $ less /etc/hosts
[06:32:51] <hedonil>	 Svetlana: .. and a more detailed mysql-tutorial with examples can be found here -> https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Shared_Resources/MySQL_queries
[06:35:38] <hedonil>	 Svetlana: and if you want to check your query or lookup table definitions, you can find cure here -> http://tools.wmflabs.org/tools-info/optimizer.py
[06:36:38] <hedonil>	 sveta: all "svetlana" messages for you ;)
[06:37:32] <Svetlana>	 yep
[06:38:13] <Pathoschild>	 ebraminio: I think all the wikis on s7 are broken. It's been like that all day; you can work around the issue by connecting to c1.labsdb for now.
[06:40:12] <ebraminio>	 Pathoschild: Thank you.
[09:13:04] <sveta>	 is there a list of the hostnames of wikis somewhere so that I can retrieve it programmatically without saving in a config file by hand?
[09:13:33] <sveta>	 o.o
[10:53:52] <wikibugs>	 3Tool Labs tools / 3Commons Delinker: COMMONSDELINKER-19 CommonsDelinker should not replace images if a different version of the target exists locally - 10https://bugzilla.wikimedia.org/61656#c3 (10Steinsplitter) If an image is deleted on Commons, the bot goes around and removes all files with that name from...
[10:56:22] <wikibugs>	 3Tool Labs tools / 3Commons Delinker: CommonsDelinker should not replace/remove images if a different version of the target exists locally - 10https://bugzilla.wikimedia.org/61656 (10Steinsplitter)
[11:40:43] <fluff>	 hm, on tools, i get 404 requesting special:whatlinkshere
[11:43:36] <fluff>	 think i found the problem.
[14:08:06] <wikibugs>	 3Tool Labs tools / 3Commons Delinker: CommonsDelinker should not replace/remove images if a different version of the target exists locally - 10https://bugzilla.wikimedia.org/61656#c4 (10Stefan2) (In reply to Steinsplitter from comment #3) > If an image is deleted on Commons, the bot goes around and removes a...
[16:06:56] <icinga-wm>	 PROBLEM - ToolLabs: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL: tools.tools-master.diskspace._var.byte_avail.value (10.00%)  
[16:30:21] <gifti>	 Doc_Taxon has the following problem in tools.taxonbot: webservice start returns 'already running', stopping it won't return and qstat shows nothing
[16:30:50] <gifti>	 what can they do?
[16:47:09] <Doc_Taxon>	 Hi, anybody here?
[16:48:22] <gifti>	 no, it's sunday
[16:49:33] <hedonil>	 Doc_Taxon: maybe try $ qdel -f  lighttpd-taxonbot   , then try restart again
[16:49:50] <hedonil>	 f=force
[16:50:24] <Doc_Taxon>	 lighttpd without dot before?
[16:50:33] <hedonil>	 yep
[16:51:52] <Doc_Taxon>	 The job lighttpd-taxonbot of user(s) tools.taxonbot does not exist
[16:52:18] <hedonil>	 ok, ry restart again
[16:53:02] <hedonil>	 qdel -f (sometimes) helps to make grid forget about a job
[16:53:18] <Doc_Taxon>	 okay
[16:53:39] <Doc_Taxon>	 webservice restart is still running ... and running
[16:53:58] <Doc_Taxon>	 ... and running
[16:55:35] <Doc_Taxon>	 ... running on and on
[16:56:46] <Doc_Taxon>	 hey hedonil, du sprichst doch auch deutsch, hm?
[16:57:14] <hedonil>	 Doc_Taxon: don't forget to change the socket of your fcgi configuration 
[16:57:25] <hedonil>	 Doc_Taxon: looks like copied from gifti
[16:57:47] <Doc_Taxon>	 Aber ich hab's geändert
[16:58:00] <Doc_Taxon>	 auf weblinksuche1
[16:58:53] <Doc_Taxon>	  /tmp/tcl.socket.weblinksuche1
[16:58:56] <hedonil>	 hmm - mod_fastcgi.c.978) bind failed for: unix:/tmp/tcl.socket.weblinksuche-0 Address already in use
[16:59:06] <Doc_Taxon>	 ja richtig
[16:59:18] <Doc_Taxon>	 hab ich auch schon gelesen
[16:59:41] <hedonil>	 vielleicht 'ne alte message, ersetz mal tcl mit foobar
[17:00:17] <Doc_Taxon>	 erledigt
[17:00:39] <Doc_Taxon>	 und jetzt?
[17:00:55] <hedonil>	 maaal schaun ;)
[17:01:35] <hedonil>	 logfile einträge sind alt  09:41:05: ...
[17:01:49] <Doc_Taxon>	 dann geb ich mal webservice stop ein
[17:02:12] <hedonil>	 ja, erst mal stop
[17:02:13] <gifti>	 qstat -q webgrid-lighttpd -j lighttpd-taxonbot
[17:02:25] <gifti>	 shows something ...
[17:02:45] <Doc_Taxon>	 und was?
[17:02:55] <gifti>	 maybe qdel 4192966
[17:03:06] <gifti>	 guck doch
[17:04:45] <Doc_Taxon>	 läuft gerade werbservice stop
[17:05:00] <Doc_Taxon>	 deshalb fragte ich
[17:05:46] <Doc_Taxon>	 you do not have the necessary privileges to delete the job "4192966"
[17:06:16] <gifti>	 mhhh!
[17:06:31] <gifti>	 das ist dein normaler benutzer
[17:08:15] <Doc_Taxon>	 ist das nicht uid 4802
[17:09:26] <gifti>	 ?
[17:10:00] <gifti>	 problem geloest ...
[17:10:22] <Doc_Taxon>	 gifti : was is gelöst
[17:10:54] <gifti>	 jetzt musst du nur noch als taxonbot den job loeschen
[17:11:08] <Doc_Taxon>	 ws hast denn gemacht
[17:11:17] <gifti>	 ich nix
[17:11:22] <Doc_Taxon>	 ich auch nix
[17:11:32] <gifti>	 ich hab nur das problem erkannt
[17:11:38] <Doc_Taxon>	 erzähl
[17:11:42] <gifti>	 huschhusch!
[17:12:18] <gifti>	 mach bitte als taxonbot qstat, da muesste der job auftauchen
[17:12:41] <Doc_Taxon>	 nix wird return-ed
[17:12:41] <gifti>	 (also nicht als tools.taxonbot)
[17:12:54] <Doc_Taxon>	 ach so
[17:13:00] <gifti>	 mannmann
[17:13:35] <Doc_Taxon>	 deleted
[19:27:05] <danilo>	 why there is no tables in metawiki_p db replica? 'SHOW TABLES' returns 'empty set'
[19:29:15] <danilo>	 I have a bot that is broken becouse there is no tables in metawiki_p
[19:33:16] <danilo>	 someone? Coren? metawiki_p is down^
[19:56:03] <Betacommand>	 danilo: file a bug and send an email to the mailing list, since its sunday most people are not around
[19:56:09] <Pathoschild>	 danilo: it's been broken for a couple of days; haven't heard an ETA on a fix. You can use c1.labsdb as a workaround for now.
[19:56:17] <Pathoschild>	 It affects all or most of s6 and s7.
[19:57:11] <danilo>	 ok, thanks
[19:57:54] <wikibugs>	 3Wikimedia Labs / 3Infrastructure: Replica MySQL:  Views completely missing from some wiki's - 10https://bugzilla.wikimedia.org/71041#c8 (10Jesse (Pathoschild)) All s6 and s7 wikis I checked are affected.
[19:57:59] <Pathoschild>	 Betacommand: ^ that's the relevant bug.
[20:49:51] <icinga-wm>	 PROBLEM - ToolLabs: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL: tools.tools-master.diskspace._var.byte_avail.value (10.00%)  
[21:02:07] <icinga-wm>	 PROBLEM - ToolLabs: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL: tools.tools-master.diskspace._var.byte_avail.value (10.00%)  
[21:29:17] <icinga-wm>	 PROBLEM - ToolLabs: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL: tools.tools-master.diskspace._var.byte_avail.value (11.11%)  
[21:41:17] <icinga-wm>	 PROBLEM - ToolLabs: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL: tools.tools-master.diskspace._var.byte_avail.value (11.11%)  
[22:05:48] <icinga-wm>	 PROBLEM - ToolLabs: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL: tools.tools-master.diskspace._var.byte_avail.value (11.11%)  
[22:32:17] <icinga-wm>	 PROBLEM - ToolLabs: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL: tools.tools-master.diskspace._var.byte_avail.value (11.11%)  
[22:45:28] <icinga-wm>	 PROBLEM - ToolLabs: Low disk space on /var on labmon1001 is CRITICAL: CRITICAL: tools.tools-master.diskspace._var.byte_avail.value (11.11%)