[00:07:53] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 11 Mar 2014 08:47:37 PM UTC [00:18:14] (03PS1) 10Dzahn: lint maintenance.pp - part 1 - modes and titles [operations/puppet] - 10https://gerrit.wikimedia.org/r/118790 [00:18:16] (03PS1) 10Dzahn: lint maintenance.pp-part 2-double quoted strings [operations/puppet] - 10https://gerrit.wikimedia.org/r/118791 [00:35:35] (03PS1) 10Dzahn: lint admins.pp-pt01-replace tabs and nothing else [operations/puppet] - 10https://gerrit.wikimedia.org/r/118794 [00:49:55] (03CR) 10Dzahn: [C: 04-1] "this should stay, but check again when fenari is dead :p" [operations/puppet] - 10https://gerrit.wikimedia.org/r/96424 (owner: 10Dzahn) [00:55:38] fenari will live forever in our hearts. [00:55:51] but not as much loved as eiximenis [01:08:45] <^d> robh: Can we have a fenari2? [01:32:43] doesnt match naming scheme, name denied! [01:36:35] <^d> robh: fenari1001? [02:01:50] !log LocalisationUpdate failed: mwversionsinuse returned empty list [02:19:20] !log LocalisationUpdate failed: mwversionsinuse returned empty list [02:36:43] morebots is dead [03:08:53] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 11 Mar 2014 08:47:37 PM UTC [03:20:03] PROBLEM - MySQL Slave Delay on db1007 is CRITICAL: CRIT replication delay 301 seconds [03:20:23] PROBLEM - MySQL Replication Heartbeat on db1007 is CRITICAL: CRIT replication delay 302 seconds [03:22:03] PROBLEM - MySQL Slave Delay on db1007 is CRITICAL: CRIT replication delay 306 seconds [03:22:23] PROBLEM - MySQL Replication Heartbeat on db1007 is CRITICAL: CRIT replication delay 307 seconds [03:56:46] (03PS1) 10Tim Landscheidt: WIP: labsdeprepo: Allow more than one local repository [operations/puppet] - 10https://gerrit.wikimedia.org/r/118796 [04:10:03] RECOVERY - MySQL Slave Delay on db1007 is OK: OK replication delay 85 seconds [04:10:23] RECOVERY - MySQL Replication Heartbeat on db1007 is OK: OK replication delay 45 seconds [06:09:53] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 11 Mar 2014 08:47:37 PM UTC [08:37:13] RECOVERY - check_job_queue on terbium is OK: JOBQUEUE OK - all job queues below 200,000 [08:40:13] PROBLEM - check_job_queue on terbium is CRITICAL: JOBQUEUE CRITICAL - the following wikis have more than 199,999 jobs: , Total (201714) [09:10:53] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 11 Mar 2014 08:47:37 PM UTC [09:26:58] (03Abandoned) 10Odder: Account creation throttle for ptwiki outreach event [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/118364 (owner: 10Odder) [09:43:18] (03PS4) 10Nemo bis: Use $wgTranslatePageTranslationULS [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/115879 [09:46:51] I wonder if there's a way to make that move forward. With N. being part-time, Translate config seems abandoned [11:21:14] (03PS8) 10Nemo bis: Add cron job to run characterEditStats.php on multilingual wikis weekly [operations/puppet] - 10https://gerrit.wikimedia.org/r/117250 [11:21:36] (03CR) 10Nemo bis: "rebased" [operations/puppet] - 10https://gerrit.wikimedia.org/r/117250 (owner: 10Nemo bis) [12:11:53] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 11 Mar 2014 08:47:37 PM UTC [12:28:30] (03Restored) 10Ori.livneh: Enable web fonts by default on Hebrew Wikisource [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/115153 (owner: 10Odder) [12:36:26] hey Nemo_bis, do you know any hewikisource folks that are on irc at these hours? [12:37:30] no [12:38:06] how horrendous would it be if i granted myself the staff bit and simple showed what i meant by importing the font in site.css? [12:38:27] any admin / interfaceeditor could still remove it if i horribly misunderstood something [12:38:37] *simply [12:39:40] oh hey, Nikerabbit is here [12:40:43] Nikerabbit, for https://gerrit.wikimedia.org/r/#/c/115153/, rather than enabling web-fonts, which would mean that any page on the wiki could potentially load any number of fonts, [12:42:11] what about adding this to Common.css: [12:42:13] @import url(http://fonts.googleapis.com/earlyaccess/alefhebrew.css); [12:43:05] i don't have any love for google; we could load it from our servers obviously. it just spared me the trouble of having to figure out what the optimal way is to arrange font file types in 2014 [12:43:40] odder is pretty miffed at me, but i'm really not sure why [12:45:15] I'll just post on gerrit [12:48:26] (03CR) 10Ori.livneh: "To clarify my earlier reservation, I'll explain that enabling webfonts means any page on the wiki can in potential load any number of font" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/115153 (owner: 10Odder) [13:17:32] (03PS9) 10Ori.livneh: Add cron job to run characterEditStats.php on multilingual wikis weekly [operations/puppet] - 10https://gerrit.wikimedia.org/r/117250 (owner: 10Nemo bis) [13:19:45] (03CR) 10Ori.livneh: "PS9:" [operations/puppet] - 10https://gerrit.wikimedia.org/r/117250 (owner: 10Nemo bis) [13:20:53] ori: such a rule is clearly forbidden, we can't load resources from Google [13:21:36] i used google most because i take it that they have computed the optimal ordering of fonts [13:21:41] If you're interested, a precedent has been Roan temporarily whitelisting a font extension for upload on a wiki, uploading the font as normal file and then embedding it [13:21:44] but we host all these files locally [13:21:54] so I can rewrite the file to reference local URLs [13:22:05] i was first trying to understand if i was completely being crazy or what [13:22:31] You can try. I suggest the approach above that was used by Roan [13:22:41] it doesn't need to be uploaded, because the files are part of ULS, which has stable URLs [13:22:59] Ok, you can just add the script there in MediaWiki namespace then [13:23:27] If the community doesn't like it they'll delete it and let us know [13:23:56] sounds like a plan, let me cook up the css [13:24:35] (just make clear in your edit summary that it's not an imposition but just a gift you make them, that they can use or not; otherwise they'll just shut up and we'll never know) [13:30:54] do you have the right to interface edit there? [13:31:00] i don't like tweaking with my privs [13:31:39] oh well, what the hell [13:34:16] (03CR) 10Nemo bis: "Thanks! The file in public_html is not necessary, publishing on noc.wikimedia.org was the original idea but the Meta-Wiki page is better. " [operations/puppet] - 10https://gerrit.wikimedia.org/r/117250 (owner: 10Nemo bis) [13:34:33] !log promoted self (User:Ori.livneh) to sysop on hewikisource to try and install a solution for 60939; will revoke shortly. [13:35:59] CodeEditor doesn't do RTL properly *sigh* [13:38:49] Ah sorry, was having coffee [13:41:31] Next time you can/should ask one of the 58 persons with globalgroupmembership / userrights-interwiki e.g. http://tools.wmflabs.org/pathoschild-contrib/globalgroups/index.php#sysadmin [13:54:42] where is morebots? [13:55:41] https://wikitech.wikimedia.org/wiki/Server_Admin_Log misses the last !log actions [14:02:58] i hate fonts [14:03:25] all fonts, in all scripts [14:36:02] (03CR) 10Hoo man: [C: 031] "Nice start! This should probably be merged soon to avoid endless rebases." [operations/puppet] - 10https://gerrit.wikimedia.org/r/118794 (owner: 10Dzahn) [14:39:05] (03CR) 10Ori.livneh: "I tried to implement this in ^ Nikerabbit, Nemo_bis [14:52:06] !log stripped temporary sysop and bureaucrat privs from my account (Ori.livneh) on hewikisource [14:52:32] !talking to a wall since morebots isn't here [14:54:41] !log restarted morebots; too lazy to check logs to try and ascertain why and where it went [14:54:50] Logged the message, Master [14:55:37] !log promoted self (User:Ori.livneh) to sysop/'creat on hewikisource to try and install a solution for 60939; then revoked both some 20 minutes later. [14:55:46] Logged the message, Master [14:57:10] 8 am! new record. good night. [14:58:22] heh... good night, ori :D [15:12:53] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 11 Mar 2014 08:47:37 PM UTC [15:15:56] (03CR) 10Hoo man: [C: 04-1] "I want to have this double checked, before anything happens. DO NOT DEPLOY YET!" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/118654 (owner: 10Gerrit Patch Uploader) [15:20:19] ori: what happens if we rename a font? [15:23:07] ori: I guess the way you did it works, but I think we should have more control over the fonts and css so we can change things like font names or the exact css being used [15:51:08] Nikerabbit: you could combine the dynamic approach with a single font, perhaps [15:52:29] so, you still use ULS library calls to resolve a lang name to a font resource, but you restrict it to $wgContentLanguage or something [15:53:53] or if devise some other way of inferring from the environment how many fonts (if any) you may load by default [15:53:59] just ideas [15:55:42] if the hewikisource community wants automatic font loading, and they understand what that entails, and it provides you with an additional production wiki to test changes on, i'd be supporting of re-enabling it [15:56:22] i'm still not persuaded it's necessary, but at the same time i don't really grok the complexity entailed in doing this properly [16:01:19] (03CR) 10Ori.livneh: " ori: what happens if we rename a font?" [operations/mediawiki-config] - 10https://gerrit.wikimedia.org/r/115153 (owner: 10Odder) [16:35:23] PROBLEM - Disk space on labstore1001 is CRITICAL: DISK CRITICAL - free space: /exp/dumps 379005 MB (3% inode=99%): [18:13:53] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 11 Mar 2014 08:47:37 PM UTC [19:04:20] PROBLEM - Puppet freshness on mw1212 is CRITICAL: Last successful Puppet run was Sat 15 Mar 2014 07:00:26 PM UTC [19:06:20] PROBLEM - Puppet freshness on mw1212 is CRITICAL: Last successful Puppet run was Sat 15 Mar 2014 07:00:26 PM UTC [19:08:20] PROBLEM - Puppet freshness on mw1212 is CRITICAL: Last successful Puppet run was Sat 15 Mar 2014 07:00:26 PM UTC [19:10:20] PROBLEM - Puppet freshness on mw1212 is CRITICAL: Last successful Puppet run was Sat 15 Mar 2014 07:00:26 PM UTC [19:12:20] PROBLEM - Puppet freshness on mw1212 is CRITICAL: Last successful Puppet run was Sat 15 Mar 2014 07:00:26 PM UTC [19:14:20] PROBLEM - Puppet freshness on mw1212 is CRITICAL: Last successful Puppet run was Sat 15 Mar 2014 07:00:26 PM UTC [19:16:20] PROBLEM - Puppet freshness on mw1212 is CRITICAL: Last successful Puppet run was Sat 15 Mar 2014 07:00:26 PM UTC [19:18:20] PROBLEM - Puppet freshness on mw1212 is CRITICAL: Last successful Puppet run was Sat 15 Mar 2014 07:00:26 PM UTC [19:20:21] PROBLEM - Puppet freshness on mw1212 is CRITICAL: Last successful Puppet run was Sat 15 Mar 2014 07:00:26 PM UTC [19:22:20] PROBLEM - Puppet freshness on mw1212 is CRITICAL: Last successful Puppet run was Sat 15 Mar 2014 07:00:26 PM UTC [19:24:20] PROBLEM - Puppet freshness on mw1212 is CRITICAL: Last successful Puppet run was Sat 15 Mar 2014 07:00:26 PM UTC [19:26:20] PROBLEM - Puppet freshness on mw1212 is CRITICAL: Last successful Puppet run was Sat 15 Mar 2014 07:00:26 PM UTC [19:28:20] PROBLEM - Puppet freshness on mw1212 is CRITICAL: Last successful Puppet run was Sat 15 Mar 2014 07:00:26 PM UTC [19:30:20] PROBLEM - Puppet freshness on mw1212 is CRITICAL: Last successful Puppet run was Sat 15 Mar 2014 07:00:26 PM UTC [19:30:40] RECOVERY - Puppet freshness on mw1212 is OK: puppet ran at Sat Mar 15 19:30:33 UTC 2014 [19:32:20] PROBLEM - Puppet freshness on mw1212 is CRITICAL: Last successful Puppet run was Sat 15 Mar 2014 07:30:33 PM UTC [20:00:10] RECOVERY - Puppet freshness on mw1212 is OK: puppet ran at Sat Mar 15 20:00:03 UTC 2014 [20:11:00] PROBLEM - MySQL Processlist on db1021 is CRITICAL: CRIT 0 unauthenticated, 0 locked, 1 copy to table, 526 statistics [20:12:00] RECOVERY - MySQL Processlist on db1021 is OK: OK 0 unauthenticated, 0 locked, 0 copy to table, 6 statistics [21:02:13] “Sorry! This site is experiencing technical difficulties.” [21:02:16] On Wikidata. [21:02:27] Oh, access again [21:02:41] No more error message than that? [21:02:52] Too many connections or something. :/ [21:03:07] mmm, db errors then possibly [21:04:36] (03CR) 10Matanya: [C: 031] lint admins.pp-pt01-replace tabs and nothing else [operations/puppet] - 10https://gerrit.wikimedia.org/r/118794 (owner: 10Dzahn) [21:05:03] Yup, a huge spam of them a few minutes ago [21:07:33] reedy@fenari:/home/wikipedia/operations-software/dbtree$ git remote -v [21:07:33] origin ssh://gerrit.wikimedia.org:29418/operations/software.git (fetch) [21:07:39] Where the hell is the dbtree repo? [21:10:00] PROBLEM - HTTP on fenari is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:14:40] PROBLEM - Puppet freshness on labstore1001 is CRITICAL: Last successful Puppet run was Tue 11 Mar 2014 08:47:37 PM UTC [21:14:50] RECOVERY - HTTP on fenari is OK: HTTP OK: HTTP/1.1 200 OK - 4775 bytes in 0.072 second response time [21:14:50] (03CR) 10Matanya: [C: 031] "no so friendly commit message, can you change it please?" [operations/puppet] - 10https://gerrit.wikimedia.org/r/118791 (owner: 10Dzahn) [21:17:02] (03CR) 10Matanya: [C: 031] "not a big fan of small increment lints, i prefer in batch, but oh well. please make the commit message nicer." [operations/puppet] - 10https://gerrit.wikimedia.org/r/118790 (owner: 10Dzahn) [21:17:03] just massively out of date locally [21:19:43] (03PS1) 10Reedy: Remove PMTPA remnants [operations/software] - 10https://gerrit.wikimedia.org/r/118939 [21:36:52] (03PS1) 10Reedy: Update to latest jquery point releases [operations/software] - 10https://gerrit.wikimedia.org/r/118940 [21:37:38] (03PS2) 10Reedy: Update to latest jquery point releases [operations/software] - 10https://gerrit.wikimedia.org/r/118940 [21:44:02] Reedy: How is the migration going? [22:48:10] RECOVERY - Disk space on virt11 is OK: DISK OK [23:48:01] andrewbogott_afk: do you know what broke the w: interwiki link on wikitech? the interwiki map there needs a special process, it was set up by apergos https://wikitech.wikimedia.org/wiki/Geolocation