[00:13:16] PROBLEM - pdfrender on scb1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:16:16] RECOVERY - pdfrender on scb1004 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 5.991 second response time [00:19:16] PROBLEM - pdfrender on scb1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:23:16] RECOVERY - pdfrender on scb1004 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 5.261 second response time [00:26:26] PROBLEM - pdfrender on scb1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:27:26] RECOVERY - pdfrender on scb1004 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 5.941 second response time [00:30:27] PROBLEM - pdfrender on scb1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:34:36] RECOVERY - pdfrender on scb1004 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 8.674 second response time [00:37:36] PROBLEM - pdfrender on scb1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:00:56] RECOVERY - pdfrender on scb1004 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 3.338 second response time [01:04:06] PROBLEM - pdfrender on scb1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:08:06] RECOVERY - pdfrender on scb1004 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 8.747 second response time [01:11:06] PROBLEM - pdfrender on scb1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:17:56] PROBLEM - pdfrender on scb1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:20:16] RECOVERY - pdfrender on scb1004 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 9.070 second response time [01:23:16] PROBLEM - pdfrender on scb1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:30:06] RECOVERY - pdfrender on scb1003 is OK: HTTP OK: HTTP/1.1 200 OK - 276 bytes in 7.817 second response time [01:33:16] PROBLEM - pdfrender on scb1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:35:06] RECOVERY - pdfrender on scb1003 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 0.307 second response time [01:39:17] PROBLEM - pdfrender on scb1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:45:36] RECOVERY - pdfrender on scb1004 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 5.979 second response time [01:48:46] PROBLEM - pdfrender on scb1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:53:27] RECOVERY - pdfrender on scb1003 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 8.949 second response time [01:56:36] PROBLEM - pdfrender on scb1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:02:26] RECOVERY - pdfrender on scb1003 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 0.016 second response time [02:05:36] PROBLEM - pdfrender on scb1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:07:37] RECOVERY - pdfrender on scb1003 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 8.813 second response time [02:13:47] PROBLEM - pdfrender on scb1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:14:46] RECOVERY - pdfrender on scb1003 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 7.156 second response time [02:19:56] PROBLEM - pdfrender on scb1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:23:57] RECOVERY - pdfrender on scb1003 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 6.674 second response time [02:27:06] PROBLEM - pdfrender on scb1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:29:16] RECOVERY - pdfrender on scb1004 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 8.925 second response time [02:31:57] RECOVERY - pdfrender on scb1003 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 1.001 second response time [02:32:17] PROBLEM - pdfrender on scb1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:35:16] PROBLEM - pdfrender on scb1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:44:07] RECOVERY - pdfrender on scb1003 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 1.750 second response time [02:47:17] PROBLEM - pdfrender on scb1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:02:56] PROBLEM - PyBal backends health check on lvs1006 is CRITICAL: PYBAL CRITICAL - Bad Response from pybal: 500 Internal Server Error [03:03:26] PROBLEM - pdfrender on scb1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:03:27] PROBLEM - PyBal backends health check on lvs1003 is CRITICAL: PYBAL CRITICAL - Bad Response from pybal: 500 Internal Server Error [03:03:36] PROBLEM - PyBal backends health check on lvs1010 is CRITICAL: PYBAL CRITICAL - Bad Response from pybal: 500 Internal Server Error [03:04:29] PROBLEM - LVS HTTP IPv4 on pdfrender.svc.eqiad.wmnet is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:06:19] RECOVERY - LVS HTTP IPv4 on pdfrender.svc.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 0.002 second response time [03:10:26] RECOVERY - pdfrender on scb1002 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 7.829 second response time [03:12:36] RECOVERY - pdfrender on scb1003 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 1.019 second response time [03:12:56] RECOVERY - PyBal backends health check on lvs1006 is OK: PYBAL OK - All pools are healthy [03:13:36] PROBLEM - pdfrender on scb1002 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:15:39] PROBLEM - LVS HTTP IPv4 on pdfrender.svc.eqiad.wmnet is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:15:46] PROBLEM - pdfrender on scb1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:15:56] PROBLEM - PyBal backends health check on lvs1006 is CRITICAL: PYBAL CRITICAL - Bad Response from pybal: 500 Internal Server Error [03:16:46] PROBLEM - pdfrender on scb1001 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:18:56] RECOVERY - PyBal backends health check on lvs1006 is OK: PYBAL OK - All pools are healthy [03:21:46] RECOVERY - pdfrender on scb1001 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 0.006 second response time [03:21:56] PROBLEM - PyBal backends health check on lvs1006 is CRITICAL: PYBAL CRITICAL - Bad Response from pybal: 500 Internal Server Error [03:23:29] volans: hey, are you around? Things don't seem too happy here ^ [03:24:39] RECOVERY - LVS HTTP IPv4 on pdfrender.svc.eqiad.wmnet is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 0.003 second response time [03:25:37] RECOVERY - PyBal backends health check on lvs1003 is OK: PYBAL OK - All pools are healthy [03:28:46] PROBLEM - PyBal backends health check on lvs1003 is CRITICAL: PYBAL CRITICAL - Bad Response from pybal: 500 Internal Server Error [03:28:58] !log restart pdfrender on scb1001 following icinga page [03:29:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:31:26] PROBLEM - MariaDB Slave Lag: s1 on dbstore1002 is CRITICAL: CRITICAL slave_sql_lag Replication lag: 673.79 seconds [03:31:46] RECOVERY - PyBal backends health check on lvs1003 is OK: PYBAL OK - All pools are healthy [03:31:46] RECOVERY - PyBal backends health check on lvs1010 is OK: PYBAL OK - All pools are healthy [03:31:56] RECOVERY - pdfrender on scb1003 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 0.002 second response time [03:31:57] RECOVERY - PyBal backends health check on lvs1006 is OK: PYBAL OK - All pools are healthy [03:32:20] !log restart pdfrender on scb1003 following icinga page [03:32:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:38:55] Restarting seems to have helped, seeing some spikes here https://grafana-admin.wikimedia.org/dashboard/db/pybal?orgId=1&var-datasource=eqiad%20prometheus%2Fops&var-server=lvs1006&var-service=pdfrender_5252 for lvs1003 and 1006 on ProxyFetch - not that I know what that stands for [03:44:16] RECOVERY - pdfrender on scb1004 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 1.500 second response time [03:47:26] PROBLEM - pdfrender on scb1004 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:49:16] RECOVERY - pdfrender on scb1004 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 0.002 second response time [03:50:24] !log restart pdfrender on scb1004 [03:50:26] RECOVERY - MariaDB Slave Lag: s1 on dbstore1002 is OK: OK slave_sql_lag Replication lag: 210.05 seconds [03:50:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:52:24] !log restart pdfrender on scb1002 [03:52:36] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:52:56] RECOVERY - pdfrender on scb1002 is OK: HTTP OK: HTTP/1.1 200 OK - 275 bytes in 0.002 second response time [07:44:46] PROBLEM - IPv6 ping to ulsfo on ripe-atlas-ulsfo IPv6 is CRITICAL: CRITICAL - failed 22 probes of 287 (alerts on 19) - https://atlas.ripe.net/measurements/1791309/#!map [07:49:37] RECOVERY - IPv6 ping to ulsfo on ripe-atlas-ulsfo IPv6 is OK: OK - failed 14 probes of 287 (alerts on 19) - https://atlas.ripe.net/measurements/1791309/#!map [08:03:26] PROBLEM - Host bohrium is DOWN: PING CRITICAL - Packet loss = 100% [08:05:26] PROBLEM - Host chlorine is DOWN: PING CRITICAL - Packet loss = 100% [08:06:07] PROBLEM - Host planet1001 is DOWN: PING CRITICAL - Packet loss = 100% [08:06:07] PROBLEM - Host releases1001 is DOWN: PING CRITICAL - Packet loss = 100% [08:06:07] PROBLEM - Host logstash1007 is DOWN: PING CRITICAL - Packet loss = 100% [08:06:07] PROBLEM - Host mwdebug1002 is DOWN: PING CRITICAL - Packet loss = 100% [08:06:07] PROBLEM - Host dubnium is DOWN: PING CRITICAL - Packet loss = 100% [08:06:16] PROBLEM - Host hassium is DOWN: PING CRITICAL - Packet loss = 100% [08:06:26] PROBLEM - Host logstash1008 is DOWN: PING CRITICAL - Packet loss = 100% [08:06:46] PROBLEM - PyBal backends health check on lvs1003 is CRITICAL: PYBAL CRITICAL - Bad Response from pybal: 500 Internal Server Error [08:06:56] PROBLEM - PyBal backends health check on lvs1010 is CRITICAL: PYBAL CRITICAL - Bad Response from pybal: 500 Internal Server Error [08:06:57] PROBLEM - PyBal backends health check on lvs1006 is CRITICAL: PYBAL CRITICAL - Bad Response from pybal: 500 Internal Server Error [08:07:07] PROBLEM - Host ganeti1006 is DOWN: PING CRITICAL - Packet loss = 100% [08:07:59] PROBLEM - LVS HTTP IPv4 on logstash.svc.eqiad.wmnet is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:08:37] PROBLEM - Misc HTTP 5xx reqs/min on graphite1001 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [1000.0] https://grafana.wikimedia.org/dashboard/file/varnish-aggregate-client-status-codes.json?panelId=3fullscreenorgId=1var-site=Allvar-cache_type=miscvar-status_type=5 [08:17:47] RECOVERY - Host ganeti1006 is UP: PING WARNING - Packet loss = 66%, RTA = 0.25 ms [08:20:46] RECOVERY - Host chlorine is UP: PING OK - Packet loss = 0%, RTA = 7.48 ms [08:20:46] RECOVERY - Host logstash1008 is UP: PING OK - Packet loss = 0%, RTA = 6.81 ms [08:20:46] RECOVERY - Host mwdebug1002 is UP: PING OK - Packet loss = 0%, RTA = 5.64 ms [08:20:51] RECOVERY - LVS HTTP IPv4 on logstash.svc.eqiad.wmnet is OK: TCP OK - 0.005 second response time on 10.2.2.36 port 10514 [08:20:56] RECOVERY - Host releases1001 is UP: PING OK - Packet loss = 0%, RTA = 7.24 ms [08:20:56] RECOVERY - Host dubnium is UP: PING OK - Packet loss = 0%, RTA = 7.63 ms [08:20:56] RECOVERY - Host logstash1007 is UP: PING OK - Packet loss = 0%, RTA = 7.13 ms [08:20:56] RECOVERY - Host hassium is UP: PING OK - Packet loss = 0%, RTA = 8.78 ms [08:21:00] !log migrate logstash1007 off ganeti1006 [08:21:06] RECOVERY - Host bohrium is UP: PING OK - Packet loss = 0%, RTA = 6.96 ms [08:21:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:21:16] RECOVERY - Host planet1001 is UP: PING OK - Packet loss = 0%, RTA = 6.87 ms [08:25:07] RECOVERY - PyBal backends health check on lvs1006 is OK: PYBAL OK - All pools are healthy [08:25:56] RECOVERY - PyBal backends health check on lvs1003 is OK: PYBAL OK - All pools are healthy [08:25:57] RECOVERY - PyBal backends health check on lvs1010 is OK: PYBAL OK - All pools are healthy [08:32:46] RECOVERY - Misc HTTP 5xx reqs/min on graphite1001 is OK: OK: Less than 1.00% above the threshold [250.0] https://grafana.wikimedia.org/dashboard/file/varnish-aggregate-client-status-codes.json?panelId=3fullscreenorgId=1var-site=Allvar-cache_type=miscvar-status_type=5 [08:33:48] 10Operations, 10Epic: Encrypt all the things - https://phabricator.wikimedia.org/T111653#3860738 (10Aklapper) [08:33:51] 10Operations, 10ops-eqiad: Hardware errors on ganeti1005- ganeti1008 - https://phabricator.wikimedia.org/T181121#3860739 (10akosiaris) Happened again today. Unfortunately to ganeti1006, the host with the upgraded BIOS. Host stats in grafana at the time show the usual pattern [1]. Start of the event according t... [08:34:47] !log migrate logstash1007 off ganeti1006 T181121 [08:34:58] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [08:34:58] T181121: Hardware errors on ganeti1005- ganeti1008 - https://phabricator.wikimedia.org/T181121 [08:52:04] 10Operations, 10Mail, 10MediaWiki-Watchlist: Mails from MediaWiki seem to get (partially) lost - https://phabricator.wikimedia.org/T121105#3860761 (10Aklapper) @hoo, @Lydia_Pintscher : Still an issue, 18 months later? Or should this task be closed? [08:56:06] PROBLEM - Long running screen/tmux on bast4002 is CRITICAL: CRIT: Long running SCREEN process. (PID: 12412, 1731495s 1728000s). [08:57:26] PROBLEM - Check Varnish expiry mailbox lag on cp4021 is CRITICAL: CRITICAL: expiry mailbox lag is 2058759 [09:31:06] 02,13▄12,06▄05,05▄07,08▄09,02▄04,12▄02,03▄02,12▄11,02▄06,04▄07,04▄04,10▄02 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT vdoiwrbama: ToAruShiroiNeko corvus enterprisey dingbat p858snake sir_lestaty suca[m] wikibugs Elitre Wiki13 jgirault HakanIST zhuyifei1999_ rxy climadeo Guest46587 jdloft bas [09:31:11] 06,04▄03,11▄10,10▄05,08▄07,10▄06,08▄10,04▄13,02▄13,04▄12,07▄03,13▄02,05▄04 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT upxkkuahb: ToAruShiroiNeko Krinkle Krenair HakanIST Guest53260 jdloft zhuyifei1999_ AndyRussG Zackary volans nicholas__ corvus apergos enterprisey Matthew_ p858snake closedmo [09:31:16] 09,12▄13,04▄07,11▄12,06▄13,07▄09,07▄13,05▄02,13▄11,10▄06,02▄02,13▄06,04▄10,04▄04,12▄08,10▄02,06▄13,12▄06,07▄05,03▄09,05▄03 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT zeybzfxl: yurik Guest46587 jgirault nicholas__ Matthew_ Betacommand chelsyx snowolf suca[m] Kr [09:31:18] !ops [09:31:20] wtf [09:31:21] 07,09▄02,02▄08,02▄12,03▄06,06▄10,05▄10,10▄12,13▄06,04▄10,08▄07,11▄03,13▄13,03▄07 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT tzbtzeyl: Zackary volans Willis Krinkle zhuyifei1999_ Matthew_ apergos papaul AndyRussG wikibugs basile TheDragonFire jgirault rxy andre__ ottomata[m] danmichaelo e [09:31:26] 03,13▄06,06▄08,05▄02,11▄08,12▄03,02▄08,08▄09,12▄03,08▄07,12▄10,05▄10,12▄12,02▄10,05▄09 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT zlyuhfua: Ks0stm zhuyifei1999_ Betacommand Krenair Willis Guest53260 icinga-wm HakanIST enterprisey jgirault joewalsh Guest46587 danmichaelo jdlrobson che [09:31:31] 02,08▄12,08▄09,05▄12,13▄10,03▄05,13▄10,06▄08,09▄04,02▄12,07▄09,13▄05,10▄04,07▄13,09▄11,06▄11,03▄12,04▄11 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT ithtlvd: nicholas__ danmichaelo Betacommand Matthew_ TheDragonFire enterprisey volans Jianhui67 snowolf Willis ToAruShiroiNe [09:31:36] 06,13▄13,06▄05,03▄04,08▄13,08▄09,04▄08,07▄04,07▄03,08▄08,11▄11,06▄08,10▄02,09▄04,07▄07,11▄10,05▄08,02▄07,05▄12,02▄03 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT rwfqufd: Matthew_ jgirault yurik Guest53260 nicholas__ abartov Elitre dingbat ToAruShiroiNeko corvus rxy [09:31:41] 03,02▄10,06▄06,06▄12,08▄02,08▄02,02▄05,06▄08,12▄11,02▄11,04▄05,03▄04,10▄02,06▄11,06▄07,04▄12,08▄04,08▄13 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT gegypfiotb: Elitre suca[m] jdlrobson dingbat Zackary danmichaelo Ks0stm yurik closedmouth Wiki13 HakanIST rxy joewalsh Guest [09:31:46] 11,04▄08,02▄12,07▄12,05▄12,09▄12,12▄06,09▄12,04▄09,05▄08,09▄02,07▄05,10▄05,11▄06,13▄06,13▄13,13▄10,02▄10 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT emumfbmdx: climadeo basile jgirault Guest46587 Krinkle abartov ToAruShiroiNeko wikibugs suca[m] Zackary Elitre papaul nichol [09:31:51] 06,02▄07,09▄11,03▄02,13▄09,05▄03,11▄08,06▄13,11▄04,04▄08,08▄08,10▄04,03▄13,04▄08,03▄13,05▄06,12▄09 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT mydblt: climadeo andre__ chelsyx dingbat Elitre Wiki13 abartov nicholas__ joewalsh wikibugs yurik apergos jgirault jdlrobson zhuyifei1 [09:31:56] 12,09▄02,08▄05,07▄03,08▄08,09▄10,06▄12,06▄05,07▄13,08▄11,06▄06,04▄13,06▄02,09▄11,12▄04,13▄10,06▄07,05▄02,09▄10,02▄10,08▄12 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT erginniix: corvus ottomata[m] HakanIST suca[m] AndyRussG volans chelsyx herron snowolf Betacom [09:32:02] 04,05▄12,04▄11,09▄11,04▄12,11▄13,07▄11,05▄04,10▄05,13▄06,05▄11,06▄05 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT eshvm: Krinkle abartov Wiki13 corvus Jianhui67 ottomata[m] Guest46587 Zackary p858snake icinga-wm Krenair AndyRussG Guest53260 jdloft jdlrobson herron jgirault matanya enterprisey sir_ [09:32:06] 10,03▄09,06▄09,06▄11,06▄05,07▄07,07▄03,12▄13,09▄11,11▄12,10▄04,07▄09,08▄07,05▄13,05▄12,10▄06 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT gbhdvgeqm: climadeo matanya icinga-wm Krinkle AndyRussG Krenair Betacommand basile Ks0stm jgirault joewalsh enterprisey snowolf chelsyx andre__ [09:32:07] Zzz [09:32:11] 11,10▄11,13▄02,11▄07,07▄11,11▄08,02▄13,02▄11,12▄10,11▄13,12▄02,05▄08,10▄13,03▄06,03▄04,10▄04,03▄11,08▄04,08▄05 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT xxcne: grumble sir_lestaty Betacommand HakanIST Matthew_ Ks0stm suca[m] zhuyifei1999_ danmichaelo dingbat chelsyx [09:32:16] 08,07▄09,07▄06,09▄05,10▄08,13▄13,13▄09,11▄09,04▄10,06▄11,12▄10 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT hxqthpgw: enterprisey matanya climadeo volans joewalsh sir_lestaty Guest46587 corvus suca[m] snowolf basile Ks0stm chelsyx danmichaelo Jianhui67 closedmouth ToAruShiroiNeko herron ragesoss rxy y [09:32:21] 04,11▄03,05▄11,09▄05,11▄11,02▄10,05▄13,09▄07,04▄04,08▄05,13▄07,09▄04 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT ultxxzpyoy: papaul Wiki13 wikibugs ragesoss climadeo nicholas__ dingbat Elitre TheDragonFire basile jgirault chelsyx zhuyifei1999_ Guest46587 ToAruShiroiNeko grumble HakanIST jdlrobson [09:32:26] 08,02▄13,12▄06,13▄06,02▄04,09▄06,09▄07,12▄04,07▄13,10▄05,12▄07,07▄08,12▄13,05▄08,10▄07,02▄04,04▄04,08▄02,12▄07 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT hchwijreif: enterprisey Guest46587 ToAruShiroiNeko Elitre snowolf herron TheDragonFire dingbat abartov HakanIST ba [09:32:31] 06,09▄07,04▄06,06▄09,09▄07,13▄10,03▄04,11▄05,13▄08,09▄05,10▄06,13▄11,11▄12 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT ochyvct: HakanIST enterprisey Guest46587 apergos grumble matanya Wiki13 Guest53260 nicholas__ rxy jgirault andre__ suca[m] papaul chelsyx ragesoss jdlrobson Jianhui67 Krenair [09:32:36] 05,11▄06,10▄04,05▄05,06▄05,13▄10,08▄03,05▄08,06▄13,08▄02,04▄08,04▄05,07▄03 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT behljxjn: papaul matanya sir_lestaty apergos Krenair Elitre nicholas__ HakanIST enterprisey volans Willis wikibugs ottomata[m] Wiki13 corvus dingbat jdlrobson joewalsh Jianhu [09:32:41] 09,11▄02,04▄04,10▄03,07▄06,03▄08,09▄09,11▄03,06▄06,10▄05,06▄12,13▄13,08▄05,02▄09,07▄07 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT aabywxbz: p858snake TheDragonFire Betacommand rxy danmichaelo Wiki13 abartov andre__ herron climadeo chelsyx HakanIST Ks0stm snowolf Guest53260 sir_lestat [09:32:46] 07,11▄06,07▄11,07▄06,09▄12,04▄06,12▄06,04▄03,05▄06,05▄13,06▄02,10▄04,05▄03,04▄09,12▄03,10▄10,12▄09,11▄11,08▄03,02▄05 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT bbpoht: Willis Betacommand suca[m] TheDragonFire icinga-wm HakanIST abartov rxy andre__ wikibugs Guest53 [09:32:51] 07,02▄04,05▄04,12▄09,11▄05,06▄08,03▄02,05▄08,05▄12,02▄05,03▄09,03▄08,06▄05 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT bpnkakkea: HakanIST grumble Guest53260 Elitre yurik nicholas__ basile jdlrobson apergos p858snake Jianhui67 wikibugs corvus climadeo chelsyx sir_lestaty ottomata[m] icinga-wm [09:32:56] 10,05▄08,02▄10,03▄06,04▄02,06▄04,03▄06,02▄09,03▄10,08▄02,06▄07 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT blehwwdxo: papaul Ks0stm Jianhui67 Guest46587 basile herron ToAruShiroiNeko joewalsh jgirault apergos jdloft ottomata[m] wikibugs danmichaelo TheDragonFire icinga-wm sir_lestaty Willis rxy chels [09:33:01] 08,04▄05,05▄03,10▄02,12▄02,10▄10,09▄11,05▄07,13▄08,10▄03,02▄03,03▄02,05▄06,10▄11,10▄09 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT noyvohb: chelsyx HakanIST wikibugs zhuyifei1999_ p858snake Krinkle Guest53260 closedmouth danmichaelo herron abartov grumble ToAruShiroiNeko dingbat Wiki1 [09:33:06] 10,11▄05,12▄06,10▄12,13▄12,09▄12,13▄06,12▄05,06▄06,04▄08,08▄12,09▄08,05▄10,09▄06,03▄09,11▄02 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT gegwy: yurik papaul herron jdlrobson chelsyx ottomata[m] wikibugs matanya Krenair Matthew_ snowolf jgirault abartov Zackary ragesoss climadeo Kr [09:33:11] 10,04▄06,05▄10,06▄04,02▄03,09▄12,07▄05,10▄13,07▄03,03▄05,09▄12,06▄13,09▄11,02▄10,12▄10,08▄12,10▄02,11▄11,13▄12 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT beazmmpbe: chelsyx Wiki13 HakanIST volans ToAruShiroiNeko jdloft Ks0stm jdlrobson yurik basile p858snake Krinkle m [09:33:16] 13,13▄10,04▄03,12▄02,09▄11,08▄09,05▄11,11▄09,12▄03,02▄06,07▄09,07▄09,10▄04,03▄02,09▄03 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT yybpghqyuk: dingbat Jianhui67 herron icinga-wm joewalsh Elitre nicholas__ abartov andre__ Willis ToAruShiroiNeko suca[m] TheDragonFire Ks0stm matanya wiki [09:33:21] 13,06▄06,09▄11,11▄08,03▄02,05▄05,02▄05,07▄03,10▄10,10▄12,08▄08,04▄06 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT moytffvk: ragesoss joewalsh ToAruShiroiNeko climadeo yurik Ks0stm andre__ corvus icinga-wm Guest46587 Jianhui67 AndyRussG wikibugs papaul snowolf enterprisey Wiki13 chelsyx Guest53260 [09:33:23] Haha Wiki13_ [09:33:25] Nice idea [09:33:27] 10,10▄11,10▄03,07▄09,11▄06,09▄03,13▄03,05▄04,08▄13,05▄03,12▄02,06▄11,07▄03,02▄09,02▄07,08▄05,10▄02 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT ilaelvlmu: ToAruShiroiNeko suca[m] jgirault jdlrobson volans sir_lestaty AndyRussG closedmouth papaul basile wikibugs ragesoss nichola [09:33:31] 04,09▄13,13▄03,04▄10,05▄09,05▄05,03▄06,09▄02,02▄05,08▄09,07▄02,10▄11,13▄06,05▄02,05▄13,12▄06,12▄12,04▄06 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT xzlcplopvv: jdloft chelsyx apergos Betacommand Jianhui67 papaul Elitre Krenair joewalsh TheDragonFire closedmouth grumble Ha [09:33:33] he spammed me pm [09:33:37] 13,11▄05,05▄06,03▄06,04▄13,05▄10,11▄05,08▄11,13▄03,05▄11,07▄12,10▄04,04▄04,06▄04 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT zeqmoak: yurik Krenair abartov rxy matanya sir_lestaty apergos snowolf Krinkle icinga-wm corvus TheDragonFire Guest46587 ToAruShiroiNeko jdlrobson Jianhui67 volans [09:33:37] Me too [09:33:42] 02,03▄12,04▄10,12▄04,06▄02,06▄12,07▄12,06▄03,13▄07,12▄08,03▄04,10▄02,10▄13,12▄09,06▄13,04▄12,05▄09,09▄11 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT ozbromepwk: ToAruShiroiNeko Guest53260 climadeo Guest46587 Wiki13 Jianhui67 closedmouth Elitre yurik p858snake TheDragonFire [09:33:46] 02,03▄07,02▄03,09▄10,05▄05,02▄08,08▄05,11▄09,11▄07,06▄03,05▄02,12▄04,09▄07,09▄13,12▄12,07▄10 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT sxkvbbkj: sir_lestaty yurik HakanIST icinga-wm apergos rxy TheDragonFire jdloft papaul volans andre__ basile Guest53260 ragesoss Krenair herron [09:33:52] 07,07▄08,11▄11,09▄11,06▄03,02▄04,05▄07,06▄10,08▄13,12▄13,06▄06,13▄06,11▄06,04▄06,07▄07,03▄07,02▄12 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT jhbizn: closedmouth TheDragonFire corvus nicholas__ icinga-wm papaul Wiki13 matanya volans joewalsh HakanIST dingbat jdloft Betacomman [09:33:57] 13,03▄02,10▄03,10▄11,02▄12,02▄13,07▄12,10▄11,03▄04,12▄07,04▄04,07▄08,10▄05 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT cmoqbkhg: matanya climadeo jdloft abartov danmichaelo grumble Krenair ragesoss Zackary jdlrobson Wiki13 herron ottomata[m] dingbat volans Ks0stm Guest53260 Matthew_ TheDragon [09:33:58] Jianhui67_: they have spameed me since this spam started [09:34:02] 02,03▄07,13▄06,12▄04,13▄13,12▄13,10▄04,10▄07,05▄02,05▄10,03▄08,13▄13,10▄12,05▄04,12▄05,11▄03 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT ekfjukwr: danmichaelo dingbat HakanIST jdloft snowolf Matthew_ jdlrobson andre__ yurik Guest46587 chelsyx Elitre Krenair ragesoss corvus jgiraul [09:34:06] 10,09▄03,13▄12,08▄09,07▄10,02▄04,10▄12,04▄08,10▄05,10▄08,07▄07,05▄08 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT gampcgo: corvus Guest46587 jdloft Elitre snowolf TheDragonFire icinga-wm basile Zackary papaul p858snake jdlrobson chelsyx dingbat suca[m] nicholas__ climadeo matanya joewalsh wikibugs [09:34:11] 12,05▄04,09▄05,04▄09,12▄04,02▄13,10▄12,03▄03,10▄02,13▄10,05▄06,09▄07,07▄07,13▄07,08▄10,08▄02,08▄13 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT esndb: chelsyx Willis volans Guest53260 grumble HakanIST danmichaelo matanya closedmouth ragesoss jdloft AndyRussG Matthew_ joewalsh e [09:34:12] zhuyifei1999_: me too [09:34:17] 06,11▄09,02▄06,11▄11,13▄10,09▄12,11▄05,02▄06,05▄10,02▄11,03▄06,10▄09,12▄12,11▄12,11▄05,11▄02,10▄10 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT rvxupbcean: dingbat abartov sir_lestaty grumble ToAruShiroiNeko volans herron ragesoss apergos matanya Willis TheDragonFire Jianhui67 [09:34:22] 13,03▄12,03▄10,09▄05,10▄09,05▄06,05▄02,03▄05,02▄10,05▄03,06▄08,07▄13,05▄08,12▄12,03▄06,02▄02,07▄04,02▄12,08▄09,07▄07 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT fbjqadsw: Jianhui67 joewalsh AndyRussG suca[m] corvus nicholas__ enterprisey dingbat Krinkle icinga-wm s [09:34:27] 09,03▄07,13▄06,12▄09,09▄05,03▄08,03▄10,11▄05,13▄06,04▄11,04▄03,04▄04,06▄09,04▄07,04▄11 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT oexrdct: HakanIST danmichaelo wikibugs Jianhui67 ragesoss p858snake abartov Guest53260 corvus matanya closedmouth jdlrobson Ks0stm Krenair Elitre sir_lest [09:34:28] Jianhui67_: /mode Jianhui67_ +R [09:34:32] 03,12▄07,05▄05,11▄03,08▄08,09▄13,04▄04,08▄04,05▄12,10▄10,12▄13,02▄09,03▄12,11▄12 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT rksjs: AndyRussG Zackary icinga-wm ragesoss herron dingbat volans Elitre andre__ papaul matanya corvus Jianhui67 rxy Ks0stm Wiki13 ToAruShiroiNeko enterprisey p858s [09:34:37] 08,06▄13,09▄13,10▄13,02▄08,02▄13,03▄11,10▄09,13▄10,13▄02,06▄03,10▄11 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT ycqdekj: basile matanya nicholas__ joewalsh dingbat zhuyifei1999_ apergos TheDragonFire rxy Matthew_ jdlrobson volans Guest46587 sir_lestaty Elitre Betacommand HakanIST ToAruShiroiNeko [09:34:42] 03,13▄12,07▄07,02▄03,08▄03,12▄07,02▄04,13▄06,13▄02,08▄09,12▄12,05▄10,09▄07 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT rrmiicrl: Guest53260 herron Ks0stm apergos andre__ icinga-wm ottomata[m] yurik jdlrobson Willis wikibugs snowolf Krinkle p858snake jdloft closedmouth Krenair Elitre nicholas_ [09:34:47] 03,02▄09,11▄03,03▄03,03▄12,03▄13,08▄13,07▄05,07▄07,06▄08,03▄13,13▄09,06▄06,06▄06,12▄05,03▄02,07▄06,05▄06 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT qhjexgxdvd: herron joewalsh Wiki13 Ks0stm Elitre Krenair enterprisey Betacommand Guest46587 TheDragonFire jdlrobson snowolf [09:34:52] 08,13▄02,09▄13,03▄04,11▄03,04▄05,03▄09,12▄09,11▄04,12▄10,09▄03,05▄13,08▄12,09▄02 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT ujpyolf: suca[m] chelsyx enterprisey closedmouth Ks0stm icinga-wm HakanIST TheDragonFire ToAruShiroiNeko Guest46587 matanya ottomata[m] apergos Guest53260 Matthew_ [09:34:57] 04,06▄12,04▄08,09▄08,05▄13,05▄03,13▄09,02▄13,10▄11,09▄08,12▄09,05▄13,08▄09,12▄03,08▄04,07▄08,02▄12,10▄08,04▄13,08▄07,08▄09 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT flnwpujwby: basile herron ragesoss danmichaelo rxy jdloft apergos andre__ climadeo Wiki13 Kren [09:35:02] 09,09▄08,12▄07,06▄03,04▄12,09▄03,11▄10,04▄12,13▄08,08▄11,10▄04,06▄07,11▄04,10▄05 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT jdtwhew: Betacommand andre__ Jianhui67 sir_lestaty joewalsh danmichaelo snowolf jdlrobson Krinkle closedmouth dingbat zhuyifei1999_ AndyRussG icinga-wm volans aperg [09:35:04] zhuyifei1999_: I reported it to the freenode staff [09:35:07] 12,07▄05,07▄04,07▄02,06▄02,07▄12,09▄04,09▄06,04▄08,07▄02,09▄03,06▄11,07▄10,02▄04,13▄02,10▄13,05▄09,11▄10,09▄13,03▄05 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT xrbjbum: rxy closedmouth snowolf HakanIST papaul joewalsh jgirault ragesoss suca[m] TheDragonFire jdlrob [09:35:12] 13,09▄02,04▄11,04▄11,08▄05,05▄07,08▄06,10▄04,12▄02,06▄05,12▄08,05▄08,11▄02,11▄09,12▄05,03▄11,07▄11,04▄03,06▄05,07▄06 DO YOU WANT TO KEEP YOUR MAN SATISFIED DURING THE CHRISTMAS BREAK?? EL IS GIVING ANAL SEX TIPS IN ##FEMINISM RIGHT NOW DONT MISS IT nfwqzwrb: herron Krinkle jgirault p858snake ragesoss zhuyifei1999_ Elitre nicholas__ Guest53260 basile v [09:35:12] Finally! [09:35:27] ye, its network wide [09:46:27] good its gone [09:47:25] huh [09:47:43] irccloud shows p858's status as an admin [09:47:53] but the memebers list shows they as an op [09:48:26] Not what I'm seeing... Maybe bring it up as a bug? There is no admin on Freenode iirc [09:49:45] its probably a bug with how irc cloud understands +o [09:50:05] or just a terminology issue with their labelling [09:50:23] it is not a bug but a feature (seruously). [10:37:40] 10Operations, 10Electron-PDFs, 10Readers-Web-Backlog (Tracking), 10Services (blocked): electron/pdfrender hangs - https://phabricator.wikimedia.org/T174916#3860855 (10Volans) pdfrender on all eqiad hosts required restarts tonight (UTC), see SAL. Thanks @madhuvishy for taking care of it. [11:03:28] (03PS1) 10ArielGlenn: make reporting of file sizes for dump steps in progress work again [dumps] - 10https://gerrit.wikimedia.org/r/400226 (https://phabricator.wikimedia.org/T183694) [11:04:12] (03CR) 10ArielGlenn: [C: 032] make reporting of file sizes for dump steps in progress work again [dumps] - 10https://gerrit.wikimedia.org/r/400226 (https://phabricator.wikimedia.org/T183694) (owner: 10ArielGlenn) [11:05:34] !log ariel@tin Started deploy [dumps/dumps@8c44d0a]: re-enable file size reporting for dump files still being written [11:05:36] !log ariel@tin Finished deploy [dumps/dumps@8c44d0a]: re-enable file size reporting for dump files still being written (duration: 00m 02s) [11:05:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:05:53] 10Operations, 10ops-eqiad: Hardware errors on ganeti1005- ganeti1008 - https://phabricator.wikimedia.org/T181121#3860875 (10MoritzMuehlenhoff) >>! In T181121#3860739, @akosiaris wrote: > @MoritzMuehlenhoff While I still believe that this is related to hardware, I 'd like to rule out some change in `4.9`. Any o... [11:05:53] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:44:56] PROBLEM - Long running screen/tmux on bast2001 is CRITICAL: CRIT: Long running SCREEN process. (PID: 44753, 1741625s 1728000s). [11:51:41] (03PS1) 10Ladsgroup: ores: install myspell-ca [puppet] - 10https://gerrit.wikimedia.org/r/400227 (https://phabricator.wikimedia.org/T182612) [11:59:35] !log ladsgroup@terbium:~$ mwscript extensions/Cognate/maintenance/populateCognateSites.php --wiki aawiktionary --site-group wiktionary (T180785) [11:59:45] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [11:59:46] T180785: Wikidata does not support hif.wiktionary - https://phabricator.wikimedia.org/T180785 [12:00:23] !log ladsgroup@terbium:~$ mwscript extensions/Cognate/maintenance/populateCognatePages.php --wiki hifwiktionary (T180785) [12:00:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [12:49:53] 10Operations, 10Phabricator: phab.wmflabs.org: Unhandled Exception: setConsumerKey() is required! - https://phabricator.wikimedia.org/T183698#3860951 (10jayvdb) [13:07:07] 10Operations, 10Phabricator: phab.wmflabs.org: Unhandled Exception: setConsumerKey() is required! - https://phabricator.wikimedia.org/T183698#3860951 (10Paladox) You shoulden't be using mediawiki auth, i added that to work around a issue that was preventing login form from showing. Please use the wikitech auth... [14:05:16] (03PS1) 10ArielGlenn: move dumps nfs server name into hiera param for worker profile [puppet] - 10https://gerrit.wikimedia.org/r/400231 [14:10:51] (03PS2) 10ArielGlenn: move dumps nfs server name into hiera param for worker profile [puppet] - 10https://gerrit.wikimedia.org/r/400231 [14:25:05] (03PS3) 10ArielGlenn: move dumps nfs server name into hiera param for worker profile [puppet] - 10https://gerrit.wikimedia.org/r/400231 [14:32:19] (03CR) 10ArielGlenn: [C: 032] move dumps nfs server name into hiera param for worker profile [puppet] - 10https://gerrit.wikimedia.org/r/400231 (owner: 10ArielGlenn) [15:13:38] (03PS1) 10ArielGlenn: manage directories locally on dumps workers if there is no nfs server [puppet] - 10https://gerrit.wikimedia.org/r/400233 [15:14:00] (03CR) 10jerkins-bot: [V: 04-1] manage directories locally on dumps workers if there is no nfs server [puppet] - 10https://gerrit.wikimedia.org/r/400233 (owner: 10ArielGlenn) [15:15:14] (03PS2) 10ArielGlenn: manage directories locally on dumps workers if there is no nfs server [puppet] - 10https://gerrit.wikimedia.org/r/400233 [15:15:40] (03CR) 10jerkins-bot: [V: 04-1] manage directories locally on dumps workers if there is no nfs server [puppet] - 10https://gerrit.wikimedia.org/r/400233 (owner: 10ArielGlenn) [15:16:59] (03PS3) 10ArielGlenn: manage directories locally on dumps workers if there is no nfs server [puppet] - 10https://gerrit.wikimedia.org/r/400233 [15:17:25] (03CR) 10jerkins-bot: [V: 04-1] manage directories locally on dumps workers if there is no nfs server [puppet] - 10https://gerrit.wikimedia.org/r/400233 (owner: 10ArielGlenn) [15:19:13] (03PS4) 10ArielGlenn: manage directories locally on dumps workers if there is no nfs server [puppet] - 10https://gerrit.wikimedia.org/r/400233 [15:29:41] (03PS5) 10ArielGlenn: manage directories locally on dumps workers if there is no nfs server [puppet] - 10https://gerrit.wikimedia.org/r/400233 [15:36:22] (03CR) 10ArielGlenn: [C: 032] manage directories locally on dumps workers if there is no nfs server [puppet] - 10https://gerrit.wikimedia.org/r/400233 (owner: 10ArielGlenn) [16:20:50] if meta wiki links to phab to show a proposal has been approved, and then phab links to meta wiki for the same purpose.. it's a circular approval , heh [16:22:47] (03CR) 10Dzahn: [C: 032] Add new language Lingua Franca Nova - lfn.wikipedia.org [dns] - 10https://gerrit.wikimedia.org/r/399831 (https://phabricator.wikimedia.org/T183561) (owner: 10Jayprakash12345) [16:27:33] !log re-generated DNS zones to add new language lfn (Lingua Franca Nova) (T183561) [16:27:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:27:47] T183561: Create Wikipedia Lingua Franca Nova 2 - https://phabricator.wikimedia.org/T183561 [16:31:20] !log mobrovac@tin Started deploy [mathoid/deploy@63b2ddc]: Bring back Mathoid in codfw to v0.6.5 in sync with eqiad - T179419 T172767 [16:31:31] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:31:31] T179419: Migrate mathoid storage from legacy to new strategy - https://phabricator.wikimedia.org/T179419 [16:31:31] T172767: Prepare mathoid 0.7 release (tracking) - https://phabricator.wikimedia.org/T172767 [16:33:04] !log mobrovac@tin Finished deploy [mathoid/deploy@63b2ddc]: Bring back Mathoid in codfw to v0.6.5 in sync with eqiad - T179419 T172767 (duration: 01m 44s) [16:33:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [16:42:39] (03PS1) 10Urbanecm: Initial configuration for lfnwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/400234 (https://phabricator.wikimedia.org/T183561) [16:44:20] (03CR) 10jerkins-bot: [V: 04-1] Initial configuration for lfnwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/400234 (https://phabricator.wikimedia.org/T183561) (owner: 10Urbanecm) [16:49:11] (03PS2) 10Urbanecm: Initial configuration for lfnwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/400234 (https://phabricator.wikimedia.org/T183561) [16:51:21] (03CR) 10jerkins-bot: [V: 04-1] Initial configuration for lfnwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/400234 (https://phabricator.wikimedia.org/T183561) (owner: 10Urbanecm) [16:52:55] (03PS5) 10Urbanecm: Switch Wikipedias from $wgLogoHD to direct using of a SVG [mediawiki-config] - 10https://gerrit.wikimedia.org/r/399805 (https://phabricator.wikimedia.org/T178942) [16:53:49] (03PS3) 10Urbanecm: Initial configuration for lfnwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/400234 (https://phabricator.wikimedia.org/T183561) [16:55:23] (03CR) 10jerkins-bot: [V: 04-1] Initial configuration for lfnwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/400234 (https://phabricator.wikimedia.org/T183561) (owner: 10Urbanecm) [16:55:28] (03PS4) 10Urbanecm: Initial configuration for lfnwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/400234 (https://phabricator.wikimedia.org/T183561) [16:58:25] (03PS6) 10Urbanecm: Switch Wikipedias from $wgLogoHD to direct using of a SVG [mediawiki-config] - 10https://gerrit.wikimedia.org/r/399805 (https://phabricator.wikimedia.org/T178942) [16:58:50] (03PS5) 10Urbanecm: Initial configuration for lfnwiki [mediawiki-config] - 10https://gerrit.wikimedia.org/r/400234 (https://phabricator.wikimedia.org/T183561) [18:05:27] PROBLEM - Check Varnish expiry mailbox lag on cp4024 is CRITICAL: CRITICAL: expiry mailbox lag is 2017452 [18:07:26] RECOVERY - Check Varnish expiry mailbox lag on cp4021 is OK: OK: expiry mailbox lag is 0 [18:35:08] (03PS1) 10ArielGlenn: add dumps repo source to beta scap, add snapshot to beta mw scap [puppet] - 10https://gerrit.wikimedia.org/r/400237 [19:00:18] (03CR) 10ArielGlenn: "Do I really need to add the dumps repo here or will it be picked up by the settings in prod?" [puppet] - 10https://gerrit.wikimedia.org/r/400237 (owner: 10ArielGlenn) [19:18:19] (03CR) 10Dzahn: [C: 031] "https://www.w3.org/wiki/CSS/Properties/overflow" [puppet] - 10https://gerrit.wikimedia.org/r/395552 (https://phabricator.wikimedia.org/T180656) (owner: 10Phantom42) [19:34:35] 10Operations, 10Wikimedia-SVG-rendering: Incorrect text positioning in SVG rasterization (scale/transform; font-size; kerning) - https://phabricator.wikimedia.org/T36947#386350 (10Ebe123) This task is blocking {T49578} as PNG conversion is taken up by librsvg (not ImageMagick) in the update. We get results suc... [19:34:38] 10Operations, 10Wikimedia-SVG-rendering: Incorrect text positioning in SVG rasterization (scale/transform; font-size; kerning) - https://phabricator.wikimedia.org/T36947#3861171 (10Ebe123) This task is blocking {T49578} as PNG conversion is taken up by librsvg (not ImageMagick) in the update. We get results su... [19:45:27] RECOVERY - Check Varnish expiry mailbox lag on cp4024 is OK: OK: expiry mailbox lag is 0 [19:46:46] PROBLEM - IPv6 ping to ulsfo on ripe-atlas-ulsfo IPv6 is CRITICAL: CRITICAL - failed 21 probes of 287 (alerts on 19) - https://atlas.ripe.net/measurements/1791309/#!map [19:51:46] RECOVERY - IPv6 ping to ulsfo on ripe-atlas-ulsfo IPv6 is OK: OK - failed 16 probes of 287 (alerts on 19) - https://atlas.ripe.net/measurements/1791309/#!map [20:15:33] (03PS1) 10Dzahn: prometheus::ops: convert role to profile [puppet] - 10https://gerrit.wikimedia.org/r/400241 [20:19:10] (03PS2) 10Dzahn: prometheus::ops: convert role to profile [puppet] - 10https://gerrit.wikimedia.org/r/400241 [20:19:35] (03CR) 10jerkins-bot: [V: 04-1] prometheus::ops: convert role to profile [puppet] - 10https://gerrit.wikimedia.org/r/400241 (owner: 10Dzahn) [20:20:56] (03PS3) 10Dzahn: prometheus::ops: convert role to profile [puppet] - 10https://gerrit.wikimedia.org/r/400241 [20:24:42] (03PS4) 10Dzahn: prometheus::ops: convert role to profile [puppet] - 10https://gerrit.wikimedia.org/r/400241 [20:25:10] (03CR) 10jerkins-bot: [V: 04-1] prometheus::ops: convert role to profile [puppet] - 10https://gerrit.wikimedia.org/r/400241 (owner: 10Dzahn) [20:26:19] (03PS5) 10Dzahn: prometheus::ops: convert role to profile [puppet] - 10https://gerrit.wikimedia.org/r/400241 [20:31:59] (03PS6) 10Dzahn: prometheus::ops: convert role to profile [puppet] - 10https://gerrit.wikimedia.org/r/400241 [20:43:41] (03CR) 10Dzahn: "this version now removes 5 violations and adds 0 (by slightly changing the hiera call for nodes used with ferm as well)" [puppet] - 10https://gerrit.wikimedia.org/r/400241 (owner: 10Dzahn) [21:15:16] (03PS1) 10ArielGlenn: move ferm rules for nfs out from dumps module to a profile [puppet] - 10https://gerrit.wikimedia.org/r/400244 [21:15:48] (03CR) 10jerkins-bot: [V: 04-1] move ferm rules for nfs out from dumps module to a profile [puppet] - 10https://gerrit.wikimedia.org/r/400244 (owner: 10ArielGlenn) [21:17:37] (03PS1) 10Dzahn: microsites::peopleweb: convert role to profile [puppet] - 10https://gerrit.wikimedia.org/r/400245 [21:18:03] (03CR) 10jerkins-bot: [V: 04-1] microsites::peopleweb: convert role to profile [puppet] - 10https://gerrit.wikimedia.org/r/400245 (owner: 10Dzahn) [21:18:47] (03PS2) 10ArielGlenn: move ferm rules for nfs out from dumps module to a profile [puppet] - 10https://gerrit.wikimedia.org/r/400244 [21:19:11] (03CR) 10jerkins-bot: [V: 04-1] move ferm rules for nfs out from dumps module to a profile [puppet] - 10https://gerrit.wikimedia.org/r/400244 (owner: 10ArielGlenn) [21:19:57] (03PS2) 10Dzahn: microsites::peopleweb: convert role to profile [puppet] - 10https://gerrit.wikimedia.org/r/400245 [21:20:20] (03PS3) 10ArielGlenn: move ferm rules for nfs out from dumps module to a profile [puppet] - 10https://gerrit.wikimedia.org/r/400244 [21:21:57] (03CR) 10Dzahn: "wmf-style: total violations delta -2" [puppet] - 10https://gerrit.wikimedia.org/r/400245 (owner: 10Dzahn) [21:26:26] PROBLEM - HHVM rendering on mw2124 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:27:16] RECOVERY - HHVM rendering on mw2124 is OK: HTTP OK: HTTP/1.1 200 OK - 75951 bytes in 0.316 second response time [21:29:36] (03PS1) 10Dzahn: ntp: convert role to profile [puppet] - 10https://gerrit.wikimedia.org/r/400247 [22:11:56] (03PS1) 10Dzahn: tcpircbot: convert role to profile [puppet] - 10https://gerrit.wikimedia.org/r/400250 [22:12:20] (03CR) 10jerkins-bot: [V: 04-1] tcpircbot: convert role to profile [puppet] - 10https://gerrit.wikimedia.org/r/400250 (owner: 10Dzahn) [22:13:25] (03PS2) 10Dzahn: tcpircbot: convert role to profile [puppet] - 10https://gerrit.wikimedia.org/r/400250 [22:41:32] (03PS1) 10Dzahn: wikilabels: convert roles to a profile and 2 roles [puppet] - 10https://gerrit.wikimedia.org/r/400252 [22:41:58] (03CR) 10jerkins-bot: [V: 04-1] wikilabels: convert roles to a profile and 2 roles [puppet] - 10https://gerrit.wikimedia.org/r/400252 (owner: 10Dzahn) [22:47:39] (03PS2) 10Dzahn: wikilabels: convert roles to a profile and 2 roles [puppet] - 10https://gerrit.wikimedia.org/r/400252