It's not a bug … It's a convenience issue

Random tech things that I come across.

Archive for category Internet

A look back: Roughly 9 years of email behavior analyzed

Jun 30

Posted by Ryan in Internet | Comments off

In passing I have seen several plugins/apps/sites that offer to retrieve your email history and present some sort of analysis. My wife mentioned this recently and I thought this idea sounded like fun so I did a little processing on my own email. I have email records going back to late 2003 – just about the time I started undergrad. So with some grep/java/gnuplot magic I present to you a graph of the timestamps for every email I have sent since then. I just processed sent emails because my behavior should be more evident than in received.

Analysis

The graph contains a red dataset – my personal emails and a green dataset – emails sent from my university account. The Y axis is time of day and the X axis is the date. Each dot is an email sent at that time on the given day.

Region-by-region

  • The first region that you see is before I started logging school email. It just has personal emails… spanning all times of day/night. Seriously? I sent more email from 0000-0800 then 1600-2400???
  • Second region introduces university emails. During this time I also worked for the university part-time 0900-1800. Nothing too surprising here.
  • Third region is what I call early grad school. A general increase in both of my email volumes.
  • Fourth region – the ASSERT period. The ASSERT project was a massive 2-2.5 year research lab project with a lot of people working on it and a lot of work. Check out the huge increase in school emails. I bet you could spot deadlines in there too.
  • Fifth region – Summer internship
  • Sixth region – Late grad school. Post-quals, post-course work, post-ASSERT, just relaxed set my own schedule time
  • Final region – Real Job. Strict no-email hours, waking up early, less email, yup, that’s real life for ya.

Global observations

  • Maybe I imagine it but I swear you see that the time that I start sending emails in the morning gradually becomes later and later — same for cease time. What/who is to blame? Grad school or my GF/fiance/wife?
  • I sent more school email than personal: 5,164 messages to 4,849 messages
  • You can vaguely spot summer breaks. Look for swaths of lower green density bordered by bursts.
  • Sadly you can’t really spot certain events like: marriage, quals, proposal or defense.

Statistics

Number of emails sent per year:

Year Personal Count School Count
2003 – 57
2004 145 310
2005 412 310
2006 474 354
2007 728 815
2008 831 1,378
2009 804 1,140
2010 893 438
2011 438 464
2012 – to May 68 59

Filter duplicate emails with Procmail

Mar 13

Posted by Ryan in Internet, Software | Comments off

At school I am on multiple email lists based upon my classification. Unfortunately the school does not have the list server configured to ensure that only one copy of any given message is delivered to a user. So when administrators send messages to list1, list2 and list3 all at once I get three copies of the same email!

This has bugged me for so long and I have put off finding a way to filter it because I figured it would be pain. But it wasn’t! All you need is a new procmail recipe like the following:

PMDIR=$HOME/.procmail
:0 Whc:
msgid.lock | formail -D 4096 $PMDIR/idcache
:0 a:
Mail/duplicates

Make sure $HOME/.procmail exists before expecting this to work!

This tidbit was thanks to the Sysadmin Ramblings blog.

http://ramblings.narrabilis.com/wp/filtering-duplicate-emails-with-procmail/

Yahoo! keeps saying I send SPAM!

Nov 29

Posted by Ryan in Internet, Rant | Comments off

Yahoo! is going to make me go insane! Why do I always have problems with Yahoo’s customer service and policies? (See Yahoo! Flickr Rant.)

So ever since I switched my mail server to the new hosting platform last May I have had issues with Yahoo marking email destined for their users as SPAM. Along with that they periodically decide to defer delivery of my messages. So I frequently see these messages in my logs:

Nov 10 09:11:21 mail sendmail[27028]: nAAFBEZG027025: to=<_removed_ @yahoo.com>, ctladdr=<_removed_ @mymegabyte.com> (501/501), delay=00:00:05, xdelay=00:00:05, mailer=esmtp, pri=122274, relay=c.mx.mail.yahoo.com. [68.142.202.247], dsn=4.0.0, stat=Deferred: 421 Message from (64.79.204.78) temporarily deferred - 4.16.50. Please refer to http://help.yahoo.com/help/us/mail/defer/defer-06.html
---- OR this new one ----
Nov 24 08:38:02 mail sendmail[2816]: nAOEbtbr002813: to=<_removed_ @yahoo.com>, ctladdr=<_removed_ @mymegabyte.com> (503/503), delay=00:00:05, xdelay=00:00:05, mailer=esmtp, pri=124010, relay=f.mx.mail.yahoo.com. [98.137.54.237], dsn=4.0.0, stat=Deferred: 421 4.7.0 [TS02] Messages from 64.79.204.78 temporarily deferred due to user complaints - 4.16.56.1; see http://postmaster.yahoo.com/421-ts02.html

These started as soon as I switched to my new mail server. I can only assume this is because the previous user of this IP address sent SPAM. Since I took over this mail server has never sent SPAM. I require authentication before accepting mail to relay so only my users can send mail. The mail server also uses DKIM and SPF so I’ve done everything I can to keep the server secure and prevent spoofing.

Its not like I have thousands of users that are acting up either. This server sends mail me and my close family only!

Well why am I complaining? I have contacted Yahoo! multiple times to resolve this but I always get the same form-mail response asking for more information. I reply to that and never hear back. Lets take a look at this.
8/25/09 – I filled out their form suggested by the reject message.
8/25/09 – I received their form-mail response asking for more information.
8/25/09 – I replied to their response with all requested information.
8/27/09 – I sent a follow-up email stating that I had not received a reply in 48 hours.
9/03/09 – I politely followed up again asking for their assistance to get this resolved and pointed out that a week has passed without any reply from them.
9/10/09 – I send my final follow-up. Still never heard back.

Isn’t that pathetic? Well I got annoyed again this week so I’m going to try the process again to see if I have better luck. This time when I filled out the web-form I gave the case # from my previous incident and once again explained my situation in detail. I even answered all of the questions in the form-mail reply that I got last time up-front! So how is this one going? Read the bottom of this post.

I don’t see why Yahoo! postmaster staff is a black hole! When I first got this IP it was on multiple blacklists and other providers were blocking my outbound mail. But that wasn’t a big deal because I followed their published procedures and got the block lifted in a reasonable period of time. Yahoo! has failed to cooperate and their customers are being inconvenienced by it.

I can’t help but link this issue with the Flickr rant I had in the past. For all I know they are two completely separate entities with no common control. Yet two of my worst internet customer service experiences have come with their parent!

Yahoo! Postmaster Contact Log

11/28/09 – I fill out the web-form.
11/28/09 – I receive a form-mail reply. GRR!
11/28/09 – I reply to the form-mail message.
11/30/09 – I found someone else reported this back in 2006. Their results weren’t pretty. http://www.ahfx.net/weblog/107
12/01/09 – I receive an automated email with a survey link for feedback regarding my experience. I respond with incredibly negative feedback. My patience with Yahoo! postmaster == 0!
12/01/09 – I follow-up with an email to Yahoo! postmaster asking for a helpful reply. I also directed them to read this.
12/01/09 – I guess I don’t like having relaxed evenings at home. I sent yahoo! postmaster a new online request. This time I selected the “other” category. I was courteous and specific. I explained email is delivered to yahoo! user’s SPAM folder and I occasionally received mail delivery deferred warnings. I hope to receive a helpful response.
12/02/09 – I receive a reply from yahoo! postmaster in the form of a questionnaire that I must fill out to have my server and sending practices evaluated for re-classification.
12/03/09 – I replied to the yahoo! postmaster with complete answers to their 11 question application for evaluation. Cross your fingers.
12/07/09 – Sent a polite follow-up to my complete application.
12/09/09 – Filled out a whole new web-form with a follow-up. Its like they never get replies sent via email?
12/10/09 – 12/12/09 – I have contacted yahoo postmaster at least once per day during this time. I love the maze of forms. They send me one form I fill it out then they reply saying unfortunately they couldn’t help me because I didn’t provide all of the requested information and enclose a new form in the email. This is the world’s biggest run-around but I won’t give up!
12/16/09 – I filled out the requested form and replied. I didn’t hear back so I sent them my responses again. This time I got a form saying they replaced their form with a new form that is better and asked me to fill this one out. Woo!
12/17/09 – SUCCESS! They replied today and said that they have updated the record for my IP address in their database. A quick test email went to the inbox of a brand new yahoo! user. Finally!

Yahoo! Mail Delivery Tips

Since I fought quite hard to get my server de-blacklisted the least I can do is pass tips on to the rest of you who may need the same thing.

  1. Is your IP ‘blacklisted’? Send an email from your server to a yahoo account. View the “full headers” of the message from the yahoo account. Does it contain a line “X-YahooFilteredBulk: ”? If so then yes.. you’re black listed.
  2. If you’re black listed fill out the following form. http://help.yahoo.com/l/us/yahoo/mail/postmaster/bulkv2.html Yeah, I know your thinking “But I’m not a bulk sender.” Neither am I but I had to fill out this form. WARNING: Do not leave any field blank on the form. If any field is blank (even if it does not apply to you) they will reject it. The business name field? I wrote my domain name. The privacy policy? I wrote one and posted it(they rejected my app when I didn’t have one!) http://www.bbbonline.org/Privacy/sample_privacy.asp I cannot emphasize enough fill out everything. I had the form rejected for not providing information that is not applicable to me as a non-bulk sender using a personal mail server. Eventually I got through by filling everything out just as a business would have to.
  3. Sign all outgoing emails with DKIM
  4. Publish SPF records
  5. Enroll on the Yahoo! Complaint Feedback Loop
  6. Make sure the reverse DNS record for your outbound mail server points to a hostname that has the same IP.
  7. Don’t send SPAM.