It's not a bug … It's a convenience issue

Filter duplicate emails with Procmail

Mar 13

Posted by Ryan in Internet, Software | Comments off

At school I am on multiple email lists based upon my classification. Unfortunately the school does not have the list server configured to ensure that only one copy of any given message is delivered to a user. So when administrators send messages to list1, list2 and list3 all at once I get three copies of the same email!

This has bugged me for so long and I have put off finding a way to filter it because I figured it would be pain. But it wasn’t! All you need is a new procmail recipe like the following:

PMDIR=$HOME/.procmail
:0 Whc:
msgid.lock | formail -D 4096 $PMDIR/idcache
:0 a:
Mail/duplicates

Make sure $HOME/.procmail exists before expecting this to work!

This tidbit was thanks to the Sysadmin Ramblings blog.

http://ramblings.narrabilis.com/wp/filtering-duplicate-emails-with-procmail/

Yahoo! keeps saying I send SPAM!

Nov 29

Posted by Ryan in Internet, Rant | Comments off

Yahoo! is going to make me go insane! Why do I always have problems with Yahoo’s customer service and policies? (See Yahoo! Flickr Rant.)

So ever since I switched my mail server to the new hosting platform last May I have had issues with Yahoo marking email destined for their users as SPAM. Along with that they periodically decide to defer delivery of my messages. So I frequently see these messages in my logs:

Nov 10 09:11:21 mail sendmail[27028]: nAAFBEZG027025: to=<_removed_ @yahoo.com>, ctladdr=<_removed_ @mymegabyte.com> (501/501), delay=00:00:05, xdelay=00:00:05, mailer=esmtp, pri=122274, relay=c.mx.mail.yahoo.com. [68.142.202.247], dsn=4.0.0, stat=Deferred: 421 Message from (64.79.204.78) temporarily deferred - 4.16.50. Please refer to http://help.yahoo.com/help/us/mail/defer/defer-06.html ---- OR this new one ---- Nov 24 08:38:02 mail sendmail[2816]: nAOEbtbr002813: to=<_removed_ @yahoo.com>, ctladdr=<_removed_ @mymegabyte.com> (503/503), delay=00:00:05, xdelay=00:00:05, mailer=esmtp, pri=124010, relay=f.mx.mail.yahoo.com. [98.137.54.237], dsn=4.0.0, stat=Deferred: 421 4.7.0 [TS02] Messages from 64.79.204.78 temporarily deferred due to user complaints - 4.16.56.1; see http://postmaster.yahoo.com/421-ts02.html

These started as soon as I switched to my new mail server. I can only assume this is because the previous user of this IP address sent SPAM. Since I took over this mail server has never sent SPAM. I require authentication before accepting mail to relay so only my users can send mail. The mail server also uses DKIM and SPF so I’ve done everything I can to keep the server secure and prevent spoofing.

Its not like I have thousands of users that are acting up either. This server sends mail me and my close family only!

Well why am I complaining? I have contacted Yahoo! multiple times to resolve this but I always get the same form-mail response asking for more information. I reply to that and never hear back. Lets take a look at this.
8/25/09 – I filled out their form suggested by the reject message.
8/25/09 – I received their form-mail response asking for more information.
8/25/09 – I replied to their response with all requested information.
8/27/09 – I sent a follow-up email stating that I had not received a reply in 48 hours.
9/03/09 – I politely followed up again asking for their assistance to get this resolved and pointed out that a week has passed without any reply from them.
9/10/09 – I send my final follow-up. Still never heard back.

Isn’t that pathetic? Well I got annoyed again this week so I’m going to try the process again to see if I have better luck. This time when I filled out the web-form I gave the case # from my previous incident and once again explained my situation in detail. I even answered all of the questions in the form-mail reply that I got last time up-front! So how is this one going? Read the bottom of this post.

I don’t see why Yahoo! postmaster staff is a black hole! When I first got this IP it was on multiple blacklists and other providers were blocking my outbound mail. But that wasn’t a big deal because I followed their published procedures and got the block lifted in a reasonable period of time. Yahoo! has failed to cooperate and their customers are being inconvenienced by it.

I can’t help but link this issue with the Flickr rant I had in the past. For all I know they are two completely separate entities with no common control. Yet two of my worst internet customer service experiences have come with their parent!

Yahoo! Postmaster Contact Log

11/28/09 – I fill out the web-form.
11/28/09 – I receive a form-mail reply. GRR!
11/28/09 – I reply to the form-mail message.
11/30/09 – I found someone else reported this back in 2006. Their results weren’t pretty. http://www.ahfx.net/weblog/107
12/01/09 – I receive an automated email with a survey link for feedback regarding my experience. I respond with incredibly negative feedback. My patience with Yahoo! postmaster == 0!
12/01/09 – I follow-up with an email to Yahoo! postmaster asking for a helpful reply. I also directed them to read this.
12/01/09 – I guess I don’t like having relaxed evenings at home. I sent yahoo! postmaster a new online request. This time I selected the “other” category. I was courteous and specific. I explained email is delivered to yahoo! user’s SPAM folder and I occasionally received mail delivery deferred warnings. I hope to receive a helpful response.
12/02/09 – I receive a reply from yahoo! postmaster in the form of a questionnaire that I must fill out to have my server and sending practices evaluated for re-classification.
12/03/09 – I replied to the yahoo! postmaster with complete answers to their 11 question application for evaluation. Cross your fingers.
12/07/09 – Sent a polite follow-up to my complete application.
12/09/09 – Filled out a whole new web-form with a follow-up. Its like they never get replies sent via email?
12/10/09 – 12/12/09 – I have contacted yahoo postmaster at least once per day during this time. I love the maze of forms. They send me one form I fill it out then they reply saying unfortunately they couldn’t help me because I didn’t provide all of the requested information and enclose a new form in the email. This is the world’s biggest run-around but I won’t give up!
12/16/09 – I filled out the requested form and replied. I didn’t hear back so I sent them my responses again. This time I got a form saying they replaced their form with a new form that is better and asked me to fill this one out. Woo!
12/17/09 – SUCCESS! They replied today and said that they have updated the record for my IP address in their database. A quick test email went to the inbox of a brand new yahoo! user. Finally!

Yahoo! Mail Delivery Tips

Since I fought quite hard to get my server de-blacklisted the least I can do is pass tips on to the rest of you who may need the same thing.

Is your IP ‘blacklisted’? Send an email from your server to a yahoo account. View the “full headers” of the message from the yahoo account. Does it contain a line “X-YahooFilteredBulk: ”? If so then yes.. you’re black listed.
If you’re black listed fill out the following form. http://help.yahoo.com/l/us/yahoo/mail/postmaster/bulkv2.html Yeah, I know your thinking “But I’m not a bulk sender.” Neither am I but I had to fill out this form. WARNING: Do not leave any field blank on the form. If any field is blank (even if it does not apply to you) they will reject it. The business name field? I wrote my domain name. The privacy policy? I wrote one and posted it(they rejected my app when I didn’t have one!) http://www.bbbonline.org/Privacy/sample_privacy.asp I cannot emphasize enough fill out everything. I had the form rejected for not providing information that is not applicable to me as a non-bulk sender using a personal mail server. Eventually I got through by filling everything out just as a business would have to.
Sign all outgoing emails with DKIM
Publish SPF records
Enroll on the Yahoo! Complaint Feedback Loop
Make sure the reverse DNS record for your outbound mail server points to a hostname that has the same IP.
Don’t send SPAM.

Why I don’t hate Windows 7

Sep 13

Posted by Ryan in Operating Systems, Windows | 2 Comments

Last weekend I upgraded my laptop to Windows 7. I usually don’t get caught up in the hype of a new Operating System so this was odd for me. I waited well over 2 years before upgrading Windows 2000 to XP and about a year before trying Vista.

Why I was ready to switch

Prior to the upgrade I was running Server 2008 on my laptop. As I mentioned before I liked this setup because some of the fluff features from the operating system are removed and I get a system built for an administrator’s perspective. Unfortunately, at least for my uses Server 2008 was a step back from Server 2003. Server 2003 just worked for me. Server 2008 took a fair amount of finagling to make work and even then there were oddities. Such as a tripled hibernate/resume time (45-60s!). Another thing that bugged me was the network icon in the system tray. On every Vista or Server 2008 system that I have used there is a noticeable delay from the time you click on the icon to the time that any sort of menu opens. This often led to frustrating click sequences where I am thinking “Did I miss?”. So after a year of dealing with these annoyances and others I decided to give Windows 7 a try.

How the upgrade went

When “upgrading” from a server OS to a non-server OS a full reinstall of Windows is required. That was fine because I prefer full reformats when I change OS anyways. To ease this transition I always keep my main hard drive split into two partitions. The first partition contains all of my programs, windows and the associated documents folders. The second partition contains all my personal collections like pictures and mp3s. So when I do a reinstall I have to remember to backup is my Windows user profile. (C:\users\me or C:\Documents and Settings\me).

Owned by robocopy

To prepare for the reformat I wanted to backup my full user profile directory so that I can more quickly restore individual program settings. Trying to grab these one-by-one would for sure leave something forgotten. So I created a new user account with administrator privileges, rebooted and logged in to the new account. I then went to the folder I wanted to backup and typed “xcopy /?” to remember the command line switches. It printed a nice big deprecation notice saying that I should use the new tool robocopy. So I looked up its parameters and ran it with /copyall. Before starting the copy I made sure that my entire profile would fit on an external disk with a couple GB to spare. After 20 minutes I had a feeling something was wrong so I checked my free disk space. Sure enough something had gone wrong my disk was almost full. I canceled the copy and then investigated. Somehow robocopy found/made a back reference and was recursively copying the same folder over and over and over. So my spare disk was full of AppData\Local Settings\AppData\Local Settings\AppData …. etc. So I tried to delete this folder. Windows silently failed. No error message no freed disk space. Grrr. So I went into the tree and started recursively deleting the folder by hand. This worked until Windows gave me an error message that the file name was too long. That made me think “How the heck did you put it there in the first place?!?!” The only way I could correct this was to reformat my backup drive then re-backup everything! What a hassle!

Windows 7 Install

The install of Windows 7 went very smoothly. I had no trouble specifying that I wanted a partition reformatted and the OS installed there. After the install all but one driver worked out of the box. The missing driver was video since my laptop contains an ATI Mobility Fire GL T2 video card ( for which you are supposed to only get drivers from your laptop manufacturer). In the past I got around this by using OmegaDrivers.net or Mobility Modder from DriverHeaven.net. Well this time neither of those worked. The MM modified driver installed but upon reboot my USB mouse wouldn’t work. I unplugged the mouse and replugged it and got it to work. But Windows disabled my video driver due to a conflict. Boo! After 30 minutes of trying things I finally went to device manager and hit the upgrade driver option. There I chose the “let me pick” option. I manually selected display adapters and unchecked the box for “only display compatible drivers”. Then I found the desktop version of my driver in the ATI list. I told Windows to install that and it worked beautifully. On my next reboot windows even offered to install a newer version off of Windows Update. Now I’m running a properly accelerated graphics environment with in the box drivers. Wow! That’s a first!

After the driver issue was resolved my install was very smooth an easy. I did make one significant customization and that was to disable a handful of unnecessary services just like I do for every install. This reduces some of my memory overhead.

Using Windows 7

Aero

Windows 7 Aero

I find Windows 7 to be a breeze to use. The most notable change is the task bar at the bottom. Gone are the days of a quick launch menu and little rectangles for each of your programs. Now the quick launch bar and your running programs are merged into one panel. The running programs have a frosted look to them as you see in the pictures. The Aero interface is nicely tuned and doesn’t add any noticeable burden to my 5 year old laptop. This laptop wasn’t low-end when I bought it but it is certainly behind today’s standards.

Hibernation

I’m extremely pleased to have quick hibernation and resume back as well. Now my 2GB of RAM hibernates in < 15s and resumes in the same time.

Network Center

The Network Center icon lag is officially gone! Now when I click the Windows wireless icon I instantly get a menu with a list of available wireless networks. I can click any network to join and the remainder of the join process is carried out with that popup menu. So simple and quick! This I could explain to my grandmother over the phone.

Windows 7 Wireless Networks

New Windows key mappings

Another bonus in Windows 7 is the addition of several new Windows keys shortcuts.

Win+Up: Maximize
Win+Down: Restore / Minimize
Win+Left: Snap to left
Win+Right: Snap to right
Win+Shift+Left: Jump to left monitor
Win+Shift+Right: Jump to right monitor
Win+Home: Minimize / Restore all other windows
Win+T : Focus the first taskbar entry
Win+Space: Peek at the desktop
Win+G: Bring gadgets to the top of the Z-order
Win+P: External display options (mirror, extend desktop, etc)

I am very happy with the Win + P shortcut. I can’t tell you how many delays I have sat through where a presenter at a conference/meeting brings a laptop and connects to the project and is confused when the projector shows a blank screen. Many audience members try to be helpful and shout things like “Press Fn and F7” or “Fn and F9”, etc. These possibilities are endless because every laptop manufacturer has picked a different key for this functionality. Now Windows makes it simple. So anyone can just press Win + P to see the same screen. Microsoft got this right!

Windows 7 Projector Connection

The other new mappings are nice but I’ll have to keep reading the list before I can ever remember to work them into my routine.

Safely removing a thumbdrive

Windows 7 Safely Remove Hardware

It is about time Microsoft make this simpler and quicker!!!!! Prior to Win 7 you had to click the tray icon, tell it to safely remove hardware (what else would I click it for !?!) then a dialog opens with cryptic device names like “USB Mass Storage” (fine for 1 device but what if I’m connected to a card reader too?) then click stop then it opens another dialog to tell you more precisely what you’re about to remove then you click stop again. Jeez! Most of the time I just yanked the thumbdrive rather than deal with that. In Win7 the process is smooth! When you click the safe removal icon a single menu comes up showing you all of the information previously distributed in multiple dialogs. You click the item you want and you’re done! Thank you Microsoft!

Windows 7 Wrap-up

So in one week I am very happy that I upgraded. This was well worth the hassle of a day’s downtime for reinstall. This kind of step forward is how every OS upgrade should be!