Kilg.us – Fantasy Stat Tracker …Tracker

I got a couple emails yesterday that some daily boxscore email stats were incorrect. Looking into the matter, it was immediately obvious what had happen. As I worked on the transition of Kilg.us to its new servers, I had accidentally left a database connection pointing to the new dev database, instead of the old, production database. As such, the stats weren’t up-to-date when the daily emails were generated. I simply pointed the connection back to the correct database and figured all was well.

This morning there have again been reports of incorrect stats. I double-checked that the database connection was pointing to the correct IP address–it was. Then I remembered one of the many reasons I’m moving Kilg.us: the hosting company uses a non-standard port for database connections! Why? Who the hell knows. But I had forgotten to explicitly change back to their port. I’ve done that now. I’ve tested the connection. All seems to be well.

On the migration topic, Kilg.us is up and fully functional on its new servers. I’m testing manually with day-to-day usage to try to identify any aspects or configurations I may have missed. Right now, my simple tests are showing that Kilg.us runs between 2.2 and 5 times as fast on the new servers (adjusting as best I can for load considerations). I’m working on some minor changes that will decrease the bandwidth Kilg.us requires because the new servers are billed by data transfer. I’ll give at least 24 hours notice before I make the migration. Depending on how quickly the domain propagates, Kilg.us may be down for some of you for up to 24 hours during the migration. More details to come.

By now, everyone should have received their daily boxscores. If not, please comment below.

We were not without bumps, unfortunately. That I am aware of so far:

• As many a 50 boxscores were sent out multiple times (apologies!)
• The public SMTP I tried to use apparently uses a 24-hour window (rather than a day) for its quotas

The duplicate boxscores was just a goof by me. Last night as I was running my final tests, I commented-out the line that removes an email from the queue after sending because I didn’t want to keep re-entering data into the system before each test. I forgot to un-comment that line when I was done. Because of that, when the system pulled the first 50 boxscores to send, it still left them in the queue. When it went to pull the next 50, it returned the same group. I noticed this after a few times through the process, so hopefully everyone that got duplicates only saw a trickle–not a deluge–of them.

The SMTP issue is frustrating. In my testing, I was able to generate the full list of emails. I assumed the quota would reset at midnight. Apparently that wasn’t the case. So hundreds last night plus hundreds this morning meant the sending account was locked out mid-process. To resolve this, I used one of my personal accounts to complete the send. Many of you will have seen messages come from “matthew@encoredigitalmedia.com” rather than Kilg.us. I figured this was better than making you miss a day’s boxscore.

I have spent most of the last two days trying to get a dedicated Kilg.us SMTP server up and running so I don’t need to worry about stupid quotas and such. Unfortunately, I have absolutely no experience with setting up SMTP and have had just about zero success–hence the use of a public SMTP to get this moving. If anyone knows SMTP and can offer a helping hand, I would be greatly appreciative! Get in touch by email (admin@kilg.us), comment here, or on Twitter (@kilgustracker) or Facebook.

All that said, I think all emails should have been delivered and I have received feedback from some people whose emails weren’t rendering properly before that they look good now. All in all, I’m going to call this one a draw. I’ll keep plugging away today in hopes of a perfect push tomorrow.

Tomorrow morning Kilg.us will use a entirely new process for generating and sending daily boxscore emails! So let’s all cross our fingers and hope it works…

I have spent all of this week re-designing and re-building how emails are generated, stored, and sent. Suffice to say, the new system is significantly more complex than the old one. When I’ve had a chance to breath and am not so tired, I’ll go into more detail on how it works.

My expectations of the new system are that it will:

• Scale to accommodate as many users/emails as desired (the old system failed in the 200-300 range)
• Allow of future portability if Kilg.us needs to change hosting
• Implement all components to identify HTML vs. Text email so boxscores render properly on all clients
• Allow for side-by-side HTML and Text versions (text version is forthcoming)
• Provide redundancy of data, so if the process fails it can be re-started without losing a day’s data
• Reduce likelihood of Kilg.us mail servers being identified as SPAM and black-listed
• Lower computation overhead

It sounds almost too good to be true! Tomorrow’s push will be the first full run trying to send out to everyone’s unique addresses. All the testing so far has been delivering to a kilg.us address (or small tests to outside domains). I’ll be keeping an eye on things in the morning to try to ensure everything goes smoothly. If you do not receive your normal boxscore in an acceptably legible format, please post comments here with details (DETAILS–without them I can’t do anything–what email client, what team name, what was wrong, screenshots, any little detail).

I’ve been frantically working away on a new email system for Kilg.us the last couple days. The goal is to streamline the process, reduce execution time, reduce likelihood for failure, and improve the quality of the email delivered to your inbox. I’m about 20 hours into the effort, but spent 6 hours going in circles last night…very frustrating. As is often the case, I slept on it and have some new ideas this morning, so I’m again optimistic that I’ll be rolling it out in the coming days. Check back for updates.

In the mean time, a number of people have been reporting that their emails are not being displayed as HTML in their email clients. I haven’t tried to troubleshoot this as I am replacing the system anyway. If you are experiencing this, it wouldn’t hurt to comment with what email client you use (Thunderbird, Outlook, Gmail, iPhone Mail, etc.). We might notice a pattern and be able to share some advice with one another.

As far as I know, emails have gone out properly for three straight days now. If you’re not seeing them, leave a comment, ping me on twitter (@kilgustracker), or shoot me an email (admin@kilg.us).

As many of you experienced, this morning the Kilg.us daily boxscores went horribly wrong and began sending piles of duplicate mails to everyone. If I haven’t apologized to you yet: I’m sorry. I try very hard to make sure any changes I make to Kilg.us won’t have an adverse effect on users. I bungled this one, but at least I think I’ve figured out what happened.

I mentioned previously that I moved the daily boxscore email script to the new Kilg.us server. That is why emails went out to anyone. On the new server PHP is installed differently from the old server. As a result, when I run the CRON job to generate the emails, I needed to use an application to call the script rather than just calling PHP to execute it. I chose to use a program called wget. The idea is that wget makes a call to a URL and that URL (a PHP file) generates and sends the emails. Before scheduling the morning’s emails, I tested to ensure the process worked. When I tested, though, I only used a sub-set of data. I didn’t really need 350 emails coming into my inbox, so I tested with a couple emails each for those sent to a team owner and those sent to a team viewer. That worked great.

When the CRON job ran this morning, everything seemed to go well until duplicates started showing up. A second round, then a third, then a fourth and so on. Interestingly, each wave was 15 minutes apart. As it turns out, if wget can’t complete a request (in this case a VERY long request for 350 emails), it tries again. By default it will try up to 20 times to fetch a file. Because the script takes so long to run, I believe it exceeded the server time-out.  When the script timed out, wget requested it a second time, then a third, then a fourth and so on. I believe this is what caused the duplicate emails.

I’ve made three adjustments to address this. First, I increased the server time-out for this process. Second, I have changed the CRON job to tell wget never to retry the fetch if it fails. Third, I’ve broken the massive emails process into multiple chunks. This is a temporary fix until I convert to using PEAR::Mail to more intelligently manage the process. I’m done for tonight, but that will probably be the priority tomorrow.

So that’s all of today’s work. Six hours sunk, but I think the emails will work in the morning. I’ve pushed back the time that the emails go out by a couple hours to they’ll line up with when I roll out of bed in the morning. If things go off-track, I’ll be able to curtail things faster than today. If all goes well, I’ll move the schedule back to the early morning hour so everyone has the boxscore email when they get up in the morning.

Keep your fingers crossed…

On the positive side, email boxscores finally went out this morning. And based on the number of emails I’ve received, they reached a lot of people! Unfortunately, after the initial batch they continued to send another 6 or 7 times. I believe I have that process stopped.

At first glance, I don’t know why this happened. The script that sends the emails is unchanged from what used to work (just migrated to the new server). That would seem to suggest there was a problem with the CRON job scheduling. I’ve deleted all CRON jobs from the server and will be doing some more research to try to determine what went wrong before I re-schedule the script.

I’m sorry to all of you that were flooded by emails this morning. I’ll do my best to ensure it doesn’t happen again.

Things are proceeding well with the migration of Kilg.us to a new host. The cloud server is up, the web server is installed, database server configured, postfix installed, SSH/SFTP running. I’ve even migrated the codebase with success, although it is all still running on the old Kilg.us database. Tomorrow I hope to get the database copied over and see how things work.

For tomorrow I have moved the CRON job that mails daily boxscores to the new server. An initial test of it was successful. I’m optimistic that tomorrow morning everyone that wants boxscore emails should receive them.

This morning the boxscore email script failed while running. I believe I’ve tracked the problem to a time-out error.

As many of you have surely noticed, Kilg.us is taking MUCH longer to render team pages than it did last year–2-6 times as long in my experience. Each time a team page is rendered, Kilg.us checks to make sure it has the latest stats for each player. If it doesn’t, it pulls in the latest and updates the database. This same process occurs when the daily boxscores are generated. This process is taking much longer than it used to. What used to take a few minutes is now trending over 90.

I’m not sure what I will do about this, yet. As I see it now, there are four options:

1. optimize the current process
2. replace the continuous look-up mechanism with a scheduled process
3. change hosting provider
4. change stats provider

I will likely start with #1, but I hold little hope of its success. #2 would suck as it could mean Kilg.us stats are a few minutes out of date at any given time. I’m also not sure it would actually decrease the load. I would love to do #3 as I’ve never been impressed with Kilg.us’s current host, but better options cost more money and the time to make the transition isn’t insignificant. Number 4 is also a fanciful option as I am yet to find another free source of live stats and paid options cost tens of thousands of dollars.

When I have a solution–or at least a path forward–selected I will post again. Until then…sorry.

Daily email boxscores should resume tomorrow morning–they send about 6am Eastern time. The emails contain your team’s stats for the previous day’s games. If you’d like to receive the daily boxscore emails for one or more of your teams, visit the Manage my Settings page and check out the Kilg.us Boxscores section.

If you think you’ve signed up for the emails and don’t receive one for your team tomorrow morning, please post a comment to this thread so I can investigate.

Happy baseball season!

I’m still working to diagnose the issues with daily boxscore emails. Based on my research, I think I have narrowed the issue down to daily boxscore emails for friends’ teams (although I still don’t know why). Is there anyone who has signed up for daily boxscore emails for their own teams that is not receiving them?