Kilg.us – Fantasy Stat Tracker …Tracker

As many of you experienced, this morning the Kilg.us daily boxscores went horribly wrong and began sending piles of duplicate mails to everyone. If I haven’t apologized to you yet: I’m sorry. I try very hard to make sure any changes I make to Kilg.us won’t have an adverse effect on users. I bungled this one, but at least I think I’ve figured out what happened.

I mentioned previously that I moved the daily boxscore email script to the new Kilg.us server. That is why emails went out to anyone. On the new server PHP is installed differently from the old server. As a result, when I run the CRON job to generate the emails, I needed to use an application to call the script rather than just calling PHP to execute it. I chose to use a program called wget. The idea is that wget makes a call to a URL and that URL (a PHP file) generates and sends the emails. Before scheduling the morning’s emails, I tested to ensure the process worked. When I tested, though, I only used a sub-set of data. I didn’t really need 350 emails coming into my inbox, so I tested with a couple emails each for those sent to a team owner and those sent to a team viewer. That worked great.

When the CRON job ran this morning, everything seemed to go well until duplicates started showing up. A second round, then a third, then a fourth and so on. Interestingly, each wave was 15 minutes apart. As it turns out, if wget can’t complete a request (in this case a VERY long request for 350 emails), it tries again. By default it will try up to 20 times to fetch a file. Because the script takes so long to run, I believe it exceeded the server time-out.  When the script timed out, wget requested it a second time, then a third, then a fourth and so on. I believe this is what caused the duplicate emails.

I’ve made three adjustments to address this. First, I increased the server time-out for this process. Second, I have changed the CRON job to tell wget never to retry the fetch if it fails. Third, I’ve broken the massive emails process into multiple chunks. This is a temporary fix until I convert to using PEAR::Mail to more intelligently manage the process. I’m done for tonight, but that will probably be the priority tomorrow.

So that’s all of today’s work. Six hours sunk, but I think the emails will work in the morning. I’ve pushed back the time that the emails go out by a couple hours to they’ll line up with when I roll out of bed in the morning. If things go off-track, I’ll be able to curtail things faster than today. If all goes well, I’ll move the schedule back to the early morning hour so everyone has the boxscore email when they get up in the morning.

Keep your fingers crossed…

On the positive side, email boxscores finally went out this morning. And based on the number of emails I’ve received, they reached a lot of people! Unfortunately, after the initial batch they continued to send another 6 or 7 times. I believe I have that process stopped.

At first glance, I don’t know why this happened. The script that sends the emails is unchanged from what used to work (just migrated to the new server). That would seem to suggest there was a problem with the CRON job scheduling. I’ve deleted all CRON jobs from the server and will be doing some more research to try to determine what went wrong before I re-schedule the script.

I’m sorry to all of you that were flooded by emails this morning. I’ll do my best to ensure it doesn’t happen again.

Things are proceeding well with the migration of Kilg.us to a new host. The cloud server is up, the web server is installed, database server configured, postfix installed, SSH/SFTP running. I’ve even migrated the codebase with success, although it is all still running on the old Kilg.us database. Tomorrow I hope to get the database copied over and see how things work.

For tomorrow I have moved the CRON job that mails daily boxscores to the new server. An initial test of it was successful. I’m optimistic that tomorrow morning everyone that wants boxscore emails should receive them.

This morning on Facebook I mentioned a few goals for fixes and improvements to Kilg.us this weekend. Those items were:

1. update MLB player/team lists
2. get email boxscores working
3. speed up stat collection
4. trim players that show up on the “Major League Baseball” team rosters (right now it includes players throughout all the minors, with no respect to whether they might actually appear in a real game…

I’m pleased to report that numbers 1 and 4 are soundly completed! As of this morning, all active MLB players should be listed in Kilg.us and associated with their correct teams. If you follow any of the “Major League Baseball” teams, those rosters should also be cleaned up to only show players on that team’s 40-man roster.

Unfortunately, 2 and 3 are stills works in progress. I have made some changes to the model for how player objects are created. The result of this change is that player stats are only refreshed from MLB.com when needed, rather than any time a player is referenced. In practice, loading a team page requires that a player’s stats be up-to-the-second. When a daily boxscore email is generated in the wee hours of the morning, though, there is no need to go gather latest stats–they haven’t changed in the hours since the games all ended.

In my testing so far, team pages are loading 10-40% faster now. I’m still not happy with their performance, but at least it is progress. When I try to generate all the daily emails, those are running significantly faster. Before today, the process was trending towards 90 minutes, timing out, and failing. Now the process is hammering through 80 (of ~360) emails in the first minute and sending emails properly. At that point, the server slows to a crawl and, while it continues to output the email content, it refuses to send the emails.

I considered reaching out to the hosting company for help in identifying why their system was doing this, but I’ve given up. The company (AMS Computer Services) and infrastructure that hosts Kilg.us sucks. I’m going to try moving Kilg.us to a cloud hosting solution from Rackspace. I’ve spent a good part of the day getting the new server up and configured. Tomorrow I should be able to start moving Kilg.us over, testing, troubleshooting, and seeing how it performs. It’s going to be a long, tedious, frustrating process but it can’t possibly be worse than sticking with the status quo.

In the meantime, my fingers are crossed that our daily boxscore emails will fire off properly in the morning…but I’m not holding my breath.

The reason I’ve mentioned in the past for not making the move to a new hosting solution (besides the time and effort) is that it will cost more than the current option. Because Kilg.us is going to cost me more to host now, I’ve added a PayPal donation button to give anyone interested the option in helping to support Kilg.us. I’m not hoping to make anything off Kilg.us, but it would be great it is wasn’t a financial burden. If you like using Kilg.us, I hope you’ll consider giving a little bit to help offset my costs. $5, $10, $1, anything will help!

I hope to have an update with good news by tomorrow evening as I start to migrate Kilg.us.

This morning the boxscore email script failed while running. I believe I’ve tracked the problem to a time-out error.

As many of you have surely noticed, Kilg.us is taking MUCH longer to render team pages than it did last year–2-6 times as long in my experience. Each time a team page is rendered, Kilg.us checks to make sure it has the latest stats for each player. If it doesn’t, it pulls in the latest and updates the database. This same process occurs when the daily boxscores are generated. This process is taking much longer than it used to. What used to take a few minutes is now trending over 90.

I’m not sure what I will do about this, yet. As I see it now, there are four options:

1. optimize the current process
2. replace the continuous look-up mechanism with a scheduled process
3. change hosting provider
4. change stats provider

I will likely start with #1, but I hold little hope of its success. #2 would suck as it could mean Kilg.us stats are a few minutes out of date at any given time. I’m also not sure it would actually decrease the load. I would love to do #3 as I’ve never been impressed with Kilg.us’s current host, but better options cost more money and the time to make the transition isn’t insignificant. Number 4 is also a fanciful option as I am yet to find another free source of live stats and paid options cost tens of thousands of dollars.

When I have a solution–or at least a path forward–selected I will post again. Until then…sorry.

Daily email boxscores should resume tomorrow morning–they send about 6am Eastern time. The emails contain your team’s stats for the previous day’s games. If you’d like to receive the daily boxscore emails for one or more of your teams, visit the Manage my Settings page and check out the Kilg.us Boxscores section.

If you think you’ve signed up for the emails and don’t receive one for your team tomorrow morning, please post a comment to this thread so I can investigate.

Happy baseball season!

With Spring Training upon us, I’m about to start gearing up Kilg.us for the new season. No big plans for upgrades currently. Kilg.us seemed to settle into a nice place mid-way through last year, so I’m hoping that will continue to be the case for 2011.

The one upgrade I am considering is converting/extending Kilg.us to work as a Facebook App. I’m not entirely sure of the constraints or repercussions of such a move yet, but it’s the only item leaping out at me as a fun upgrade. If you have any other ideas, send them my way.

Remember to follow Kilgus on Twitter (@KilgusTracker) and Facebook for all the latest news and updates.

I’m still working to diagnose the issues with daily boxscore emails. Based on my research, I think I have narrowed the issue down to daily boxscore emails for friends’ teams (although I still don’t know why). Is there anyone who has signed up for daily boxscore emails for their own teams that is not receiving them?

Kilg.us passed a few milestones on usage yesterday. Here are some fun facts:

• Over 1000 teams being managed
• Over 700 unique users
• Over 100 industries formed

In context, we ended the 2009 season just shy of 500 users, right about 700 teams, and around 60 industries.

Keep spreading the word! Knowing lots of people are getting use out of Kilg.us helps keep me motivated to continue improving it!

I haven’t posted about any Kilg.us changes for a couple days, but there has been a little going on. In my defense, I started a “Road to the Show” career in MLB 2010 The Show, so I MIGHT have been spending a little more time honing by batting skills than publishing about Kilg.us changes.

On the fixes side, there were some issues around gathering players and adding them to Kilg.us. Those have been revised now, so any player on a 40-man roster should be showing up (and then some). If you don’t see someone that you want on your team, post a comment or let me know.

There is an outstanding issue with rookies, though. I think I’ve narrowed it to the stored procedure that saves player stats, but all rookie pitchers show a blank (no zeros, no stats, completely blank) stat line regardless of their performance. This is next on the list of things to fix.

Next up, when players were being collected from MLB.com, they were added to the official “Major League Baseball” owner’s teams if they didn’t currently exist on the appropriate roster. They weren’t, however, being removed from their previous team if they had been traded or were a free agent (rather than a rookie). This never seemed to come up last year, but there code I changed to fix the problem hasn’t been touched in nearly a year, so I’m not quite sure the root of the problem. Hopefully it is resolved going forward.

On that note, keep in mind that you can follow any MLB team’s stats by adding the owner “Major League Baseball” to your friends. That particular owner has all 30 teams with rosters that are automatically updated to have all the latest players.

Among new features for 2010, there are three more items I hope to get out by opening day (or very shortly thereafter):

1. Apply the new theme to the …Tracker blog
2. Implement Lucene for Add Player searches
3. Create a Season-to-date view of a team’s stats (cumulative for the season, not respective of adds/removals during the year; this was a request from a Strat-o-matic owner–hopefully others will find use in it)

Opening Day is barely a week away!