Kilg.us – Fantasy Stat Tracker …Tracker

I intended to migrate Kilg.us to its new servers tomorrow, May 3. If all goes well, the limiting factor on how long Kilg.us will be down is the domain transfer. This can take up to 48 hours, but will go through in 1-3 hours for most people. I intend to take Kilg.us off-line as soon as I get home from work tomorrow (~6pm mountain time). With this approach, I expect the migration and any subsequent troubleshooting should be finished tomorrow night.

Please check Facebook and Twitter (@kilgustracker) for updates on the migration.

By now, everyone should have received their daily boxscores. If not, please comment below.

We were not without bumps, unfortunately. That I am aware of so far:

• As many a 50 boxscores were sent out multiple times (apologies!)
• The public SMTP I tried to use apparently uses a 24-hour window (rather than a day) for its quotas

The duplicate boxscores was just a goof by me. Last night as I was running my final tests, I commented-out the line that removes an email from the queue after sending because I didn’t want to keep re-entering data into the system before each test. I forgot to un-comment that line when I was done. Because of that, when the system pulled the first 50 boxscores to send, it still left them in the queue. When it went to pull the next 50, it returned the same group. I noticed this after a few times through the process, so hopefully everyone that got duplicates only saw a trickle–not a deluge–of them.

The SMTP issue is frustrating. In my testing, I was able to generate the full list of emails. I assumed the quota would reset at midnight. Apparently that wasn’t the case. So hundreds last night plus hundreds this morning meant the sending account was locked out mid-process. To resolve this, I used one of my personal accounts to complete the send. Many of you will have seen messages come from “matthew@encoredigitalmedia.com” rather than Kilg.us. I figured this was better than making you miss a day’s boxscore.

I have spent most of the last two days trying to get a dedicated Kilg.us SMTP server up and running so I don’t need to worry about stupid quotas and such. Unfortunately, I have absolutely no experience with setting up SMTP and have had just about zero success–hence the use of a public SMTP to get this moving. If anyone knows SMTP and can offer a helping hand, I would be greatly appreciative! Get in touch by email (admin@kilg.us), comment here, or on Twitter (@kilgustracker) or Facebook.

All that said, I think all emails should have been delivered and I have received feedback from some people whose emails weren’t rendering properly before that they look good now. All in all, I’m going to call this one a draw. I’ll keep plugging away today in hopes of a perfect push tomorrow.

As many of you experienced, this morning the Kilg.us daily boxscores went horribly wrong and began sending piles of duplicate mails to everyone. If I haven’t apologized to you yet: I’m sorry. I try very hard to make sure any changes I make to Kilg.us won’t have an adverse effect on users. I bungled this one, but at least I think I’ve figured out what happened.

I mentioned previously that I moved the daily boxscore email script to the new Kilg.us server. That is why emails went out to anyone. On the new server PHP is installed differently from the old server. As a result, when I run the CRON job to generate the emails, I needed to use an application to call the script rather than just calling PHP to execute it. I chose to use a program called wget. The idea is that wget makes a call to a URL and that URL (a PHP file) generates and sends the emails. Before scheduling the morning’s emails, I tested to ensure the process worked. When I tested, though, I only used a sub-set of data. I didn’t really need 350 emails coming into my inbox, so I tested with a couple emails each for those sent to a team owner and those sent to a team viewer. That worked great.

When the CRON job ran this morning, everything seemed to go well until duplicates started showing up. A second round, then a third, then a fourth and so on. Interestingly, each wave was 15 minutes apart. As it turns out, if wget can’t complete a request (in this case a VERY long request for 350 emails), it tries again. By default it will try up to 20 times to fetch a file. Because the script takes so long to run, I believe it exceeded the server time-out.  When the script timed out, wget requested it a second time, then a third, then a fourth and so on. I believe this is what caused the duplicate emails.

I’ve made three adjustments to address this. First, I increased the server time-out for this process. Second, I have changed the CRON job to tell wget never to retry the fetch if it fails. Third, I’ve broken the massive emails process into multiple chunks. This is a temporary fix until I convert to using PEAR::Mail to more intelligently manage the process. I’m done for tonight, but that will probably be the priority tomorrow.

So that’s all of today’s work. Six hours sunk, but I think the emails will work in the morning. I’ve pushed back the time that the emails go out by a couple hours to they’ll line up with when I roll out of bed in the morning. If things go off-track, I’ll be able to curtail things faster than today. If all goes well, I’ll move the schedule back to the early morning hour so everyone has the boxscore email when they get up in the morning.

Keep your fingers crossed…

On the positive side, email boxscores finally went out this morning. And based on the number of emails I’ve received, they reached a lot of people! Unfortunately, after the initial batch they continued to send another 6 or 7 times. I believe I have that process stopped.

At first glance, I don’t know why this happened. The script that sends the emails is unchanged from what used to work (just migrated to the new server). That would seem to suggest there was a problem with the CRON job scheduling. I’ve deleted all CRON jobs from the server and will be doing some more research to try to determine what went wrong before I re-schedule the script.

I’m sorry to all of you that were flooded by emails this morning. I’ll do my best to ensure it doesn’t happen again.

Things are proceeding well with the migration of Kilg.us to a new host. The cloud server is up, the web server is installed, database server configured, postfix installed, SSH/SFTP running. I’ve even migrated the codebase with success, although it is all still running on the old Kilg.us database. Tomorrow I hope to get the database copied over and see how things work.

For tomorrow I have moved the CRON job that mails daily boxscores to the new server. An initial test of it was successful. I’m optimistic that tomorrow morning everyone that wants boxscore emails should receive them.

No new features on Kilg.us today, but hopefully the existing features are working a little better.

If you were online tracking your team tonight and had any Rangers or Athletics on your team, you may have noticed some odd behavior. It took a couple hours, but I think double-header stats should all be compiling correctly. I ended up working through a series of iterations over the process of collecting all the stats between games 1 and 2, adding them, and being sure not to collect stats over and over if they haven’t changed.

It was also pointed out this morning that, if a team didn’t have any players on it, errors would be thrown on the Dashboard and Team pages. That issue has been resolved. The issue manifested from an error in the team creation process. A team created with an apostrophe in it’s name would throw an error. The team would not be created and an empty, unusable team would be associated with the Owner. I deleted all those blank teams so if you used to have one on your Dashboard, hopefully it’s gone now. You can now create teams with apostrophes in their names.

Finally, when approving an Industry request, a 404 error (page not found) was reported. Turns out a typo had found it’s way into the path to the script. That has been fixed. I had noticed no new industries in the last couple days, though, so apologies to anyone else who encountered the 404 error in the last couple days.

If you encounter any errors or unexpect behavior while you’re using Kilg.us, please shoot me an email with a description so I can get it fixed up. Thanks to everyone who helped identify the bugs today!

With 2009 Spring Training fast approaching, I’ve begun work to improve the Fantasy Baseball Stattracker for the new season. As you’ve probably already noticed, the Stattracker has a name: Kilg.us! There is a brief overview of where the name came from on the site. I’ll likely move that over to the blog eventually.

Besides the name change and subsequent move to Kilg.us, there are a number of back-end upgrades already in place.

1. The database has been upgraded to a newer version that allows for stronger password encryption.
2. A new login script has been added that takes advantage of the stronger password encryption so as soon as you log in, your old account will be upgraded with the new encryption.
3. This blog is now running the latest WorkPress – 2.7. Between the upgrade and the server change there were a few hurdles to overcome, but I think everything is up and running.

Among the front-end changes I anticipate before the 2009 season starts are improvements to the account sign-up process, SSL implementation of the sign-up and log-on processes, and an iPhone-optimized web application. I’d like to build a real iPhone app for Kilg.us, but with no budget I probably won’t try to tackle the $99 iPhone developer fee.

If there are other improvements that would make the site better for you and your friends, please post a comment and I’ll see what I can do.

Here’s to an exciting 2009!

It’s been a little while since I’ve made any changes to the Fantasy Stat Tracker, but I had a few minutes today and thought of a great improvement. Previously, when viewing your team you could identify which players were on the “bench” for a given day. Unfortunately, that designation as a “bench” player was specific to the current, loaded instance of the Team page. When you reloaded the page (to see updated stats), you lost your bench designations and needed to re-check your bench players.

As of today, that is no longer the case. When you tag a player as being on the bench, they will stay on the bench until you explicitly remove them. After benching a player, when you reload the page they will still be on the bench. If you log-out and log back in tomorrow, they’ll still be on the bench. Until you un-check the little “bench” box next to the player’s name, their stats won’t count in your calculated totals.

This, for me, is probably the biggest annoyance I’ve had with the Stat Tracker, so I’m quite pleased it is resolved. Hopefully this will be helpful to everyone else using the tool as well.

As an added bonus, when a player is on the bench they will automatically be dropped to the bottom of the player list so their stats are easily ignored as you scroll through your other players.

Today I implemented a couple minor updates to the Fantasy Stat Tracker. The first was a slight re-organization of the “Select Stats” page. Previously the page had been a long list of all the stats available, batting stats stacked upon pitching stats. In order to shorten the page and reduce the amount of scolling needed, batting and pitching stats now sit side-by-side on the page. The button to submit your selections has been moved from the left to the right side of the page to facilitate forward motion.

The second update today was to the mechanism for calculating when to grab updated stats from mlb.com and when to use the stored stats from the database. Specifically it surrounds the Date/Time functions I was using. The function to check when the remote stats file was updated returns a Date that looks something like “Thu, 12 Jun 2008 00:47:18 GMT”. Previously I’d been doing a simple comparison of times based on that format. As long as the Date/Time values that are being compared are from the same day, this works fine. The complication, however, occurs when the day rolls over. And, because these dates use GMT, the day rolls over in the early evening here in Denver (and elsewhere in the USA). From Monday (Mon) to Tuesday (Tue) this works fine (“Tue” follows “Mon” alphabetically, so a stat file starting “Tue…” will be recognized as newer than a stat file starting “Mon…”). So Monday to Tuesday, Tuesday to Wednesday, Friday to Saturday, and Saturday to Sunday work fine. For the other day changes, though, the later day is earlier alphabetically then the former day, so the application determines that the out-dated stats in the database are actually the more recent one.

To work around this, I included a function that can read the alpha-numeric date format mentioned above and convert it to a purely numeric form. By converting to pure numbers, the problem of day names is removed and the application can determine—to the second—which set of stats is newer. This will resolve any odd behaviors you may have noticed when some of your players’ stats stop updating.