Kilg.us – Fantasy Stat Tracker …Tracker

I’m a little tepid to call the Kilg.us migration a success, but things seem to have worked. The site, blog, databases, scheduled processes, accounts, teams, stats…they’re all moved over. And as best I can tell, they are all working. I’ll be keeping a very close eye on things through the week in case anything goes awry, but we seem to be in good shape for now. If you notice any odd behavior, errors, or encounter problems PLEASE comment here, post to Facebook or Twitter, or send me an email so I can get to the root of things.

Now that the migration is complete, I’ll also finally get to some upgrades! First on the list are some new statistical categories. Please keep in mind that Kilg.us uses a free data source from MLB that only offers limited information. I pull as much out of that source as I can, but some stats just aren’t available. Also on the list is some tweaking to the sign-up/log-in form. I have been receiving quite a few emails about issues with it, most specifically with Internet Explorer 8. If you encounter problem signing-up or logging-in in the meantime, try a browser other than IE8.

Just for fun: Kilg.us now manages over 2300 teams for more than 1300 users! And with the new hosting infrastructure, my testing is showing 3-4 second rendering times for team stat pages compared to 13-23 seconds at the old host. Light speed!

I hope you all continue to enjoy using Kilg.us. Our new hosting should make your stat tracking experience faster and better than ever!

Tomorrow morning Kilg.us will use a entirely new process for generating and sending daily boxscore emails! So let’s all cross our fingers and hope it works…

I have spent all of this week re-designing and re-building how emails are generated, stored, and sent. Suffice to say, the new system is significantly more complex than the old one. When I’ve had a chance to breath and am not so tired, I’ll go into more detail on how it works.

My expectations of the new system are that it will:

• Scale to accommodate as many users/emails as desired (the old system failed in the 200-300 range)
• Allow of future portability if Kilg.us needs to change hosting
• Implement all components to identify HTML vs. Text email so boxscores render properly on all clients
• Allow for side-by-side HTML and Text versions (text version is forthcoming)
• Provide redundancy of data, so if the process fails it can be re-started without losing a day’s data
• Reduce likelihood of Kilg.us mail servers being identified as SPAM and black-listed
• Lower computation overhead

It sounds almost too good to be true! Tomorrow’s push will be the first full run trying to send out to everyone’s unique addresses. All the testing so far has been delivering to a kilg.us address (or small tests to outside domains). I’ll be keeping an eye on things in the morning to try to ensure everything goes smoothly. If you do not receive your normal boxscore in an acceptably legible format, please post comments here with details (DETAILS–without them I can’t do anything–what email client, what team name, what was wrong, screenshots, any little detail).

I’ve been frantically working away on a new email system for Kilg.us the last couple days. The goal is to streamline the process, reduce execution time, reduce likelihood for failure, and improve the quality of the email delivered to your inbox. I’m about 20 hours into the effort, but spent 6 hours going in circles last night…very frustrating. As is often the case, I slept on it and have some new ideas this morning, so I’m again optimistic that I’ll be rolling it out in the coming days. Check back for updates.

In the mean time, a number of people have been reporting that their emails are not being displayed as HTML in their email clients. I haven’t tried to troubleshoot this as I am replacing the system anyway. If you are experiencing this, it wouldn’t hurt to comment with what email client you use (Thunderbird, Outlook, Gmail, iPhone Mail, etc.). We might notice a pattern and be able to share some advice with one another.

As far as I know, emails have gone out properly for three straight days now. If you’re not seeing them, leave a comment, ping me on twitter (@kilgustracker), or shoot me an email (admin@kilg.us).

As many of you experienced, this morning the Kilg.us daily boxscores went horribly wrong and began sending piles of duplicate mails to everyone. If I haven’t apologized to you yet: I’m sorry. I try very hard to make sure any changes I make to Kilg.us won’t have an adverse effect on users. I bungled this one, but at least I think I’ve figured out what happened.

I mentioned previously that I moved the daily boxscore email script to the new Kilg.us server. That is why emails went out to anyone. On the new server PHP is installed differently from the old server. As a result, when I run the CRON job to generate the emails, I needed to use an application to call the script rather than just calling PHP to execute it. I chose to use a program called wget. The idea is that wget makes a call to a URL and that URL (a PHP file) generates and sends the emails. Before scheduling the morning’s emails, I tested to ensure the process worked. When I tested, though, I only used a sub-set of data. I didn’t really need 350 emails coming into my inbox, so I tested with a couple emails each for those sent to a team owner and those sent to a team viewer. That worked great.

When the CRON job ran this morning, everything seemed to go well until duplicates started showing up. A second round, then a third, then a fourth and so on. Interestingly, each wave was 15 minutes apart. As it turns out, if wget can’t complete a request (in this case a VERY long request for 350 emails), it tries again. By default it will try up to 20 times to fetch a file. Because the script takes so long to run, I believe it exceeded the server time-out.  When the script timed out, wget requested it a second time, then a third, then a fourth and so on. I believe this is what caused the duplicate emails.

I’ve made three adjustments to address this. First, I increased the server time-out for this process. Second, I have changed the CRON job to tell wget never to retry the fetch if it fails. Third, I’ve broken the massive emails process into multiple chunks. This is a temporary fix until I convert to using PEAR::Mail to more intelligently manage the process. I’m done for tonight, but that will probably be the priority tomorrow.

So that’s all of today’s work. Six hours sunk, but I think the emails will work in the morning. I’ve pushed back the time that the emails go out by a couple hours to they’ll line up with when I roll out of bed in the morning. If things go off-track, I’ll be able to curtail things faster than today. If all goes well, I’ll move the schedule back to the early morning hour so everyone has the boxscore email when they get up in the morning.

Keep your fingers crossed…

This evening I completed work to allow Owners to track their Kilg.us teams’ season-to-date statistics. This has been a popular request throughout the past year.  Your team’s season-to-date stats are now available from each Team page by clicking on the “Season-to-Date” tab just below your team name. Clicking the “Today” tab will return you to your live, up-to-the-minute daily stats.

Season-to-Date statistics are for the players CURRENTLY on your roster. Kilg.us does not track the changes you make to your roster throughout the season. The Season-to-Date statistics center around a Strat-O-Matic scoring model. I’ve never played Strat-O-Matic, but–as described to me–roster moves aren’t allowed during the season. Rosters are locked in before the season and only change in the off-season.

For those playing in a standard fantasy baseball league, these Season-to-Date stats will be different than your fantasy team’s total stats (accumulating stats for players in your lineup on any given day). For Owners in this situation, season-to-date stats will make it a touch easier to view all your players’ season performance, but won’t be of direct use when tracking your team’s performance.

Adding Season-to-Date stats wasn’t a trivial matter. I had to make changes to more than a half-dozen of Kilg.us’ most important objects. In addition to changes to Kilg.us objects, the database was also expanded to include tables to store the season stats for players. At the moment, the objects are writing to the database via SQL statements. In the coming days I’ll update to use stored procedures like the rest of the database communications.

I have tried to test for unexpected repercussions from the object and database changes, but please let me know if you encounter any odd behavior by posting a comment or emailing me.

Looks like last night’s 2010 Kilg.us re-launch wasn’t spotless–the script to generate and send email box scores didn’t run correctly this morning. Part of the new 2010 efforts included re-structuring the organization of classes in the file system to follow more standard practices. It looks like the script in question was missing a reference to the class it needed to determine the date for which it should be sending the box scores.

I’ve updated the script and expect it to run tomorrow without a hitch. I don’t intend to run the script to generate box score emails today because it is still spring training. If you have an urgent and dire need for a box score email, let me know and I’ll see what can be done.

Tomorrow (3/17) evening I plan to launch the 2010 updates for Kilg.us. Launch will probably commence around 10pm Mountain time. The launch will replace the entire Kilg.us code base with a new iteration. This will likely result in at least a few minutes–and as much as a few hours–of downtime.

I will blog and tweet (@kilgusTracker) updates to announce when the launch is completed.

In addition to the Kilg.us updates, I will be publishing a series of blog posts to describe the changes and how to use the new features of Kilg.us. For current users, you will find the interaction mechanisms to be very familiar. A new visual style is the most disruptive change. As the season progresses I’ll be getting back to refining the functionality of the tool

I’m in the process of implementing a series of updates to Kilg.us for the 2010 season. The biggest noticeable difference will be an entire new look and feel. The new layout will allow for 1, 2, and 3-column pages that should let Kilg.us more efficiently use space.

Other changes in progress include:

• Integrated log-in/sign-up function
• User-defined backgrounds
• More flexibility in the “invite friends” functionality
• Updated log-in flow
• Drag-and-drop bench
• Integration with Zend framework

Among the other items I’m planning to get to before opening day:

• Add OPS to statistics available for tracking
• Implement Lucene search for “Find Friends”
• SSL for log-in flow

If you have other suggestions, please add a comment or drop me a long at admin@kilg.us

I’ll go into more detail about the updated features over the coming weeks. In the mean time, I’m just trying to get things done before the season kicks-off!

One of the longest running requests of Kilg.us has been to include game progress information. I can definitely see the value in knowing whether the game your pitcher was pitching in is done yet or not (will there be a win coming or is it all over already?). This is a different kind of statistic than what Kilg.us otherwise collects, so it has been low on the priority list. As I got tired of stored procedures earlier today, though, it finally sounded like an appealing project.

The scoreboard is displayed across the top of the page in the banner area. It displays abbreviated linescores: team, runs, hits, errors, and inning (or “F” if Final). The banner area only has room for 8 scores at a time, so there are left and right arrows to slide the scoreboard side-to-side to view all games.

The first draft of the implementation pulls XML data from MLB each time a page is loaded that displays the scoreboard. Currently the Kilg.us homepage, your Dashboard, and all Team pages include the scoreboard. The data is collected from the XML, looped through, and populated into a list/table combination of mark-up. Some JQuery sets up click events on the arrow buttons when the page loads (if they are needed; if 8 or less games are occuring that day, no arrows will appear) and calculates how far the scoreboard needs to slide to display all games.

I only had time to perform a very quick testing on IE8, Safari, and Chrome (in addition to Firefox, on which all my development occurs) so if you experience any problems with your browser, please let me know.

More development will be ongoing with the scoreboard. I expect I will update the process to store the game scores so Kilg.us doesn’t need to continue pulling in game scores even after all game are done. It will function similar to how player stats are collected. It will first check if the XML feed has been updated. If not, it will pull data from the database. If so, it will update the database and use those stats.

A lot of my weekend was spent making substantially un-noticable upgrades to Kilg.us. Most of the time was spent stepping through my objects and making minor improvements as I noticed opportunities. Among the changes, on-base percentages should be calculated more accurately now. Previously Kilg.us was checking for the existence of at least one at-bat before calculating to ensure no division by zero. Obviously, if a player starts off his day with a walk or two, this would result in no OBP calculation. The code now correctly checks for either at-bats OR walks before calculating.

The second collection of behind-the-scenes changes was creation of a new series of stored procedures. Kilg.us uses 8 different methods to save and update batter and pitchers statistics. The new stored procedures help streamline the process and reduce the amount of SQL (structured query language–the programming language used to manipulate databases) included in-line with the PHP code. Now those 8 methods can use the same set of stored procedures to interact with the database instead of each doing the interaction their own unique way. These changes probably won’t improve performance substantially, but they do abstract the code a little further which will help with maintenance and future improvements.