Kilg.us – Fantasy Stat Tracker …Tracker

Now that Kilg.us is migrated and–seemingly–stable, I can actually make some improvements!

First up are some stats additions. The easiest request was to include total bases. Because this is calculated based on data we already have, I was able to implement it pretty quickly. To enable Total Bases for any of your teams, visit the “Select Stats” page for that team. You will see a checkbox for “TB.” Checking that box and saving your changes should add a TB column right after strikeouts.

The most commonly requested stats that Kilg.us does not currently have are games played. While this data isn’t conveniently available in the MLB XML data Kilg.us uses, I think I have determined a way to deduce whether a player has played in the day’s game(s) or not. That will take significantly more effort, though, so it will have to wait. Perhaps this weekend I’ll find some time.

I have moved Kilg.us back to referencing the database on the new servers. Once again, things seem to be going well. Hopefully that will still be the case in a couple hours!

I did not attempt to copy over the old database. There is a lot of complexity involved in merging databases and I just didn’t have the patience after yesterday. As a result, if you added or removed players, moved them on or off the bench, or manipulated a team or your account in the last 24 hours, those changes will be lost. You will need to re-apply any changes.

To the couple people that signed up for Kilg.us today–I’m sorry, but you’ll need to do it again! I promise this isn’t a regular occurrence. In fact, I’ve never deleted an account before!

The only outstanding issue that I am aware of from the migration is the appearance of players that are not on the bench. They are rendering as gray and “benched” right now, but the team is still calculating their stats in totals. I’m working on that now and assume it will be a quick fix.

If you notice any other anomalies, please comment here, email me, or post to Facebook or Twitter (@kilgustracker).

**UPDATE 10:40PM**

It appears that batter stats are being collected properly for individuals, but tallied improperly for teams.

It appears that pitcher stats are not being collected properly for individuals, but tallies are correct for teams (based on day-old data).

Still investigating.

**UPDATE 11:19PM**

The culprits seem to be the stored procedures that Kilg.us tries to use for storing stats. All the procedures seem to be failing with the move to the new server. Luckily, Kilg.us included fallback string concatenation mechanisms for exactly this situation. I have disabled all the stored procedure calls and everything now seems to be working.

I’m a little tepid to call the Kilg.us migration a success, but things seem to have worked. The site, blog, databases, scheduled processes, accounts, teams, stats…they’re all moved over. And as best I can tell, they are all working. I’ll be keeping a very close eye on things through the week in case anything goes awry, but we seem to be in good shape for now. If you notice any odd behavior, errors, or encounter problems PLEASE comment here, post to Facebook or Twitter, or send me an email so I can get to the root of things.

Now that the migration is complete, I’ll also finally get to some upgrades! First on the list are some new statistical categories. Please keep in mind that Kilg.us uses a free data source from MLB that only offers limited information. I pull as much out of that source as I can, but some stats just aren’t available. Also on the list is some tweaking to the sign-up/log-in form. I have been receiving quite a few emails about issues with it, most specifically with Internet Explorer 8. If you encounter problem signing-up or logging-in in the meantime, try a browser other than IE8.

Just for fun: Kilg.us now manages over 2300 teams for more than 1300 users! And with the new hosting infrastructure, my testing is showing 3-4 second rendering times for team stat pages compared to 13-23 seconds at the old host. Light speed!

I hope you all continue to enjoy using Kilg.us. Our new hosting should make your stat tracking experience faster and better than ever!

I intended to migrate Kilg.us to its new servers tomorrow, May 3. If all goes well, the limiting factor on how long Kilg.us will be down is the domain transfer. This can take up to 48 hours, but will go through in 1-3 hours for most people. I intend to take Kilg.us off-line as soon as I get home from work tomorrow (~6pm mountain time). With this approach, I expect the migration and any subsequent troubleshooting should be finished tomorrow night.

Please check Facebook and Twitter (@kilgustracker) for updates on the migration.

Tomorrow morning Kilg.us will use a entirely new process for generating and sending daily boxscore emails! So let’s all cross our fingers and hope it works…

I have spent all of this week re-designing and re-building how emails are generated, stored, and sent. Suffice to say, the new system is significantly more complex than the old one. When I’ve had a chance to breath and am not so tired, I’ll go into more detail on how it works.

My expectations of the new system are that it will:

• Scale to accommodate as many users/emails as desired (the old system failed in the 200-300 range)
• Allow of future portability if Kilg.us needs to change hosting
• Implement all components to identify HTML vs. Text email so boxscores render properly on all clients
• Allow for side-by-side HTML and Text versions (text version is forthcoming)
• Provide redundancy of data, so if the process fails it can be re-started without losing a day’s data
• Reduce likelihood of Kilg.us mail servers being identified as SPAM and black-listed
• Lower computation overhead

It sounds almost too good to be true! Tomorrow’s push will be the first full run trying to send out to everyone’s unique addresses. All the testing so far has been delivering to a kilg.us address (or small tests to outside domains). I’ll be keeping an eye on things in the morning to try to ensure everything goes smoothly. If you do not receive your normal boxscore in an acceptably legible format, please post comments here with details (DETAILS–without them I can’t do anything–what email client, what team name, what was wrong, screenshots, any little detail).

This morning on Facebook I mentioned a few goals for fixes and improvements to Kilg.us this weekend. Those items were:

1. update MLB player/team lists
2. get email boxscores working
3. speed up stat collection
4. trim players that show up on the “Major League Baseball” team rosters (right now it includes players throughout all the minors, with no respect to whether they might actually appear in a real game…

I’m pleased to report that numbers 1 and 4 are soundly completed! As of this morning, all active MLB players should be listed in Kilg.us and associated with their correct teams. If you follow any of the “Major League Baseball” teams, those rosters should also be cleaned up to only show players on that team’s 40-man roster.

Unfortunately, 2 and 3 are stills works in progress. I have made some changes to the model for how player objects are created. The result of this change is that player stats are only refreshed from MLB.com when needed, rather than any time a player is referenced. In practice, loading a team page requires that a player’s stats be up-to-the-second. When a daily boxscore email is generated in the wee hours of the morning, though, there is no need to go gather latest stats–they haven’t changed in the hours since the games all ended.

In my testing so far, team pages are loading 10-40% faster now. I’m still not happy with their performance, but at least it is progress. When I try to generate all the daily emails, those are running significantly faster. Before today, the process was trending towards 90 minutes, timing out, and failing. Now the process is hammering through 80 (of ~360) emails in the first minute and sending emails properly. At that point, the server slows to a crawl and, while it continues to output the email content, it refuses to send the emails.

I considered reaching out to the hosting company for help in identifying why their system was doing this, but I’ve given up. The company (AMS Computer Services) and infrastructure that hosts Kilg.us sucks. I’m going to try moving Kilg.us to a cloud hosting solution from Rackspace. I’ve spent a good part of the day getting the new server up and configured. Tomorrow I should be able to start moving Kilg.us over, testing, troubleshooting, and seeing how it performs. It’s going to be a long, tedious, frustrating process but it can’t possibly be worse than sticking with the status quo.

In the meantime, my fingers are crossed that our daily boxscore emails will fire off properly in the morning…but I’m not holding my breath.

The reason I’ve mentioned in the past for not making the move to a new hosting solution (besides the time and effort) is that it will cost more than the current option. Because Kilg.us is going to cost me more to host now, I’ve added a PayPal donation button to give anyone interested the option in helping to support Kilg.us. I’m not hoping to make anything off Kilg.us, but it would be great it is wasn’t a financial burden. If you like using Kilg.us, I hope you’ll consider giving a little bit to help offset my costs. $5, $10, $1, anything will help!

I hope to have an update with good news by tomorrow evening as I start to migrate Kilg.us.

Since the release of season-to-date stats, a lot of people have been asking for games played and games started to be included. I’m doing some research to find a reliable place to find such a statistic. If I can find it–and figure out how it plays into a paradigm with daily stats, too–I will incorporate it. The level of effort to include it is pretty minimal, assuming it can be found. More details to come.

Kilg.us now retains your last view state for each one of your teams. This serves as a low-complexity mechanism for allowing you to define a default view for your teams–whether it be Today’s stats or Season-to-Date stats.

Each time you switch between viewing daily and season stats for your teams, Kilg.us now makes a record. Which ever view you looked at last will be the view it presents when you return to your team the next time.

This seemed like a simpler interface than adding an additional control to toggle back-and-forth between settings. While a separate control would add slightly more flexibility, I think this mechanism will be just as effective and require less effort for both users and myself.

So if you always want to see Season-to-Date stats for your team, all you have to do is click over to that tab the first time, and for the rest of eternity you’ll see Season-to-Date stats (until you manually switch back to daily stats). If you only ever want to see daily stats, you actually don’t have to do anything (daily stats are the default until you select otherwise).

To accomplish this, the Teams table in the database was extended to include a “View” field. In addition, the Team object was extended to allow for setting and getting that View value. Each time you access a team page, the page checks for a view variable in the query-string (these are what the “Today” and “Season-to-Date” tabs use to set the view state). If that variable exists, it updates the default setting for the team. If it doesn’t exist if asks the Team object what the default state is and uses that instead.

Earlier this Spring I changed the search function for finding friends and creating Industries. The biggest change was moving to a system that uses the Lucene search engine. Part of that engine requires that an index be built for the data to be searched. This was my first implementation of Lucene, so I didn’t have experience with maintaining that index of data. As it turns out, each time the data within the database changes, the index needs to be rebuilt to include all the correct information.

Initially, I wrote a short script to build the index. As more and more people signed-up for Kilg.us, though, the data became out of date because it wasn’t automatically re-indexing.

I have updated the processes for creating an account and changing user settings to display information publicly. Both processes now include a step that re-indexes user data. This should ensure that the user index is always up-to-date and all users with their information shared will be findable in the Find Friends search.

I haven’t posted about any Kilg.us changes for a couple days, but there has been a little going on. In my defense, I started a “Road to the Show” career in MLB 2010 The Show, so I MIGHT have been spending a little more time honing by batting skills than publishing about Kilg.us changes.

On the fixes side, there were some issues around gathering players and adding them to Kilg.us. Those have been revised now, so any player on a 40-man roster should be showing up (and then some). If you don’t see someone that you want on your team, post a comment or let me know.

There is an outstanding issue with rookies, though. I think I’ve narrowed it to the stored procedure that saves player stats, but all rookie pitchers show a blank (no zeros, no stats, completely blank) stat line regardless of their performance. This is next on the list of things to fix.

Next up, when players were being collected from MLB.com, they were added to the official “Major League Baseball” owner’s teams if they didn’t currently exist on the appropriate roster. They weren’t, however, being removed from their previous team if they had been traded or were a free agent (rather than a rookie). This never seemed to come up last year, but there code I changed to fix the problem hasn’t been touched in nearly a year, so I’m not quite sure the root of the problem. Hopefully it is resolved going forward.

On that note, keep in mind that you can follow any MLB team’s stats by adding the owner “Major League Baseball” to your friends. That particular owner has all 30 teams with rosters that are automatically updated to have all the latest players.

Among new features for 2010, there are three more items I hope to get out by opening day (or very shortly thereafter):

1. Apply the new theme to the …Tracker blog
2. Implement Lucene for Add Player searches
3. Create a Season-to-date view of a team’s stats (cumulative for the season, not respective of adds/removals during the year; this was a request from a Strat-o-matic owner–hopefully others will find use in it)

Opening Day is barely a week away!