Kilg.us – Fantasy Stat Tracker …Tracker

Over the last couple weeks, I have received numerous emails from people having trouble signing-in to their accounts. With a little digging I was able to find a consistent theme: they were all using Internet Explorer 8 or 9.

Specifically what was happening was that a person would type their existing username into the sign-in form and would then be prompted to create an account. As everyone is probably aware, Kilg.us uses the same form for logging in and signing up. If you enter an existing username, it asks you to log-in. If you enter a username that doesn’t exist, it asks you to sign-up.

To manage this functionality, Kilg.us generates an AJAX request after you enter your username then move focus to the password field. This is creates a “change” event on the username field. When that event happens, Kilg.us asks the database whether the username you entered exists. The database returns a response in the form of XML. The XML has one message: the username is valid (it exists in the database) or it is invalid (it doesn’t exist in the database).

Once the XML response is returned, some JavaScript parses it to read whether it says “valid” or “invalid”. If the value is “invalid” the form asks you to create an account. The JavaScript to read this follows standard practices to navigate the document object model (DOM) and find the value in question. This works in Firefox, Chrome, Safari, Konqueror, Mobile Safari, Opera, Android Browser, Blackberry Browser, etc., etc. Apparently, how Internet Explorer handles the XML is different. It doesn’t create all the parameters that the standard approach would.

This shouldn’t come as a surprise to anyone.

What I ultimately found is that Internet Explorer will parse the value out and save it in a “text” parameter for the XML object. Rather than navigating the DOM in a standard fashion, I’m now taking a shortcut and checking if that “text” parameter exists on the XML object. If it does, I read it in as the returned value. If it doesn’t (obviously, all the other browsers to generate this rogue parameter), I continue to navigate the tree and get the value in the proper manner. This only works because the only value of the XML is the “valid” or “invalid” message.

The end result: the log-in/sign-up form logic should now be working again in IE. And still working in every other browser. If you find that not to be the case, please let me know (comment, email, Facebook, Tweet).

I have moved Kilg.us back to referencing the database on the new servers. Once again, things seem to be going well. Hopefully that will still be the case in a couple hours!

I did not attempt to copy over the old database. There is a lot of complexity involved in merging databases and I just didn’t have the patience after yesterday. As a result, if you added or removed players, moved them on or off the bench, or manipulated a team or your account in the last 24 hours, those changes will be lost. You will need to re-apply any changes.

To the couple people that signed up for Kilg.us today–I’m sorry, but you’ll need to do it again! I promise this isn’t a regular occurrence. In fact, I’ve never deleted an account before!

The only outstanding issue that I am aware of from the migration is the appearance of players that are not on the bench. They are rendering as gray and “benched” right now, but the team is still calculating their stats in totals. I’m working on that now and assume it will be a quick fix.

If you notice any other anomalies, please comment here, email me, or post to Facebook or Twitter (@kilgustracker).

**UPDATE 10:40PM**

It appears that batter stats are being collected properly for individuals, but tallied improperly for teams.

It appears that pitcher stats are not being collected properly for individuals, but tallies are correct for teams (based on day-old data).

Still investigating.

**UPDATE 11:19PM**

The culprits seem to be the stored procedures that Kilg.us tries to use for storing stats. All the procedures seem to be failing with the move to the new server. Luckily, Kilg.us included fallback string concatenation mechanisms for exactly this situation. I have disabled all the stored procedure calls and everything now seems to be working.

I intended to migrate Kilg.us to its new servers tomorrow, May 3. If all goes well, the limiting factor on how long Kilg.us will be down is the domain transfer. This can take up to 48 hours, but will go through in 1-3 hours for most people. I intend to take Kilg.us off-line as soon as I get home from work tomorrow (~6pm mountain time). With this approach, I expect the migration and any subsequent troubleshooting should be finished tomorrow night.

Please check Facebook and Twitter (@kilgustracker) for updates on the migration.

I got a couple emails yesterday that some daily boxscore email stats were incorrect. Looking into the matter, it was immediately obvious what had happen. As I worked on the transition of Kilg.us to its new servers, I had accidentally left a database connection pointing to the new dev database, instead of the old, production database. As such, the stats weren’t up-to-date when the daily emails were generated. I simply pointed the connection back to the correct database and figured all was well.

This morning there have again been reports of incorrect stats. I double-checked that the database connection was pointing to the correct IP address–it was. Then I remembered one of the many reasons I’m moving Kilg.us: the hosting company uses a non-standard port for database connections! Why? Who the hell knows. But I had forgotten to explicitly change back to their port. I’ve done that now. I’ve tested the connection. All seems to be well.

On the migration topic, Kilg.us is up and fully functional on its new servers. I’m testing manually with day-to-day usage to try to identify any aspects or configurations I may have missed. Right now, my simple tests are showing that Kilg.us runs between 2.2 and 5 times as fast on the new servers (adjusting as best I can for load considerations). I’m working on some minor changes that will decrease the bandwidth Kilg.us requires because the new servers are billed by data transfer. I’ll give at least 24 hours notice before I make the migration. Depending on how quickly the domain propagates, Kilg.us may be down for some of you for up to 24 hours during the migration. More details to come.

By now, everyone should have received their daily boxscores. If not, please comment below.

We were not without bumps, unfortunately. That I am aware of so far:

• As many a 50 boxscores were sent out multiple times (apologies!)
• The public SMTP I tried to use apparently uses a 24-hour window (rather than a day) for its quotas

The duplicate boxscores was just a goof by me. Last night as I was running my final tests, I commented-out the line that removes an email from the queue after sending because I didn’t want to keep re-entering data into the system before each test. I forgot to un-comment that line when I was done. Because of that, when the system pulled the first 50 boxscores to send, it still left them in the queue. When it went to pull the next 50, it returned the same group. I noticed this after a few times through the process, so hopefully everyone that got duplicates only saw a trickle–not a deluge–of them.

The SMTP issue is frustrating. In my testing, I was able to generate the full list of emails. I assumed the quota would reset at midnight. Apparently that wasn’t the case. So hundreds last night plus hundreds this morning meant the sending account was locked out mid-process. To resolve this, I used one of my personal accounts to complete the send. Many of you will have seen messages come from “matthew@encoredigitalmedia.com” rather than Kilg.us. I figured this was better than making you miss a day’s boxscore.

I have spent most of the last two days trying to get a dedicated Kilg.us SMTP server up and running so I don’t need to worry about stupid quotas and such. Unfortunately, I have absolutely no experience with setting up SMTP and have had just about zero success–hence the use of a public SMTP to get this moving. If anyone knows SMTP and can offer a helping hand, I would be greatly appreciative! Get in touch by email (admin@kilg.us), comment here, or on Twitter (@kilgustracker) or Facebook.

All that said, I think all emails should have been delivered and I have received feedback from some people whose emails weren’t rendering properly before that they look good now. All in all, I’m going to call this one a draw. I’ll keep plugging away today in hopes of a perfect push tomorrow.

Things are proceeding well with the migration of Kilg.us to a new host. The cloud server is up, the web server is installed, database server configured, postfix installed, SSH/SFTP running. I’ve even migrated the codebase with success, although it is all still running on the old Kilg.us database. Tomorrow I hope to get the database copied over and see how things work.

For tomorrow I have moved the CRON job that mails daily boxscores to the new server. An initial test of it was successful. I’m optimistic that tomorrow morning everyone that wants boxscore emails should receive them.

This morning the boxscore email script failed while running. I believe I’ve tracked the problem to a time-out error.

As many of you have surely noticed, Kilg.us is taking MUCH longer to render team pages than it did last year–2-6 times as long in my experience. Each time a team page is rendered, Kilg.us checks to make sure it has the latest stats for each player. If it doesn’t, it pulls in the latest and updates the database. This same process occurs when the daily boxscores are generated. This process is taking much longer than it used to. What used to take a few minutes is now trending over 90.

I’m not sure what I will do about this, yet. As I see it now, there are four options:

1. optimize the current process
2. replace the continuous look-up mechanism with a scheduled process
3. change hosting provider
4. change stats provider

I will likely start with #1, but I hold little hope of its success. #2 would suck as it could mean Kilg.us stats are a few minutes out of date at any given time. I’m also not sure it would actually decrease the load. I would love to do #3 as I’ve never been impressed with Kilg.us’s current host, but better options cost more money and the time to make the transition isn’t insignificant. Number 4 is also a fanciful option as I am yet to find another free source of live stats and paid options cost tens of thousands of dollars.

When I have a solution–or at least a path forward–selected I will post again. Until then…sorry.

Yesterday I started receiving notes that players’ season at-bats in Kilg.us were topping out at 127, even when MLB was reporting a higher value. The reason was a simple oversight in database design. All the numeric values were defined as “tinyint” within the database. This is what is used throughout the daily stats, so it was copied over when the season stats tables were created.

Tinyint is a data-type that only allows for values from -127 to 127. This is great for daily stats (the only daily stat I could imagine passing 127 would be a pitch-count), but doesn’t work so well for season stats. I have changed the data-type to “smallint” for all fields that could exceed 127 in a full-season. Smallint will allow values up to 32767.

Season-to-date stats are collected on a scheduled basis, every 15 minutes. In the next few minutes we should see season-to-date at-bat values correcting themselves. I’ll be monitoring this to ensure it happens so I can diagnose further problems if it doesn’t work.

Thanks to all of you that chimed in with this issue and for providing details so I could correct it quickly!

A comment yesterday brought to my attention that Kilg.us was not showing WHIP values for individual players on their Season-to-Date stats. This was an oversight in an assumption I made about the data Kilg.us was collecting and storing.

Each time Kilg.us renders stats, it checks to see if the latest stats are in the database or if they need to be collected. If the latest ones are in the database, it displays those. If not, it collects the new ones, saves them, and displays them. The XML data from MLB contains the season WHIP stat for each pitcher. So I assumed I was also storing that value in the database.

As it turns out, I wasn’t. The database stores the counting values (innings, hits, walks, etc). Kilg.us  then calculates the average stats (WHIP, ERA, k/9, etc.) based on the counting stats. In the Pitcher object, it was attempting to pull the WHIP value from the dataset returned, which works when collecting new stats from MLB. When the latest stats are in the database, though, that WHIP value doesn’t exist, so “-.–” is displayed instead.

I have updated the Pitcher object to calculate the WHIP value whenever it is needed. This means WHIP should be displaying in Season-to-Date stat views now. If you find otherwise, let me know!