Kilg.us – Fantasy Stat Tracker …Tracker

A blog about the development of Kilg.us – The Fantasy Baseball Stat Tracker

Archive for the ‘Upgrades’ Category

SendGrid Roll-out Successful

Tuesday, April 17th, 2012

Last night I upgraded the email sending mechanism in Kilg.us to use SendGrid. This morning all email boxscores look to have gone out as planned.

As a number people have noted, “as planned” may be less that what we all would like. One of the things SendGrid will help me do is see how many emails are being generated, sent, delivered, read, etc. This will allow me to better manage how daily boxscore emails work.

The first issue to address are those users that have signed up for boxscore emails but haven’t received them. The old email provider had a much lower limit on the number of messages that could be sent per batch and per day. This lead to some less than ideal compromises in designing the system. Basically, rather than just flooding the email server with all the boxscore emails, I had to throttle the process, building a queuing system and sending out small batches of emails. As the system neared its daily limits, some people’s emails would go un-sent.

In addition to the migration to SendGrid last night, this morning I have updated the queuing and sending system. It now queues everything the same, but tomorrow will send one large batch of messages (using an anti-flood function to ensure the email server doesn’t get overwhelmed). This SHOULD ensure that everyone receives every email they have requested! We’ll check back in tomorrow morning to see if it worked.

As a tack on, I switched this WordPress installation to also use SendGrid which should make it less likely that I miss comments and questions people post on here (sorry if I missed your question or comment!).

Moving to SendGrid

Monday, April 16th, 2012

In the next day or so Kilg.us will be moving its email handling to SendGrid. This should give me better transparency into what messages are being send, received, and bounced back. Right now things are a bit of a manual process which makes it difficult to diagnose one-off errors (particularly with daily boxscore emails).

If all goes well, I’ll probably make the change tonight. Theoretically, there should be no change for anyone but me. Let’s hope it works out that way!

New Stat: Total Bases

Thursday, May 5th, 2011

Now that Kilg.us is migrated and–seemingly–stable, I can actually make some improvements!

First up are some stats additions. The easiest request was to include total bases. Because this is calculated based on data we already have, I was able to implement it pretty quickly. To enable Total Bases for any of your teams, visit the “Select Stats” page for that team. You will see a checkbox for “TB.” Checking that box and saving your changes should add a TB column right after strikeouts.

The most commonly requested stats that Kilg.us does not currently have are games played. While this data isn’t conveniently available in the MLB XML data Kilg.us uses, I think I have determined a way to deduce whether a player has played in the day’s game(s) or not. That will take significantly more effort, though, so it will have to wait. Perhaps this weekend I’ll find some time.

Database back up and running

Wednesday, May 4th, 2011

I have moved Kilg.us back to referencing the database on the new servers. Once again, things seem to be going well. Hopefully that will still be the case in a couple hours!

I did not attempt to copy over the old database. There is a lot of complexity involved in merging databases and I just didn’t have the patience after yesterday. As a result, if you added or removed players, moved them on or off the bench, or manipulated a team or your account in the last 24 hours, those changes will be lost. You will need to re-apply any changes.

To the couple people that signed up for Kilg.us today–I’m sorry, but you’ll need to do it again! I promise this isn’t a regular occurrence. In fact, I’ve never deleted an account before!

The only outstanding issue that I am aware of from the migration is the appearance of players that are not on the bench. They are rendering as gray and “benched” right now, but the team is still calculating their stats in totals. I’m working on that now and assume it will be a quick fix.

If you notice any other anomalies, please comment here, email me, or post to Facebook or Twitter (@kilgustracker).

**UPDATE 10:40PM**

It appears that batter stats are being collected properly for individuals, but tallied improperly for teams.

It appears that pitcher stats are not being collected properly for individuals, but tallies are correct for teams (based on day-old data).

Still investigating.

**UPDATE 11:19PM**

The culprits seem to be the stored procedures that Kilg.us tries to use for storing stats. All the procedures seem to be failing with the move to the new server. Luckily, Kilg.us included fallback string concatenation mechanisms for exactly this situation. I have disabled all the stored procedure calls and everything now seems to be working.

Migration Success!

Tuesday, May 3rd, 2011

I’m a little tepid to call the Kilg.us migration a success, but things seem to have worked. The site, blog, databases, scheduled processes, accounts, teams, stats…they’re all moved over. And as best I can tell, they are all working. I’ll be keeping a very close eye on things through the week in case anything goes awry, but we seem to be in good shape for now. If you notice any odd behavior, errors, or encounter problems PLEASE comment here, post to Facebook or Twitter, or send me an email so I can get to the root of things.

Now that the migration is complete, I’ll also finally get to some upgrades! First on the list are some new statistical categories. Please keep in mind that Kilg.us uses a free data source from MLB that only offers limited information. I pull as much out of that source as I can, but some stats just aren’t available. Also on the list is some tweaking to the sign-up/log-in form. I have been receiving quite a few emails about issues with it, most specifically with Internet Explorer 8. If you encounter problem signing-up or logging-in in the meantime, try a browser other than IE8.

Just for fun: Kilg.us now manages over 2300 teams for more than 1300 users! And with the new hosting infrastructure, my testing is showing 3-4 second rendering times for team stat pages compared to 13-23 seconds at the old host. Light speed!

I hope you all continue to enjoy using Kilg.us. Our new hosting should make your stat tracking experience faster and better than ever!

Kilg.us Migration

Monday, May 2nd, 2011

I intended to migrate Kilg.us to its new servers tomorrow, May 3. If all goes well, the limiting factor on how long Kilg.us will be down is the domain transfer. This can take up to 48 hours, but will go through in 1-3 hours for most people. I intend to take Kilg.us off-line as soon as I get home from work tomorrow (~6pm mountain time). With this approach, I expect the migration and any subsequent troubleshooting should be finished tomorrow night.

Please check Facebook and Twitter (@kilgustracker) for updates on the migration.

Cross your fingers!

Friday, April 15th, 2011

Tomorrow morning Kilg.us will use a entirely new process for generating and sending daily boxscore emails! So let’s all cross our fingers and hope it works…

I have spent all of this week re-designing and re-building how emails are generated, stored, and sent. Suffice to say, the new system is significantly more complex than the old one. When I’ve had a chance to breath and am not so tired, I’ll go into more detail on how it works.

My expectations of the new system are that it will:

  • Scale to accommodate as many users/emails as desired (the old system failed in the 200-300 range)
  • Allow of future portability if Kilg.us needs to change hosting
  • Implement all components to identify HTML vs. Text email so boxscores render properly on all clients
  • Allow for side-by-side HTML and Text versions (text version is forthcoming)
  • Provide redundancy of data, so if the process fails it can be re-started without losing a day’s data
  • Reduce likelihood of Kilg.us mail servers being identified as SPAM and black-listed
  • Lower computation overhead

It sounds almost too good to be true! Tomorrow’s push will be the first full run trying to send out to everyone’s unique addresses. All the testing so far has been delivering to a kilg.us address (or small tests to outside domains). I’ll be keeping an eye on things in the morning to try to ensure everything goes smoothly. If you do not receive your normal boxscore in an acceptably legible format, please post comments here with details (DETAILS–without them I can’t do anything–what email client, what team name, what was wrong, screenshots, any little detail).

Update on Boxscore Emails and More

Saturday, April 9th, 2011

This morning on Facebook I mentioned a few goals for fixes and improvements to Kilg.us this weekend. Those items were:

  1. update MLB player/team lists
  2. get email boxscores working
  3. speed up stat collection
  4. trim players that show up on the “Major League Baseball” team rosters (right now it includes players throughout all the minors, with no respect to whether they might actually appear in a real game…

I’m pleased to report that numbers 1 and 4 are soundly completed! As of this morning, all active MLB players should be listed in Kilg.us and associated with their correct teams. If you follow any of the “Major League Baseball” teams, those rosters should also be cleaned up to only show players on that team’s 40-man roster.

Unfortunately, 2 and 3 are stills works in progress. I have made some changes to the model for how player objects are created. The result of this change is that player stats are only refreshed from MLB.com when needed, rather than any time a player is referenced. In practice, loading a team page requires that a player’s stats be up-to-the-second. When a daily boxscore email is generated in the wee hours of the morning, though, there is no need to go gather latest stats–they haven’t changed in the hours since the games all ended.

In my testing so far, team pages are loading 10-40% faster now. I’m still not happy with their performance, but at least it is progress. When I try to generate all the daily emails, those are running significantly faster. Before today, the process was trending towards 90 minutes, timing out, and failing. Now the process is hammering through 80 (of ~360) emails in the first minute and sending emails properly. At that point, the server slows to a crawl and, while it continues to output the email content, it refuses to send the emails.

I considered reaching out to the hosting company for help in identifying why their system was doing this, but I’ve given up. The company (AMS Computer Services) and infrastructure that hosts Kilg.us sucks. I’m going to try moving Kilg.us to a cloud hosting solution from Rackspace. I’ve spent a good part of the day getting the new server up and configured. Tomorrow I should be able to start moving Kilg.us over, testing, troubleshooting, and seeing how it performs. It’s going to be a long, tedious, frustrating process but it can’t possibly be worse than sticking with the status quo.

In the meantime, my fingers are crossed that our daily boxscore emails will fire off properly in the morning…but I’m not holding my breath.

The reason I’ve mentioned in the past for not making the move to a new hosting solution (besides the time and effort) is that it will cost more than the current option. Because Kilg.us is going to cost me more to host now, I’ve added a PayPal donation button to give anyone interested the option in helping to support Kilg.us. I’m not hoping to make anything off Kilg.us, but it would be great it is wasn’t a financial burden. If you like using Kilg.us, I hope you’ll consider giving a little bit to help offset my costs. $5, $10, $1, anything will help!

I hope to have an update with good news by tomorrow evening as I start to migrate Kilg.us.

Games Played/Started Stats

Sunday, April 18th, 2010

Since the release of season-to-date stats, a lot of people have been asking for games played and games started to be included. I’m doing some research to find a reliable place to find such a statistic. If I can find it–and figure out how it plays into a paradigm with daily stats, too–I will incorporate it. The level of effort to include it is pretty minimal, assuming it can be found. More details to come.

Default Team Stat View

Sunday, April 11th, 2010

Kilg.us now retains your last view state for each one of your teams. This serves as a low-complexity mechanism for allowing you to define a default view for your teams–whether it be Today’s stats or Season-to-Date stats.

Each time you switch between viewing daily and season stats for your teams, Kilg.us now makes a record. Which ever view you looked at last will be the view it presents when you return to your team the next time.

This seemed like a simpler interface than adding an additional control to toggle back-and-forth between settings. While a separate control would add slightly more flexibility, I think this mechanism will be just as effective and require less effort for both users and myself.

So if you always want to see Season-to-Date stats for your team, all you have to do is click over to that tab the first time, and for the rest of eternity you’ll see Season-to-Date stats (until you manually switch back to daily stats). If you only ever want to see daily stats, you actually don’t have to do anything (daily stats are the default until you select otherwise).

To accomplish this, the Teams table in the database was extended to include a “View” field. In addition, the Team object was extended to allow for setting and getting that View value. Each time you access a team page, the page checks for a view variable in the query-string (these are what the “Today” and “Season-to-Date” tabs use to set the view state). If that variable exists, it updates the default setting for the team. If it doesn’t exist if asks the Team object what the default state is and uses that instead.