Kilg.us – Fantasy Stat Tracker …Tracker

A blog about the development of Kilg.us – The Fantasy Baseball Stat Tracker

Posts Tagged ‘hosting’

Server Issues this Morning

Sunday, August 25th, 2013

This morning a lot of people awoke to find Kilg.us was kicking out errors and was showing all players had been lost from all team. Frankly, not a nice way to start the morning!

Luckily, the issue was resolved with a quick reboot of the server. It seems to have been an issue with one of the database tables, but I’ll be performing some additional diagnostics to try to determine what caused all the errors this morning. For now things should be back up and running.

Sorry for the inconvenience!

That’s a wrap!

Monday, October 29th, 2012

The 2012 baseball season has come to a close. As such, Kilg.us will no longer be sending out daily box score emails.

It’s been a tumultuous year for Kilg.us. In particular, we experienced a lot of accuracy issues with our data source (free MLB.com published XML feeds) and I heard more than ever about the lack of this stat or that stat.

I stopped using Kilg.us for my personal needs a couple years ago (I originally built it as a personal tool). Going into 2013, I’m on the fence as to whether to continue maintaining Kilg.us. Obviously there are costs associated with keeping the servers up and running (thanks to all of you that have donated to help cover hosting costs!), but even more important is the time it takes to field email questions and comments, fix things when they go wrong, and monitor the performance of servers, databases, and email services. We’ll see how that goes in a few months when the excitement of a new season ramps up.

In the meantime, keep your chins up. Baseball is only a few cold months away!

Migration Success!

Tuesday, May 3rd, 2011

I’m a little tepid to call the Kilg.us migration a success, but things seem to have worked. The site, blog, databases, scheduled processes, accounts, teams, stats…they’re all moved over. And as best I can tell, they are all working. I’ll be keeping a very close eye on things through the week in case anything goes awry, but we seem to be in good shape for now. If you notice any odd behavior, errors, or encounter problems PLEASE comment here, post to Facebook or Twitter, or send me an email so I can get to the root of things.

Now that the migration is complete, I’ll also finally get to some upgrades! First on the list are some new statistical categories. Please keep in mind that Kilg.us uses a free data source from MLB that only offers limited information. I pull as much out of that source as I can, but some stats just aren’t available. Also on the list is some tweaking to the sign-up/log-in form. I have been receiving quite a few emails about issues with it, most specifically with Internet Explorer 8. If you encounter problem signing-up or logging-in in the meantime, try a browser other than IE8.

Just for fun: Kilg.us now manages over 2300 teams for more than 1300 users! And with the new hosting infrastructure, my testing is showing 3-4 second rendering times for team stat pages compared to 13-23 seconds at the old host. Light speed!

I hope you all continue to enjoy using Kilg.us. Our new hosting should make your stat tracking experience faster and better than ever!

Kilg.us Migration

Monday, May 2nd, 2011

I intended to migrate Kilg.us to its new servers tomorrow, May 3. If all goes well, the limiting factor on how long Kilg.us will be down is the domain transfer. This can take up to 48 hours, but will go through in 1-3 hours for most people. I intend to take Kilg.us off-line as soon as I get home from work tomorrow (~6pm mountain time). With this approach, I expect the migration and any subsequent troubleshooting should be finished tomorrow night.

Please check Facebook and Twitter (@kilgustracker) for updates on the migration.

Well that’s embarrassing

Sunday, April 24th, 2011

I got a couple emails yesterday that some daily boxscore email stats were incorrect. Looking into the matter, it was immediately obvious what had happen. As I worked on the transition of Kilg.us to its new servers, I had accidentally left a database connection pointing to the new dev database, instead of the old, production database. As such, the stats weren’t up-to-date when the daily emails were generated. I simply pointed the connection back to the correct database and figured all was well.

This morning there have again been reports of incorrect stats. I double-checked that the database connection was pointing to the correct IP address–it was. Then I remembered one of the many reasons I’m moving Kilg.us: the hosting company uses a non-standard port for database connections! Why? Who the hell knows. But I had forgotten to explicitly change back to their port. I’ve done that now. I’ve tested the connection. All seems to be well.

On the migration topic, Kilg.us is up and fully functional on its new servers. I’m testing manually with day-to-day usage to try to identify any aspects or configurations I may have missed. Right now, my simple tests are showing that Kilg.us runs between 2.2 and 5 times as fast on the new servers (adjusting as best I can for load considerations). I’m working on some minor changes that will decrease the bandwidth Kilg.us requires because the new servers are billed by data transfer. I’ll give at least 24 hours notice before I make the migration. Depending on how quickly the domain propagates, Kilg.us may be down for some of you for up to 24 hours during the migration. More details to come.

Cross your fingers!

Friday, April 15th, 2011

Tomorrow morning Kilg.us will use a entirely new process for generating and sending daily boxscore emails! So let’s all cross our fingers and hope it works…

I have spent all of this week re-designing and re-building how emails are generated, stored, and sent. Suffice to say, the new system is significantly more complex than the old one. When I’ve had a chance to breath and am not so tired, I’ll go into more detail on how it works.

My expectations of the new system are that it will:

  • Scale to accommodate as many users/emails as desired (the old system failed in the 200-300 range)
  • Allow of future portability if Kilg.us needs to change hosting
  • Implement all components to identify HTML vs. Text email so boxscores render properly on all clients
  • Allow for side-by-side HTML and Text versions (text version is forthcoming)
  • Provide redundancy of data, so if the process fails it can be re-started without losing a day’s data
  • Reduce likelihood of Kilg.us mail servers being identified as SPAM and black-listed
  • Lower computation overhead

It sounds almost too good to be true! Tomorrow’s push will be the first full run trying to send out to everyone’s unique addresses. All the testing so far has been delivering to a kilg.us address (or small tests to outside domains). I’ll be keeping an eye on things in the morning to try to ensure everything goes smoothly. If you do not receive your normal boxscore in an acceptably legible format, please post comments here with details (DETAILS–without them I can’t do anything–what email client, what team name, what was wrong, screenshots, any little detail).

Why it went wrong

Monday, April 11th, 2011

As many of you experienced, this morning the Kilg.us daily boxscores went horribly wrong and began sending piles of duplicate mails to everyone. If I haven’t apologized to you yet: I’m sorry. I try very hard to make sure any changes I make to Kilg.us won’t have an adverse effect on users. I bungled this one, but at least I think I’ve figured out what happened.

I mentioned previously that I moved the daily boxscore email script to the new Kilg.us server. That is why emails went out to anyone. On the new server PHP is installed differently from the old server. As a result, when I run the CRON job to generate the emails, I needed to use an application to call the script rather than just calling PHP to execute it. I chose to use a program called wget. The idea is that wget makes a call to a URL and that URL (a PHP file) generates and sends the emails. Before scheduling the morning’s emails, I tested to ensure the process worked. When I tested, though, I only used a sub-set of data. I didn’t really need 350 emails coming into my inbox, so I tested with a couple emails each for those sent to a team owner and those sent to a team viewer. That worked great.

When the CRON job ran this morning, everything seemed to go well until duplicates started showing up. A second round, then a third, then a fourth and so on. Interestingly, each wave was 15 minutes apart. As it turns out, if wget can’t complete a request (in this case a VERY long request for 350 emails), it tries again. By default it will try up to 20 times to fetch a file. Because the script takes so long to run, I believe it exceeded the server time-out.  When the script timed out, wget requested it a second time, then a third, then a fourth and so on. I believe this is what caused the duplicate emails.

I’ve made three adjustments to address this. First, I increased the server time-out for this process. Second, I have changed the CRON job to tell wget never to retry the fetch if it fails. Third, I’ve broken the massive emails process into multiple chunks. This is a temporary fix until I convert to using PEAR::Mail to more intelligently manage the process. I’m done for tonight, but that will probably be the priority tomorrow.

So that’s all of today’s work. Six hours sunk, but I think the emails will work in the morning. I’ve pushed back the time that the emails go out by a couple hours to they’ll line up with when I roll out of bed in the morning. If things go off-track, I’ll be able to curtail things faster than today. If all goes well, I’ll move the schedule back to the early morning hour so everyone has the boxscore email when they get up in the morning.

Keep your fingers crossed…

 

Migration and Boxscores

Sunday, April 10th, 2011

Things are proceeding well with the migration of Kilg.us to a new host. The cloud server is up, the web server is installed, database server configured, postfix installed, SSH/SFTP running. I’ve even migrated the codebase with success, although it is all still running on the old Kilg.us database. Tomorrow I hope to get the database copied over and see how things work.

For tomorrow I have moved the CRON job that mails daily boxscores to the new server. An initial test of it was successful. I’m optimistic that tomorrow morning everyone that wants boxscore emails should receive them.

Update on Boxscore Emails and More

Saturday, April 9th, 2011

This morning on Facebook I mentioned a few goals for fixes and improvements to Kilg.us this weekend. Those items were:

  1. update MLB player/team lists
  2. get email boxscores working
  3. speed up stat collection
  4. trim players that show up on the “Major League Baseball” team rosters (right now it includes players throughout all the minors, with no respect to whether they might actually appear in a real game…

I’m pleased to report that numbers 1 and 4 are soundly completed! As of this morning, all active MLB players should be listed in Kilg.us and associated with their correct teams. If you follow any of the “Major League Baseball” teams, those rosters should also be cleaned up to only show players on that team’s 40-man roster.

Unfortunately, 2 and 3 are stills works in progress. I have made some changes to the model for how player objects are created. The result of this change is that player stats are only refreshed from MLB.com when needed, rather than any time a player is referenced. In practice, loading a team page requires that a player’s stats be up-to-the-second. When a daily boxscore email is generated in the wee hours of the morning, though, there is no need to go gather latest stats–they haven’t changed in the hours since the games all ended.

In my testing so far, team pages are loading 10-40% faster now. I’m still not happy with their performance, but at least it is progress. When I try to generate all the daily emails, those are running significantly faster. Before today, the process was trending towards 90 minutes, timing out, and failing. Now the process is hammering through 80 (of ~360) emails in the first minute and sending emails properly. At that point, the server slows to a crawl and, while it continues to output the email content, it refuses to send the emails.

I considered reaching out to the hosting company for help in identifying why their system was doing this, but I’ve given up. The company (AMS Computer Services) and infrastructure that hosts Kilg.us sucks. I’m going to try moving Kilg.us to a cloud hosting solution from Rackspace. I’ve spent a good part of the day getting the new server up and configured. Tomorrow I should be able to start moving Kilg.us over, testing, troubleshooting, and seeing how it performs. It’s going to be a long, tedious, frustrating process but it can’t possibly be worse than sticking with the status quo.

In the meantime, my fingers are crossed that our daily boxscore emails will fire off properly in the morning…but I’m not holding my breath.

The reason I’ve mentioned in the past for not making the move to a new hosting solution (besides the time and effort) is that it will cost more than the current option. Because Kilg.us is going to cost me more to host now, I’ve added a PayPal donation button to give anyone interested the option in helping to support Kilg.us. I’m not hoping to make anything off Kilg.us, but it would be great it is wasn’t a financial burden. If you like using Kilg.us, I hope you’ll consider giving a little bit to help offset my costs. $5, $10, $1, anything will help!

I hope to have an update with good news by tomorrow evening as I start to migrate Kilg.us.

Unexpected Boxscores

Saturday, June 13th, 2009

Did you receive boxscore emails last night and again this morning? If so: 1) please accept my apologies, 2) please let me know either by commenting here or emailing.

It appears that Kilg.us’ hosting provider executed all scheduled tasks about 11pm MDT last night. This includes the application to fire off boxscores, as well as some housekeeping processes that happen on the website. I’ll be looking into why this happened and provide an update if I learn anything.