Kilg.us – Fantasy Stat Tracker …Tracker

A blog about the development of Kilg.us – The Fantasy Baseball Stat Tracker

Posts Tagged ‘infrastructure’

Email Boxscore Issues

Monday, May 18th, 2015

Numerous people have notified me of an issue with daily email boxscores. It seems that since early May, many have not been delivered properly. This is on my radar. Unfortunately, I have not had a chance to dive in to resolve it. Trust that as soon as I find time, I will get it fixed and post an update here about what happened.

SendGrid Roll-out Successful

Tuesday, April 17th, 2012

Last night I upgraded the email sending mechanism in Kilg.us to use SendGrid. This morning all email boxscores look to have gone out as planned.

As a number people have noted, “as planned” may be less that what we all would like. One of the things SendGrid will help me do is see how many emails are being generated, sent, delivered, read, etc. This will allow me to better manage how daily boxscore emails work.

The first issue to address are those users that have signed up for boxscore emails but haven’t received them. The old email provider had a much lower limit on the number of messages that could be sent per batch and per day. This lead to some less than ideal compromises in designing the system. Basically, rather than just flooding the email server with all the boxscore emails, I had to throttle the process, building a queuing system and sending out small batches of emails. As the system neared its daily limits, some people’s emails would go un-sent.

In addition to the migration to SendGrid last night, this morning I have updated the queuing and sending system. It now queues everything the same, but tomorrow will send one large batch of messages (using an anti-flood function to ensure the email server doesn’t get overwhelmed). This SHOULD ensure that everyone receives every email they have requested! We’ll check back in tomorrow morning to see if it worked.

As a tack on, I switched this WordPress installation to also use SendGrid which should make it less likely that I miss comments and questions people post on here (sorry if I missed your question or comment!).

Moving to SendGrid

Monday, April 16th, 2012

In the next day or so Kilg.us will be moving its email handling to SendGrid. This should give me better transparency into what messages are being send, received, and bounced back. Right now things are a bit of a manual process which makes it difficult to diagnose one-off errors (particularly with daily boxscore emails).

If all goes well, I’ll probably make the change tonight. Theoretically, there should be no change for anyone but me. Let’s hope it works out that way!

I love Kilgus! Can you include [STATX]?

Friday, April 6th, 2012

This is by far the most common topic when people email me about Kilg.us. I’m thrilled that so many people are enjoying the tool (we start the year with over 1800 members, nearly 500 users per day), but these are bummer emails to receive.

Unfortunately, 95% of the time the answer: No, sorry.

As a free tool, Kilg.us needs to rely on whatever free stats we can find. As it turns out, MLB publishes some limited live data (I believe for their GameDay feature) that we’ve been able to tap. While it is great that the data is free, the bummer is that it isn’t comprehensive.

Kilg.us is already pulling every stat available from MLB and even calculating a whole series more that aren’t made available in the data feed.

If you’re favorite stat isn’t included, it’s almost certainly because we just don’t have access to it. At the start of each season, I double-check the feeds to see if any new stats are available. This year, there is nothing new.

So, if there is something that can be calculated with the data already in Kilg.us, let me know and I’ll be happy to add it! If you want a new counting stat added, though, I can’t add it.

Maybe someday an angel investor will throw a pile of money at Kilg.us and we can afford to purchase a real data feed (~$25k/year). Until then, we just have to make due with what we have.

Hopefully it’s enough to be useful to you!

(That said, don’t forget to click “Select Stats” for each of your teams to see all the stats that are available: currently 32 categories!)

Database Failure

Wednesday, May 4th, 2011

Well, things seemed to go great with the migration. I posted to the blog that everything was running great. An hour later the database server crapped out. I’ve spent the last 4 hours trying to recover with no luck. As best I can tell, the permissions tables in the database got screwed up. I’m completely baffled as to how it could have happened. I’m very frustrated by this, but I need to sleep so it will have to wait for tomorrow.

In the meantime, I have pointed to the old database server. Kilg.us code is still running on the new servers, but it is looking to the old servers for data. This probably means that Kilg.us will continue to run at about the speed it used to.

Tomorrow I will attempt to export the databases from the other server again and re-import into the new database servers.

Sorry for the unexpected, additional down-time after I re-launched Kilg.us. If I can find a cause for the problem, I’ll definitely pass it along with any update tomorrow evening.

Migration Success!

Tuesday, May 3rd, 2011

I’m a little tepid to call the Kilg.us migration a success, but things seem to have worked. The site, blog, databases, scheduled processes, accounts, teams, stats…they’re all moved over. And as best I can tell, they are all working. I’ll be keeping a very close eye on things through the week in case anything goes awry, but we seem to be in good shape for now. If you notice any odd behavior, errors, or encounter problems PLEASE comment here, post to Facebook or Twitter, or send me an email so I can get to the root of things.

Now that the migration is complete, I’ll also finally get to some upgrades! First on the list are some new statistical categories. Please keep in mind that Kilg.us uses a free data source from MLB that only offers limited information. I pull as much out of that source as I can, but some stats just aren’t available. Also on the list is some tweaking to the sign-up/log-in form. I have been receiving quite a few emails about issues with it, most specifically with Internet Explorer 8. If you encounter problem signing-up or logging-in in the meantime, try a browser other than IE8.

Just for fun: Kilg.us now manages over 2300 teams for more than 1300 users! And with the new hosting infrastructure, my testing is showing 3-4 second rendering times for team stat pages compared to 13-23 seconds at the old host. Light speed!

I hope you all continue to enjoy using Kilg.us. Our new hosting should make your stat tracking experience faster and better than ever!

Kilg.us Migration

Monday, May 2nd, 2011

I intended to migrate Kilg.us to its new servers tomorrow, May 3. If all goes well, the limiting factor on how long Kilg.us will be down is the domain transfer. This can take up to 48 hours, but will go through in 1-3 hours for most people. I intend to take Kilg.us off-line as soon as I get home from work tomorrow (~6pm mountain time). With this approach, I expect the migration and any subsequent troubleshooting should be finished tomorrow night.

Please check Facebook and Twitter (@kilgustracker) for updates on the migration.

So close…

Saturday, April 16th, 2011

By now, everyone should have received their daily boxscores. If not, please comment below.

We were not without bumps, unfortunately. That I am aware of so far:

  • As many a 50 boxscores were sent out multiple times (apologies!)
  • The public SMTP I tried to use apparently uses a 24-hour window (rather than a day) for its quotas

The duplicate boxscores was just a goof by me. Last night as I was running my final tests, I commented-out the line that removes an email from the queue after sending because I didn’t want to keep re-entering data into the system before each test. I forgot to un-comment that line when I was done. Because of that, when the system pulled the first 50 boxscores to send, it still left them in the queue. When it went to pull the next 50, it returned the same group. I noticed this after a few times through the process, so hopefully everyone that got duplicates only saw a trickle–not a deluge–of them.

The SMTP issue is frustrating. In my testing, I was able to generate the full list of emails. I assumed the quota would reset at midnight. Apparently that wasn’t the case. So hundreds last night plus hundreds this morning meant the sending account was locked out mid-process. To resolve this, I used one of my personal accounts to complete the send. Many of you will have seen messages come from “matthew@encoredigitalmedia.com” rather than Kilg.us. I figured this was better than making you miss a day’s boxscore.

I have spent most of the last two days trying to get a dedicated Kilg.us SMTP server up and running so I don’t need to worry about stupid quotas and such. Unfortunately, I have absolutely no experience with setting up SMTP and have had just about zero success–hence the use of a public SMTP to get this moving. If anyone knows SMTP and can offer a helping hand, I would be greatly appreciative! Get in touch by email (admin@kilg.us), comment here, or on Twitter (@kilgustracker) or Facebook.

All that said, I think all emails should have been delivered and I have received feedback from some people whose emails weren’t rendering properly before that they look good now. All in all, I’m going to call this one a draw. I’ll keep plugging away today in hopes of a perfect push tomorrow.

Migration and Boxscores

Sunday, April 10th, 2011

Things are proceeding well with the migration of Kilg.us to a new host. The cloud server is up, the web server is installed, database server configured, postfix installed, SSH/SFTP running. I’ve even migrated the codebase with success, although it is all still running on the old Kilg.us database. Tomorrow I hope to get the database copied over and see how things work.

For tomorrow I have moved the CRON job that mails daily boxscores to the new server. An initial test of it was successful. I’m optimistic that tomorrow morning everyone that wants boxscore emails should receive them.

Update on Boxscore Emails and More

Saturday, April 9th, 2011

This morning on Facebook I mentioned a few goals for fixes and improvements to Kilg.us this weekend. Those items were:

  1. update MLB player/team lists
  2. get email boxscores working
  3. speed up stat collection
  4. trim players that show up on the “Major League Baseball” team rosters (right now it includes players throughout all the minors, with no respect to whether they might actually appear in a real game…

I’m pleased to report that numbers 1 and 4 are soundly completed! As of this morning, all active MLB players should be listed in Kilg.us and associated with their correct teams. If you follow any of the “Major League Baseball” teams, those rosters should also be cleaned up to only show players on that team’s 40-man roster.

Unfortunately, 2 and 3 are stills works in progress. I have made some changes to the model for how player objects are created. The result of this change is that player stats are only refreshed from MLB.com when needed, rather than any time a player is referenced. In practice, loading a team page requires that a player’s stats be up-to-the-second. When a daily boxscore email is generated in the wee hours of the morning, though, there is no need to go gather latest stats–they haven’t changed in the hours since the games all ended.

In my testing so far, team pages are loading 10-40% faster now. I’m still not happy with their performance, but at least it is progress. When I try to generate all the daily emails, those are running significantly faster. Before today, the process was trending towards 90 minutes, timing out, and failing. Now the process is hammering through 80 (of ~360) emails in the first minute and sending emails properly. At that point, the server slows to a crawl and, while it continues to output the email content, it refuses to send the emails.

I considered reaching out to the hosting company for help in identifying why their system was doing this, but I’ve given up. The company (AMS Computer Services) and infrastructure that hosts Kilg.us sucks. I’m going to try moving Kilg.us to a cloud hosting solution from Rackspace. I’ve spent a good part of the day getting the new server up and configured. Tomorrow I should be able to start moving Kilg.us over, testing, troubleshooting, and seeing how it performs. It’s going to be a long, tedious, frustrating process but it can’t possibly be worse than sticking with the status quo.

In the meantime, my fingers are crossed that our daily boxscore emails will fire off properly in the morning…but I’m not holding my breath.

The reason I’ve mentioned in the past for not making the move to a new hosting solution (besides the time and effort) is that it will cost more than the current option. Because Kilg.us is going to cost me more to host now, I’ve added a PayPal donation button to give anyone interested the option in helping to support Kilg.us. I’m not hoping to make anything off Kilg.us, but it would be great it is wasn’t a financial burden. If you like using Kilg.us, I hope you’ll consider giving a little bit to help offset my costs. $5, $10, $1, anything will help!

I hope to have an update with good news by tomorrow evening as I start to migrate Kilg.us.