Kilg.us – Fantasy Stat Tracker …Tracker

A blog about the development of Kilg.us – The Fantasy Baseball Stat Tracker

Posts Tagged ‘stats’

Daily Boxscore Email Update

Tuesday, May 26th, 2015

I apologize for the delay on getting this issue addressed.

As noted previously, daily boxscore emails stopped going out earlier this month. I’ve finally had time to dig into the issue.

First, if you are subscribed for daily emails, you should have received a boxscore email around 9:30am Mountain time today. If you didn’t receive such an email (and confirmed it isn’t in a SPAM folder), please comment here or email me.

To publish daily boxscore emails, Kilg.us goes through a few steps. First, it makes sure all the latest stats have been collected. It then starts a queue for composing the emails. Each team is checked for subscribers and email content is saved to a database table. Once complete, the queue is closed (the stages of opening and closing this queue are recorded). The next step is to systematically go through that DB table and send the emails. As each email is sent, it is deleted from the table. The emails are sent in batches, 50 at a time.

A few observations:

  • The database shows that the composition queue was completed each day of the outage.
  • The database is stuffed full of all the emails that haven’t gone out in the last few weeks.
  • SendGrid (the 3rd party that actually sends the emails at Kilg.us’ request) shows that a full batch (~550) emails went out on May 1. May 2 dropped to 50, followed by 200, 50, 100, and then to 0 in subsequent days. These increments of 50 suggest something is going wrong with our send batches.
  • I used the exact same scripts to successfully send the emails today (manually executed, rather than scheduled).

All told, I haven’t identified a specific cause of the outage. The send process can take a while so it is possible we ran into a timeout along the way. The server has been updated, scrubbed, and rebooted so it should be in tip-top shape for tomorrow. Fingers crossed that emails will go out as scheduled. If not, I at least know I can manually trigger the process daily until I have time to dig back in deeper.

2015 Season

Monday, April 6th, 2015

The new season is upon us! I’ve had a number of people checking in to see if Kilg.us will be functioning again this year. It looks like our data feeds are still working, so at this point I don’t have plans to shut Kilg.us down.

Daily boxscore emails should resume tomorrow. If you would like to change your settings for which teams you receive, please access your account settings.

As a quick reminder: Kilg.us is in maintenance mode. If something goes down, I’ll try to fix it in a timely manner. At this point I don’t actively use Kilg.us, though, so it isn’t a high priority. Last year I posted some things to keep in mind about the operations of Kilg.us. Please take a look.

Good luck with your 2015 season!

Spring Training Stats

Friday, April 5th, 2013

I’ve been getting a flood of emails about spring training statistics showing up for players that haven’t played in the big leagues yet this year.

Kilg.us collects all of its stats for an XML feed that MLB publishes. Every time you request the stats for your team(s), Kilg.us checks to see what the latest stats are and loads them into the system.

Unfortunately, it seems that MLB leaves springs stats listed for all players that appear in the spring but are then sent to the minors (or DL). Until they appear in a big league game, MLB will continue to show springs stats in their feed.

This is just how it works.

If you have a player on your team that is showing spring stats, please move that player to your bench. The player’s individual stats will still show from spring, but this will remove the player’s stats from your team totals.

Missing and Delayed Player Stats

Sunday, August 19th, 2012

Over the last few days I’ve received a lot of emails about Kilg.us not properly updating some player’s stats. Reports include missing stats, delayed reporting of stats, and out-dated season stats. I’ve not been able to investigate every report (timeliness is obviously critical in a real-time stat tracker!), but in each instance I’ve been able to check, the MLB data source is, in fact, reporting the incorrect, incomplete, or out-dated states.

Unfortunately, there is nothing I can do about the incorrect stats feed. This is one of the down-sides of using a free, unsupported stats feed. We just have to live with whatever it provides us.

It seems like most people have had success checking later in the evening or first thing in the morning. Hopefully that will suffice in getting everyone the stats they need.

With any luck, there is just a glitch in the MLB.com stats feeds and they will right themselves soon. In the four or so years Kilg.us has been up and running, though, I haven’t seen this happen before, so I’m not quite sure what to expect.

Stat Updates

Thursday, July 19th, 2012

Over the last few weeks I’ve received a number of messages that some players that haven’t played in the big leagues are showing season stats for 2012. All of that data comes from MLB, so if they show season stats in their data feeds, Kilg.us shows it as well. Usually these are spring training stats that just haven’t been updated.

I have made an effort to go through and delete any season stats for players that have been reported. If you see this behavior for any additional players, please let me know via email or in comments here.

I love Kilgus! Can you include [STATX]?

Friday, April 6th, 2012

This is by far the most common topic when people email me about Kilg.us. I’m thrilled that so many people are enjoying the tool (we start the year with over 1800 members, nearly 500 users per day), but these are bummer emails to receive.

Unfortunately, 95% of the time the answer: No, sorry.

As a free tool, Kilg.us needs to rely on whatever free stats we can find. As it turns out, MLB publishes some limited live data (I believe for their GameDay feature) that we’ve been able to tap. While it is great that the data is free, the bummer is that it isn’t comprehensive.

Kilg.us is already pulling every stat available from MLB and even calculating a whole series more that aren’t made available in the data feed.

If you’re favorite stat isn’t included, it’s almost certainly because we just don’t have access to it. At the start of each season, I double-check the feeds to see if any new stats are available. This year, there is nothing new.

So, if there is something that can be calculated with the data already in Kilg.us, let me know and I’ll be happy to add it! If you want a new counting stat added, though, I can’t add it.

Maybe someday an angel investor will throw a pile of money at Kilg.us and we can afford to purchase a real data feed (~$25k/year). Until then, we just have to make due with what we have.

Hopefully it’s enough to be useful to you!

(That said, don’t forget to click “Select Stats” for each of your teams to see all the stats that are available: currently 32 categories!)

New Stat: Total Bases

Thursday, May 5th, 2011

Now that Kilg.us is migrated and–seemingly–stable, I can actually make some improvements!

First up are some stats additions. The easiest request was to include total bases. Because this is calculated based on data we already have, I was able to implement it pretty quickly. To enable Total Bases for any of your teams, visit the “Select Stats” page for that team. You will see a checkbox for “TB.” Checking that box and saving your changes should add a TB column right after strikeouts.

The most commonly requested stats that Kilg.us does not currently have are games played. While this data isn’t conveniently available in the MLB XML data Kilg.us uses, I think I have determined a way to deduce whether a player has played in the day’s game(s) or not. That will take significantly more effort, though, so it will have to wait. Perhaps this weekend I’ll find some time.

So close…

Saturday, April 16th, 2011

By now, everyone should have received their daily boxscores. If not, please comment below.

We were not without bumps, unfortunately. That I am aware of so far:

  • As many a 50 boxscores were sent out multiple times (apologies!)
  • The public SMTP I tried to use apparently uses a 24-hour window (rather than a day) for its quotas

The duplicate boxscores was just a goof by me. Last night as I was running my final tests, I commented-out the line that removes an email from the queue after sending because I didn’t want to keep re-entering data into the system before each test. I forgot to un-comment that line when I was done. Because of that, when the system pulled the first 50 boxscores to send, it still left them in the queue. When it went to pull the next 50, it returned the same group. I noticed this after a few times through the process, so hopefully everyone that got duplicates only saw a trickle–not a deluge–of them.

The SMTP issue is frustrating. In my testing, I was able to generate the full list of emails. I assumed the quota would reset at midnight. Apparently that wasn’t the case. So hundreds last night plus hundreds this morning meant the sending account was locked out mid-process. To resolve this, I used one of my personal accounts to complete the send. Many of you will have seen messages come from “matthew@encoredigitalmedia.com” rather than Kilg.us. I figured this was better than making you miss a day’s boxscore.

I have spent most of the last two days trying to get a dedicated Kilg.us SMTP server up and running so I don’t need to worry about stupid quotas and such. Unfortunately, I have absolutely no experience with setting up SMTP and have had just about zero success–hence the use of a public SMTP to get this moving. If anyone knows SMTP and can offer a helping hand, I would be greatly appreciative! Get in touch by email (admin@kilg.us), comment here, or on Twitter (@kilgustracker) or Facebook.

All that said, I think all emails should have been delivered and I have received feedback from some people whose emails weren’t rendering properly before that they look good now. All in all, I’m going to call this one a draw. I’ll keep plugging away today in hopes of a perfect push tomorrow.

Cross your fingers!

Friday, April 15th, 2011

Tomorrow morning Kilg.us will use a entirely new process for generating and sending daily boxscore emails! So let’s all cross our fingers and hope it works…

I have spent all of this week re-designing and re-building how emails are generated, stored, and sent. Suffice to say, the new system is significantly more complex than the old one. When I’ve had a chance to breath and am not so tired, I’ll go into more detail on how it works.

My expectations of the new system are that it will:

  • Scale to accommodate as many users/emails as desired (the old system failed in the 200-300 range)
  • Allow of future portability if Kilg.us needs to change hosting
  • Implement all components to identify HTML vs. Text email so boxscores render properly on all clients
  • Allow for side-by-side HTML and Text versions (text version is forthcoming)
  • Provide redundancy of data, so if the process fails it can be re-started without losing a day’s data
  • Reduce likelihood of Kilg.us mail servers being identified as SPAM and black-listed
  • Lower computation overhead

It sounds almost too good to be true! Tomorrow’s push will be the first full run trying to send out to everyone’s unique addresses. All the testing so far has been delivering to a kilg.us address (or small tests to outside domains). I’ll be keeping an eye on things in the morning to try to ensure everything goes smoothly. If you do not receive your normal boxscore in an acceptably legible format, please post comments here with details (DETAILS–without them I can’t do anything–what email client, what team name, what was wrong, screenshots, any little detail).

Update on Boxscore Emails and More

Saturday, April 9th, 2011

This morning on Facebook I mentioned a few goals for fixes and improvements to Kilg.us this weekend. Those items were:

  1. update MLB player/team lists
  2. get email boxscores working
  3. speed up stat collection
  4. trim players that show up on the “Major League Baseball” team rosters (right now it includes players throughout all the minors, with no respect to whether they might actually appear in a real game…

I’m pleased to report that numbers 1 and 4 are soundly completed! As of this morning, all active MLB players should be listed in Kilg.us and associated with their correct teams. If you follow any of the “Major League Baseball” teams, those rosters should also be cleaned up to only show players on that team’s 40-man roster.

Unfortunately, 2 and 3 are stills works in progress. I have made some changes to the model for how player objects are created. The result of this change is that player stats are only refreshed from MLB.com when needed, rather than any time a player is referenced. In practice, loading a team page requires that a player’s stats be up-to-the-second. When a daily boxscore email is generated in the wee hours of the morning, though, there is no need to go gather latest stats–they haven’t changed in the hours since the games all ended.

In my testing so far, team pages are loading 10-40% faster now. I’m still not happy with their performance, but at least it is progress. When I try to generate all the daily emails, those are running significantly faster. Before today, the process was trending towards 90 minutes, timing out, and failing. Now the process is hammering through 80 (of ~360) emails in the first minute and sending emails properly. At that point, the server slows to a crawl and, while it continues to output the email content, it refuses to send the emails.

I considered reaching out to the hosting company for help in identifying why their system was doing this, but I’ve given up. The company (AMS Computer Services) and infrastructure that hosts Kilg.us sucks. I’m going to try moving Kilg.us to a cloud hosting solution from Rackspace. I’ve spent a good part of the day getting the new server up and configured. Tomorrow I should be able to start moving Kilg.us over, testing, troubleshooting, and seeing how it performs. It’s going to be a long, tedious, frustrating process but it can’t possibly be worse than sticking with the status quo.

In the meantime, my fingers are crossed that our daily boxscore emails will fire off properly in the morning…but I’m not holding my breath.

The reason I’ve mentioned in the past for not making the move to a new hosting solution (besides the time and effort) is that it will cost more than the current option. Because Kilg.us is going to cost me more to host now, I’ve added a PayPal donation button to give anyone interested the option in helping to support Kilg.us. I’m not hoping to make anything off Kilg.us, but it would be great it is wasn’t a financial burden. If you like using Kilg.us, I hope you’ll consider giving a little bit to help offset my costs. $5, $10, $1, anything will help!

I hope to have an update with good news by tomorrow evening as I start to migrate Kilg.us.