Subscribe by email

Q&A with Hal Higdon

Each week, coach and author, Hal Higdon answers your questions about running. Here's the latest:

Saturday
Nov172012

Simple Server Density Plugin for Windows to Monitor MSMQ

We've been steadily growing our dependence on Server Density (SD) as a means to monitor our cloud and hosted offerings.  While there are a number of supported plugins and 3rd party offerings, most likely you will run into a situation where you need to do something custom. The plugin architecture for Server Density is pretty awesome. The criteria for a good API is ease of use and a short ramp time. Simply put, when it comes to documentation, samples, and support; Server Density nails it.

First I downloaded the sample C# application and reviewed their support article on writing a custom plugin. Luckily the sample application was using System.Diagnotics which seemed like a good start for writing my plugin.  We use MSMQ for async operations like our Post Workout Notifications, or for offloading heavy work from the webserver.  Its easy to use if you have are in Windows, using some sort of AD, and need reliable queueing.  Anyway, this isn't a post on queue software (that's a whole other post), but suffice to say we need to monitor the size of the queue.  Occasionally, the Windows Process that processes the queue stops receiving messages, due to an elusive bug that is difficult to track down.  In the meantime, we can monitor the queue to look for signs that our queue processor has failed.

Below is the simple plugin used to report on all MSMQ queues for the current server.

   1:  using System;
   2:  using System.Collections.Generic;
   3:  using BoxedIce.ServerDensity.Agent.PluginSupport;
   4:  using System.Diagnostics;
   5:  using System.Messaging;
   6:   
   7:  namespace ServerDensity.MSMQ
   8:  {
   9:      public class Check : ICheck
  10:      {
  11:          public string Key
  12:          {
  13:              get { return "MSMQ"; }
  14:          }
  15:   
  16:          public object DoCheck()
  17:          {
  18:              PerformanceCounterCategory myCat = new PerformanceCounterCategory("MSMQ Queue");
  19:   
  20:              IDictionary<string, object> values = new Dictionary<string, object>();
  21:   
  22:              foreach (var name in myCat.GetInstanceNames())
  23:              {
  24:                  var cntr = new PerformanceCounter (myCat.CategoryName, "Messages in Queue", name);                                     
  25:                  var key = cntr.InstanceName.Replace (@"\", @"\\");
  26:                  values.Add(key, cntr.NextValue());                
  27:              }
  28:   
  29:              return values;
  30:          }
  31:      }
  32:  }

 

While writing this plugin, I encountered a few bugs. First with System.Diagnostics API, where Counters wanted a specific name. Second was a more subtle bug with the output of our dictionary. Turns out I was getting the following error in the Windows Event Log:  "URL http://trainingpeaks.serverdensity.com/postback/ returned: JSON Error: Syntax error."  My hunch was that the cntr.InstanceName wasn't properly JSON encoded since it inculdes AD paths. I roped Server Density Support into this, just to see if/how they would respond. Their team was on it right away, giving me the link to help debug my plugin. Their debug logging provided me just what I needed: the JSON output they send to their servers. Some simple Googling turned up JSONLint, which easily pointed me to the bad JSON. I simply needed to replace '\' with '\\' in my string, and I was good to go.

Here is a screen shot of our Server Density Plugin:

The final step was to create an alert on the queue of interest to email/SMS the DevOps crew if this queue gets too big. 

Well, that's enough for this Saturday morning. I'm going on a ride. :)

 

Trent Niemeyer has over 10 years of development experience at small web companies. As the VP of Technology at TrainingPeaks, Trent's role is to incubate new technologies and new ideas in order to scale the TrainingPeaks platform and grow our company. Trent has been racing triathlon for 5 years, with 4 Ironmans and about a dozen 70.3's under his belt. You can follow him on Twitter @trentniemeyer.

Friday
Nov162012

Our Transition to HTML5: Part 1

As a company whose flagship product is a large-scale, thick-client Flash application, we are keenly aware of the need to move forward to more progressive web development technologies, namely Javascript and HTML5.

For starters, we dove into what HTML5 development means: in our eyes, it has gotten to mean more than just the latest incarnation of the HTML standard. It’s more than a collection of new standards and buzzwords like localStorage, Canvas, RESTful APIs, or OAuth 2. It’s a new paradigm for the efficient development of large-scale applications, written in a language that until very recently was known as the evil cause of jQuery-spaghetti. Now, HTML5 has become an incredibly powerful object-oriented platform for developing high-quality software based on conventions. Convention, over compilers and configuration, has become Javascript’s way of ensuring high quality code, and fortunately there is no lack of community efforts. The blend of convention through frameworks like Backbone.JS and methods like BDD and the dynamic qualities of Javascript are opening up extremely powerful ways to create client-side software.

Let’s get into the details!

The search for a framework

With only minimal Javascript background and an existing basic home-spun Javascript framework, we were already aware of Javascript’s limitations. Coming from a highly structured C++ and C# background, our team quickly recognized the dangers of creating thick Javascript clients without a strategy. Spaghetti code, dependency management nightmares, often-repeated pieces of code, closures, global scope, spelling errors causing dynamic property errors, and highly coupled modules are all issues that will quickly degrade the quality of the whole project until it becomes entirely unmanageable.

This conclusion quickly led us to looking for an established, open-source client-side framework based on one of the widely known and understood MV**-paradigms. A cleanly-designed framework would allow us to fully decouple code dealing with the backend from rendering code and UX code.

The sheer amount of available open-source libraries for this purpose was mind-boggling and overwhelming, which initially led us to just picking our current favorite, and starting to create some experimental applications.

Backbone.JS was the "chosen one", and so far we have not regretted the choice. It's proven to be extremely lightweight, but with enough structure and convention to give us what we needed. On top of that, thanks to the dynamic nature of Javascript, it would be very easy to expand and modify Backbone as necessary.

BackboneJS

The first application we started developing with Backbone was our current Public Activity Viewer. The TrainingPeaks Public Activity Viewer was a great candidate because it is a  a single-page, single-purpose application with dynamic content and some cool user interaction, but rather limited use of a RESTful API, with no model updates, creations or user input.

What does BackboneJS offer? Client-side MVC. A decoupled framework of Models, Views, and Controllers; just, without the Controllers. Backbone’s MVC is their own interpretation of the design pattern and does not include traditional Controllers. Instead, the Backbone.Views are what would traditionally be controllers, i.e. modules that bind views to models and handle user interactions. The traditional MVC Views are replaced by Backbone Templates, which are by default Underscore templates that can be substituted with any JS templating engine.

Backbone.Models are much like traditional MVC models, and with properly implemented RESTful backend do not require a lot of manual coding: just some sensible model defaults and initialization code, and some model property validation code. If poorly architected, some manual parsing of request or response objects to conform to your existing API is required. The last bit of Backbone are its Routers, nothing but a hash-tag navigation engine that forwards page navigation to the appropriate Controller/View.

Here's one of our very simple sample models:

   var PublicFile = Backbone.Model.extend(
   {
       url: function ()
       {
           return "/TPWebServices/REST/PublicWorkouts/Workout/" + this.id;
       }
   });


One our our sample views:

   var StatsView = Backbone.View.extend(
   {
       rangeName: "Total",

       initialize: function (options)
       {
           _.bindAll(this);

           this.user = options.user;

           this.model.bind("fetching", this.fetching);
           this.model.bind("fetchComplete", this.render);
       },

       fetchNewData: function (options)
       {
           if (!options)
               return;

           var start = options.get ? options.get("Begin") : options.Begin;
           var end = options.get ? options.get("End") : options.End;
           this.rangeName = options.get ? options.get("Name") : (options.Name ? options.Name : "");
           this.model.set("Begin", start, { silent: true });
           this.model.set("End", end, { silent: true });
           this.model.fetch({ data: { "start": start, "end": end} });
       },

       fetching: function ()
       {
           $(this.el).empty();
           var html = fetchingTemplate({});
           $(this.el).html(html);
       },

       render: function ()
       {
           $(this.el).empty();

           if (this.model.get("Begin") === null)
               return;

           var data = this.model.toJSON();
           data.Units = this.user.get("units");
           data.WorkoutType = this.user.get("workoutType");

           if (this.rangeName !== "") data.RangeLabel = this.rangeName;

           var html = statsViewTemplate(data);
           $(this.el).append(html);
       }
   });

Backbone.Router

Our first instinct was to follow any one of the thousands of tutorials out there, and our Router quickly became ungainly:

var AppRouter = Backbone.Router.extend(
{
   sharedModels: {},
   sharedViews: {},

   // Event Aggregator is used to bubble up events to this central location.
   eventsAggregator: function (event, options)
   {
       switch (event)
       {
           //.....
        }
   },

   routes:
   {
       "*badFacebookToken": "index",
       "": "index"
   },

   index: function ()
   {
           var token = this.getQueryVariable("token");

           if (!token)
               token = this.getTokenFromPath();

           if (this.sharedModels.thePublicFile == null)
           {
               // Get the LastModified date (in epoch format)
               var lastmodifiedepoch = $("#main-wrapper").data("lastmodified");

               // …....
              this.createViewsAndModels(“index”);
   },

   createViewsAndModels: function(route)
   {
           var ws = new WorkoutSummaryModel(this.sharedModels.thePublicFile.get("WorkoutSummary"));
           var fs = new FlatSamplesModel(this.sharedModels.thePublicFile.get("FlatSamples"));
           var r = new RangesModel(this.sharedModels.thePublicFile.get("Ranges"));

           var statsJsonArray = this.sharedModels.thePublicFile.get("StatsCacheFeed");
           var stats = new StatsModel(statsJsonArray[0]);
           stats.set("id", this.sharedModels.thePublicFile.get("id"));
           stats.localStorage = this.cache.Stats;

           var comments = new CommentsModel(this.sharedModels.thePublicFile.get("Comments"));

           Preferences.model.get().set("Units", ws.get("Person").Units, { silent: true });
           Preferences.model.get().set("WorkoutType", ws.get("Activity"), { silent: true });

           var mapView = null;
           if (fs.get("HasLatLngData"))
           {
               mapView = new MapView(
               {
                   name: "theOneMap",
                   model: fs,
                   el: $("#map")
               });
           }
           else
               $("#map").parent().hide();

           //.........
   }
};

We looked at solutions to this, and other problems, beyond just simply refactoring into smaller methods, and found a host of design patterns and Backbone extensions that suited us.

Backbone.Marionette

One of the solutions we found to the above problem was to create our own custom Backbone extensions. We created a “Backbone.Hacks” repository and started adding object prototypes for Controllers, Mediator, EventAggregator, and several other Backbone hacks to customize it to our needs. As we progressed in our project, we found the need for more of these custom extensions, and more importantly, we noticed a lot of repeated code for basic things like rendering or setting up event handlers. We addressed those issues by creating things like default implementations for Backbone.Views for simple json-to-template rendering. This pattern drove us to look for existing extensions to Backbone.

Enter Backbone.Marionette by Derick Bailey. Marionette is a fantastic, powerful extension to Backbone to handle all of the above mentioned issues and more. Marionette introduces an Application prototype, a singleton that handles app-wide event binding and layout management. Without going too in-depth on what Marionette does (read about it on its official GitHub page), I've provided some sample Marionette code from our application below.

Marionette.Application

Small utility prototype that handles app initialization, acts as a main container for app-wide objects, and acts as a global event handler.

   App.js

   var fileviewerApp = new Marionette.Application();

   fileviewerApp.addRegions(
   {
       comments: "#comments",
       graph: "#graph"
   });

    fileviewerApp.on("initialize:before", function ()
   {
       this.getCurrentUnits = function () { ... };
       this.CacheStore = {};
   });


   fileviewerApp.on("initialize:after", function ()
   {
       var lastmodifiedepoch = $("#main-wrapper").data("lastmodified");

       // Instantiate the Cache stores.
       this.CacheStore.Activity = new Backbone.Cache({ name: "Activity", provider: window.localStorage, cacheBy: Backbone.Cache.CacheBy.ID, lastModifiedOnServer: lastmodifiedepoch });
       this.CacheStore.Stats = new Backbone.Cache({ name: "Stats", provider: new InMemoryCache(), cacheBy: Backbone.Cache.CacheBy.TIME_RANGE, lastModifiedOnServer: lastmodifiedepoch });

       if (!Backbone.History.started && Backbone.history)
           Backbone.history.start();
   });

 

   Main.js

   require([ "app" ], function(TheApp)
   {
        TheApp.start();
    } 

Marionette.ItemView and Marionette.CollectionView

These are really awesome & simple default implementations of a Backbone.View for a single model or a collection. A few lines of code, as follows, allow us to create a default view based on a template, fill it with model data, and render it, all automatically behind the scenes.

   var SingleCommentView = Marionette.ItemView.extend({ template: commentsTemplate });
   var CommentsView = Marionette.CollectionView.extend({itemView: SingleCommentView});
   var view = new CommentsView({ model: myModel });
   App.regionName.show(view);

Module Dependencies

The second issue we ran into was dependency management. How could we guarantee the definition for a BackboneWorkoutModel would be loaded before its corresponding BackboneWorkoutView, but after the main application entry point created the necessary wrappers? Messy code like this was not uncommon:

Model.js

window.App.Models = window.App.Models || {};
window.App.Models.MyNewModel = Backbone.Models.extend(
{
   initialize: function()
   {
       //...
   }
});


View.js

window.App.Views = window.App.Views || {};
window.App.Views.MyNewView = Backbone.View.extend({});


Module management like this makes it very hard to guarantee dependency order, and made it even harder to run unit tests without having to load the entire application in one monolithic block and without the ability to mock objects.

Cue RequireJS.

RequireJS

Require is a module loading library that follows the CommonJS & AMD standards and allow for asynchronous loading of modules with guaranteed load order & dependency resolution - we love RequireJS! We’re not going to lie, it has caused us some headaches integrating with some of the frameworks out there, specifically unit testing and code coverage frameworks, and it currently still breaks IntelliSense and other code-completion tools, but it is an extremely powerful way to define and load libraries statically and dynamically and to unify & minify your entire project. The benefits far outweigh the cons for now, and it looks like there's work in progress to make IntelliSense work around RequireJS.

A require module looks like this:

module.js

define(
“moduleName”,
[ “dependency1Module”, “dependency2Module” ],
function(Dependency1, Dependency2)
{
   var module =
   {
       // …
   };

   return module;
});


The "moduleName" is optional and not recommended - Require automatically fills it in when not present which makes it easier to move modules from folder to folder when refactoring. 
The array of dependencies is a string list of module names that can either be resolved by a require.config() object or by path/module.js resolution. The anonymous function is the callback executed when all the dependencies have been resolved, and those dependencies are passed in as arguments to the callback.

Next Steps

We still have a lot of questions, and we don't have all the answers yet. This is all for today, but we have more coming in Part 2 of this series about transitioning to HTML5! Issues & concepts we will be addressing include:

  • Unit Test framework and a discussion of Code Coverage
  • HTML templating engines
  • Internationalization
  • Minification & build environments
  • Applications of design patterns, mostly based on this article: http://addyosmani.com/resources/essentialjsdesignpatterns/book/
  • Etc.

Are you interested in being a part of this process? Do you love endurance sports and are you passionate about creating the world's best software for coaches and athletes? TrainingPeaks is hiring developers!

As a Software Engineer at TrainingPeaks, Bernardo Fanti is currently helping to move TrainingPeaks to the Cloud and HTML5. An aerospace engineer from Italy with a passion for software and high-adrenaline sports, Bernardo is also an MMA fighter who loves cooking, mountains, skiing, climbing, and gardening. Follow Bernardo on Twitter.

 

Tuesday
Oct092012

File migration: MS SQL to MongoDB in EC2

Recently we have solidified the move of all file data from MS SQL to MongoDB. First, you might wonder why anyone would store files in a relational database. There are a couple of advantages: 

  • Complete data portability. We can have all of our data backed up with one file. Along those lines restoring is easier, a process we use in our user acceptance/beta environment.
  • Storing millions of files in an OS can be massively inefficient, especially in Windows.

So why did we switch? In a nutshell: scalability. Our database had grown to 1.3 TB, of which 1 TB was all file data. We were faced with having to scale our solution out vertically, which in SQL Server can be an expensive proposition. In 2012 things only get worse with per core licensing. We researched a number of solutions and decided on MongoDB for the following reasons:

  • GridFS: A database built with BLOBs in mind.
  • Replication: Instant failover, and read scalability.
  • Future Growth Potential:  We could grow portions of our system into this new environment.
  • Open Source Community with Paid Support: A strong model for developing lasting and flexible technologies.

To make the move to MongoDB more cost effective we looked at moving to IaaS. EC2 was our provider of choice simply because they blow everyone else out of the water when it comes to full scalability and features. Similar to MongoDB, the move to EC2 has lots of potential to scale into.

Below are the challenges we'd have with the migration strategy:

  • Potential latency issues, since we are going outside of our hosted environment.
  • Engineering overhead of moving into a Linux production environment (we are mostly a Windows shop), EC2, and MongoDB itself.

The first step was to figure a migration path that would allow us to transition into Mongo slowly. The plan was to encapsulate the codebase that retrieves/puts files with code that will stick new files and recently accessed files into Mongo. Once that was stable and working we'd write a process that would migrate all old files over to Mongo in a long running background task.

First sticking point is the 1TB limit on EBS Storage volumes. Being so close to that limit with the current dataset, we had to scale horizontally. MongoDB offers sharding, but we were afraid of the backup/restore limitations, and the extra complexity added by sharding vs simple replication.

Instead, we decided to do a software RAID with LVM (https://wiki.archlinux.org/index.php/Software_RAID_and_LVM). RAID has multiple benefits: it helps buffer us from the relatively high failure rate of EBS Volumes
(http://aws.amazon.com/ebs/), allow us to create larger volumes, and increases read speed. We tested multiple RAID setups, starting with a MongoDB recommended RAID1+0 setup:

mdadm --verbose --create /dev/md0 --level=10 --chunk=256 --raid-devices=4 /dev/sdh1 /dev/sdh2 /dev/sdh3 /dev/sdh4

We quickly moved away from this implementation when we realized that mdadm does not allow us to hot grow a RAID0 based array. Our research brought us into LVM territory, and we decided on the following setup (pseudo shell code):

for x in {0..3}
mdadm --verbose --create /dev/md$x --level=1 --raid-devices=2 /dev/sdh$((2 *x)) /dev/sdh$((2*x + 1))
done


With four underlying RAID1 devices we now created a Striped Logical Volume using the LVM, practically obtaining a dynamically growable RAID10 array without the pain of dealing with mdadm to resize the array. Resizing is as easy as adding two new EBS volumes, raiding them together into a RAID1 array using mdadm, and expanding the Logical Volume.

Our Mongo Replica Set configuration includes 1 primary, 1 secondary, and 1 four hour delayed secondary. With the equivalent of a RAID10 configuration, that means a lot of EBS volumes!

That was the plan, and what follows are the lessons we learned during the transition.

  • We overlooked the importance of keeping our environments distinct and portable. Remember how SQL Server was convenient in storing everything with one backup?  Having files missing on our internal testing and local dev environments caused many unnecessary headaches. Simply put, each environment need be fully self contained, and changing our processes was a pain.
  • We opted out of MongoDB's backup option and decided to do a file system backup. Figuring the backup has been the most time consuming issue we've run into. With regards to standing up a mongo replica from a backup archive, lots of information points us to restoring the data directory and deleting the local files, but that didn't seem to work. What we found works is re-seeding the replica config, so we created more shell scripts that blow away our Beta Mongo environment, restore the archive to mongo’s data path, and re-seed replica configuration on startup.
  • We use RSync for backups (taking hourly and daily diffs). This is accomplished by taking an LVM snapshot of the entire mongo data volume (we have combined data path and journal path, to avoid having to fsync+lock the database), and RSync’ing to a backup server that handles storing the archives. The most important rsync setting we are using is the --link-dest setting which allows us to save massive amounts of disk space by using *nix hard links to copy file. Once a week we backup the entire dataset to Glacier. Glacier is awesome, but pretty new. I hope to see emerging 3rd party tools come out in the future, but for now we have a pretty awesome C# app built for Mono - latest updates coming soon.
  • We are constantly testing our backups as our beta environment is refreshed weekly.
  • Secondary replica read. We do use safeMode=true when writing to mongo, but we don’t block until replication happens and return as soon as the write to the primary succeeds. Asynchronous DB writes might be a longer term solution, but the way our app is currently built we require instant synchronous writes to happen. This causes trouble in a connection-pooled environment where a different connection might try to read that same data from a secondary. We learned that lesson the hard way and have since trimmed down our oplog and, most importantly, leveraged a Redis cache server for recent writes/reads. Redis cache is awesome and better than Memcache since you can store files larger than 2MB. The biggest downside to Redis so far is the lack of clustering. I am eagerly anticipating clustering in Redis. So far Redis + Linux has been really reliable, and its not a critical path for our application should it go down.
  • Replica nodes in MongoDB are pretty sweet. You can just stand up a new node, it handles syncing for you, and it comes online after *mostly* syncing. That being said, test your failover. The C# driver had a bug when you configured your server to use hostnames, but the client used IP's. The app got confused when failing over, and our site was unable to operate without file data. Maybe this is why Netflix has a chaos monkey. We didn't implement a chaos monkey, but we do test failover now.

In summary, the move to MongoDB has been positive and we’ve been satisfied with the performance so far. While we’re only using the GridFS at the moment, we have plans to move logging and eventually select application bits. Additionally we’ve whetted our appetite for AWS and have plans to move more of our infrastructure to the IaaS/PaaS offerings in EC2.

As a Software Engineer at TrainingPeaks, Bernardo Fanti is currently helping to move TrainingPeaks to the Cloud and HTML5. An aerospace engineer from Italy with a passion for software and high-adrenaline sports, Bernardo is also an MMA fighter who loves cooking, mountains, skiing, climbing, and gardening. Follow Bernardo on Twitter 

 

 

Thursday
Oct042012

Introducing the TrainingPeaks Tech Blog

We, the engineering team at TrainingPeaks, think we’re working with some interesting software on some interesting problem sets. We wanted to share those thoughts with the rest of the tech community to showcase what we are doing, and to share the lessons learned as we develop and improve upon the world's best training and nutrition software. 2013 is on track to bring major development changes, so now is a great time to get some of these thoughts and lessons out there in the form of our new TrainingPeaks Tech blog, at blog.trainingpeaks.com/tech.

To kick things off, here are some of the technology and infrastructure changes we plan to make in the near term, and will likely be fodder for upcoming content:

  • Migrate the TrainingPeaks Flex application to a JS/HTML5 web app
  • Move towards IaaS and PaaS in EC2
  • Build a new RESTful API and put SOAP services to bed
  • Expand our fleet of mobile offerings with native iOS and Android applications
  • Answer BigData questions with tools like Hadoop

While we’re at it, you should also know that we are hiring. We’re looking for talented folks with the following skill sets and experience to help us make the technology transitions I described above:

  • Javascript Application Development: backbone, bootstrap, jquery, etc
  • Thick Client Development (Windows,Mac,Flex)
  • HTML5, CSS3
  • C#/ASP.NET
  • C++
  • iOS
  • Android
  • EC2 DevOps
  • Unix DevOps
  • Windows DevOps
  • MongoDB Administration and Application Development
  • Redis Administration and Development

If you have these qualifications, and live and breathe endurance sports like we do, come check out our Careers page.

So, bookmark this URL, add us to your RSS feed, and keep an eye out - more to come here at the TP Tech Blog.