david mosher

Kevin Pierce showed me some handy scripting when I was updating some svn stuff.

Basically, this sets up a for loop that's iterating over some directories that I wanted to move into a sub directory. I hadn't done much bash scripting, but this is much more efficient than typing out the lines 1 at a time.

Useful! :)

I saw Posterous through the TweetDeck update and was pleasantly intrigued by the interface. Couldn't have a simpler concept for publishing personal stuff if you ask me, so I decided to give it a try and see what it looked like. I'm pretty impressed to be honest, driving content via email, sms, or any other number of input methods and having it appear automagically on a website somewhere with panache is a pretty cool idea.

Update: I've since dug into the Posterous interface and I've been even more impressed. It was easy to add a DNS record for www.davemo.com in EasyDNS and have it point to this site in a matter of minutes. Yay :)

Facebook connect integration, auto embedding of media including images, video, music and the ability to post via so many methods. This just makes blogging as easy as sending email.

[ Note: this post was originally started in September/November, 2008. It's been sitting in my drafts for quite a while and I decided it was time to finish it up. ]

We finished off a short 1 week development sprint last Friday [edit: sometime in Sep/Nov 2008] which culminated in the current live version of the VendAsta website. Despite such a short dev-cycle, I feel we were able to accomplish most of our research goals.

One of the tools we used that I didn't have much experience with prior was Yahoo Pipes. We had somewhat of a unique problem to solve; we wanted to be able to draw from a variety of Feed sources that would filter through to different sections of the site. Before I dove into using Pipes my gut feeling was just to use some sort of feed parsing library in Python that was compatible with AppEngine. One of the requirements for implementing the feed sources was that it be very easy to update should a feed need to be changed. If we stuck with the Python library implementation and statically coded some feed sources, changes would take significantly more overhead to implement.

We started investigating Pipes as a platform to aggregate all of the content we wanted to re-publish and were impressed by what we found. It turns out that the Pipes interface is much better at providing an easy-to-update set of feed sources for non-programmers. If you haven't looked at the Pipes interface I highly recommend you do so. I think Pipes offers so much advanced filtering and splicing functionality in a really tight package, so I'll attempt to break down some of the modules we used specifically for the VendAsta website.

The image above is a shot of the Pipes interface, and more specifically the Pipe that we use to aggregate all of the feed-based content on the VendAsta website. I should clarify, by feed-based content I'm referring to all of the official employee, product, and corporate blogs. Yes you read that right, employee blog posts (when tagged appropriately) end up on our corporate website. But let's set aside the trust/ethical issues related to that and focus on the technology we're using. There are a few components to the root pipe we use to collect all this content, and the Pipes documentation isn't always entirely clear on the best way to use things, so I will attempt to decompose it and discuss each module in detail.

Feed Auto-Discovery

We have a URL mounted on VendAsta.com that provides only 1 function: spitting out a list of link tags in the following format:

<link rel="alternate" type="application/rss+xml" title="Allan Wolinski" href="http://www.orthodrome.ca">

Internally, we store a list of all employees with some meta information like blogURL, linkedinURL etc. This url is what we provide to the auto-discovery module so it can do it's magic and go out to all of the rss+xml links and gather a list of feeds from the blogs. Here's a sample of the output you get from this URL in the Pipes debug console.

Once we have this list of blogs and the links to them, we can pass the output from the auto-discovery module on to the Loop + Fetch Site Feed modules.

Loop + Fetch Site Feed

The loop module is pretty self explanatory, you use it when you want to iterate over a group of results and operate on them somehow. It's kind of similar to providing an AJAX onSuccess callback; you send out a request for the content (the iteration) and you define a 'callback' by dragging a supporting pipes function into the available box to process each item. In our case, we're using the Fetch Site Feed module from the toolbox to snag a feed from each site in our blogroll. We configure the module to emit all results to our next module, the Filter, which does most of the magic in determining which content ends up on the corporate website.

Filter

The filter module works really well for grabbing a set of elements based on a certain set of criteria; in our case we use the Regexp option to explicitly allow any content with the tag/category 'VendAsta'. 

Union, Sort, Rename + Pipe Output

The final layer of our pipe involves combining results from some other static sources independent of filters (VendAsta blog and the StepRep blog), sorting them based on the date published descending, renaming a specific RSS feed field (item.dc:creator) to 'author' to fix a problem in the feed reading library we're using server side (feedparser.py) and finally the output of all these operations: 'Pipe Output' module. All these modules are really easy to string together with the Pipes interface and are clickable at any point during the creation which will toggle the debug output to stop at the currently selected module. This is very useful for seeing what the modules you are using do at each step of the process.

Pipes as abstract components ties it all together

So now that we have a completed Pipes module, we can do a number of things with it. We can get the output in RSS or JSON format, define a custom url for the pipe, and even give it a name and make it public so other people can use it in their mashups. However, the coolest feature IMHO is that you can use any created pipes module as an abstracted pipes component on it's own which can be part of another pipes component that you can then perform further operations on. We chose to separate a few of the layers in our feed hierarchy this way for simplicity sake and to keep it loosely coupled if we wanted to make changes to what feeds appear on certain pages. Here's an example of our previous Blogroll pipe being used as input for further filtering and display on 1 section of the VendAsta website: the Projects page.

We covered a lot of ground on the MyFrontSteps project this week. We managed to migrate our Photo Management / Uploading solution on Homebook from SWFUpload and a lot of hacked together jQuery to a more elegant solution (albeit one that could still use some refactoring - but what code can't use refactoring really). We decided to use the "still-in-beta" YUI Uploader coupled with some custom jQuery. The implementation seems much more cohesive than our previous solution but there were a few gotchas we rant into when trying to get things working properly in IE7.

YUI Uploader

I know this component is still in beta support from Yahoo, but I had given it a very brief try in our supported browser baseline (currently FF2+, Safari3+, IE7+) and it seemed to work really well. All of the YUI stuff we're using works really well and the graded browser support and testing the YUI team puts behind their work makes implementation that much more reliable. That being said, no piece of software is perfect and we ran into a pretty frustrating bug implementing the YUI Uploader in a specific area of the application.

We have a number of context menus in Homebook that appear to the user on mouseover. This particular thumbnail represents the default image for a users Home, and clicking it prompts the user with a file dialog to select a new photo which is then uploaded via the YUI Uploader. When instantiated, the uploader inserts a transparent Flash object using the SWFObject plugin into the markup in the container specified. In our case we had the following markup:

    ul class='mfs-context-menu invisible' id='default-image-menu'
        li
           a
             span class='uploader' id='upload-profile-picture' /span
             'Change Profile Picture'
          /a
       /li
    /ul

We use a class of invisible to set the css property 'display:none;' on the 'ul' so it's hidden by default when the page loads. When the page loads and the script triggers, the uploader sits nicely in the span tag directly above the link in the li tag. The user then clicks on the transparent flash object and the upload can start. Now, this approach works fine in every browser EXCEPT IE7. For some reason when the uploader is instantiated in IE7 if it becomes hidden and then shown again at any point the YUI Uploader loses all binding to the functions that trigger submission to the server side url. I tried doing some searching to see if this was documented anywhere and came up empty everywhere I looked.

My best guess is that the IE7 security model doesn't like the fact that an 'embed' tag with the Flash object is being manipulated from JavaScript and decided to turn off something that disables the uploader from completing an upload. The file dialog will still popup, and the embedded Flash remains in the DOM but the uploader doesn't trigger sending of the files to the server. It took us a while to figure out what was going on, and the only way we tracked it down was by a process of elimination commenting out lines of code in the uploader instantiation script until we discovered it was our context menu show/hide trigger that made the uploader behave this way. So, the solution in our case was to move the uploader 'span' tag outside of the 'li' but still inside the thumbnail container, give it a fixed width and height directly above the image (like a transparent rectangle) and bind mouseover/mouseout events to it that triggered show/hide of the 'li' element with the 'a' inside:

    ul class='mfs-context-menu invisible' id='default-image-menu'
        span class='uploader' id='upload-profile-picture' /span
        li a 'Change Profile Picture' /a /li
    /ul

This way the flash element never gets hidden, and IE7 doesn't do anything funky to it so the uploader remains operational.

Brett and I recently began refactoring a significant amount of the JavaScript that is currently in MyFrontSteps and Homebook. One of the areas we identified as needing improvement was controlling when scripts get loaded in the page; it's a challenging subject especially when utilizing Django templates which can extend and include bits of HTML that are both static and dynamic. We're not finished the refactoring quite yet but I thought it would be valuable to blog about the lessons we've learned early on about how to manage JavaScript loading without having script tags all over the place.

Manual Dependency Management is Hard

Not too long ago, Yahoo put out a list of guidelines that web developers can use to enhance the performance of their websites. I won't get into it in this post but there is a Firefox plugin called YSlow available that can automate some of the performance checking. The recommendation I'm going to focus on here is the one regarding moving scripts as close to the bottom of the page as possible. I'll just quote the Yahoo doc as they explain it very clearly:

The problem caused by scripts is that they block parallel downloads. The HTTP/1.1 specification suggests that browsers download no more than two components in parallel per hostname. If you serve your images from multiple hostnames, you can get more than two downloads to occur in parallel. While a script is downloading, however, the browser won't start any other downloads, even on different hostnames. In some situations it's not easy to move scripts to the bottom. If, for example, the script uses document.writeto insert part of the page's content, it can't be moved lower in the page. There might also be scoping issues. In many cases, there are ways to workaround these situations.

In the case of the code we're using on MyFrontSteps we had all kinds of Django templates with includes and templatetags that include other little bits of dynamic JavaScript that was adding markup to the DOM and manipulating DOM content. Some of this code would appear prior to certain dependent scripts being loaded which created a real nightmare for us as we had to manually track down the position of the dependent scripts in the page that was fed to the browser after a variable number of layers of templates and includes. We decided to research a better way to manage all these scripts and since we had been using the YUI library for many other parts of the website we settled on the YUI loader.

YUI Loader Makes Dependency Management Easy

The YUI Loader Utility is a client-side JavaScript component that allows you to load specific YUI components and their dependencies into your page via script. YUI Loader can operate as a holistic solution by loading all of your necessary YUI components, or it can be used to add one or more components to a page on which some YUI content already exists.

The great thing about the YUI Loader is that you can use it to manage dependency sorting for all of the YUI components you use on a page but the killer feature imho is the fact that it allows you to create custom modules to load your own JS and CSS libraries. The basic pattern we're using in our root level Django template is as follows:

1. Load all CSS Files at the top of the template
2. Place all YUI Loader and other statically included scripts at the bottom of the page prior to the closing BODY tag
3. Define the YUI Loader instance
4. Define custom modules for the YUI loader
5. Provide a Django template block to allow manipulation of the onSuccess callback
6. Call the YUI Loaders .insert() method to trigger insertion of all dependency sorted scripts into the DOM

The advantage to using the loader in conjunction with our root Django template is clear; all of our scripts are loaded at the bottom of the page and CSS is loaded at the top. When a user requests the pages they will start receiving the content from the server immediately and not have to wait for scripts to process as the loader dynamically inserts them into the HEAD element after the page content has been rendered. Another trick we're using to control when scripts get executed is by manipulating how the onSuccess callback of the loader works. Here's the code:

// Instantiate and configure YUI Loader:
MFSLoader = new YAHOO.util.YUILoader({
    base: "",
    require: ["base","reset-fonts-grids","MFS"],
    loadOptional: false,
    combine: true,
    filter: "MIN",
    allowRollup: true,
    onSuccess: function() {
        while( MFSLoader.onReady.length )  {
            MFSLoader.onReady.shift()();
        }
    }
});

// Define a list of executables that we
// can add to prior to page load completion
MFSLoader.onReady = [];

// Custom Modules for Loader
MFSLoader.addModule({
    name: 'MFS', type: 'js', varName: 'MFS',
    path: '{% vurl "/static/script/MFS.js" %}',
    requires: ['jQuery', 'jQueryUI']
});

{# Override this block if you need script at the global level. #}
{% block global.script %}{% endblock %}
// Trigger insertion of all dependency sorted scripts into the DOM
MFSLoader.insert();

As you can see we have a few of the custom modules defined here, which include a 'requires' attribute in the configuration object. This lets the YUI Loader know that these files require the other modules to be loaded. The Loader then determines sort order for inclusion based on all required modules and goes to work inserting the scripts for you.

Tying it all Together

The key to making this work in the varying levels of Django templates is to manipulate the onSuccess callback. When the YUI Loader finishes loading all the dependency sorted scripts it will execute this callback and allow you to then execute any additional code that depends on the dynamically inserted scripts. We have a number of namespaced JS objects for MyFrontSteps (MFS.js, MFS.DataGrid.js etc..) and when we need to call functions defined in these objects we do so by extending the global.script block in our child template and pushing a new anonymous function onto the MFSLoader.onReady list we defined in the root template. Here's a simple example:

{# A CHILD TEMPLATE #}
{% block global.script %}
    #include the parent templates global.script contents
    {{ block.super }} 

    // Load our required features
    MFSLoader.require( 'MFS.DataGrid', 'uploader', 'MFS.Uploader' );
    // Push an anonymous function onto the list
    // to get executed when all required scripts have been loaded
    MFSLoader.onReady.push( function()  {
        MFS.DataGrid.initPhotosGrid();
    });
{% endblock %}

Once the page is rendered and the YUI Loader has finished parsing the loaded scripts the code we have in the onSuccess callback pops all the anonymous functions off the list which causes them to be executed.

{# THE ONSUCCESS CALLBACK #}
onSuccess: function() {
    while( MFSLoader.onReady.length )  {
        MFSLoader.onReady.shift()();
    }
}

We still have a number of files to refactor, but the performance benefits we've seen so far using this approach have really made us confident that this is the way to go. Using a dependency manager that works for custom JS / CSS libraries is just so much more liberating than having to manually keep track of where scripts are getting executed. Because we're also migrating all of our code to external library files it makes things that much easier to debug in Firebug as well. :)

If I had captured a snapshot of my application usage habits during each year of the last few years I think I would have discovered something like this:

• 2005: (Windows): Photoshop, Fireworks, Dreamweaver, Firefox
• 2006: (Windows/Linux): Fireworks, Komodo, Linux Terminal, Notepad, PHPMySQL, WAMP, Firefox (+Firebug)
• 2007: (Windows/OS X): Fireworks, Notepad, IntelliJ IDEA, Visual Studio, Firefox (+Firebug)
• 2008: (OS X): Fireworks, Textmate, OSX Terminal, Firefox (+Firebug)

I've got nothing really insightful about these pseudo-stats, it just kind of struck me as I was working that I really don't use much of an IDE at all now.

There are some interesting things that happen when a job comes to you as opposed to the alternative (read: job hunting trying to find a company that fits you). When you fit the company things click. Perhaps I'm being a bit presumptuous with that last sentence, but it's truly how I feel working at VendAsta. This feeling is one of the things that has solidified this idea of positive dissatisfaction in my mind and how it can be a huge benefit to people in any field of work, but even more so to those in the world of software development.

Our small development team for the MFS (My Front Steps) project is nearly completed now and I'm amazed at how well things are coming together. We have a very well rounded team of developers who all seem to be on the same page about things. Our experiences are varied and cover a broad spectrum of disciplines which gives great depth to such a small team. However, I believe the common uniting factor in what is going to make us such a great development team is the fact that we all realize that we haven't arrived in terms of our abilities. We don't know it all. I certainly don't know it all, and I'll switch to the first person from this point on as I don't want to put words in anyone's mouth.

I define "Positive Dissatisfaction" as a state of mind that reflects an individuals desire to continually better themselves by coming to the realization that they cannot be satisfied with the knowledge they have attained at any given point in time. I feel that in order to truly succeed in the field of software development that I have to continually admit that there is always a better way to do things than the way I think things should be done. My goal is to reinvent myself every day with the knowledge I learn from others, the web, and, more generally, my life experience.

I believe that Positive Dissatisfaction fits great with the Scrum development methodology. Short iterations offer a huge opportunity for team members to improve their development practices and experience the benefits of learning quickly from one another. The fact that the team is self regulating in terms of task load, allocation, and followup promotes a great sense of ownership in the project that reinforces a sense of continual refinement of processes. I certainly don't want to be contributing code/resources to the project that was the result of some half-assed effort. All of these factors help me to keep myself in a continual state of Positive Dissatisfaction (have I said that enough yet?) so that I'm attempting to contribute with a 110% effort.

When you fit the company, it really makes you want to contribute that way :)

On Work

I've been at VendAsta for a week now and I'm happy to say that my initial impressions haven't changed all that much. It's really fulfilling to go in every day and feel totally engaged and encouraged by both the work you're doing and the people you're working with. I've heard some people say negative things about VendAsta; things like "they won't be around longer than 8 months" or "they're just a start up, how can they hope to accomplish anything". I find most of the people saying these things are either uninformed about what it is we're working on or just spiteful for some reason. Well, to those of you in either camp let me enlighten you.

VendAsta is about building quality software. VendAsta is about empowering people by allowing them to work on the things they are passionate about. Right now the team I'm working on is building a social marketing tool that will enable homeowners and home industry service providers to share their experiences on whatever social network they happen to be using. There is a lot of room for movement in this area of the social networking sphere, particularly because nobody else has really taken advantage of the online experience as it relates to our homes and home experiences. Rennovations, improvements, parties, appliances you purchased, decorating tips... the list of things people can share about their homes goes on and on. So when I hear people predicting the downfall of VendAsta I really think they have no clue about how much potential there is in the market segment we're working in. Oh and did I mention, that's only 1/2 of what we do? :)

On Macs

I made the switch to working on a Mac at home about 8 months ago. It was more of a novelty at the time, but I needed a change and was tired of "tweaking" my PC when I got home from work. I did manage to learn some important tricks about OS X and get a glimpse of what actually "working" on one would be like. Flash forward some 9 months and I work full time on a MacBook Pro developing with some of the most intriguing technologies out there. The only thing holding me back to Windows for the longest time was the games, and I find now that most of my gaming plays better on my iMac anyways. The keyboard shortcuts are kind of a pain to get used to, but once you realize you actually have an additional modifier key to work with things just kind of click.

On Python

Python is neat. It's kind of like JavaScript with less of the syntax cruft (hooray for no more semicolons). It's also kind of like C (I think some of the base libraries are actually written in C) and while I don't have a lot of C experience, the one class I did take was very enjoyable. It's like you get all the power in the lower level like C but without all the syntax nightmares; you also have access to a lot of libraries. Writing python is kind of like writing pseudo code, which is a good fit for me because I always thought I was better at coming up with pseudo code algorithms than actually programming.

If you're interested in learning I can't recommend this book enough: Dive Into Python. The best part is, you can download a full PDF of the book entirely free! I'm only about half way through so far but it's written in a very easy to understand style and comes with many code examples in the zip file. If you're following along using the examples on a Mac it's even better because Leopard comes already installed with Python 2.5. Yes, working with Python is good stuff.

On Google App Engine (GAE)

Google App Engine is an interesting beast. I've read some articles already about how it's changing development paradigms and forcing developers to think about scalability and data access in a completely different way. I don't purport to be a database expert but from what I've read GAE uses a storage system called BigTable that differs radically from typical RDBS (Relational Database Systems). While this doesn't affect me directly (yet) I find it interesting to read about problems like scalability. The GAE DataStore API is really simple yet effective, and so far the documentation has been really easy to read and the code examples really easy to follow. A few posts in the Google Groups section for GAE have some further detail on performance testing and the results are intriguing. My only beef so far with GAE is with running Django on it and the fact that there seems to be a few established ways of running it but no "standard" way:

1. Google Code - Running Django on Google App Engine
2. GC Project - Google-AppEngine-Django
3. Guido van Rossum - Rietveld (sample django on GAE project)

I've been working hard to try and figure out some of the best practices for running Django on GAE but I suppose a lot of this is subject to potential change considering GAE is still in PREVIEW RELEASE mode and the Django 1.0 release is still a few weeks away. I don't anticipate too much change though as there is quite a bit of the API already established. I guess it's a good thing we're using the latest SVN releases of Django instead of the default 0.96, 0.97 releases GAE comes with by default.

On Django

I tried doing some tutorials on the Django website a few months back and while it was really cool seeing all the "magic" stuff that you got for free with things like the admin interface, ORM layer etc... I don't think I really appreciated the power of what it can do until I started reading Dive Into Python. Since Django is written in Python it really helps to have an understanding of the language at a more rudimentary level to see exactly how Django works and to avoid the "wtf?! how did it do that?" questions I was having previously. It seems to me that the guys who built Django really understand what's tedious about building web applications and they've created this set of tools to make web application development much more enjoyable.

I personally enjoy working on backend stuff like configuration, deployment, ORM setup and working with Django + GAE is really nice for all of that, but at the same time part of me really enjoys constructing the CSS, XHTML templates, implementing jQuery and YUI for effects and interface widgets, and actually designing the graphics... all of the more frontend type work. I consider myself a Sweeper (read the first paragraph), although more of a frontend leaning Sweeper, but working with the tools and technologies I am able to work with here at VendAsta has really empowered me to "sweep" from frontend to backend and picking up all kinds of knowledge everywhere in between. Just one more thing that makes me really feel like I've finally found a job that fits with my never ending thirst to work in both worlds. Did I mention, we're hiring?

Normally I think most people would associate work with emotions other than bliss. It's not often in life you can find yourself working in a job with people who are on the same page as you. It's also not too often in life you can find yourself working in a job where the things you are passionately interested in are what people want you to work on. Combine the two previously mentioned "hard-to-find" qualities with a brand new MacBook Pro and a healthy salary (there's something to be said for getting paid well for the things you love to do) and you have what I would consider the recipe for the perfect job; aka: work bliss.

Today is my first day at VendAsta and already I've been able to configure my work environment the way I want (3 cheers for TextMate, SVN, FireBug, FireWorks) after being told if there was anything I need to just let them know and they'd have no problem getting it. So I got right into things, checked out a few projects from SVN that we're working on utilizing Django and Google App Engine. Normally at a new job you'd probably have a lot of red tape and "new hire" overhead to go through; not at VendAsta. <sarcasm>I couldn't believe there was *gasp* documentation for the projects!</sarcasm> There's something to be said for working with quality processes like Scrum and Agile, and great tools like JIRA and GoogleApps at a software development company; try working without these things, it's just not fun.

I think the thing I've been impressed with most in my first day at VendAsta is the fact that there are people who are willing to listen to my ideas. I don't feel intimidated at all, the culture here is very welcoming and open and the tools provided make it very easy to actually want to be as productive as you can be. Obviously it's only the first day on the job so I can't expect you all to take everything I'm saying at face value. Time will tell how things pan out, but after the first day I can definitely say that my first day @ VendAsta has been far more enlightening and empowering than a couple of years working for other companies :)