Thinking asynchronously

I started playing with Node.js yesterday. It was easy to pick up, but the hardest part was getting used to thinking in a completely asynchronous manner. I’ve done a ton of work making Windows Forms programs behave asynchronously, but in traditional WinForms programming (before C# async) the process didn’t really require special thought. I’d write isolated synchronous code, test it, then write the framework necessary to make my UI call it asynchronously and respond to the completion events.

Node.js requires me to think in a completely different way, though looking backwards I can still see how it is analagous. Here’s an example from yesterday’s experiments.

While I was getting familiar with Node.js, I wrote some code that didn’t do what I expected.

var badStream = fs.createReadStream('data.xml');
var buffer = badStream.read()
console.log(buffer.toString());

So far I’d only looked at examples of HTTP requests in Node, and noticed those tended to use the data event on streams to respond as data arrived. I was in a case where I knew all my data could easily fit into memory, so I figured I’d stick to something simple and synchronous. The read() function sounded synchronous, so I used it. But every time I ran this code, the console would print “null”. I was especially confounded because the same code worked as expected in the REPL. Worse, some code I had written earlier used this approach and had worked.

I’d love a Node.js expert to confirm this, but I think I understand it now. In Node.js, you are not expected to attempt to use streams (or at least filesystem streams) synchronously like this. The createReadStream() function creates the stream, but doesn’t guarantee any data has been read into the internal buffer. And the read() function doesn’t block until data is available, it returns null instead. This code has a race condition and will work if the buffer happens to be loaded before you call read(). That’s why it worked in the REPL and my previous programs. (It also makes sense that the REPL might force some data to be loaded on creation to facilitate faster hacking.)

When I returned to the documentation with a fresh mind, the proper solution was clear to me.

var goodStream = fs.createReadStream('data.xml');
goodStream.on('readable', function() {
	var buffer = goodStream.read();
	console.log(buffer.toString());
});

Creating the stream doesn’t mean it’s ready. But it does emit the readable event when it is ready. So if you want to load data, you have to create the stream and call read() in response to one of the events that indicates data is ready. Then read() will return the data.

Another thing that caused me pain until I learned to think asynchronously was ensuring an order to stream operations. I needed to compress some data befor sending it to a server, so to test my understanding I wanted to make a program to round-trip compress then decompress. It looked like this:

var inputStream = fs.createReadStream('data.xml')
inputStream.on('end', function() {
	console.log("Created output file.");
}
var gzip = zlib.createGZip();
inputStream.pipe(gzip);
var outputStream = fs.createWriteStream('data.gz');
gzip.pipe(outputStream);

console.log("Done writing, time to read.");
var compressedInput = fs.createReadStream('data.gz');
var unzip = zlib.createGunzip();
compressedInput.pipe(unzip).on('data', function(chunk) {
	console.log(chunk.toString());
});

The first behavior I didn’t understand was my output statements were out of order. I got “Done writing…” before “Created output file.” The second behavior I didn’t understand is I was not seeing any decompressed output. I had no clue what was wrong and don’t really have a Node.js mentor to ask, so I hacked and hacked at it until I realized both of my problems were related to attacking the problem synchronously.

The first problem: The statements are out of order because the “Done writing…” output will happen synchronously while the “Created…” output happens after piping the file completes. It’s our good old buddy ‘race condition’. Especially concerning is this means the output file might not be ready before the code tries to read it.

The second problem was a little weird, and I’d like confirmation I understand it. I thought the last lines were saying, “Set up a pipe() operation that sends data from the compressed input to the unzipper, and as data arrives publish the decompressed data.” Instead, the data was being piped from input to the unzipper and producing a third stream, and I was setting up my result on that stream. But nothing in the program ever causes that result stream to flow data, so the data event is never emitted. Oops.

The solution to the first problem was to think asynchronously and tell Node.js it should only operate on the output file when the output file stream says it’s done creating the file. The solution to the second was to stop using shortcuts and be more explicit with where I call on():

var inputStream = fs.createReadStream('data.xml')
var gzip = zlib.createGzip();
var outputStream = fs.createWriteStream('data.gz');
outputStream.on('finish', function() {
	console.log("Done writing, reading data back:");
	var compressedInput = fs.createReadStream('data.gz');
	var unzip = zlib.createGunzip();
	unzip.on('data', function(chunk) {
		console.log(chunk.toString());
	});

	compressedInput.pipe(unzip);
});

inputStream.pipe(gzip).pipe(outputStream);

This sets up the relationships I wanted. Data flows from the input file to the gzipper to the output file. When that is finished, the code to read the data back is scheduled. Data flows from the compressed file to the gunzipper, and as it is published by the unzipper it is logged to the console.

It was sort of tough to figure this stuff out on my own, but I think I get it now. If I extract the decompression code to a function in the above example, it’s easy to see how Node.js is still similar to what I was used to. This is how I visualize the organization of a Node.js application now:

  1. Testable utility functions that do something synchronously.
  2. Definitions of objects that will act as data sources and sinks.
  3. Setup of responses to events on the objects created in (2) using functions from (1).
  4. Operations on the objects in (2). (These cause the events in (3) to be emitted.

I’m not sure if that’s the best mental model, but it’s working for me now. It’s tough to do this without a mentor.

Jonathan’s Card

The feel-good news item last week was Jonathan’s Card. Apparently Starbucks lets people use their phones as a sort of loyalty + gift card by displaying a barcode. Some guy made his barcode public and offered to let anyone use it. What happened was a kind of Starbucks fan circlejerk as some people used up the money and others replenished it. Starbucks let it go on until recently; some say it was a publicity stunt, others believe it was good publicity for them. Jonathan himself framed it as a social experiment.

Here’s the thing about experiments: they conclude, and the results tell you something. There’s plenty of reasonable hypotheses for this but I think they all boil down to, “Will people use a near-anonymous source of giving and receiving money for good or evil?” for various values of good and evil. For example, if no one deposited money, the conclusion would be, “People are too greedy for this to work.”

Sam Odio performed an experiment of his own. He wrote a script to periodically check the balance of the account and when it reached some amount he had the surplus transferred to his own card. He transferred this surplus to Starbucks gift cards and auctioned the cards on eBay, with proceeds going to a charity. His reasoning was the social experiment was silly because its only side effect was buying coffee for people who could already afford it.

The Jonathan’s Card community is enraged. I’m quite amused by this; most people’s rants start something like this:

I put $10 on the card with the intent of buying myself some coffee. When I got to Starbucks the card was empty! Sam Odio stole my money, he’s a thief!

Let’s paint a different story that happened numerous times and no one complained. Someone put $10 on the card, then left for Starbucks. When they arrived, only $2 were on the card. This happened numerous times, I have no doubts. Why was it OK for unknown individuals to spend the $10 on coffee, but theft for Odio to spend it on charity? Anyone could transfer money from Jonathan’s Card to their own. Sam Odio did nothing unique, and never emptied the card. If anything, this was a flaw in the experiment, not humanity.

Since the card allowed transfer of funds, it was more like a community bank account. If you want to buy yourself some coffee, you don’t put it in the community account.  You take your $10 and hand it to the cashier. Anyone who put their money on Jonathan’s card should have been comfortable with that money being used for anything at all. If you specifically wanted it to go towards coffee, you should have sought a mechanism that guaranteed the use. Some people have taken to buying Starbucks gift cards and leaving them in random locations.

I think the experiment’s conclusion shows that such a community bank account can work: the card rarely stayed empty for long periods and I’m sure plenty of moochers were kept caffeinated. But if a technique for using the money for non-coffee purchases exists, it will be exploited. Something like this card might have been a convenient way to launder some money, but no one seems to have been worried about that.

Starbucks has shut the card down. If it was a marketing stunt Odio likely helped them realize how this card could be misused. If it wasn’t a marketing stunt I think it reflects poorly on Starbucks: the experiment was great while it was money spent on Starbucks merch and a violation as soon as it was spent on anything else. I think it would have been interesting to see what happened after Odio started his experiment; others would likely copy it. Would they make sure to never empty the card, or would they be greedy and make it impossible for money to stay on the card? We’ll never know because the experiment was shut down as soon as the result didn’t look favorable to Starbucks.

New Article: Converting Numbers to Strings

I’m not keeping up my blog like I should. To try and fix this, I’m going to attack my backlog of “articles to end forum posts”. I’m active on some beginners’ VB .NET forums, and some questions come up very frequently. It’s frustrating when it’s one that requires a lot of explanation like, “Why does setting the property on this class change the property on this other variable?”. I’m going to try and attack one of these each month or (ideally) each week. Maybe by then I’ll be in the habit.

Today’s article: Converting Strings to Numbers. It’s a beginners’ article, so don’t expect anything profound. My main target is newbie VB .NET developers but I threw in some C# information where it behaves differently.

The iPhone changed my life.

Linkbait!

Monday I ended my relationship with Sprint. I spent a whole 8 hours as a free man then signed my soul away to AT&T. It’s been one of the best purchases I’ve made.

I had a BlackBerry Curve 8300, and it was a very frustrating phone. It did text messaging and email well enough, but using the internet was nearly impossible. Opera Mini was the only usable browser; the BlackBerry browser took several minutes to load even simple sites. At some point the phone decided enough was enough and I lost the ability to install applications. It would start the process then claim it was out of memory after 10 minutes or so.

I’ve always neglected the social side of the internet, and this didn’t help. There was a Facebook application, but using it was a pain and I never quite got the hang of it. The Twitter app was obviously not designed for the trackball-based BB phones, and I was probably at the very bottom of the requirements because it generally took a minute or two to load individual tweets. I had a data plan but every time I tried to use data I was foiled. Perhaps that’s why Sprint sold me unlimited data.

I’ve used my iPhone constantly since purchasing it. It’s been very liberating. Twitter’s usable. Facebook is a breeze. I can use Gowalla and a ton of other silly apps I could only glimpse through cracks in the smartphone wall. Now my music’s always with me, I can always take a photo, and I can always take a quick note if I need to. This was all *possible* with my Blackberry, but it took so much longer to get everything done I never did.

Time to fire Chekhov’s gun; how did this phone change my life? It’s caused me to expand my social interactions by a teeny bit, and this is a breakthrough for someone who usually doesn’t leave the house. My wife’s been out of town for a week. Usually when she’s gone my weekend schedule consists of waking up, playing video games, then going to bed. This weekend was a triumph. I went downtown and worked on some Gowalla trips. I wouldn’t have gone to the Zilker Kite Festival alone had it not been for the desire to liveblog the experience via tweets (unfortunately AT&T’s network wasn’t up to the task; I hope for their sake they do better at SXSW.) In addition, my increased Facebook use has reopened connections with people I haven’t talked to in years. I noticed someone was coming in for SXSW and we’re going to meet. If I still had my BlackBerry I’d have been ignorant they were coming to Austin. If I’ve had this positive an experience in a week, I can’t wait to discover what other great experiences are in store.

RIM’s newest attempt at relevance.

RIM just announced they’re going to port their BlackBerry Messenger (BBM) application to Android and iPhone. This news is a great indicator of why I believe RIM won’t make it through the smartphone wars.

I bought a BlackBerry curve in 2009 because I was too cool to buy an iPhone and Android was still too young for me to take the plunge. I corrected this error on Monday, the exact date of expiration on my Sprint contract. BBM’s a great example of why using the phone was a miserable experience. In my opinion, it’s only purposes are to keep you chained to BB phones out of fear and make you beg your friends to switch to BB phones as well. If RIM had spent their effort on designing a good SMS experience instead, they wouldn’t have to port a BBM application and if that idea were applied across the entire phone they might have had a chance against the new smartphones.

Here’s how an SMS conversation with a person who doesn’t have a BlackBerry worked on my phone:

  • Find the person in the contacts list.
  • “Right-click” their name to bring up a menu with 3 screens of information on it; find “Send SMS Message” and click it.
  • Now type the message and send it; this dumps you back to the contacts application.
  • When they reply, the message is dumped in the general log with missed calls, voice mails, and the spam SMS that BlackBerry sends once or twice a month.
  • To reply to that message, right-click the message and find “reply” in the 2-screen high menu that pops up.

Note that the messages weren’t treated as a conversation, they were discrete phone events. If I were conversing with more than one person, the messages in both conversations would intermingle in the log. Oh, and each message only got one line in the log, so only about 20 characters could be seen. It was miserable. You didn’t *have* to have a person in your contacts to send an SMS, but in 2 years I only successfully pulled that off once after a 10-minute Google search session, so I got in the habit of adding a contact if I wanted to SMS someone.

In BBM, here was the process:

  • Somehow obtain the person’s special BB identifier.
  • Input their 12 digit hex code into the BBM app and now they’re a contact.
  • Now all of your messaging with this person appears as a linear conversation like an IM window.

The only real problem here was obtaining the person’s BB id. It was a hex code with 12 digits or so; imagine if you had to have someone’s MAC address to send an email! The ID was buried in the system options application; I can’t complain too much about this because Apple shoved the iPhone’s MAC information in here too. It wasn’t in anything useful like “My BB ID”, it was in something generic like “Settings>Advanced Options>About”. It took me 15 minutes to find it the first time and always involved a web search afterwards.

Here’s how it works for *everyone* in iPhone, and I bet Android’s similar:

  • Type a number.
  • Type the message.
  • Now the conversation happens in an IM-like window.

Nice, no hex codes! If the person is in my contacts, I can start typing their name and the iPhone automatically searches contacts so I can select them quicker.

The only reason BBM is needed is the poor messaging experience on BB phones. If RIM had spent their effort on making a good messaging experience instead of a special application limited to BB phones, they wouldn’t have a need to port the application to 2 different platforms in an attempt to make it easier for BB users to communicate with everyone else. Even if I had plenty of friends with BB phones, I wouldn’t download this app. I don’t need two messaging applications. I have one that works great. It’s a shame neither of their messaging applications is a good solution, but I think if it’s difficult they should demand better from RIM and switch platforms if no changes are made.

Screencasting on Windows

I’m very active on the Xtreme VB Talk forums; it’s the first programming forum I ever joined and I enjoy sharing my knowledge with other developers. This year I’m trying to transform my blog from a cobweb collector to something I might want to show off, and part of that involves posting more about coding. Posting on the blog lets me incorporate screenshots and videos more easily, so I’d like to start augmenting some of my tutorials here. For that, I need something that lets me do screen recording. Unfortunately, I didn’t find the state of Windows screencasting to be all that good. Here’s a list of the programs I’ve tried and what I think about them.

Requirements

These are the requirements I have for screencasting software.

  • Full screen recording is nice, I’d like to be able to restrict it to a region or window as well.
  • I mostly want to record VS 2010 development, so it has to work with Windows.
  • I don’t care if it can record Vista/Win7 chrome.
  • Control over the framerate is nice; if I’m recording 5 minutes worth of typing code it’s not exactly vital that I get 24 FPS.
  • Transcoding to a Youtube-friendly format would be really nice.
  • Documentation about how to get the best possible quality on Youtube would be nice.
  • The ability to add annotations is required.
  • The ability to speed up or slow down portions of the video is required.
  • I’d prefer one tool over a collection of tools.
  • I’m unwilling to spend more than $125 with a slight margin if features are compelling.

The Contenders

To round out my stable of bullet lists, here’s the list of software I’ve tried or will discuss:

  • Fraps
  • Jing (Free)
  • CamStudio
  • Camtasia
  • Expression Encoder 4
  • BB Flashback Express
  • ScreenFlow

Fraps

This is probably the most famous screen recording tool on Windows. You can get it to record your desktop by clicking a checkbox that makes it monitor the DWM. However, it doesn’t record my mouse cursor, which is really nice for screencasting. It can only record the entire desktop. Any editing of the video must be done with a separate tool.

I’d really only recommend Fraps if you’re planning on recording yourself playing PC games; even then you’re going to have to do some post-recording work unless you fancy uploading 10GB to Youtube.

Jing

I don’t recall how I found out about Jing. It’s made by TechSmith, whose SnagIt is the best screenshot software I’ve ever used. I had high hopes for Jing because the SnagIt’s annotation capabilities were really good. There is a free version of Jing and a Pro version that costs $14.95 per year; I’m reviewing the free version. Both versions make you sign up for a screencast.com account before you can use the software; if you don’t like signing up for new services don’t bother with Jing. I don’t see any real benefit to screencast.com so it looks like a way for them to scrape some emails from signups.

Jing itself is pretty easy to use. It lets you pick the whole screen, a region, or a window, and it can record audio as well. The free version limits you to 5 minutes; that’s pretty harsh but it’s plenty of time to get a feel for how it works. The free version only lets you save to a Flash file or upload to screencast.com. The free version watermarks the Flash files it produces with some branding that appears briefly before the video starts. I’ve used Jing at work to save time writing out the steps to reproduce a bug; it’s perfect for this task.

For screencasting, I’m not sold. The Pro version only adds an MPEG-4 codec, removes the 5 minute limit, and lets you directly upload videos to Youtube. There’s no editing or annotation capabilities, and I have no idea what the Youtube quality would look like since there’s no way to evaluate the Pro features.

I’d recommend Jing for helping you fill out bug reports or making quick tutorial videos for family members that need help using some option in a program. The Flash export is really convenient for distribution. For anything complicated, I’m not sure it’s worth a recurring $14.95.

CamStudio

This is a GPL product so it’s free. I haven’t used it recently, but I used it in the past. I seem to remember it did good full-screen and region recording, and it could pan the region to follow the mouse cursor; it was designed in days when video was quite a luxury.

My main problem with CamStudio is a lack of annotation tools. It can output to AVI, SWF, or its own codec, but all of these will have to be edited in some other tool if you want to annotate your video.

It’s definitely a good choice if you know how to edit and transcode video with some other tool; you don’t get cheaper than free.

Camtasia

This is another TechSmith product; it’s Jing’s big brother. It’s got an impressive video editor and tons of options for exporting video and uploading it straight to Youtube. But it’s $300 and far out of my price range. If I blogged for a living I might be interested. It’s definitely one of the most capable-looking tools for Windows.

Expression Encoder 4

I may have used Expression Encoder 3 when I was working with it; I’m not sure. It’s a Microsoft product related to Expression Blend. There’s a free version and a paid version; it looks like the main difference is the free version limits you to 10-minute screencasts and doesn’t support IIS smooth streaming.

EE is not really an editing tool; it lets you glue different videos together and cut pieces out but you can’t do much more. Its main focus is on converting video to formats that Silverlight can stream well; one of those is H.264. I was mostly intimidated by it because it seemed like it was intended for people that already knew what they were doing. The configuration dialogs weren’t even as much help as a man page, so I found myself changing random numbers and hoping the outcome was good.

I can’t recommend EE because it looks like it’s just a fancy replacement for Windows Media Encoder and it doesn’t make much of an effort to be friendly to someone that just wants to turn their 4GB avi into something that takes less than a day to upload to Youtube.

BB Flashback Express

This one’s relatively unknown but has a lot going for it. Express is free, Standard is $89, and Pro is $199. I’ve only tried Express.

Express has the standard stable of screen recording features and is limited to AVI and Flash formats for export. Standard allows you to create WMV or H.264 video, both of which are nice additions. Standard also promises a nice set of editing features including annotation and re-recording audio. Pro lets you redo mouse movements which is really neat; it also enables frame-by-frame editing and changing the speed of playback.

BB Flashback Pro has all of the features I want, but $189 is far outside my budget considering I don’t blog for a living. Standard has some nice features, but it feels like paying $90 for only half of what I want is silly.

ScreenFlow

This is some software for Mac OS X that looks pretty promising. It’s $99 and has all of the features I want. The only trouble is I need something that works with Windows. I’ve been thinking about picking up a MacBook Air, and I’m curious if I could run Windows in a VM and record that.

Conclusion

There’s no solution that fits my requirements on Windows. The only software that provides all of the editing features costs more than $150. Spending $100 can get me close, but I feel like that’s a waste. While there are several free options, editing video using these options involves mastering several tools. And I’ve never found a good guide for making a screencast look good on Youtube other than vague recommendations like “make your video 1080p” or “use H.264”. I’d appreciate any help on this.

Early Adoption, Smartphones, and Chauvinism

I stumbled across an MSDN Magazine article from a tweet that pointed out it made the absurd statement that all people with smartphones are early adopters. That was strange enough, but then I found myself disgusted with an apparent attempt to prove why there are so few women in Computer Science.

I’m surprised they didn’t edit out the “if you have a smartphone you’re an early adopter” comment. If you trust Wikipedia, smartphones existed in 1992. RIM’s BlackBerry was out in 2002, and Windows CE was the OS for “Microsoft Windows Powered Smartphone 2002”. iPhone’s been out since 2007. I’m not so sure I’d qualify people buying a 3-year-old technology as early adopters, and I’d certainly avoid saying it for an 18-year-old technology. Are VB6 developers still early adopters? I suppose if you narrow the definition of “smartphone” to “Windows Phone 7” it works. At least MS doesn’t have Google’s arrogance and didn’t create a lame new term like “superphone”. I understand his point that smartphones tend to employ cutting-edge technologies, but if we apply that as the definition of early adoption then anyone with a PC is still an early adopter.

The rest of the article turns patronizing and downright sexist. I agree with the premise: we need to make smartphones to appeal to the masses, not just for profitability but because smartphones have the capability to reshape our lives. Mr. Platt suggests we should focus on making smartphones appeal to women to accomplish this. I assumed this would be followed with statistics that showed smartphone usage is a male-dominated area; no such data was referenced so I assume Mr. Platt is relying on his experience. What does Mr. Platt believe appeals to the modern woman? Grocery shopping! Managing appointments for the family! Nurturing children! Making sure the husband’s supper is warm when he gets home! I’m surprised the article doesn’t suggest an application that suggests which sandwich my wife should make for me, or sound an alarm when GPS detects her out of the kitchen to remind her where she belongs. Maybe Mr. Platt believes we should make Hello Kitty themed phones in addition to writing apps to help those of the other gender feel less threatened by this manly device. Oh, but he’s not entirely sexist. He did mention that the woman works a job before tending to her domestic duties.

I suppose I’m trying to get offended so I can write something sensational but I think he could have picked a better way to phrase his example. In particular, I think one should be sensitive when speaking specifically about women then focusing on tasks that line up with traditional gender roles in American society. There’s more to the market of people without smartphones than homemakers. To many people, a smartphone is just an expensive cell phone. Others don’t like the idea of being connected to the internet 24/7. A very large percentage of the country can’t afford smartphones even with the subsidized pricing (My smartphone bill could buy me lunch for a month.) The way to open the market is not to appeal to stereotypes. The way to open the market is to showcase how a smartphone can dramatically improve your quality of life by keeping track of menial information and coordinating with other people’s data. Scoble wrote a good post I can’t find right now outlining scenarios where a smartphone can really make life better. I’ll paraphrase from memory the kind of scenario that would get people to buy smartphones:

We’re in the minivan driving to the conference. My kid’s hungry, so I say “Phone, find me a place to stop and eat.” The phone looks at my GPS location and route and finds several restaurants along the way. It finds a Burger King in 2 miles but notices I tend to eat at McDonald’s more and picks one of those 4 miles away. “Does it have a playplace, I need one of those.” The phone checks a database, realizes there’s no playplace at this McDonald’s, and asks if I’d mind visiting one that’s 10 miles ahead instead. We stop and spend 30 minutes; the phone notices this will make us arrive 30 minutes later than expected and that I had a meeting I might be late to; it sends a notification to the participants to give me some extra time.

That’s the game-changing kind of functionality we need. Scoble wasn’t making this point (he was discussing how data islands are preventing this scenario) but it’s the future I want. If I wanted to do this kind of work today I’d have to pull over, check the map, find the restaurants, then start using individual websites to determine which restaurant had a playplace. My GPS can find the restaurants, but it doesn’t know which ones to suggest. It can’t tell me the closest McDonald’s is closed for renovations but there’s another one 12 miles ahead. It’d be up to me to check my route, recalculate arrival times, and pull over (again!) to notify my colleagues I’m running late. Ubiquitous web connectivity enables a device to perform these tasks for me, and a smartphone is the strongest candidate for a web-enabled device I’ll keep on my person at all times.

So I agree that practical apps are what will win the masses over. There is a need for apps that are recipe books, pantry managers, shopping list generators, and train schedule monitors. But it’s wrong to say these apps are important because women need them. Everyone needs them.

I’m additionally disappointed that Mr. Platt is writing about this as if it’s some revelation. When I went to Apple’s online app store, I found categories like “Apps for Cooks”, “Apps for Students”, and “Apps for Working Out”. The “Apps for Students” page suggests a National Geographic reference application, a scheduler and homework planner, and a couple of other useful applications for students. Apple knows real people are using their phones and has created lists that appeal to these real people. Android’s market is a step down from that; their online presence gives me categories, but isn’t searchable and isn’t organized well. I expected some kind of encyclopedia application at the top of “Reference”, but instead it’s “Worship Music”, “Final Fantasy XIV Guide”, and “Black Ops Intel Plus”. That definitely has the “this tool is for geeks only” vibe to it. What of Microsoft’s app store? If I don’t want to download Zune, all I get is “Featured”, “Top Paid”, and “Top Free”. A few of the apps look useful, but it really gives the impression that WP7 is a gaming platform for geeks. (Android’s app store is probably better on an Android too. I think it’s stupid to require your user to download software/purchase a phone to see your store.

It seems Apple’s the only company that’s figured out what Mr. Platt is saying and following up on it. Apple’s the only company that lets me get a phoneless view of their store that makes me feel like their device is intended for normal people. I suppose that’s why the article was written: if you narrow your definition of smartphone to “Windows Phone 7”, you find a world of early adopters and developers that need to write for the masses. This is why I’m waiting a while before picking up a WP7: it’s still going through the “geek app” phase Mr. Platt suggests the iPhone is still langushing in. No, real developers are writing influential apps for the iPhone. If it’s happening on Android or WP7, I can’t see it without buying a phone. They might want to see about that.

Update: Check out Word Lens: an application that translates text in an image from one language to another. Fancy fart app for the iPhone, no?

Perilous Parallel.For

I’ve been playing a lot with the Task Parallel Library because I’m really excited about the new Task-Based Asynchronous Pattern. Someone on the VB forum asked a question that was more or less about parallelizing a task, and I endeavored to make an example that compared the performance of some approaches.

When I got to the one that used Parallel.For, I started having some random incorrect results. Here’s some code that reproduces what I was seeing:

using System.Threading.Tasks;
using System.Threading;

static class Class1
{
    private const int NumberOfValues = 100;

    public static void Main()
    {
        int[] values = new int[NumberOfValues];
        int expectedSum = 0;
        for (int i = 0; i < values.Length; i++)
        {
            values[i] = i;
            expectedSum += i;
        }

        for (int i = 0; i < 10; i++)
        {
            int sum = 0;
            Parallel.For(0, values.Length, (int value) =>
                {
                    sum += value;
                    Thread.Sleep(10);
                });
            System.Console.WriteLine("{0}/{1}", expectedSum, sum);
        }
    }

}

The frequency of incorrect results seemed to depend on the presence and length of the Sleep() call. This told me it was most definitely a problem with concurrent access to the sum variable. My first attempt to fix it involved making a class with a lock:

class Sum
{
    private readonly object LockObject = new object();
    private int _sum;

    public int Sum
    {
        get
        {
            lock(LockObject) { return _sum; }
        }
        set
        {
            lock(LockObject) { _sum = value; }
        }
    }
}

It didn’t help. This is where I actually started converting the project to C# and planning out a StackOverflow post, but after a few more minutes of thought I figured out what was happening.

When sum += value executes, it looks like one atomic step, but it’s not. It has to execute in at least two steps: it must fetch the current value of sum, it must apply the + operator to that result, then it must assign the result to sum. There is a potential for some thread to assign a new value to sum after some other thread has retrieved it; this will cause some results to be lost. The Sum class above didn’t solve this because the thread synchronization only blocks concurrent access to the get and set individually. To solve the problem, I had to block access to the get and set simultaneously:

// ...snip

for (int i = 0; i < 10; i++)
{
    _sum = 0;
    Parallel.For(0, values.Length, (int value) =>
        {
            lock (LockObject)
            {
                _sum += value;
            }
            Thread.Sleep(10);
        });
    System.Console.WriteLine("{0}/{1}", expectedSum, _sum);
}

This works as expected (I later updated it to use Interlocked.Add()). But if you look closely, it seems to destroy the concurrency! The point of the code is to sum an array of values, but if each thread can only execute serially there’s no value to having threads. What gives?

I didn’t believe summation is impossible to parallelize because addition is commutative. However, since adding a value to the sum must be atomic, I feel like it might not be something that can be done in parallel. When I remove the Sleep() call, this approach is faster than a single-threaded approach but much slower than some other approaches I took using the thread pool; those let each thread tally its own sum then added each thread’s individual sum at the end. I wish I could think of a way to do that elegantly with Parallel.For().

With the Sleep() calls inserted, Parallel.For() is dramatically faster than any of the other approaches I tried; I assume this is because the Sleep() calls don’t block concurrent access and the TPL might be shuffling tasks out if they’re idle. While the Sleep() call may seem useless, it could represent 10ms worth of work that I have to do once I’ve updated _sum; if I cache any values needed in the lock statement I could still gain some benefit from working with the cached values outside of the lock statement.

So if you’re using Parallel.For(), be mindful of whether any variables you touch are safe for concurrent access. If they aren’t, consider different approaches.

I guess I have to weigh in on Silverlight

I’m going to assume you’re familiar with the Microsoft stack and moderately familiar with the current furor over the shift in Silverlight’s strategy. If you’re not, this is going to be a boring post and you should move on.

I see a few camps right now. There are people heavily invested in SL that feel a bit upset. There are some people heavily invested in SL that feel this is a misunderstanding. There are some people that state the writing on the wall’s always been there. Let’s focus on the last people, because they have the only position that isn’t summarized by its description.

Silverlight was interpreted as many things at release; I’ve always felt like it was supposed to be a Flash competitor. Regardless, the message has been that Silverlight was there to fill in some gaps that standard HTML didn’t fill well. Indeed, there’s some things it still does that HTML 5 and its related technologies don’t address well. Microsoft’s new opinion is that Silverlight was never intended to take the place of simple HTML 5 applications, and instead was only intended for web applications that required a level of integration with the system that could not be achieved with HTML, even the superhero HTML 5. The more I think about it, the more I can’t understand why I didn’t see this message shine through from the beginning. Perhaps it’s the fault of their evangelism of SL for all things web, but that’s a different discussion.

The thing I don’t like about the “Silverlight is rightfully being snubbed in favor of HTML 5” arguments is many proponents ignore the fact that Silverlight is still a good option for those applications that need tight integration with the system. Too many arguments chastize developers for picking SL in the first place. Too many arguments celebrate the passing of SL (which isn’t dead yet) and can’t wait for the world in which they build their awesome HTML 5 applications. Let’s talk about the SL application I’m heavily invested in.

I have a hand in a part of LabVIEW Web UI Builder. LabVIEW is a graphical dataflow programming language primarily targeted at engineers. It has excellent support for analysis of large data sets, superb interoperability with many types of hardware, and implemented intuitive support for multicore parallelism before the MS stack even realized it was something worth working on. The Web UI Builder is an implementation of the programming language in Silverlight with stripped-down support; it’s not intended to replace LV but it’s going to be very handy for some remote configuration scenarios that are tricky to handle with plain old native code. You can put a web browser on pretty much anything, and since SL’s client-side that device can serve SL content without requiring much in the way of muscle. It’s a great fit.

I see no way this could have been implemented in HTML, CSS, and JS. We require the ability to let the user create a UI, use those UI elements to create a diagram that represents program code, compile that code into something, and run that. It’s nice to have access to the user’s hard drive for this stuff. In the ideal case, it’d be nice to be able to work with hardware connected to the system. Last I checked, that’s not really on anyone’s roadmap for HTML and related technologies. While this is the case, there will always be room for Silverlight.

Will Microsoft toss aside Silverlight like they did VB6 when .NET released? I’m not sure. The initial statement gave that impression, and the clarification seemed to skirt around the issue. I’d like to think it’s all an issue of poor planning on the part of MS. It’s possible that their statements seem (to an insider) that the question of continued Silverlight support is answered. Unfortunately, they’re preaching to a crowd that’s seen numerous well-accepted technologies pushed aside to make room for the latest. I don’t know that the murmurs are going to die down until details about SL 5 start to slip. April’s quite a long way to go. I think most of the discussion would die down if MS made it immediately clear that WP7 and the shift to HTML 5 isn’t going to take resources away from developing Silverlight as an excellent no-brainer deployment for cross-platform applications that need system integration. Right now Java and Mono are the only real story for that, and neither puts money in Microsoft’s pocket.

I’m writing SL code so long as my managers feel it’s the right tool for the job. I’m not an independent developer able to change stacks on a whim, and I don’t like that most of the community seems to ignore those of us that have roots and can’t switch jobs on a whim. I’m aware of the excellent work going on in the non-Microsoft world and I like to keep an eye on Ruby, Python, and a handful of other languages just in case. But I’m happy with my job, I like the people I work with, and I don’t have any problems with the Microsoft stack that are specific to Microsoft. I don’t aim to fix things that aren’t broken.

Fiddling with the blog

I want to fiddle with the theme or find a theme that works better than the default one. If stuff looks broken, I’m probably going to fix it.

Probably.

*Update*
I’m done with what I want to fool with tonight; I need to do some research for the rest.