Friday, November 16, 2012

Who built ORCA? That's the wrong question

Ten days after an extremely disappointing election, the pundit class is pointing fingers in every direction. What caused the Romney loss? Was it the ground game? The media putting its thumb on the scales? Failure to press Benghazi as an issue? Ideology? Apathy? Gaffes? There's been plenty of good and bad pontificating on all of the above. Certainly the much-vaunted Project Orca was a failure. But why? I'd like to take a closer look today, and address the recent "who built it" scapegoat effort.

As someone who has managed web-based software application products and projects since there have been web-based software applications, and a Project Orca observer on the ground in Ohio on election day, I thought I'd bring my perspective to the problem. Not to spoil the ending, but I believe that the application itself was the least of Team Romney's problems on election day, and were overshadowed by problems with deployment and fatal problems with the concept itself.

In a nutshell, Project Orca was an innovative GOTV (get out the vote) project that was designed to gain efficiencies in election-day phone bank efforts. Traditional campaigns have staffed phone banks that call likely voters (for their party) on election day to make sure they vote. Twice a day, election officials post lists of who has voted (on paper, in the polling location). In the past, sophisticated campaigns have put people on the ground to view the lists and relay information to the folks at HQ, so that the people on the phones don't waste time calling those who have already voted. Obviously, this can be an inefficient and unwieldy process.

Orca, then, was conceived to streamline this information flow. With a mobile application, Team Romney observers could mark who voted in real time as each person requested their ballot. This would allow them to have a more targeted GOTV call list, updated continuously throughout the day, so that the Romney camp would call fewer people who already voted than the Obama camp. In a game of inches, this efficiency was worth money.

So what happened? There are some good first-hand accounts of failures out there on the web, and I'll borrow from some of those and append my own experience.

Observers didn't get proper credentials. There was quite a bit of information sent to us prior to Election Day. The day before, I received my credentials to allow me to observe. Not everyone received these, and it seems that this was a problem localized to certain areas (as the credentialing process is local). This is a huge failure, though it seems not one that occurred at the national level, and not one having anything to do with the application.

Observers got information, including voter lists, very late (the night before). Team Romney prepared (in a way) for an application failure by giving physical lists to observers (which were handy), and a mechanism to call in who voted (completely impractical). I had about 30 pages to print the night before, and many had lists much longer. It's a toll on the home printer and many people weren't prepared to do so, and had a very short window of time to resolve it. Why so late? It appears to have been intentional so that those who voted early could be removed, and the most up-to-date list could be given to the observer. More on this later.

Some observers could not log into the application. We had four observers at my particular polling location. Three of us had no trouble, the fourth had password issues. Apparently the entire state of Colorado could not log in, and there seemed to be pockets of people all over who had the wrong password information. Again, a huge failure, though it rests on those managing the rollout rather than those who designed and built the app.

However, the password reset utility did not work. You have to anticipate password issues in any web-based application. Always. When people called to say they could not log in, the people supporting the application could not reset or give the observer the proper password. This is one of those last-mile product features that is routine, but often overlooked, and it has to work. This failure does rest squarely on whoever developed the app.

There wasn't a proper redirect. The login URL was https, for a secure connection. However, most folks  typing the URL into their browser (including me at first) ignored the "s"  and used a regular http address. We got a vague error. Huge failure for not automatically redirecting anyone who used http to https automatically. However, this was a failure of whoever deployed the application, not the people who built it (we do not know that they were the same).

It wasn't a "native app." There are plenty of instances of people complaining that they couldn't find the app in the Android or Apple stores. True, but a somewhat unfair criticism. This was a browser-based application, and we were told in our training that you wouldn't find it in an app store. Given the short window to deliver the product, this was almost certainly the right way to go. However, my training session included mostly retirees. Did they count on this demographic being the ones who were going to be having to use the app? Was that distinction clear to everyone? Probably not, though I'm not sure what could have been done to alleviate the problem.

Except... observers couldn't try out the app in advance. Lots of us are planners, and want to know how things are going to work ahead of time. The application was not made available until Election Day. So issues like using the correct URL, having the right password, knowing not to look in the app store, didn't surface until it counted. I'm sure there were secrecy concerns and logistics concerns that fed into this decision, but it cost them.


The application crashed. The entire system went down somewhere in the neighborhood of 4:30 (based on my recollection). In the aftermath, it appears, inexplicably, that the traffic from 37,000 volunteers in the polling locations, and who knows how many accessing it on the back end, was being handled by a single web server. This is less than junior varsity. Redundancy is rule #1 in any mission-critical web deployment. Hardware is cheap. Team Romney reportedly spent $70 million on this application. But for want of a few thousand to build a proper deployment with multiple servers and proper load balancing and failover precautions, the entire effort was crippled (though I'll get to the reality of that in a moment). Again, this has nothing to do with who built the application, and everything to do with who managed the deployment. 

That said, I had no problems with it (until 4:30), and neither did the other three workers at my location who were able to log in. It was very easy to use, and seemed a vast improvement from having to phone in results when lists are posted at 11am and 4pm. Until I realized it was all wrong.

Shortly after 4pm, after the second list was posted, a poll worker who went through the list announced that there were only about 250 people left in the precinct who hadn't voted. That seemed wrong to me. I checked the app, which I believed contained the same information as the hardcopy lists, and it showed there were at least 800 who hadn't yet voted. I had a sinking feeling, and realized that the app didn't have any of the absentee or early voters filtered out. In other words, the whole purpose of the exercise was to whittle down the list. Yet a massive block of names that could have been easily crossed off the night before, wasn't. Assuming the people manning the phone banks were using the same information we had (that was the point right?) this was a huge strategic fail.

The posted lists had an "A" next to each person who had requested an absentee ballot or voted early. The majority were updated by computer, but many were put in by hand as poll workers accounted for "late" early voters (I voted the day before, and the A was penciled in next to my name). All of the machine-printed A's were filtered out of the hardcopy lists we received the night before. But the application did not filter them out. And nobody received instructions to mark them off in the application. Technically, having the A simply meant they requested the ballot. It does not mean they sent it in. However the hand-written A's definitely denoted early voters. In any case, it was a significant number, and they should have all been filtered out of the call lists.

It was very slow at our location at this point (everyone had already voted, and we didn't realize it!). We scrambled to divide up the lists and start updating all the absentee information. One of us called Romney HQ. The first responder said, no I don't think you have to worry about those, but let me check. Then he came back with: yes, keep crossing off the absentees! And then the application crashed.

But even if it hadn't, we were leaving at best a two-hour window for people to dial potential voters (at their home numbers) and hope to get them back out of the house to go vote by 7:00. It was very late in the game by that point, which is why the application crashing, while a massive failure, probably had little impact on actual results.

In retrospect,  given the huge number of early voters in Ohio, the campaign would have been far better served in having people like me work overnight to cross last-minute early/absentee voters off the list than to worry about the incremental changes over the course of the day. To get a bit Biblical, the project focused on the speck and not the beam in their eye.

 But the failure is bigger than that. If the concept worked perfectly, how effective would it have been? Could a revised version of Orca be a model for the future? No, I don't think so. Fewer and fewer people have home phones in the first place. That trend is not about to reverse itself. Those who do laugh about the only calls they receive being election-related. How many are answering by election day, with caller ID being ubiquitous? Is a message left on a machine from a live person on election day so much different from a robocall every day prior? It can't possibly be the best use of $70 million.

Team Obama worked proactively to target prospective voters to make them want to get out and vote. Team Romney focused on reactively getting people who don't care enough to vote out to the polls by badgering them before the clock strikes seven. Which firm built the Orca application? Who cares? A slightly flawed application met with a disastrous implementation in the service of an entirely outdated strategy. Scapegoating application development will only give us answers leading us in the wrong direction.

Tuesday, September 04, 2012

Fact Check

So it's interesting that all these news organizations are starting the Orwellian "fact check" campaigns. Isn't that part of every journalist's job? Are they admittedly only checking the facts in the one article each day? Sounds about right. But now we've gone beyond cherry-picking whose facts to check, and have wrapped opinion journalism snugly in the context of a fact check. Here's one from ABC on Romney's speech:

http://abcnews.go.com/Politics/OTUS/fact-check-mitt-romneys-republican-national-convention-speech/story?id=17119764#.UEYSFiIWoRY

Please tell me which fact he got wrong. Some of these are jaw-dropping, like the one about the Poles and Czechs. BTW, what do Vaclav Havel and Lech Walesa think about it? Dare they ask? Read the tortured logic on how the Medicare cuts aren't really cuts, and the only word that will spring to mind is "shill."

Tuesday, August 28, 2012

GOP Convention Notes: Day 1

Kelly Ayotte had a slower delivery than a unionized pizza shop. Good grief.

Gilchrist's accent reminds me of home.

Kasich knocked it out of the park at the Romney rally on Saturday, borrowed heavily from it for today, but wasn't nearly as good in such a formal setting.

Would love to catch Gwen Ifil's mic open after she finished with Terry Branstead.

Mark Shields seems completely baffled by the purpose of the convention.

Wow. Huge ovation for Scott Walker's entrance. Well deserved. 94% of employers think Wisconsin headed in the right direction vs just 10% previously.

225th anniversary if US Constitution coming up. I wonder if anyone will care when we hit 250?

Every question from Judy and Gwen is challenging. Does anyone doubt that tune will change in a few weeks? "is it hard being so smart? What's on your iPod?"

Senator Barrasso has some gravitas. Shoved Shields' EPA question neatly back down his throat. Wow, they are jumping all over themselves to ask him things to trip him up, but he has been unfazed.

Shields is monologuing Democratic talking points now. Ridiculous.

Santorum: in 1923 there were no government benefits for immigrants, except one: freedom.

The president plan didn't work for America because that's not how America works.

Families, education, hard work.

Santorum hits on the legality of Obama changing welfare law. Good point that is overlooked.

Nice riff on the hard-working hands that built America. Well done. Not much about Romney, but a nice speech highlighting a conservative vision.

Judy Woodruff is clearly troubled by her chat with Newt.

Ted Cruz getting a rousing response.
He resembles a preacher more than a politician. Maybe it's because he has convictions.

Free markets, fiscal responsibility and individual liberty.

Gwen just asked Newt what advice he would give Obama in defeating Romney. What?? Annoyed they kept trying to hit Newt about the GOP having few minorities, with Susanna Martinez speaking in the background getting no airtime.

Would pay good money to see an Ann Romney - Michelle Obama debate. She's doing very well. Didn't realize her dad was an immigrant.

Not a storybook marriage - a real marriage.

Wow, she is very good. Very credible.

"If the last four years had been more successful, do you really think they'd be attacking Mitts success?"

She may be articulating capitalism better than anyone else I've seen recently. Fantastic defense of Bain.

Thursday, June 16, 2011

Baffling

Catching upon some of James Delingpole's work for The Telegraph in which he notes the industrial blight that is about to be unnecessarily perpetrated upon the people of Wales in the form of wind turbines. Beautiful countryside will become disfigured, for the purpose of gaining an unreliable energy source. When you think of how many people live on an island as small as the U.K, it's remarkable such tracts of virtually unmolested land exist at all. But they do, and we'll mar them in the name of being green.

Of course, the irony of this is that we can't drill for oil in the Arctic National Wildlife Refuge because if it's "pristine-ness." That the actual area where oil will be extracted is possibly the ugliest natural place on planet earth, that no human being would ever willingly venture there were it not for oil, that the entire industrial footprint would be the size of a medium-sized airport seem not to matter to these zealots who seem to think that  all the trouble in the world stems from American oil companies.

The idea of "American Exceptionalism" has been discussed a bit recently, and rests on a belief that we are, at the core of who we are and how we govern ourselves... different. It used to mean different in terms of being more free. Unfortunately, today we are losing many of our personal and economic freedoms, and find that the only area where we are truly different from everyone else is our unwillingness to use our natural energy resources. A tragedy is unfolding before us, and future historians, though they will have the entire record available to them, will shake their heads in wonder as they strap on their skates to get their exercise on the frozen Thames.

Monday, January 03, 2011

The Man with the Mustache

Well it's 2011. That means it's time to start seriously talking about 2012, no? Jay Nordlinger at National Review makes a good case for John Bolton. I can't say I can disagree with anything about the man; I do question his electability. Still, why not latch onto this guy during primary season instead of anybody else? When Palin and Huckabee are mentioned as possible strong candidates, I shudder. Probably because I believe it. I read something interesting the other day about the two strains of candidates emerging in the Republican primary season: ideological and managerial. Palin heads up the ideological wing and thus has huge Tea Party support. Romney heads the managerial wing and gets more traditional (and RINO) support. The managerial type is going to be more apt to get crossover milquetoast votes... those same voters that switch back and forth depending upon which way the wind is blowing.

However, I think the ideological side can attract votes too, if it's positioned strongly enough. Very few people really want huge, bloated government and higher taxes. But we have no Reagan in our midst for people to rally to. Palin is not gonna do it. Huckabee is a joke and the worst kind of candidate in that he is primarily a social conservative, and a fiscal conservative in name only. Meaning he can't win, and if somehow he did, he'd give us the same tax-and-spend BS we've had previously.

I would sleep pretty well knowing John Bolton is in charge. And I would pay pretty decent money to watch that October 2012 foreign policy debate between him and Obama. Of course, given that the debates are put on by the networks, I would guess that should Bolton miraculously get the nomination, the debate formats will miraculously change to put less emphasis on foreign policy. Still, one can dream.

http://www.nationalreview.com/articles/256081/man-mustache-jay-nordlinger

Wednesday, November 03, 2010

So...

Disappointed in some of the races. We got the ones we were supposed to get, but not many of the true toss-ups. Toomey and Kirk being the exceptions. The Tea Party certainly inspired a lot of the change, but Miller, Angle, and O'Donnell going down in flames does not bode well for 2012. On the other hand, I'm excited about Rubio, excited about Kasich, and excited about Col. Allen West winning a House seat in Florida. Disappointed McClung (the rocket scientist in Arizona) couldn't beat the incumbent, but she was badly outspent. Amazed Barney Frank beat Bielat by such a huge margin given what a scumbag Frank is and what a smart, stand-up guy Bielat is. But it's on par with Rangel winning with 85% of the vote. He's a criminal and a gangster, but he's our criminal. That must be the thought process. If you don't believe government should be about laws being applied to people equally, and think it should be about changing outcomes and getting more for your group, then why not vote for the gangster? Similarly, Mark Dayton winning in Minnesota is yet another statewide electoral disgrace from the Land of a Thousand Flakes. And then there's Jerry Brown. Jerry Brown?? Seriously? Jerry f-ing Brown? Also disappointed Joe DioGuardi got trounced in NY, but looking at the cash, he got outspent 6-1.



Tuesday, November 02, 2010

It stands to reason

I sit puzzling over why it seems that Republicans are the only ones who are serious about tossing out the unethical in their ranks. Charlie Rangel, Mark Dayton, Barney Frank spring to mind as I watch the returns. But it occurred to me that if you subscribe to the concept of a "gangster government" as Michael Barone termed the Obama administration, it doesn't make any sense to vote for anyone but your own gangsters.