Who built ORCA? That's the wrong question
Ten days after an extremely disappointing election, the pundit class is pointing fingers in every direction. What caused the Romney loss? Was it the ground game? The media putting its thumb on the scales? Failure to press Benghazi as an issue? Ideology? Apathy? Gaffes? There's been plenty of good and bad pontificating on all of the above. Certainly the much-vaunted Project Orca was a failure. But why? I'd like to take a closer look today, and address the recent "who built it" scapegoat effort.
As someone who has managed web-based software application products and projects since there have been web-based software applications, and a Project Orca observer on the ground in Ohio on election day, I thought I'd bring my perspective to the problem. Not to spoil the ending, but I believe that the application itself was the least of Team Romney's problems on election day, and were overshadowed by problems with deployment and fatal problems with the concept itself.
In a nutshell, Project Orca was an innovative GOTV (get out the vote) project that was designed to gain efficiencies in election-day phone bank efforts. Traditional campaigns have staffed phone banks that call likely voters (for their party) on election day to make sure they vote. Twice a day, election officials post lists of who has voted (on paper, in the polling location). In the past, sophisticated campaigns have put people on the ground to view the lists and relay information to the folks at HQ, so that the people on the phones don't waste time calling those who have already voted. Obviously, this can be an inefficient and unwieldy process.
Orca, then, was conceived to streamline this information flow. With a mobile application, Team Romney observers could mark who voted in real time as each person requested their ballot. This would allow them to have a more targeted GOTV call list, updated continuously throughout the day, so that the Romney camp would call fewer people who already voted than the Obama camp. In a game of inches, this efficiency was worth money.
So what happened? There are some good first-hand accounts of failures out there on the web, and I'll borrow from some of those and append my own experience.
Observers didn't get proper credentials. There was quite a bit of information sent to us prior to Election Day. The day before, I received my credentials to allow me to observe. Not everyone received these, and it seems that this was a problem localized to certain areas (as the credentialing process is local). This is a huge failure, though it seems not one that occurred at the national level, and not one having anything to do with the application.
Observers got information, including voter lists, very late (the night before). Team Romney prepared (in a way) for an application failure by giving physical lists to observers (which were handy), and a mechanism to call in who voted (completely impractical). I had about 30 pages to print the night before, and many had lists much longer. It's a toll on the home printer and many people weren't prepared to do so, and had a very short window of time to resolve it. Why so late? It appears to have been intentional so that those who voted early could be removed, and the most up-to-date list could be given to the observer. More on this later.
Some observers could not log into the application. We had four observers at my particular polling location. Three of us had no trouble, the fourth had password issues. Apparently the entire state of Colorado could not log in, and there seemed to be pockets of people all over who had the wrong password information. Again, a huge failure, though it rests on those managing the rollout rather than those who designed and built the app.
However, the password reset utility did not work. You have to anticipate password issues in any web-based application. Always. When people called to say they could not log in, the people supporting the application could not reset or give the observer the proper password. This is one of those last-mile product features that is routine, but often overlooked, and it has to work. This failure does rest squarely on whoever developed the app.
There wasn't a proper redirect. The login URL was https, for a secure connection. However, most folks typing the URL into their browser (including me at first) ignored the "s" and used a regular http address. We got a vague error. Huge failure for not automatically redirecting anyone who used http to https automatically. However, this was a failure of whoever deployed the application, not the people who built it (we do not know that they were the same).
It wasn't a "native app." There are plenty of instances of people complaining that they couldn't find the app in the Android or Apple stores. True, but a somewhat unfair criticism. This was a browser-based application, and we were told in our training that you wouldn't find it in an app store. Given the short window to deliver the product, this was almost certainly the right way to go. However, my training session included mostly retirees. Did they count on this demographic being the ones who were going to be having to use the app? Was that distinction clear to everyone? Probably not, though I'm not sure what could have been done to alleviate the problem.
Except... observers couldn't try out the app in advance. Lots of us are planners, and want to know how things are going to work ahead of time. The application was not made available until Election Day. So issues like using the correct URL, having the right password, knowing not to look in the app store, didn't surface until it counted. I'm sure there were secrecy concerns and logistics concerns that fed into this decision, but it cost them.
The application crashed. The entire system went down somewhere in the neighborhood of 4:30 (based on my recollection). In the aftermath, it appears, inexplicably, that the traffic from 37,000 volunteers in the polling locations, and who knows how many accessing it on the back end, was being handled by a single web server. This is less than junior varsity. Redundancy is rule #1 in any mission-critical web deployment. Hardware is cheap. Team Romney reportedly spent $70 million on this application. But for want of a few thousand to build a proper deployment with multiple servers and proper load balancing and failover precautions, the entire effort was crippled (though I'll get to the reality of that in a moment). Again, this has nothing to do with who built the application, and everything to do with who managed the deployment.
That said, I had no problems with it (until 4:30), and neither did the other three workers at my location who were able to log in. It was very easy to use, and seemed a vast improvement from having to phone in results when lists are posted at 11am and 4pm. Until I realized it was all wrong.
Shortly after 4pm, after the second list was posted, a poll worker who went through the list announced that there were only about 250 people left in the precinct who hadn't voted. That seemed wrong to me. I checked the app, which I believed contained the same information as the hardcopy lists, and it showed there were at least 800 who hadn't yet voted. I had a sinking feeling, and realized that the app didn't have any of the absentee or early voters filtered out. In other words, the whole purpose of the exercise was to whittle down the list. Yet a massive block of names that could have been easily crossed off the night before, wasn't. Assuming the people manning the phone banks were using the same information we had (that was the point right?) this was a huge strategic fail.
The posted lists had an "A" next to each person who had requested an absentee ballot or voted early. The majority were updated by computer, but many were put in by hand as poll workers accounted for "late" early voters (I voted the day before, and the A was penciled in next to my name). All of the machine-printed A's were filtered out of the hardcopy lists we received the night before. But the application did not filter them out. And nobody received instructions to mark them off in the application. Technically, having the A simply meant they requested the ballot. It does not mean they sent it in. However the hand-written A's definitely denoted early voters. In any case, it was a significant number, and they should have all been filtered out of the call lists.
It was very slow at our location at this point (everyone had already voted, and we didn't realize it!). We scrambled to divide up the lists and start updating all the absentee information. One of us called Romney HQ. The first responder said, no I don't think you have to worry about those, but let me check. Then he came back with: yes, keep crossing off the absentees! And then the application crashed.
But even if it hadn't, we were leaving at best a two-hour window for people to dial potential voters (at their home numbers) and hope to get them back out of the house to go vote by 7:00. It was very late in the game by that point, which is why the application crashing, while a massive failure, probably had little impact on actual results.
In retrospect, given the huge number of early voters in Ohio, the campaign would have been far better served in having people like me work overnight to cross last-minute early/absentee voters off the list than to worry about the incremental changes over the course of the day. To get a bit Biblical, the project focused on the speck and not the beam in their eye.
But the failure is bigger than that. If the concept worked perfectly, how effective would it have been? Could a revised version of Orca be a model for the future? No, I don't think so. Fewer and fewer people have home phones in the first place. That trend is not about to reverse itself. Those who do laugh about the only calls they receive being election-related. How many are answering by election day, with caller ID being ubiquitous? Is a message left on a machine from a live person on election day so much different from a robocall every day prior? It can't possibly be the best use of $70 million.
Team Obama worked proactively to target prospective voters to make them want to get out and vote. Team Romney focused on reactively getting people who don't care enough to vote out to the polls by badgering them before the clock strikes seven. Which firm built the Orca application? Who cares? A slightly flawed application met with a disastrous implementation in the service of an entirely outdated strategy. Scapegoating application development will only give us answers leading us in the wrong direction.
As someone who has managed web-based software application products and projects since there have been web-based software applications, and a Project Orca observer on the ground in Ohio on election day, I thought I'd bring my perspective to the problem. Not to spoil the ending, but I believe that the application itself was the least of Team Romney's problems on election day, and were overshadowed by problems with deployment and fatal problems with the concept itself.
In a nutshell, Project Orca was an innovative GOTV (get out the vote) project that was designed to gain efficiencies in election-day phone bank efforts. Traditional campaigns have staffed phone banks that call likely voters (for their party) on election day to make sure they vote. Twice a day, election officials post lists of who has voted (on paper, in the polling location). In the past, sophisticated campaigns have put people on the ground to view the lists and relay information to the folks at HQ, so that the people on the phones don't waste time calling those who have already voted. Obviously, this can be an inefficient and unwieldy process.
Orca, then, was conceived to streamline this information flow. With a mobile application, Team Romney observers could mark who voted in real time as each person requested their ballot. This would allow them to have a more targeted GOTV call list, updated continuously throughout the day, so that the Romney camp would call fewer people who already voted than the Obama camp. In a game of inches, this efficiency was worth money.
So what happened? There are some good first-hand accounts of failures out there on the web, and I'll borrow from some of those and append my own experience.
Observers didn't get proper credentials. There was quite a bit of information sent to us prior to Election Day. The day before, I received my credentials to allow me to observe. Not everyone received these, and it seems that this was a problem localized to certain areas (as the credentialing process is local). This is a huge failure, though it seems not one that occurred at the national level, and not one having anything to do with the application.
Observers got information, including voter lists, very late (the night before). Team Romney prepared (in a way) for an application failure by giving physical lists to observers (which were handy), and a mechanism to call in who voted (completely impractical). I had about 30 pages to print the night before, and many had lists much longer. It's a toll on the home printer and many people weren't prepared to do so, and had a very short window of time to resolve it. Why so late? It appears to have been intentional so that those who voted early could be removed, and the most up-to-date list could be given to the observer. More on this later.
Some observers could not log into the application. We had four observers at my particular polling location. Three of us had no trouble, the fourth had password issues. Apparently the entire state of Colorado could not log in, and there seemed to be pockets of people all over who had the wrong password information. Again, a huge failure, though it rests on those managing the rollout rather than those who designed and built the app.
However, the password reset utility did not work. You have to anticipate password issues in any web-based application. Always. When people called to say they could not log in, the people supporting the application could not reset or give the observer the proper password. This is one of those last-mile product features that is routine, but often overlooked, and it has to work. This failure does rest squarely on whoever developed the app.
There wasn't a proper redirect. The login URL was https, for a secure connection. However, most folks typing the URL into their browser (including me at first) ignored the "s" and used a regular http address. We got a vague error. Huge failure for not automatically redirecting anyone who used http to https automatically. However, this was a failure of whoever deployed the application, not the people who built it (we do not know that they were the same).
It wasn't a "native app." There are plenty of instances of people complaining that they couldn't find the app in the Android or Apple stores. True, but a somewhat unfair criticism. This was a browser-based application, and we were told in our training that you wouldn't find it in an app store. Given the short window to deliver the product, this was almost certainly the right way to go. However, my training session included mostly retirees. Did they count on this demographic being the ones who were going to be having to use the app? Was that distinction clear to everyone? Probably not, though I'm not sure what could have been done to alleviate the problem.
Except... observers couldn't try out the app in advance. Lots of us are planners, and want to know how things are going to work ahead of time. The application was not made available until Election Day. So issues like using the correct URL, having the right password, knowing not to look in the app store, didn't surface until it counted. I'm sure there were secrecy concerns and logistics concerns that fed into this decision, but it cost them.
The application crashed. The entire system went down somewhere in the neighborhood of 4:30 (based on my recollection). In the aftermath, it appears, inexplicably, that the traffic from 37,000 volunteers in the polling locations, and who knows how many accessing it on the back end, was being handled by a single web server. This is less than junior varsity. Redundancy is rule #1 in any mission-critical web deployment. Hardware is cheap. Team Romney reportedly spent $70 million on this application. But for want of a few thousand to build a proper deployment with multiple servers and proper load balancing and failover precautions, the entire effort was crippled (though I'll get to the reality of that in a moment). Again, this has nothing to do with who built the application, and everything to do with who managed the deployment.
That said, I had no problems with it (until 4:30), and neither did the other three workers at my location who were able to log in. It was very easy to use, and seemed a vast improvement from having to phone in results when lists are posted at 11am and 4pm. Until I realized it was all wrong.
Shortly after 4pm, after the second list was posted, a poll worker who went through the list announced that there were only about 250 people left in the precinct who hadn't voted. That seemed wrong to me. I checked the app, which I believed contained the same information as the hardcopy lists, and it showed there were at least 800 who hadn't yet voted. I had a sinking feeling, and realized that the app didn't have any of the absentee or early voters filtered out. In other words, the whole purpose of the exercise was to whittle down the list. Yet a massive block of names that could have been easily crossed off the night before, wasn't. Assuming the people manning the phone banks were using the same information we had (that was the point right?) this was a huge strategic fail.
The posted lists had an "A" next to each person who had requested an absentee ballot or voted early. The majority were updated by computer, but many were put in by hand as poll workers accounted for "late" early voters (I voted the day before, and the A was penciled in next to my name). All of the machine-printed A's were filtered out of the hardcopy lists we received the night before. But the application did not filter them out. And nobody received instructions to mark them off in the application. Technically, having the A simply meant they requested the ballot. It does not mean they sent it in. However the hand-written A's definitely denoted early voters. In any case, it was a significant number, and they should have all been filtered out of the call lists.
It was very slow at our location at this point (everyone had already voted, and we didn't realize it!). We scrambled to divide up the lists and start updating all the absentee information. One of us called Romney HQ. The first responder said, no I don't think you have to worry about those, but let me check. Then he came back with: yes, keep crossing off the absentees! And then the application crashed.
But even if it hadn't, we were leaving at best a two-hour window for people to dial potential voters (at their home numbers) and hope to get them back out of the house to go vote by 7:00. It was very late in the game by that point, which is why the application crashing, while a massive failure, probably had little impact on actual results.
In retrospect, given the huge number of early voters in Ohio, the campaign would have been far better served in having people like me work overnight to cross last-minute early/absentee voters off the list than to worry about the incremental changes over the course of the day. To get a bit Biblical, the project focused on the speck and not the beam in their eye.
But the failure is bigger than that. If the concept worked perfectly, how effective would it have been? Could a revised version of Orca be a model for the future? No, I don't think so. Fewer and fewer people have home phones in the first place. That trend is not about to reverse itself. Those who do laugh about the only calls they receive being election-related. How many are answering by election day, with caller ID being ubiquitous? Is a message left on a machine from a live person on election day so much different from a robocall every day prior? It can't possibly be the best use of $70 million.
Team Obama worked proactively to target prospective voters to make them want to get out and vote. Team Romney focused on reactively getting people who don't care enough to vote out to the polls by badgering them before the clock strikes seven. Which firm built the Orca application? Who cares? A slightly flawed application met with a disastrous implementation in the service of an entirely outdated strategy. Scapegoating application development will only give us answers leading us in the wrong direction.
Comments