Algorithms nowadays do more than determining which books, DVDs, CDs or whatever you might want at Amazon. While that’s always amusing for me, given my catholic tastes (and no, I don’t like Johnny Cash, but thanks for keeping on suggesting him), it’s more puzzling when it decides that my partner – who recently bought a tent – must now want several more. We’re not planning on building a tented city anywhere, but Amazon’s algorithms struggle with the difference between wants and needs, no matter how many books about Maslow’s Heirarchy their parent site may sell.

Commonly overlooked but increasingly ubiquitous in our lives, algorithms deserve more of our attention. And better understanding: the nearest convenient dictionaries define an algorithm as “A process or set of rules to be followed in calculations or other problem-solving operations, esp. by a computer” or – in the ‘defined for kids’ option – as “a step-by-step method for solving a problem (as finding the greatest common divisor) or accomplishing a goal”. As the embedding of algorithmic ‘solutions’ (in the Private Eye column sense) goes, suggesting a book or two is relatively harmless: I can combine my catholicism with a discriminatory touch. But the human tendency to codify has seen them implemented in places where the potential is more significant, and not necessarily in a positive way.

Here’s David Schwartz, writing in the Financial Times last week:

High-frequency trading – and its effect on stock-market trends – continues to worry me. It is entirely computer-driven, with trades being triggered by complex mathematical computations. Automated trading systems typically buy or sell shares within a fraction of a second after an opportunity is spotted. Some trades are closed a few seconds later.

[… ] I have no opinion about the value of HF trading in normal times when shares drift by manageable amounts. But when markets move strongly in a single direction like we saw in recent weeks, I have a strong suspicion that HF trading adds to the problem.

Jane Wakefield, a BBC Technology correspondent, had been less sanguine just 3 days earlier when she wrote:

In the so-called Flash Crash of 2.45 [pm] on May 6 2010, a five minute dip in the markets caused momentary chaos.

A rogue trader was blamed for the 10% Dow Jones index fall but in reality, it was the computer program that the unnamed trader was using that was really to blame.

The algorithm sold 75,000 stocks with a value of £2.6bn in just 20 minutes, causing other super-fast trading algorithms to follow suit.

(Schwartz had similarly mentioned the Flash Crash, although less explicitly linking cause and effect.)

As Dr Mark Parkinson pointed out at his blog recently, actually it’s more complicated than that – although possibly not in a very encouraging way. (While you read the rest of this post and any links made from it, bear in mind the value of your personal pension pot: you may find that doing so focuses your attention.) Mark’s cultural antennae had been piqued by the concept of ‘sentiment tracking’ – which is apparently something to the effect of ‘the operation and impact of emotion at a societal level’ (even if it does sound rather more like a detuned, pre-watershed version of kerb crawling).

Mark thoughtfully provided a link to research paper that had been published in the Journal of Computational Science (you can read it here in PDF format), in which the authors “investigate whether measurements of collective mood states derived from largescale Twitter feeds are correlated to the value of the Dow Jones Industrial Average (DJIA) over time” – although readers should be warned that its contents include the kind of mathematical formulae that look as if the typesetter not only fell asleep, but had a rather restless snooze as (s)he rolled about on the keys. The article questions the assertion of the Efficient Market Hyopthesis [EMH] that the market responds only to news lightens the mood a little, albeit blackly (I did laugh at the idea of an “efficient market hypothesis” in the light of the last few years, but then I remembered my pension …)

While I’d already happily conceded that you could view Twitter as a kind of collective mood ring for our times, I struggle slightly with the idea of it as a crystal ball. Recent press coverage in the UK, following the recent riots, seemed to – once everyone had recovered from their initial shock/outrage/raised eyebrow – conclude that social media had mostly followed events rather than predicting or instigating them. (Apart from a fascinating insight into one cross-section of the world’s breakfast and shopping habits, Twitter is also the 21st century home of amateur reportage.) The concept of it as a driving factor on the world’s stock markets was a little too startling, especially given its own tendency to act as a rapidly churning combination of roller coaster and amplifier. (Those of you more comfortable with a good old-fashioned book might want to read Guardian columnist Grace Dent’s recent book about the Twitter experience for more on this.)

Undoubtedly useful in some situations – tweeting with your phone’s keys set not to click from under a hotel bed during the attacks on Mumbai, for example – Twitter’s chaotic use of hash-tags, promotion of what’s ‘trending’ above all else, its cliques and its attraction to those not much given to calm rational reflection and a keen sense of the distinction between public and private all marked it down, for me at least, as a poor choice of influencer on the financial prospects of nations. All things being equal (as economists were once prone to say), it’d be a risky gamble: at least with toss ha’penny you still get the ha’penny …

I don’t doubt that large groups of people vaguely sharing a similar emotion can have an impact on the mood, but my concerns for my economic wellbeing increased quite sharply when I read part of the authors’ conclusion:

Information can with great ease and at very short time-scales travel along the ties in an online social networking environment thereby exerting an equalizing effect on investment strategies (and mood) of individual and professional traders alike. On the other hand, rumors and misleading information can spread with equal if not greater efficiency across the same social networking ties as demonstrated by the prevalence of so called “astroturfing” and “Twitter bombing” campaigns.This puts large groups of individuals that can not rely on the infrastructure of professional traders at a significant disadvantage and may in fact increase market volatility.

It was that “infrastructure of professional traders” that really got me, given that 70% of trading happens through the actions of ‘clever’ little bits of maths rather than the insightful minds of skilled and experienced human operators. It was quite disillusioning that the accumulated mood of Twitter – which, by the way, most of us will experience entirely differently depending on who we’re following – is seen as influencing anyone’s investment strategies. Shouldn’t we be Twitter bombing to campaign for Emotional Intelligence training for everyone employed in financial services, and assessing training candidates on the emotional impact on them of the news that several thousand Lady Gaga fans are ‘disappointed’ with her new frock/song/video/choice of bacon rasher?

As The Economist pointed out in highlighting the foibles and inaccuracies of TripAdvisor – another source of ‘advice’ for our age:

The holy grail would clearly be a comprehensive tool to ensure that every review on the site—both good and bad—is genuine. The use of software to spot fake reviews can help, as can the use of algorithms and simple human perusal.

The Cold War may be over, but its habits live on: we live in a potential Golden Age of Disinformation, either planted for commercial benefit, or misinterpreted or misread through the lack of thought in filtering our ‘information’ and where it comes from. (David Shenk was years ahead of the curve on this when he wrote Data Smog, although Eli Periser and Kevin Slavin are bringing these issues back to the public eye in recent months: if more of us tweet about this, we might even get someone’s attention.) Perhaps we could take a tip from today’s spooks: most of them have to hand in their mobile as they enter the workplace as a security risk.

More importantly, shouldn’t we be thinking a little more carefully about how our cherished new tools are shaping us? Evolving to the point where our economy heads south if enough of us click ‘Dislike’ on the latest ephemeral output of a passing celeb doesn’t sound like progress.

Add to: Facebook | Digg | Del.icio.us | Stumbleupon | Reddit | Blinklist | Twitter | Technorati | Furl | Newsvine

Add to FacebookAdd to NewsvineAdd to DiggAdd to Del.icio.usAdd to StumbleuponAdd to RedditAdd to BlinklistAdd to TwitterAdd to TechnoratiAdd to Furl