Friday 5 August 2011

Automated sentiment analysis gives poor showing in accuracy test

2 June 2010 | By Brian Tarran

Coin flip

UK— Automated sentiment analysis is less accurate then flipping a coin when it comes to determining whether brand mentions in social media are positive or negative, according to a white paper from FreshMinds.
Tests of a range of different social media monitoring tools conducted by the research consultancy found that comments were, on average, correctly categorised only 30% of the time.
FreshMinds’ experiment involved tools from Alterian, Biz360, Brandwatch, Nielsen, Radian6, Scoutlabs and Sysomos. The products were tested on how well they assessed comments made about the coffee chain Starbucks, with the comments also having been manually coded.
On aggregate the results look good, said FreshMinds. Accuracy levels were between 60% and 80% when the automated tools were reporting whether a brand mention was either positive, negative or neutral.
“However, this masks what is really going on here,” writes Matt Rhodes, a director of sister company FreshNetworks, in a blog post. “In our test case on the Starbucks brand, approximately 80% of all comments we found were neutral in nature.
“For brands, the positive and negative conversations are of most importance and it is here that automated sentiment analysis really fails,” Rhodes said.
Excluding the neutral comments, FreshMinds manually coded conversations that the tools judged to be either positive or negative in tone. “We were shocked that, without ‘training the tools’, they could be so wrong,” said the firm. “While positive sentiment was more consistently categorised than negative, not one tool achieved the 60-80% accuracy we saw at the aggregate level.
“To get real value from any social media monitoring tool, ongoing human refinement and interpretation is essential,” said the company.
The full whitepaper can be download online here. Get the lowdown on social media monitoring here.