Where is your data hiding?

When attempting to generate value from data, many organisations intuitively  turn to areas of their business that produce lots of it. However, these data rich areas are not necessarily where one should look for valuable insights.

A good rule of thumb to follow when attempting to build an analytical capability within an organisation is to constantly ask oneself: “What is the purpose of establishing this particular process?”. If the answer to the questions happens to be “to analyse data” or “to deliver value, but delivery mechanism is not defined”, then it is likely that your efforts fell prey to a common mistake. Data analysis capabilities do not bring value! A detailed business plan, with well-defined purpose, and data analysis, which is integrated into the decision-making processes of the organisation, is what creates value. Blindly analysing data without a purpose, in hopes of finding much-needed insight is a waste of resources. And this brings up a second question: “What data should be analysed?”.

All businesses collect data and large amounts of it. However, it is a logical fallacy to think that the area with the most data will produce the most insight. Additionally, integrating external data into business operations may prove to be a significant return on investment. Which data pool we select depends entirely on what purpose we want to achieve. In some cases, acquiring professional strategic advice is the key to success in data analytics efforts.

~Alexey Mitko

Categories: Uncategorized

Multi-professional approach to Data Analytics

March 21, 2013 Leave a comment

Data analytics, Enterprise Intelligence, Continuous Assurance, Regression Analysis, Data Life Cycle are terms that you may hear when discussing potential approaches to addressing the Big Data Challenge. Unfortunately, the term “Big Data Challenge” is a misleading one, for it implies that there is only a single problem that needs solution, while in fact there is a number of unique circumstances that companies face, each requiring its own tailored approach. In this post I will aim to highlight the main areas of concern for Big Data specialists and some of the tools that have been developed to address these problems.

Before we begin, it is important to understand that a number of professions aim to fill the need for data analytics capability in business. Accountants, Actuaries, Internal Auditors, External Auditors, Statisticians, Mathematicians, Data warehouse specialists, Programmers, Risk Managers and Consultants, all of these professionals feel the need to contribute to the discussion. As you can imagine there is a great variety of problems faced and each profession has developed its own set of tools to cope with these challenges. Many of the professions struggle to adapt, in many cases statistical analysis has become more prominent, with Statisticians and Actuaries taking a lead and fewer professionals in the accounting field or consulting having the necessary skills. In other cases, professions come into conflict , with some professionals feeling that their domain is being taken over. As such, there is no single way to distinguish underlying domains of the Big Data Challenge, but I will try to do my best to reconcile various views.

What is the Big Data Challenge?

Most commonly, Big Data is described as a an explosion in the amount or frequency of data generated by modern enterprises. However, this is not a useful definition for it describes only the fact of occurrence and not repercussions of such a change.

I would postulate that this data explosion affects us in the following ways:

1. It is harder to find relevant information now, than when data was less abundant, because we need to dedicate more resources to searching.

2. It is harder to ensure consistency and compatibility of records, than when data was less abundant, because there are more ways in which data is collected.

3. It is harder to detect meaningful patterns within the data, than when data was less abundant, because the volume and speed of transactions require additional processing capabilities.

What solutions are out there?

As you can imagine, each organisation has its unique challenges, each challenge has several solutions, depending on the type of data, urgency, market conditions, and even people involved. As such, it is very difficult to create discrete rules that would classify each type of problem and advise a particular solution. This framework is aimed to be a rough guide, rather than a prescription.

1. Getting data warehouses in order and enabling easier access

Believe it or not, but data storage, data accuracy and ease of data access have been a topic of discussion in the computer science profession for decades. Database structure has had a considerable evolutionary history over the past 50 years. In short, databases became quicker, more tolerant to errors and more flexible. Unfortunately, not all organisations have cutting edge databases. A great variety of legacy systems and improper ways of using existing systems introduce number of errors into the datasets, errors that need to be remedied if further analysis is to take place. The explosion in data volumes exacerbated the situation by placing additional volume strain, as well as accuracy and operational requirements (as, for example, is the case for distributed databases). A number of new and established firms responded in a variety of ways to this challenge, either by developing new database technologies or by dedicating more processing and accuracy verification resources. This area has traditionally been addressed by IT professional.

Further reading on this topic can be found here.

2. More advanced and specialised search engines

In a way mini Big Data problems have been around for centuries. When the first printing press was invented, an explosion in print media warranted creation of libraries and subsequent catalog systems. Similar experience gave birth to phonebooks. And Google, in its brilliance, brought order to the informational chaos of the early Internet. Since then several new technologies emerged in order to tackle the challenge of finding the correct piece of information within a cluster of related data. Examples of companies involved in this field include IBM (and its famous Watson Computer), Sinequa (with its unified information access tools), and Recommind (with automatic categorisation tools), just to name a few. Each approach uses different underlying technologies, and if your Big Data problem falls into Search Engine category, you need to do additional research to understand which technology would work best in your circumstances.

3. Pattern recognition and detection – new and old data analysis techniques

Another domain of Big Data is the need (or an opportunity?) to detect patterns within the data with a view of making forward-looking predictions or to detect anomalies. A range of situations where this capability might be useful is virtually limitless and varies from pricing, customer management and production planning to fraud detection and equipment monitoring. However, methods that address this issue fall into three main categories.

First method is data visualisation. This method is very intuitive and appealing, since we can perceive visual information very rapidly. Majority of data visualisation techniques focus around enabling rapid prototyping of visual models, some examples can be found here. These techniques allow to pinpoint outliers and trends, but rely heavily on personal interpretation. Additionally, not all phenomena can be expressed in a visual way, with some patterns taking form of multi-dimensional multi-order relationships. Furthermore, a great deal of training and experience is needed for these visual models to be used correctly. Human brain excels at finding visual patterns, however in some cases it is susceptible to finding false positives, Astrology being one example.

Second method is mathematical modelling. This approach leverages a number of well-known statistical techniques starting from various types of regression and drawing heavily on differential equations. This approach has proven to be effective in a number of applications, such as its integration with ERP systems. However, it is very expensive and complex to implement. The level of mathematical expertise required and specialised nature of the models often restrict application of this method to a high value and high impact projects. Furthermore, most models of this type have limited dynamic flexibility, and if underlying relationships change the model becomes obsolete. As such this method is most appropriate for specialised application in relatively stable environments.

Third method is automated software modelling or sometimes called artificial intelligence modelling. Instead of hiring a team of mathematicians to build a model, several companies are developing software packages that are themselves capable of  choosing what factors are most important in modelling a particular environment. Most notable example of a company engaged in this area is Numenta. While this approach can be orders of magnitude cheaper, compared to traditional statistical approaches, its usefulness rests with high velocity temporal data applications, such as modelling electricity usage, credit card transactions or monitoring equipment status. This software is also capable to dynamically adapt to underlying changes in relationships within data.

Final words

As can be seen from the above list of solutions, the Big Data Challenge is a fragmented problem. Each particular situation demands careful problem classification and selecting appropriate tools to address it. I believe that these tools fall into the three categories described above and that each category is experiencing rapid evolution. The challenge facing many businesses today is navigating through this complex environment, and hopefully this article helps them to do so.

~Alexey Mitko

The Land of Hairless Carpets

March 19, 2013 Leave a comment

During my Bachelor studies my economics professor shared an interesting story, the lessons of which I’m just beginning to grasp. Back in early 1900’s, when vacuum cleaners were a recent invention, many vacuum cleaner producers were competing on the suction power of their devices. In the beginning, competition on the power dimension made perfect sense, after all, the higher suction power provided for better cleaning. Over time and as technology progressed vacuum cleaners grew more powerful and eventually became capable of tearing hairs out of the carpets they cleaned. Unfortunately, consumers didn’t know at what point vacuum cleaners become carpet barbers, thus Consumer Protection Agency had to step in and restrict how vacuum cleaners ought to be marketed.

What is the moral of the story? Initially important, but subsequently outdated competitive dimension may actually siphon your resources, which in turn could have been used for true research and innovation. The mistake of competing on irrelevant factor is often made when company loses the sight of its purpose. If in the example above the purpose of the company was to provide an easy and efficient cleaning tool, then vacuum’s power is an important factor, but to a point. As history has it, eventually dust bags and cyclone vacuums were created, and overall weight of vacuum cleaners was reduced as well. Product innovation cycles through competitive factors, companies that fail to recognise that end up in the land of hairless carpets.

These principles are not limited to vacuum cleaners! Similar cycles can be observed in the mobile phone industry. Every time a new smartphone comes out its hardware is carefully examined. Over the years, cpu power and RAM capacity were legitimate competitive dimensions. If your engineers were able to produce faster, lighter phones, without increasing power consumption, then the resulting improvements contributed directly to customer experience. The end product was more fluid and could boast better graphics experience. But these competitive dimensions have diminishing returns. What if human eye cannot tell the difference between a super definition display and ultra definition one? What if all smartphones on the market are capable of super smooth performance? After all, once response times become minuscule,  even orders of magnitude improvements become had to notice. It is quite possible that current smartphone race is reaching its hardware limits and companies that are not careful may miss the next competitive dimension.


Categories: Alex, Authors, Strategy

Ethical battle lines of Marketing

February 22, 2013 Leave a comment

Marketing has been a core component of business since probably a second after business was invented. At its core Marketing aims to deliver a message to a specific group of people and that message often tries to persuade those individuals to consider purchasing the product or service in question. Competition in the marketplace makes Marketing a vital business function. After all, even if your company makes the best product, but nobody knows about it, chances are that your competitors will be able to reap greater rewards. Under competitive pressure it is often essential for marketers to promote the product to its potential, but they cannot cross the line into misleading. In the US misleading advertisements are often termed “false advertisement” or “deceptive advertisement” and are regulated by the Federal  Trade Commission. Other countries have similar laws and regulatory bodies, some initiatives that aim to protect consumers are www.isitfair.eu in EU and Australian Competition & Consumer Commission in Australia.

It is clear that you can advertise in a deceptive way and that there are regulatory and consumer bodies set up to protect the world from such practices. But what exactly is deceptive? Over the course of history and with the help of the legislative system certain marketing practices became accepted as deceptive. For example, marketing one product and substituting it for something else, creating a pyramid scheme or forging trust marks. These examples are numerous, but they came about and became accepted as deceptive because injured parties sought compensation in court in the years past. But what if the matter is so minor and legal process so expensive that nobody bothers to seek compensation? Or what if deception cannot be easily proven? I would speculate that there remains a subset of marketing practices that could be called dubious, while regulatory bodies are too busy policing more serious cases.

Just how prevalent these practices are? How used are we to them? I will include several examples below, but feel free to add additional illustrations into the comments.

1. That food looks so good on TV, but not so well in real life. Actually there are companies that specialise in replicating popular menu items in plastic. One such company is www.fake-foods.com. Not all replicated foods are used in tv commercials, some are used as restaurant displays and children toys, but some  do end up being stars of 30-second movies. Using plastic props in commercials makes perfect sense, since they don’t spoil or wither with time and able to tolerate high-powered  stage lights without melting. But is that deceptive? After all, the food I buy at the store is not plastic and would look quite different on TV.

2. People that look good on TV and look well in real life, but never used the product advertised to achieve their results. I’ll let the actual ad prove my point. Is that deceptive? Those people are in great shape, but they probably achieved such physique either genetically or by going to the gym, an exact opposite of what ad claims it can do for you. Could it be that it’s just natural to get models for your commercial? And since everyone does it is would be against industry practice try to do otherwise. Besides consumers are aware enough to understand distinction between marketing and reality. To address these arguments, I would like you to have a look at this ad. Did those people used the actual product to achieve their results? Do you think they had hairdressers on the set when filming the commercial? Another example can be seen here.

3. Perfume is all about the smell right? Perfume ads are an interesting example. On one hand what is sold is a fragrance, a physical product, that has nothing to do with the model on the ad or the shape of the bottle. In fact, very few people would be able to connect the ad and the smell of the product. So is it deception to use pretty models to sell your product? Should you not include testers in every magazine ad? Not quite, while how perfume is advertised has little or nothing to do with the actual smell, something else is sold along with it. That something is value created by the add itself solely in the mind of the consumer. By looking at the model, the elegance of the bottle some consumers derive satisfaction because they are able to imagine themselves as belonging to that lifestyle image. The feeling that consumers buys along with the perfume is purely subjective, created only in his/her mind, but it is real and paid for.

Danger with some of these practices is that we become used to them and as the result transpose marketing reality into our actual lives. Any other examples?


Quick communication exercise

February 5, 2013 Leave a comment

This is a small exercise that I picked up, illustrating how easily we are susceptible to misunderstandings, even in situations where no misunderstandings are to be expected.

“A lake exercise”

  1. Imagine a lake, let it be classical and generic.
  2. Now all of your team members should do the same.
  3. Let each team member describe his or her lake. Let them elaborate on its size, scenery, vessels or colour.
  4. As they do so, it becomes quite apparent that our descriptions of lakes are rather different.

The important question to ask is: “How can we trust no misunderstandings to occur if such a simple term as a lake creates such a variation in description. We often assume that common terms are universally understood. However, words such as simple, convenient, good or bad are subject to exactly the same interpretation. In a way no single person speaks the same language, simply because meanings attached to the words are based on the unique experiences of that particular individual.

Thus it is important to ensure that extra care is taken in situations where parties come from different backgrounds. Senior and junior, consultant and client, ambitious and timid, all of these personal differences create different lakes.

~ Alexey Mitko

Categories: Uncategorized

How long is your attention span?

January 21, 2013 Leave a comment

Human brain has an interesting tendency, it strives to create explanations (or better word would be models) for the environment it observes. Evolutionary speaking, this trait is highly advantageous, it allows humans to adapt and predict their environment, thus increasing chances of survival. But it also creates false positives, builds models and finds order where none exist, as for example in creating constellations out of a random spray of stars in the night sky. While the night sky is an example of how humans find order in dimensional world, we should also note that false positives may exist in the temporal continuum.

Consider, for example, a recent graduate, who becomes a young aspiring commodities trader circa 2000. It would not take long for her to learn that her peers who advise their clients to buy gold are getting substantially bigger paychecks than she does. So she asks her coworkers for their strategies, their models and explanations for how gold behaves. Wether or not those explanations are correct does not actually matter, as long as they predict higher gold price and do not substantially deviate from industry thinking, none of them can be proven wrong. Fast forward to 2012, our graduate is now a 32-year-old successful senior manager, who built her career in gold trading, and received hefty bonuses for the past 10 years. The reason for her success is her unyielding faith in the value of the yellow metal, her peers who doubted that the gold rise in the past are working for her now, since they were not promoted as quickly.

What should interest us in this discussion is the gold pricing model that this manager developed in her head over the years. It can be as simple as a single sentence, or a complex mathematical model, but the key question we must ask is whether or not it is biased. Is it biased? After all this manager was rewarded for a very particular behaviour for the past 10 years, her peers were punished for predicting lower gold prices, her whole career was based on 10 years of rising gold prices, the echelon of her analysts, senior and junior, have similar behaviour cultivated in them since the first day they join the firm. Are they biased? Would they not try to look optimistically on any kind of evidence presented to them, not try to find an explanation for why the gold should rise again after a dip, just as it did before? Furthermore, this slightly skewed view propagates through the ranks with each one adding an additional twist of optimism until reality becomes grossly distorted. How can a person, in a position of power, be objective when all he/she was presented with during her career is a single side of the coin? Can human attention span be long enough to incorporate macroeconomic forces, which take decades to play out, into mental models that humans so eagerly construct?

For most of us gold trading is of little relevance, but principles at play apply to situations that rest much closer to home. Consider for example a banker, who is measured on his yearly performance, while the loans that he makes are for periods longer than 10 years. Would he care if the loan is defaulted on 7 years from now? Probably not, the loan book could have been sold on or he could have changed jobs several times since then. You can realize how this sort of behaviour could quickly become a reckless one. If banking is not personal enough for you, consider a manager who is responsible for plant maintenance. He is presented with a choice, to save some money now or spend it to upgrade plant’s equipment, if he saves it now then he is rewarded, albeit at the risk that the machine will have a higher chance of a breakdown (but when? 1,2 or maybe 3 years from now?). How long is his attention span?

In conclusion I would like to share several quotes that drive home some of the principles discussed.

Abraham Maslow “I suppose it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail.”

Ralph Linton “The last thing a fish would ever notice would be water.”

Daniel Day-Lewis  “Perhaps I’m particularly serious, because I’m not unaware of the potential absurdity of what I’m doing.”

Several movies: Anatomy of a DisasterHow the banks never lose

And several interesting articles: Morgan Stanley, Warning to Banks



gold price charts provided by goldprice.org

Lean Communication

December 3, 2012 Leave a comment

I recently discussed the topic of new forms of media in the context of communication with a former colleague of mine. E-Mail, short messages, instant messages and even Facebook-messages infiltrated the workplace within the last two decades. Undoubtedly, those forms of communications bring a lot of advantages to our daily business and make the transfer of data not only easier, but also a lot faster.

But, have you ever realized that there is a downside to quick and easy communication channels? Most of you would probably agree that the number of letters we receive on a daily basis shrunk extremely with the rise of E-Mail. However, that shrinkage is small compared to the increase in electronic messages received. Because it is so easy to write and send an E-Mail the use of it is inflationary. Especially if someone is available via Smartphone we tend to write a short mail or message. Even though, it is believed to be the fastest and easiest way, is it really? Sending several messages back and forth can be very time-consuming and annoying. On top of that important E-Mails are always in danger of drowning in the flood of “not as important” (if you’d like to call it that way) E-Mail that flush through Inboxes every day. Therefore it might sometimes be better just to do the oldschool-way and grab the phone to discuss a topic rather than doing it electronically. And if we see this way: Someone who’s always online with his smartphone to receive E-Mails is also always available via phone.

In times of Lean-Management, Lean-Manufacturing,… everything’s aim is to be lean, so why don’t we try to use Lean-Communication?