Searching words of wisdom

I’m currently working on a project that includes a lot of text as part of the data.

Traditionally computer systems and databases don’t like free text; they prefer values that can be added, evaluated and easily stored. Of course there is much to be said for that; assigning a value enables us to clarify our thoughts and opinions. But we can’t communicate everything with numbers and capturing free text provides a great deal of rich understanding. It also enables us to store information which may not be considered relevant right now, but could be valuable in the future or to other people.

Happily, searching text has become a great deal more sophisticated than when I first started working with business systems. Databases such as Microsoft’s SQL Server now have the ability to search for meaning as well as words, to rank search results in terms of likely relevance, and to filter out the “joining words” that don’t add anything to the search process.

When we search for things on the internet we use the same functionality. We expect to key in a search term and find relevant information from a variety of sources. We rarely stop to consider the software technology that is behind making it happen, even though it is hugely powerful and is the piece of the jigsaw that makes all the internet information useful.

Our electronic world has made it so quick and simple to send an email to anywhere in the world, so a huge amount of information is being conveyed in free text. Documents can easily be created and stored on the billions of PCs in businesses everywhere - the issue now is to make the information accessible to the people who need it.

How well do you know your customers?

I guess the answer to that question will depend on what type of business you are in. If, like me, you are in a service business, the answer might be “pretty well, thank you!”

But if the question were phrased slightly differently: “What characteristics do you look for in new customers?” that might not be so easy to answer. Attracting new customers isn’t always straightforward. And all our customers are different, aren’t they? Well yes, and no.

Yes, in that we are all unique. No, in that we share characteristics that make us more or less likely to purchase from any given business. If we can understand those common characteristics then we are better placed to attract and keep profitable customers.

Most companies have a lot of data about their customers, but very little insight into what these common characteristics are. Yet with a bit of analysis these shared characteristics can be uncovered. It’s called data mining, data analysis, analytics, or segmentation depending on who you talk to. But it’s a potentially profitable part of any new business campaign.

Database guys only want sets

As I queued for breakfast last Friday, the T-shirt in front me read “Database guys only want sets.”

Databases, it seems, are a bloke thing. This is news to me having been fascinated by databases since before SQL Server 6.5. I idly wondered what a gal’s database T-shirt might read as I looked around the room. A sea of male faces stared back at me - all of them looking as if sets were the furthest thing from their minds.

I was at SQLBits Goes West – the latest SQLBits conference for those, like me, who work with Microsoft SQL Server in its various guises.

The conference was being held in Newport, Wales, at the iconic Celtic Manor Resort. It’s the first thing you see as you enter Wales from the M4. It nestles in the steep hillside, towering above the motorway below. I’ve driven past it many times, but this was my first visit inside. It is grand and spacious and its imposing front hall vaguely reminded me of The Shining. Jack Nicholson, fortunately, was nowhere in sight.

SQLBits is a terrific idea. Organised by a bunch of enthusiastic SQL Server Microsoft Valued Professionals, it is not-for-profit and all-for-education. And bacon butties.

Last Friday’s excellent presentations were centred on the latest Beta releases from Microsoft: Office 2010 and SQL Server 2008 R2. There are some deeply exciting new features which open up Business Intelligence to many more users. This has to be a good thing. It remains to be seen whether Microsoft’s strategy of BI for the masses pays off but from where I'm sitting it's looking good.

Data mining - digging for gold

Just as coal is the work-horse of modern energy production, so the relational database is the work-horse of modern business. Alright, one is black and dusty and the other is, well, virtual and clean, but the end result is the same - reserves waiting to be mined, whether the reserves are coal or data.

Reserves which may contain gold for their owners.

Before you put on your hard hat with the lamp on the front to break open the server, I’m talking metaphorical gold - metaphorical gold which could be worth a great deal more to your business than the real thing.

First – let’s consider the reserves, which unless you are actually a mining company will be the data stored in databases within your company. Then, let’s look at what the gold might be that’s hidden in the data.

There are few businesses which have not installed a database, whether for managing customers, accounts or stock. As more enterprise-wide systems are installed the amount of data being generated is phenomenal. Some of that data will be immediately accessible through reporting tools. But what could happen if those databases were joined together? What if you could see the sales information together with the customer management information? Or the training data together with sales data? At the risk of mixing enough metaphors to make soup, that would really be cooking with gas ….

But whether it’s one database, or a number joined together, how do you go about looking for gold? Indeed what does gold look like in data terms?

How you find gold is by using a technique called data mining, and what it looks like all depends on your business. It may be customers who are more likely to book a particular type of show in your theatre, or finding which products to bundle together to maximise sales and profit. Or it could be something completely different – depending on what business you are in. The applications for data mining are many and varied and are limited only by business owners' imagination and ambitions.

Data mining is now more accessible and affordable than ever. Products such as Microsoft SQL Server 2008 and 2005 put data mining within the reach of most companies – large or small.

Get in touch if you want to dig for gold in your data. Hard hats with lamps supplied.