open-source vs closed-source – how to make a choice.

Nothing is likely to raise the blood pressure in a collection of search experts quicker than a lively debate over the virtues of open versus closed source search. Here at Dahu Towers, we of course have our own views and thought a gentle discussion  might be a nice jolly way of starting off our new blog. Our view, without wishing to be too trite, is that the decision to use open source search depends largely on whether you have an open source problem or not. The traditional view is that open source is for those wishing to avoid a licence fee and that are happy to roll up their sleeves and get down-and-dirty with the code. As with all things in life, its really not that simple. Firstly, consider what open source and indeed closed-source looks like at the moment. Open source search is good. Very good. Great in fact. There, i’ve said it. Its feature-rich, very scaleable and very reliable. We at Dahu towers are using open-source for our own product development. We also consult on closed-source search however.

The reality is that the boundaries of open-source solutions (Solr, Lucene, ELasticSearch et al) are much more sparse than the core of a typical closed-source solution. Open source search tends to solve the technical core issues of search but doesn’t really provide all the additional features and facilities that the closed source vendors tend to provide. It would be interesting to look at a typical closed-source vendor’s engineering department and see how much of their time is spent engineering in the kernel or core of their product and how much time is spent in the periphery.

So back to the question – “do you have an open source problem?”. The traditional answer (from the vendors at least) has always been that open source is for solving a) simple problems and b) for tech. savvy organisations. ‘a’ is simply not the case – some of the largest and most complex solutions are based on open source, and for ‘b’, this is somewhat of a myth. It is true however that for open source you will be gathering a number of components together and for anything but the simplest requirement, you will be doing (or having done for you) some customisations. There is now quite an eco system out there of tools for content gathering, processing, search User Interface creation, taxonomy and ontology management (include our own deep-content offerings). So a primary question needs to be “is what you are doing a specialisation that exists in the closed-source world or are you breaking new ground?“. Its no surprise that most startups that are using search in innovative ways will opt for open-source. If you are going to have to take the back off and start tinkering about on something truly novel, then often, closed-source will not offer you any particular advantage.

If however your problem is a specialisation that is already catered for in the closed-source world than it is worth looking at the vendors and doing a thorough investigation of the costs (and we mean all the costs – the total cost of ownership not just the licence cost or lack of it). The vendors of closed-source solutions all have their specialisations. Some are particularly strong in true enterprise search, offering business-level management tools, connectors for all the content sources you are likely to need, and easily customisable User Interfaces. Some are particularly strong in E-commerce, providing campaign management tools, support for managing catalogues with real-time price updates. Some specialise in specific domains like Legal and Compliance and provide tools and methods aimed specifically at these verticals. The way that open-source software is developed tends towards generic search facilities. Any thing that was specific to a particular solution or domain is unlikely to make it into the main trunk of an open-source search product unless it can be abstracted and made available as a feature of universal application. This keeps the core of the open-source engines powerful, but devoid of any vertical or domain bias. It is possible that in the future we might start to see flavours of search aimed at,say, the compliance market, although right now, we doubt it.

The closed-source vendors identify their specific markets and pursue them, often with almost religious zeal, without having the same sort of agnostic constraints. This is often seen in the way the specific content is connected to, filtered and processed and presented, often out-of-the-box. Innovation on the other hand can be hindered. As we mentioned, many startups with a search bias will chose open-source precisely because of the low-level flexibility.

Another important consideration is the rate of change in your application. For example, if you are a large enterprise looking to deploy a number of search-based solutions at a global and departmental level, then a closed-source vendor with a soup-to-nuts solution will be cheaper in the long run. If you are looking to build something relatively static for a point-solution, then open-source will be cheaper. Large-scale enterprises tend to standardise on solutions to keep costs down. Single technology platforms require one set of skills and with a full-featured solution, and while this may enforce some (potentially unpopular) constraints, it ultimately lowers support costs. In controlled environments, continual development using a wide range of tools and modules can be seen as anarchy – it can certainly cause difficulty in control, and introduces potential duplication of effort and skill-sets.

In conclusion, we feel that open-source is not just for the tech-savy, the frugal and those with simple problems. Closed source is not just for those who are technology adverse and those with deep pockets. We expect to see more inovation based on open source, and we expect to see a lot more specialisation in the closed-source community.