The whole world relies on search to get access to relevant information and data. Handling this large amount of data presents a huge challenge. To get relevant data, we hit the database which is an extremely slow process. Moreover, there is a need to separately perform data encryption. Elasticsearch is a great technology which offers fast search results.
Elasticsearch is a Lucene-based search server. It performs data indexing in an encrypted format, making search more secure and fast. Internally it stores data in JSON format. The Indexed data can be queried using APIs in (Java + Curl) language. Elasticsearch provides an interface to browse through the clusters, indexes and data which can be accessed using the URL http://localhost:9200/_plugin/head. Every record inserted in Elasticsearch is called a Document which is associated with a unique identifier called Document Id.
In one of the applications I worked on, I mapped the database identity of corresponding record to the document Id. This helped bridge the gap between Elasticsearch data and data stored in RDBMS at a few functional points. If you don’t map the Document Id, Elasticsearch associates the document with a self-generated unique identifier.
Pre-requisites
Elasticsearch Setup: Unzip the downloaded setup, navigate to bin folder and start the server by running elasticsearch.bat file.
We can use the following dependency for integration in Java application:
Maven dependency:
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-webmvc</artifactId>
<version>3.1.1.RELEASE</version>
</dependency>
Configuration code
Internally, elasticsearch performs data sync for all available connections running in the same network. To avoid this auto sync process, we can configure different cluster name in the elasticsearch.yml file as mentioned below:
cluster.name: cards
port.number:9300
Cluster is created when we start the Elasticsearch server. The inserted data is divided in various shards for effective search; the ClusterStatus indicates the state of shards. ClusterStatus green indicates that data has been indexed correctly and provides efficient search.
Properties configured using properties file in our Java application named as
searchserver.properties are as follows:
search.index.path= /opt/elasticsearch-1.3.0/data
(Note: Default indexed data is located in the data folder. With the help of elasticsearch.yml file or property file we can configure the location as well)
search.cluster.name= cards
socket.transport.address=localhost
search.port.number=9300
search.index.name=card_index
Source Code
Elasticsearch provides various APIs to perform operations like data insertion, data update and search, among others.
In the application mentioned above, I had inserted specific data into Elastic Search. For that I created a Java entity holding fields that were searchable in application. The Java entity was converted to its JSON representation using GSON and then inserted in Elasticsearch.
The entity definition is as below:
import java.util.Collection;
public class shopperCard{
private long id;
private String userName;
private boolean whetherActive;
private Long cardNumber;
private Collection<Products> products;
public long getId() {
return id;
}
public void setId(long id) {
this.id = id;
}
public String getUserName() {
return userName;
}
public void setUserName(String userName) {
this.userName = userName;
}
public boolean isWhetherActive() {
return whetherActive;
}
public void setWhetherActive(boolean whetherActive) {
this.whetherActive = whetherActive;
}
public Long getCardNumber() {
return cardNumber;
}
public void setCardNumber(Long cardNumber) {
this.cardNumber = cardNumber;
}
public Collection<Products> getProducts() {
return products;
}
public void setProducts(Collection<Products> products) {
this.products = products;
}
}
While performing search on a particular field, you need to specify the field name. The field name is the same as the name specified in the ShopperCard class. In the application, I also had to enable search for a field like product name which was in the Products class. Products were not mandatory when I inserted ShopperCard. So it was mapped “null” at times, but the name was searchable as per our search algorithm product name.
While performing field search, Elasticsearch searches for the field name in the document .If field name does not exist it results in an error .To avoid this, I mapped an empty Product object [handled in setter method].
E.g.: Product[
name= null;
Id=null;
]
JSON Representation:
{
id: 1
userName: John
cardNumber: 1234567890
whetherActive: true
products: [
{
name: Vegetables
id: 2
}
]
}
{
id: 2
userName: Williams
cardNumber: 1234567567
whetherActive: false
products: [
{
name: cosmetics
id: 2
}
]
}
Java Code
DataSearchResource.java
Public class DataSearchResource{
@Autowired
public ElasticSearchService elasticSearchService;
/**
* Method to add Shopper Card
*
* @param shopperCard - {@link ShopperCard}
*/
public void addShopperCard(ShopperCard shopperCard) {
Gson gson = new GsonBuilder().serializeNulls().setPrettyPrinting().create();
String card = gson.toJson(shopperCard);
Long cardId = shopperCard.getId();
elasticSearchService.add(cardId, card);
}
/**
* Method to Update ShopperCard record
*
* @param shopperCard
* - {@link ShopperCard}
*/
public void updateShopperCard(ShopperCard shopperCard){
Gson gson = new GsonBuilder().serializeNulls().setPrettyPrinting().create();
String card = gson.toJson(shopperCard);
Long cardId = shopperCard.getId();
elasticSearchService. update(cardId, card);
}
/**
* Delete shopperCard by id
* @param cardId
*/
public void deleteShopperCard(Long cardId){
elasticSearchService. delete(cardId);
}
/**
* Method to fetch shopperCard details for entered search keyword.
*
* @param searchKeyword
* - search keyword
*/
public List<ShopperCard> fetchShopperCardDetails(String searchKeyword){
BoolSearchQuery searchQuery= elasticSearchService .buildSearchQuery(searchKeyword,”userName”);
SortBuilder sortBuilder = SortBuilders.fieldSort(“userName”).order(SortOrder.ASC).missing("_last");
List<String>searchResult=elasticSearchService.performFieldSearch(searchQuery,sortBuilder);
Gson gson= new Gson();
List<ShopperCard> cards;
for(String card: searchResult){
cards = Arrays. asList ( gson.fromJson(card,ShopperCard.class));
}
return cards;}
}
ElasticSearchServiceImlp.java
public class ElasticSearchServiceImpl implements ElasticSearchSerivce{
private final Settings settings = ImmutableSettings. settingsBuilder()
.put("cluster.name", I18NUtility.getMessage("search.cluster.name")).put("client.transport.sniff", true).build();
private final Client client = new TransportClient(settings).addTransportAddress(new InetSocketTransportAddress(I18NUtility.getMessage("socket.transport.address"), Integer.valueOf(I18NUtility.getMessage("search.port.number"))));
public Client getClient() {
return client;
}
/**
* Method to add document
*
* @param cardId
* - unique identifier
* @param shopperData
* - JSON of {@link ShopperCard}
*/
public void add(Long cardId, String shopperData) {
try {
if (cardId != null) {
getClient().prepareIndex(I18NUtility.getMessage("search.index.name"), I18NUtility.getMessage("search. index type"),cardId + "").setSource(shopperData).setRefresh(true).execute().actionGet();
}
} catch (Exception e) {
throw new NoNodeAvailableException("Error occurred while creating record");
}
}
/**
* Method to update document
*
* @param cardId
* - unique identifier
* @param updatedRecord
*/
public void update(Long cardId, String updatedRecord) {
try {
getClient().prepareUpdate(I18NUtility.getMessage("search.index.name"), I18NUtility.getMessage("search.index.type"),cardId + "").setDoc(updatedRecord).setRefresh(true).execute().actionGet();
}
catch (Exception e) {
throw new NoNodeAvailableException("Error occurred while updating record", e);
}
}
/**
* Method to delete shopper Card record
*
* @param cardId
* - unique identifier
*/
public void delete(Long cardId) {
try {
DeleteResponse response = getClient()
.prepareDelete(I18NUtility.getMessage("search.index.name"), I18NUtility.getMessage("search.index.type"),cardId + "").execute().actionGet();
}
catch (Exception e) {
throw new NoNodeAvailableException("Error occurred while updating record", e);
}
}
/**
* Method to perform search
*
* @param searchQuery
* - search query
* @param sortBuilder
* - sort builder to perform data sorting
* @return - list of matching records
* @throws Exception
*/
public List<String> performFieldSearch(QueryBuilder searchQuery, SortBuilder sortBuilder) throws Exception {
try {
CountResponse countresponse = client.prepareCount(I18NUtility.getMessage("search.index.name"))
.setQuery(QueryBuilders.matchAllQuery()).execute().actionGet();
int recordCount = (int) countresponse.getCount();
SearchResponse response = getClient().prepareSearch(I18NUtility.getMessage("search.index.name"))
.setTypes(I18NUtility.getMessage("search.index.type")).setSearchType(SearchType.QUERY_AND_FETCH).setQuery(searchQuery).addSort(sortBuilder). .setSize(recordCount) .execute().actionGet();
SearchHit[] searchHits = response.getHits().getHits();
List<String> searchResults = new ArrayList<String>();
for (SearchHit searchHit : searchHits) {
String searchResult = searchHit.getSourceAsString();
searchResults.add(searchResult);
}
return searchResults;
}
catch (Exception e) {
throw new NoNodeAvailableException("Error occurred while searching", e.getCause());
}
}
}
/**
* Builds search query based on search keyword and filed name
*
* @param searchKeyword
* - search keyword
* @param fieldName
* - field name on which search has to be performed
*/
public BoolQueryBuilder buildSearchQuery(String searchKeyword, String fieldName){
BoolQueryBuilder searchQuery = QueryBuilders.boolQuery ();
searchQuery.must (QueryBuilders.queryString(searchString).phraseSlop(searchString .length()).field(fieldName));
searchQuery.mustNot(QueryBuilders.termQuery("whetherActive", Boolean.FALSE));
return searchQuery;
}
}
Note: When search query is fired by default Elastic search returns a maximum of 10 records even if there are more similar records. We can configure this by setting the setSize parameter.
References:
http://www.elasticsearch.org/guide/en/elasticsearch/client/java-api