A huge percentage of the data generated today is unstructured. The overall Big Data and Hadoop wave happened because it had the ability to handle unstructured data. However, most of the Hadoop or Big Data systems today handle at most semi-structured data – JSON’s, Avro, Parquet, etc. When it comes to making sense of unstructured data with no proper structure, the amount of effort required is significant.
Text analytics comes to the rescue when we talk about making sense of a largely unstructured corpus of data. And NLP plays an influential role in text analytics. Advancements in the area of NLP are further helping mine this huge data to extract insights and help get a comprehensive view. While data analytics of structured content helps understand who, what, when, and where; the content analytics distinctively adds to the why and the how.
NLP is a field of Artificial Intelligence wherein the computer can understand, analyze, manipulate, and potentially generate human language. Therefore NLG, NLU, etc. are all part of NLP. Developing NLP applications can be quite challenging. Computers traditionally need to be communicated with using unambiguous and highly structured commands. Human speech is exactly the opposite. It is often ambiguous, and the linguistic structure can depend on many complex variables, including slang, social influences, and dialects.
NLP is widely used for mainstream applications such as,
- Information Retrieval – Like Google, but more intelligent and even tuned to a context or a domain. This could also take the form of a Q&A for the end-user.
- Information extraction – This could be known as entity extraction, like retrieval of origin location, GST codes, invoice number, passing entity, etc. from invoice notes or even extraction of your flight information from confirmation emails sent by airlines.
- Machine Translations – Conversion from one language to another.
- Sentiment Analytics – Can be used to arrive at insights on how a new product is being received by the users and the overall product roadmap, employee satisfaction, etc.
- And many other applications like text summarization, speech recognition, and more.
2018, I believe, has been the busiest year for advancement in NLP research. With so much of activity happening around the BERT model and now the next-gen OpenAI GTP-2 model, it can get difficult to know what this means from a practical business perspective.
Can this research be taken as-is and applied to daily mainstream applications? With the underlying technology evolving so rapidly, does it make sense to invest time developing an approach which might become absolute with the next breakthrough in research or the next research paper?
Most cloud players now also have API-driven services that can be quickly integrated with business applications. Amazon’s Comprehend, Azure’s cognitive services like Speech and Language, Google’s Natural Language API are all services that can be very easily leveraged to develop AI-enabled applications.
As we help our customers make sense of the vast amounts of unstructured data that they directly or indirectly collect, we work across the applied NLP areas mentioned above. Each work packet that we have solved is unique in the sense of the business problem it is trying to solve, the NLP techniques that are involved, and the approach that is used. What is common amongst these is that all of them show the tremendous power that NLP can bring when used in the right way. It helps businesses overcome potential barriers by helping unearth missing insights.
In this blog, I have attempted describing these unique problems and the approaches used to solve them. I have tried to give you a good idea of the trends in this space, which can help you make an informed decision about applying the latest solutions/frameworks/methodologies to your business problems. The work being done in the NLP space is very exciting, and I hope it excites you just as much to begin this journey and take a dip in the vast ocean of opportunities that NLP and text analytics presents.
And if you are looking for a partner to work with on any similar problems, feel free to reach out to us!
