{"id":14064,"date":"2018-02-07T09:15:18","date_gmt":"2018-02-07T16:15:18","guid":{"rendered":"https:\/\/www.fullcontact.com\/?p=14064"},"modified":"2022-04-21T02:02:21","modified_gmt":"2022-04-21T08:02:21","slug":"experimentation-leads-innovation-machine-learning-fullcontact","status":"publish","type":"post","link":"https:\/\/www.fullcontact.com\/blog\/customer-experience\/experimentation-leads-innovation-machine-learning-fullcontact\/","title":{"rendered":"When Experimentation Leads to Innovation: Machine Learning at FullContact"},"content":{"rendered":"

At FullContact<\/a>, we\u2019re always experimenting with new technologies and techniques. Machine learning has come into vogue as of late, and has shown some impressive results within our company and without. Recently, we had an opportunity to apply some machine learning to improve our handling of job title data we find throughout the web. The choice, however, was not an easy one. To someone unfamiliar with the technology, machine learning can seem complex, foreign, expensive, and hard. Training data, precision and recall, neural networks, scary-sounding academic papers — it\u2019s overwhelming. Given such a large investment, is it really the right choice? In this post, I\u2019ll try to explain our thought process at FullContact on solving a tough problem, and how we decided to use machine learning.<\/span><\/p>\n

The Problem<\/h3>\n

FullContact handles job title data in a variety of forms. Understanding job titles at a deep level is critical for contact search, analytics, and formatting for use in our APIs<\/a> and consumer products. One problem that we had historically not solved was department classification. Given a job title, we needed to classify it into a corporate department. <\/span><\/p>\n

For instance, a job title input of \u201csoftware engineer\u201d should return a department classification of \u201cengineering.\u201d \u201cController\u201d should return the \u201cfinance\u201d department. \u201cCustomer Success Manager\u201d is \u201cCustomer Support. <\/span><\/p>\n

This solution may seem simple, but it quickly becomes complicated after further thought. For instance, what rules could we use to classify a job title into the \u201cengineering\u201d department? Maybe if the job title contains the word \u201cengineer\u201d? This works for \u201csoftware engineer\u201d but fails for \u201cprogrammer,\u201d which should also be an engineering job. This approach classifies \u201csupport engineer\u201d as \u201cengineering,” when at many companies that position is a customer support job. The story is similar for many other professions, and simple rules like \u201ccontains the word engineer\u201d quickly become a quagmire of logic and exceptions.<\/span><\/p>\n

This was the state our department classification methodology. It worked for common job titles, but it was painful to maintain and had undefined behavior for job titles outside the norm. Addressing bugs in this system was usually \u201cadd another special case,\u201d because fixing the underlying logic was either not useful or too complex.<\/span><\/p>\n

The Challenge<\/h3>\n

After talking about the problem, we decided that it was worth trying a prototype using machine learning. This was not a decision made lightly. We were well aware of the drawbacks of machine learning:<\/span><\/p>\n