{"id":18427,"date":"2020-05-22T10:00:49","date_gmt":"2020-05-22T16:00:49","guid":{"rendered":"https:\/\/www.fullcontact.com\/?p=18427"},"modified":"2025-01-06T04:35:02","modified_gmt":"2025-01-06T11:35:02","slug":"resolve-building-the-identity-resolution-engine-part-3","status":"publish","type":"post","link":"https:\/\/www.fullcontact.com\/blog\/engineering\/resolve-building-the-identity-resolution-engine-part-3\/","title":{"rendered":"Resolve: Building the Identity Resolution Engine (Part 3)"},"content":{"rendered":"

The following is the third installment of a multi-part series offering a glimpse of the behind-the-scenes advancements we achieved to deliver our Resolve<\/a> API. The first post <\/a>covered the origin of our Identity Graph<\/a> and Resolve API, and in the second post<\/a>, we delved into the power of our new persistent PersonID and the ability to Bring Your Own ID.<\/p>\n

Part 3: Building Trust Through Security<\/h2>\n

To properly support our customers in this new era of privacy and security, we went back to the drawing board to invent new systems and patterns designed to maximize security around our customers\u2019 data. Some of those patterns were in play in the previous posts with the engineered design of the PersonID<\/a>.\u00a0<\/span><\/p>\n

Beyond IDs, we honed a centralized microservice to be the only system capable of encrypting and decrypting sensitive information. Being centralized eliminates the need to ever share the private keys with other microservices, thus limiting exposure of leaking the sensitive keys. \"\"<\/span><\/p>\n

Being a centralized, purpose-built micro service adheres to Service Oriented Architecture philosophy, but also creates new challenges to surmount. For us, given the amount of places encryption and decryption are used we had to engineer this system to scale. As part of our batch remapping process to re-resolve archived person fragments, we needed it to work inside Spark jobs which couldn\u2019t take days to run, metering out 1,000 requests a second.<\/span><\/p>\n

No, we needed this API to handle volumes of <\/span>1,000,000 requests\/second<\/b>. As part of our Resolve release, we pushed the limits of our APIs and parts of AWS to achieve this. We created S3 wrappers that worked natively inside Spark, much like s3a:\/\/<\/code>, and enhanced the decryption service to seamlessly autoscale up and down. The client and server were designed to block API calls and slowly increase the traffic until our decryption rate could be sustained at 1MM qps.<\/span><\/p>\n

Specific to the Resolve API, as our customers provide us with their own identity fragments, we wanted to make sure we didn\u2019t have their PII lying around in un-encrypted databases. We baked more security into our design by only retaining person fragments in an archived form. This eliminates the potential for using live queries to retrieve these fragments. To further add security to our customers\u2019 data, using the above-discussed encryption service we encrypt each stream prior to archival with encryption keys unique to each customer. From a technical perspective, we felt that at this point our customers\u2019 data was maximally protected.<\/span><\/p>\n

Lastly, with the addition of accepting Record IDs (RIDs) and PersonIDs (PIDs) as input to our APIs, customers also benefit from the secure design of the API. Where a customer previously needed to send PII over the wire to enrich or resolve a record, an opaque identifier will now suffice. More specifically, if a Customer Record is loaded into the Resolve API with a RID, that RID is all that is required to perform an enrich query in <\/span>person.enrich<\/span>. We believe that with capabilities as simple as this we can slowly begin to increase general security within the industry.<\/span><\/p>\n

 <\/p>\n

In the final post<\/a> of the series, we talk about the ways we try to achieve peak performance of our systems and discuss some of the technology choices we made on the backend. Thanks for reading and we\u2019ll see you next week!<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

The following is the third installment of a multi-part series offering a glimpse of the behind-the-scenes advancements we achieved to deliver our Resolve API. The first post covered the origin of our Identity Graph and Resolve API, and in the second post, we delved into the power of our new persistent PersonID and the ability […]<\/p>\n","protected":false},"author":42,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_improvement_type_select":"improve_an_existing","_thumb_yes_seoaic":false,"_frame_yes_seoaic":false,"seoaic_generate_description":"","seoaic_improve_instructions_prompt":"","seoaic_rollback_content_improvement":"","seoaic_idea_thumbnail_generator":"","thumbnail_generated":false,"thumbnail_generate_prompt":"","seoaic_article_description":"","seoaic_article_subtitles":[],"footnotes":""},"categories":[679,656],"tags":[5638,5639,674,75,290,98,478,50],"class_list":["post-18427","post","type-post","status-publish","format-standard","hentry","category-resolve","category-engineering","tag-personid","tag-record-id","tag-resolve","tag-api-challenge","tag-identity-2","tag-identity-resolution","tag-identity-graph","tag-api"],"acf":[],"yoast_head":"\nResolve: Building the Identity Resolution Engine (Part 3) | FullContact<\/title>\n<meta name=\"description\" content=\"The following is the third installment of a multi-part series offering a glimpse of the behind-the-scenes advancements we achieved to deliver our Resolve\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.fullcontact.com\/blog\/engineering\/resolve-building-the-identity-resolution-engine-part-3\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Resolve: Building the Identity Resolution Engine (Part 3)\" \/>\n<meta property=\"og:description\" content=\"The following is the third installment of a multi-part series offering a glimpse of the behind-the-scenes advancements we achieved to deliver our Resolve\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.fullcontact.com\/blog\/engineering\/resolve-building-the-identity-resolution-engine-part-3\/\" \/>\n<meta property=\"og:site_name\" content=\"FullContact\" \/>\n<meta property=\"article:published_time\" content=\"2020-05-22T16:00:49+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-01-06T11:35:02+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.fullcontact.com\/wp-content\/uploads\/2020\/05\/eng-blog-chart@2x-1-300x261.png\" \/>\n<meta name=\"author\" content=\"Ken Michie\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@fullcontact\" \/>\n<meta name=\"twitter:site\" content=\"@fullcontact\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Ken Michie\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.fullcontact.com\/blog\/engineering\/resolve-building-the-identity-resolution-engine-part-3\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.fullcontact.com\/blog\/engineering\/resolve-building-the-identity-resolution-engine-part-3\/\"},\"author\":{\"name\":\"Ken Michie\",\"@id\":\"https:\/\/www.fullcontact.com\/#\/schema\/person\/763f69f799a4cd2a2188fea684c3f337\"},\"headline\":\"Resolve: Building the Identity Resolution Engine (Part 3)\",\"datePublished\":\"2020-05-22T16:00:49+00:00\",\"dateModified\":\"2025-01-06T11:35:02+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.fullcontact.com\/blog\/engineering\/resolve-building-the-identity-resolution-engine-part-3\/\"},\"wordCount\":560,\"publisher\":{\"@id\":\"https:\/\/www.fullcontact.com\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.fullcontact.com\/blog\/engineering\/resolve-building-the-identity-resolution-engine-part-3\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.fullcontact.com\/wp-content\/uploads\/2020\/05\/eng-blog-chart@2x-1-300x261.png\",\"keywords\":[\"PersonID\",\"Record ID\",\"Resolve\",\"api challenge\",\"identity\",\"identity resolution\",\"identity graph\",\"API\"],\"articleSection\":[\"Resolve\",\"Engineering\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.fullcontact.com\/blog\/engineering\/resolve-building-the-identity-resolution-engine-part-3\/\",\"url\":\"https:\/\/www.fullcontact.com\/blog\/engineering\/resolve-building-the-identity-resolution-engine-part-3\/\",\"name\":\"Resolve: Building the Identity Resolution Engine (Part 3) | FullContact\",\"isPartOf\":{\"@id\":\"https:\/\/www.fullcontact.com\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.fullcontact.com\/blog\/engineering\/resolve-building-the-identity-resolution-engine-part-3\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.fullcontact.com\/blog\/engineering\/resolve-building-the-identity-resolution-engine-part-3\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.fullcontact.com\/wp-content\/uploads\/2020\/05\/eng-blog-chart@2x-1-300x261.png\",\"datePublished\":\"2020-05-22T16:00:49+00:00\",\"dateModified\":\"2025-01-06T11:35:02+00:00\",\"description\":\"The following is the third installment of a multi-part series offering a glimpse of the behind-the-scenes advancements we achieved to deliver our Resolve\",\"breadcrumb\":{\"@id\":\"https:\/\/www.fullcontact.com\/blog\/engineering\/resolve-building-the-identity-resolution-engine-part-3\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.fullcontact.com\/blog\/engineering\/resolve-building-the-identity-resolution-engine-part-3\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.fullcontact.com\/blog\/engineering\/resolve-building-the-identity-resolution-engine-part-3\/#primaryimage\",\"url\":\"https:\/\/www.fullcontact.com\/wp-content\/uploads\/2020\/05\/eng-blog-chart@2x-1.png\",\"contentUrl\":\"https:\/\/www.fullcontact.com\/wp-content\/uploads\/2020\/05\/eng-blog-chart@2x-1.png\",\"width\":533,\"height\":464},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.fullcontact.com\/blog\/engineering\/resolve-building-the-identity-resolution-engine-part-3\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.fullcontact.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Resolve: Building the Identity Resolution Engine (Part 3)\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.fullcontact.com\/#website\",\"url\":\"https:\/\/www.fullcontact.com\/\",\"name\":\"FullContact\",\"description\":\"Relationships, reimagined.\",\"publisher\":{\"@id\":\"https:\/\/www.fullcontact.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.fullcontact.com\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.fullcontact.com\/#organization\",\"name\":\"FullContact\",\"url\":\"https:\/\/www.fullcontact.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.fullcontact.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.fullcontact.com\/wp-content\/uploads\/2019\/11\/fc-logo@2x.png\",\"contentUrl\":\"https:\/\/www.fullcontact.com\/wp-content\/uploads\/2019\/11\/fc-logo@2x.png\",\"width\":200,\"height\":38,\"caption\":\"FullContact\"},\"image\":{\"@id\":\"https:\/\/www.fullcontact.com\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/x.com\/fullcontact\",\"https:\/\/www.linkedin.com\/company\/fullcontact-inc-\",\"https:\/\/www.youtube.com\/user\/FullContactAPI\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.fullcontact.com\/#\/schema\/person\/763f69f799a4cd2a2188fea684c3f337\",\"name\":\"Ken Michie\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.fullcontact.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/59c32a659aa924a55d4df202ed7d2ddefc1aefe5a6e7b369e15a5c222fe58c93?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/59c32a659aa924a55d4df202ed7d2ddefc1aefe5a6e7b369e15a5c222fe58c93?s=96&d=mm&r=g\",\"caption\":\"Ken Michie\"},\"url\":\"https:\/\/www.fullcontact.com\/blog\/author\/kenmichie\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Resolve: Building the Identity Resolution Engine (Part 3) | FullContact","description":"The following is the third installment of a multi-part series offering a glimpse of the behind-the-scenes advancements we achieved to deliver our Resolve","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.fullcontact.com\/blog\/engineering\/resolve-building-the-identity-resolution-engine-part-3\/","og_locale":"en_US","og_type":"article","og_title":"Resolve: Building the Identity Resolution Engine (Part 3)","og_description":"The following is the third installment of a multi-part series offering a glimpse of the behind-the-scenes advancements we achieved to deliver our Resolve","og_url":"https:\/\/www.fullcontact.com\/blog\/engineering\/resolve-building-the-identity-resolution-engine-part-3\/","og_site_name":"FullContact","article_published_time":"2020-05-22T16:00:49+00:00","article_modified_time":"2025-01-06T11:35:02+00:00","og_image":[{"url":"https:\/\/www.fullcontact.com\/wp-content\/uploads\/2020\/05\/eng-blog-chart@2x-1-300x261.png","type":"","width":"","height":""}],"author":"Ken Michie","twitter_card":"summary_large_image","twitter_creator":"@fullcontact","twitter_site":"@fullcontact","twitter_misc":{"Written by":"Ken Michie","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.fullcontact.com\/blog\/engineering\/resolve-building-the-identity-resolution-engine-part-3\/#article","isPartOf":{"@id":"https:\/\/www.fullcontact.com\/blog\/engineering\/resolve-building-the-identity-resolution-engine-part-3\/"},"author":{"name":"Ken Michie","@id":"https:\/\/www.fullcontact.com\/#\/schema\/person\/763f69f799a4cd2a2188fea684c3f337"},"headline":"Resolve: Building the Identity Resolution Engine (Part 3)","datePublished":"2020-05-22T16:00:49+00:00","dateModified":"2025-01-06T11:35:02+00:00","mainEntityOfPage":{"@id":"https:\/\/www.fullcontact.com\/blog\/engineering\/resolve-building-the-identity-resolution-engine-part-3\/"},"wordCount":560,"publisher":{"@id":"https:\/\/www.fullcontact.com\/#organization"},"image":{"@id":"https:\/\/www.fullcontact.com\/blog\/engineering\/resolve-building-the-identity-resolution-engine-part-3\/#primaryimage"},"thumbnailUrl":"https:\/\/www.fullcontact.com\/wp-content\/uploads\/2020\/05\/eng-blog-chart@2x-1-300x261.png","keywords":["PersonID","Record ID","Resolve","api challenge","identity","identity resolution","identity graph","API"],"articleSection":["Resolve","Engineering"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.fullcontact.com\/blog\/engineering\/resolve-building-the-identity-resolution-engine-part-3\/","url":"https:\/\/www.fullcontact.com\/blog\/engineering\/resolve-building-the-identity-resolution-engine-part-3\/","name":"Resolve: Building the Identity Resolution Engine (Part 3) | FullContact","isPartOf":{"@id":"https:\/\/www.fullcontact.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.fullcontact.com\/blog\/engineering\/resolve-building-the-identity-resolution-engine-part-3\/#primaryimage"},"image":{"@id":"https:\/\/www.fullcontact.com\/blog\/engineering\/resolve-building-the-identity-resolution-engine-part-3\/#primaryimage"},"thumbnailUrl":"https:\/\/www.fullcontact.com\/wp-content\/uploads\/2020\/05\/eng-blog-chart@2x-1-300x261.png","datePublished":"2020-05-22T16:00:49+00:00","dateModified":"2025-01-06T11:35:02+00:00","description":"The following is the third installment of a multi-part series offering a glimpse of the behind-the-scenes advancements we achieved to deliver our Resolve","breadcrumb":{"@id":"https:\/\/www.fullcontact.com\/blog\/engineering\/resolve-building-the-identity-resolution-engine-part-3\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.fullcontact.com\/blog\/engineering\/resolve-building-the-identity-resolution-engine-part-3\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.fullcontact.com\/blog\/engineering\/resolve-building-the-identity-resolution-engine-part-3\/#primaryimage","url":"https:\/\/www.fullcontact.com\/wp-content\/uploads\/2020\/05\/eng-blog-chart@2x-1.png","contentUrl":"https:\/\/www.fullcontact.com\/wp-content\/uploads\/2020\/05\/eng-blog-chart@2x-1.png","width":533,"height":464},{"@type":"BreadcrumbList","@id":"https:\/\/www.fullcontact.com\/blog\/engineering\/resolve-building-the-identity-resolution-engine-part-3\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.fullcontact.com\/"},{"@type":"ListItem","position":2,"name":"Resolve: Building the Identity Resolution Engine (Part 3)"}]},{"@type":"WebSite","@id":"https:\/\/www.fullcontact.com\/#website","url":"https:\/\/www.fullcontact.com\/","name":"FullContact","description":"Relationships, reimagined.","publisher":{"@id":"https:\/\/www.fullcontact.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.fullcontact.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.fullcontact.com\/#organization","name":"FullContact","url":"https:\/\/www.fullcontact.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.fullcontact.com\/#\/schema\/logo\/image\/","url":"https:\/\/www.fullcontact.com\/wp-content\/uploads\/2019\/11\/fc-logo@2x.png","contentUrl":"https:\/\/www.fullcontact.com\/wp-content\/uploads\/2019\/11\/fc-logo@2x.png","width":200,"height":38,"caption":"FullContact"},"image":{"@id":"https:\/\/www.fullcontact.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/x.com\/fullcontact","https:\/\/www.linkedin.com\/company\/fullcontact-inc-","https:\/\/www.youtube.com\/user\/FullContactAPI"]},{"@type":"Person","@id":"https:\/\/www.fullcontact.com\/#\/schema\/person\/763f69f799a4cd2a2188fea684c3f337","name":"Ken Michie","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.fullcontact.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/59c32a659aa924a55d4df202ed7d2ddefc1aefe5a6e7b369e15a5c222fe58c93?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/59c32a659aa924a55d4df202ed7d2ddefc1aefe5a6e7b369e15a5c222fe58c93?s=96&d=mm&r=g","caption":"Ken Michie"},"url":"https:\/\/www.fullcontact.com\/blog\/author\/kenmichie\/"}]}},"_links":{"self":[{"href":"https:\/\/www.fullcontact.com\/wp-json\/wp\/v2\/posts\/18427","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.fullcontact.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.fullcontact.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.fullcontact.com\/wp-json\/wp\/v2\/users\/42"}],"replies":[{"embeddable":true,"href":"https:\/\/www.fullcontact.com\/wp-json\/wp\/v2\/comments?post=18427"}],"version-history":[{"count":0,"href":"https:\/\/www.fullcontact.com\/wp-json\/wp\/v2\/posts\/18427\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.fullcontact.com\/wp-json\/wp\/v2\/media?parent=18427"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.fullcontact.com\/wp-json\/wp\/v2\/categories?post=18427"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.fullcontact.com\/wp-json\/wp\/v2\/tags?post=18427"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}