{"id":19350,"date":"2021-03-11T09:45:27","date_gmt":"2021-03-11T16:45:27","guid":{"rendered":"https:\/\/www.fullcontact.com\/?p=19350"},"modified":"2023-03-28T04:54:07","modified_gmt":"2023-03-28T10:54:07","slug":"redriving-the-databus","status":"publish","type":"post","link":"https:\/\/www.fullcontact.com\/blog\/engineering\/redriving-the-databus\/","title":{"rendered":"(Re)Driving The Databus"},"content":{"rendered":"

In this blog, we\u2019ll explore the backend processes and architecture that power <\/span><\/i>Resolve<\/span><\/i><\/a> while discussing some challenges we faced along the way.<\/span><\/i><\/p>\n

When designing FullContact\u2019s newest product, <\/span>Resolve<\/span><\/a>, we borrowed several concepts from our Enrich<\/a> platform and adapted them to support key differences between the two products.\u00a0<\/span><\/p>\n

Our existing enrichment platform, which primarily serves read-only data, uses HBase as its primary data store. A common task automated through Airflow when new datasets are periodically ingested or refreshed internally is to completely rebuild the HBase cluster from the underlying data by generating HFiles via EMR and creating a new read-only database. This enables us to \u201conline\u201d new datasets.\u00a0<\/span><\/p>\n

The Airflow DAG at a high level looks like this:<\/span><\/p>\n

\"\"<\/p>\n

Steps for Launching new HBase via Airflow<\/b><\/p>\n

In fact, multiple large internal databases here at FullContact use this process. This lets us easily leverage our data pipeline for big data processing and switch databases with new and\/or refreshed identifiers with zero downtime. We took a similar approach for our Resolve platform. However, with data being written by customers instead of just reading internal data, we were presented with several challenges.<\/span><\/p>\n

Before we continue, it\u2019s also worth mentioning another key difference with data storage–we decided to use Cassandra in place of HBase, the common database for key FullContact platforms.\u00a0<\/span><\/p>\n

Reaching back to computer science theory, the <\/span>CAP theorem<\/span><\/a> tells us for any database, you can provide at most two of the three properties: Consistency, Availability, and Partition tolerance. HBase covers C and P of the CAP Theorem, allowing consistent reads and partition tolerance while Cassandra covers A and P through consistent hashing and eventual consistency.<\/span><\/p>\n

For Enrich, all customer’s queries read from a statically compiled database containing enrichment data. Given the lack of scalability needs and desire to have very consistent data, we chose HBase.<\/span><\/p>\n

We intentionally sacrificed consistency for partition tolerance for our Resolve product since customers accessing their individual FullContact Identity Streme\u2122<\/a> have higher volumes of dynamic requests. As we explore below, customer records are written to two places: the Cassandra database and archived to S3.\u00a0<\/span><\/p>\n

Additionally, when PersonIDs<\/a> are generated, they\u2019re consistently generated for a given individual customer\u2019s account. Each customer provides Record IDs on their side, so in the worst case, if Record IDs are re-mapped or PersonIDs are re-generated, we can again minimize the consistency concern. Giving up a small amount of consistency lets us focus our attention on A and P: Availability and Partition tolerance with Cassandra.<\/span><\/p>\n

Cassandra also gives us what we need:\u00a0<\/span><\/p>\n