{"id":17304,"date":"2019-09-05T14:07:37","date_gmt":"2019-09-05T14:07:37","guid":{"rendered":"https:\/\/www.fullcontact.com\/?p=17304"},"modified":"2023-02-08T04:56:35","modified_gmt":"2023-02-08T11:56:35","slug":"our-process-of-deploying-running-and-maintaining-druid","status":"publish","type":"post","link":"https:\/\/www.fullcontact.com\/blog\/engineering\/our-process-of-deploying-running-and-maintaining-druid\/","title":{"rendered":"Our Process of Deploying, Running and Maintaining Druid"},"content":{"rendered":"<p><span style=\"font-weight: 400;\">FullContact is a people company and a data company. We believe that businesses can deliver higher quality products through better relationships and 1-1 customized service. Our Identity Resolution products (both batch and API) are what enables that to happen. We have several different APIs but they can be boiled down to two simple use cases: <\/span><\/p>\n<ul>\n<li><span style=\"font-weight: 400;\"><strong>Resolve &#8211; <\/strong>Who does this identifier connect to?<\/span><\/li>\n<li><span style=\"font-weight: 400;\"><strong>Enrich<\/strong> &#8211; W<i>hat other information do you know about that person that can help the business better connect?<\/i>\u00a0<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">Our API is embedded by customers when they want to make real-time decisions about their customers and users: what types of content might this person be interested in, have they talked about our business on social media, etc. These real-time queries and responses are made to our API, <em>api.fullcontact.com<\/em>. The client passes one or more identifiers (email, social handle, etc.) and a response is returned with several pieces of information that enriches and adds additional information about the person. These enrichment pieces can be things like age, interests, demographics and are grouped together into Insights Bundles (formerly Data Packs).<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Batch enrichment works much the same way but is more asynchronous and capable of dealing with millions of records all at once. A client of ours will securely ship us a file of identifiers, we process it, match it to our graph and append additional columns of enrichment data as defined by the agreement with the customer and the Insights Bundles they want to pay for.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">In both of these cases, API and batch, we had similar problems that needed to be solved:<\/span><\/p>\n<ol>\n<li><span style=\"font-weight: 400;\">How many calls did a client make to us over a given time period? We needed to find out how many 200s (matches), and 404s they received.<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Out of the data that was returned to them how many of each Insights Bundle did they receive?<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Based on the Insights Bundles returned, how much data has the client consumed towards their committed plan?<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Out of the Insights Bundles returned, how much do we owe our upstream data providers?<\/span><\/li>\n<\/ol>\n<p><span style=\"font-weight: 400;\">While we could store all of the data needed to compute these things in S3 and run large aggregation jobs using something like Hive or Spark, we wanted to do better. Our new usage dashboard requires that we have fast access to this usage data and can aggregate across several dimensions: account ID, Insights Bundles, time range, and more. In order to meet all of the above requirements, we built a streaming architecture that unifies usage data from both API and batch onto a Kafka topic and eventually makes it into <strong>Druid<\/strong>, an OLAP high-performance system where we can slice and dice the data in nearly any way we want.<\/span><\/p>\n<h2>Architecture<\/h2>\n<p><span style=\"font-weight: 400;\">Let&#8217;s examine the life of a usage event and how it flows through our architecture.<\/span><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-17305 size-large\" src=\"https:\/\/www.fullcontact.com\/wp-content\/uploads\/2019\/09\/architecture-diagram-620x304.png\" alt=\"\" width=\"620\" height=\"304\" \/><\/p>\n<h3><span style=\"font-weight: 400;\">First, from the API side:<\/span><\/h3>\n<ol>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">A client with an authenticated FullContact API token makes a call to our API to resolve email<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Our internal API layer processes the request, figures out the result, and returns it to the client. <\/span>\n<ol>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">As part of this process, an Avro message is emitted to a Kafka usage topic.<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">This Avro message is registered in the schema registry.<\/span><\/li>\n<\/ol>\n<\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\"><a href=\"https:\/\/github.com\/pinterest\/secor\">Secor<\/a> (an open source project from Pinterest) is running in our Kubernetes cluster as a stateful set, reads from the Kafka topic and archives the data to S3 in columnar parquet storage.<\/span>\n<ol>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">AWS Glue runs regular crawlers on this S3 data and registers tables on top of it, making it available for later querying if needed in Athena and other tools.<\/span><\/li>\n<\/ol>\n<\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">\u00a0A <strong>Druid Kafka Indexer <\/strong>supervisor runs on Druid and launches the Peon workers on the data nodes that do the actual ingestion. These jobs read the data, apply basic transformations, and roll up aggregations and store it in the appropriate segments.<\/span><\/li>\n<\/ol>\n<p><span style=\"font-weight: 400;\">From the batch side:<\/span><\/p>\n<ol>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">A client securely <strong>delivers their bulk file<\/strong> with IDs to be enriched.<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">A <strong>result file<\/strong> is created by the data pipeline system.<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">A <strong>usage summary report file<\/strong> is generated that has summarized statistics for each row in the results file describing how many Insights Bundles were returned on that row and the source of each Insights Bundle. This file is persisted in S3.<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">When the batch file is approved and delivered back to the customer, <strong>a process is run<\/strong> that reads the summary report file and creates standard Avro usage messages and streams them to the same Kafka topic used in the API side<\/span><\/li>\n<\/ol>\n<p><span style=\"font-weight: 400;\">From a user accessing our developer dashboard to view their usage at <em>dashboard.fullcontact.com<\/em>:<\/span><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-17315 size-img-small\" src=\"https:\/\/www.fullcontact.com\/wp-content\/uploads\/2019\/09\/Screen-Shot-2019-09-05-at-6.41.52-PM-750x467.png\" alt=\"\" width=\"750\" height=\"467\" \/><\/p>\n<p>&nbsp;<\/p>\n<ol>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">User logins into the <strong>usage dashboard<\/strong>.<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Our web service backend passes their account ID and requested time ranges to our <strong>usage middle tier service<\/strong>.<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">The usage middle tier service has several canned Druid JSON queries that are filled in and then submitted to the <strong>Druid Query Broker<\/strong>.\u00a0<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Results from the <strong>Druid Query Broker<\/strong> are passed back to the client dashboard and rendered in various charts.\u00a0<\/span><\/li>\n<\/ol>\n<h2><b>Deploying, Running and Maintaining Druid<\/b><\/h2>\n<p><span style=\"font-weight: 400;\">Like you would with any new technology we went through several different phases of trying out and experimenting with Druid to actually getting it to a state we felt comfortable going with to production.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">The Druid project offers a \u201c<a href=\"https:\/\/druid.apache.org\/docs\/latest\/tutorials\/index.html\">quick start<\/a>\u201d configuration that is meant to be run on a small local instance. We found this to be useful to stand up Druid locally on our laptops to try out very simple configurations.\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Once we graduated from that and wanted to ingest from a larger Kafka topic we manually configured Druid on a i2.xlarge instance we had unused reservation capacity for. Since this instance is quite a bit larger (has a decent amount more ephemeral disk and more CPU and memory) we modified some of the quickstart JVM parameters to use this additional capacity. In this initial configuration we were only using local deep storage, a local zookeeper and local in memory database for the metastore; all configurations you probably shouldn\u2019t rely on in production.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Before going to production we wanted our infrastructure to have a few properties:<\/span><\/p>\n<ol>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Be completely automated and immutable, if we want to change something we can spin up new instances.<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Be able to replace or scale a single component without losing data.<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Be monitored and have automated alerts in case of error conditions.<\/span><\/li>\n<\/ol>\n<p><span style=\"font-weight: 400;\">We went with the common deployment approach of classifying the various Druid subprocesses into three different types of nodes:<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Master Node<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Coordinator (port 8081)<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Overlord (port 8090)<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">Query Node<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Broker (port 8082)<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Router (port 8088)<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">Data Node<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Historical (port 8083)<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Middle Manager (port 8091)<\/span><\/li>\n<\/ul>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-17307 size-large\" src=\"https:\/\/www.fullcontact.com\/wp-content\/uploads\/2019\/09\/node-druid-apache-620x422.png\" alt=\"\" width=\"620\" height=\"422\" \/><\/p>\n<p><span style=\"font-weight: 400;\">The three different kinds of nodes are described using ansible playbooks and roles. We bake a single Druid AMI that contains the configuration and JVMs that can support any one of the above three nodes then supply user data variables that the systemd services use to decide which components to launch.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">We configured a RDS Mysql database for the metastore and a completely external and highly available 3 node zookeeper cluster to coordinate. Externalizing these components and adding S3 deep storage means we can completely replace any of the nodes above without losing any data.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">For monitoring our shiny new Druid instance we setup the following prometheus components:<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">Prometheus Node Exporter<\/span>\n<ul>\n<li>provides common system metrics like CPU, Memory, Network, disk usage<\/li>\n<\/ul>\n<\/li>\n<li style=\"font-weight: 400;\">Prometheus Druid Exporter\n<ul>\n<li>exports druid specific like number of queries, query durations, failed queries and missing segments<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">We have a simple dashboard in grafana that will show us some of the important Druid metrics that are getting scraped to prometheus:<\/span><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"wp-image-17348 size-large aligncenter\" src=\"https:\/\/www.fullcontact.com\/wp-content\/uploads\/2019\/09\/Druid-Prod-620x348-1-620x348.png\" alt=\"\" width=\"620\" height=\"348\" \/><\/p>\n<p><span style=\"font-weight: 400;\">For alerting, we are using prometheus-alert manager to alert us when the missing segments count (the number of segments that the historical process has been unable to register) climbs above some specified value:<\/span><\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-17309 size-large\" src=\"https:\/\/www.fullcontact.com\/wp-content\/uploads\/2019\/09\/Druid-Missing-Segments-620x98.png\" alt=\"\" width=\"620\" height=\"98\" \/><\/p>\n<p><span style=\"font-weight: 400;\">Once we have this Druid cluster setup we can submit our ingestion spec to Druid that defines how to ingest the stream from Kafka. In our initial setup of of storing segments per day with no roll ups we were storing about 2GB per day. Once we started rolling up similar events occurring during the same minute using HLLSketchBuild we were able to drop this down to around 150MB. Given this fairly low data footprint and the possibility for us to define more aggressive rollup windows once data ages we feel confident we will be able to store all the usage data we need at a relatively low cost.<\/span><\/p>\n<h2><b>Why Avro, why Schema registry?<\/b><\/h2>\n<p><span style=\"font-weight: 400;\">FullContact historically uses Google Protobuf wrapped in a few extra layers of serialization for passing messages around on Kafka topics. Early on in the usage system design we decided to go with Avro. Why Avro?\u00a0<\/span><\/p>\n<p><span style=\"font-weight: 400;\">One of the out of the box tools you can get with the open source Confluent Kafka distribution is the Schema registry. The schema registry is a lightweight REST service that can be run on the Kafka brokers that will keep track of the schema definition that each topic should use. Several Kafka clients and other third party tools like Secor and Druid integrate nicely with the schema registry. Going this route let us avoid writing any custom Ser\/De logic.\u00a0<\/span><\/p>\n<h2><b>Druid Migration v0.14 to v0.15<\/b><\/h2>\n<p><span style=\"font-weight: 400;\">We initially started the 3 node Druid cluster with v14 which was soon outdated by v15. We tried to predict the steps for migration so that we could accomplish migration and keep our existing data with no downtime to the cluster. We decided to bring up a v15 cluster in parallel and migrate the existing data to the new cluster. One of the major changes to Druid v15 is in the way how the segment information is stored.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">Prior to v15 there are two places where the segment information is stored:<\/span><\/p>\n<ul>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">As descriptor.json in deep storage<\/span><\/li>\n<li style=\"font-weight: 400;\"><span style=\"font-weight: 400;\">As loadspec in Metadata store<\/span><\/li>\n<\/ul>\n<p><span style=\"font-weight: 400;\">With v15, Druid now stores this information only in Metadata store. This lead to the deprecation of the tool `insert-segments-to-db-tool`. This brought a challenge with manual migration of metadata since we could no longer follow the guides we were able to find online.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">To work around this we created a new Druid Cluster with S3 deep storage and metadata store. Then we copied the old deep storage and metastore to the new instances. Now the metastore contains the loadspec for each segment that point to old deep storage.<\/span><\/p>\n<p><code><br \/>\n{<br \/>\n\"loadSpec\": {<br \/>\n\"type\":\"s3_zip\",<br \/>\n\"bucket\":\"druid-deepstorage-data\",<br \/>\n\"key\":\"druid\/segments\/usage_v0_query\/2019-07-18T00:00:00.000Z_2019-07-19T00:00:00.000Z\/2019-07-18T00:00:00.143Z\/6\/e458d343-e666-4665-85aa-f4ba2276da1e\/index.zip\"<br \/>\n}<br \/>\n}<br \/>\n<\/code><\/p>\n<p><span style=\"font-weight: 400;\">After examining the metadata for the segments you can see it is all pointing to the old deep storage location. In order to update the segment metadata to point to the new deep storage location we created a simple python script to scan through the database and update the records:<\/span><\/p>\n<p><code><br \/>\n#!\/usr\/bin\/python<br \/>\nimport MySQLdb<br \/>\nimport json<br \/>\nimport pathlib<br \/>\ndb = MySQLdb.connect(host=\"druid-**.qpcxrads.us-east-1.rds.amazonaws.com\",    # your host, usually localhost<br \/>\nuser=\"druid\",         # your username<br \/>\npasswd=\"**\",  # your password<br \/>\ndb=\"druid\")        # name of the database<br \/>\n# you must create a Cursor object. It will let<br \/>\n# you execute all the queries you need<br \/>\ncur = db.cursor()<br \/>\n# Use all the SQL you like<br \/>\ncur.execute(\"SELECT * FROM druid_segments\")<br \/>\n# print all the first cell of all the rows<br \/>\nfor row in cur.fetchall():<br \/>\nprint row[0]<br \/>\nloadSpec=json.loads(row[8])['loadSpec']<br \/>\nif loadSpec['bucket']=='druid-deepstorage-data': #Old deep storage<br \/>\nloadSpec['bucket']='druid-deepstorage-0-15-data' #New Deep Storage<br \/>\npayload=json.loads(row[8])<br \/>\npayload['loadSpec']=loadSpec<br \/>\nsql = \"UPDATE druid_segments SET payload = %s WHERE id = %s\"<br \/>\nval = (json.dumps(payload), row[0])<br \/>\ncur.execute(sql, val)<br \/>\ndb.commit()<br \/>\ndb.close()<br \/>\n<\/code><\/p>\n<p><span style=\"font-weight: 400;\">Once this change is made, starting the Druid Master, Data and Query servers will bring up the cluster running with old data. Since this migration of v14 to v15 we have experimented with a few more data migration scenarios with Druid during our most recent Hack Week:<\/span><\/p>\n<ul>\n<li>Standing up a secondary &#8220;sandbox cluster&#8221; that shares the same deep storage as prod for all historical data and its own deep storage for new data<\/li>\n<li><span style=\"font-weight: 400;\">Ingesting batch data from Parquet (work in progress)\u00a0<\/span><\/li>\n<\/ul>\n<h2><b>Summary and Next Steps<\/b><\/h2>\n<p><span style=\"font-weight: 400;\">Learning Druid has been a fun and informative process. This new journey feels similar to back in the day when I was learning the basics of Hadoop. So much to learn and so much potential with the technology. As you have probably noticed from our writeup the scale of our Druid cluster is fairly small (there are some descriptions of larger organizations running 1000+ node clusters) and we are only using some of the most basic features. We are excited to explore all of the additional features of Druid that can help give us even more real-time streaming insight to our data.<\/span><\/p>\n<p><span style=\"font-weight: 400;\">FullContact is hiring! We have several engineering roles open on our careers page. One of those is on the <a href=\"https:\/\/www.fullcontact.com\/careers\/\"><strong>Foundation Integrations<\/strong><\/a> team (the same one that is working on the systems described in this blog). So if you want to join a fast moving team working on Druid, JVM microservices (Clojure and Java) all deployed on AWS apply now and shoot me a note at <a href=\"mailto:jeremy.plichta@fullcontact.com\">jeremy.plichta@fullcontact.com<\/a>.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"<p>FullContact is a people company and a data company. We believe that businesses can deliver higher quality products through better relationships and 1-1 customized service. Our Identity Resolution products (both batch and API) are what enables that to happen. We have several different APIs but they can be boiled down to two simple use cases: [&hellip;]<\/p>\n","protected":false},"author":91,"featured_media":17326,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_improvement_type_select":"improve_an_existing","_thumb_yes_seoaic":false,"_frame_yes_seoaic":false,"seoaic_generate_description":"","seoaic_improve_instructions_prompt":"","seoaic_rollback_content_improvement":"","seoaic_idea_thumbnail_generator":"","thumbnail_generated":false,"thumbnail_generate_prompt":"","seoaic_article_description":"","seoaic_article_subtitles":[],"footnotes":""},"categories":[656],"tags":[609,610,611,612,50,72],"class_list":["post-17304","post","type-post","status-publish","format-standard","hentry","category-engineering","tag-avro","tag-code","tag-druid","tag-fullcontact-engineering","tag-api","tag-devs"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v27.1 (Yoast SEO v27.1.1) - https:\/\/yoast.com\/product\/yoast-seo-premium-wordpress\/ -->\n<title>Our Process of Deploying, Running and Maintaining Druid | FullContact<\/title>\n<meta name=\"description\" content=\"Learn FullContact&#039;s process of deploying, running and maintaining druid.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.fullcontact.com\/blog\/engineering\/our-process-of-deploying-running-and-maintaining-druid\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Our Process of Deploying, Running and Maintaining Druid\" \/>\n<meta property=\"og:description\" content=\"Learn FullContact&#039;s process of deploying, running and maintaining druid.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.fullcontact.com\/blog\/engineering\/our-process-of-deploying-running-and-maintaining-druid\/\" \/>\n<meta property=\"og:site_name\" content=\"FullContact\" \/>\n<meta property=\"article:published_time\" content=\"2019-09-05T14:07:37+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-02-08T11:56:35+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.fullcontact.com\/wp-content\/uploads\/2020\/07\/party-data-blog-li.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1200\" \/>\n\t<meta property=\"og:image:height\" content=\"630\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Jeremy Plichta\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@fullcontact\" \/>\n<meta name=\"twitter:site\" content=\"@fullcontact\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Jeremy Plichta\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"11 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.fullcontact.com\/blog\/engineering\/our-process-of-deploying-running-and-maintaining-druid\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.fullcontact.com\/blog\/engineering\/our-process-of-deploying-running-and-maintaining-druid\/\"},\"author\":{\"name\":\"Jeremy Plichta\",\"@id\":\"https:\/\/www.fullcontact.com\/#\/schema\/person\/72c64648c1d628849506fa594a520e11\"},\"headline\":\"Our Process of Deploying, Running and Maintaining Druid\",\"datePublished\":\"2019-09-05T14:07:37+00:00\",\"dateModified\":\"2023-02-08T11:56:35+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.fullcontact.com\/blog\/engineering\/our-process-of-deploying-running-and-maintaining-druid\/\"},\"wordCount\":2016,\"publisher\":{\"@id\":\"https:\/\/www.fullcontact.com\/#organization\"},\"image\":{\"@id\":\"https:\/\/www.fullcontact.com\/blog\/engineering\/our-process-of-deploying-running-and-maintaining-druid\/#primaryimage\"},\"thumbnailUrl\":\"\",\"keywords\":[\"Avro\",\"Code\",\"Druid\",\"FullContact Engineering\",\"API\",\"developers\"],\"articleSection\":[\"Engineering\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.fullcontact.com\/blog\/engineering\/our-process-of-deploying-running-and-maintaining-druid\/\",\"url\":\"https:\/\/www.fullcontact.com\/blog\/engineering\/our-process-of-deploying-running-and-maintaining-druid\/\",\"name\":\"Our Process of Deploying, Running and Maintaining Druid | FullContact\",\"isPartOf\":{\"@id\":\"https:\/\/www.fullcontact.com\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.fullcontact.com\/blog\/engineering\/our-process-of-deploying-running-and-maintaining-druid\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.fullcontact.com\/blog\/engineering\/our-process-of-deploying-running-and-maintaining-druid\/#primaryimage\"},\"thumbnailUrl\":\"\",\"datePublished\":\"2019-09-05T14:07:37+00:00\",\"dateModified\":\"2023-02-08T11:56:35+00:00\",\"description\":\"Learn FullContact's process of deploying, running and maintaining druid.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.fullcontact.com\/blog\/engineering\/our-process-of-deploying-running-and-maintaining-druid\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.fullcontact.com\/blog\/engineering\/our-process-of-deploying-running-and-maintaining-druid\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.fullcontact.com\/blog\/engineering\/our-process-of-deploying-running-and-maintaining-druid\/#primaryimage\",\"url\":\"\",\"contentUrl\":\"\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.fullcontact.com\/blog\/engineering\/our-process-of-deploying-running-and-maintaining-druid\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.fullcontact.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Our Process of Deploying, Running and Maintaining Druid\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.fullcontact.com\/#website\",\"url\":\"https:\/\/www.fullcontact.com\/\",\"name\":\"FullContact\",\"description\":\"Relationships, reimagined.\",\"publisher\":{\"@id\":\"https:\/\/www.fullcontact.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.fullcontact.com\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.fullcontact.com\/#organization\",\"name\":\"FullContact\",\"url\":\"https:\/\/www.fullcontact.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.fullcontact.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.fullcontact.com\/wp-content\/uploads\/2019\/11\/fc-logo@2x.png\",\"contentUrl\":\"https:\/\/www.fullcontact.com\/wp-content\/uploads\/2019\/11\/fc-logo@2x.png\",\"width\":200,\"height\":38,\"caption\":\"FullContact\"},\"image\":{\"@id\":\"https:\/\/www.fullcontact.com\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/x.com\/fullcontact\",\"https:\/\/www.linkedin.com\/company\/fullcontact-inc-\",\"https:\/\/www.youtube.com\/user\/FullContactAPI\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.fullcontact.com\/#\/schema\/person\/72c64648c1d628849506fa594a520e11\",\"name\":\"Jeremy Plichta\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.fullcontact.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/da25040c6b6f787c42fda93950ff50ef24ce1de3063a3943928f7e6b67db954a?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/da25040c6b6f787c42fda93950ff50ef24ce1de3063a3943928f7e6b67db954a?s=96&d=mm&r=g\",\"caption\":\"Jeremy Plichta\"},\"url\":\"https:\/\/www.fullcontact.com\/blog\/author\/jeremyplichta\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Our Process of Deploying, Running and Maintaining Druid | FullContact","description":"Learn FullContact's process of deploying, running and maintaining druid.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.fullcontact.com\/blog\/engineering\/our-process-of-deploying-running-and-maintaining-druid\/","og_locale":"en_US","og_type":"article","og_title":"Our Process of Deploying, Running and Maintaining Druid","og_description":"Learn FullContact's process of deploying, running and maintaining druid.","og_url":"https:\/\/www.fullcontact.com\/blog\/engineering\/our-process-of-deploying-running-and-maintaining-druid\/","og_site_name":"FullContact","article_published_time":"2019-09-05T14:07:37+00:00","article_modified_time":"2023-02-08T11:56:35+00:00","og_image":[{"width":1200,"height":630,"url":"https:\/\/www.fullcontact.com\/wp-content\/uploads\/2020\/07\/party-data-blog-li.png","type":"image\/png"}],"author":"Jeremy Plichta","twitter_card":"summary_large_image","twitter_creator":"@fullcontact","twitter_site":"@fullcontact","twitter_misc":{"Written by":"Jeremy Plichta","Est. reading time":"11 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.fullcontact.com\/blog\/engineering\/our-process-of-deploying-running-and-maintaining-druid\/#article","isPartOf":{"@id":"https:\/\/www.fullcontact.com\/blog\/engineering\/our-process-of-deploying-running-and-maintaining-druid\/"},"author":{"name":"Jeremy Plichta","@id":"https:\/\/www.fullcontact.com\/#\/schema\/person\/72c64648c1d628849506fa594a520e11"},"headline":"Our Process of Deploying, Running and Maintaining Druid","datePublished":"2019-09-05T14:07:37+00:00","dateModified":"2023-02-08T11:56:35+00:00","mainEntityOfPage":{"@id":"https:\/\/www.fullcontact.com\/blog\/engineering\/our-process-of-deploying-running-and-maintaining-druid\/"},"wordCount":2016,"publisher":{"@id":"https:\/\/www.fullcontact.com\/#organization"},"image":{"@id":"https:\/\/www.fullcontact.com\/blog\/engineering\/our-process-of-deploying-running-and-maintaining-druid\/#primaryimage"},"thumbnailUrl":"","keywords":["Avro","Code","Druid","FullContact Engineering","API","developers"],"articleSection":["Engineering"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.fullcontact.com\/blog\/engineering\/our-process-of-deploying-running-and-maintaining-druid\/","url":"https:\/\/www.fullcontact.com\/blog\/engineering\/our-process-of-deploying-running-and-maintaining-druid\/","name":"Our Process of Deploying, Running and Maintaining Druid | FullContact","isPartOf":{"@id":"https:\/\/www.fullcontact.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.fullcontact.com\/blog\/engineering\/our-process-of-deploying-running-and-maintaining-druid\/#primaryimage"},"image":{"@id":"https:\/\/www.fullcontact.com\/blog\/engineering\/our-process-of-deploying-running-and-maintaining-druid\/#primaryimage"},"thumbnailUrl":"","datePublished":"2019-09-05T14:07:37+00:00","dateModified":"2023-02-08T11:56:35+00:00","description":"Learn FullContact's process of deploying, running and maintaining druid.","breadcrumb":{"@id":"https:\/\/www.fullcontact.com\/blog\/engineering\/our-process-of-deploying-running-and-maintaining-druid\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.fullcontact.com\/blog\/engineering\/our-process-of-deploying-running-and-maintaining-druid\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.fullcontact.com\/blog\/engineering\/our-process-of-deploying-running-and-maintaining-druid\/#primaryimage","url":"","contentUrl":""},{"@type":"BreadcrumbList","@id":"https:\/\/www.fullcontact.com\/blog\/engineering\/our-process-of-deploying-running-and-maintaining-druid\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.fullcontact.com\/"},{"@type":"ListItem","position":2,"name":"Our Process of Deploying, Running and Maintaining Druid"}]},{"@type":"WebSite","@id":"https:\/\/www.fullcontact.com\/#website","url":"https:\/\/www.fullcontact.com\/","name":"FullContact","description":"Relationships, reimagined.","publisher":{"@id":"https:\/\/www.fullcontact.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.fullcontact.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.fullcontact.com\/#organization","name":"FullContact","url":"https:\/\/www.fullcontact.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.fullcontact.com\/#\/schema\/logo\/image\/","url":"https:\/\/www.fullcontact.com\/wp-content\/uploads\/2019\/11\/fc-logo@2x.png","contentUrl":"https:\/\/www.fullcontact.com\/wp-content\/uploads\/2019\/11\/fc-logo@2x.png","width":200,"height":38,"caption":"FullContact"},"image":{"@id":"https:\/\/www.fullcontact.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/x.com\/fullcontact","https:\/\/www.linkedin.com\/company\/fullcontact-inc-","https:\/\/www.youtube.com\/user\/FullContactAPI"]},{"@type":"Person","@id":"https:\/\/www.fullcontact.com\/#\/schema\/person\/72c64648c1d628849506fa594a520e11","name":"Jeremy Plichta","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.fullcontact.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/da25040c6b6f787c42fda93950ff50ef24ce1de3063a3943928f7e6b67db954a?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/da25040c6b6f787c42fda93950ff50ef24ce1de3063a3943928f7e6b67db954a?s=96&d=mm&r=g","caption":"Jeremy Plichta"},"url":"https:\/\/www.fullcontact.com\/blog\/author\/jeremyplichta\/"}]}},"_links":{"self":[{"href":"https:\/\/www.fullcontact.com\/wp-json\/wp\/v2\/posts\/17304","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.fullcontact.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.fullcontact.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.fullcontact.com\/wp-json\/wp\/v2\/users\/91"}],"replies":[{"embeddable":true,"href":"https:\/\/www.fullcontact.com\/wp-json\/wp\/v2\/comments?post=17304"}],"version-history":[{"count":0,"href":"https:\/\/www.fullcontact.com\/wp-json\/wp\/v2\/posts\/17304\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.fullcontact.com\/wp-json\/"}],"wp:attachment":[{"href":"https:\/\/www.fullcontact.com\/wp-json\/wp\/v2\/media?parent=17304"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.fullcontact.com\/wp-json\/wp\/v2\/categories?post=17304"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.fullcontact.com\/wp-json\/wp\/v2\/tags?post=17304"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}