diff options
Diffstat (limited to 'test')
46 files changed, 3343 insertions, 42 deletions
diff --git a/test/test-pages/003-metadata-preferred/expected.html b/test/test-pages/003-metadata-preferred/expected.html index 6b03dd3..b282bdd 100644 --- a/test/test-pages/003-metadata-preferred/expected.html +++ b/test/test-pages/003-metadata-preferred/expected.html @@ -1,4 +1,5 @@ <article> + <h2>Test document title</h2> <p> Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, diff --git a/test/test-pages/004-metadata-space-separated-properties/expected.html b/test/test-pages/004-metadata-space-separated-properties/expected.html index 6b03dd3..b282bdd 100644 --- a/test/test-pages/004-metadata-space-separated-properties/expected.html +++ b/test/test-pages/004-metadata-space-separated-properties/expected.html @@ -1,4 +1,5 @@ <article> + <h2>Test document title</h2> <p> Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, diff --git a/test/test-pages/base-url-base-element-relative/expected.html b/test/test-pages/base-url-base-element-relative/expected.html index 44ae2de..1a8d412 100644 --- a/test/test-pages/base-url-base-element-relative/expected.html +++ b/test/test-pages/base-url-base-element-relative/expected.html @@ -1,5 +1,5 @@ <article> - + <h2>Lorem</h2> <p> Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, diff --git a/test/test-pages/base-url-base-element/expected.html b/test/test-pages/base-url-base-element/expected.html index 3bc1ab4..3a03ddb 100644 --- a/test/test-pages/base-url-base-element/expected.html +++ b/test/test-pages/base-url-base-element/expected.html @@ -1,5 +1,5 @@ <article> - + <h2>Lorem</h2> <p> Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, diff --git a/test/test-pages/base-url/expected.html b/test/test-pages/base-url/expected.html index 504b249..e9da69d 100644 --- a/test/test-pages/base-url/expected.html +++ b/test/test-pages/base-url/expected.html @@ -1,5 +1,5 @@ <article> - + <h2>Lorem</h2> <p> Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, diff --git a/test/test-pages/citylab-1/expected.html b/test/test-pages/citylab-1/expected.html index b57f2e5..2ad9f86 100644 --- a/test/test-pages/citylab-1/expected.html +++ b/test/test-pages/citylab-1/expected.html @@ -12,7 +12,12 @@ <span itemprop="caption">The Moulin Rouge cabaret in Paris</span> <span itemprop="creator">Benoit Tessier/Reuters</span> </figcaption> </figure> - + <div> + <h2 itemprop="headline"> + Why Neon Is the Ultimate Symbol of the 20th Century + </h2> + + </div> <h2 itemprop="description"> The once-ubiquitous form of lighting was novel when it first emerged in the early 1900s, though it has since come to represent decline. </h2> diff --git a/test/test-pages/daringfireball-1/expected.html b/test/test-pages/daringfireball-1/expected.html index 601e17b..7fbd053 100644 --- a/test/test-pages/daringfireball-1/expected.html +++ b/test/test-pages/daringfireball-1/expected.html @@ -1,5 +1,5 @@ <div id="Box"> - + <h2>About This Site</h2> <p>Daring Fireball is written and produced by John Gruber.</p> <p> <a href="http://fakehost/graphics/author/addison-bw.jpg"> <img src="http://fakehost/graphics/author/addison-bw-425.jpg" alt="Photograph of the author."></a> diff --git a/test/test-pages/different-sources-for-images/expected.html b/test/test-pages/different-sources-for-images/expected.html index aa03b79..e5a6261 100644 --- a/test/test-pages/different-sources-for-images/expected.html +++ b/test/test-pages/different-sources-for-images/expected.html @@ -1,5 +1,5 @@ <article> - + <h2>Lorem</h2> <p> Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, diff --git a/test/test-pages/dropbox-blog/expected-images.json b/test/test-pages/dropbox-blog/expected-images.json new file mode 100644 index 0000000..9a202b5 --- /dev/null +++ b/test/test-pages/dropbox-blog/expected-images.json @@ -0,0 +1,5 @@ +[ + "https:\/\/aem.dropbox.com\/cms\/content\/dam\/dropbox\/tech-blog\/en-us\/2020\/11\/atf\/diagrams\/Techblog-ATF-Social.png", + "http:\/\/fakehost\/cms\/content\/dam\/dropbox\/tech-blog\/en-us\/2020\/11\/atf\/diagrams\/Techblog-ATF-720x844px-1.png", + "http:\/\/fakehost\/cms\/content\/dam\/dropbox\/tech-blog\/en-us\/2020\/11\/atf\/diagrams\/Techblog-ATF-720x225px-2.png" +]
\ No newline at end of file diff --git a/test/test-pages/dropbox-blog/expected-metadata.json b/test/test-pages/dropbox-blog/expected-metadata.json new file mode 100644 index 0000000..d3d9913 --- /dev/null +++ b/test/test-pages/dropbox-blog/expected-metadata.json @@ -0,0 +1,8 @@ +{ + "Author": "Arun Sai Krishnan", + "Direction": null, + "Excerpt": "I joined Dropbox not long after graduating with a Master\u2019s degree in computer science. Aside from an internship, this was my first big-league engineering job. My team had already begun designing a critical internal service that most of our software would use: It would handle asynchronous computing requests behind the scenes, powering everything from dragging a file into a Dropbox folder to scheduling a marketing campaign.", + "Image": "https:\/\/aem.dropbox.com\/cms\/content\/dam\/dropbox\/tech-blog\/en-us\/2020\/11\/atf\/diagrams\/Techblog-ATF-Social.png", + "Title": "How we designed Dropbox\u2019s ATF - an async task framework", + "SiteName": null +}
\ No newline at end of file diff --git a/test/test-pages/dropbox-blog/expected.html b/test/test-pages/dropbox-blog/expected.html new file mode 100644 index 0000000..e15acfa --- /dev/null +++ b/test/test-pages/dropbox-blog/expected.html @@ -0,0 +1,527 @@ +<div> + <div> + <p> + I joined Dropbox not long after graduating with a Master’s degree in computer science. Aside from an internship, this was my first big-league engineering job. My team had already begun designing a critical internal service that most of our software would use: It would handle asynchronous computing requests behind the scenes, powering everything from dragging a file into a Dropbox folder to scheduling a marketing campaign. + </p> + <p> + This Asynchronous Task Framework (ATF) would replace multiple bespoke async systems used by different engineering teams. It would reduce redundant development, incompatibilities, and reliance on legacy software. There were no open-source projects or buy-not-build solutions that worked well for our use case and scale, so we had to create our own. ATF is both an important and interesting challenge, though, so we were happy to design, build and deploy our own in-house service. + </p> + <p> + ATF not only had to work well, it had to work well at scale: It would be a foundational building block of Dropbox infrastructure. It would need to handle 10,000 async tasks per second from the start, and be architected for future growth. It would need to support nearly 100 unique async task types from the start, again with room to grow. There were at least two dozen engineering teams that would want to use it for entirely different parts of our codebase, for many products and services. + </p> + <p> + As any engineer would, we Googled to see what other companies with mega-scale services had done to handle async tasks. We were disappointed to find little material published by engineers who built supersized async services. + </p> + <p> + Now that ATF is deployed and currently serving 9,000 async tasks scheduled per second and in use by 28 engineering teams internally, we’re glad to fill that information gap. We’ve documented Dropbox ATF thoroughly, as a reference and guide for the engineering community seeking their own async solutions. + </p> + </div> + <div> + <p id="introduction"> + <h2> + Introduction + </h2> + </p> + </div> + <div> + <p> + Scheduling asynchronous tasks on-demand is a critical capability that powers many features and internal platforms at Dropbox. Async Task Framework (ATF) is the infrastructural system that supports this capability at Dropbox through a callback-based architecture. ATF enables developers to define callbacks, and schedule tasks that execute against these pre-defined callbacks. + </p> + <p> + Since its introduction over a year ago, ATF has gone on to become an important building block in the Dropbox infrastructure, used by nearly 30 internal teams across our codebase. It currently supports 100+ use cases which require either immediate or delayed task scheduling. + </p> + </div> + <div> + <p id="glossary"> + <h2> + Glossary + </h2> + </p> + </div> + <div> + <p> + Some basic terms repeatedly used in this post, defined as used in the context of this discussion. + </p> + <p> + <b>Lambda:</b> A callback implementing business logic. + </p> + <p> + <span><b>Task:</b> Unit of execution of a lambda. Each asynchronous job scheduled with ATF is a task.</span> + </p> + <p> + <span><b>Collection:</b> A labeled subset of tasks belonging to a lambda. If <span>send email</span> is implemented as a lambda, then <span>password reset email</span> and <span>marketing email</span> would be collections.</span> + </p> + <p> + <span><b> Priority:</b> Labels defining priority of execution of tasks within a lambda. </span> + </p> + </div> + <div> + <p id="features"> + <h2> + Features + </h2> + </p> + </div> + <div> + <p> + <b>Task scheduling</b><br> + Clients can schedule tasks to execute at a specified time. Tasks can be scheduled for immediate execution, or delayed to fit the use case. + </p> + <p> + <b>Priority based execution</b><br> + Tasks should be associated with a priority. Tasks with higher priority should get executed before tasks with a lower priority once they are ready for execution. + </p> + <p> + <b>Task gating</b><br> + ATF enables the the gating of tasks based on lambda, or a subset of tasks on a lambda based on collection. Tasks can be gated to be completely dropped or paused until a suitable time for execution. + </p> + <p> + <b>Track task status</b><br> + Clients can query the status of a scheduled task. + </p> + </div> + <div> + <p id="system-guarantees"> + <h2> + System guarantees + </h2> + </p> + </div> + <div> + <p> + <b>At-least once task execution<br></b> The ATF system guarantees that a task is executed at least once after being scheduled. Execution is said to be complete once the user-defined callback signals task completion to the ATF system. + </p> + <p> + <b>No concurrent task execution<br></b> The ATF system guarantees that at most one instance of a task will be actively executing at any given in point. This helps users write their callbacks without designing for concurrent execution of the same task from different locations. + </p> + <p> + <b>Isolation<br></b> Tasks in a given lambda are isolated from the tasks in other lambdas. This isolation spans across several dimensions, including worker capacity for task execution and resource use for task scheduling. Tasks on the same lambda but different priority levels are also isolated in their resource use for task scheduling. + </p> + <p> + <b>Delivery latency<br></b> 95% of tasks begin execution within five seconds from their scheduled execution time. + </p> + <p> + <b>High availability for task scheduling<br></b> The ATF service is 99.9% available to accept task scheduling requests from any client. + </p> + </div> + <div> + <p id="-lambda-requirements"> + <h2> + Lambda requirements + </h2> + </p> + </div> + <div> + <p> + Following are some restrictions we place on the callback logic (lambda): + </p> + <p> + <b>Idempotence</b><br> + A single task on a lambda can be executed multiple times within the ATF system. Developers should ensure that their lambda logic and correctness of task execution in clients are not affected by this. + </p> + <p> + <b>Resiliency</b><br> + Worker processes which execute tasks might die at any point during task execution. ATF retries abruptly interrupted tasks, which could also be retried on different hosts. Lambda owners must design their lambdas such that retries on different hosts do not affect lambda correctness. + </p> + <p> + <b>Terminal state handling<br></b> ATF retries tasks until they are signaled to be complete from the lambda logic. Client code can mark a task as successfully completed, fatally terminated, or retriable. It is critical that lambda owners design clients to signal task completion appropriately to avoid misbehavior such as infinite retries. + </p> + </div> + <div> + <p id="architecture"> + <h2> + Architecture + </h2> + </p> + </div> + <div> + <figure> + <img src="http://fakehost/cms/content/dam/dropbox/tech-blog/en-us/2020/11/atf/diagrams/Techblog-ATF-720x844px-1.png" aria-hidden="false" alt="Async Task Framework (ATF) [Fig 1]" height="1688" width="1440"> + <figcaption> + Async Task Framework (ATF) [Fig 1] + </figcaption> + </figure> + </div> + <div> + <p> + In this section, we describe the high-level architecture of ATF and give brief description of its different components. (See Fig. 1 above.) In this section, we describe the high-level architecture of ATF and give brief description of its different components. (See Fig. 1 above.) Dropbox <a href="https://dropbox.tech/infrastructure/courier-dropbox-migration-to-grpc">uses gRPC</a> for remote calls and our in-house <a href="https://dropbox.tech/infrastructure/reintroducing-edgestore">Edgestore</a> to store tasks. + </p> + <p> + ATF consists of the following components: + </p> + <ul> + <li>Frontend + </li> + <li>Task Store + </li> + <li>Store Consumer + </li> + <li>Queue + </li> + <li>Controller + </li> + <li>Executor + </li> + <li>Heartbeat and Status Controller (HSC)<span><br></span> + </li> + </ul> + <p> + <span><b>Frontend</b><br> + This is the service that schedules requests via an RPC interface. The frontend accepts RPC requests from clients and schedules tasks by interacting with ATF’s task store described below.</span><br> + </p> + <p> + <b>Task Store<br></b> ATF tasks are stored in and triggered from the task store. The task store could be any generic data store with indexed querying capability. In ATF’s case, We use our in-house metadata store Edgestore to power the task store. More details can be found in the <a href="https://paper.dropbox.com/doc/How-we-designed-Dropboxs-ATF-an-async-task-framework--A~wmq5aW48OkHns4LzkM~o6zAg-cf95JuxevqilF2iWWATj6#:uid=395988446153757833740421&h2=Data-model">D</a><a href="https://paper.dropbox.com/doc/How-we-designed-Dropboxs-ATF-an-async-task-framework--A~wmq5aW48OkHns4LzkM~o6zAg-cf95JuxevqilF2iWWATj6#:uid=395988446153757833740421&h2=Data-model">ata</a> <a href="https://paper.dropbox.com/doc/How-we-designed-Dropboxs-ATF-an-async-task-framework--A~wmq5aW48OkHns4LzkM~o6zAg-cf95JuxevqilF2iWWATj6#:uid=395988446153757833740421&h2=Data-model">M</a><a href="https://paper.dropbox.com/doc/How-we-designed-Dropboxs-ATF-an-async-task-framework--A~wmq5aW48OkHns4LzkM~o6zAg-cf95JuxevqilF2iWWATj6#:uid=395988446153757833740421&h2=Data-model">odel</a> section below. + </p> + <p> + <b>Store Consumer<br></b> The Store Consumer is a service that periodically polls the task store to find tasks that are ready for execution and pushes them onto the right queues, as described in the queue section below. These could be tasks that are newly ready for execution, or older tasks that are ready for execution again because they either failed in a retriable way on execution, or were dropped elsewhere within the ATF system. + </p> + <p> + Below is a simple walkthrough of the Store Consumer’s function: + </p> + </div> + <div> + <pre><code>repeat every second: + 1. poll tasks ready for execution from task store + 2. push tasks onto the right queues + 3. update task statuses</code></pre> + </div> + <div> + <p> + The Store Consumer polls tasks that failed in earlier execution attempts. This helps with the at-least-once guarantee that the ATF system provides. More details on how the Store Consumer polls new and previously failed tasks is presented in the <a href="https://paper.dropbox.com/doc/How-we-designed-Dropboxs-ATF-an-async-task-framework--A~wmq5aW48OkHns4LzkM~o6zAg-cf95JuxevqilF2iWWATj6#:uid=342792671048375002388848&h2=Lifecycle-of-a-task">Lifecycle of a task</a> section below. + </p> + <p> + <b>Queue<br></b> ATF uses AWS <a href="https://aws.amazon.com/sqs/">Simple Queue Service</a> (SQS) to queue tasks internally. These queues act as a buffer between the Store Consumer and Controllers (described below). Each <span><lambda, priority></span> pair gets a dedicated SQS queue. The total number of SQS queues used by ATF is <span>#lambdas x #priorities</span>. + </p> + <p> + <b>Controller<br></b> Worker hosts are physical hosts dedicated for task execution. Each worker host has one controller process responsible for polling tasks from SQS queues in a background thread, and then pushing them onto process local buffered queues. The Controller is only aware of the lambdas it is serving and thus polls only the limited set of necessary queues. + </p> + <p> + The Controller serves tasks from its process local queue as a response to <span>NextWork</span> RPCs. This is the layer where execution level task prioritization occurs. The Controller has different process level queues for tasks of different priorities and can thus prioritize tasks in response to <span>NextWork</span> RPCs. + </p> + <p> + <b>Executor<br></b> The Executor is a process with multiple threads, responsible for the actual task execution. Each thread within an Executor process follows this simple loop: + </p> + </div> + <div> + <pre><code>while True: + w = get_next_work() + do_work(w)</code></pre> + </div> + <div> + <p> + Each worker host has a single Controller process and multiple executor processes. Both the Controller and Executors work in a “pull” model, in which active loops continuously long-poll for new work to be done. + </p> + <p> + <b>Heartbeat and Status Controller (HSC)</b><br> + The HSC serves RPCs for claiming a task for execution (<span>ClaimTask</span>), setting task status after execution (<span>SetResults</span>) and heartbeats during task execution (<span>Heartbeat</span>). <span>ClaimTask</span> requests originate from the Controllers in response to <span>NextWork</span> requests. <span>Heartbeat</span> and <span>SetResults</span> requests originate from executor processes during and after task execution. The HSC interacts with the task store to update the task status on the kind of request it receives. + </p> + </div> + <div> + <p id="data-model"> + <h2> + Data model + </h2> + </p> + </div> + <div> + <p> + ATF uses our in-house metadata store, Edgestore, as a task store. Edgestore objects can be Entities or Associations (<span>assoc</span>), each of which can have user-defined attributes. Associations are used to represent relationships between entities. Edgestore supports indexing only on attributes of associations. + </p> + <p> + Based on this design, we have two kinds of ATF-related objects in Edgestore. The ATF association stores scheduling information, such as the next scheduled timestamp at which the Store Consumer should poll a given task (either for the first time or for a retry). The ATF entity stores all task related information that is used to track the task state and payload for task execution. We query on associations from the Store Consumer in a pull model to pick up tasks ready for execution. + </p> + </div> + <div> + <p id="lifecycle-of-a-task"> + <h2> + Lifecycle of a task + </h2> + </p> + </div> + <div> + <ol> + <li>Client performs a <span>Schedule</span> RPC call to <b>Frontend</b> with task information, including execution time. + </li> + <li>Frontend creates Edgestore <span>entity</span> and <span>assoc</span> for the task. + </li> + <li>When it is time to process the task, <b>Store Consumer</b> pulls the task from <b>Edgestore</b> and pushes it to a related <b>SQS</b> queue. + </li> + <li> + <b>Executor</b> makes <span>NextWork</span> RPC call to <b>Controller</b>, which pulls tasks from the <b>SQS</b> queue, makes a <span>ClaimTask</span> RPC to the HSC and then returns the task to the <b>Executor</b>. + </li> + <li> + <b>Executor</b> invokes the callback for the task. While processing, <b>Executor</b> performs <span>Heartbeat</span> RPC calls to <b>Heartbeat and Status Controller (HSC)</b>. Once processing is done, <b>Executor</b> performs <span>TaskStatus</span> RPC call to <b>HSC</b>. + </li> + <li>Upon getting <span>Heartbeat</span> and <span>TaskStatus</span> RPC calls, <b>HSC</b> updates the <b>Edgestore</b> entity and <span>assoc</span>. + </li> + </ol> + <p> + Every state update in the lifecycle of a task is accompanied by an update to the next trigger timestamp in the <span>assoc</span>. This ensures that the Store Consumer pulls the task again if there is no change in state of the task within the next trigger timestamp. This helps ATF achieve its at-least-once delivery guarantee by ensuring that no task is dropped. + </p> + <p> + Following are the task entity and association states in ATF and their corresponding timestamp updates: + </p> + <table readabilityDataTable="1"> + <tbody> + <tr> + <td> + <p> + <b>Entity status</b> + </p> + </td> + <td> + <p> + <b>Assoc status</b> + </p> + </td> + <td> + <p> + <b>next trigger timestamp in Assoc</b> + </p> + </td> + <td> + <p> + <b>Comment</b> + </p> + </td> + </tr> + <tr> + <td> + <p> + <span>new</span> + </p> + </td> + <td> + <p> + <span>new</span> + </p> + </td> + <td> + <p> + <span>scheduled_timestamp</span> of the task + </p> + </td> + <td> + <p> + Pick up new tasks that are ready. + </p> + </td> + </tr> + <tr> + <td> + <p> + <span>enqueued</span> + </p> + </td> + <td> + <p> + <span>started</span> + </p> + </td> + <td> + <p> + <span>enqueued_timestamp</span> + <span>enqueue_timeout</span> + </p> + </td> + <td> + <p> + Re-enqueue task if it has been in <span>enqueued</span> state for too long. This can happen if the queue loses data or the controller goes down after polling the queue and before the task is claimed. + </p> + </td> + </tr> + <tr> + <td> + <p> + <span>claimed</span> + </p> + </td> + <td> + <p> + <span>started</span> + </p> + </td> + <td> + <p> + <span>claimed_timestamp</span> + <span>claim_timeout</span> + </p> + </td> + <td> + <p> + Re-enqueue if task is claimed but never transfered to <span>processing</span>. This can happen if Controller is down after claiming a task. Task status is changed to <span>enqueued</span> after re-enqueue. + </p> + </td> + </tr> + <tr> + <td> + <p> + <span>processing</span> + </p> + </td> + <td> + <p> + <span>started</span> + </p> + </td> + <td> + <p> + <span>heartbeat_timestamp</span> + <span>heartbeat_timeout</span>` + </p> + </td> + <td> + <p> + Re-enqueue if task hasn’t sent <span>heartbeat</span> for too long. This can happen if Executor is down. Task status is changed to <span>enqueued</span> after re-enqueue. + </p> + </td> + </tr> + <tr> + <td> + <p> + <span>retriable failure</span> + </p> + </td> + <td> + <p> + started + </p> + </td> + <td> + <p> + compute <span>next_timestamp</span> according to backoff logic + </p> + </td> + <td> + <p> + Exponential backoff for tasks with retriable failure. + </p> + </td> + </tr> + <tr> + <td> + <p> + <span>success</span> + </p> + </td> + <td> + <p> + <span>completed</span> + </p> + </td> + <td> + <p> + N/A + </p> + </td> + <td> + + </td> + </tr> + <tr> + <td> + <p> + <span>fatal_failure</span> + </p> + </td> + <td> + <p> + <span>completed</span> + </p> + </td> + <td> + <p> + N/A + </p> + </td> + <td> + + </td> + </tr> + </tbody> + </table> + <p> + The store consumer polls for tasks based on the following query: + </p> + <p> + <span>assoc_status= && next_timestamp<=time.now()<br></span> + </p> + <p> + Below is the state machine that defines task state transitions: <br> + </p> + </div> + <div> + <figure> + <img src="http://fakehost/cms/content/dam/dropbox/tech-blog/en-us/2020/11/atf/diagrams/Techblog-ATF-720x225px-2.png" aria-hidden="false" alt="Task State Transitions [Fig 2]" height="450" width="1440"> + </figure> + </div> + <div> + <p id="-achieving-guarantees"> + <h2> + Achieving guarantees + </h2> + </p> + </div> + <div> + <p> + <b>At-least-once task execution<br></b> At-least-once execution is guaranteed in ATF by retrying a task until it completes execution (which is signaled by a <span>Success</span> or a <span>FatalFailure</span> state). All ATF system errors are implicitly considered retriable failures, and lambda owners have an option of marking tasks with a <span>RetriableFailure</span> state. Tasks might be dropped from the ATF execution pipeline in different parts of the system through transient RPC failures and failures on dependencies like Edgestore or SQS. These transient failures at different parts of the system do not affect the at-least-once guarantee, though, because of the system of timeouts and re-polling from Store Consumer. + </p> + <p> + <b>No concurrent task execution<br></b> Concurrent task execution is avoided through a combination of two methods in ATF. First, tasks are explicitly claimed through an exclusive task state (<span>Claimed</span>) before starting execution. Once the task execution is complete, the task status is updated to one of <span>Success</span>, <span>FatalFailure</span> or <span>RetriableFailure</span>. A task can be claimed only if its existing task state is <span>Enqueued</span> (retried tasks go to the <span>Enqueued</span> state as well once they are re-pushed onto SQS). + </p> + <p> + However, there might be situations where once a long running task starts execution, its heartbeats might fail repeatedly yet the task execution continues. ATF would retry this task by polling it from the store consumer because the heartbeat timeouts would’ve expired. This task can then be claimed by another worker and lead to concurrent execution. <br> + </p> + <p> + To avoid this situation, there is a termination logic in the Executor processes whereby an Executor process terminates itself as soon as three consecutive heartbeat calls fail. Each heartbeat timeout is large enough to eclipse three consecutive heartbeat failures. This ensures that the Store Consumer cannot pull such tasks before the termination logic ends them—the second method that helps achieve this guarantee. + </p> + <p> + <b>Isolation<br></b> Isolation of lambdas is achieved through dedicated worker clusters, dedicated queues, and dedicated per-lambda scheduling quotas. In addition, isolation across different priorities within the same lambda is likewise achieved through dedicated queues and scheduling bandwidth. + </p> + <p> + <b>Delivery latency<br></b> ATF use cases do not require ultra-low task delivery latencies. Task delivery latencies on the order of a couple of seconds are acceptable. Tasks ready for execution are periodically polled by the Store Consumer and this period of polling largely controls the task delivery latency. Using this as a tuning lever, ATF can achieve different delivery latencies as required. Increasing poll frequency reduces task delivery latency and vice versa. Currently, we have calibrated ATF to poll for ready tasks once every two seconds. + </p> + </div> + <div> + <p id="ownership-model"> + <h2> + Ownership model + </h2> + </p> + </div> + <p> + ATF is designed to be a self-serve framework for developers at Dropbox. The design is very intentional in driving an ownership model where lambda owners own all aspects of their lambdas’ operations. To promote this, all lambda worker clusters are owned by the lambda owners. They have full control over operations on these clusters, including code deployments and capacity management. Each executor process is bound to one lambda. Owners have the option of deploying multiple lambdas on their worker clusters simply by spawning new executor processes on their hosts. + </p> + <div> + <p id="-extending-atf"> + <h2> + Extending ATF + </h2> + </p> + </div> + <div> + <p> + As described above, ATF provides an infrastructural building block for scheduling asynchronous tasks. With this foundation established, ATF can be extended to support more generic use cases and provide more features as a framework. Following are some examples of what could be built as an extension to ATF. + </p> + <p> + <b>Periodic task execution<br></b> Currently, ATF is a system for one-time task scheduling. Building support for periodic task execution as an extension to this framework would be useful in unlocking new capabilities for our clients. + </p> + <p> + <b>Better support for task chaining<br></b> Currently, it is possible to chain tasks on ATF by scheduling a task onto ATF that then schedules other tasks onto ATF during its execution. Although it is possible to do this in the current ATF setup, visibility and control on this chaining is absent at the framework level. Another natural extension here would be to better support task chaining through framework-level visibility and control, to make this use case a first class concept in the ATF model. + </p> + <p> + <b>Dead letter queues for misbehaving tasks<br></b> One common source of maintenance overhead we observe on ATF is that some tasks get stuck in infinite retry loops due to occasional bugs in lambda logic. This requires manual intervention from the ATF framework owners in some cases where there are a large number of tasks stuck in such loops, occupying a lot of the scheduling bandwidth in the system. Typical manual actions in response to such a situation include pausing execution of the lambdas with misbehaving tasks, or dropping them outright. + </p> + <p> + One way to reduce this operational overhead and provide an easy interface for lambda owners to recover from such incidents would be to create dead letter queues filled with such misbehaving tasks. The ATF framework could impose a maximum number of retries before tasks are pushed onto the dead letter queue. We could create and expose tools that make it easy to reschedule tasks from the dead letter queue back into the ATF system, once the associated lambda bugs are fixed.<br> + </p> + </div> + <div> + <p id="conclusion"> + <h2> + Conclusion + </h2> + </p> + </div> + <p> + We hope this post helps engineers elsewhere to develop better async task frameworks of their own. Many thanks to everyone who worked on this project: Anirudh Jayakumar, Deepak Gupta, Dmitry Kopytkov, Koundinya Muppalla, Peng Kang, Rajiv Desai, Ryan Armstrong, Steve Rodrigues, Thomissa Comellas, Xiaonan Zhang and Yuhuan Du.<br> + + </p> + </div>
\ No newline at end of file diff --git a/test/test-pages/dropbox-blog/source.html b/test/test-pages/dropbox-blog/source.html new file mode 100644 index 0000000..83f2b05 --- /dev/null +++ b/test/test-pages/dropbox-blog/source.html @@ -0,0 +1,868 @@ +<!DOCTYPE html> +<html lang="en" xml:lang="en" data-cms-lang="en-us" xmlns="http://www.w3.org/1999/xhtml"> + <head> + <meta charset="utf-8" /> + <title> + How we designed Dropbox ATF: an async task framework - Dropbox + </title> + <meta name="data-tags" content="Async,Edgestore,Infrastructure,Task Scheduling" /> + <meta name="data-tagTaxonomy" content="Async; Edgestore; Infrastructure; Task Scheduling;" /> + <meta name="page-id" content="infrastructure-asynchronous-task-scheduling-at-dropbox" /> + <meta name="topic" content="Infrastructure" /> + <meta name="publishDate" content="2020-11-11 12:00:00.000-0600" /> + <meta name="author" content="Arun Sai Krishnan" /> + <link rel="canonical" href="https://dropbox.tech/infrastructure/asynchronous-task-scheduling-at-dropbox" /> + <link rel="icon" href="https://cfl.dropboxstatic.com/static/images/favicon-vflUeLeeY.ico" type="image/x-icon" /> + <meta property="og:url" content="https://dropbox.tech/infrastructure/asynchronous-task-scheduling-at-dropbox" /> + <meta property="og:type" content="article" /> + <meta property="og:title" content="How we designed Dropbox’s ATF - an async task framework" /> + <meta property="og:image" content="https://aem.dropbox.com/cms/content/dam/dropbox/tech-blog/en-us/2020/11/atf/diagrams/Techblog-ATF-Social.png" /> + <meta name="twitter:card" content="summary_large_image" /> + <meta content="width=device-width,initial-scale=1.0,user-scalable=no" name="viewport" /> + <link rel="alternate" hreflang="en-us" href="https://dropbox.tech/infrastructure/asynchronous-task-scheduling-at-dropbox" /><!-- /* Enable rebrand styles */ +<sly data-sly-use.inheritUtil="com.dropbox.aem.common.models.utils.InheritanceUtilUse" + data-sly-test.pageStyle="" /> --> + <link rel="stylesheet" href="/cms/etc.clientlibs/settings/wcm/designs/dropbox-common/clientlib-cms-common.757d73acbd22d3e2bf4eeb953c16c4d5.css" type="text/css" /> + <link rel="stylesheet" href="/cms/etc.clientlibs/settings/wcm/designs/dropbox-tech-blog/clientlib-all.fc2ae6db413129b3901dd5a89e64f347.css" type="text/css" /><!--Knotch Integration should be added in header--> + + <script src="https://www.knotch-cdn.com/unit/latest/knotch.min.js" data-account="33c0d4ac-b5bc-4168-a95b-e963ec65974d" async="async"></script> + <link rel="stylesheet" href="/cms/etc.clientlibs/settings/wcm/designs/dropbox-tech-blog/clientlib-article-content.22c503f9a8a000fceab6a403af5ce96f.css" type="text/css" /> + <style> + <![CDATA[ + body.stormcrow-animate{opacity:1;} + ]]> + </style> + </head> + <body class="tech-blog-article-page__page stormcrow-animate" data-article-uuid="d4052e45-cbcb-4ebb-b834-eb377ab543e8"> + <input type="hidden" id="wcmRunmode" name="wcmRunmode" value="publish,prod" /> + <script type="text/javascript"> + //<![CDATA[ + + var attr = "tealium_event$cms".split(","); + + var utag_data = {} + attr.forEach(function (item) { + if (item && item.indexOf("$") > -1) + utag_data[item.split("$")[0]] = item.split("$")[1]; + }) + //]]> + </script> + <script type="text/javascript"> + //<![CDATA[ + + (function (a, b, c, d) { + a = "\/\/tags.tiqcdn.com\/utag\/dropbox\/tech\u002Dblog\/prod\/utag.js"; + b = document; + c = 'script'; + d = b.createElement(c); + d.src = a; + d.type = 'text/java' + c; + d.async = true; + a = b.getElementsByTagName(c)[0]; + a.parentNode.insertBefore(d, a); + })(); + //]]> + </script> + <header class="dr-header"> + <div class="dr-header__sticky-container"> + <section class="dr-header__section dr-flex dr-flex--align-center dr-padding-right-40 dr-padding-left-40 dr-header__sticky-content-container dr-header__sticky-content-container--opened dr-container--surface"> + <div class="dr-flex-1"> + <a class="dr-link dr-link--no-underline dr-link--no-underline-hover dr-typography-t1" href="https://dropbox.tech/">Dropbox.Tech</a> + </div><button class="dr-header__item dr-nav__menu-toggle-button dr-button dr-hide-from-md dr-flex dr-flex-align-center dr-flex-justify-center" data-dr-tooltip="Menu" data-dr-tooltip-theme="white"><svg viewbox="0 0 28 28" fill="none" xmlns="http://www.w3.org/2000/svg" class="dr-width-100 dr-height-100"> + <rect x="0.501831" y="8" width="28" height="2" fill="white"></rect> + <rect x="0.500977" y="18" width="28" height="2" fill="white"></rect></svg></button> + <nav class="dr-show-block-from-md dr-nav__nav"> + <button class="dr-button dr-nav__menu-close-button"><svg width="20" height="20" viewbox="0 0 20 20" fill="none" xmlns="http://www.w3.org/2000/svg"> + <path fill-rule="evenodd" clip-rule="evenodd" d="M19.2875 2.15983L17.6683 0.566406L9.82597 8.28403L2.33211 0.909344L0.71294 2.50277L8.2068 9.87746L0.666992 17.2974L2.28617 18.8908L9.82597 11.4709L17.7143 19.2337L19.3334 17.6403L11.4451 9.87746L19.2875 2.15983Z" fill="white"></path></svg></button> + <ul class="dr-unstyled-list dr-typography-t2 dr-flex dr-nav__nav-list"> + <li class="dr-header__item dr-position-relative dr-header__item--with-subnav dr-header__list-item"> + <button class="dr-button dr-button--link dr-header__link--with-subnav">Topics</button> + <ul class="dr-unstyled-list dr-display-none dr-header__subnav dr-position-absolute dr-container--surface dr-padding-top-30 dr-padding-left-40 dr-padding-bottom-20 dr-padding-right-40 dr-font-weight-500"> + <li class="dr-header__list-item dr-header__list-item--subnav"> + <a href="https://dropbox.tech/application" class="dr-display-block dr-link dr-link--no-underline dr-container--application dr-link--primary">Application</a> + </li> + <li class="dr-header__list-item dr-header__list-item--subnav"> + <a href="https://dropbox.tech/frontend" class="dr-display-block dr-link dr-link--no-underline dr-container--frontend dr-link--primary">Front End</a> + </li> + <li class="dr-header__list-item dr-header__list-item--subnav"> + <a href="https://dropbox.tech/infrastructure" class="dr-display-block dr-link dr-link--no-underline dr-container--infrastructure dr-link--primary">Infrastructure</a> + </li> + <li class="dr-header__list-item dr-header__list-item--subnav"> + <a href="https://dropbox.tech/machine-learning" class="dr-display-block dr-link dr-link--no-underline dr-container--machine-learning dr-link--primary">Machine Learning</a> + </li> + <li class="dr-header__list-item dr-header__list-item--subnav"> + <a href="https://dropbox.tech/mobile" class="dr-display-block dr-link dr-link--no-underline dr-container--mobile dr-link--primary">Mobile</a> + </li> + <li class="dr-header__list-item dr-header__list-item--subnav"> + <a href="https://dropbox.tech/security" class="dr-display-block dr-link dr-link--no-underline dr-container--security dr-link--primary">Security</a> + </li> + </ul> + </li> + <li class="dr-header__item dr-header__list-item"> + <a href="https://dropbox.tech/developers" class="dr-link dr-link--no-underline dr-nav__main-category">Developers</a> + </li> + <li class="dr-header__item dr-header__list-item"> + <a class="dr-link dr-link--no-underline dr-header__link" href="http://dropbox.com/jobs" target="_blank">Jobs</a> + </li> + </ul> + </nav><button data-dark-mode-switcher="" class="dr-header__item dr-header__dark-mode-switcher dr-button dr-button--link dr-cursor-pointer" data-dr-tooltip="Dark Mode" data-dr-tooltip-theme="white" type="button"><img alt="" height="30" src="/cms/etc.clientlibs/settings/wcm/designs/dropbox-tech-blog/clientlib-all/resources/button_dark-mode-new.svg" width="30" class="dr-header__mode-image" /></button> <button class="dr-header__item dr-header__search-button dr-button dr-button--link dr-cursor-pointer" data-dr-tooltip="Search" data-dr-tooltip-theme="white" type="button"><img alt="" height="20" src="/cms/etc.clientlibs/settings/wcm/designs/dropbox-tech-blog/clientlib-all/resources/button_search-new.svg" width="20" /></button> <!--search-result-page-only--> + <!--search-result-page-only--> + <div class="dr-header__search dr-display-none"> + <button class="dr-header__search-close-button dr-header__item dr-button dr-button--link dr-cursor-pointer" type="button"><svg width="20" height="20" viewbox="0 0 20 20" fill="none" xmlns="http://www.w3.org/2000/svg"> + <path fill-rule="evenodd" clip-rule="evenodd" d="M19.2875 2.15983L17.6683 0.566406L9.82597 8.28403L2.33211 0.909344L0.71294 2.50277L8.2068 9.87746L0.666992 17.2974L2.28617 18.8908L9.82597 11.4709L17.7143 19.2337L19.3334 17.6403L11.4451 9.87746L19.2875 2.15983Z" fill="white"></path></svg></button> + <div class="dr-header__search-form-container"> + <form action="https://dropbox.tech/search-results.html" class="dr-header__search-form dr-container__content dr-width-100"> + <input autocomplete="off" class="dr-header__search-input dr-typography-t3" name="q" placeholder="Search" required="true" type="text" /> + <p class="dr-header__search-hint dr-margin-top-30 dr-margin-bottom-0 dr-typography-t5 dr-display-none"> + // Press enter to search + </p> + </form> + </div> + </div> + </section> + </div> + </header> + <div class="dr-article-hero"> + <div class="dr-article-hero__background-container dr-container--infrastructure"> + <picture class="dr-article-hero__background dr-article-hero__background--regular"><source media="( max-width: 375px )" srcset="/content/dam/dropbox/tech-blog/en-us/2020/11/atf/header/Infrastructure-ATF-375x150-light.png" /> <source media="( max-width: 376px ) and ( max-width: 1199px )" srcset="/content/dam/dropbox/tech-blog/en-us/2020/11/atf/header/Infrastructure-ATF-1024x250-light.png" /> <img class="dr-article-hero__background-image" src="/cms/content/dam/dropbox/tech-blog/en-us/2020/11/atf/header/Infrastructure-ATF-1440x305-light.png" alt="" /></picture> <picture class="dr-article-hero__background dr-article-hero__background--dark"><source media="( max-width: 375px )" srcset="/content/dam/dropbox/tech-blog/en-us/2020/11/atf/header/Infrastructure-ATF-375x150-dark.png" /> <source media="( max-width: 376px ) and ( max-width: 1199px )" srcset="/content/dam/dropbox/tech-blog/en-us/2020/11/atf/header/Infrastructure-ATF-1024x250-dark.png" /> <img class="dr-article-hero__background-image" src="/cms/content/dam/dropbox/tech-blog/en-us/2020/11/atf/header/Infrastructure-ATF-1440x305-dark.png" alt="" /></picture> + </div> + <section class="dr-container__content"> + <h1 class="dr-display-inline dr-typography-t15 dr-container--surface dr-article-hero__title"> + <span class="dr-article-hero__title-container dr-container--infrastructure">How we designed Dropbox ATF: an async task framework</span> + </h1> + <div class="dr-typography-no-space dr-margin-top-10 dr-margin-md-top-20"> + <span class="dr-typography-t5">// By Arun Sai Krishnan • Nov 11, 2020</span> + </div> + </section> + </div> + <div class="dr-article-content"> + <div class="dr-article-content__scroll-tracker-container dr-container--infrastructure"> + <div class="dr-article-content__scroll-tracker"></div> + </div> + <div class="dr-container__content"> + <div class="dr-article-content__content-container dr-padding-md-left-80 dr-padding-md-right-80 dr-typography-t12"> + <nav class="dr-article-content__side-nav dr-article-content__side-nav--initial dr-typography-t5"> + <ol class="dr-article-content__side-nav-list dr-margin-0"> + <li class="dr-article-content__side-nav-list-item dr-margin-bottom-5"> + <a href="#introduction" class="dr-link dr-link--no-underline dr-article-content__side-nav-link">Introduction</a> + </li> + <li class="dr-article-content__side-nav-list-item dr-margin-bottom-5"> + <a href="#glossary" class="dr-link dr-link--no-underline dr-article-content__side-nav-link">Glossary</a> + </li> + <li class="dr-article-content__side-nav-list-item dr-margin-bottom-5"> + <a href="#features" class="dr-link dr-link--no-underline dr-article-content__side-nav-link">Features</a> + </li> + <li class="dr-article-content__side-nav-list-item dr-margin-bottom-5"> + <a href="#system-guarantees" class="dr-link dr-link--no-underline dr-article-content__side-nav-link">System guarantees</a> + </li> + <li class="dr-article-content__side-nav-list-item dr-margin-bottom-5"> + <a href="#-lambda-requirements" class="dr-link dr-link--no-underline dr-article-content__side-nav-link">Lambda requirements</a> + </li> + <li class="dr-article-content__side-nav-list-item dr-margin-bottom-5"> + <a href="#architecture" class="dr-link dr-link--no-underline dr-article-content__side-nav-link">Architecture</a> + </li> + <li class="dr-article-content__side-nav-list-item dr-margin-bottom-5"> + <a href="#data-model" class="dr-link dr-link--no-underline dr-article-content__side-nav-link">Data model</a> + </li> + <li class="dr-article-content__side-nav-list-item dr-margin-bottom-5"> + <a href="#lifecycle-of-a-task" class="dr-link dr-link--no-underline dr-article-content__side-nav-link">Lifecycle of a task</a> + </li> + <li class="dr-article-content__side-nav-list-item dr-margin-bottom-5"> + <a href="#-achieving-guarantees" class="dr-link dr-link--no-underline dr-article-content__side-nav-link">Achieving guarantees</a> + </li> + <li class="dr-article-content__side-nav-list-item dr-margin-bottom-5"> + <a href="#ownership-model" class="dr-link dr-link--no-underline dr-article-content__side-nav-link">Ownership model</a> + </li> + <li class="dr-article-content__side-nav-list-item dr-margin-bottom-5"> + <a href="#-extending-atf" class="dr-link dr-link--no-underline dr-article-content__side-nav-link">Extending ATF</a> + </li> + <li class="dr-article-content__side-nav-list-item dr-margin-bottom-5"> + <a href="#conclusion" class="dr-link dr-link--no-underline dr-article-content__side-nav-link">Conclusion</a> + </li> + </ol> + </nav> + <div class="dr-article-content__content"> + <div class="aem-Grid aem-Grid--12 aem-Grid--default--12"> + <div class="text parbase aem-GridColumn aem-GridColumn--default--12"> + <p> + I joined Dropbox not long after graduating with a Master’s degree in computer science. Aside from an internship, this was my first big-league engineering job. My team had already begun designing a critical internal service that most of our software would use: It would handle asynchronous computing requests behind the scenes, powering everything from dragging a file into a Dropbox folder to scheduling a marketing campaign. + </p> + <p> + This Asynchronous Task Framework (ATF) would replace multiple bespoke async systems used by different engineering teams. It would reduce redundant development, incompatibilities, and reliance on legacy software. There were no open-source projects or buy-not-build solutions that worked well for our use case and scale, so we had to create our own. ATF is both an important and interesting challenge, though, so we were happy to design, build and deploy our own in-house service. + </p> + <p> + ATF not only had to work well, it had to work well at scale: It would be a foundational building block of Dropbox infrastructure. It would need to handle 10,000 async tasks per second from the start, and be architected for future growth. It would need to support nearly 100 unique async task types from the start, again with room to grow. There were at least two dozen engineering teams that would want to use it for entirely different parts of our codebase, for many products and services.  + </p> + <p> + As any engineer would, we Googled to see what other companies with mega-scale services had done to handle async tasks. We were disappointed to find little material published by engineers who built supersized async services. + </p> + <p> + Now that ATF is deployed and currently serving 9,000 async tasks scheduled per second and in use by 28 engineering teams internally, we’re glad to fill that information gap. We’ve documented Dropbox ATF thoroughly, as a reference and guide for the engineering community seeking their own async solutions. + </p> + </div> + <div class="section aem-GridColumn aem-GridColumn--default--12"> + <div class="dr-article-content__section" id="introduction"> + <h2 class="dr-article-content__section-title"> + Introduction + </h2> + </div> + </div> + <div class="text parbase aem-GridColumn aem-GridColumn--default--12"> + <p> + Scheduling asynchronous tasks on-demand is a critical capability that powers many features and internal platforms at Dropbox. Async Task Framework (ATF) is the infrastructural system that supports this capability at Dropbox through a callback-based architecture. ATF enables developers to define callbacks, and schedule tasks that execute against these pre-defined callbacks. + </p> + <p> + Since its introduction over a year ago, ATF has gone on to become an important building block in the Dropbox infrastructure, used by nearly 30 internal teams across our codebase. It currently supports 100+ use cases which require either immediate or delayed task scheduling.  + </p> + </div> + <div class="section aem-GridColumn aem-GridColumn--default--12"> + <div class="dr-article-content__section" id="glossary"> + <h2 class="dr-article-content__section-title"> + Glossary + </h2> + </div> + </div> + <div class="text parbase aem-GridColumn aem-GridColumn--default--12"> + <p> + Some basic terms repeatedly used in this post, defined as used in the context of this discussion. + </p> + <p> + <b>Lambda:</b> A callback implementing business logic. + </p> + <p> + <span><b>Task:</b> Unit of execution of a lambda. Each asynchronous job scheduled with ATF is a task.</span> + </p> + <p> + <span><b>Collection:</b> A labeled subset of tasks belonging to a lambda. If <span class="dr-code">send email</span> is implemented as a lambda, then <span class="dr-code">password reset email</span> and <span class="dr-code">marketing email</span> would be collections.</span> + </p> + <p> + <span><b> Priority:</b> Labels defining priority of execution of tasks within a lambda. </span> + </p> + </div> + <div class="section aem-GridColumn aem-GridColumn--default--12"> + <div class="dr-article-content__section" id="features"> + <h2 class="dr-article-content__section-title"> + Features + </h2> + </div> + </div> + <div class="text parbase aem-GridColumn aem-GridColumn--default--12"> + <p> + <b>Task scheduling</b><br /> + Clients can schedule tasks to execute at a specified time. Tasks can be scheduled for immediate execution, or delayed to fit the use case. + </p> + <p> + <b>Priority based execution</b><br /> + Tasks should be associated with a priority. Tasks with higher priority should get executed before tasks with a lower priority once they are ready for execution. + </p> + <p> + <b>Task gating</b><br /> + ATF enables the the gating of tasks based on lambda, or a subset of tasks on a lambda based on collection. Tasks can be gated to be completely dropped or paused until a suitable time for execution. + </p> + <p> + <b>Track task status</b><br /> + Clients can query the status of a scheduled task. + </p> + </div> + <div class="section aem-GridColumn aem-GridColumn--default--12"> + <div class="dr-article-content__section" id="system-guarantees"> + <h2 class="dr-article-content__section-title"> + System guarantees + </h2> + </div> + </div> + <div class="text parbase aem-GridColumn aem-GridColumn--default--12"> + <p> + <b>At-least once task execution<br /></b> The ATF system guarantees that a task is executed at least once after being scheduled. Execution is said to be complete once the user-defined callback signals task completion to the ATF system. + </p> + <p> + <b>No concurrent task execution<br /></b> The ATF system guarantees that at most one instance of a task will be actively executing at any given in point. This helps users write their callbacks without designing for concurrent execution of the same task from different locations. + </p> + <p> + <b>Isolation<br /></b> Tasks in a given lambda are isolated from the tasks in other lambdas. This isolation spans across several dimensions, including worker capacity for task execution and resource use for task scheduling. Tasks on the same lambda but different priority levels are also isolated in their resource use for task scheduling. + </p> + <p> + <b>Delivery latency<br /></b> 95% of tasks begin execution within five seconds from their scheduled execution time. + </p> + <p> + <b>High availability for task scheduling<br /></b> The ATF service is 99.9% available to accept task scheduling requests from any client. + </p> + </div> + <div class="section aem-GridColumn aem-GridColumn--default--12"> + <div class="dr-article-content__section" id="-lambda-requirements"> + <h2 class="dr-article-content__section-title"> + Lambda requirements + </h2> + </div> + </div> + <div class="text parbase aem-GridColumn aem-GridColumn--default--12"> + <p> + Following are some restrictions we place on the callback logic (lambda): + </p> + <p> + <b>Idempotence</b><br /> + A single task on a lambda can be executed multiple times within the ATF system. Developers should ensure that their lambda logic and correctness of task execution in clients are not affected by this. + </p> + <p> + <b>Resiliency</b><br /> + Worker processes which execute tasks might die at any point during task execution. ATF retries abruptly interrupted tasks, which could also be retried on different hosts. Lambda owners must design their lambdas such that retries on different hosts do not affect lambda correctness. + </p> + <p> + <b>Terminal state handling<br /></b> ATF retries tasks until they are signaled to be complete from the lambda logic. Client code can mark a task as successfully completed, fatally terminated, or retriable. It is critical that lambda owners design clients to signal task completion appropriately to avoid misbehavior such as infinite retries.  + </p> + </div> + <div class="section aem-GridColumn aem-GridColumn--default--12"> + <div class="dr-article-content__section" id="architecture"> + <h2 class="dr-article-content__section-title"> + Architecture + </h2> + </div> + </div> + <div class="image c04-image aem-GridColumn aem-GridColumn--default--12"> + <div class="dr-image image cq-dd-image"> + <figure class="dr-margin-0 dr-display-inline-block"> + <img src="/cms/content/dam/dropbox/tech-blog/en-us/2020/11/atf/diagrams/Techblog-ATF-720x844px-1.png" aria-hidden="false" alt="Async Task Framework (ATF) [Fig 1]" height="1688" width="1440" /> + <figcaption class="dr-typography-t5 dr-color-ink-60"> + Async Task Framework (ATF) [Fig 1] + </figcaption> + </figure> + </div> + </div> + <div class="text parbase aem-GridColumn aem-GridColumn--default--12"> + <p> + In this section, we describe the high-level architecture of ATF and give brief description of its different components. (See Fig. 1 above.) In this section, we describe the high-level architecture of ATF and give brief description of its different components. (See Fig. 1 above.) Dropbox <a href="https://dropbox.tech/infrastructure/courier-dropbox-migration-to-grpc">uses gRPC</a> for remote calls and our in-house <a href="https://dropbox.tech/infrastructure/reintroducing-edgestore">Edgestore</a> to store tasks. + </p> + <p> + ATF consists of the following components:  + </p> + <ul> + <li>Frontend + </li> + <li>Task Store + </li> + <li>Store Consumer + </li> + <li>Queue + </li> + <li>Controller + </li> + <li>Executor + </li> + <li>Heartbeat and Status Controller (HSC)<span><br /></span> + </li> + </ul> + <p> + <span><b>Frontend</b><br /> + This is the service that schedules requests via an RPC interface. The frontend accepts RPC requests from clients and schedules tasks by interacting with ATF’s task store described below.</span><br /> + </p> + <p> + <b>Task Store<br /></b> ATF tasks are stored in and triggered from the task store. The task store could be any generic data store with indexed querying capability. In ATF’s case, We use our in-house metadata store Edgestore to power the task store. More details can be found in the <a href="https://paper.dropbox.com/doc/How-we-designed-Dropboxs-ATF-an-async-task-framework--A~wmq5aW48OkHns4LzkM~o6zAg-cf95JuxevqilF2iWWATj6#:uid=395988446153757833740421&h2=Data-model">D</a><a href="https://paper.dropbox.com/doc/How-we-designed-Dropboxs-ATF-an-async-task-framework--A~wmq5aW48OkHns4LzkM~o6zAg-cf95JuxevqilF2iWWATj6#:uid=395988446153757833740421&h2=Data-model">ata</a> <a href="https://paper.dropbox.com/doc/How-we-designed-Dropboxs-ATF-an-async-task-framework--A~wmq5aW48OkHns4LzkM~o6zAg-cf95JuxevqilF2iWWATj6#:uid=395988446153757833740421&h2=Data-model">M</a><a href="https://paper.dropbox.com/doc/How-we-designed-Dropboxs-ATF-an-async-task-framework--A~wmq5aW48OkHns4LzkM~o6zAg-cf95JuxevqilF2iWWATj6#:uid=395988446153757833740421&h2=Data-model">odel</a> section below. + </p> + <p> + <b>Store Consumer<br /></b> The Store Consumer is a service that periodically polls the task store to find tasks that are ready for execution and pushes them onto the right queues, as described in the queue section below. These could be tasks that are newly ready for execution, or older tasks that are ready for execution again because they either failed in a retriable way on execution, or were dropped elsewhere within the ATF system.  + </p> + <p> + Below is a simple walkthrough of the Store Consumer’s function:  + </p> + </div> + <div class="dr-code-container aem-GridColumn aem-GridColumn--default--12"> + <button class="dr-code-container__copy-button dr-button dr-typography-t17">Copy</button> + <pre class="dr-code-container__pre"><code class="dr-code-container__code dr-typography-t5">repeat every second: + 1. poll tasks ready for execution from task store + 2. push tasks onto the right queues + 3. update task statuses</code></pre> + </div> + <div class="text parbase aem-GridColumn aem-GridColumn--default--12"> + <p> + The Store Consumer polls tasks that failed in earlier execution attempts. This helps with the at-least-once guarantee that the ATF system provides. More details on how the Store Consumer polls new and previously failed tasks is presented in the <a href="https://paper.dropbox.com/doc/How-we-designed-Dropboxs-ATF-an-async-task-framework--A~wmq5aW48OkHns4LzkM~o6zAg-cf95JuxevqilF2iWWATj6#:uid=342792671048375002388848&h2=Lifecycle-of-a-task">Lifecycle of a task</a> section below. + </p> + <p> + <b>Queue<br /></b> ATF uses AWS <a href="https://aws.amazon.com/sqs/" style="background-color: rgb(255,255,255);">Simple Queue Service</a> (SQS) to queue tasks internally. These queues act as a buffer between the Store Consumer and Controllers (described below). Each <span class="dr-code"><lambda, priority></span>  pair gets a dedicated SQS queue. The total number of SQS queues used by ATF is <span class="dr-code">#lambdas x #priorities</span>. + </p> + <p> + <b>Controller<br /></b> Worker hosts are physical hosts dedicated for task execution. Each worker host has one controller process responsible for polling tasks from SQS queues in a background thread, and then pushing them onto process local buffered queues. The Controller is only aware of the lambdas it is serving and thus polls only the limited set of necessary queues.  + </p> + <p> + The Controller serves tasks from its process local queue as a response to <span class="dr-code">NextWork</span> RPCs. This is the layer where execution level task prioritization occurs. The Controller has different process level queues for tasks of different priorities and can thus prioritize tasks in response to <span class="dr-code">NextWork</span> RPCs. + </p> + <p> + <b>Executor<br /></b> The Executor is a process with multiple threads, responsible for the actual task execution. Each thread within an Executor process follows this simple loop: + </p> + </div> + <div class="dr-code-container aem-GridColumn aem-GridColumn--default--12"> + <button class="dr-code-container__copy-button dr-button dr-typography-t17">Copy</button> + <pre class="dr-code-container__pre"><code class="dr-code-container__code dr-typography-t5">while True: + w = get_next_work() + do_work(w)</code></pre> + </div> + <div class="text parbase aem-GridColumn aem-GridColumn--default--12"> + <p> + Each worker host has a single Controller process and multiple executor processes. Both the Controller and Executors work in a “pull” model, in which active loops continuously long-poll for new work to be done. + </p> + <p> + <b>Heartbeat and Status Controller (HSC)</b><br /> + The HSC serves RPCs for claiming a task for execution (<span class="dr-code">ClaimTask</span>), setting task status after execution (<span class="dr-code">SetResults</span>) and heartbeats during task execution (<span class="dr-code">Heartbeat</span>). <span class="dr-code">ClaimTask</span> requests originate from the Controllers in response to <span class="dr-code">NextWork</span> requests. <span class="dr-code">Heartbeat</span> and <span class="dr-code">SetResults</span> requests originate from executor processes during and after task execution. The HSC interacts with the task store to update the task status on the kind of request it receives. + </p> + </div> + <div class="section aem-GridColumn aem-GridColumn--default--12"> + <div class="dr-article-content__section" id="data-model"> + <h2 class="dr-article-content__section-title"> + Data model + </h2> + </div> + </div> + <div class="text parbase aem-GridColumn aem-GridColumn--default--12"> + <p> + ATF uses our in-house metadata store, Edgestore, as a task store. Edgestore objects can be Entities or Associations (<span class="dr-code">assoc</span>), each of which can have user-defined attributes. Associations are used to represent relationships between entities. Edgestore supports indexing only on attributes of associations. + </p> + <p> + Based on this design, we have two kinds of ATF-related objects in Edgestore. The ATF association stores scheduling information, such as the next scheduled timestamp at which the Store Consumer should poll a given task (either for the first time or for a retry). The ATF entity stores all task related information that is used to track the task state and payload for task execution. We query on associations from the Store Consumer in a pull model to pick up tasks ready for execution. + </p> + </div> + <div class="section aem-GridColumn aem-GridColumn--default--12"> + <div class="dr-article-content__section" id="lifecycle-of-a-task"> + <h2 class="dr-article-content__section-title"> + Lifecycle of a task + </h2> + </div> + </div> + <div class="text parbase aem-GridColumn aem-GridColumn--default--12"> + <ol> + <li>Client performs a <span class="dr-code">Schedule</span> RPC call to <b>Frontend</b> with task information, including execution time.  + </li> + <li>Frontend creates Edgestore <span class="dr-code">entity</span> and <span class="dr-code">assoc</span> for the task.  + </li> + <li>When it is time to process the task, <b>Store Consumer</b> pulls the task from <b>Edgestore</b> and pushes it to a related <b>SQS</b> queue.  + </li> + <li> + <b>Executor</b> makes <span class="dr-code">NextWork</span> RPC call to <b>Controller</b>, which pulls tasks from the <b>SQS</b> queue, makes a <span class="dr-code">ClaimTask</span> RPC to the HSC and then returns the task to the <b>Executor</b>.  + </li> + <li> + <b>Executor</b> invokes the callback for the task. While processing, <b>Executor</b> performs <span class="dr-code">Heartbeat</span> RPC calls to <b>Heartbeat and Status Controller (HSC)</b>. Once processing is done, <b>Executor</b> performs <span class="dr-code">TaskStatus</span> RPC call to <b>HSC</b>.  + </li> + <li>Upon getting <span class="dr-code">Heartbeat</span> and <span class="dr-code">TaskStatus</span> RPC calls, <b>HSC</b> updates the <b>Edgestore</b> entity and <span class="dr-code">assoc</span>. + </li> + </ol> + <p> + Every state update in the lifecycle of a task is accompanied by an update to the next trigger timestamp in the <span class="dr-code">assoc</span>. This ensures that the Store Consumer pulls the task again if there is no change in state of the task within the next trigger timestamp. This helps ATF achieve its at-least-once delivery guarantee by ensuring that no task is dropped. + </p> + <p> + Following are the task entity and association states in ATF and their corresponding timestamp updates: + </p> + <table> + <tbody> + <tr> + <td> + <p> + <b>Entity status</b> + </p> + </td> + <td> + <p> + <b>Assoc status</b> + </p> + </td> + <td> + <p> + <b>next trigger timestamp in Assoc</b> + </p> + </td> + <td> + <p> + <b>Comment</b> + </p> + </td> + </tr> + <tr> + <td> + <p> + <span class="dr-code">new</span> + </p> + </td> + <td> + <p> + <span class="dr-code">new</span> + </p> + </td> + <td> + <p> + <span class="dr-code">scheduled_timestamp</span> of the task + </p> + </td> + <td> + <p> + Pick up new tasks that are ready.  + </p> + </td> + </tr> + <tr> + <td> + <p> + <span class="dr-code">enqueued</span> + </p> + </td> + <td> + <p> + <span class="dr-code">started</span> + </p> + </td> + <td> + <p> + <span class="dr-code">enqueued_timestamp</span> + <span class="dr-code">enqueue_timeout</span> + </p> + </td> + <td> + <p> + Re-enqueue task if it has been in <span class="dr-code">enqueued</span> state for too long. This can happen if the queue loses data or the controller goes down after polling the queue and before the task is claimed. + </p> + </td> + </tr> + <tr> + <td> + <p> + <span class="dr-code">claimed</span> + </p> + </td> + <td> + <p> + <span class="dr-code">started</span> + </p> + </td> + <td> + <p> + <span class="dr-code">claimed_timestamp</span> + <span class="dr-code">claim_timeout</span> + </p> + </td> + <td> + <p> + Re-enqueue if task is claimed but never transfered to <span class="dr-code">processing</span>. This can happen if Controller is down after claiming a task. Task status is changed to <span class="dr-code">enqueued</span> after re-enqueue. + </p> + </td> + </tr> + <tr> + <td> + <p> + <span class="dr-code">processing</span> + </p> + </td> + <td> + <p> + <span class="dr-code">started</span> + </p> + </td> + <td> + <p> + <span class="dr-code">heartbeat_timestamp</span> + <span class="dr-code">heartbeat_timeout</span>` + </p> + </td> + <td> + <p> + Re-enqueue if task hasn’t sent <span class="dr-code">heartbeat</span> for too long. This can happen if Executor is down. Task status is changed to <span class="dr-code">enqueued</span> after re-enqueue.  + </p> + </td> + </tr> + <tr> + <td> + <p> + <span class="dr-code">retriable failure</span> + </p> + </td> + <td> + <p> + started + </p> + </td> + <td> + <p> + compute <span class="dr-code">next_timestamp</span> according to backoff logic + </p> + </td> + <td> + <p> + Exponential backoff for tasks with retriable failure.  + </p> + </td> + </tr> + <tr> + <td> + <p> + <span class="dr-code">success</span> + </p> + </td> + <td> + <p> + <span class="dr-code">completed</span> + </p> + </td> + <td> + <p> + N/A + </p> + </td> + <td> + <p> +   + </p> + </td> + </tr> + <tr> + <td> + <p> + <span class="dr-code">fatal_failure</span> + </p> + </td> + <td> + <p> + <span class="dr-code">completed</span> + </p> + </td> + <td> + <p> + N/A + </p> + </td> + <td> + <p> +   + </p> + </td> + </tr> + </tbody> + </table> + <p> + The store consumer polls for tasks based on the following query: + </p> + <p> + <span class="dr-code">assoc_status= && next_timestamp<=time.now()<br /></span> + </p> + <p> + Below is the state machine that defines task state transitions: <br /> + </p> + </div> + <div class="image c04-image aem-GridColumn aem-GridColumn--default--12"> + <div class="dr-image image cq-dd-image"> + <figure class="dr-margin-0 dr-display-inline-block"> + <img src="/cms/content/dam/dropbox/tech-blog/en-us/2020/11/atf/diagrams/Techblog-ATF-720x225px-2.png" aria-hidden="false" alt="Task State Transitions [Fig 2]" height="450" width="1440" /> + </figure> + </div> + </div> + <div class="section aem-GridColumn aem-GridColumn--default--12"> + <div class="dr-article-content__section" id="-achieving-guarantees"> + <h2 class="dr-article-content__section-title"> + Achieving guarantees + </h2> + </div> + </div> + <div class="text parbase aem-GridColumn aem-GridColumn--default--12"> + <p> + <b>At-least-once task execution<br /></b> At-least-once execution is guaranteed in ATF by retrying a task until it completes execution (which is signaled by a <span class="dr-code">Success</span> or a <span class="dr-code">FatalFailure</span> state). All ATF system errors are implicitly considered retriable failures, and lambda owners have an option of marking tasks with a <span class="dr-code">RetriableFailure</span> state. Tasks might be dropped from the ATF execution pipeline in different parts of the system through transient RPC failures and failures on dependencies like Edgestore or SQS. These transient failures at different parts of the system do not affect the at-least-once guarantee, though, because of the system of timeouts and re-polling from Store Consumer. + </p> + <p> + <b>No concurrent task execution<br /></b> Concurrent task execution is avoided through a combination of two methods in ATF. First, tasks are explicitly claimed through an exclusive task state (<span class="dr-code">Claimed</span>) before starting execution. Once the task execution is complete, the task status is updated to one of <span class="dr-code">Success</span>, <span class="dr-code">FatalFailure</span> or <span class="dr-code">RetriableFailure</span>. A task can be claimed only if its existing task state is <span class="dr-code">Enqueued</span> (retried tasks go to the <span class="dr-code">Enqueued</span> state as well once they are re-pushed onto SQS). + </p> + <p> + However, there might be situations where once a long running task starts execution, its heartbeats might fail repeatedly yet the task execution continues. ATF would retry this task by polling it from the store consumer because the heartbeat timeouts would’ve expired. This task can then be claimed by another worker and lead to concurrent execution. <br /> + </p> + <p> + To avoid this situation, there is a termination logic in the Executor processes whereby an Executor process terminates itself as soon as three consecutive heartbeat calls fail. Each heartbeat timeout is large enough to eclipse three consecutive heartbeat failures. This ensures that the Store Consumer cannot pull such tasks before the termination logic ends them—the second method that helps achieve this guarantee. + </p> + <p> + <b>Isolation<br /></b> Isolation of lambdas is achieved through dedicated worker clusters, dedicated queues, and dedicated per-lambda scheduling quotas. In addition, isolation across different priorities within the same lambda is likewise achieved through dedicated queues and scheduling bandwidth. + </p> + <p> + <b>Delivery latency<br /></b> ATF use cases do not require ultra-low task delivery latencies. Task delivery latencies on the order of a couple of seconds are acceptable. Tasks ready for execution are periodically polled by the Store Consumer and this period of polling largely controls the task delivery latency. Using this as a tuning lever, ATF can achieve different delivery latencies as required. Increasing poll frequency reduces task delivery latency and vice versa. Currently, we have calibrated ATF to poll for ready tasks once every two seconds. + </p> + </div> + <div class="section aem-GridColumn aem-GridColumn--default--12"> + <div class="dr-article-content__section" id="ownership-model"> + <h2 class="dr-article-content__section-title"> + Ownership model + </h2> + </div> + </div> + <div class="text parbase aem-GridColumn aem-GridColumn--default--12"> + <p> + ATF is designed to be a self-serve framework for developers at Dropbox. The design is very intentional in driving an ownership model where lambda owners own all aspects of their lambdas’ operations. To promote this, all lambda worker clusters are owned by the lambda owners. They have full control over operations on these clusters, including code deployments and capacity management. Each executor process is bound to one lambda. Owners have the option of deploying multiple lambdas on their worker clusters simply by spawning new executor processes on their hosts. + </p> + </div> + <div class="section aem-GridColumn aem-GridColumn--default--12"> + <div class="dr-article-content__section" id="-extending-atf"> + <h2 class="dr-article-content__section-title"> + Extending ATF + </h2> + </div> + </div> + <div class="text parbase aem-GridColumn aem-GridColumn--default--12"> + <p> + As described above, ATF provides an infrastructural building block for scheduling asynchronous tasks. With this foundation established, ATF can be extended to support more generic use cases and provide more features as a framework. Following are some examples of what could be built as an extension to ATF.  + </p> + <p> + <b>Periodic task execution<br /></b> Currently, ATF is a system for one-time task scheduling. Building support for periodic task execution as an extension to this framework would be useful in unlocking new capabilities for our clients. + </p> + <p> + <b>Better support for task chaining<br /></b> Currently, it is possible to chain tasks on ATF by scheduling a task onto ATF that then schedules other tasks onto ATF during its execution. Although it is possible to do this in the current ATF setup, visibility and control on this chaining is absent at the framework level. Another natural extension here would be to better support task chaining through framework-level visibility and control, to make this use case a first class concept in the ATF model. + </p> + <p> + <b>Dead letter queues for misbehaving tasks<br /></b> One common source of maintenance overhead we observe on ATF is that some tasks get stuck in infinite retry loops due to occasional bugs in lambda logic. This requires manual intervention from the ATF framework owners in some cases where there are a large number of tasks stuck in such loops, occupying a lot of the scheduling bandwidth in the system. Typical manual actions in response to such a situation include pausing execution of the lambdas with misbehaving tasks, or dropping them outright. + </p> + <p> + One way to reduce this operational overhead and provide an easy interface for lambda owners to recover from such incidents would be to create dead letter queues filled with such misbehaving tasks. The ATF framework could impose a maximum number of retries before tasks are pushed onto the dead letter queue. We could create and expose tools that make it easy to reschedule tasks from the dead letter queue back into the ATF system, once the associated lambda bugs are fixed.<br /> + </p> + </div> + <div class="section aem-GridColumn aem-GridColumn--default--12"> + <div class="dr-article-content__section" id="conclusion"> + <h2 class="dr-article-content__section-title"> + Conclusion + </h2> + </div> + </div> + <div class="text parbase aem-GridColumn aem-GridColumn--default--12"> + <p> + We hope this post helps engineers elsewhere to develop better async task frameworks of their own. Many thanks to everyone who worked on this project: Anirudh Jayakumar, Deepak Gupta, Dmitry Kopytkov, Koundinya Muppalla, Peng Kang, Rajiv Desai, Ryan Armstrong, Steve Rodrigues, Thomissa Comellas, Xiaonan Zhang and Yuhuan Du.<br /> +   + </p> + </div> + </div> + </div> + <hr class="dr-typography-t5 dr-margin-top-50 dr-article-content__divider" /> + <div class="dr-typography-t5"></div> + <div class="dr-typography-t5 dr-margin-top-20"> + // Tags<br /> + <ul class="dr-unstyled-list dr-margin-top-10 dr-typography-t4"> + <li class="dr-container--infrastructure dr-display-inline-block dr-margin-right-10 dr-margin-bottom-10"> + <a class="dr-link dr-pill dr-pill--primary dr-link--no-underline" href="https://dropbox.tech/infrastructure">Infrastructure</a> + </li> + <li class="dr-display-inline-block dr-margin-right-10"> + <a class="dr-link dr-pill dr-link--no-underline" href="https://dropbox.tech/tag-results.task-scheduling">Task Scheduling</a> + </li> + <li class="dr-display-inline-block dr-margin-right-10"> + <a class="dr-link dr-pill dr-link--no-underline" href="https://dropbox.tech/tag-results.async">Async</a> + </li> + <li class="dr-display-inline-block dr-margin-right-10"> + <a class="dr-link dr-pill dr-link--no-underline" href="https://dropbox.tech/tag-results.edgestore">Edgestore</a> + </li> + </ul> + </div> + <div class="dr-typography-t5 dr-margin-top-20 dr-hide-from-md"> + // Copy link<br /> + <div class="dr-article-content__social-links-tooltip dr-display-none"> + Link copied + </div><button class="dr-button dr-button--link dr-link dr-link--no-underline dr-article-content__copy-link" data-dr-tooltip="Copy link"><img alt="Copy link" class="dr-display-block dr-invert-on-theme-dark" src="/cms/etc.clientlibs/settings/wcm/designs/dropbox-tech-blog/clientlib-article-content/resources/copy.svg" /></button> + </div> + <div class="dr-article-content__social-links"> + <ul class="dr-article-content__social-links-list dr-unstyled-list"> + <li class="dr-margin-bottom-20"> + <div class="dr-article-content__social-links-tooltip dr-typography-t5 dr-display-none"> + Link copied + </div><button class="dr-button dr-display-block dr-link dr-link--no-underline dr-article-content__copy-link dr-button--link" data-dr-tooltip="Copy link" data-dr-tooltip-position="cl" data-dr-tooltip-theme="bw"><img alt="Copy link" class="dr-display-block dr-invert-on-theme-dark" src="/cms/etc.clientlibs/settings/wcm/designs/dropbox-tech-blog/clientlib-article-content/resources/copy.svg" /></button> + </li> + <li class="dr-margin-bottom-20"> + <a class="dr-link dr-display-block dr-link--no-underline dr-article-content__share-link dr-article-content__twitter-link" data-dr-tooltip="Share on Twitter" data-dr-tooltip-position="cl" data-dr-tooltip-theme="bw" href="https://twitter.com/intent/tweet/?text=How%20we%20designed%20Dropbox%20ATF%3A%20an%20async%20task%20framework&url=https://dropbox.tech/infrastructure/asynchronous-task-scheduling-at-dropbox" target="_blank"><img alt="Share on Twitter" class="dr-display-block dr-invert-on-theme-dark" src="/cms/etc.clientlibs/settings/wcm/designs/dropbox-tech-blog/clientlib-article-content/resources/twitter.svg" /></a> + </li> + <li class="dr-margin-bottom-20"> + <a class="dr-link dr-display-block dr-link--no-underline dr-article-content__share-link dr-article-content__facebook-link" data-dr-tooltip="Share on Facebook" data-dr-tooltip-position="cl" data-dr-tooltip-theme="bw" href="https://facebook.com/sharer/sharer.php?u=https://dropbox.tech/infrastructure/asynchronous-task-scheduling-at-dropbox" target="_blank"><img alt="Share on Facebook" class="dr-display-block dr-invert-on-theme-dark" src="/cms/etc.clientlibs/settings/wcm/designs/dropbox-tech-blog/clientlib-article-content/resources/facebook.svg" /></a> + </li> + <li> + <a class="dr-link dr-display-block dr-link--no-underline dr-article-content__share-link dr-article-content__linkedin-link" data-dr-tooltip="Share on Linkedin" data-dr-tooltip-position="cl" data-dr-tooltip-theme="bw" href="https://www.linkedin.com/shareArticle?mini=true&url=https://dropbox.tech/infrastructure/asynchronous-task-scheduling-at-dropbox&title=How%20we%20designed%20Dropbox%20ATF%3A%20an%20async%20task%20framework&source=https://dropbox.tech/infrastructure/asynchronous-task-scheduling-at-dropbox" target="_blank"><img alt="Share on Linkedin" class="dr-display-block dr-invert-on-theme-dark" src="/cms/etc.clientlibs/settings/wcm/designs/dropbox-tech-blog/clientlib-article-content/resources/linkedin.svg" /></a> + </li> + </ul> + </div> + </div> + </div> + </div> + <div class="aem-Grid aem-Grid--12 aem-Grid--default--12"> + <div class="plain-html c17-plain-html aem-GridColumn aem-GridColumn--default--12"> + <div class="knotch_placeholder"></div> + </div> + </div> + <footer class="dr-footer"> + <div class="dr-container--surface"> + <section class="dr-container__content dr-footer__container"> + <div class="dr-newsletter-subscription__succeed dr-display-none dr-typography-t5"> + <hr class="dr-newsletter-subscription__form-divider" /> + <div class="dr-margin-bottom-30 dr-margin-top-30"> + <!--// Thanks for subscribing.--> + <div class="dr-show-block-from-lg"> + <img src="/cms/content/dam/dropbox/tech-blog/en-us/subscribe/thanksforsubscribing_desktop.png" title="subscription__success" alt="subscription__success" /> + </div> + <div class="dr-show-block-from-md dr-hide-from-lg dr-hide-from-sm"> + <img src="/cms/content/dam/dropbox/tech-blog/en-us/subscribe/thanksforsubscribing_tablet.png" title="subscription__success" alt="subscription__success" /> + </div> + <div class="dr-show-block-from-sm dr-hide-from-lg dr-hide-from-md"> + <img src="/cms/content/dam/dropbox/tech-blog/en-us/subscribe/thanksforsubscribing_mobile.png" title="subscription__success" alt="subscription__success" /> + </div> + </div> + <hr class="dr-newsletter-subscription__form-divider" /> + </div> + <form role="form" class="dr-typography-t5 dr-newsletter-subscription__form" novalidate=""> + <hr class="dr-newsletter-subscription__form-divider" /> + <div class="dr-margin-top-30 dr-margin-bottom-30 dr-margin-md-top-10 dr-margin-md-bottom-10"> + // Subscribe to email updates by category + </div> + <div class="dr-margin-left-25"> + <p class="dr-newsletter-subscription__topic-error dr-display-none dr-color-tangerine"> + Select at least one topic + </p><label class="dr-newsletter-subscription__form-label" for="newsletterForm.application"><input class="dr-newsletter-subscription__form-checkbox dr-input" id="newsletterForm.application" name="categories[ ]" type="checkbox" value="Application" data-mid="127814" />Application</label> <label class="dr-newsletter-subscription__form-label" for="newsletterForm.frontend"><input class="dr-newsletter-subscription__form-checkbox dr-input" id="newsletterForm.frontend" name="categories[ ]" type="checkbox" value="Front End" data-mid="127842" />Front End</label> <label class="dr-newsletter-subscription__form-label" for="newsletterForm.infrastructure"><input class="dr-newsletter-subscription__form-checkbox dr-input" id="newsletterForm.infrastructure" name="categories[ ]" type="checkbox" value="Infrastructure" data-mid="127826" />Infrastructure</label> <label class="dr-newsletter-subscription__form-label" for="newsletterForm.machine-learning"><input class="dr-newsletter-subscription__form-checkbox dr-input" id="newsletterForm.machine-learning" name="categories[ ]" type="checkbox" value="Machine Learning" data-mid="127830" />Machine Learning</label><br class="dr-show-block-from-md" /> + <label class="dr-newsletter-subscription__form-label" for="newsletterForm.mobile"><input class="dr-newsletter-subscription__form-checkbox dr-input" id="newsletterForm.mobile" name="categories[ ]" type="checkbox" value="Mobile" data-mid="127834" />Mobile</label> <label class="dr-newsletter-subscription__form-label" for="newsletterForm.security"><input class="dr-newsletter-subscription__form-checkbox dr-input" id="newsletterForm.security" name="categories[ ]" type="checkbox" value="Security" data-mid="127838" />Security</label> <label class="dr-newsletter-subscription__form-label" for="newsletterForm.developers"><input class="dr-newsletter-subscription__form-checkbox dr-input" id="newsletterForm.developers" name="categories[ ]" type="checkbox" value="Developers" data-mid="129642" />Developers</label> <label class="dr-newsletter-subscription__form-label" for="newsletterForm.all"><input class="dr-newsletter-subscription__form-checkbox dr-newsletter-subscription__form-checkbox--all dr-input" id="newsletterForm.all" type="checkbox" />All</label> + </div> + <p class="dr-newsletter-subscription__error dr-display-none dr-color-tangerine"> + Error occurred!<br /> + Please try again later + </p> + <p class="dr-newsletter-subscription__email-error dr-display-none dr-color-tangerine"> + Enter a valid address + </p> + <div class="dr-newsletter-subscription__email-container dr-margin-bottom-20 dr-margin-top-40 dr-margin-md-top-0"> + <div> + // Type your email address + </div><input autocomplete="off" class="dr-newsletter-subscription__form-input dr-flex-1" name="email" type="email" /> + <div class="dr-newsletter-subscription__actions-container"> + <div class="dr-newsletter-subscription__loading dr-display-none"> + Submitting... + </div><button type="submit" disabled="disabled" class="dr-newsletter-subscription__form-submit dr-button dr-typography-t5">Subscribe</button> + </div> + </div> + <hr class="dr-newsletter-subscription__form-divider" /> + </form> + <div class="dr-grid dr-grid--md-2"> + <div> + <a href="https://dropbox.com" target="_blank" class="dr-margin-bottom-20 dr-display-block"><img alt="Dropbox" height="40" src="/cms/etc.clientlibs/settings/wcm/designs/dropbox-tech-blog/clientlib-all/resources/logo_dropbox.svg" width="164" /></a> + </div> + <ul class="dr-footer-links dr-unstyled-list dr-typography-t10 dr-grid dr-grid--2 dr-grid--column-gap-15"> + <li> + <a class="dr-link dr-link--no-underline" href="http://dropbox.com/jobs" target="_blank">Jobs</a> + </li> + <li> + <a class="dr-link dr-link--no-underline" href="https://medium.com/@Dropbox" target="_blank">Medium</a> + </li> + <li> + <a class="dr-link dr-link--no-underline" href="https://www.dropbox.com/privacy" target="_blank">Privacy</a> + </li> + <li> + <a class="dr-link dr-link--no-underline" href="https://twitter.com/Dropbox" target="_blank">twitter</a> + </li> + <li> + <a class="dr-link dr-link--no-underline" href="https://www.dropbox.com/terms" target="_blank">Terms</a> + </li> + <li> + <a class="dr-link dr-link--no-underline" href="https://www.instagram.com/dropbox" target="_blank">Instagram</a> + </li> + <li> + <a class="dr-link dr-link--no-underline" href="https://blog.dropbox.com/" target="_blank">Work In Progress</a> + </li> + </ul> + </div> + </section> + </div> + </footer> + <div id="u04-snapengage-config" data-snapengage-widget-id="d5c1efed-d0ef-4fca-8c7d-faff398ad272" data-proactive-chat="false" style="display:none;"></div> + <script type="text/javascript" src="/cms/etc.clientlibs/settings/wcm/designs/dropbox-common/clientlib-cms-common.7f3cf4624fd698d8bfec572c3c993880.js"></script> + <script type="text/javascript" src="/cms/etc.clientlibs/settings/wcm/designs/dropbox-tech-blog/clientlib-all.3230e3eaa6e5a90686710bfde829f620.js"></script> + <script type="text/javascript" src="/cms/etc.clientlibs/settings/wcm/designs/dropbox-tech-blog/clientlib-article-content.2c12dd2925c2dcad6bde22d2ff271137.js"></script> + <script type="application/javascript"> + <![CDATA[ + document.body.classList.remove('stormcrow-animate'); + ]]> + </script> <noscript></noscript> + </body> +</html> diff --git a/test/test-pages/ehow-1/expected.html b/test/test-pages/ehow-1/expected.html index 736401e..d73c3c5 100644 --- a/test/test-pages/ehow-1/expected.html +++ b/test/test-pages/ehow-1/expected.html @@ -9,7 +9,11 @@ <figure> <img src="http://img-aws.ehowcdn.com/640/cme/photography.prod.demandstudios.com/16149374-814f-40bc-baf3-ca20f149f0ba.jpg" alt="Glass cloche terrariums" title="Glass cloche terrariums" data-credit="Lucy Akins " longdesc="http://s3.amazonaws.com/photography.prod.demandstudios.com/16149374-814f-40bc-baf3-ca20f149f0ba.jpg"> </figure> <figcaption> Glass cloche terrariums (Lucy Akins) </figcaption> </div> - + <div id="relatedContentUpper" data-module="rcp_top"> + <header class> + <h3>Other People Are Reading</h3> </header> + + </div> <div> <p><span>What You'll Need:</span></p><ul> <li>Cloche</li> <li>Planter saucer, small shallow dish or desired platform</li> diff --git a/test/test-pages/ehow-2/expected.html b/test/test-pages/ehow-2/expected.html index f2c0762..48823ad 100644 --- a/test/test-pages/ehow-2/expected.html +++ b/test/test-pages/ehow-2/expected.html @@ -114,7 +114,12 @@ <figcaption> Mark Stout/iStock/Getty Images </figcaption> </div> - + <div id="relatedContentUpper" data-module="rcp_top"> + <header class> + <h3>Other People Are Reading</h3> + </header> + + </div> </span> </span> diff --git a/test/test-pages/embedded-videos/expected.html b/test/test-pages/embedded-videos/expected.html index 5ad8919..4dc9164 100644 --- a/test/test-pages/embedded-videos/expected.html +++ b/test/test-pages/embedded-videos/expected.html @@ -1,12 +1,12 @@ <article> - + <h2>Lorem</h2> <p>Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.</p> - <h2>Videos</h2> + <p>At root</p> <iframe width="560" height="315" src="https://www.youtube.com/embed/LtOGa5M8AuU" frameborder="0" allowfullscreen></iframe> <iframe width="560" height="315" src="https://www.youtube-nocookie.com/embed/LtOGa5M8AuU" frameborder="0" allowfullscreen></iframe> diff --git a/test/test-pages/google-sre-book-1/expected-images.json b/test/test-pages/google-sre-book-1/expected-images.json new file mode 100644 index 0000000..0637a08 --- /dev/null +++ b/test/test-pages/google-sre-book-1/expected-images.json @@ -0,0 +1 @@ +[]
\ No newline at end of file diff --git a/test/test-pages/google-sre-book-1/expected-metadata.json b/test/test-pages/google-sre-book-1/expected-metadata.json new file mode 100644 index 0000000..41b1ff7 --- /dev/null +++ b/test/test-pages/google-sre-book-1/expected-metadata.json @@ -0,0 +1,8 @@ +{ + "Author": "Written by Rob Ewaschuk\n Edited by Betsy Beyer", + "Direction": null, + "Excerpt": "Google\u2019s SRE teams have some basic principles and best practices for building successful monitoring and alerting systems. This chapter offers guidelines for what issues should interrupt a human via a page, and how to deal with issues that aren\u2019t serious enough to trigger a page.", + "Image": null, + "Title": "Google - Site Reliability Engineering", + "SiteName": null +}
\ No newline at end of file diff --git a/test/test-pages/google-sre-book-1/expected.html b/test/test-pages/google-sre-book-1/expected.html new file mode 100644 index 0000000..1461f27 --- /dev/null +++ b/test/test-pages/google-sre-book-1/expected.html @@ -0,0 +1,458 @@ +<section data-type="chapter" id="maia-main" role="main"> + <h2> + Monitoring Distributed Systems + </h2> + + <p> + Google’s SRE teams have some basic principles and best practices for building successful monitoring and alerting systems. This chapter offers guidelines for what issues should interrupt a human via a page, and how to deal with issues that aren’t serious enough to trigger a page. + </p> + <section data-type="sect1" id="definitions-2ksZhN"> + <h2> + Definitions + </h2> + <p> + <a data-type="indexterm" data-primary="monitoring distributed systems" data-secondary="terminology" id="id-DnC1SWFMhD"></a>There’s no uniformly shared vocabulary for discussing all topics related to monitoring. Even within Google, usage of the following terms varies, but the most common interpretations are listed here. + </p> + <dl> + + <dd> + <p> + Collecting, processing, aggregating, and displaying real-time quantitative data about a system, such as query counts and types, error counts and types, processing times, and server lifetimes. + </p> + </dd> + + <dd> + <p> + <a data-type="indexterm" data-primary="white-box monitoring" id="id-9nCjSDS4tZILhX"></a>Monitoring based on metrics exposed by the internals of the system, including logs, interfaces like the Java Virtual Machine Profiling Interface, or an HTTP handler that emits internal statistics. + </p> + </dd> + + <dd> + <p> + <a data-type="indexterm" data-primary="black-box monitoring" id="id-zdCxSrSgTWIdhb"></a>Testing externally visible behavior as a user would see it. + </p> + </dd> + + <dd> + <p> + <a data-type="indexterm" data-primary="dashboards" data-secondary="defined" id="id-VMCPS2SribIkh4"></a>An application (usually web-based) that provides a summary view of a service’s core metrics. A dashboard may have filters, selectors, and so on, but is prebuilt to expose the metrics most important to its users. The dashboard might also display team information such as ticket queue length, a list of high-priority bugs, the current on-call engineer for a given area of responsibility, or recent pushes. + </p> + </dd> + + <dd> + <p> + <a data-type="indexterm" data-primary="alerts" data-secondary="defined" id="id-wqC7SvSPUAIVhQ"></a>A notification intended to be read by a human and that is pushed to a system such as a bug or ticket queue, an email alias, or a pager. Respectively, these alerts are classified as <em>tickets</em>, <em>email alerts</em>,<sup><a data-type="noteref" id="id-LvQuvtYS7UvI8h4-marker" href="#id-LvQuvtYS7UvI8h4">22</a></sup> and <em>pages</em>. + </p> + </dd> + + <dd> + <p> + <a data-type="indexterm" data-primary="root cause" data-secondary="defined" id="id-PnCpSaSKsgIjho"></a>A defect in a software or human system that, if repaired, instills confidence that this event won’t happen again in the same way. A given incident might have multiple root causes: for example, perhaps it was caused by a combination of insufficient process automation, software that crashed on bogus input, <em>and</em> insufficient testing of the script used to generate the configuration. Each of these factors might stand alone as a root cause, and each should be repaired. + </p> + </dd> + <dt id="node-and-machine"> + Node and machine + </dt> + <dd> + <p> + <a data-type="indexterm" data-primary="machines" data-secondary="defined" id="id-XmC9SkSlfnI1hK"></a>Used interchangeably to indicate a single instance of a running kernel in either a physical server, virtual machine, or container. There might be multiple <em>services</em> worth monitoring on a single machine. The services may either be: + </p> + <ul> + <li>Related to each other: for example, a caching server and a web server + </li> + <li>Unrelated services sharing hardware: for example, a code repository and a master for a configuration system like <a href="https://puppetlabs.com/puppet/puppet-open-source" target="_blank">Puppet</a> or <a href="https://www.chef.io/chef/" target="_blank">Chef</a> + </li> + </ul> + </dd> + + <dd> + <p> + Any change to a service’s running software or its configuration. + </p> + </dd> + </dl> + </section> + <section data-type="sect1" id="why-monitor-pWsBTZ"> + <h2> + Why Monitor? + </h2> + <p> + <a data-type="indexterm" data-primary="monitoring distributed systems" data-secondary="benefits of monitoring" id="id-kVCkSpFnTl"></a>There are many reasons to monitor a system, including: + </p> + <dl> + + <dd> + <p> + How big is my database and how fast is it growing? How quickly is my daily-active user count growing? + </p> + </dd> + + <dd> + <p> + Are queries faster with Acme Bucket of Bytes 2.72 versus Ajax DB 3.14? How much better is my memcache hit rate with an extra node? Is my site slower than it was last week? + </p> + </dd> + + <dd> + <p> + Something is broken, and somebody needs to fix it right now! Or, something might break soon, so somebody should look soon. + </p> + </dd> + + <dd> + <p> + <a data-type="indexterm" data-primary="dashboards" data-secondary="benefits of" id="id-rjCXSOS0iDIGT8"></a>Dashboards should answer basic questions about your service, and normally include some form of the four golden signals (discussed in <a data-type="xref" href="#xref_monitoring_golden-signals">The Four Golden Signals</a>). + </p> + </dd> + + <dd> + <p> + Our latency just shot up; what else happened around the same time? + </p> + </dd> + </dl> + <p> + System monitoring is also helpful in supplying raw input into business analytics and in facilitating analysis of security breaches. Because this book focuses on the engineering domains in which SRE has particular expertise, we won’t discuss these applications of monitoring here. + </p> + <p> + Monitoring and alerting enables a system to tell us when it’s broken, or perhaps to tell us what’s about to break. When the system isn’t able to automatically fix itself, we want a human to investigate the alert, determine if there’s a real problem at hand, mitigate the problem, and determine the root cause of the problem. Unless you’re performing security auditing on very narrowly scoped components of a system, you should never trigger an alert simply because "something seems a bit weird." + </p> + <p> + Paging a human is a quite expensive use of an employee’s time. If an employee is at work, a page interrupts their workflow. If the employee is at home, a page interrupts their personal time, and perhaps even their sleep. When pages occur too frequently, employees second-guess, skim, or even ignore incoming alerts, sometimes even ignoring a "real" page that’s masked by the noise. Outages can be prolonged because other noise interferes with a rapid diagnosis and fix. Effective alerting systems have good signal and very low noise. + </p> + </section> + <section data-type="sect1" id="setting-reasonable-expectations-for-monitoring-o8svcM"> + <h2> + Setting Reasonable Expectations for Monitoring + </h2> + <p> + <a data-type="indexterm" data-primary="monitoring distributed systems" data-secondary="setting expectations for" id="id-4nCqSYFQcE"></a>Monitoring a complex application is a significant engineering endeavor in and of itself. Even with substantial existing infrastructure for instrumentation, collection, display, and alerting in place, a Google SRE team with 10–12 members typically has one or sometimes two members whose primary assignment is to build and maintain monitoring systems for their service. This number has decreased over time as we generalize and centralize common monitoring infrastructure, but every SRE team typically has at least one “monitoring person.” (That being said, while it can be fun to have access to traffic graph dashboards and the like, SRE teams carefully avoid any situation that requires someone to “stare at a screen to watch for problems.”) + </p> + <p> + <a data-type="indexterm" data-primary="post hoc analysis" id="id-JnCDSjIVcG"></a>In general, Google has trended toward simpler and faster monitoring systems, with better tools for <em>post hoc</em> analysis. We avoid "magic" systems that try to learn thresholds or automatically detect causality. Rules that detect unexpected changes in end-user request rates are one counterexample; while these rules are still kept as simple as possible, they give a very quick detection of a very simple, specific, severe anomaly. Other uses of monitoring data such as capacity planning and traffic prediction can tolerate more fragility, and thus, more complexity. Observational experiments conducted over a very long time horizon (months or years) with a low sampling rate (hours or days) can also often tolerate more fragility because occasional missed samples won’t hide a long-running trend. + </p> + <p> + <a data-type="indexterm" data-primary="dependency hierarchies" id="id-9nCjSOtmcj"></a>Google SRE has experienced only limited success with complex dependency hierarchies. We seldom use rules such as, "If I know the database is slow, alert for a slow database; otherwise, alert for the website being generally slow." Dependency-reliant rules usually pertain to very stable parts of our system, such as our system for draining user traffic away from a datacenter. For example, "If a datacenter is drained, then don’t alert me on its latency" is one common datacenter alerting rule. Few teams at Google maintain complex dependency hierarchies because our infrastructure has a steady rate of continuous refactoring. + </p> + <p> + Some of the ideas described in this chapter are still aspirational: there is always room to move more rapidly from symptom to root cause(s), especially in ever-changing systems. So while this chapter sets out some goals for monitoring systems, and some ways to achieve these goals, it’s important that monitoring systems—especially the critical path from the onset of a production problem, through a page to a human, through basic triage and deep debugging—be kept simple and comprehensible by everyone on the team. + </p> + <p> + Similarly, to keep noise low and signal high, the elements of your monitoring system that direct to a pager need to be very simple and robust. Rules that generate alerts for humans should be simple to understand and represent a clear failure. + </p> + </section> + <section data-type="sect1" id="symptoms-versus-causes-g0sEi4"> + <h2> + Symptoms Versus Causes + </h2> + <p> + <a data-type="indexterm" data-primary="monitoring distributed systems" data-secondary="symptoms vs. causes" id="id-JnCDSlFmiG"></a>Your monitoring system should address two questions: what’s broken, and why? + </p> + <p> + The "what’s broken" indicates the symptom; the "why" indicates a (possibly intermediate) cause. <a data-type="xref" href="#table_monitoring_symptoms">Table 6-1</a> lists some hypothetical symptoms and corresponding causes. + </p> + <table id="table_monitoring_symptoms" readabilityDataTable="1"> + <caption> + <span>Table 6-1.</span> Example symptoms and causes + </caption> + <thead> + <tr> + <th> + <strong>Symptom</strong> + </th> + <th> + <strong>Cause</strong> + </th> + </tr> + </thead> + <tbody> + <tr> + <td> + <p> + <strong>I’m serving HTTP 500s or 404s</strong> + </p> + </td> + <td> + <p> + Database servers are refusing connections + </p> + </td> + </tr> + <tr> + <td> + <p> + <strong>My responses are slow</strong> + </p> + </td> + <td> + <p> + CPUs are overloaded by a bogosort, or an Ethernet cable is crimped under a rack, visible as partial packet loss + </p> + </td> + </tr> + <tr> + <td> + <p> + <strong>Users in Antarctica aren’t receiving animated cat GIFs</strong> + </p> + </td> + <td> + <p> + Your Content Distribution Network hates scientists and felines, and thus blacklisted some client IPs + </p> + </td> + </tr> + <tr> + <td> + <p> + <strong>Private content is world-readable</strong> + </p> + </td> + <td> + <p> + A new software push caused ACLs to be forgotten and allowed all requests + </p> + </td> + </tr> + </tbody> + </table> + <p> + "What" versus "why" is one of the most important distinctions in writing good monitoring with maximum signal and minimum noise. + </p> + </section> + <section data-type="sect1" id="black-box-versus-white-box-q8sJuw"> + <h2> + Black-Box Versus White-Box + </h2> + <p> + <a data-type="indexterm" data-primary="monitoring distributed systems" data-secondary="blackbox vs. whitebox" id="id-9nCjSvFVuj"></a><a data-type="indexterm" data-primary="white-box monitoring" id="id-ZbC1FMFEu7"></a><a data-type="indexterm" data-primary="black-box monitoring" id="id-zdCXIGFvuy"></a>We combine heavy use of white-box monitoring with modest but critical uses of black-box monitoring. The simplest way to think about black-box monitoring versus white-box monitoring is that black-box monitoring is symptom-oriented and represents active—not predicted—problems: "The system isn’t working correctly, right now." White-box monitoring depends on the ability to inspect the innards of the system, such as logs or HTTP endpoints, with instrumentation. White-box monitoring therefore allows detection of imminent problems, failures masked by retries, and so forth. + </p> + <p> + Note that in a multilayered system, one person’s symptom is another person’s cause. For example, suppose that a database’s performance is slow. Slow database reads are a symptom for the database SRE who detects them. However, for the frontend SRE observing a slow website, the same slow database reads are a cause. Therefore, white-box monitoring is sometimes symptom-oriented, and sometimes cause-oriented, depending on just how informative your white-box is. + </p> + <p> + When collecting telemetry for debugging, white-box monitoring is essential. If web servers seem slow on database-heavy requests, you need to know both how fast the web server perceives the database to be, and how fast the database believes itself to be. Otherwise, you can’t distinguish an actually slow database server from a network problem between your web server and your database. + </p> + <p> + For paging, black-box monitoring has the key benefit of forcing discipline to only nag a human when a problem is both already ongoing and contributing to real symptoms. On the other hand, for not-yet-occurring but imminent problems, black-box monitoring is fairly useless. + </p> + </section> + <section data-type="sect1" id="xref_monitoring_golden-signals"> + <h2> + The Four Golden Signals + </h2> + <p> + <a data-type="indexterm" data-primary="monitoring distributed systems" data-secondary="four golden signals of" id="id-ZbCxSMFjU7"></a>The four golden signals of monitoring are latency, traffic, errors, and saturation. If you can only measure four metrics of your user-facing system, focus on these four. + </p> + <dl> + + <dd> + <p> + <a data-type="indexterm" data-primary="service latency" data-secondary="monitoring for" id="id-yYCASJS9FKIWUb"></a><a data-type="indexterm" data-primary="latency" data-secondary="monitoring for" id="id-VMCpF2SXFbIwU4"></a><a data-type="indexterm" data-primary="request latency" id="id-rjCeIOSKFDIaU8"></a><a data-type="indexterm" data-primary="user requests" data-secondary="request latency monitoring" id="id-wqCDtvSGFAIMUQ"></a>The time it takes to service a request. It’s important to distinguish between the latency of successful requests and the latency of failed requests. For example, an HTTP 500 error triggered due to loss of connection to a database or other critical backend might be served very quickly; however, as an HTTP 500 error indicates a failed request, factoring 500s into your overall latency might result in misleading calculations. On the other hand, a slow error is even worse than a fast error! Therefore, it’s important to track error latency, as opposed to just filtering out errors. + </p> + </dd> + + <dd> + <p> + <a data-type="indexterm" data-primary="user requests" data-secondary="traffic analysis" id="id-rjCXSOSxtDIaU8"></a><a data-type="indexterm" data-primary="traffic analysis" id="id-wqC4FvSBtAIMUQ"></a>A measure of how much demand is being placed on your system, measured in a high-level system-specific metric. For a web service, this measurement is usually HTTP requests per second, perhaps broken out by the nature of the requests (e.g., static versus dynamic content). For an audio streaming system, this measurement might focus on network I/O rate or concurrent sessions. For a key-value storage system, this measurement might be transactions and retrievals per <span>second</span>. + </p> + </dd> + + <dd> + <p> + <a data-type="indexterm" data-primary="error rates" id="id-x1C4SjSlTLIMUJ"></a><a data-type="indexterm" data-primary="user requests" data-secondary="monitoring failures" id="id-PnCxFaS0TgIVUo"></a>The rate of requests that fail, either explicitly (e.g., HTTP 500s), implicitly (for example, an HTTP 200 success response, but coupled with the wrong content), or by policy (for example, "If you committed to one-second response times, any request over one second is an error"). Where protocol response codes are insufficient to express all failure conditions, secondary (internal) protocols may be necessary to track partial failure modes. Monitoring these cases can be drastically different: catching HTTP 500s at your load balancer can do a decent job of catching all completely failed requests, while only end-to-end system tests can detect that you’re serving the wrong content. + </p> + </dd> + + <dd> + <p> + <a data-type="indexterm" data-primary="saturation" id="id-OnCNS2S4iDIYU8"></a>How "full" your service is. A measure of your system fraction, emphasizing the resources that are most constrained (e.g., in a memory-constrained system, show memory; in an I/O-constrained system, show I/O). Note that many systems degrade in performance before they achieve 100% utilization, so having a utilization target is essential. + </p> + <p> + In complex systems, saturation can be supplemented with higher-level load measurement: can your service properly handle double the traffic, handle only 10% more traffic, or handle even less traffic than it currently receives? For very simple services that have no parameters that alter the complexity of the request (e.g., "Give me a nonce" or "I need a globally unique monotonic integer") that rarely change configuration, a static value from a load test might be adequate. As discussed in the previous paragraph, however, most services need to use indirect signals like CPU utilization or network bandwidth that have a known upper bound. Latency increases are often a leading indicator of saturation. Measuring your 99th percentile response time over some small window (e.g., one minute) can give a very early signal of saturation. + </p> + <p> + Finally, saturation is also concerned with predictions of impending saturation, such as "It looks like your database will fill its hard drive in 4 hours." + </p> + </dd> + </dl> + <p> + If you measure all four golden signals and page a human when one signal is problematic (or, in the case of saturation, nearly problematic), your service will be at least decently covered by monitoring. + </p> + </section> + <section data-type="sect1" id="worrying-about-your-tail-or-instrumentation-and-performance-Yms9Ck"> + <h2> + Worrying About Your Tail (or, Instrumentation and Performance) + </h2> + <p> + <a data-type="indexterm" data-primary="monitoring distributed systems" data-secondary="instrumentation and performance" id="id-zdCxSGFQCy"></a><a data-type="indexterm" data-primary="performance" data-secondary="monitoring" id="id-yYCyFpFdCr"></a>When building a monitoring system from scratch, it’s tempting to design a system based upon the mean of some quantity: the mean latency, the mean CPU usage of your nodes, or the mean fullness of your databases. The danger presented by the latter two cases is obvious: CPUs and databases can easily be utilized in a very imbalanced way. The same holds for latency. If you run a web service with an average latency of 100 ms at 1,000 requests per second, 1% of requests might easily take 5 seconds.<sup><a data-type="noteref" id="id-QQLuAIXFxCz-marker" href="#id-QQLuAIXFxCz">23</a></sup> If your users depend on several such web services to render their page, the 99th percentile of one backend can easily become the median response of your <span>frontend</span>. + </p> + <p> + The simplest way to differentiate between a slow average and a very slow "tail" of requests is to collect request counts bucketed by latencies (suitable for rendering a histogram), rather than actual latencies: how many requests did I serve that took between 0 ms and 10 ms, between 10 ms and 30 ms, between 30 ms and 100 ms, between 100 ms and 300 ms, and so on? Distributing the histogram boundaries approximately exponentially (in this case by factors of roughly 3) is often an easy way to visualize the distribution of your requests. + </p> + </section> + <section data-type="sect1" id="choosing-an-appropriate-resolution-for-measurements-vJsBsE"> + <h2> + Choosing an Appropriate Resolution for Measurements + </h2> + <p> + <a data-type="indexterm" data-primary="monitoring distributed systems" data-secondary="resolution" id="id-yYCASpFxsr"></a>Different aspects of a system should be measured with different levels of granularity. For example: + </p> + <ul> + <li>Observing CPU load over the time span of a minute won’t reveal even quite long-lived spikes that drive high tail latencies. + </li> + <li>On the other hand, for a web service targeting no more than 9 hours aggregate downtime per year (99.9% annual uptime), probing for a 200 (success) status more than once or twice a minute is probably unnecessarily frequent. + </li> + <li>Similarly, checking hard drive fullness for a service targeting 99.9% availability more than once every 1–2 minutes is probably unnecessary. + </li> + </ul> + <p> + Take care in how you structure the granularity of your measurements. Collecting per-second measurements of CPU load might yield interesting data, but such frequent measurements may be very expensive to collect, store, and analyze. If your monitoring goal calls for high resolution but doesn’t require extremely low latency, you can reduce these costs by performing internal sampling on the server, then configuring an external system to collect and aggregate that distribution over time or across servers. You might: + </p> + <ol> + <li>Record the current CPU utilization each second. + </li> + <li>Using buckets of 5% granularity, increment the appropriate CPU utilization bucket each second. + </li> + <li>Aggregate those values every minute. + </li> + </ol> + <p> + This strategy allows you to observe brief CPU hotspots without incurring very high cost due to collection and retention. + </p> + </section> + <section data-type="sect1" id="as-simple-as-possible-no-simpler-lqskHx"> + <h2> + As Simple as Possible, No Simpler + </h2> + <p> + <a data-type="indexterm" data-primary="monitoring distributed systems" data-secondary="avoiding complexity in" id="id-VMCPSrFpHm"></a>Piling all these requirements on top of each other can add up to a very complex monitoring system—your system might end up with the following levels of complexity: + </p> + <ul> + <li>Alerts on different latency thresholds, at different percentiles, on all kinds of different metrics + </li> + <li>Extra code to detect and expose possible causes + </li> + <li>Associated dashboards for each of these possible causes + </li> + </ul> + <p> + The sources of potential complexity are never-ending. Like all software systems, monitoring can become so complex that it’s fragile, complicated to change, and a maintenance burden. + </p> + <p> + Therefore, design your monitoring system with an eye toward simplicity. In choosing what to monitor, keep the following guidelines in mind: + </p> + <ul> + <li>The rules that catch real incidents most often should be as simple, predictable, and reliable as possible. + </li> + <li>Data collection, aggregation, and alerting configuration that is rarely exercised (e.g., less than once a quarter for some SRE teams) should be up for removal. + </li> + <li>Signals that are collected, but not exposed in any prebaked dashboard nor used by any alert, are candidates for removal. + </li> + </ul> + <p> + In Google’s experience, basic collection and aggregation of metrics, paired with alerting and dashboards, has worked well as a relatively standalone system. (In fact Google’s monitoring system is broken up into several binaries, but typically people learn about all aspects of these binaries.) It can be tempting to combine monitoring with other aspects of inspecting complex systems, such as detailed system profiling, single-process debugging, tracking details about exceptions or crashes, load testing, log collection and analysis, or traffic inspection. While most of these subjects share commonalities with basic monitoring, blending together too many results in overly complex and fragile systems. As in many other aspects of software engineering, maintaining distinct systems with clear, simple, loosely coupled points of integration is a better strategy (for example, using web APIs for pulling summary data in a format that can remain constant over an extended period of time). + </p> + </section> + <section data-type="sect1" id="tying-these-principles-together-nqsJfw"> + <h2> + Tying These Principles Together + </h2> + <p> + The principles discussed in this chapter can be tied together into a philosophy on monitoring and alerting that’s widely endorsed and followed within Google SRE teams. While this monitoring philosophy is a bit aspirational, it’s a good starting point for writing or reviewing a new alert, and it can help your organization ask the right questions, regardless of the size of your organization or the complexity of your service or system. + </p> + <p> + <a data-type="indexterm" data-primary="monitoring distributed systems" data-secondary="creating rules for" id="id-wqC7SDIvfj"></a>When creating rules for monitoring and alerting, asking the following questions can help you avoid false positives and pager burnout:<sup><a data-type="noteref" id="id-a82udF8IBfx-marker" href="#id-a82udF8IBfx">24</a></sup> + </p> + <ul> + <li>Does this rule detect <em>an otherwise undetected condition</em> that is urgent, actionable, and actively or imminently user-visible?<sup><a data-type="noteref" id="id-0vYuEFpSjSMtLfG-marker" href="#id-0vYuEFpSjSMtLfG">25</a></sup> + </li> + <li>Will I ever be able to ignore this alert, knowing it’s benign? When and why will I be able to ignore this alert, and how can I avoid this scenario? + </li> + <li>Does this alert definitely indicate that users are being negatively affected? Are there detectable cases in which users aren’t being negatively impacted, such as drained traffic or test deployments, that should be filtered out? + </li> + <li>Can I take action in response to this alert? Is that action urgent, or could it wait until morning? Could the action be safely automated? Will that action be a long-term fix, or just a short-term workaround? + </li> + <li>Are other people getting paged for this issue, therefore rendering at least one of the pages unnecessary? + </li> + </ul> + <p> + <a data-type="indexterm" data-primary="monitoring distributed systems" data-secondary="monitoring philosophy" id="id-PnCpSwhJfa"></a>These questions reflect a fundamental philosophy on pages and pagers: + </p> + <ul> + <li>Every time the pager goes off, I should be able to react with a sense of urgency. I can only react with a sense of urgency a few times a day before I become fatigued. + </li> + <li>Every page should be actionable. + </li> + <li>Every page response should require intelligence. If a page merely merits a robotic response, it shouldn’t be a page. + </li> + <li>Pages should be about a novel problem or an event that hasn’t been seen before. + </li> + </ul> + <p> + Such a perspective dissipates certain distinctions: if a page satisfies the preceding four bullets, it’s irrelevant whether the page is triggered by white-box or black-box monitoring. This perspective also amplifies certain distinctions: it’s better to spend much more effort on catching symptoms than causes; when it comes to causes, only worry about very definite, very imminent causes. + </p> + </section> + <section data-type="sect1" id="monitoring-for-the-long-term-NbsNS8"> + <h2> + Monitoring for the Long Term + </h2> + <p> + <a data-type="indexterm" data-primary="monitoring distributed systems" data-secondary="challenges of" id="id-wqC7SPFMSj"></a>In modern production systems, monitoring systems track an ever-evolving system with changing software architecture, load characteristics, and performance targets. An alert that’s currently exceptionally rare and hard to automate might become frequent, perhaps even meriting a hacked-together script to resolve it. At this point, someone should find and eliminate the root causes of the problem; if such resolution isn’t possible, the alert response deserves to be fully automated. + </p> + <p> + It’s important that decisions about monitoring be made with long-term goals in mind. Every page that happens today distracts a human from improving the system for tomorrow, so there is often a case for taking a short-term hit to availability or performance in order to improve the long-term outlook for the system. Let’s take a look at two case studies that illustrate this trade-off. + </p> + <section data-type="sect2" id="bigtable-sre-a-tale-of-over-alerting-dbsXtjSM"> + + <p> + <a data-type="indexterm" id="MDSbig6" data-primary="monitoring distributed systems" data-secondary="case studies"></a><a data-type="indexterm" data-primary="Bigtable" id="id-XmCpFOFytySv"></a>Google’s internal infrastructure is typically offered and measured against a service level objective (SLO; see <a data-type="xref" href="http://fakehost/sre/sre-book/chapters/service-level-objectives">Service Level Objectives</a>). Many years ago, the Bigtable service’s SLO was based on a synthetic well-behaved client’s mean performance. Because of problems in Bigtable and lower layers of the storage stack, the mean performance was driven by a "large" tail: the worst 5% of requests were often significantly slower than the rest. + </p> + <p> + Email alerts were triggered as the SLO approached, and paging alerts were triggered when the SLO was exceeded. Both types of alerts were firing voluminously, consuming unacceptable amounts of engineering time: the team spent significant amounts of time triaging the alerts to find the few that were really actionable, and we often missed the problems that actually affected users, because so few of them did. Many of the pages were non-urgent, due to well-understood problems in the infrastructure, and had either rote responses or received no response. + </p> + <p> + To remedy the situation, the team used a three-pronged approach: while making great efforts to improve the performance of Bigtable, we also temporarily dialed back our SLO target, using the 75th percentile request latency. We also disabled email alerts, as there were so many that spending time diagnosing them was infeasible. + </p> + <p> + This strategy gave us enough breathing room to actually fix the longer-term problems in Bigtable and the lower layers of the storage stack, rather than constantly fixing tactical problems. On-call engineers could actually accomplish work when they weren’t being kept up by pages at all hours. Ultimately, temporarily backing off on our alerts allowed us to make faster progress toward a better service. + </p> + </section> + <section data-type="sect2" id="gmail-predictable-scriptable-responses-from-humans-BVs1h4SD"> + + <p> + <a data-type="indexterm" data-primary="Gmail" id="id-XmC9SOFZhySv"></a>In the very early days of Gmail, the service was built on a retrofitted distributed process management system called Workqueue, which was originally created for batch processing of pieces of the search index. Workqueue was "adapted" to long-lived processes and subsequently applied to Gmail, but certain bugs in the relatively opaque codebase in the scheduler proved hard to beat. + </p> + <p> + At that time, the Gmail monitoring was structured such that alerts fired when individual tasks were “de-scheduled” by Workqueue. This setup was less than ideal because even at that time, Gmail had many, many thousands of tasks, each task representing a fraction of a percent of our users. We cared deeply about providing a good user experience for Gmail users, but such an alerting setup was unmaintainable. + </p> + <p> + To address this problem, Gmail SRE built a tool that helped “poke” the scheduler in just the right way to minimize impact to users. The team had several discussions about whether or not we should simply automate the entire loop from detecting the problem to nudging the rescheduler, until a better long-term solution was achieved, but some worried this kind of workaround would delay a real fix. + </p> + <p> + This kind of tension is common within a team, and often reflects an underlying mistrust of the team’s self-discipline: while some team members want to implement a “hack” to allow time for a proper fix, others worry that a hack will be forgotten or that the proper fix will be deprioritized indefinitely. This concern is credible, as it’s easy to build layers of unmaintainable technical debt by patching over problems instead of making real fixes. Managers and technical leaders play a key role in implementing true, long-term fixes by supporting and prioritizing potentially time-consuming long-term fixes even when the initial “pain” of paging subsides. + </p> + <p> + Pages with rote, algorithmic responses should be a red flag. Unwillingness on the part of your team to automate such pages implies that the team lacks confidence that they can clean up their technical debt. This is a major problem worth escalating.<a data-type="indexterm" data-primary data-startref="MDSbig6" id="id-oPCASqT2hLSk"></a> + </p> + </section> + <section data-type="sect2" id="the-long-run-MQsWTMS7"> + + <p> + <a data-type="indexterm" data-primary="monitoring distributed systems" data-secondary="short- vs. long-term availability" id="id-jyCxSoFETNSd"></a>A common theme connects the previous examples of Bigtable and Gmail: a tension between short-term and long-term availability. Often, sheer force of effort can help a rickety system achieve high availability, but this path is usually short-lived and fraught with burnout and dependence on a small number of heroic team members. Taking a controlled, short-term decrease in availability is often a painful, but strategic trade for the long-run stability of the system. It’s important not to think of every page as an event in isolation, but to consider whether the overall <em>level</em> of paging leads toward a healthy, appropriately available system with a healthy, viable team and long-term outlook. We review statistics about page frequency (usually expressed as incidents per shift, where an incident might be composed of a few related pages) in quarterly reports with management, ensuring that decision makers are kept up to date on the pager load and overall health of their teams. + </p> + </section> + </section> + <section data-type="sect1" id="conclusion-8ksvFj"> + <h2> + Conclusion + </h2> + <p> + A healthy monitoring and alerting pipeline is simple and easy to reason about. It focuses primarily on symptoms for paging, reserving cause-oriented heuristics to serve as aids to debugging problems. Monitoring symptoms is easier the further "up" your stack you monitor, though monitoring saturation and performance of subsystems such as databases often must be performed directly on the subsystem itself. Email alerts are of very limited value and tend to easily become overrun with noise; instead, you should favor a dashboard that monitors all ongoing subcritical problems for the sort of information that typically ends up in email alerts. A dashboard might also be paired with a log, in order to analyze historical correlations. + </p> + <p> + Over the long haul, achieving a successful on-call rotation and product includes choosing to alert on symptoms or imminent real problems, adapting your targets to goals that are actually achievable, and making sure that your monitoring supports rapid diagnosis. + </p> + </section> + + </section>
\ No newline at end of file diff --git a/test/test-pages/google-sre-book-1/source.html b/test/test-pages/google-sre-book-1/source.html new file mode 100644 index 0000000..6fb2024 --- /dev/null +++ b/test/test-pages/google-sre-book-1/source.html @@ -0,0 +1,742 @@ +<!DOCTYPE html> +<html lang="en" xmlns="http://www.w3.org/1999/xhtml" xml:lang="en"> + <head> + <meta charset="utf-8" /> + <meta content="initial-scale=1, minimum-scale=1, width=device-width" name="viewport" /> + <title> + Google - Site Reliability Engineering + </title> + <meta name="referrer" content="no-referrer" /> + <link rel="apple-touch-icon-precomposed" sizes="180x180" href="https://lh3.googleusercontent.com/Yf2DCX8RKda6r4Jml9DLMByS2zQCBFs3kQpvBfN8UgIh4YVWIYSYIQOoTxJriyuM26cT5PDjyEb5aynDQ0Xyz46yHKnfg8JlUbDW" /> + <link rel="stylesheet" href="//fonts.googleapis.com/css?family=Google+Sans:400|Roboto:400,400italic,500,500italic,700,700italic|Roboto+Mono:400,500,700|Material+Icons" /> + <link rel="icon" type="image/png" sizes="32x32" href="https://lh3.googleusercontent.com/Yf2DCX8RKda6r4Jml9DLMByS2zQCBFs3kQpvBfN8UgIh4YVWIYSYIQOoTxJriyuM26cT5PDjyEb5aynDQ0Xyz46yHKnfg8JlUbDW" /> + <link rel="icon" type="image/png" sizes="16x16" href="https://lh3.googleusercontent.com/Yf2DCX8RKda6r4Jml9DLMByS2zQCBFs3kQpvBfN8UgIh4YVWIYSYIQOoTxJriyuM26cT5PDjyEb5aynDQ0Xyz46yHKnfg8JlUbDW" /> + <link rel="shortcut icon" href="https://lh3.googleusercontent.com/Yf2DCX8RKda6r4Jml9DLMByS2zQCBFs3kQpvBfN8UgIh4YVWIYSYIQOoTxJriyuM26cT5PDjyEb5aynDQ0Xyz46yHKnfg8JlUbDW" /> + <link href="/sre/sre-book/static/css/index.min.css?cache=0ffc48d" rel="stylesheet" /> + <script> + <![CDATA[ + (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){ + (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o), + m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m) + })(window,document,'script','https://www.google-analytics.com/analytics.js','ga'); + + ga('create', 'UA-75468017-1', 'auto'); + ga('send', 'pageview'); + ]]> + </script> + <script src="/sre/sre-book/static/js/detect.min.js?cache=4cb778b"></script> + </head> + <body> + <main> + <div ng-controller="HeaderCtrl as headerCtrl"> + <div id="curtain" class="menu-closed"></div> + <div class="header clearfix"> + <a id="burger-menu" class="expand"></a> + <h2 class="chapter-title"> + Chapter 6 - Monitoring Distributed Systems + </h2> + </div> + <div id="overlay-element" class="expands"> + <div class="logo"> + <a href="https://www.google.com"><img src="https://lh3.googleusercontent.com/YoVRtLOHMSRYQZ3OhFL8RIamcjFYbmQXX4oAQx02MRqqY9zlKNvsuZpS73khXiOqTH3qrFW27VrERJJIHTjPk-tAh46q8-Fd4w6qlw" alt="Google" /></a> + </div> + <ol id="drop-down" class="dropdown-content hide"> + <li> + <a class="menu-buttons" href="/sre/sre-book/toc/">Table of Contents</a> + </li> + <li> + <a href="/sre/sre-book/chapters/foreword" class="menu-buttons">Foreword</a> + </li> + <li> + <a href="/sre/sre-book/chapters/preface" class="menu-buttons">Preface</a> + </li> + <li> + <a href="/sre/sre-book/chapters/part1" class="menu-buttons">Part I - Introduction</a> + </li> + <li> + <a href="/sre/sre-book/chapters/introduction" class="menu-buttons">1. Introduction</a> + </li> + <li> + <a href="/sre/sre-book/chapters/production-environment" class="menu-buttons">2. The Production Environment at Google, from the Viewpoint of an SRE</a> + </li> + <li> + <a href="/sre/sre-book/chapters/part2" class="menu-buttons">Part II - Principles</a> + </li> + <li> + <a href="/sre/sre-book/chapters/embracing-risk" class="menu-buttons">3. Embracing Risk</a> + </li> + <li> + <a href="/sre/sre-book/chapters/service-level-objectives" class="menu-buttons">4. Service Level Objectives</a> + </li> + <li> + <a href="/sre/sre-book/chapters/eliminating-toil" class="menu-buttons">5. Eliminating Toil</a> + </li> + <li class="active"> + <a href="/sre/sre-book/chapters/monitoring-distributed-systems" class="menu-buttons">6. Monitoring Distributed Systems</a> + </li> + <li> + <a href="/sre/sre-book/chapters/automation-at-google" class="menu-buttons">7. The Evolution of Automation at Google</a> + </li> + <li> + <a href="/sre/sre-book/chapters/release-engineering" class="menu-buttons">8. Release Engineering</a> + </li> + <li> + <a href="/sre/sre-book/chapters/simplicity" class="menu-buttons">9. Simplicity</a> + </li> + <li> + <a href="/sre/sre-book/chapters/part3" class="menu-buttons">Part III - Practices</a> + </li> + <li> + <a href="/sre/sre-book/chapters/practical-alerting" class="menu-buttons">10. Practical Alerting</a> + </li> + <li> + <a href="/sre/sre-book/chapters/being-on-call" class="menu-buttons">11. Being On-Call</a> + </li> + <li> + <a href="/sre/sre-book/chapters/effective-troubleshooting" class="menu-buttons">12. Effective Troubleshooting</a> + </li> + <li> + <a href="/sre/sre-book/chapters/emergency-response" class="menu-buttons">13. Emergency Response</a> + </li> + <li> + <a href="/sre/sre-book/chapters/managing-incidents" class="menu-buttons">14. Managing Incidents</a> + </li> + <li> + <a href="/sre/sre-book/chapters/postmortem-culture" class="menu-buttons">15. Postmortem Culture: Learning from Failure</a> + </li> + <li> + <a href="/sre/sre-book/chapters/tracking-outages" class="menu-buttons">16. Tracking Outages</a> + </li> + <li> + <a href="/sre/sre-book/chapters/testing-reliability" class="menu-buttons">17. Testing for Reliability</a> + </li> + <li> + <a href="/sre/sre-book/chapters/software-engineering-in-sre" class="menu-buttons">18. Software Engineering in SRE</a> + </li> + <li> + <a href="/sre/sre-book/chapters/load-balancing-frontend" class="menu-buttons">19. Load Balancing at the Frontend</a> + </li> + <li> + <a href="/sre/sre-book/chapters/load-balancing-datacenter" class="menu-buttons">20. Load Balancing in the Datacenter</a> + </li> + <li> + <a href="/sre/sre-book/chapters/handling-overload" class="menu-buttons">21. Handling Overload</a> + </li> + <li> + <a href="/sre/sre-book/chapters/addressing-cascading-failures" class="menu-buttons">22. Addressing Cascading Failures</a> + </li> + <li> + <a href="/sre/sre-book/chapters/managing-critical-state" class="menu-buttons">23. Managing Critical State: Distributed Consensus for Reliability</a> + </li> + <li> + <a href="/sre/sre-book/chapters/distributed-periodic-scheduling" class="menu-buttons">24. Distributed Periodic Scheduling with Cron</a> + </li> + <li> + <a href="/sre/sre-book/chapters/data-processing-pipelines" class="menu-buttons">25. Data Processing Pipelines</a> + </li> + <li> + <a href="/sre/sre-book/chapters/data-integrity" class="menu-buttons">26. Data Integrity: What You Read Is What You Wrote</a> + </li> + <li> + <a href="/sre/sre-book/chapters/reliable-product-launches" class="menu-buttons">27. Reliable Product Launches at Scale</a> + </li> + <li> + <a href="/sre/sre-book/chapters/part4" class="menu-buttons">Part IV - Management</a> + </li> + <li> + <a href="/sre/sre-book/chapters/accelerating-sre-on-call" class="menu-buttons">28. Accelerating SREs to On-Call and Beyond</a> + </li> + <li> + <a href="/sre/sre-book/chapters/dealing-with-interrupts" class="menu-buttons">29. Dealing with Interrupts</a> + </li> + <li> + <a href="/sre/sre-book/chapters/operational-overload" class="menu-buttons">30. Embedding an SRE to Recover from Operational Overload</a> + </li> + <li> + <a href="/sre/sre-book/chapters/communication-and-collaboration" class="menu-buttons">31. Communication and Collaboration in SRE</a> + </li> + <li> + <a href="/sre/sre-book/chapters/evolving-sre-engagement-model" class="menu-buttons">32. The Evolving SRE Engagement Model</a> + </li> + <li> + <a href="/sre/sre-book/chapters/part5" class="menu-buttons">Part V - Conclusions</a> + </li> + <li> + <a href="/sre/sre-book/chapters/lessons-learned" class="menu-buttons">33. Lessons Learned from Other Industries</a> + </li> + <li> + <a href="/sre/sre-book/chapters/conclusion" class="menu-buttons">34. Conclusion</a> + </li> + <li> + <a href="/sre/sre-book/chapters/availability-table" class="menu-buttons">Appendix A. Availability Table</a> + </li> + <li> + <a href="/sre/sre-book/chapters/service-best-practices" class="menu-buttons">Appendix B. A Collection of Best Practices for Production Services</a> + </li> + <li> + <a href="/sre/sre-book/chapters/incident-document" class="menu-buttons">Appendix C. Example Incident State Document</a> + </li> + <li> + <a href="/sre/sre-book/chapters/postmortem" class="menu-buttons">Appendix D. Example Postmortem</a> + </li> + <li> + <a href="/sre/sre-book/chapters/launch-checklist" class="menu-buttons">Appendix E. Launch Coordination Checklist</a> + </li> + <li> + <a href="/sre/sre-book/chapters/production-meeting" class="menu-buttons">Appendix F. Example Production Meeting Minutes</a> + </li> + <li> + <a href="/sre/sre-book/chapters/bibliography" class="menu-buttons">Bibliography</a> + </li> + </ol> + </div> + </div> + <div id="maia-main" role="main"> + <div class="maia-teleport" id="content"></div> + <div class="content"> + <section data-type="chapter" id="chapter_monitoring"> + <h1 class="heading jumptargets"> + Monitoring Distributed Systems + </h1> + <p class="byline author"> + Written by Rob Ewaschuk<br /> + Edited by Betsy Beyer + </p> + <p> + Google’s SRE teams have some basic principles and best practices for building successful monitoring and alerting systems. This chapter offers guidelines for what issues should interrupt a human via a page, and how to deal with issues that aren’t serious enough to trigger a page. + </p> + <section data-type="sect1" id="definitions-2ksZhN"> + <h1 class="heading jumptargets"> + Definitions + </h1> + <p> + <a data-type="indexterm" data-primary="monitoring distributed systems" data-secondary="terminology" id="id-DnC1SWFMhD"></a>There’s no uniformly shared vocabulary for discussing all topics related to monitoring. Even within Google, usage of the following terms varies, but the most common interpretations are listed here. + </p> + <dl> + <dt class="subheaders jumptargets" id="monitoring"> + Monitoring + </dt> + <dd> + <p> + Collecting, processing, aggregating, and displaying real-time quantitative data about a system, such as query counts and types, error counts and types, processing times, and server lifetimes. + </p> + </dd> + <dt class="subheaders jumptargets" id="white-box-monitoring"> + White-box monitoring + </dt> + <dd> + <p> + <a data-type="indexterm" data-primary="white-box monitoring" id="id-9nCjSDS4tZILhX"></a>Monitoring based on metrics exposed by the internals of the system, including logs, interfaces like the Java Virtual Machine Profiling Interface, or an HTTP handler that emits internal statistics. + </p> + </dd> + <dt class="subheaders jumptargets" id="black-box-monitoring"> + Black-box monitoring + </dt> + <dd> + <p> + <a data-type="indexterm" data-primary="black-box monitoring" id="id-zdCxSrSgTWIdhb"></a>Testing externally visible behavior as a user would see it. + </p> + </dd> + <dt class="subheaders jumptargets" id="dashboard"> + Dashboard + </dt> + <dd> + <p> + <a data-type="indexterm" data-primary="dashboards" data-secondary="defined" id="id-VMCPS2SribIkh4"></a>An application (usually web-based) that provides a summary view of a service’s core metrics. A dashboard may have filters, selectors, and so on, but is prebuilt to expose the metrics most important to its users. The dashboard might also display team information such as ticket queue length, a list of high-priority bugs, the current on-call engineer for a given area of responsibility, or recent pushes. + </p> + </dd> + <dt class="subheaders jumptargets" id="alert"> + Alert + </dt> + <dd> + <p> + <a data-type="indexterm" data-primary="alerts" data-secondary="defined" id="id-wqC7SvSPUAIVhQ"></a>A notification intended to be read by a human and that is pushed to a system such as a bug or ticket queue, an email alias, or a pager. Respectively, these alerts are classified as <em>tickets</em>, <em>email alerts</em>,<sup><a class="jumptarget" data-type="noteref" id="id-LvQuvtYS7UvI8h4-marker" href="#id-LvQuvtYS7UvI8h4">22</a></sup> and <em>pages</em>. + </p> + </dd> + <dt class="subheaders jumptargets" id="root-cause"> + Root cause + </dt> + <dd> + <p> + <a data-type="indexterm" data-primary="root cause" data-secondary="defined" id="id-PnCpSaSKsgIjho"></a>A defect in a software or human system that, if repaired, instills confidence that this event won’t happen again in the same way. A given incident might have multiple root causes: for example, perhaps it was caused by a combination of insufficient process automation, software that crashed on bogus input, <em>and</em> insufficient testing of the script used to generate the configuration. Each of these factors might stand alone as a root cause, and each should be repaired. + </p> + </dd> + <dt class="subheaders jumptargets" id="node-and-machine"> + Node and machine + </dt> + <dd> + <p> + <a data-type="indexterm" data-primary="machines" data-secondary="defined" id="id-XmC9SkSlfnI1hK"></a>Used interchangeably to indicate a single instance of a running kernel in either a physical server, virtual machine, or container. There might be multiple <em>services</em> worth monitoring on a single machine. The services may either be: + </p> + <ul> + <li>Related to each other: for example, a caching server and a web server + </li> + <li>Unrelated services sharing hardware: for example, a code repository and a master for a configuration system like <a href="https://puppetlabs.com/puppet/puppet-open-source" target="_blank">Puppet</a> or <a href="https://www.chef.io/chef/" target="_blank">Chef</a> + </li> + </ul> + </dd> + <dt class="subheaders jumptargets" id="push"> + Push + </dt> + <dd> + <p> + Any change to a service’s running software or its configuration. + </p> + </dd> + </dl> + </section> + <section data-type="sect1" id="why-monitor-pWsBTZ"> + <h1 class="heading jumptargets"> + Why Monitor? + </h1> + <p> + <a data-type="indexterm" data-primary="monitoring distributed systems" data-secondary="benefits of monitoring" id="id-kVCkSpFnTl"></a>There are many reasons to monitor a system, including: + </p> + <dl> + <dt class="subheaders jumptargets" id="analyzing-long-term-trends"> + Analyzing long-term trends + </dt> + <dd> + <p> + How big is my database and how fast is it growing? How quickly is my daily-active user count growing? + </p> + </dd> + <dt class="subheaders jumptargets" id="comparing-over-time-or-experiment-groups"> + Comparing over time or experiment groups + </dt> + <dd> + <p> + Are queries faster with Acme Bucket of Bytes 2.72 versus Ajax DB 3.14? How much better is my memcache hit rate with an extra node? Is my site slower than it was last week? + </p> + </dd> + <dt class="subheaders jumptargets" id="alerting"> + Alerting + </dt> + <dd> + <p> + Something is broken, and somebody needs to fix it right now! Or, something might break soon, so somebody should look soon. + </p> + </dd> + <dt class="subheaders jumptargets" id="building-dashboards"> + Building dashboards + </dt> + <dd> + <p> + <a data-type="indexterm" data-primary="dashboards" data-secondary="benefits of" id="id-rjCXSOS0iDIGT8"></a>Dashboards should answer basic questions about your service, and normally include some form of the four golden signals (discussed in <a data-type="xref" href="#xref_monitoring_golden-signals">The Four Golden Signals</a>). + </p> + </dd> + <dt class="subheaders jumptargets" id="conducting-ad-hoc-retrospective-analysis-ie-debugging"> + Conducting <i class="italic">ad hoc</i> retrospective analysis (i.e., debugging) + </dt> + <dd> + <p> + Our latency just shot up; what else happened around the same time? + </p> + </dd> + </dl> + <p> + System monitoring is also helpful in supplying raw input into business analytics and in facilitating analysis of security breaches. Because this book focuses on the engineering domains in which SRE has particular expertise, we won’t discuss these applications of monitoring here. + </p> + <p> + Monitoring and alerting enables a system to tell us when it’s broken, or perhaps to tell us what’s about to break. When the system isn’t able to automatically fix itself, we want a human to investigate the alert, determine if there’s a real problem at hand, mitigate the problem, and determine the root cause of the problem. Unless you’re performing security auditing on very narrowly scoped components of a system, you should never trigger an alert simply because "something seems a bit weird." + </p> + <p> + Paging a human is a quite expensive use of an employee’s time. If an employee is at work, a page interrupts their workflow. If the employee is at home, a page interrupts their personal time, and perhaps even their sleep. When pages occur too frequently, employees second-guess, skim, or even ignore incoming alerts, sometimes even ignoring a "real" page that’s masked by the noise. Outages can be prolonged because other noise interferes with a rapid diagnosis and fix. Effective alerting systems have good signal and very low noise. + </p> + </section> + <section data-type="sect1" id="setting-reasonable-expectations-for-monitoring-o8svcM"> + <h1 class="heading jumptargets"> + Setting Reasonable Expectations for Monitoring + </h1> + <p> + <a data-type="indexterm" data-primary="monitoring distributed systems" data-secondary="setting expectations for" id="id-4nCqSYFQcE"></a>Monitoring a complex application is a significant engineering endeavor in and of itself. Even with substantial existing infrastructure for instrumentation, collection, display, and alerting in place, a Google SRE team with 10–12 members typically has one or sometimes two members whose primary assignment is to build and maintain monitoring systems for their service. This number has decreased over time as we generalize and centralize common monitoring infrastructure, but every SRE team typically has at least one “monitoring person.” (That being said, while it can be fun to have access to traffic graph dashboards and the like, SRE teams carefully avoid any situation that requires someone to “stare at a screen to watch for problems.”) + </p> + <p> + <a data-type="indexterm" data-primary="post hoc analysis" id="id-JnCDSjIVcG"></a>In general, Google has trended toward simpler and faster monitoring systems, with better tools for <em>post hoc</em> analysis. We avoid "magic" systems that try to learn thresholds or automatically detect causality. Rules that detect unexpected changes in end-user request rates are one counterexample; while these rules are still kept as simple as possible, they give a very quick detection of a very simple, specific, severe anomaly. Other uses of monitoring data such as capacity planning and traffic prediction can tolerate more fragility, and thus, more complexity. Observational experiments conducted over a very long time horizon (months or years) with a low sampling rate (hours or days) can also often tolerate more fragility because occasional missed samples won’t hide a long-running trend. + </p> + <p> + <a data-type="indexterm" data-primary="dependency hierarchies" id="id-9nCjSOtmcj"></a>Google SRE has experienced only limited success with complex dependency hierarchies. We seldom use rules such as, "If I know the database is slow, alert for a slow database; otherwise, alert for the website being generally slow." Dependency-reliant rules usually pertain to very stable parts of our system, such as our system for draining user traffic away from a datacenter. For example, "If a datacenter is drained, then don’t alert me on its latency" is one common datacenter alerting rule. Few teams at Google maintain complex dependency hierarchies because our infrastructure has a steady rate of continuous refactoring. + </p> + <p> + Some of the ideas described in this chapter are still aspirational: there is always room to move more rapidly from symptom to root cause(s), especially in ever-changing systems. So while this chapter sets out some goals for monitoring systems, and some ways to achieve these goals, it’s important that monitoring systems—especially the critical path from the onset of a production problem, through a page to a human, through basic triage and deep debugging—be kept simple and comprehensible by everyone on the team. + </p> + <p> + Similarly, to keep noise low and signal high, the elements of your monitoring system that direct to a pager need to be very simple and robust. Rules that generate alerts for humans should be simple to understand and represent a clear failure. + </p> + </section> + <section data-type="sect1" id="symptoms-versus-causes-g0sEi4"> + <h1 class="heading jumptargets"> + Symptoms Versus Causes + </h1> + <p> + <a data-type="indexterm" data-primary="monitoring distributed systems" data-secondary="symptoms vs. causes" id="id-JnCDSlFmiG"></a>Your monitoring system should address two questions: what’s broken, and why? + </p> + <p> + The "what’s broken" indicates the symptom; the "why" indicates a (possibly intermediate) cause. <a data-type="xref" href="#table_monitoring_symptoms">Table 6-1</a> lists some hypothetical symptoms and corresponding causes. + </p> + <table id="table_monitoring_symptoms" class="pagebreak-before"> + <caption class="jumptarget"> + <span class="label">Table 6-1.</span> Example symptoms and causes + </caption> + <thead> + <tr> + <th> + <strong>Symptom</strong> + </th> + <th> + <strong>Cause</strong> + </th> + </tr> + </thead> + <tbody> + <tr> + <td> + <p> + <strong>I’m serving HTTP 500s or 404s</strong> + </p> + </td> + <td> + <p> + Database servers are refusing connections + </p> + </td> + </tr> + <tr> + <td> + <p> + <strong>My responses are slow</strong> + </p> + </td> + <td> + <p> + CPUs are overloaded by a bogosort, or an Ethernet cable is crimped under a rack, visible as partial packet loss + </p> + </td> + </tr> + <tr> + <td> + <p> + <strong>Users in Antarctica aren’t receiving animated cat GIFs</strong> + </p> + </td> + <td> + <p> + Your Content Distribution Network hates scientists and felines, and thus blacklisted some client IPs + </p> + </td> + </tr> + <tr> + <td> + <p> + <strong>Private content is world-readable</strong> + </p> + </td> + <td> + <p> + A new software push caused ACLs to be forgotten and allowed all requests + </p> + </td> + </tr> + </tbody> + </table> + <p> + "What" versus "why" is one of the most important distinctions in writing good monitoring with maximum signal and minimum noise. + </p> + </section> + <section data-type="sect1" id="black-box-versus-white-box-q8sJuw"> + <h1 class="heading jumptargets"> + Black-Box Versus White-Box + </h1> + <p> + <a data-type="indexterm" data-primary="monitoring distributed systems" data-secondary="blackbox vs. whitebox" id="id-9nCjSvFVuj"></a><a data-type="indexterm" data-primary="white-box monitoring" id="id-ZbC1FMFEu7"></a><a data-type="indexterm" data-primary="black-box monitoring" id="id-zdCXIGFvuy"></a>We combine heavy use of white-box monitoring with modest but critical uses of black-box monitoring. The simplest way to think about black-box monitoring versus white-box monitoring is that black-box monitoring is symptom-oriented and represents active—not predicted—problems: "The system isn’t working correctly, right now." White-box monitoring depends on the ability to inspect the innards of the system, such as logs or HTTP endpoints, with instrumentation. White-box monitoring therefore allows detection of imminent problems, failures masked by retries, and so forth. + </p> + <p> + Note that in a multilayered system, one person’s symptom is another person’s cause. For example, suppose that a database’s performance is slow. Slow database reads are a symptom for the database SRE who detects them. However, for the frontend SRE observing a slow website, the same slow database reads are a cause. Therefore, white-box monitoring is sometimes symptom-oriented, and sometimes cause-oriented, depending on just how informative your white-box is. + </p> + <p> + When collecting telemetry for debugging, white-box monitoring is essential. If web servers seem slow on database-heavy requests, you need to know both how fast the web server perceives the database to be, and how fast the database believes itself to be. Otherwise, you can’t distinguish an actually slow database server from a network problem between your web server and your database. + </p> + <p> + For paging, black-box monitoring has the key benefit of forcing discipline to only nag a human when a problem is both already ongoing and contributing to real symptoms. On the other hand, for not-yet-occurring but imminent problems, black-box monitoring is fairly useless. + </p> + </section> + <section data-type="sect1" id="xref_monitoring_golden-signals"> + <h1 class="heading jumptargets"> + The Four Golden Signals + </h1> + <p> + <a data-type="indexterm" data-primary="monitoring distributed systems" data-secondary="four golden signals of" id="id-ZbCxSMFjU7"></a>The four golden signals of monitoring are latency, traffic, errors, and saturation. If you can only measure four metrics of your user-facing system, focus on these four. + </p> + <dl> + <dt class="subheaders jumptargets" id="latency"> + Latency + </dt> + <dd> + <p> + <a data-type="indexterm" data-primary="service latency" data-secondary="monitoring for" id="id-yYCASJS9FKIWUb"></a><a data-type="indexterm" data-primary="latency" data-secondary="monitoring for" id="id-VMCpF2SXFbIwU4"></a><a data-type="indexterm" data-primary="request latency" id="id-rjCeIOSKFDIaU8"></a><a data-type="indexterm" data-primary="user requests" data-secondary="request latency monitoring" id="id-wqCDtvSGFAIMUQ"></a>The time it takes to service a request. It’s important to distinguish between the latency of successful requests and the latency of failed requests. For example, an HTTP 500 error triggered due to loss of connection to a database or other critical backend might be served very quickly; however, as an HTTP 500 error indicates a failed request, factoring 500s into your overall latency might result in misleading calculations. On the other hand, a slow error is even worse than a fast error! Therefore, it’s important to track error latency, as opposed to just filtering out errors. + </p> + </dd> + <dt class="subheaders jumptargets" id="traffic"> + Traffic + </dt> + <dd> + <p> + <a data-type="indexterm" data-primary="user requests" data-secondary="traffic analysis" id="id-rjCXSOSxtDIaU8"></a><a data-type="indexterm" data-primary="traffic analysis" id="id-wqC4FvSBtAIMUQ"></a>A measure of how much demand is being placed on your system, measured in a high-level system-specific metric. For a web service, this measurement is usually HTTP requests per second, perhaps broken out by the nature of the requests (e.g., static versus dynamic content). For an audio streaming system, this measurement might focus on network I/O rate or concurrent sessions. For a key-value storage system, this measurement might be transactions and retrievals per <span class="keep-together">second</span>. + </p> + </dd> + <dt class="subheaders jumptargets" id="errors"> + Errors + </dt> + <dd> + <p> + <a data-type="indexterm" data-primary="error rates" id="id-x1C4SjSlTLIMUJ"></a><a data-type="indexterm" data-primary="user requests" data-secondary="monitoring failures" id="id-PnCxFaS0TgIVUo"></a>The rate of requests that fail, either explicitly (e.g., HTTP 500s), implicitly (for example, an HTTP 200 success response, but coupled with the wrong content), or by policy (for example, "If you committed to one-second response times, any request over one second is an error"). Where protocol response codes are insufficient to express all failure conditions, secondary (internal) protocols may be necessary to track partial failure modes. Monitoring these cases can be drastically different: catching HTTP 500s at your load balancer can do a decent job of catching all completely failed requests, while only end-to-end system tests can detect that you’re serving the wrong content. + </p> + </dd> + <dt class="subheaders jumptargets" id="saturation"> + Saturation + </dt> + <dd> + <p> + <a data-type="indexterm" data-primary="saturation" id="id-OnCNS2S4iDIYU8"></a>How "full" your service is. A measure of your system fraction, emphasizing the resources that are most constrained (e.g., in a memory-constrained system, show memory; in an I/O-constrained system, show I/O). Note that many systems degrade in performance before they achieve 100% utilization, so having a utilization target is essential. + </p> + <p> + In complex systems, saturation can be supplemented with higher-level load measurement: can your service properly handle double the traffic, handle only 10% more traffic, or handle even less traffic than it currently receives? For very simple services that have no parameters that alter the complexity of the request (e.g., "Give me a nonce" or "I need a globally unique monotonic integer") that rarely change configuration, a static value from a load test might be adequate. As discussed in the previous paragraph, however, most services need to use indirect signals like CPU utilization or network bandwidth that have a known upper bound. Latency increases are often a leading indicator of saturation. Measuring your 99th percentile response time over some small window (e.g., one minute) can give a very early signal of saturation. + </p> + <p> + Finally, saturation is also concerned with predictions of impending saturation, such as "It looks like your database will fill its hard drive in 4 hours." + </p> + </dd> + </dl> + <p> + If you measure all four golden signals and page a human when one signal is problematic (or, in the case of saturation, nearly problematic), your service will be at least decently covered by monitoring. + </p> + </section> + <section data-type="sect1" id="worrying-about-your-tail-or-instrumentation-and-performance-Yms9Ck"> + <h1 class="heading jumptargets"> + Worrying About Your Tail (or, Instrumentation and Performance) + </h1> + <p> + <a data-type="indexterm" data-primary="monitoring distributed systems" data-secondary="instrumentation and performance" id="id-zdCxSGFQCy"></a><a data-type="indexterm" data-primary="performance" data-secondary="monitoring" id="id-yYCyFpFdCr"></a>When building a monitoring system from scratch, it’s tempting to design a system based upon the mean of some quantity: the mean latency, the mean CPU usage of your nodes, or the mean fullness of your databases. The danger presented by the latter two cases is obvious: CPUs and databases can easily be utilized in a very imbalanced way. The same holds for latency. If you run a web service with an average latency of 100 ms at 1,000 requests per second, 1% of requests might easily take 5 seconds.<sup><a class="jumptarget" data-type="noteref" id="id-QQLuAIXFxCz-marker" href="#id-QQLuAIXFxCz">23</a></sup> If your users depend on several such web services to render their page, the 99th percentile of one backend can easily become the median response of your <span class="keep-together">frontend</span>. + </p> + <p> + The simplest way to differentiate between a slow average and a very slow "tail" of requests is to collect request counts bucketed by latencies (suitable for rendering a histogram), rather than actual latencies: how many requests did I serve that took between 0 ms and 10 ms, between 10 ms and 30 ms, between 30 ms and 100 ms, between 100 ms and 300 ms, and so on? Distributing the histogram boundaries approximately exponentially (in this case by factors of roughly 3) is often an easy way to visualize the distribution of your requests. + </p> + </section> + <section data-type="sect1" id="choosing-an-appropriate-resolution-for-measurements-vJsBsE"> + <h1 class="heading jumptargets"> + Choosing an Appropriate Resolution for Measurements + </h1> + <p> + <a data-type="indexterm" data-primary="monitoring distributed systems" data-secondary="resolution" id="id-yYCASpFxsr"></a>Different aspects of a system should be measured with different levels of granularity. For example: + </p> + <ul> + <li>Observing CPU load over the time span of a minute won’t reveal even quite long-lived spikes that drive high tail latencies. + </li> + <li>On the other hand, for a web service targeting no more than 9 hours aggregate downtime per year (99.9% annual uptime), probing for a 200 (success) status more than once or twice a minute is probably unnecessarily frequent. + </li> + <li>Similarly, checking hard drive fullness for a service targeting 99.9% availability more than once every 1–2 minutes is probably unnecessary. + </li> + </ul> + <p> + Take care in how you structure the granularity of your measurements. Collecting per-second measurements of CPU load might yield interesting data, but such frequent measurements may be very expensive to collect, store, and analyze. If your monitoring goal calls for high resolution but doesn’t require extremely low latency, you can reduce these costs by performing internal sampling on the server, then configuring an external system to collect and aggregate that distribution over time or across servers. You might: + </p> + <ol> + <li>Record the current CPU utilization each second. + </li> + <li>Using buckets of 5% granularity, increment the appropriate CPU utilization bucket each second. + </li> + <li>Aggregate those values every minute. + </li> + </ol> + <p> + This strategy allows you to observe brief CPU hotspots without incurring very high cost due to collection and retention. + </p> + </section> + <section data-type="sect1" id="as-simple-as-possible-no-simpler-lqskHx"> + <h1 class="heading jumptargets"> + As Simple as Possible, No Simpler + </h1> + <p> + <a data-type="indexterm" data-primary="monitoring distributed systems" data-secondary="avoiding complexity in" id="id-VMCPSrFpHm"></a>Piling all these requirements on top of each other can add up to a very complex monitoring system—your system might end up with the following levels of complexity: + </p> + <ul> + <li>Alerts on different latency thresholds, at different percentiles, on all kinds of different metrics + </li> + <li>Extra code to detect and expose possible causes + </li> + <li>Associated dashboards for each of these possible causes + </li> + </ul> + <p> + The sources of potential complexity are never-ending. Like all software systems, monitoring can become so complex that it’s fragile, complicated to change, and a maintenance burden. + </p> + <p> + Therefore, design your monitoring system with an eye toward simplicity. In choosing what to monitor, keep the following guidelines in mind: + </p> + <ul> + <li>The rules that catch real incidents most often should be as simple, predictable, and reliable as possible. + </li> + <li>Data collection, aggregation, and alerting configuration that is rarely exercised (e.g., less than once a quarter for some SRE teams) should be up for removal. + </li> + <li>Signals that are collected, but not exposed in any prebaked dashboard nor used by any alert, are candidates for removal. + </li> + </ul> + <p> + In Google’s experience, basic collection and aggregation of metrics, paired with alerting and dashboards, has worked well as a relatively standalone system. (In fact Google’s monitoring system is broken up into several binaries, but typically people learn about all aspects of these binaries.) It can be tempting to combine monitoring with other aspects of inspecting complex systems, such as detailed system profiling, single-process debugging, tracking details about exceptions or crashes, load testing, log collection and analysis, or traffic inspection. While most of these subjects share commonalities with basic monitoring, blending together too many results in overly complex and fragile systems. As in many other aspects of software engineering, maintaining distinct systems with clear, simple, loosely coupled points of integration is a better strategy (for example, using web APIs for pulling summary data in a format that can remain constant over an extended period of time). + </p> + </section> + <section data-type="sect1" id="tying-these-principles-together-nqsJfw"> + <h1 class="heading jumptargets"> + Tying These Principles Together + </h1> + <p> + The principles discussed in this chapter can be tied together into a philosophy on monitoring and alerting that’s widely endorsed and followed within Google SRE teams. While this monitoring philosophy is a bit aspirational, it’s a good starting point for writing or reviewing a new alert, and it can help your organization ask the right questions, regardless of the size of your organization or the complexity of your service or system. + </p> + <p> + <a data-type="indexterm" data-primary="monitoring distributed systems" data-secondary="creating rules for" id="id-wqC7SDIvfj"></a>When creating rules for monitoring and alerting, asking the following questions can help you avoid false positives and pager burnout:<sup><a class="jumptarget" data-type="noteref" id="id-a82udF8IBfx-marker" href="#id-a82udF8IBfx">24</a></sup> + </p> + <ul> + <li>Does this rule detect <em>an otherwise undetected condition</em> that is urgent, actionable, and actively or imminently user-visible?<sup><a class="jumptarget" data-type="noteref" id="id-0vYuEFpSjSMtLfG-marker" href="#id-0vYuEFpSjSMtLfG">25</a></sup> + </li> + <li>Will I ever be able to ignore this alert, knowing it’s benign? When and why will I be able to ignore this alert, and how can I avoid this scenario? + </li> + <li>Does this alert definitely indicate that users are being negatively affected? Are there detectable cases in which users aren’t being negatively impacted, such as drained traffic or test deployments, that should be filtered out? + </li> + <li>Can I take action in response to this alert? Is that action urgent, or could it wait until morning? Could the action be safely automated? Will that action be a long-term fix, or just a short-term workaround? + </li> + <li>Are other people getting paged for this issue, therefore rendering at least one of the pages unnecessary? + </li> + </ul> + <p> + <a data-type="indexterm" data-primary="monitoring distributed systems" data-secondary="monitoring philosophy" id="id-PnCpSwhJfa"></a>These questions reflect a fundamental philosophy on pages and pagers: + </p> + <ul> + <li>Every time the pager goes off, I should be able to react with a sense of urgency. I can only react with a sense of urgency a few times a day before I become fatigued. + </li> + <li>Every page should be actionable. + </li> + <li>Every page response should require intelligence. If a page merely merits a robotic response, it shouldn’t be a page. + </li> + <li>Pages should be about a novel problem or an event that hasn’t been seen before. + </li> + </ul> + <p> + Such a perspective dissipates certain distinctions: if a page satisfies the preceding four bullets, it’s irrelevant whether the page is triggered by white-box or black-box monitoring. This perspective also amplifies certain distinctions: it’s better to spend much more effort on catching symptoms than causes; when it comes to causes, only worry about very definite, very imminent causes. + </p> + </section> + <section data-type="sect1" id="monitoring-for-the-long-term-NbsNS8"> + <h1 class="heading jumptargets"> + Monitoring for the Long Term + </h1> + <p> + <a data-type="indexterm" data-primary="monitoring distributed systems" data-secondary="challenges of" id="id-wqC7SPFMSj"></a>In modern production systems, monitoring systems track an ever-evolving system with changing software architecture, load characteristics, and performance targets. An alert that’s currently exceptionally rare and hard to automate might become frequent, perhaps even meriting a hacked-together script to resolve it. At this point, someone should find and eliminate the root causes of the problem; if such resolution isn’t possible, the alert response deserves to be fully automated. + </p> + <p> + It’s important that decisions about monitoring be made with long-term goals in mind. Every page that happens today distracts a human from improving the system for tomorrow, so there is often a case for taking a short-term hit to availability or performance in order to improve the long-term outlook for the system. Let’s take a look at two case studies that illustrate this trade-off. + </p> + <section data-type="sect2" id="bigtable-sre-a-tale-of-over-alerting-dbsXtjSM"> + <h2 class="subheaders jumptargets"> + Bigtable SRE: A Tale of Over-Alerting + </h2> + <p> + <a data-type="indexterm" id="MDSbig6" data-primary="monitoring distributed systems" data-secondary="case studies"></a><a data-type="indexterm" data-primary="Bigtable" id="id-XmCpFOFytySv"></a>Google’s internal infrastructure is typically offered and measured against a service level objective (SLO; see <a data-type="xref" href="/sre/sre-book/chapters/service-level-objectives">Service Level Objectives</a>). Many years ago, the Bigtable service’s SLO was based on a synthetic well-behaved client’s mean performance. Because of problems in Bigtable and lower layers of the storage stack, the mean performance was driven by a "large" tail: the worst 5% of requests were often significantly slower than the rest. + </p> + <p> + Email alerts were triggered as the SLO approached, and paging alerts were triggered when the SLO was exceeded. Both types of alerts were firing voluminously, consuming unacceptable amounts of engineering time: the team spent significant amounts of time triaging the alerts to find the few that were really actionable, and we often missed the problems that actually affected users, because so few of them did. Many of the pages were non-urgent, due to well-understood problems in the infrastructure, and had either rote responses or received no response. + </p> + <p> + To remedy the situation, the team used a three-pronged approach: while making great efforts to improve the performance of Bigtable, we also temporarily dialed back our SLO target, using the 75th percentile request latency. We also disabled email alerts, as there were so many that spending time diagnosing them was infeasible. + </p> + <p> + This strategy gave us enough breathing room to actually fix the longer-term problems in Bigtable and the lower layers of the storage stack, rather than constantly fixing tactical problems. On-call engineers could actually accomplish work when they weren’t being kept up by pages at all hours. Ultimately, temporarily backing off on our alerts allowed us to make faster progress toward a better service. + </p> + </section> + <section data-type="sect2" id="gmail-predictable-scriptable-responses-from-humans-BVs1h4SD"> + <h2 class="subheaders jumptargets"> + Gmail: Predictable, Scriptable Responses from Humans + </h2> + <p> + <a data-type="indexterm" data-primary="Gmail" id="id-XmC9SOFZhySv"></a>In the very early days of Gmail, the service was built on a retrofitted distributed process management system called Workqueue, which was originally created for batch processing of pieces of the search index. Workqueue was "adapted" to long-lived processes and subsequently applied to Gmail, but certain bugs in the relatively opaque codebase in the scheduler proved hard to beat. + </p> + <p> + At that time, the Gmail monitoring was structured such that alerts fired when individual tasks were “de-scheduled” by Workqueue. This setup was less than ideal because even at that time, Gmail had many, many thousands of tasks, each task representing a fraction of a percent of our users. We cared deeply about providing a good user experience for Gmail users, but such an alerting setup was unmaintainable. + </p> + <p> + To address this problem, Gmail SRE built a tool that helped “poke” the scheduler in just the right way to minimize impact to users. The team had several discussions about whether or not we should simply automate the entire loop from detecting the problem to nudging the rescheduler, until a better long-term solution was achieved, but some worried this kind of workaround would delay a real fix. + </p> + <p> + This kind of tension is common within a team, and often reflects an underlying mistrust of the team’s self-discipline: while some team members want to implement a “hack” to allow time for a proper fix, others worry that a hack will be forgotten or that the proper fix will be deprioritized indefinitely. This concern is credible, as it’s easy to build layers of unmaintainable technical debt by patching over problems instead of making real fixes. Managers and technical leaders play a key role in implementing true, long-term fixes by supporting and prioritizing potentially time-consuming long-term fixes even when the initial “pain” of paging subsides. + </p> + <p> + Pages with rote, algorithmic responses should be a red flag. Unwillingness on the part of your team to automate such pages implies that the team lacks confidence that they can clean up their technical debt. This is a major problem worth escalating.<a data-type="indexterm" data-primary="" data-startref="MDSbig6" id="id-oPCASqT2hLSk"></a> + </p> + </section> + <section data-type="sect2" id="the-long-run-MQsWTMS7"> + <h2 class="subheaders jumptargets"> + The Long Run + </h2> + <p> + <a data-type="indexterm" data-primary="monitoring distributed systems" data-secondary="short- vs. long-term availability" id="id-jyCxSoFETNSd"></a>A common theme connects the previous examples of Bigtable and Gmail: a tension between short-term and long-term availability. Often, sheer force of effort can help a rickety system achieve high availability, but this path is usually short-lived and fraught with burnout and dependence on a small number of heroic team members. Taking a controlled, short-term decrease in availability is often a painful, but strategic trade for the long-run stability of the system. It’s important not to think of every page as an event in isolation, but to consider whether the overall <em>level</em> of paging leads toward a healthy, appropriately available system with a healthy, viable team and long-term outlook. We review statistics about page frequency (usually expressed as incidents per shift, where an incident might be composed of a few related pages) in quarterly reports with management, ensuring that decision makers are kept up to date on the pager load and overall health of their teams. + </p> + </section> + </section> + <section data-type="sect1" id="conclusion-8ksvFj"> + <h1 class="heading jumptargets"> + Conclusion + </h1> + <p> + A healthy monitoring and alerting pipeline is simple and easy to reason about. It focuses primarily on symptoms for paging, reserving cause-oriented heuristics to serve as aids to debugging problems. Monitoring symptoms is easier the further "up" your stack you monitor, though monitoring saturation and performance of subsystems such as databases often must be performed directly on the subsystem itself. Email alerts are of very limited value and tend to easily become overrun with noise; instead, you should favor a dashboard that monitors all ongoing subcritical problems for the sort of information that typically ends up in email alerts. A dashboard might also be paired with a log, in order to analyze historical correlations. + </p> + <p> + Over the long haul, achieving a successful on-call rotation and product includes choosing to alert on symptoms or imminent real problems, adapting your targets to goals that are actually achievable, and making sure that your monitoring supports rapid diagnosis. + </p> + </section> + <div class="footnotes" data-type="footnotes"> + <p data-type="footnote" id="id-LvQuvtYS7UvI8h4"> + <sup><a class="jumptargets" href="#id-LvQuvtYS7UvI8h4-marker">22</a></sup>Sometimes known as "alert spam," as they are rarely read or acted on. + </p> + <p data-type="footnote" id="id-QQLuAIXFxCz"> + <sup><a class="jumptargets" href="#id-QQLuAIXFxCz-marker">23</a></sup>If 1% of your requests are 50x the average, it means that the rest of your requests are about twice as fast as the average. But if you’re not measuring your distribution, the idea that most of your requests are near the mean is just hopeful thinking. + </p> + <p data-type="footnote" id="id-a82udF8IBfx"> + <sup><a class="jumptargets" href="#id-a82udF8IBfx-marker">24</a></sup>See <em>Applying Cardiac Alarm Management Techniques to Your On-Call</em> <a data-type="xref" href="/sre/sre-book/chapters/bibliography#Hol14" target="_blank">[Hol14]</a> for an example of alert fatigue in another context. + </p> + <p data-type="footnote" id="id-0vYuEFpSjSMtLfG"> + <sup><a class="jumptargets" href="#id-0vYuEFpSjSMtLfG-marker">25</a></sup>Zero-redundancy (<em>N</em> + 0) situations count as imminent, as do "nearly full" parts of your service! For more details about the concept of redundancy, see <a href="https://en.wikipedia.org/wiki/N%2B1_redundancy" target="_blank"><em class="hyperlink">https://en.wikipedia.org/wiki/N%2B1_redundancy</em></a>. + </p> + </div> + </section> + </div> + </div> + <div class="footer"> + <div class="maia-aux"> + <div class="previous"> + <a href="/sre/sre-book/chapters/eliminating-toil"> + <p class="footer-caption"> + Previous + </p> + <p class="chapter-link"> + Chapter 5 - Eliminating Toil + </p></a> + </div> + <div class="next"> + <a href="/sre/sre-book/chapters/automation-at-google"> + <p class="footer-caption"> + Next + </p> + <p class="chapter-link"> + Chapter 7 - The Evolution of Automation at Google + </p></a> + </div> + <p class="footer-link"> + Copyright © 2017 Google, Inc. Published by O'Reilly Media, Inc. Licensed under <a href="https://creativecommons.org/licenses/by-nc-nd/4.0/" target="_blank">CC BY-NC-ND 4.0</a> + </p> + </div> + </div> + </main> + <script src="//ajax.googleapis.com/ajax/libs/angularjs/1.6.6/angular.min.js"></script> + <script src="//ajax.googleapis.com/ajax/libs/angularjs/1.6.6/angular-animate.min.js"></script> + <script src="//ajax.googleapis.com/ajax/libs/angularjs/1.6.6/angular-touch.min.js"></script> + <script src="/sre/sre-book/static/js/index.min.js?cache=5b7f90b"></script> + </body> +</html> diff --git a/test/test-pages/keep-tabular-data/expected.html b/test/test-pages/keep-tabular-data/expected.html index 930387c..0f86741 100644 --- a/test/test-pages/keep-tabular-data/expected.html +++ b/test/test-pages/keep-tabular-data/expected.html @@ -1,7 +1,5 @@ <div> - <h2> - Friday Facts #282 - 0.17 in sight - </h2> + <p>Posted by kovarex, TOGos, Ernestas, Albert on 2019-02-15, <a href="http://fakehost/blog/">all posts</a></p> <h2>The release plan <span size="2">(kovarex)</span> diff --git a/test/test-pages/lazy-image-1/expected.html b/test/test-pages/lazy-image-1/expected.html index 2092395..1c98070 100644 --- a/test/test-pages/lazy-image-1/expected.html +++ b/test/test-pages/lazy-image-1/expected.html @@ -4,14 +4,18 @@ </p> </div> - + <h2 id="0231"> + Why CPU monitoring is important? + </h2> <p id="d2c1"> I work at <a href="http://voodoo.io/" target="_blank" rel="noopener nofollow">Voodoo</a>, a French company that creates mobile video games. We have a lot of challenges with performance, availability, and scalability because of the insane amount of traffic our infrastructure supports (billions of events/requests per day …… no joke!). In this setting, every metric is important and gives us a lot of information about the state of our system. </p> <p id="0e89"> When working with Node.js one of the most critical resources to monitor is the CPU. Most of the time, when working on a low traffic API or project we don’t realize how many simple lines of code can have a huge impact on CPU. On the other hand, when traffic increases, a simple mistake can cost dearly. </p> - + <h2 id="292e"> + Resources + </h2> <p id="1efa"> What kind of resources does your application need? In most cases, we focus on memory and CPU. Good monitoring of these two elements is mandatory for an application running on production. </p> @@ -53,7 +57,9 @@ <p id="0728"> And what next? We don’t have data about the state of the instance when the CPU usage has increased. So we can’t determine why we had this peak, at least not without an important time of debugging, comparing log, etc. This is exactly why you need to use CPU profiling. </p> - + <h2 id="8d00"> + CPU profiling: what’s the difference with CPU monitoring? + </h2> <blockquote> <p> “Most commonly, profiling information serves to aid program optimization. Profiling is achieved by instrumenting either the program source code or its binary executable form using a tool called a profiler” @@ -65,7 +71,9 @@ <p id="91c5"> It will help you to track the exact file, line, and function which takes the most time to execute. </p> - + <h2 id="088b"> + What about existing solutions? + </h2> <h2 id="dd40"> Add arguments to Node.js </h2> @@ -146,7 +154,9 @@ Unfortunately, CPU issues have a worrying tendency to occur on production, and when you are not in front of your screen. </p> </blockquote> - + <h2 id="13ef"> + Inspector + </h2> <p id="294e"> “Inspector” refers to an API thanks to which you can debug your application. By debugging we mean to be able to connect directly to the core of Node.js to collect real-time data about the process. </p> @@ -181,7 +191,9 @@ <figure> </figure> - + <h2 id="848b"> + And now, CPU profiling on-demand! + </h2> <p id="6933"> We have an API that we want to test with autocannon tool. At this step, our project is able to serve around 200 requests in 20 seconds. There is probably a mistake somewhere in the code which slows down our application. </p> @@ -264,7 +276,9 @@ </p> </div> </figure> - + <h2 id="98b9"> + More than just CPU profiling + </h2> <p id="e1ad"> With the inspector module, you can do much more than just CPU profiling, here is a non-exhaustive list: </p> @@ -276,7 +290,9 @@ <li id="b896">use the debugger in real-time </li> </ul> - + <h2 id="06d2"> + Warnings + </h2> <p id="731b"> Every tool, even the most powerful, comes with its own disadvantages. If you enable the profiler and/or the debugger on your production you have to keep an eye on two things: </p> @@ -295,7 +311,9 @@ <p id="7999"> Using the inspector in Node.js it’s like opening the door of the core of your application. You should be very careful about who can use features like CPU profiling and/or the debugger. Never make the inspector “public” as being able to launch a feature from an unsafe route (not protected with an authentification mechanism). Even the collected data can be seen as critical, never send it to a system you do not trust. </p> - + <h2 id="5618"> + Conclusion + </h2> <p id="ae1a"> CPU profiling is really a must-have tool for every developer. And now, with some precautions, we can run it on production thanks to the amazing work done by the V8 and Node.js team. </p> @@ -305,7 +323,9 @@ <p id="0aba"> I will write another article about using CPU profiling and the inspector on production on a high traffic project. </p> - + <h2 id="3c5b"> + Sources & links + </h2> <ul class> <li id="d86d"> <a href="https://nodejs.org/api/inspector.html" target="_blank" rel="noopener nofollow">https://nodejs.org/api/inspector.html</a> diff --git a/test/test-pages/medium-1/expected.html b/test/test-pages/medium-1/expected.html index 4b399f0..eb62327 100644 --- a/test/test-pages/medium-1/expected.html +++ b/test/test-pages/medium-1/expected.html @@ -1,5 +1,5 @@ <div> - <h2 name="3c62" id="3c62" data-align="center">Open Journalism Project:</h2> + <h4 name="425a" id="425a" data-align="center"><em>Better Student Journalism</em></h4> diff --git a/test/test-pages/metadata-content-missing/expected-images.json b/test/test-pages/metadata-content-missing/expected-images.json new file mode 100644 index 0000000..0637a08 --- /dev/null +++ b/test/test-pages/metadata-content-missing/expected-images.json @@ -0,0 +1 @@ +[]
\ No newline at end of file diff --git a/test/test-pages/metadata-content-missing/expected-metadata.json b/test/test-pages/metadata-content-missing/expected-metadata.json new file mode 100644 index 0000000..ed10c30 --- /dev/null +++ b/test/test-pages/metadata-content-missing/expected-metadata.json @@ -0,0 +1,8 @@ +{ + "Author": "Creator Name", + "Direction": null, + "Excerpt": "Preferred description", + "Image": null, + "Title": "My title", + "SiteName": null +}
\ No newline at end of file diff --git a/test/test-pages/metadata-content-missing/expected.html b/test/test-pages/metadata-content-missing/expected.html new file mode 100644 index 0000000..b282bdd --- /dev/null +++ b/test/test-pages/metadata-content-missing/expected.html @@ -0,0 +1,19 @@ +<article> + <h2>Test document title</h2> + <p> + Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod + tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, + quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo + consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse + cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non + proident, sunt in culpa qui officia deserunt mollit anim id est laborum. + </p> + <p> + Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod + tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, + quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo + consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse + cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non + proident, sunt in culpa qui officia deserunt mollit anim id est laborum. + </p> + </article>
\ No newline at end of file diff --git a/test/test-pages/metadata-content-missing/source.html b/test/test-pages/metadata-content-missing/source.html new file mode 100644 index 0000000..1e5c1ad --- /dev/null +++ b/test/test-pages/metadata-content-missing/source.html @@ -0,0 +1,33 @@ +<!DOCTYPE html> +<html> + <head> + <meta charset="utf-8"/> + <title>Title Element</title> + <meta property="x:title dc:title" content="My title"/> + <meta property="dc:creator twitter:site_name" content="Creator Name"/> + <meta name="author" content="FAIL"/> + <meta property="og:description twitter:description"/> + <meta property="dc:description og:description" content="Preferred description"/> + </head> + <body> + <article> + <h1>Test document title</h1> + <p> + Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod + tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, + quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo + consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse + cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non + proident, sunt in culpa qui officia deserunt mollit anim id est laborum. + </p> + <p> + Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod + tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, + quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo + consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse + cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non + proident, sunt in culpa qui officia deserunt mollit anim id est laborum. + </p> + </article> + </body> +</html> diff --git a/test/test-pages/normalize-spaces/expected-images.json b/test/test-pages/normalize-spaces/expected-images.json new file mode 100644 index 0000000..0637a08 --- /dev/null +++ b/test/test-pages/normalize-spaces/expected-images.json @@ -0,0 +1 @@ +[]
\ No newline at end of file diff --git a/test/test-pages/normalize-spaces/expected-metadata.json b/test/test-pages/normalize-spaces/expected-metadata.json new file mode 100644 index 0000000..59c9941 --- /dev/null +++ b/test/test-pages/normalize-spaces/expected-metadata.json @@ -0,0 +1,8 @@ +{ + "Author": null, + "Direction": null, + "Excerpt": "Lorem\n ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod\n\ttab here\n incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam,\n quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo\n consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse\n cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non\n proident, sunt in culpa qui officia deserunt mollit anim id est laborum.", + "Image": null, + "Title": "Normalize space test", + "SiteName": null +}
\ No newline at end of file diff --git a/test/test-pages/normalize-spaces/expected.html b/test/test-pages/normalize-spaces/expected.html new file mode 100644 index 0000000..18739c3 --- /dev/null +++ b/test/test-pages/normalize-spaces/expected.html @@ -0,0 +1,26 @@ +<article> + <h2>Lorem</h2> + <p> + Lorem + ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod + tab here + incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, + quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo + consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse + cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non + proident, sunt in culpa qui officia deserunt mollit anim id est laborum. + </p> + <h2>Foo</h2> + <p> + Tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, + quis nostrud exercitation + + + + + ullamco laboris nisi ut aliquip ex ea commodo + consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse + cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non + proident, sunt in culpa qui officia deserunt mollit anim id est laborum. + </p> + </article>
\ No newline at end of file diff --git a/test/test-pages/normalize-spaces/source.html b/test/test-pages/normalize-spaces/source.html new file mode 100644 index 0000000..b230798 --- /dev/null +++ b/test/test-pages/normalize-spaces/source.html @@ -0,0 +1,35 @@ +<!DOCTYPE html> +<html> +<head> + <meta charset="utf-8"/> + <title>Normalize space test</title> +</head> +<body> + <article> + <h1>Lorem</h1> + <div> + Lorem + ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod + tab here + incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, + quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo + consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse + cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non + proident, sunt in culpa qui officia deserunt mollit anim id est laborum. + </div> + <h2>Foo</h2> + <div> + Tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, + quis nostrud exercitation + + + + + ullamco laboris nisi ut aliquip ex ea commodo + consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse + cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non + proident, sunt in culpa qui officia deserunt mollit anim id est laborum. + </div> + </article> +</body> +</html> diff --git a/test/test-pages/replace-font-tags/expected.html b/test/test-pages/replace-font-tags/expected.html index c4700f3..cb19cbf 100644 --- a/test/test-pages/replace-font-tags/expected.html +++ b/test/test-pages/replace-font-tags/expected.html @@ -1,5 +1,5 @@ <article> - + <h2>Lorem</h2> <p><span face="Arial" size="2"><span face="Times" size="10">Lorem</span> ipsum dolor</span> sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo diff --git a/test/test-pages/rtl-1/expected.html b/test/test-pages/rtl-1/expected.html index eba1686..9c611c9 100644 --- a/test/test-pages/rtl-1/expected.html +++ b/test/test-pages/rtl-1/expected.html @@ -1,4 +1,6 @@ -<div><article> +<div> + <article> + <h2>Lorem</h2> <p> Lorem ipsum dolor sit amet. </p> @@ -8,4 +10,5 @@ <p> Tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. </p> - </article></div>
\ No newline at end of file + </article> + </div>
\ No newline at end of file diff --git a/test/test-pages/rtl-2/expected.html b/test/test-pages/rtl-2/expected.html index eba1686..9c611c9 100644 --- a/test/test-pages/rtl-2/expected.html +++ b/test/test-pages/rtl-2/expected.html @@ -1,4 +1,6 @@ -<div><article> +<div> + <article> + <h2>Lorem</h2> <p> Lorem ipsum dolor sit amet. </p> @@ -8,4 +10,5 @@ <p> Tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. </p> - </article></div>
\ No newline at end of file + </article> + </div>
\ No newline at end of file diff --git a/test/test-pages/rtl-3/expected.html b/test/test-pages/rtl-3/expected.html index 7544888..7892601 100644 --- a/test/test-pages/rtl-3/expected.html +++ b/test/test-pages/rtl-3/expected.html @@ -1,4 +1,6 @@ -<div dir="rtl"><article> +<div dir="rtl"> + <article> + <h2>Lorem</h2> <p> Lorem ipsum dolor sit amet. </p> @@ -8,4 +10,5 @@ <p> Tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. </p> - </article></div>
\ No newline at end of file + </article> + </div>
\ No newline at end of file diff --git a/test/test-pages/rtl-4/expected.html b/test/test-pages/rtl-4/expected.html index 14724e7..0d55d33 100644 --- a/test/test-pages/rtl-4/expected.html +++ b/test/test-pages/rtl-4/expected.html @@ -1,4 +1,6 @@ -<div><article dir="rtl"> +<div> + <article dir="rtl"> + <h2>Lorem</h2> <p> Lorem ipsum dolor sit amet. </p> @@ -8,4 +10,5 @@ <p> Tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. </p> - </article></div>
\ No newline at end of file + </article> + </div>
\ No newline at end of file diff --git a/test/test-pages/social-buttons/expected.html b/test/test-pages/social-buttons/expected.html index 6d135be..30c09b2 100644 --- a/test/test-pages/social-buttons/expected.html +++ b/test/test-pages/social-buttons/expected.html @@ -1,4 +1,5 @@ <article> + <h2>Lorem ipsum dolor</h2> <p>Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo diff --git a/test/test-pages/style-tags-removal/expected.html b/test/test-pages/style-tags-removal/expected.html index d3a8806..a5ba2c6 100644 --- a/test/test-pages/style-tags-removal/expected.html +++ b/test/test-pages/style-tags-removal/expected.html @@ -1,4 +1,6 @@ <article> + <h2>Lorem</h2> + <p> Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, @@ -7,6 +9,7 @@ cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. </p> + <h2>Foo</h2> <p> Tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, diff --git a/test/test-pages/title-and-h1-discrepancy/expected.html b/test/test-pages/title-and-h1-discrepancy/expected.html index e4fa77a..8eded27 100644 --- a/test/test-pages/title-and-h1-discrepancy/expected.html +++ b/test/test-pages/title-and-h1-discrepancy/expected.html @@ -1,4 +1,5 @@ <article> + <h2>This is a long title with a colon: But the final text here is different</h2> <p> Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod diff --git a/test/test-pages/tmz-1/expected.html b/test/test-pages/tmz-1/expected.html index 1f7479a..6849dfe 100644 --- a/test/test-pages/tmz-1/expected.html +++ b/test/test-pages/tmz-1/expected.html @@ -1,6 +1,6 @@ <div id="post-2015_02_26_lupita-nyongo-pearl-dress-stolen-oscars"> <p> - <h2>Lupita Nyong'o</h2> + <h4>$150K Pearl Oscar Dress ... STOLEN!!!!</h4> diff --git a/test/test-pages/topicseed-1/expected-images.json b/test/test-pages/topicseed-1/expected-images.json new file mode 100644 index 0000000..61e175c --- /dev/null +++ b/test/test-pages/topicseed-1/expected-images.json @@ -0,0 +1,3 @@ +[ + "https:\/\/topicseed.com\/static\/9c97da26f6eeee98fc2e628ca3416226\/57090\/content-depth-seo.png" +]
\ No newline at end of file diff --git a/test/test-pages/topicseed-1/expected-metadata.json b/test/test-pages/topicseed-1/expected-metadata.json new file mode 100644 index 0000000..cfac459 --- /dev/null +++ b/test/test-pages/topicseed-1/expected-metadata.json @@ -0,0 +1,8 @@ +{ + "Author": null, + "Direction": null, + "Excerpt": "Content writers and marketers find it hard to write a lot of content about a very specific topic. They lose a lot of points on their content depth because they would rather focus on pushing thin content about plenty of topics.", + "Image": "https:\/\/topicseed.com\/static\/9c97da26f6eeee98fc2e628ca3416226\/57090\/content-depth-seo.png", + "Title": "Content Depth \u2014 Write Comprehensively About Your Core Topics", + "SiteName": "topicseed" +}
\ No newline at end of file diff --git a/test/test-pages/topicseed-1/expected.html b/test/test-pages/topicseed-1/expected.html new file mode 100644 index 0000000..18cd128 --- /dev/null +++ b/test/test-pages/topicseed-1/expected.html @@ -0,0 +1,93 @@ +<div> + <ul> + <li> + <a href="#assess-how-deep-is-your-content">Assess How Deep Is Your Content</a> + </li> + <li> + <a href="#rewrite-with-content-depth-in-mind">Rewrite With Content Depth In Mind</a> + </li> + <li> + <a href="#yes-content-depth-and-breadth-overlap">Yes, Content Depth and Breadth Overlap</a> + </li> + <li> + <a href="#depth-of-content--quality--frequency">Depth of Content = Quality + Frequency</a> + </li> + </ul> + <p> + <strong>Content depth</strong> is an arbitrary score or rating of how comprehensive the coverage of a specific topic is within a piece of content. <strong>Content breadth</strong> is an arbitrary grading of how many related subjects are you covering within your content. + </p> + <p> + And this distinction is important to make and establish from the beginning. Effective <a href="https://topicseed.com/blog/what-is-topical-authority" target="_blank" rel="nofollow noopener noreferrer"><strong>topical authority</strong></a> can only be gained when you use both content depth and content breadth in your overall content strategy for rapid search engine optimization gains. However, because most content writers prefer to write a little bit about many things rather than write a lot about one thing, you end up with a too little substance spread very thin. + </p> + <p> + Content depth should be the urgent priority for your content marketing strategy, and clearly defined in your <a href="http://fakehost/blog/content-briefs">content briefs</a>. Start by dominating your own core topics, before venturing across the pond and write about linked subject matters. Otherwise, you are the opposite of an authority as the definition states that an authority is <em>“a person with extensive or specialized knowledge about a subject; an expert”.</em> Lastly, do not mistake article depth vs. article length: a blog post’s extreme wordcount has nothing to do with its content depth. + </p> + <h2 id="assess-how-deep-is-your-content"> + <a href="#assess-how-deep-is-your-content" aria-label="assess how deep is your content permalink"></a>Assess How Deep Is Your Content + </h2> + <p> + The first task on your list, right now, is to shortlist your core topics. What are you trying to be an expert on? Then, go through each one of your pieces of content and understand how well each blog post is covering its focus topic(s). Not how many times specific keywords appear, or how well the article is outlined and structured. + </p> + <p> + Put yourself in the shoes of an ignorant reader who seeks information. Read your article. <strong>And ask yourself how in-depth was the content you have written?</strong> I know the excuse you will come up with: this was written for beginners, therefore, it shouldn’t be too in-depth. And you are correct. Not every blog post is about absolute content depth otherwise we would only write one 10,000-word-long article, once and for all. But then, how well your beginner-level content pointing to your expert-level content? + </p> + <p> + In other words, each article should reach an incredible level of content depth for its expertise level. And then, provide further reading <em>(i.e. links)</em> to gain more knowledge, and depth. A lot of content editors write a beginner’s blog post and wait to see it perform well in order to write a more advanced sequel. Wrong. Give all the value so search engines can grade you highly on their authority scale for your core topics. Yes, it is a risk and you may write a dozen of articles on a specific topic that will never really rank at the top of SERPs, but <strong>reaching content depth is the first step towards SEO gains</strong>. + </p> + <p> + Remember that <strong>skyscraper content</strong> and <strong>10x content</strong> are not necessarily the answer. These content writing strategies state that in order to beat another piece of content, you need to write 10x more. Either in quantity with a 10x word count or in quality by putting times more information within your own piece of content. Such articles often become unreadable and discourage visitors from absorbing all the knowledge. The best alternative is the create <a href="https://topicseed.com/blog/how-broad-should-topics-be-for-pillar-pages" target="_blank" rel="nofollow noopener noreferrer">pillar pages</a> centered around core topics, and several articles dealing with each specific section in depth. This is <strong>deep content powered by a <a href="https://topicseed.com/blog/internal-linking-strategies-for-topic-clustering" target="_blank" rel="nofollow noopener noreferrer">smart internal linking strategy</a></strong> and search engines love that in this day and age where attention spans are short! <em>With that being said, avoid writing 600-word articles!</em> + </p> + <h2 id="rewrite-with-content-depth-in-mind"> + <a href="#rewrite-with-content-depth-in-mind" aria-label="rewrite with content depth in mind permalink"></a>Rewrite With Content Depth In Mind + </h2> + <p> + Once you know which articles are lacking depth of knowledge and information, it is time to rethink each one. For each article, make a list of what essential pieces of information or data are missing. Then decide where to fit them, and decide whether the article would benefit from a full rewrite or not. As a rule of thumb, if you need to change a third of your article, you may need to rewrite it entirely. Of course, this does not mean erasing all work done prior, but it means starting afresh! Trying to <strong>fit deep content into an existing blog</strong> post gives you constraints so doing it from scratch can actually be easier to fight thin content. + </p> + + <p> + As explained above, make sure you do not force yourself to write a much longer article to reach a <a href="https://moz.com/blog/blog-post-length-frequency" target="_blank" rel="nofollow noopener noreferrer">magic word count</a>. And if you do, it has to be natural. In many cases, articles written months or years ago may need some upkeeping: trimming the fat and removing parts that are not bringing much value. Replace these with your newer and deeper content. + </p> + <p> + All content writers know that when you open Google Docs, WordPress, or your text editor of choice, you will inevitably count your focus keywords’ frequency. Although I understand (yet question) the value of keywords in modern SEO, do not become obsessed with reaching a magic number for your keywords. No reader coming from Google is out there counting how often your keywords are appearing. And search engine algorithms will penalize you for writing for robots, rather than humans. + </p> + <p> + With the massive rise of voice searches, <a href="http://fakehost/blog/featured-snippets-using-questions">users tend to use full questions for their search queries</a>. What used to be <code>top bottled water brands</code> is now <code>OK google, what is the best bottled-water brand in Texas</code>? The point being, <a href="https://topicseed.com/blog/keyword-search-volume-overrated" target="_blank" rel="nofollow noopener noreferrer"><strong>keywords are losing traction</strong></a> to leave space for a more natural language understanding of a blog post’s textual content, and meaning. + </p> + <h2 id="yes-content-depth-and-breadth-overlap"> + <a href="#yes-content-depth-and-breadth-overlap" aria-label="yes content depth and breadth overlap permalink"></a>Yes, Content Depth and Breadth Overlap + </h2> + <p> + <em>“A topic can be defined as the company it keeps.”</em> A very accurate saying loved by ontologists within the fields of computational linguistics, and information science. In simpler terms, a topic and all the terminology it is encompassing will inevitably overlap with related topics. Which, in turn, will form <a href="https://topicseed.com/blog/topic-clusters-relationships" target="_blank" rel="nofollow noopener noreferrer"><strong>topic clusters</strong></a>. + </p> + <p> + For example, it is obvious that despite being two different topics, <code>digital advertising</code> and <code>content marketing</code> share some common phrases and terms. Inevitably, a website picking one as its core topic will use words in some blog posts that will identify the article as belonging to both topics, with a specific weight for each. + </p> + <p> + A keyword, phrase, or term, is not a prisoner to a single concept at all. This is how algorithms in natural language understanding can understand how two topics are related (e.g. read about <a href="https://en.wikipedia.org/wiki/Topic_model" target="_blank" rel="nofollow noopener noreferrer"><em>topic modeling</em></a>). Each topic has a specific <strong>vocabulary</strong>, a list of words and phrases commonly used in its context, and some of these terms are present in different vocabularies. + </p> + <p> + Therefore, content depth and content breadth are not to be opposed. Content marketers should use both strategies in order to reach ultimate <strong>topical authority</strong> over their choice of subject matters. + </p> + <h2 id="depth-of-content--quality--frequency"> + <a href="#depth-of-content--quality--frequency" aria-label="depth of content quality frequency permalink"></a>Depth of Content = Quality + Frequency + </h2> + <p> + Up until recently, long-form blog posts generally were <strong>evergreen articles</strong> that generated a constant stream of organic traffic for a website. This was a lead magnet generation strategy which worked well: hire a writer, include the right keywords, reach over a 5,000-word word count, and hit publish. Then, wait. + </p> + <p> + Nowadays, in-depth content requires more effort over time in order to pay off. Writing a big article, as good as it is, will not get your anywhere near the level of <a href="https://topicseed.com/blog/topical-seo" target="_blank" rel="nofollow noopener noreferrer">topical breadth</a> required by Google to rank you first. Instead, your content marketing plan should be about having: + </p> + <ul> + <li>a <strong>comprehensive pillar page</strong> covering a unique topic, and + </li> + <li> + <strong>narrow-focused children articles</strong> to dig deeper. + </li> + </ul> + <p> + Search engines also look at how often you publish about a specific topic, and when was the last time it was written about. Nobody likes a <a href="https://www.copypress.com/blog/avoiding-blog-graveyard/" target="_blank" rel="nofollow noopener noreferrer">graveyard blog</a>, it just makes the reader lose trust; as if the writer was not good enough, therefore had no traffic, before entirely giving up. Deep content requires a sustained effort on your part to always new find ways to write about a specific subject. Sure, it will be easy at first. But what about five years later? Well, you will still need to hit publish, all about the very same topics you already covered years ago. + </p> + <p> + Tools and platforms such as topicseed are here to <a href="https://topicseed.com/blog/how-to-find-new-blog-post-ideas" target="_blank" rel="nofollow noopener noreferrer">help you find new article ideas</a> pertaining to your core topics within a few clicks and a few minutes. The number of web pages, Wikipedia articles, and pieces of content, our machine-learning algorithms can analyze in seconds would take you months to digest. Our <em>topicgraph</em> finds closely related concepts in order for your domain to <strong>reach topical authority through content depth and content breadth</strong>. + </p> + </div>
\ No newline at end of file diff --git a/test/test-pages/topicseed-1/source.html b/test/test-pages/topicseed-1/source.html new file mode 100644 index 0000000..fca1c00 --- /dev/null +++ b/test/test-pages/topicseed-1/source.html @@ -0,0 +1,400 @@ +<!DOCTYPE html> +<html xmlns="http://www.w3.org/1999/xhtml"> + <head> + <meta charset="utf-8" /> + <meta http-equiv="x-ua-compatible" content="ie=edge" /> + <meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no" /> + <style data-href="/styles.4554427a2ff1ec4c9b41.css"> + <![CDATA[ + .index-module--Section--2wwdv{padding:2% 4%;background-color:#fff}.index-module--Section--2wwdv:nth-child(odd){background-color:#f9f9f9}.index-module--Section--2wwdv h1,.index-module--Section--2wwdv h2,.index-module--Section--2wwdv h3,.index-module--Section--2wwdv h4,.index-module--Section--2wwdv h5{font-family:Lato}.index-module--Section--2wwdv h2{color:#00ced1;text-shadow:0 0 1rem rgba(187,235,234,.64);line-height:1.25;font-size:2rem;margin:1.5em 0;text-align:center}.index-module--OurLocation--39dZN{margin-top:3rem}.index-module--OurLocation--39dZN a{color:#00ced1;font-weight:500;text-decoration:none}.index-module--OurLocation--39dZN .index-module--Pin--1M4gn{text-align:center}.index-module--OurLocation--39dZN .index-module--Pin--1M4gn img{height:64px;width:64px}.index-module--OurLocation--39dZN .index-module--Location--3wrrA p{font-size:1.2rem;padding:0;margin:0 auto;line-height:1.2;max-width:600px;font-weight:100;font-style:italic}.index-module--SectionExcerpt--36zdZ{max-width:800px;margin:auto auto 3rem;font-size:1.5em;line-height:1.5;text-align:center}.index-module--Listing--2-O97{font-size:1.2em;line-height:1.5;display:block;list-style:none;margin:0;padding:0}.index-module--Listing--2-O97>li{display:block;text-align:center;padding:1rem;border-radius:10px}.index-module--ListingIcon--2Hsao{padding-bottom:1rem}.index-module--ListingIcon--2Hsao img{width:64px;height:64px;display:block;margin:auto}.index-module--SubListingTitle--HZOQi{font-weight:700;padding:1rem 0;display:block}.index-module--SubListing--MrsDd{list-style:none;margin:0;padding:0}.index-module--SubListing--MrsDd li{text-transform:uppercase;font-size:.6em;letter-spacing:1px;margin:0;padding:.35rem 0}@media (min-width:1200px){.index-module--Listing--2-O97{display:flex}.index-module--Listing--2-O97>li{flex:1 1}}.index-module--Container--3Xdxi{overflow:hidden;max-width:100%;padding:2.5% 0 4%}.index-module--Container--2hTRH{background-color:#f9f9f9}.index-module--OptinArea--oMoVV{max-width:900px;background-color:transparent;margin:0 auto;border-radius:10px;padding:0 1rem}.index-module--Steps--2Plqq{margin:auto;text-align:left;padding:5% 1rem 0;max-width:400px;line-height:1}.index-module--Steps--2Plqq h4,.index-module--Steps--2Plqq li,.index-module--Steps--2Plqq ul{list-style-type:none;padding:0;margin:0}.index-module--Steps--2Plqq li{margin-bottom:1rem}.index-module--Steps--2Plqq .index-module--Inner--1eYQv{display:flex;flex-flow:nowrap;padding:1rem 1rem 0}.index-module--Steps--2Plqq .index-module--Index--2iAgE{flex-shrink:1}.index-module--Steps--2Plqq .index-module--Index--2iAgE strong{position:relative;display:inline-block;text-align:center;font-weight:700;border-radius:50%;background-color:#fff;color:#40e0d0;font-size:1em;padding:.5em .75em;box-shadow:0 0 1em rgba(64,224,208,.4)}.index-module--Steps--2Plqq .index-module--Text--1Xgj4{flex-grow:1;padding-left:1rem}.index-module--Steps--2Plqq .index-module--Text--1Xgj4 h4{padding-top:.5em;color:#00ced1;font-weight:400}.index-module--Steps--2Plqq .index-module--Text--1Xgj4 h4 em{font-style:normal;font-weight:900}.index-module--Steps--2Plqq .index-module--Text--1Xgj4 .index-module--Description--2Zoob{margin-top:.7rem;font-size:.8rem;padding-right:1rem;line-height:1.25}@media screen and (min-width:800px){.index-module--Steps--2Plqq{max-width:1000px}.index-module--Steps--2Plqq ul{display:flex}.index-module--Steps--2Plqq ul li{flex:1 1;margin-bottom:0}}.container-fluid{margin-right:auto;margin-left:auto;padding-right:2rem;padding-left:2rem}.row{box-sizing:border-box;display:flex;flex:0 1 auto;flex-direction:row;flex-wrap:wrap}.row.reverse{flex-direction:row-reverse}.col.reverse{flex-direction:column-reverse}.col-xs,.col-xs-1,.col-xs-2,.col-xs-3,.col-xs-4,.col-xs-5,.col-xs-6,.col-xs-7,.col-xs-8,.col-xs-9,.col-xs-10,.col-xs-11,.col-xs-12{box-sizing:border-box;flex:0 0 auto;padding-right:1rem;padding-left:1rem}.col-xs{flex-grow:1;flex-basis:0;max-width:100%}.col-xs-1{flex-basis:8.333%;max-width:8.333%}.col-xs-2{flex-basis:16.667%;max-width:16.667%}.col-xs-3{flex-basis:25%;max-width:25%}.col-xs-4{flex-basis:33.333%;max-width:33.333%}.col-xs-5{flex-basis:41.667%;max-width:41.667%}.col-xs-6{flex-basis:50%;max-width:50%}.col-xs-7{flex-basis:58.333%;max-width:58.333%}.col-xs-8{flex-basis:66.667%;max-width:66.667%}.col-xs-9{flex-basis:75%;max-width:75%}.col-xs-10{flex-basis:83.333%;max-width:83.333%}.col-xs-11{flex-basis:91.667%;max-width:91.667%}.col-xs-12{flex-basis:100%;max-width:100%}.col-xs-offset-1{margin-left:8.333%}.col-xs-offset-2{margin-left:16.667%}.col-xs-offset-3{margin-left:25%}.col-xs-offset-4{margin-left:33.333%}.col-xs-offset-5{margin-left:41.667%}.col-xs-offset-6{margin-left:50%}.col-xs-offset-7{margin-left:58.333%}.col-xs-offset-8{margin-left:66.667%}.col-xs-offset-9{margin-left:75%}.col-xs-offset-10{margin-left:83.333%}.col-xs-offset-11{margin-left:91.667%}.start-xs{justify-content:flex-start;text-align:start}.center-xs{justify-content:center;text-align:center}.end-xs{justify-content:flex-end;text-align:end}.top-xs{align-items:flex-start}.middle-xs{align-items:center}.bottom-xs{align-items:flex-end}.around-xs{justify-content:space-around}.between-xs{justify-content:space-between}.first-xs{order:-1}.last-xs{order:1}@media only screen and (min-width:48em){.container{width:46rem}.col-sm,.col-sm-1,.col-sm-2,.col-sm-3,.col-sm-4,.col-sm-5,.col-sm-6,.col-sm-7,.col-sm-8,.col-sm-9,.col-sm-10,.col-sm-11,.col-sm-12{box-sizing:border-box;flex:0 0 auto;padding-right:1rem;padding-left:1rem}.col-sm{flex-grow:1;flex-basis:0;max-width:100%}.col-sm-1{flex-basis:8.333%;max-width:8.333%}.col-sm-2{flex-basis:16.667%;max-width:16.667%}.col-sm-3{flex-basis:25%;max-width:25%}.col-sm-4{flex-basis:33.333%;max-width:33.333%}.col-sm-5{flex-basis:41.667%;max-width:41.667%}.col-sm-6{flex-basis:50%;max-width:50%}.col-sm-7{flex-basis:58.333%;max-width:58.333%}.col-sm-8{flex-basis:66.667%;max-width:66.667%}.col-sm-9{flex-basis:75%;max-width:75%}.col-sm-10{flex-basis:83.333%;max-width:83.333%}.col-sm-11{flex-basis:91.667%;max-width:91.667%}.col-sm-12{flex-basis:100%;max-width:100%}.col-sm-offset-1{margin-left:8.333%}.col-sm-offset-2{margin-left:16.667%}.col-sm-offset-3{margin-left:25%}.col-sm-offset-4{margin-left:33.333%}.col-sm-offset-5{margin-left:41.667%}.col-sm-offset-6{margin-left:50%}.col-sm-offset-7{margin-left:58.333%}.col-sm-offset-8{margin-left:66.667%}.col-sm-offset-9{margin-left:75%}.col-sm-offset-10{margin-left:83.333%}.col-sm-offset-11{margin-left:91.667%}.start-sm{justify-content:flex-start;text-align:start}.center-sm{justify-content:center;text-align:center}.end-sm{justify-content:flex-end;text-align:end}.top-sm{align-items:flex-start}.middle-sm{align-items:center}.bottom-sm{align-items:flex-end}.around-sm{justify-content:space-around}.between-sm{justify-content:space-between}.first-sm{order:-1}.last-sm{order:1}}@media only screen and (min-width:62em){.container{width:61rem}.col-md,.col-md-1,.col-md-2,.col-md-3,.col-md-4,.col-md-5,.col-md-6,.col-md-7,.col-md-8,.col-md-9,.col-md-10,.col-md-11,.col-md-12{box-sizing:border-box;flex:0 0 auto;padding-right:1rem;padding-left:1rem}.col-md{flex-grow:1;flex-basis:0;max-width:100%}.col-md-1{flex-basis:8.333%;max-width:8.333%}.col-md-2{flex-basis:16.667%;max-width:16.667%}.col-md-3{flex-basis:25%;max-width:25%}.col-md-4{flex-basis:33.333%;max-width:33.333%}.col-md-5{flex-basis:41.667%;max-width:41.667%}.col-md-6{flex-basis:50%;max-width:50%}.col-md-7{flex-basis:58.333%;max-width:58.333%}.col-md-8{flex-basis:66.667%;max-width:66.667%}.col-md-9{flex-basis:75%;max-width:75%}.col-md-10{flex-basis:83.333%;max-width:83.333%}.col-md-11{flex-basis:91.667%;max-width:91.667%}.col-md-12{flex-basis:100%;max-width:100%}.col-md-offset-1{margin-left:8.333%}.col-md-offset-2{margin-left:16.667%}.col-md-offset-3{margin-left:25%}.col-md-offset-4{margin-left:33.333%}.col-md-offset-5{margin-left:41.667%}.col-md-offset-6{margin-left:50%}.col-md-offset-7{margin-left:58.333%}.col-md-offset-8{margin-left:66.667%}.col-md-offset-9{margin-left:75%}.col-md-offset-10{margin-left:83.333%}.col-md-offset-11{margin-left:91.667%}.start-md{justify-content:flex-start;text-align:start}.center-md{justify-content:center;text-align:center}.end-md{justify-content:flex-end;text-align:end}.top-md{align-items:flex-start}.middle-md{align-items:center}.bottom-md{align-items:flex-end}.around-md{justify-content:space-around}.between-md{justify-content:space-between}.first-md{order:-1}.last-md{order:1}}@media only screen and (min-width:75em){.container{width:71rem}.col-lg,.col-lg-1,.col-lg-2,.col-lg-3,.col-lg-4,.col-lg-5,.col-lg-6,.col-lg-7,.col-lg-8,.col-lg-9,.col-lg-10,.col-lg-11,.col-lg-12{box-sizing:border-box;flex:0 0 auto;padding-right:1rem;padding-left:1rem}.col-lg{flex-grow:1;flex-basis:0;max-width:100%}.col-lg-1{flex-basis:8.333%;max-width:8.333%}.col-lg-2{flex-basis:16.667%;max-width:16.667%}.col-lg-3{flex-basis:25%;max-width:25%}.col-lg-4{flex-basis:33.333%;max-width:33.333%}.col-lg-5{flex-basis:41.667%;max-width:41.667%}.col-lg-6{flex-basis:50%;max-width:50%}.col-lg-7{flex-basis:58.333%;max-width:58.333%}.col-lg-8{flex-basis:66.667%;max-width:66.667%}.col-lg-9{flex-basis:75%;max-width:75%}.col-lg-10{flex-basis:83.333%;max-width:83.333%}.col-lg-11{flex-basis:91.667%;max-width:91.667%}.col-lg-12{flex-basis:100%;max-width:100%}.col-lg-offset-1{margin-left:8.333%}.col-lg-offset-2{margin-left:16.667%}.col-lg-offset-3{margin-left:25%}.col-lg-offset-4{margin-left:33.333%}.col-lg-offset-5{margin-left:41.667%}.col-lg-offset-6{margin-left:50%}.col-lg-offset-7{margin-left:58.333%}.col-lg-offset-8{margin-left:66.667%}.col-lg-offset-9{margin-left:75%}.col-lg-offset-10{margin-left:83.333%}.col-lg-offset-11{margin-left:91.667%}.start-lg{justify-content:flex-start;text-align:start}.center-lg{justify-content:center;text-align:center}.end-lg{justify-content:flex-end;text-align:end}.top-lg{align-items:flex-start}.middle-lg{align-items:center}.bottom-lg{align-items:flex-end}.around-lg{justify-content:space-around}.between-lg{justify-content:space-between}.first-lg{order:-1}.last-lg{order:1}}.no-gutter{padding:0;margin:0} + + /*! normalize.css v8.0.0 | MIT License | github.com/necolas/normalize.css */*{max-width:100%;box-sizing:border-box}html{font-size:20px;line-height:1.15;-webkit-text-size-adjust:100%}body{font-family:Source Sans Pro,"sans-serif";font-weight:300;-webkit-font-smoothing:antialiased;-moz-osx-font-smoothing:grayscale}body,h1{margin:0}h1{font-size:2rem}hr{box-sizing:content-box;height:0;overflow:visible}pre{font-family:monospace,monospace;font-size:1rem}a{background-color:transparent}abbr[title]{border-bottom:none;text-decoration:underline;-webkit-text-decoration:underline dotted;text-decoration:underline dotted}b,strong{font-weight:bolder}code,kbd,samp{font-family:monospace,monospace;font-size:1rem}small{font-size:80%}sub,sup{font-size:75%;line-height:0;position:relative;vertical-align:baseline}sub{bottom:-.25rem}sup{top:-.5rem}img{border-style:none}button,input,optgroup,select,textarea{font-family:inherit;font-size:100%;line-height:1.15;margin:0}button,input{overflow:visible}button,select{text-transform:none}[type=button],[type=reset],[type=submit],button{-webkit-appearance:button}[type=button]::-moz-focus-inner,[type=reset]::-moz-focus-inner,[type=submit]::-moz-focus-inner,button::-moz-focus-inner{border-style:none;padding:0}[type=button]:-moz-focusring,[type=reset]:-moz-focusring,[type=submit]:-moz-focusring,button:-moz-focusring{outline:1px dotted ButtonText}fieldset{padding:.35em .75em .625rem}legend{box-sizing:border-box;color:inherit;display:table;max-width:100%;padding:0;white-space:normal}progress{vertical-align:baseline}textarea{overflow:auto}[type=checkbox],[type=radio]{box-sizing:border-box;padding:0}[type=number]::-webkit-inner-spin-button,[type=number]::-webkit-outer-spin-button{height:auto}[type=search]{-webkit-appearance:textfield;outline-offset:-2px}[type=search]::-webkit-search-decoration{-webkit-appearance:none}::-webkit-file-upload-button{-webkit-appearance:button;font:inherit}details{display:block}summary{display:list-itrem}[hidden],template{display:none}.pad-s{padding:1rem}.pad-m{padding:2.5rem}.pad-l{padding:5rem}.topicseed-copy{font-family:Georgia,serif;font-weight:700;letter-spacing:1pt;background-image:url(https://topicseed.com/assets/logos/topicseed-visual-black-xs.png);background-size:1.2rem 1.2rem;background-repeat:no-repeat;background-position:100%;padding-right:1.5rem;color:#000}:root{--padding-xs:1rem;--header-size:.9rem;--max-width:1000px;--max-hero-width:70%;--side-padding:5%;--serif:Georgia,"Times New Roman",Times,serif;--sans-serif:"Source Sans Pro","Open Sans",sans;--sans-serif-title:Lato,"Source Sans Pro",sans}.row-valign-center{align-items:center;justify-content:center}.Navigation-module--navigation--2Ttnx{background-color:transparent;display:flex;font-family:Source Sans Pro;font-size:var(--header-size)}.Navigation-module--navigation--2Ttnx *{padding:0;margin:0;line-height:1;list-style-type:none}.Navigation-module--navigation--2Ttnx .Navigation-module--padding--JanqS{padding:1rem}.Navigation-module--navigation--2Ttnx .Navigation-module--branding--QzOe4{flex-shrink:1}.Navigation-module--navigation--2Ttnx .Navigation-module--branding--QzOe4 a{display:block}.Navigation-module--navigation--2Ttnx .Navigation-module--branding--QzOe4 a img{display:inline-block;height:var(--header-size)}.Navigation-module--navigation--2Ttnx .Navigation-module--branding--QzOe4 span{font-family:Georgia,Times New Roman,Times,serif;font-weight:600}.Navigation-module--navigation--2Ttnx .Navigation-module--branding--QzOe4 .Navigation-module--navLink--3Ceps{opacity:1}.Navigation-module--navigation--2Ttnx .Navigation-module--branding--QzOe4 .Navigation-module--navLink--3Ceps:hover{color:#000}.Navigation-module--navigation--2Ttnx .Navigation-module--toggler--1HP5t{display:block;padding:1rem!important;margin:0!important;font-size:1rem;outline:0;border:0;font-weight:600;cursor:pointer;float:right}.Navigation-module--navigation--2Ttnx .Navigation-module--menuToggle--2a0Fx{display:none}.Navigation-module--navigation--2Ttnx .Navigation-module--menuToggle--2a0Fx:checked+.Navigation-module--menu--3ilH3{display:block}.Navigation-module--navigation--2Ttnx .Navigation-module--navLink--3Ceps{padding:1rem 0;margin:0 1rem;display:block;color:#000;font-weight:600;text-decoration:none;opacity:.6}.Navigation-module--navigation--2Ttnx .Navigation-module--navLink--3Ceps.Navigation-module--active--tC-lF{opacity:1;border-top-color:#40e0d0}.Navigation-module--navigation--2Ttnx .Navigation-module--navLink--3Ceps.Navigation-module--active--tC-lF:hover{border-top-color:#40e0d0}.Navigation-module--navigation--2Ttnx .Navigation-module--navLink--3Ceps:hover{opacity:1;color:#40e0d0;border-top-color:#f0f0f0}.Navigation-module--navigation--2Ttnx .Navigation-module--topMenu--2WS0K{flex-grow:1}.Navigation-module--navigation--2Ttnx .Navigation-module--topMenu--2WS0K ul{display:none;text-align:center;padding:5%;position:fixed;top:3rem;left:0;height:100%;width:100%;background-color:#fff;z-index:99999}.Navigation-module--navigation--2Ttnx .Navigation-module--topMenu--2WS0K li{display:block;letter-spacing:.5pt}.Navigation-module--navigation--2Ttnx .Navigation-module--topMenu--2WS0K .Navigation-module--SigninNI--1oVXc{opacity:1!important}.Navigation-module--navigation--2Ttnx .Navigation-module--topMenu--2WS0K .Navigation-module--SigninNI--1oVXc a{background:#40e0d0;color:#fff;padding:.5rem .75em!important;opacity:1!important;border-radius:5px;border:0!important}@media screen and (min-width:771px){.Navigation-module--navigation--2Ttnx .Navigation-module--topMenu--2WS0K ul{display:block;position:static;padding:0;text-align:right}.Navigation-module--navigation--2Ttnx .Navigation-module--topMenu--2WS0K ul li{display:inline-block;letter-spacing:.5pt}.Navigation-module--navigation--2Ttnx .Navigation-module--topMenu--2WS0K .Navigation-module--toggler--1HP5t{display:none}.Navigation-module--navigation--2Ttnx .Navigation-module--navLink--3Ceps{border-top:8px solid #fafafa;opacity:.6}.Navigation-module--navigation--2Ttnx .Navigation-module--navLink--3Ceps:hover{opacity:1;border-top-color:#f0f0f0}.Navigation-module--navigation--2Ttnx .Navigation-module--navLink--3Ceps.Navigation-module--active--tC-lF{opacity:1;border-top-color:#40e0d0}.Navigation-module--navigation--2Ttnx .Navigation-module--navLink--3Ceps.Navigation-module--active--tC-lF:hover{color:#000}}.SiteFooter-module--siteFooter--2ZMf7{padding:5%;background-color:#f9f9f9;font-size:.8rem;border-top:1px solid #fefefe}.SiteFooter-module--LineOfLinks--17SUL{text-align:center}.SiteFooter-module--LineOfLinks--17SUL a{display:inline-block;color:#999;margin:0 .5rem;text-decoration:none}.SiteFooter-module--copyrightNotice--3uQ14{padding:2rem 0 0}.SiteFooter-module--copyrightNotice--3uQ14>p{width:100%;margin:0;text-align:center;color:#999;text-transform:uppercase;letter-spacing:1pt}.SimpleOptinForm-module--container--3JpKE{background-color:#fff;box-shadow:0 0 20px rgba(43,45,56,.08)!important;border:0 solid #eee;transition:all .15s ease;border-radius:5px}.SimpleOptinForm-module--container--3JpKE.SimpleOptinForm-module--fluid--1iPrt{width:100%}.SimpleOptinForm-module--container--3JpKE.SimpleOptinForm-module--maxed--1ICi6{max-width:1000px;margin:5% auto}.SimpleOptinForm-module--inner--1YsM0{padding:5%;background-color:#ff0;border-radius:5px}.SimpleOptinForm-module--inner--1YsM0 fieldset{border:0;padding:0;margin:0}.SimpleOptinForm-module--header--36bBg{text-align:center}.SimpleOptinForm-module--header--36bBg h1{display:block;text-transform:uppercase;font-weight:700;font-family:Lato;letter-spacing:1pt;font-size:1.5rem;margin-bottom:5px}.SimpleOptinForm-module--header--36bBg h2{display:block;font-size:1rem;padding:0;margin:0 0 5px;line-height:1.5}.SimpleOptinForm-module--fields--3BB2p{padding:1em 0 0}.SimpleOptinForm-module--fields--3BB2p .SimpleOptinForm-module--field--jE8AR{padding:0 0 20px;margin:5px 0;order:5;height:100%}.SimpleOptinForm-module--fields--3BB2p input[type=color],.SimpleOptinForm-module--fields--3BB2p input[type=date],.SimpleOptinForm-module--fields--3BB2p input[type=datetime-local],.SimpleOptinForm-module--fields--3BB2p input[type=datetime],.SimpleOptinForm-module--fields--3BB2p input[type=email],.SimpleOptinForm-module--fields--3BB2p input[type=month],.SimpleOptinForm-module--fields--3BB2p input[type=number],.SimpleOptinForm-module--fields--3BB2p input[type=password],.SimpleOptinForm-module--fields--3BB2p input[type=range],.SimpleOptinForm-module--fields--3BB2p input[type=search],.SimpleOptinForm-module--fields--3BB2p input[type=tel],.SimpleOptinForm-module--fields--3BB2p input[type=text],.SimpleOptinForm-module--fields--3BB2p input[type=time],.SimpleOptinForm-module--fields--3BB2p input[type=url],.SimpleOptinForm-module--fields--3BB2p input[type=week],.SimpleOptinForm-module--fields--3BB2p textarea{color:#666;border:1px solid #ccc;border-radius:3px;height:100%;width:100%;padding:1rem;font-weight:700}.SimpleOptinForm-module--fields--3BB2p select{position:relative;width:100%;height:100%;color:#666;background:#fff;line-height:1rem;border:1px solid #ccc;font-weight:700;min-height:3rem;padding-left:1rem}.SimpleOptinForm-module--btnWrapper--140rB .SimpleOptinForm-module--btn--2iMm_{width:100%;border:1px solid;border-color:#ccc #ccc #bbb;border-radius:3px;background:#e6e6e6;color:rgba(0,0,0,.8);font-size:12px;font-size:.75rem;line-height:1;cursor:pointer;outline:none;white-space:nowrap;display:inline-block;height:2.7rem;line-height:2.7rem;padding:0 1.25rem;box-shadow:0 4px 6px rgba(50,50,93,.11),0 1px 3px rgba(0,0,0,.08);background:#fff;border-radius:4px;font-size:1rem;font-weight:600;text-transform:uppercase;letter-spacing:.05rem;color:#000;text-decoration:none;transition:all .15s ease;color:#fff;background-color:#40e0d0;text-shadow:0 1px 3px rgba(36,180,126,.4)}.SimpleOptinForm-module--errors--1zBn2{background-color:#fff;color:red;padding:1rem;margin:5px}.SimpleOptinForm-module--successful--3chKn .SimpleOptinForm-module--inner--1YsM0{background:#008b8b;color:#fff;text-align:center}.SimpleOptinForm-module--successful--3chKn .SimpleOptinForm-module--icon--2twVV{max-width:30%;margin:5% auto}.SimpleOptinForm-module--successful--3chKn .SimpleOptinForm-module--icon--2twVV img{display:block}.SimpleOptinForm-module--successful--3chKn .SimpleOptinForm-module--big--2ofKD{font-weight:700;font-size:1.2rem;max-width:70%;margin:auto}.changelog-module--Timeline--2qo-C,.changelog-module--TimelineItem--Lhj7q{list-style-type:none;padding:0;margin:0}.changelog-module--Timeline--2qo-C{position:relative;padding:5%;margin:auto;max-width:900px}.changelog-module--TimelineItem--Lhj7q{border-left:2px solid #00ced1;padding-left:2rem;padding-bottom:4rem;position:relative}@media (min-width:768px){.changelog-module--TimelineItem--Lhj7q{padding-bottom:3rem}}@media (min-width:1400px){.changelog-module--TimelineItem--Lhj7q{padding-bottom:5rem}}.changelog-module--TimelineItem--Lhj7q h2{margin-top:0;line-height:.5;color:#00ced1;font-size:1.5rem;font-weight:400}.changelog-module--TimelineItem--Lhj7q p{margin:0 0 1rem}.changelog-module--TimelineItem--Lhj7q:before{content:" ";width:14px;height:14px;position:absolute;left:-8px;background-color:#00ced1;border-radius:50%}.changelog-module--TimelineItem--Lhj7q:first-child:before{left:-8px}.changelog-module--TimelineItem--Lhj7q:last-child:before{left:-6px}.changelog-module--TimelineItem--Lhj7q:last-of-type{padding-bottom:0;border:0}.changelog-module--TimelineEntry--4YMAW{padding-top:15px;display:block}.changelog-module--TimelineEntry--4YMAW+.changelog-module--TimelineEntry--4YMAW{margin-top:15px}.changelog-module--TimelineEntry--4YMAW .changelog-module--TimelineEntryLabel--21UAa{font-size:16px;margin-bottom:1.5rem;color:#1a1a1a;display:block}@media (min-width:768px){.changelog-module--TimelineEntry--4YMAW{display:flex}.changelog-module--TimelineEntry--4YMAW .changelog-module--TimelineEntryLabel--21UAa{flex-shrink:1;min-width:100px}.changelog-module--TimelineEntry--4YMAW .changelog-module--TimelineEntryData--36e08{flex-grow:1;padding-left:2rem;padding-top:.25rem}}.changelog-module--Badge--2SdHf{font-weight:700;font-size:.8rem;text-transform:uppercase;padding:5px 10px;border-radius:5px;display:inline-block;letter-spacing:1px}.changelog-module--Badge--2SdHf.changelog-module--Success--1Ed07{background-color:#3dc372;color:#fff}.changelog-module--Badge--2SdHf.changelog-module--Warning--GE_Hw{background-color:#ffb24e;color:#fff}.changelog-module--Badge--2SdHf.changelog-module--Danger--2aJE-{background-color:#e44e56;color:#fff}.index-module--section--l4Kpe{position:relative}.index-module--sectionSplash--Wjz_m{padding:0 0 2rem;background-color:#fff}.index-module--sectionSplash--Wjz_m .index-module--left--1Ebqu{padding:1rem}.index-module--sectionSplash--Wjz_m header{text-align:left!important}@media screen and (min-width:1200px){.index-module--sectionSplash--Wjz_m{padding:5% 0}.index-module--sectionSplash--Wjz_m .index-module--left--1Ebqu{padding:10% 0 10% 10%}}.index-module--sectionSplash--Wjz_m .index-module--right--1qcuo{padding:5%}.index-module--sectionSplash--Wjz_m .index-module--mainTitle--FW4Uw{font-family:Lato;font-size:1.4rem}@media screen and (min-width:800px){.index-module--sectionSplash--Wjz_m .index-module--mainTitle--FW4Uw{font-size:1.9rem}}.index-module--sectionSplash--Wjz_m .index-module--changelogLink---xSR7{margin-top:.5rem}.index-module--sectionSplash--Wjz_m .index-module--changelogLink---xSR7 span{font-size:.85rem;font-style:italic}.index-module--sectionSplash--Wjz_m .index-module--changelogLink---xSR7 a{text-decoration:none;color:#999}.index-module--sectionSplash--Wjz_m p{margin-bottom:0;padding-bottom:0}.index-module--sectionSplash--Wjz_m .index-module--subheader--3gKPq{line-height:1.4}.index-module--sectionSplash--Wjz_m .index-module--splashScreenshot--2OaaW{max-width:100%;display:block;border-radius:3px;box-shadow:0 0 20px rgba(43,45,56,.08)!important;border:0 solid #eee}.index-module--sectionSplash--Wjz_m .index-module--btnList--K3Sjr>*{margin-bottom:.25rem}.index-module--sectionSplash--Wjz_m .index-module--btnList--K3Sjr>:last-child{opacity:.66}.index-module--sectionMentions--370ct{text-align:center;background-color:#f9f9f9;padding:2rem 1rem}.index-module--sectionMentions--370ct .index-module--mentionHeader--cYdR- p{display:block;text-transform:uppercase;color:#999;font-size:1.25em;text-align:center}.index-module--sectionMentions--370ct .index-module--mentionBox--1BzX5{padding:1rem 2rem}.index-module--sectionMentions--370ct .index-module--mentionBox--1BzX5 .index-module--imgMention--3R5vh{display:block;width:100%}@media screen and (min-width:900px){.index-module--sectionMentions--370ct{padding:2rem}.index-module--sectionMentions--370ct .index-module--mentionBox--1BzX5{padding:2rem 1rem}}.index-module--sectionPersonas--14yif{padding:2rem 0}@media screen and (min-width:900px){.index-module--sectionPersonas--14yif{padding:5% 0}}.index-module--sectionPersonas--14yif h3{color:#000;margin:0 0 2rem;text-align:center;font-size:1rem}@media screen and (min-width:900px){.index-module--sectionPersonas--14yif h3{font-size:1.7rem}}@media screen and (min-width:1200px){.index-module--sectionPersonas--14yif{padding:10% 5% 5%}}.index-module--sectionPersonas--14yif .index-module--box--2C1yX{border-radius:10px;box-shadow:0 0 3rem hsla(0,0%,39.2%,.1);background-color:hsla(0,0%,78.4%,.2);padding:2rem;margin-bottom:2rem;text-align:center}.index-module--sectionPersonas--14yif .index-module--box--2C1yX .index-module--heading--2EiHG{white-space:nowrap;line-height:1;text-transform:uppercase;margin-bottom:1rem}.index-module--sectionPersonas--14yif .index-module--box--2C1yX .index-module--heading--2EiHG img{height:3rem;display:inline-block}.index-module--sectionPersonas--14yif .index-module--box--2C1yX .index-module--heading--2EiHG h4{font-family:var(--sans-serif);font-size:1.5rem}.index-module--sectionPersonas--14yif .index-module--box--2C1yX p{font-family:var(--sans-serif);margin:0;padding:0;font-size:1rem}.index-module--Feature--3DL7n{padding:5% 0}.index-module--Feature--3DL7n .index-module--container--2IQ43{margin:auto}.index-module--Feature--3DL7n .index-module--Description--3WXED li,.index-module--Feature--3DL7n .index-module--Description--3WXED ul{list-style:none;padding:0;margin:0}.index-module--Feature--3DL7n .index-module--Description--3WXED li{line-height:1.5;font-size:.8em}@media screen and (min-width:800px){.index-module--Feature--3DL7n .index-module--Description--3WXED li{font-size:1em}}.index-module--Feature--3DL7n .index-module--Description--3WXED .index-module--svgListItem--1SQhP{height:1em;width:1em;vertical-align:middle;display:inline-block;margin-right:.5rem}@media screen and (min-width:800px){.index-module--Feature--3DL7n .index-module--Description--3WXED{font-size:1.25rem}}.index-module--Feature--3DL7n .index-module--Head--IJzHL{text-align:left;padding-bottom:1rem}.index-module--Feature--3DL7n .index-module--Head--IJzHL img{width:1.5rem}@media screen and (min-width:900px){.index-module--Feature--3DL7n .index-module--Head--IJzHL img{width:3rem}}.index-module--Feature--3DL7n .index-module--Head--IJzHL h2{line-height:1;font-size:1.5rem;margin:.5rem 0;padding:0}.index-module--Feature--3DL7n .index-module--Head--IJzHL h2 em,.index-module--Feature--3DL7n .index-module--Head--IJzHL h2 strong{font-family:Lato;text-transform:uppercase;font-style:normal}.index-module--Feature--3DL7n .index-module--Head--IJzHL h2 strong{font-weight:800}.index-module--Feature--3DL7n .index-module--Head--IJzHL h2 em{font-weight:100}@media screen and (min-width:900px){.index-module--Feature--3DL7n .index-module--Head--IJzHL h2{font-size:2.3rem}.index-module--Feature--3DL7n .index-module--Head--IJzHL h2 em{display:block}}.index-module--sectionLeadGenA--2Wqpy{padding:0 5% 5%;color:#fff;text-align:center}.index-module--sectionLeadGenA--2Wqpy .index-module--box--2C1yX{background-color:#40e0d0;padding:3% 10%}.index-module--sectionLatestBlogs--3ZHEt{margin-top:5%;padding:5%;background-color:#f9f9f9}.pricing-module--Container--tuRnU{padding:0 0 2rem}.pricing-module--RedAnn--16W00{text-align:center;margin:5% 0 0}.pricing-module--RedAnn--16W00 p{display:inline-block;background:#008b8b;color:#fff;font-weight:700;padding:1rem;margin:0 1rem 1rem;border-radius:10px}@media (max-width:900px){.pricing-module--paddedColLg--1M4yw{padding:0;margin:0}}.pricing-module--Comparison--Cbw4q{max-width:1200px;margin:0 auto;text-align:center;box-shadow:0 0 2rem rgba(68,74,102,.15);border-radius:5px}@media (min-width:900px){.pricing-module--Comparison--Cbw4q{margin:5% auto 2rem}}.pricing-module--Comparison--Cbw4q table{width:100%;border-collapse:collapse;border-spacing:0}.pricing-module--Comparison--Cbw4q td,.pricing-module--Comparison--Cbw4q th{empty-cells:show;padding:1rem}.pricing-module--Comparison--Cbw4q .pricing-module--Heading--2voQH{font-size:1.1rem;font-weight:700!important;border-bottom:0!important;padding-bottom:.5rem!important;padding-top:1rem!important}@media (min-width:900px){.pricing-module--Comparison--Cbw4q .pricing-module--Heading--2voQH{padding-top:2rem!important}}.pricing-module--Comparison--Cbw4q tfoot{background-color:#f5f5f5}.pricing-module--Comparison--Cbw4q tbody .pricing-module--FeatureNameRow--33vpW{display:none}.pricing-module--Comparison--Cbw4q tbody tr.pricing-module--GroupRow--ICQ71{color:#195a5b;background:#afeeee;font-weight:700}.pricing-module--Comparison--Cbw4q tbody tr.pricing-module--GroupRow--ICQ71 td{text-align:center}.pricing-module--Comparison--Cbw4q .pricing-module--Row--1jsXC{background:#f5f5f5}.pricing-module--Comparison--Cbw4q .pricing-module--TrWords--PzU-G{font-weight:900}.pricing-module--Comparison--Cbw4q .pricing-module--BlueTick--2_BxZ,.pricing-module--Comparison--Cbw4q .pricing-module--GreenTick--OVE5b,.pricing-module--Comparison--Cbw4q .pricing-module--RedTick--3es5i{font-weight:400}.pricing-module--Comparison--Cbw4q .pricing-module--BlueTick--2_BxZ{color:#0078c1}.pricing-module--Comparison--Cbw4q .pricing-module--RedTick--3es5i{color:#8b0000}.pricing-module--Comparison--Cbw4q .pricing-module--GreenTick--OVE5b{color:#009e2c}.pricing-module--Comparison--Cbw4q th{font-weight:400;padding:0}.pricing-module--Comparison--Cbw4q tr td:first-child{font-weight:700;text-align:left}.pricing-module--Comparison--Cbw4q .pricing-module--Qbo--3r0N0,.pricing-module--Comparison--Cbw4q .pricing-module--Qbse--2LRrW,.pricing-module--Comparison--Cbw4q .pricing-module--Tl--2lWYs{color:#fff;padding:1rem;font-size:13px;border-bottom:0}.pricing-module--Comparison--Cbw4q .pricing-module--Heading--2voQH th,.pricing-module--Comparison--Cbw4q .pricing-module--Tl--2lWYs th{padding-top:2rem}.pricing-module--Comparison--Cbw4q .pricing-module--Qbse--2LRrW{background:#0078c1;border-top-left-radius:3px;border-left:0}.pricing-module--Comparison--Cbw4q .pricing-module--Qbo--3r0N0{background:#009e2c;border-top-right-radius:3px;border-right:0}.pricing-module--Comparison--Cbw4q .pricing-module--PriceButton--2nANz{display:none;margin:1rem 0}.pricing-module--Comparison--Cbw4q .pricing-module--PriceInfo--OibUh{padding:.25rem}@media (min-width:639px){.pricing-module--Comparison--Cbw4q .pricing-module--PriceInfo--OibUh{padding:.5rem 1.5rem 1.5rem}}.pricing-module--Comparison--Cbw4q .pricing-module--PriceWas--V1z1e{display:none;color:#999;text-decoration:line-through}.pricing-module--Comparison--Cbw4q .pricing-module--PriceNow--3bQ-P span{color:#00ced1;font-size:2.5rem;font-weight:700}.pricing-module--Comparison--Cbw4q .pricing-module--PriceNow--3bQ-P span .pricing-module--PriceSmall--2JZYR{font-size:.6rem}.pricing-module--Comparison--Cbw4q .pricing-module--PriceTry--1DZ0J{display:none;font-size:.8rem}.pricing-module--Comparison--Cbw4q .pricing-module--PriceTry--1DZ0J a{color:#202020}@media (max-width:900px){.pricing-module--Comparison--Cbw4q td:first-child,.pricing-module--Comparison--Cbw4q th:first-child{display:none}.pricing-module--Comparison--Cbw4q tbody tr.pricing-module--GroupRow--ICQ71{display:table-row}.pricing-module--Comparison--Cbw4q tbody tr.pricing-module--GroupRow--ICQ71 td{display:table-cell}.pricing-module--Comparison--Cbw4q tbody tr.pricing-module--FeatureNameRow--33vpW{display:table-row;background:#f7f7f7}.pricing-module--Comparison--Cbw4q .pricing-module--Row--1jsXC{background:#fff}.pricing-module--Comparison--Cbw4q .pricing-module--PriceInfo--OibUh{border-top:0!important}}@media (max-width:639px){.pricing-module--Comparison--Cbw4q .pricing-module--PriceBuy--2O6fz{padding:.5rem 1rem}.pricing-module--Comparison--Cbw4q td,.pricing-module--Comparison--Cbw4q th{padding:.7rem .5rem}.pricing-module--Comparison--Cbw4q .pricing-module--HideMobile--3uRP8{display:none}.pricing-module--Comparison--Cbw4q .pricing-module--PriceNow--3bQ-P span{font-size:16px}.pricing-module--Comparison--Cbw4q .pricing-module--PriceSmall--2JZYR{font-size:16px!important;top:0;left:0}.pricing-module--Comparison--Cbw4q .pricing-module--Qbo--3r0N0,.pricing-module--Comparison--Cbw4q .pricing-module--Qbse--2LRrW{font-size:12px;padding:1rem .5rem}.pricing-module--Comparison--Cbw4q .pricing-module--PriceBuy--2O6fz{margin-top:1rem}.pricing-module--Comparison--Cbw4q .pricing-module--Heading--2voQH{font-size:13px}}.pricing-module--Comparison--Cbw4q .pricing-module--Promo--1P4K_{position:relative;text-align:left}.pricing-module--Comparison--Cbw4q .pricing-module--Promo--1P4K_ div{font-weight:700;line-height:1.3;color:#fff;position:relative;z-index:1;height:10rem;width:15.5rem;padding:1rem 0;left:.25rem;display:flex;align-items:center}.pricing-module--Comparison--Cbw4q .pricing-module--Promo--1P4K_ div span{display:block;padding-right:6rem;font-size:1.5rem;margin:-.5rem 0 0}.pricing-module--Comparison--Cbw4q .pricing-module--Promo--1P4K_ div:before{content:"";position:absolute;top:0;left:-1.55rem;width:100%;height:100%;background:url(https://copywritely.com/wp-content/themes/copywritely/design/assets/img/new-pricing/Vector-1.svg) no-repeat;background-size:contain;z-index:-1}.pricing-module--Comparison--Cbw4q .pricing-module--Promo--1P4K_ div:after{content:"";position:absolute;top:-6px;left:-1rem;width:1rem;height:100%;background:url(https://copywritely.com/wp-content/themes/copywritely/design/assets/img/new-pricing/Vector.svg) no-repeat;background-size:contain;z-index:-2}.StartScreen-module--StartScreen--2axSL{max-width:800px;margin-left:auto;margin-right:auto;text-align:center;padding:5% 0}.StartScreen-module--PageSelector--lVxle input{font-size:1rem;height:3rem;line-height:1rem;padding-left:.4rem;-ms-box-sizing:border-box;-o-box-sizing:border-box;box-sizing:border-box;width:100%;border-radius:5px;-ms-box-shadow:inset 0 1px 2px rgba(0,0,0,.18);-o-box-shadow:inset 0 1px 2px rgba(0,0,0,.18);box-shadow:inset 0 1px 2px rgba(0,0,0,.18);-webkit-font-smoothing:antialiased;border:1px solid #bbb;border-top-color:#999;background:#fff}.StartScreen-module--PageSelector--lVxle ul{display:flex;list-style:none;padding:0;margin:0;flex-flow:row wrap}.StartScreen-module--PageSelector--lVxle ul li{flex-grow:1;display:block}.StartScreen-module--PageSelector--lVxle ul li strong{color:#999;font-weight:700;font-size:.9rem;letter-spacing:.3pt;border-radius:5px;margin:5px;background:#f9f9f9;display:block;padding:.5rem;cursor:pointer}.StartScreen-module--PageSelector--lVxle ul li strong:hover{color:#fff;background:#3ecf8e;text-shadow:0 1px 3px rgba(36,180,126,.4)}.Categories-module--CategoriesItem--2cxew,.Categories-module--CategoriesList--11a-r{padding:0;margin:0;list-style-type:none}.Categories-module--CategoriesList--11a-r{display:flex;flex-flow:row wrap;align-items:center;justify-content:center}.Categories-module--CategoryItem--1z3tl{flex-shrink:1;padding:.5rem;background:#fefbf3;border:1px solid #fff}.Categories-module--CategoryBox--3ZiK8 a{padding:5px 0 5px 1.75rem;display:inline-block;background-image:url(https://image.flaticon.com/icons/svg/148/148946.svg);background-repeat:no-repeat;background-size:1.1em 1.1rem;background-position:0;font-size:.7rem;color:#eb7937}.Header-module--header--3iSqR{padding:3% 1rem;background:#40e0d0;color:#fff}.Header-module--header--3iSqR h1{font-size:2.5rem;padding:0;margin:0;text-align:center}.Sections-module--sections--1_XZN{padding:5% 0}.Sections-module--SectionList--2P5Kd{position:relative;text-align:center;padding:0;margin:0}.Sections-module--SectionList--2P5Kd a{text-decoration:none}.Sections-module--SectionList--2P5Kd li,.Sections-module--SectionList--2P5Kd ul{list-style-type:none;padding:0;margin:0}.Sections-module--SectionList--2P5Kd .Sections-module--SectionList--2P5Kd{margin:2%}.Sections-module--SectionList--2P5Kd .Sections-module--ChildrenList--PftMZ .Sections-module--SectionBox--cSSDW{border-radius:5px}.Sections-module--SectionList--2P5Kd .Sections-module--SectionBox--cSSDW{padding:3% 2%;margin-bottom:3%}.Sections-module--SectionList--2P5Kd .Sections-module--SectionBox--cSSDW.Sections-module--depth1--3mZ7e{background:#f4f4f4}.Sections-module--SectionList--2P5Kd .Sections-module--SectionBox--cSSDW.Sections-module--depth2--3hZ_z{background:#e6e6e6}.Sections-module--SectionList--2P5Kd .Sections-module--SectionBox--cSSDW.Sections-module--depth3--yRnNL{background:#d2d2d2}.Sections-module--SectionList--2P5Kd .Sections-module--SectionBox--cSSDW.Sections-module--depth4--29pv1{background:#b7b7b7}.Sections-module--SectionList--2P5Kd :target:before{display:block;content:" ";padding-top:70px;visibility:hidden;pointer-events:none}.Sections-module--SectionList--2P5Kd .Sections-module--SectionBox--cSSDW .Sections-module--SectionTitle--2RPrv{margin:0 0 50px;padding:0;line-height:1;font-size:1rem;opacity:1;font-weight:700}.Sections-module--SectionList--2P5Kd .Sections-module--SectionBox--cSSDW .Sections-module--SectionTitle--2RPrv.Sections-module--d1--2w3x-{font-size:2rem}.Sections-module--SectionList--2P5Kd .Sections-module--SectionBox--cSSDW .Sections-module--SectionTitle--2RPrv.Sections-module--d2--30bth{font-size:1.6rem}.Sections-module--SectionList--2P5Kd .Sections-module--SectionBox--cSSDW .Sections-module--SectionTitle--2RPrv.Sections-module--d3--3A50T{font-size:1.3rem}.Sections-module--SectionList--2P5Kd .Sections-module--SectionBox--cSSDW .Sections-module--SectionTitle--2RPrv.Sections-module--d4--18dhX{font-size:1.1rem}.Sections-module--SectionList--2P5Kd .Sections-module--SectionBox--cSSDW .Sections-module--SectionTitle--2RPrv.Sections-module--d5---0OCD,.Sections-module--SectionList--2P5Kd .Sections-module--SectionBox--cSSDW .Sections-module--SectionTitle--2RPrv.Sections-module--d6--1ITSu{font-size:1rem}.Sections-module--SectionList--2P5Kd .Sections-module--SectionTitle--2RPrv small{display:inline-block;opacity:.7;margin-left:.5rem;font-size:.7rem;vertical-align:middle}.Sections-module--SectionList--2P5Kd .Sections-module--SectionDetails--3aWNI{font-size:.7rem;padding:0}.Sections-module--SectionList--2P5Kd ul.Sections-module--SectionDetailsList--vmAzD{display:flex;flex-flow:row wrap}.Sections-module--SectionList--2P5Kd ul.Sections-module--SectionDetailsList--vmAzD>li{flex:1 1}.Sections-module--SectionList--2P5Kd .Sections-module--SectionDetailsList--vmAzD>li em{font-style:normal;font-weight:700;text-transform:uppercase;margin-right:1rem;opacity:.6}.Sections-module--SectionList--2P5Kd .Sections-module--SectionDetailsList--vmAzD>li .Sections-module--all--1bdeY,.Sections-module--SectionList--2P5Kd .Sections-module--SectionDetailsList--vmAzD>li .Sections-module--own--1YvPj{opacity:.5}.Sections-module--SectionList--2P5Kd .Sections-module--SectionDetailsList--vmAzD>li .Sections-module--own--1YvPj{margin-right:.5rem}.Sections-module--SectionList--2P5Kd .Sections-module--SectionDetailsList--vmAzD>li .Sections-module--all--1bdeY{opacity:.3}.Sections-module--SectionList--2P5Kd .Sections-module--SectionDetails--3aWNI li{display:block}.Sections-module--SectionList--2P5Kd ul.Sections-module--LinkList--RV43i{display:flex;flex-flow:row wrap;padding-top:1rem;justify-content:center;align-items:center}.Sections-module--SectionList--2P5Kd li.Sections-module--linkToItem--1Lhxz{margin:0 .5em .5em 0;flex-shrink:1}.Sections-module--SectionList--2P5Kd .Sections-module--linkToItem--1Lhxz>span{cursor:pointer;padding:.3em .6rem;border:1px solid #fff;background:#fff;border-radius:4px;flex-shrink:1;display:inline-block;font-size:.7rem;color:#999}.Sections-module--SectionList--2P5Kd .Sections-module--linkToItem--1Lhxz>span:hover{background:#999;color:#fff}.Sections-module--SectionList--2P5Kd>li:last-child .Sections-module--SectionBox--cSSDW{margin-bottom:0}.Related-module--related--mNNbZ li,.Related-module--related--mNNbZ ul{list-style-type:none;padding:0;margin:0;display:block;text-align:center}.Related-module--related--mNNbZ span{display:block;text-decoration:none;cursor:pointer}.Related-module--related--mNNbZ .Related-module--linkItem--3FTnw{position:relative}.Related-module--related--mNNbZ .Related-module--bg--32Ltm{position:absolute;left:0;right:0;margin-left:auto;margin-right:auto;height:100%;background:rgba(64,224,208,.2);z-index:1}.Related-module--related--mNNbZ .Related-module--box--2DbuI{position:relative;padding:5px;border-bottom:1px solid #fff;z-index:2;text-align:center}.Related-module--related--mNNbZ .Related-module--label--1ZZ9i,.Related-module--related--mNNbZ em{position:relative;display:inline-block;z-index:2;font-size:.8rem;padding:0;margin-top:10px;vertical-align:-webkit-baseline-middle}.Related-module--related--mNNbZ .Related-module--label--1ZZ9i{font-weight:700;color:#000;padding-right:.5rem}.Related-module--related--mNNbZ em{background:#fff;color:#666;min-width:2.5rem;text-align:center;font-weight:700;border-radius:5px;margin-left:5px;font-size:.7rem;padding:5px}.TableOfContents-module--ToC--2HVzo{background-color:#fff}.TableOfContents-module--ToC--2HVzo li,.TableOfContents-module--ToC--2HVzo ol{list-style-type:none;padding:0;margin:0}.TableOfContents-module--ToC--2HVzo a{text-decoration:none}.TableOfContents-module--TableOfContentsList--2LCac{display:flex;flex-flow:row wrap;text-align:left}.TableOfContents-module--TableOfContentsList--2LCac:first-child{counter-reset:section!important;list-style-type:none!important}.TableOfContents-module--TableOfContentsList--2LCac .TableOfContents-module--Shadowed--2Iym6{background-color:#fff;border-radius:5px;box-shadow:0 0 20px rgba(43,45,56,.08)!important;border:0 solid #eee;transition:all .15s ease}.TableOfContents-module--TableOfContentsList--2LCac ol{counter-reset:section!important;list-style-type:none!important;background:hsla(0,0%,78.4%,.2);font-size:90%}.TableOfContents-module--TableOfContentsList--2LCac .TableOfContents-module--TocSectionTitle--kaY7f{margin:0;line-height:1.5;opacity:1;font-weight:300;font-size:90%}.TableOfContents-module--TableOfContentsList--2LCac .TableOfContents-module--TocSectionTitle--kaY7f a{display:block;color:#666;padding:.5rem}.TableOfContents-module--TableOfContentsList--2LCac .TableOfContents-module--TocSectionTitle--kaY7f a:hover{color:#000}.TableOfContents-module--TableOfContentsList--2LCac .TableOfContents-module--TocSectionTitle--kaY7f a:before{counter-increment:section!important;content:counters(section,".") ". "!important}.TableOfContents-module--TableOfContentsList--2LCac>li{flex-grow:1}.TableOfContents-module--TableOfContentsList--2LCac>li>.TableOfContents-module--TocSectionBox--2ZPU0{margin:1rem;background:#fefefe;border:1px solid #f9f9f9;border-radius:5px}.TableOfContents-module--TableOfContentsList--2LCac>li>.TableOfContents-module--TocSectionBox--2ZPU0>.TableOfContents-module--TocSectionTitle--kaY7f{font-weight:700;margin:0}.TableOfContents-module--TableOfContentsList--2LCac>li:last-child .TableOfContents-module--TocSectionTitle--kaY7f{border-bottom:0}.TableOfContents-module--TableOfContentsList--2LCac>li .TableOfContents-module--TableOfContents--2GtsK .TableOfContents-module--SectionBox--3-WZI.TableOfContents-module--depth1--1qbk4{background:#f4f4f4}.TableOfContents-module--TableOfContentsList--2LCac>li .TableOfContents-module--TableOfContents--2GtsK .TableOfContents-module--SectionBox--3-WZI.TableOfContents-module--depth2--_vEYp{background:#e6e6e6}.TableOfContents-module--TableOfContentsList--2LCac>li .TableOfContents-module--TableOfContents--2GtsK .TableOfContents-module--SectionBox--3-WZI.TableOfContents-module--depth3--szDg6{background:#d2d2d2}.TableOfContents-module--TableOfContentsList--2LCac>li .TableOfContents-module--TableOfContents--2GtsK .TableOfContents-module--SectionBox--3-WZI.TableOfContents-module--depth4--3OJcj{background:#b7b7b7}.BigLoader-module--BigLoader--3aHwt p{font-size:2rem;font-weight:100;max-width:600px;margin:5% auto;text-align:center}.BigLoader-module--BigLoader--3aHwt .BigLoader-module--spinner--1n6pq{margin:5em auto;width:200px;height:200px;position:relative;text-align:center;animation:BigLoader-module--sk-rotate--1cV2B 2s linear infinite}.BigLoader-module--BigLoader--3aHwt .BigLoader-module--dot1--g3Ddg,.BigLoader-module--BigLoader--3aHwt .BigLoader-module--dot2--2BVb6{width:60%;height:60%;display:inline-block;position:absolute;top:0;background-color:#40e0d0;border-radius:100%;animation:BigLoader-module--sk-bounce--1Lgu8 2s ease-in-out infinite}.BigLoader-module--BigLoader--3aHwt .BigLoader-module--dot2--2BVb6{top:auto;bottom:0;animation-delay:-1s;background-color:#40e0d0}@keyframes BigLoader-module--sk-rotate--1cV2B{to{transform:rotate(1turn);-webkit-transform:rotate(1turn)}}@keyframes BigLoader-module--sk-bounce--1Lgu8{0%,to{transform:scale(0);-webkit-transform:scale(0)}50%{transform:scale(1);-webkit-transform:scale(1)}}.PageHeader-module--pageHeader--3O4Vw{text-align:center;background-color:#f9f9f9}.PageHeader-module--inner---KwJO{padding:5%}.PageHeader-module--inner---KwJO>p{margin-left:auto;margin-right:auto;font-weight:400;line-height:1.5}.PageHeader-module--inner---KwJO>p:last-child{margin-bottom:0}@media (min-width:768px){.PageHeader-module--inner---KwJO>p{max-width:60%;font-size:1.5rem}}.FeatureListItem-module--Section--14Rwp{padding:3rem 1rem;margin:auto}@media screen and (min-width:991px){.FeatureListItem-module--Section--14Rwp{padding:5% 2rem}}@media screen and (max-width:991px){.FeatureListItem-module--Section--14Rwp .FeatureListItem-module--Row--o6opq>*{margin:0;padding:0}}.FeatureListItem-module--Section--14Rwp .FeatureListItem-module--Visual--FLmjs img{display:block;border-radius:10px;padding:0;margin-left:auto;margin-right:auto;width:80%;max-width:500px}@media screen and (min-width:991px){.FeatureListItem-module--Section--14Rwp .FeatureListItem-module--Visual--FLmjs img{width:100%;max-width:none}}.FeatureListItem-module--Section--14Rwp .FeatureListItem-module--Text--31U5h{position:relative;vertical-align:middle;text-align:center}.FeatureListItem-module--Section--14Rwp .FeatureListItem-module--Text--31U5h h2{font-size:2rem}@media screen and (min-width:991px){.FeatureListItem-module--Section--14Rwp .FeatureListItem-module--Text--31U5h h2{margin:0 0 1em}}.FeatureListItem-module--Section--14Rwp .FeatureListItem-module--Text--31U5h .FeatureListItem-module--Description--1Yj5u{padding:0 1rem}@media screen and (min-width:991px){.FeatureListItem-module--Section--14Rwp .FeatureListItem-module--Text--31U5h .FeatureListItem-module--Description--1Yj5u{padding:0}}.FeatureListItem-module--Section--14Rwp .FeatureListItem-module--Text--31U5h .FeatureListItem-module--Description--1Yj5u p{letter-spacing:.5pt;font-weight:100;color:#666;font-size:1.1rem}.FeatureListItem-module--Section--14Rwp .FeatureListItem-module--Text--31U5h .FeatureListItem-module--Description--1Yj5u p strong{letter-spacing:.1pt;font-weight:400;color:#000}@media screen and (min-width:991px){.FeatureListItem-module--Section--14Rwp .FeatureListItem-module--Text--31U5h{padding:0 5% 0 10%}}@media screen and (min-width:1200px){.FeatureListItem-module--Section--14Rwp .FeatureListItem-module--Text--31U5h{padding:0 20% 0 10%}}@media screen and (min-width:991px){.FeatureListItem-module--Section--14Rwp.FeatureListItem-module--NotReverse--3-77E .FeatureListItem-module--Text--31U5h{text-align:left}.FeatureListItem-module--Section--14Rwp.FeatureListItem-module--Reverse--2CLbw .FeatureListItem-module--Text--31U5h{text-align:right;padding:0 10% 0 5%}}@media screen and (min-width:1200px){.FeatureListItem-module--Section--14Rwp.FeatureListItem-module--Reverse--2CLbw .FeatureListItem-module--Text--31U5h{text-align:right;padding:0 10% 0 20%}}.resources-module--Container--2tk_N{background-color:#f9f9f9;padding:0 1rem 2rem;margin:0}.resources-module--Container--2tk_N ol{list-style:none;max-width:1200px;margin:0 auto;padding:0;line-height:1}.resources-module--RI--U3JIv{position:relative;box-shadow:0 0 2rem hsla(0,0%,78.4%,.3);border-radius:10px;margin-bottom:2rem}.resources-module--Index--3vmnU{display:none;text-align:center;padding:1rem}.resources-module--Index--3vmnU span{font-size:2rem;font-weight:900;font-family:Lato,sans-serif;text-rendering:optimizeLegibility}.resources-module--Text--2NHJ6{padding:1rem}.resources-module--Text--2NHJ6 h3{margin:0 0 1rem;padding:0}.resources-module--Visual--1ktgu{padding:1rem}.resources-module--Visual--1ktgu svg{padding:5%}.resources-module--Visual--1ktgu img{width:100%;border-radius:10px!important}@media screen and (min-width:900px){.resources-module--Inner--2yfpW{display:flex;margin:0;max-width:none}.resources-module--Inner--2yfpW>*{align-self:center}.resources-module--Index--3vmnU{flex:1 1}.resources-module--Text--2NHJ6{flex:5 1}.resources-module--Visual--1ktgu{width:40%;flex:4 1}.resources-module--Visual--1ktgu img{padding:0}}@media screen and (min-width:1100px){.resources-module--Index--3vmnU{display:block}}.tags-module--vignetteList--2EhMa{padding:0 5% 5%;list-style-type:none;align-items:center;justify-content:center}.tags-module--vignetteList--2EhMa>*{padding:1%;list-style-type:none}.tags-module--vignetteList--2EhMa a{line-height:1;text-decoration:none;display:block;padding:1rem;border-radius:5px;background-color:#fff;color:#00ced1;font-weight:700;text-transform:uppercase;border:3px solid #00ced1}.tags-module--vignetteList--2EhMa a:hover{background-color:#00ced1;color:#fff}.tags-module--vignetteList--2EhMa a small{background:#fff;display:inline-block;text-align:center;color:#000;font-size:.7rem;padding:5px 8px;border-radius:50%}.ArticleVignette-module--vignette--z54ew{position:relative;background-color:#fff;border-radius:5px;box-shadow:0 0 10px rgba(43,45,56,.06)!important;border:0 solid #eee;transition:all .15s ease;margin:0 0 1.5rem}.ArticleVignette-module--vignette--z54ew:hover{box-shadow:0 0 20px rgba(43,45,56,.08)!important}.ArticleVignette-module--vignette--z54ew a{text-decoration:none;color:#000}.ArticleVignette-module--visual--2tA77{display:block}.ArticleVignette-module--heroImage--3Mc3z{height:auto;max-width:100%;width:100%;display:block;border-top-right-radius:5px;border-top-left-radius:5px}.ArticleVignette-module--textual--27b6d{padding:30px;border:none}.ArticleVignette-module--title--m_Ood{margin:0;padding:0;font-size:1.25rem;font-family:Lato}.ArticleVignette-module--meta--1q63w{font-weight:400;text-align:left;font-size:.8rem;letter-spacing:1px}.ArticleVignette-module--meta--1q63w>span{display:inline-block;margin:.75rem 0 .25rem;color:#666;white-space:nowrap;overflow:hidden;text-overflow:clip;padding-right:16px}.ArticleVignette-module--meta--1q63w>span:last-child{border-right:0;padding-right:0;margin-right:0}.ArticleVignette-module--updated--1iGG3{display:none}.ArticleVignette-module--excerpt--Kdaxc{padding:0;margin:0 auto;color:#666;font-size:.9rem;line-height:1.4}.blog-module--blogHeader--2rEBf{padding:5%;text-align:center}.blog-module--subheader--1bWY5{text-align:center;max-width:800px;padding:1rem;margin:auto;font-size:1.5rem;font-weight:100;font-style:italic;color:#999}.blog-module--vignetteList--BdqsD{padding:0 5% 5%}.blog-module--vignetteList--BdqsD>*{padding:2.5%;align-items:center;justify-content:center}.tag-module--vignetteList--1tB18{padding:0 5% 5%}.tag-module--vignetteList--1tB18>*{padding:2.5%;align-items:center;justify-content:center}.tag-module--bottom--28SSj{text-align:center;padding:0 5% 5%}.index-module--TeaserOptin--FVea1{font-size:1rem}.index-module--TeaserOptin--FVea1 header{font-weight:700;padding-bottom:.5em}@media screen and (max-width:600px){.index-module--TeaserOptin--FVea1{font-size:.8em}}.index-module--container--34rxR{background-color:#fff;box-shadow:0 0 20px rgba(43,45,56,.08)!important;border:0 solid #eee;transition:all .15s ease;border-radius:5px;background:#f9f9f9}.index-module--container--34rxR fieldset,.index-module--container--34rxR p{border:0;padding:0;margin:0}.index-module--inner--3kOD7{background:#fff;padding:.5em .5em .5em .75em}.index-module--inner--3kOD7 .index-module--email--3jzSr,.index-module--inner--3kOD7 .index-module--website--2It4m{display:flex}.index-module--inner--3kOD7 .index-module--email--3jzSr>input,.index-module--inner--3kOD7 .index-module--website--2It4m>input{flex-grow:1}.index-module--inner--3kOD7 .index-module--email--3jzSr .index-module--submitter--2CsUF,.index-module--inner--3kOD7 .index-module--website--2It4m .index-module--submitter--2CsUF{flex-shrink:1}.index-module--inner--3kOD7 input[type=color],.index-module--inner--3kOD7 input[type=date],.index-module--inner--3kOD7 input[type=datetime-local],.index-module--inner--3kOD7 input[type=datetime],.index-module--inner--3kOD7 input[type=email],.index-module--inner--3kOD7 input[type=month],.index-module--inner--3kOD7 input[type=number],.index-module--inner--3kOD7 input[type=password],.index-module--inner--3kOD7 input[type=range],.index-module--inner--3kOD7 input[type=search],.index-module--inner--3kOD7 input[type=tel],.index-module--inner--3kOD7 input[type=text],.index-module--inner--3kOD7 input[type=time],.index-module--inner--3kOD7 input[type=url],.index-module--inner--3kOD7 input[type=week],.index-module--inner--3kOD7 textarea{color:#666;border:1px solid #fff;border-radius:3px;padding:0;font-weight:400;outline:none}.index-module--Steps--W963K{display:flex;flex-flow:column wrap;flex-shrink:1;padding:0 0 0 .5em;justify-content:center;flex-direction:column;text-align:center}.index-module--Steps--W963K>div{flex-shrink:1;width:10px;height:10px;border-radius:50%;margin:3px 0;border:1px solid}.index-module--Steps--W963K>div.index-module--Done--3wzyP{background:#40e0d0;border-color:#40e0d0}.index-module--Steps--W963K>div.index-module--NotDone--1xW0M{border-color:rgba(175,238,238,.5)}.index-module--btn--324SS{border:0;width:100%;border-radius:3px;background:#e6e6e6;color:rgba(0,0,0,.8);font-size:12px;font-size:.75em;cursor:pointer;outline:none;white-space:nowrap;display:inline-block;height:2.7em;line-height:1;padding:0 1.25em;box-shadow:0 4px 6px rgba(50,50,93,.11),0 1px 3px hsla(0,0%,39.2%,.08);background:#fff;border-radius:4px;font-size:1em;font-weight:600;text-transform:uppercase;letter-spacing:.05em;color:#000;text-decoration:none;transition:all .15s ease;color:#fff;background-color:#40e0d0;text-shadow:0 1px 3px rgba(36,180,126,.4)}.index-module--ErrorList--oKLl3{margin:0 0 1em}.index-module--ErrorList--oKLl3 li{color:red}.index-module--optedIn--1c026 .index-module--inside--3uwon{color:#00ced1;display:flex;font-size:1.5em;padding:1em}.index-module--optedIn--1c026 .index-module--icon--1qvDc{flex-shrink:1;display:inline-block}.index-module--optedIn--1c026 .index-module--icon--1qvDc img{display:inline-block;height:1em;width:1em}.index-module--optedIn--1c026 .index-module--text--35wJN{padding-left:1em;flex-grow:1;font-weight:700;font-size:.9em}.index-module--optedIn--1c026 .index-module--text--35wJN em{display:block;margin-top:.5em;font-size:.8em;line-height:1;font-style:normal}.index-module--optedInNok--20AKD .index-module--inside--3uwon{color:#8b0000}.index-module--Submitting--22pTe{display:flex}.index-module--Submitting--22pTe .index-module--SubmittingText--18Hyc{flex-grow:1}.index-module--Submitting--22pTe .index-module--SubmittingText--18Hyc p{margin:0;padding:.4em 0 0;text-align:center;font-size:1em;color:#40e0d0;font-weight:700}.index-module--Submitting--22pTe .index-module--SubmittingGrid--1KSJ9{flex-shrink:1}.index-module--SubmittingGrid--1KSJ9{display:inline-block;position:relative;width:64px;height:64px}.index-module--SubmittingGrid--1KSJ9 div{position:absolute;width:13px;height:13px;border-radius:50%;background:#40e0d0;animation:index-module--LoadingGrid--2ksRm 1.5s linear infinite}.index-module--SubmittingGrid--1KSJ9 div:first-child{top:6px;left:6px;animation-delay:0s}.index-module--SubmittingGrid--1KSJ9 div:nth-child(2){top:6px;left:26px;animation-delay:-.4s}.index-module--SubmittingGrid--1KSJ9 div:nth-child(3){top:6px;left:45px;animation-delay:-.8s}.index-module--SubmittingGrid--1KSJ9 div:nth-child(4){top:26px;left:6px;animation-delay:-.4s}.index-module--SubmittingGrid--1KSJ9 div:nth-child(5){top:26px;left:26px;animation-delay:-.8s}.index-module--SubmittingGrid--1KSJ9 div:nth-child(6){top:26px;left:45px;animation-delay:-1.2s}.index-module--SubmittingGrid--1KSJ9 div:nth-child(7){top:45px;left:6px;animation-delay:-.8s}.index-module--SubmittingGrid--1KSJ9 div:nth-child(8){top:45px;left:26px;animation-delay:-1.2s}.index-module--SubmittingGrid--1KSJ9 div:nth-child(9){top:45px;left:45px;animation-delay:-1.6s}@keyframes index-module--LoadingGrid--2ksRm{0%,to{opacity:1}50%{opacity:.3}}.index-module--Logo--48XJj .index-module--ToolName--1OHdX{display:inline-block;margin-left:1rem}.index-module--Name--2h7KK{margin:0;font-size:1.5rem}p.index-module--Excerpt--250I9{padding:0 5%;font-size:1rem!important;font-weight:300}.index-module--Container--lfHSi{padding:1rem}.index-module--Container--lfHSi li,.index-module--Container--lfHSi ul{list-style-type:none;padding:0;margin:0}.index-module--AddressBar--2timP{margin:1rem 0;padding:.5rem;font-size:.8rem;width:auto;display:inline-block;background-color:#fff;border-radius:10px;letter-spacing:1px}.index-module--AddressBar--2timP>.index-module--HostName--aSwf5,.index-module--AddressBar--2timP>.index-module--PathName--3Dn-s,.index-module--AddressBar--2timP>.index-module--Protocol--2v1sw{display:inline-block}.index-module--AddressBar--2timP>.index-module--HostName--aSwf5,.index-module--AddressBar--2timP>.index-module--Protocol--2v1sw{color:#006400}.index-module--AddressBar--2timP>.index-module--PathName--3Dn-s{color:#666}.index-module--ToC--NOuEy{text-align:center}.index-module--ToC--NOuEy a{display:block;padding:0 0 .5rem;text-transform:uppercase;text-decoration:none;color:#666}.index-module--ToC--NOuEy a:hover{color:#000}@media screen and (min-width:1000px){.index-module--Container--lfHSi{display:flex;padding:2rem 5%}.index-module--ToC--NOuEy{flex:1 1;align-self:flex-start;position:-webkit-sticky;position:sticky;top:0;text-align:right;margin-right:2rem;margin-bottom:2rem}.index-module--SectionArea--360Dr{flex:4 1}.index-module--SectionArea--360Dr .index-module--Section--g6OzV:last-of-type{margin-bottom:2rem}}@media screen and (min-width:1400px){.index-module--Container--lfHSi{padding:2rem 15%}}.index-module--Section--g6OzV{font-size:.9rem;padding-top:2rem}.index-module--Section--g6OzV .index-module--Empty--36Qlb{background-color:rgba(139,0,0,.1);color:#8b0000}.index-module--Section--g6OzV .index-module--Empty--36Qlb p{font-weight:700;padding:1em}@media screen and (min-width:800px){.index-module--Section--g6OzV .index-module--Empty--36Qlb p{padding:1em 1em 1em 5rem}}@media screen and (min-width:800px){.index-module--Section--g6OzV{font-size:1rem}}.index-module--Section--g6OzV .index-module--Padded--TRfUZ{padding:1rem}@media screen and (min-width:800px){.index-module--Section--g6OzV .index-module--Padded--TRfUZ{padding:2rem}}.index-module--Section--g6OzV .index-module--Inner--UoRCq{box-shadow:0 0 2.5rem hsla(0,0%,78.4%,.4);border-radius:10px}.index-module--Section--g6OzV ul{margin:0;padding:0;background-color:rgba(175,238,238,.329)}.index-module--Section--g6OzV li{padding:1em;border-bottom:1px solid #fff}@media screen and (min-width:800px){.index-module--Section--g6OzV li{padding:1rem 0 1rem 5rem}}.index-module--Section--g6OzV li .index-module--Icon--j19_I{height:.7em;vertical-align:middle}.index-module--Section--g6OzV p{font-size:1em;font-weight:300;margin:0 0 1em}.index-module--Section--g6OzV p:last-child{margin-bottom:0}.index-module--CallToAction--2wvfY{padding:4rem 0 2rem}.index-module--CallToAction--2wvfY p{padding:0;margin:0;font-weight:900}.index-module--CallToAction--2wvfY .index-module--Inner--UoRCq{box-shadow:0 0 2.5rem hsla(0,0%,78.4%,.5);border-radius:10px;padding:1rem;display:flex;align-items:center}.index-module--One--3iXeb .index-module--Inner--UoRCq{background-color:#00ced1;color:#fff;box-shadow:0 0 2.5rem rgba(0,206,209,.5)}.index-module--One--3iXeb .index-module--Inner--UoRCq .index-module--A--K9VGn{flex-grow:1;font-size:1.2em;padding-left:.5rem}.index-module--One--3iXeb .index-module--Inner--UoRCq .index-module--B--1rXOO{flex-shrink:1}.index-module--Two--NioYl .index-module--Inner--UoRCq{background-color:#673ab7;color:#fff;box-shadow:0 0 2.5rem rgba(103,58,183,.5)}.index-module--Two--NioYl .index-module--Inner--UoRCq .index-module--A--K9VGn{flex-grow:1;font-size:1.2em;padding-left:.5rem}.index-module--Two--NioYl .index-module--Inner--UoRCq .index-module--B--1rXOO{flex-shrink:1}.index-module--Point--3fnfO{position:relative;line-height:1.5}.index-module--Point--3fnfO .index-module--Title--1HSab{margin:0 0 1em;padding:0;font-weight:600;color:#2a305e;font-size:2em;line-height:1}@media screen and (min-width:800px){.index-module--Point--3fnfO{padding-left:3em}.index-module--Point--3fnfO:after,.index-module--Point--3fnfO:before{content:" ";position:absolute;border-radius:50%;left:0;top:0}.index-module--Point--3fnfO:before{width:2em;height:2em;background-color:#fff;box-shadow:0 .3em .6em 0 rgba(68,74,102,.1)}.index-module--Point--3fnfO:after{width:1em;height:1em;margin-left:.5em;margin-top:.5em}.index-module--Point--3fnfO.index-module--Purple--1T084:after{background-color:#795fdf}.index-module--Point--3fnfO.index-module--Gold--2BQoR:after{background-color:gold}.index-module--Point--3fnfO.index-module--Cyan--hXnrR:after{background-color:#0ff}.index-module--Point--3fnfO.index-module--Green--3vzjk:after{background-color:#adff2f}.index-module--Point--3fnfO.index-module--Gray--3TQb5:after{background-color:#dcdcdc}}.index-module--Loader--1ONIT{padding:5% 0;text-align:center}.index-module--Loader--1ONIT>*{margin:0;padding:0;font-weight:100}.index-module--Loader--1ONIT .index-module--Primary--7e9Vp{font-size:2rem}.index-module--Loader--1ONIT .index-module--Secondary--2p6O3{font-size:1rem}.index-module--Loader--1ONIT svg{width:300px;height:300px;margin:2rem auto;display:block}.index-module--ErrorPage--QYGDs{color:#8b0000;text-align:center;padding:5%}.Button-module--Button--3uC_E{display:inline-block;position:relative;text-align:center;font-size:1em}.Button-module--Button--3uC_E a{position:relative;cursor:pointer;outline:none;white-space:nowrap;display:inline-block;box-shadow:0 4px 6px rgba(50,50,93,.11),0 1px 3px rgba(0,0,0,.08);background:#fff;border-radius:4px;font-weight:600;text-transform:uppercase;letter-spacing:.05rem;color:#000;text-decoration:none;transition:all .15s ease;color:#fff;background:#40e0d0;text-shadow:0 1px 3px rgba(33,150,243,.4);border:1px solid #40e0d0}.Button-module--Button--3uC_E a:hover{background:#42e6d5;top:-1px}.Button-module--Button--3uC_E.Button-module--InvertedGreen--2Ek2f a{color:#40e0d0;border:1px solid #40e0d0;text-shadow:none;background:#fff}.Button-module--Button--3uC_E.Button-module--InvertedGreen--2Ek2f a:hover{background:#fcfcfc}.Button-module--Button--3uC_E.Button-module--InvertedPurple--2vyeI a{color:#673ab7;border:1px solid #673ab7;text-shadow:none;background:#fff}.Button-module--Button--3uC_E.Button-module--InvertedPurple--2vyeI a:hover{background:#fcfcfc}.Button-module--Button--3uC_E.Button-module--FluidButton--1xDTC,.Button-module--Button--3uC_E.Button-module--FluidButton--1xDTC a{display:block}.Button-module--Button--3uC_E.Button-module--Normal--OqCcE a{font-size:.9rem;padding:.75rem 1rem}.Button-module--Button--3uC_E.Button-module--Large--3-Fg7 a{font-size:1rem;padding:1rem 1.3rem}.Button-module--Button--3uC_E.Button-module--Small--3h1yA a{font-size:.8rem;padding:.6rem 1rem}.post-module--article--1SjHg h1,.post-module--article--1SjHg h2,.post-module--article--1SjHg h3,.post-module--article--1SjHg h4,.post-module--article--1SjHg h5{font-family:Lato}.post-module--article--1SjHg h2{color:#00ced1}.post-module--articleHeaderText--2QLQM{text-align:center;padding:5% 5% 0;margin:auto}.post-module--articleHeaderText--2QLQM h1{font-size:1.4rem;line-height:1.25;color:#00ced1}.post-module--excerpt--11hub{font-family:Source Sans Pro;font-weight:100;padding-top:1rem;padding-bottom:1rem;font-size:1.1rem;line-height:1.25;margin:auto}.post-module--metaList--WV8cT{list-style-type:none;margin:0;padding:0;font-style:italic}.post-module--metaList--WV8cT li{list-style-type:none;margin:0;padding:1em 0}.post-module--articleHero--yMBPa{margin:5% 0}.post-module--articleMain--2iunI{padding-bottom:3rem}.post-module--articleContents--iGVdC{font-family:Source Sans Pro}.post-module--articleContents--iGVdC>div>*{max-width:var(--max-width);padding-left:var(--side-padding);padding-right:var(--side-padding);margin:2rem auto}.post-module--articleContents--iGVdC>div>ul:first-of-type{border:1px solid #fcfcfc;padding:1rem 2rem;border-radius:10px;box-shadow:0 0 2rem rgba(0,0,0,.05);background:hsla(0,0%,78.4%,.05);margin:2rem 1rem 4rem}.post-module--articleContents--iGVdC>div>ul:first-of-type a{font-size:.8rem;text-decoration:none;color:#666}.post-module--articleContents--iGVdC>div>ul:first-of-type a:hover{color:#000}@media screen and (min-width:771px){.post-module--articleContents--iGVdC>div>ul:first-of-type a{font-size:1rem}}.post-module--articleContents--iGVdC>div>ul:first-of-type p{margin:0;padding:0}.post-module--articleContents--iGVdC>div>ul:first-of-type li{margin:0;padding:0 0 .25rem}.post-module--articleContents--iGVdC>div>ol li,.post-module--articleContents--iGVdC>div>ul li{margin-left:2rem}.post-module--articleContents--iGVdC>div>p code{background:#f0f0f0;padding:3px 8px;border-radius:5px;border:1px solid #c6c6c6;margin:0;color:#333}.post-module--articleContents--iGVdC>div>h1,.post-module--articleContents--iGVdC>div>h2,.post-module--articleContents--iGVdC>div>h3,.post-module--articleContents--iGVdC>div>h4,.post-module--articleContents--iGVdC>div>h5{line-height:1.25;margin-top:2.5em;margin-bottom:1em}.post-module--articleContents--iGVdC>div>h1 a,.post-module--articleContents--iGVdC>div>h2 a,.post-module--articleContents--iGVdC>div>h3 a,.post-module--articleContents--iGVdC>div>h4 a,.post-module--articleContents--iGVdC>div>h5 a{visibility:hidden}.post-module--articleContents--iGVdC>div>h1{font-size:2.2rem}.post-module--articleContents--iGVdC>div>h2{font-size:2rem;text-shadow:0 0 1rem rgba(187,235,234,.64)}.post-module--articleContents--iGVdC>div>h4{font-weight:400}.post-module--articleContents--iGVdC>div figcaption{font-size:.7rem;line-height:1.25;color:#cacaca;margin:.8em 2rem;text-align:center}.post-module--articleContents--iGVdC>div a{color:#009597;font-weight:400;text-decoration:underline}.post-module--articleContents--iGVdC>div a:hover{color:#00696b}@media screen and (min-width:771px){.post-module--articleHeaderText--2QLQM{padding:5% 5% 0}.post-module--articleHeaderText--2QLQM h1{font-size:2.5rem}.post-module--excerpt--11hub{font-size:1.5rem;line-height:1.5}.post-module--articleHero--yMBPa{max-width:var(--max-hero-width);margin:5% auto}.post-module--articleHero--yMBPa img{display:block;width:100%;max-width:100%;border-radius:10px!important;border:1px solid #eee}.post-module--articleContents--iGVdC{font-size:1.2rem;line-height:1.5}.post-module--articleContents--iGVdC>div>ul:first-of-type{padding:1rem 3rem;margin-left:auto;margin-right:auto}}@media screen and (min-width:1200px){.post-module--articleHeaderText--2QLQM{max-width:60%}} + ]]> + </style> + <meta name="generator" content="Gatsby 2.22.9" /> + <link rel="shortcut icon" href="/icons-5f7390f8df97283d5b517dead9b4a9d5/favicon.ico" /> + <link rel="icon" type="image/png" sizes="16x16" href="/icons-5f7390f8df97283d5b517dead9b4a9d5/favicon-16x16.png" /> + <link rel="icon" type="image/png" sizes="32x32" href="/icons-5f7390f8df97283d5b517dead9b4a9d5/favicon-32x32.png" /> + <link rel="manifest" href="/icons-5f7390f8df97283d5b517dead9b4a9d5/manifest.json" /> + <meta name="mobile-web-app-capable" content="yes" /> + <meta name="theme-color" content="#fff" /> + <meta name="application-name" content="topicseed-website" /> + <link rel="apple-touch-icon" sizes="57x57" href="/icons-5f7390f8df97283d5b517dead9b4a9d5/apple-touch-icon-57x57.png" /> + <link rel="apple-touch-icon" sizes="60x60" href="/icons-5f7390f8df97283d5b517dead9b4a9d5/apple-touch-icon-60x60.png" /> + <link rel="apple-touch-icon" sizes="72x72" href="/icons-5f7390f8df97283d5b517dead9b4a9d5/apple-touch-icon-72x72.png" /> + <link rel="apple-touch-icon" sizes="76x76" href="/icons-5f7390f8df97283d5b517dead9b4a9d5/apple-touch-icon-76x76.png" /> + <link rel="apple-touch-icon" sizes="114x114" href="/icons-5f7390f8df97283d5b517dead9b4a9d5/apple-touch-icon-114x114.png" /> + <link rel="apple-touch-icon" sizes="120x120" href="/icons-5f7390f8df97283d5b517dead9b4a9d5/apple-touch-icon-120x120.png" /> + <link rel="apple-touch-icon" sizes="144x144" href="/icons-5f7390f8df97283d5b517dead9b4a9d5/apple-touch-icon-144x144.png" /> + <link rel="apple-touch-icon" sizes="152x152" href="/icons-5f7390f8df97283d5b517dead9b4a9d5/apple-touch-icon-152x152.png" /> + <link rel="apple-touch-icon" sizes="167x167" href="/icons-5f7390f8df97283d5b517dead9b4a9d5/apple-touch-icon-167x167.png" /> + <link rel="apple-touch-icon" sizes="180x180" href="/icons-5f7390f8df97283d5b517dead9b4a9d5/apple-touch-icon-180x180.png" /> + <link rel="apple-touch-icon" sizes="1024x1024" href="/icons-5f7390f8df97283d5b517dead9b4a9d5/apple-touch-icon-1024x1024.png" /> + <meta name="apple-mobile-web-app-capable" content="yes" /> + <meta name="apple-mobile-web-app-status-bar-style" content="black-translucent" /> + <meta name="apple-mobile-web-app-title" content="topicseed-website" /> + <link rel="apple-touch-startup-image" media="(device-width: 320px) and (device-height: 480px) and (-webkit-device-pixel-ratio: 1)" href="/icons-5f7390f8df97283d5b517dead9b4a9d5/apple-touch-startup-image-320x460.png" /> + <link rel="apple-touch-startup-image" media="(device-width: 320px) and (device-height: 480px) and (-webkit-device-pixel-ratio: 2)" href="/icons-5f7390f8df97283d5b517dead9b4a9d5/apple-touch-startup-image-640x920.png" /> + <link rel="apple-touch-startup-image" media="(device-width: 320px) and (device-height: 568px) and (-webkit-device-pixel-ratio: 2)" href="/icons-5f7390f8df97283d5b517dead9b4a9d5/apple-touch-startup-image-640x1096.png" /> + <link rel="apple-touch-startup-image" media="(device-width: 375px) and (device-height: 667px) and (-webkit-device-pixel-ratio: 2)" href="/icons-5f7390f8df97283d5b517dead9b4a9d5/apple-touch-startup-image-750x1294.png" /> + <link rel="apple-touch-startup-image" media="(device-width: 414px) and (device-height: 736px) and (orientation: landscape) and (-webkit-device-pixel-ratio: 3)" href="/icons-5f7390f8df97283d5b517dead9b4a9d5/apple-touch-startup-image-1182x2208.png" /> + <link rel="apple-touch-startup-image" media="(device-width: 414px) and (device-height: 736px) and (orientation: portrait) and (-webkit-device-pixel-ratio: 3)" href="/icons-5f7390f8df97283d5b517dead9b4a9d5/apple-touch-startup-image-1242x2148.png" /> + <link rel="apple-touch-startup-image" media="(device-width: 768px) and (device-height: 1024px) and (orientation: landscape) and (-webkit-device-pixel-ratio: 1)" href="/icons-5f7390f8df97283d5b517dead9b4a9d5/apple-touch-startup-image-748x1024.png" /> + <link rel="apple-touch-startup-image" media="(device-width: 768px) and (device-height: 1024px) and (orientation: portrait) and (-webkit-device-pixel-ratio: 1)" href="/icons-5f7390f8df97283d5b517dead9b4a9d5/apple-touch-startup-image-768x1004.png" /> + <link rel="apple-touch-startup-image" media="(device-width: 768px) and (device-height: 1024px) and (orientation: landscape) and (-webkit-device-pixel-ratio: 2)" href="/icons-5f7390f8df97283d5b517dead9b4a9d5/apple-touch-startup-image-1496x2048.png" /> + <link rel="apple-touch-startup-image" media="(device-width: 768px) and (device-height: 1024px) and (orientation: portrait) and (-webkit-device-pixel-ratio: 2)" href="/icons-5f7390f8df97283d5b517dead9b4a9d5/apple-touch-startup-image-1536x2008.png" /> + <title data-react-helmet="true"> + Content Depth — Write Comprehensively About Your Core Topics | topicseed + </title> + <meta data-react-helmet="true" name="description" content="Content writers and marketers find it hard to write a lot of content about a very specific topic. They lose a lot of points on their content depth because they would rather focus on pushing thin content about plenty of topics." /> + <meta data-react-helmet="true" name="tags" content="Content Depth,Topical Authority" /> + <meta data-react-helmet="true" property="og:title" content="Content Depth — Write Comprehensively About Your Core Topics" /> + <meta data-react-helmet="true" property="og:type" content="article" /> + <meta data-react-helmet="true" property="og:url" content="https://topicseed.com/blog/content-depth-for-seo" /> + <meta data-react-helmet="true" property="og:image" content="https://topicseed.com/static/9c97da26f6eeee98fc2e628ca3416226/57090/content-depth-seo.png" /> + <meta data-react-helmet="true" property="og:description" content="Content writers and marketers find it hard to write a lot of content about a very specific topic. They lose a lot of points on their content depth because they would rather focus on pushing thin content about plenty of topics." /> + <meta data-react-helmet="true" property="og:site_name" content="topicseed" /> + <meta data-react-helmet="true" property="og:locale" content="en_US" /> + <meta data-react-helmet="true" property="og:updated_time" content="2018-06-12T23:00:00.000Z" /> + <meta data-react-helmet="true" property="article:modified_time" content="2018-06-12T23:00:00.000Z" /> + <meta data-react-helmet="true" property="article:published_time" content="2018-06-12T23:00:00.000Z" /> + <link rel="canonical" href="https://topicseed.com/blog/content-depth-for-seo" data-baseprotocol="https:" data-basehost="topicseed.com" /> + <style type="text/css"> + /*<![CDATA[*/ + .anchor { + float: left; + padding-right: 4px; + margin-left: -20px; + } + h1 .anchor svg, + h2 .anchor svg, + h3 .anchor svg, + h4 .anchor svg, + h5 .anchor svg, + h6 .anchor svg { + visibility: hidden; + } + h1:hover .anchor svg, + h2:hover .anchor svg, + h3:hover .anchor svg, + h4:hover .anchor svg, + h5:hover .anchor svg, + h6:hover .anchor svg, + h1 .anchor:focus svg, + h2 .anchor:focus svg, + h3 .anchor:focus svg, + h4 .anchor:focus svg, + h5 .anchor:focus svg, + h6 .anchor:focus svg { + visibility: visible; + } + /*]]>*/ + </style> + <script> + <![CDATA[ + document.addEventListener("DOMContentLoaded", function(event) { + var hash = window.decodeURI(location.hash.replace('#', '')) + if (hash !== '') { + var element = document.getElementById(hash) + if (element) { + var offset = element.offsetTop + // Wait for the browser to finish rendering before scrolling. + setTimeout((function() { + window.scrollTo(0, offset - 0) + }), 0) + } + } + }) + ]]> + </script> + <link rel="sitemap" type="application/xml" href="/sitemap.xml" /> + <link href="https://fonts.googleapis.com/css?family=Source+Sans+Pro:100,100i,200,200i,300,300i,400,400i,600,600i,700,700i,900,900i|Lato:300,400,400i,700" rel="stylesheet" /> + <link as="script" rel="preload" href="/webpack-runtime-0492a572fdd49b8e1c9d.js" /> + <link as="script" rel="preload" href="/framework-c078ab5cc0599490105f.js" /> + <link as="script" rel="preload" href="/app-e61cbb9e5f9aaa06b962.js" /> + <link as="script" rel="preload" href="/styles-9b6f388623a2ec93d35f.js" /> + <link as="script" rel="preload" href="/885eb025e981378eedf422d5b3545d4eebce3f62-c969f725392e18e66f50.js" /> + <link as="script" rel="preload" href="/a46159e2a7241999ec544c326fa84cca5dda2ca8-419ee48c76badafe338c.js" /> + <link as="script" rel="preload" href="/4d8bd03769955e7ffa305e9df9fd8c45ea8ce544-20cfe20204c6ef10aa5c.js" /> + <link as="script" rel="preload" href="/494aa2667243fb352b6c4f73ed1a8fa80c641942-1149805138126f0ca2a7.js" /> + <link as="script" rel="preload" href="/component---src-templates-post-js-a7f7cfc98fbcd66feb7e.js" /> + <link as="fetch" rel="preload" href="/page-data/blog/content-depth-for-seo/page-data.json" crossorigin="anonymous" /> + <link as="fetch" rel="preload" href="/page-data/app-data.json" crossorigin="anonymous" /> + </head> + <body> + <div id="___gatsby"> + <div style="outline:none" tabindex="-1" id="gatsby-focus-wrapper"> + <div> + <div> + <nav class="Navigation-module--navigation--2Ttnx"> + <div class="Navigation-module--branding--QzOe4"> + <a class="Navigation-module--navLink--3Ceps" href="/"><span>topicseed<!-- --> <img src="https://topicseed.com/assets/logos/topicseed-visual-black-xs.png" alt="topicseed logo" /></span></a> + </div> + <div class="Navigation-module--topMenu--2WS0K"> + <label class="Navigation-module--navLink--3Ceps Navigation-module--toggler--1HP5t" for="menu-toggle">☰</label><input type="checkbox" id="menu-toggle" class="Navigation-module--menuToggle--2a0Fx" aria-label="Toggle Menu" /> + <ul class="Navigation-module--menu--3ilH3"> + <li> + <a class="Navigation-module--navLink--3Ceps" href="/features">Features</a> + </li> + <li> + <a class="Navigation-module--navLink--3Ceps" href="/pricing">Pricing</a> + </li> + <li> + <a class="Navigation-module--navLink--3Ceps" href="/resources">Free SEO Tools</a> + </li> + <li> + <a class="Navigation-module--navLink--3Ceps Navigation-module--active--tC-lF" href="/blog">Blog</a> + </li> + </ul> + </div> + </nav> + </div> + <main class="siteMain"> + <article class="post-module--article--1SjHg"> + <header> + <div class="post-module--articleHeaderText--2QLQM"> + <h1> + Content Depth — Write Comprehensively About Your Core Topics + </h1> + <p class="post-module--excerpt--11hub"> + Content writers and marketers find it hard to write a lot of content about a very specific topic. They lose a lot of points on their content depth because they would rather focus on pushing thin content about plenty of topics. + </p> + <ul class="post-module--metaList--WV8cT"> + <li> + <small>On <!-- -->6/13/2018<!-- --> by <!-- -->@lazharichir</small> + </li> + </ul> + </div> + <div class="post-module--articleHero--yMBPa"> + <div class="gatsby-image-wrapper" style="position:relative;overflow:hidden"> + <div aria-hidden="true" style="width:100%;padding-bottom:50%"></div><img aria-hidden="true" src="data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAKCAIAAAA7N+mxAAAACXBIWXMAAAsSAAALEgHS3X78AAABgElEQVQoz2WSS2vCUBCF88dFfJt3dBFJ3fmKG7vTWruwIEithbQiXVmkEhvjTWq0xih10dNcShUPNzA35JtzZgjzdaHNZnM4HEajUTweVxSFZVlRFFVVRZ1IJOKhstksx3HMJbzF2W4JIbqup1IpWZIeB4PpZFIul2OxmCgKsiyjXS6X+4e3odbr9Wq1wpUslyvHue92FVkGhq95noc5JaFMJsPsQgVBgMchxDJNe7FAC9uyng3jdTzWazVk7vV6T8Mh4HQ6nUwmaXgmONHa83CoP8LfttuFgorZqpXqTauVz+cjkQhgls0Kwm/4E3i/3/m+6zjwRITv4/FtMtE0DQuLRqM8x103GjjoBX+anDlh93Ajtv3pupgZV8uyapXKXadTKpUe+v0Xw6jX6xSjk5/Fhnzf9zwP/mjxYZrmbPY+nV5pmiAKXCgwkiTR+gzeBwEYaz5f2jbMsXaXkFaziTlhxYdiQ6E4n/nPGQujNX4VbKFYLGK98AEjCDx1Bok3P/kSytq5UkC3AAAAAElFTkSuQmCC" alt="Content Depth — Write Comprehensively About Your Core Topics" style="position:absolute;top:0;left:0;width:100%;height:100%;object-fit:cover;object-position:center;opacity:1;transition-delay:500ms" /><noscript><picture><source srcset="/static/9c97da26f6eeee98fc2e628ca3416226/d9446/content-depth-seo.png 158w, /static/9c97da26f6eeee98fc2e628ca3416226/2f6e7/content-depth-seo.png 315w, /static/9c97da26f6eeee98fc2e628ca3416226/57090/content-depth-seo.png 630w, /static/9c97da26f6eeee98fc2e628ca3416226/4ace0/content-depth-seo.png 945w, /static/9c97da26f6eeee98fc2e628ca3416226/fb0fe/content-depth-seo.png 1260w, /static/9c97da26f6eeee98fc2e628ca3416226/dc9f7/content-depth-seo.png 2000w" sizes="(max-width: 630px) 100vw, 630px" /><img loading="lazy" sizes="(max-width: 630px) 100vw, 630px" srcset="/static/9c97da26f6eeee98fc2e628ca3416226/d9446/content-depth-seo.png 158w, /static/9c97da26f6eeee98fc2e628ca3416226/2f6e7/content-depth-seo.png 315w, /static/9c97da26f6eeee98fc2e628ca3416226/57090/content-depth-seo.png 630w, /static/9c97da26f6eeee98fc2e628ca3416226/4ace0/content-depth-seo.png 945w, /static/9c97da26f6eeee98fc2e628ca3416226/fb0fe/content-depth-seo.png 1260w, /static/9c97da26f6eeee98fc2e628ca3416226/dc9f7/content-depth-seo.png 2000w" src="/static/9c97da26f6eeee98fc2e628ca3416226/57090/content-depth-seo.png" alt="Content Depth — Write Comprehensively About Your Core Topics" style="position:absolute;top:0;left:0;opacity:1;width:100%;height:100%;object-fit:cover;object-position:center" /></picture></noscript> + </div> + </div> + </header> + <main class="post-module--articleMain--2iunI"> + <div class="post-module--articleContents--iGVdC"> + <div> + <ul> + <li> + <a href="#assess-how-deep-is-your-content">Assess How Deep Is Your Content</a> + </li> + <li> + <a href="#rewrite-with-content-depth-in-mind">Rewrite With Content Depth In Mind</a> + </li> + <li> + <a href="#yes-content-depth-and-breadth-overlap">Yes, Content Depth and Breadth Overlap</a> + </li> + <li> + <a href="#depth-of-content--quality--frequency">Depth of Content = Quality + Frequency</a> + </li> + </ul> + <p> + <strong>Content depth</strong> is an arbitrary score or rating of how comprehensive the coverage of a specific topic is within a piece of content. <strong>Content breadth</strong> is an arbitrary grading of how many related subjects are you covering within your content. + </p> + <p> + And this distinction is important to make and establish from the beginning. Effective <a href="https://topicseed.com/blog/what-is-topical-authority" target="_blank" rel="nofollow noopener noreferrer"><strong>topical authority</strong></a> can only be gained when you use both content depth and content breadth in your overall content strategy for rapid search engine optimization gains. However, because most content writers prefer to write a little bit about many things rather than write a lot about one thing, you end up with a too little substance spread very thin. + </p> + <p> + Content depth should be the urgent priority for your content marketing strategy, and clearly defined in your <a href="/blog/content-briefs">content briefs</a>. Start by dominating your own core topics, before venturing across the pond and write about linked subject matters. Otherwise, you are the opposite of an authority as the definition states that an authority is <em>“a person with extensive or specialized knowledge about a subject; an expert”.</em> Lastly, do not mistake article depth vs. article length: a blog post’s extreme wordcount has nothing to do with its content depth. + </p> + <h2 id="assess-how-deep-is-your-content"> + <a href="#assess-how-deep-is-your-content" aria-label="assess how deep is your content permalink" class="anchor"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"> + <path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Assess How Deep Is Your Content + </h2> + <p> + The first task on your list, right now, is to shortlist your core topics. What are you trying to be an expert on? Then, go through each one of your pieces of content and understand how well each blog post is covering its focus topic(s). Not how many times specific keywords appear, or how well the article is outlined and structured. + </p> + <p> + Put yourself in the shoes of an ignorant reader who seeks information. Read your article. <strong>And ask yourself how in-depth was the content you have written?</strong> I know the excuse you will come up with: this was written for beginners, therefore, it shouldn’t be too in-depth. And you are correct. Not every blog post is about absolute content depth otherwise we would only write one 10,000-word-long article, once and for all. But then, how well your beginner-level content pointing to your expert-level content? + </p> + <p> + In other words, each article should reach an incredible level of content depth for its expertise level. And then, provide further reading <em>(i.e. links)</em> to gain more knowledge, and depth. A lot of content editors write a beginner’s blog post and wait to see it perform well in order to write a more advanced sequel. Wrong. Give all the value so search engines can grade you highly on their authority scale for your core topics. Yes, it is a risk and you may write a dozen of articles on a specific topic that will never really rank at the top of SERPs, but <strong>reaching content depth is the first step towards SEO gains</strong>. + </p> + <p> + Remember that <strong>skyscraper content</strong> and <strong>10x content</strong> are not necessarily the answer. These content writing strategies state that in order to beat another piece of content, you need to write 10x more. Either in quantity with a 10x word count or in quality by putting times more information within your own piece of content. Such articles often become unreadable and discourage visitors from absorbing all the knowledge. The best alternative is the create <a href="https://topicseed.com/blog/how-broad-should-topics-be-for-pillar-pages" target="_blank" rel="nofollow noopener noreferrer">pillar pages</a> centered around core topics, and several articles dealing with each specific section in depth. This is <strong>deep content powered by a <a href="https://topicseed.com/blog/internal-linking-strategies-for-topic-clustering" target="_blank" rel="nofollow noopener noreferrer">smart internal linking strategy</a></strong> and search engines love that in this day and age where attention spans are short! <em>With that being said, avoid writing 600-word articles!</em> + </p> + <h2 id="rewrite-with-content-depth-in-mind"> + <a href="#rewrite-with-content-depth-in-mind" aria-label="rewrite with content depth in mind permalink" class="anchor"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"> + <path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Rewrite With Content Depth In Mind + </h2> + <p> + Once you know which articles are lacking depth of knowledge and information, it is time to rethink each one. For each article, make a list of what essential pieces of information or data are missing. Then decide where to fit them, and decide whether the article would benefit from a full rewrite or not. As a rule of thumb, if you need to change a third of your article, you may need to rewrite it entirely. Of course, this does not mean erasing all work done prior, but it means starting afresh! Trying to <strong>fit deep content into an existing blog</strong> post gives you constraints so doing it from scratch can actually be easier to fight thin content. + </p> + <div> + <aside class="index-module--TeaserOptin--FVea1"> + <header> + Optimize one of your blog posts immediately – for FREE! + </header> + <div class="index-module--container--34rxR"> + <div class="index-module--inner--3kOD7"> + <form class="index-module--website--2It4m"> + <input type="url" placeholder="Blog Post URL" value="" required="" aria-label="Blog Post URL" /> + <div class="index-module--submitter--2CsUF"> + <button type="submit" class="index-module--btn--324SS">Next</button> + </div> + <div class="index-module--Steps--W963K"> + <div class="index-module--Done--3wzyP"></div> + <div class="index-module--NotDone--1xW0M"></div> + <div class="index-module--NotDone--1xW0M"></div> + </div> + </form> + </div> + </div> + </aside> + </div> + <p> + As explained above, make sure you do not force yourself to write a much longer article to reach a <a href="https://moz.com/blog/blog-post-length-frequency" target="_blank" rel="nofollow noopener noreferrer">magic word count</a>. And if you do, it has to be natural. In many cases, articles written months or years ago may need some upkeeping: trimming the fat and removing parts that are not bringing much value. Replace these with your newer and deeper content. + </p> + <p> + All content writers know that when you open Google Docs, WordPress, or your text editor of choice, you will inevitably count your focus keywords’ frequency. Although I understand (yet question) the value of keywords in modern SEO, do not become obsessed with reaching a magic number for your keywords. No reader coming from Google is out there counting how often your keywords are appearing. And search engine algorithms will penalize you for writing for robots, rather than humans. + </p> + <p> + With the massive rise of voice searches, <a href="/blog/featured-snippets-using-questions">users tend to use full questions for their search queries</a>. What used to be <code>top bottled water brands</code> is now <code>OK google, what is the best bottled-water brand in Texas</code>? The point being, <a href="https://topicseed.com/blog/keyword-search-volume-overrated" target="_blank" rel="nofollow noopener noreferrer"><strong>keywords are losing traction</strong></a> to leave space for a more natural language understanding of a blog post’s textual content, and meaning. + </p> + <h2 id="yes-content-depth-and-breadth-overlap"> + <a href="#yes-content-depth-and-breadth-overlap" aria-label="yes content depth and breadth overlap permalink" class="anchor"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"> + <path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Yes, Content Depth and Breadth Overlap + </h2> + <p> + <em>“A topic can be defined as the company it keeps.”</em> A very accurate saying loved by ontologists within the fields of computational linguistics, and information science. In simpler terms, a topic and all the terminology it is encompassing will inevitably overlap with related topics. Which, in turn, will form <a href="https://topicseed.com/blog/topic-clusters-relationships" target="_blank" rel="nofollow noopener noreferrer"><strong>topic clusters</strong></a>. + </p> + <p> + For example, it is obvious that despite being two different topics, <code>digital advertising</code> and <code>content marketing</code> share some common phrases and terms. Inevitably, a website picking one as its core topic will use words in some blog posts that will identify the article as belonging to both topics, with a specific weight for each. + </p> + <p> + A keyword, phrase, or term, is not a prisoner to a single concept at all. This is how algorithms in natural language understanding can understand how two topics are related (e.g. read about <a href="https://en.wikipedia.org/wiki/Topic_model" target="_blank" rel="nofollow noopener noreferrer"><em>topic modeling</em></a>). Each topic has a specific <strong>vocabulary</strong>, a list of words and phrases commonly used in its context, and some of these terms are present in different vocabularies. + </p> + <p> + Therefore, content depth and content breadth are not to be opposed. Content marketers should use both strategies in order to reach ultimate <strong>topical authority</strong> over their choice of subject matters. + </p> + <h2 id="depth-of-content--quality--frequency"> + <a href="#depth-of-content--quality--frequency" aria-label="depth of content quality frequency permalink" class="anchor"><svg aria-hidden="true" focusable="false" height="16" version="1.1" viewbox="0 0 16 16" width="16"> + <path fill-rule="evenodd" d="M4 9h1v1H4c-1.5 0-3-1.69-3-3.5S2.55 3 4 3h4c1.45 0 3 1.69 3 3.5 0 1.41-.91 2.72-2 3.25V8.59c.58-.45 1-1.27 1-2.09C10 5.22 8.98 4 8 4H4c-.98 0-2 1.22-2 2.5S3 9 4 9zm9-3h-1v1h1c1 0 2 1.22 2 2.5S13.98 12 13 12H9c-.98 0-2-1.22-2-2.5 0-.83.42-1.64 1-2.09V6.25c-1.09.53-2 1.84-2 3.25C6 11.31 7.55 13 9 13h4c1.45 0 3-1.69 3-3.5S14.5 6 13 6z"></path></svg></a>Depth of Content = Quality + Frequency + </h2> + <p> + Up until recently, long-form blog posts generally were <strong>evergreen articles</strong> that generated a constant stream of organic traffic for a website. This was a lead magnet generation strategy which worked well: hire a writer, include the right keywords, reach over a 5,000-word word count, and hit publish. Then, wait. + </p> + <p> + Nowadays, in-depth content requires more effort over time in order to pay off. Writing a big article, as good as it is, will not get your anywhere near the level of <a href="https://topicseed.com/blog/topical-seo" target="_blank" rel="nofollow noopener noreferrer">topical breadth</a> required by Google to rank you first. Instead, your content marketing plan should be about having: + </p> + <ul> + <li>a <strong>comprehensive pillar page</strong> covering a unique topic, and + </li> + <li> + <strong>narrow-focused children articles</strong> to dig deeper. + </li> + </ul> + <p> + Search engines also look at how often you publish about a specific topic, and when was the last time it was written about. Nobody likes a <a href="https://www.copypress.com/blog/avoiding-blog-graveyard/" target="_blank" rel="nofollow noopener noreferrer">graveyard blog</a>, it just makes the reader lose trust; as if the writer was not good enough, therefore had no traffic, before entirely giving up. Deep content requires a sustained effort on your part to always new find ways to write about a specific subject. Sure, it will be easy at first. But what about five years later? Well, you will still need to hit publish, all about the very same topics you already covered years ago. + </p> + <p> + Tools and platforms such as topicseed are here to <a href="https://topicseed.com/blog/how-to-find-new-blog-post-ideas" target="_blank" rel="nofollow noopener noreferrer">help you find new article ideas</a> pertaining to your core topics within a few clicks and a few minutes. The number of web pages, Wikipedia articles, and pieces of content, our machine-learning algorithms can analyze in seconds would take you months to digest. Our <em>topicgraph</em> finds closely related concepts in order for your domain to <strong>reach topical authority through content depth and content breadth</strong>. + </p> + </div> + </div> + </main> + </article> + </main> + <aside class="SimpleOptinForm-module--container--3JpKE SimpleOptinForm-module--fluid--1iPrt"> + <div class="SimpleOptinForm-module--inner--1YsM0"> + <header class="SimpleOptinForm-module--header--36bBg"> + <h1> + RECEIVE TOPICAL SEO TIPS + </h1> + <h2> + Learn about topical SEO, keyword clustering, and cool SEO content tools. + </h2> + </header> + <form> + <fieldset> + <div class="SimpleOptinForm-module--fields--3BB2p"> + <div class="row"> + <div class="col-xs-12 col-sm-12 col-md-6 col-lg-6"> + <p class="SimpleOptinForm-module--field--jE8AR"> + <input type="text" placeholder="Full Name" value="" aria-label="Full Name" /> + </p> + </div> + <div class="col-xs-12 col-sm-12 col-md-6 col-lg-6"> + <p class="SimpleOptinForm-module--field--jE8AR"> + <input type="email" placeholder="Work Email" value="" required="" aria-label="Work Email" /> + </p> + </div> + </div> + <div class="row"> + <div class="col-xs-12 col-sm-12 col-md-6 col-lg-6"> + <p class="SimpleOptinForm-module--field--jE8AR"> + <input type="text" placeholder="Organization" value="" aria-label="Organization" /> + </p> + </div> + <div class="col-xs-12 col-sm-12 col-md-6 col-lg-6"> + <p class="SimpleOptinForm-module--field--jE8AR"> + <select> + <option value="Founder"> + Founder + </option> + <option value="Director"> + Director + </option> + <option value="Marketing Manager"> + Marketing Manager + </option> + <option value="Marketing Executive"> + Marketing Executive + </option> + <option value="Content Writer"> + Content Writer + </option> + <option value="Content Editor"> + Content Editor + </option> + </select> + </p> + </div> + </div> + </div> + <div class="SimpleOptinForm-module--btnWrapper--140rB"> + <div class="row"> + <div class="col-xs-12"> + <button type="submit" class="SimpleOptinForm-module--btn--2iMm_">Let's go!</button> + </div> + </div> + </div> + </fieldset> + </form> + </div> + </aside> + <footer class="SiteFooter-module--siteFooter--2ZMf7"> + <div class="row row-valign-center"> + <div class="col-xs-12 col-sm-12 col-md-12 col-lg-12"> + <div class="SiteFooter-module--LineOfLinks--17SUL"> + <a href="/blog">Blog</a><a href="/agency">Agency</a><a href="/changelog">Changelog</a><a href="/contact">Contact</a><a href="/privacy">Privacy Policy</a> + </div> + </div> + </div> + <div class="row SiteFooter-module--copyrightNotice--3uQ14"> + <p> + topicseed, 2018-2019 © All rights reserved. Operated by TOPICSEED LIMITED, a company registered in England & Wales (Company No. 11746352) + </p> + </div> + </footer> + </div> + </div> + <div id="gatsby-announcer" style="position:absolute;top:0;width:1px;height:1px;padding:0;overflow:hidden;clip:rect(0, 0, 0, 0);white-space:nowrap;border:0" aria-live="assertive" aria-atomic="true"></div> + </div> + <script id="gatsby-script-loader"> + /*<![CDATA[*/window.pagePath="/blog/content-depth-for-seo";/*]]>*/ + </script> + <script id="gatsby-chunk-mapping"> + /*<![CDATA[*/window.___chunkMapping={"app":["/app-e61cbb9e5f9aaa06b962.js"],"component---src-pages-404-js":["/component---src-pages-404-js-629775d27a2dc4ec3d2d.js"],"component---src-pages-agency-index-js":["/component---src-pages-agency-index-js-156249e08af981271f31.js"],"component---src-pages-blog-js":["/component---src-pages-blog-js-ed62d4c0a6ea689b9a52.js"],"component---src-pages-changelog-js":["/component---src-pages-changelog-js-b7fbb55334e481c3936b.js"],"component---src-pages-contact-js":["/component---src-pages-contact-js-4ceb754adec5e981fce7.js"],"component---src-pages-features-index-js":["/component---src-pages-features-index-js-27c81d134ae9eaf9e882.js"],"component---src-pages-index-js":["/component---src-pages-index-js-b673e5e0ada88bfb9d4f.js"],"component---src-pages-optimize-index-js":["/component---src-pages-optimize-index-js-4781b3602ad3a2bbb0da.js"],"component---src-pages-pricing-js":["/component---src-pages-pricing-js-77b60e8075370d840e83.js"],"component---src-pages-privacy-js":["/component---src-pages-privacy-js-d5d0d0327a6c36487ae1.js"],"component---src-pages-resources-js":["/component---src-pages-resources-js-22268944990d8ceb3017.js"],"component---src-pages-tags-js":["/component---src-pages-tags-js-32e47399c0e44d54942a.js"],"component---src-pages-wikibrowser-big-loader-js":["/component---src-pages-wikibrowser-big-loader-js-2480c4a479d098fa3c79.js"],"component---src-pages-wikibrowser-categories-js":["/component---src-pages-wikibrowser-categories-js-be322cf37349abed6674.js"],"component---src-pages-wikibrowser-header-js":["/component---src-pages-wikibrowser-header-js-27d60f97d0bfaa86d7d6.js"],"component---src-pages-wikibrowser-index-js":["/component---src-pages-wikibrowser-index-js-bd277474980a4e601169.js"],"component---src-pages-wikibrowser-page-selector-js":["/component---src-pages-wikibrowser-page-selector-js-9ffd7523bf4f0e012591.js"],"component---src-pages-wikibrowser-page-view-js":["/component---src-pages-wikibrowser-page-view-js-90d9ace6d852090ab3e0.js"],"component---src-pages-wikibrowser-related-js":["/component---src-pages-wikibrowser-related-js-268c847fb46795e93a25.js"],"component---src-pages-wikibrowser-sections-js":["/component---src-pages-wikibrowser-sections-js-29f9686b1aeaba46d44a.js"],"component---src-pages-wikibrowser-start-screen-js":["/component---src-pages-wikibrowser-start-screen-js-56c75be8ad98a4738dbc.js"],"component---src-pages-wikibrowser-table-of-contents-js":["/component---src-pages-wikibrowser-table-of-contents-js-311cd188a98614de1993.js"],"component---src-templates-post-js":["/component---src-templates-post-js-a7f7cfc98fbcd66feb7e.js"],"component---src-templates-tag-js":["/component---src-templates-tag-js-4475dea6846e23127a9d.js"]};/*]]>*/ + </script> + <script src="/component---src-templates-post-js-a7f7cfc98fbcd66feb7e.js" async="async"></script> + <script src="/494aa2667243fb352b6c4f73ed1a8fa80c641942-1149805138126f0ca2a7.js" async="async"></script> + <script src="/4d8bd03769955e7ffa305e9df9fd8c45ea8ce544-20cfe20204c6ef10aa5c.js" async="async"></script> + <script src="/a46159e2a7241999ec544c326fa84cca5dda2ca8-419ee48c76badafe338c.js" async="async"></script> + <script src="/885eb025e981378eedf422d5b3545d4eebce3f62-c969f725392e18e66f50.js" async="async"></script> + <script src="/styles-9b6f388623a2ec93d35f.js" async="async"></script> + <script src="/app-e61cbb9e5f9aaa06b962.js" async="async"></script> + <script src="/framework-c078ab5cc0599490105f.js" async="async"></script> + <script src="/webpack-runtime-0492a572fdd49b8e1c9d.js" async="async"></script> + </body> +</html> diff --git a/test/test-pages/uses-getfirstelementchild-function/expected.html b/test/test-pages/uses-getfirstelementchild-function/expected.html index b9f0750..09cd247 100644 --- a/test/test-pages/uses-getfirstelementchild-function/expected.html +++ b/test/test-pages/uses-getfirstelementchild-function/expected.html @@ -2,10 +2,10 @@ <p><img src="http://fakehost/test/logo.jpg" width="400" height="235"> </p> <div> + <br> - - + <h2><a name="general" id="general"></a>General Info</h2> <p>The Seattle Thunderbirds are excited to announce the dates of their annual Summer Hockey Clinic! This three day hockey school will feature top level instruction, both on and off-ice, from Thunderbirds players and coaches. Each day @@ -43,7 +43,7 @@ include Seattle Thunderbirds Head Coach Rob Sumner, Assistant Coach Turner Stevenson, past Thunderbirds Ryan Gibbons and Tyler Metcalfe and past and present Seattle Thunderbirds players.</p> - + <h2><a name="when" id="when"></a>When/Where</h2> <p>The Thunderbirds Summer Hockey Clinic will take place from Tuesday, August 14th, through Thursday, August 16th, at Kingsgate Ice Arena, 14326 124th Ave. NE, Kirkland, WA 98034.</p> @@ -51,7 +51,7 @@ certified helmet with full facemask. Youth players must also wear a mouth guard. Adult players may wear a half visor, however full cages and mouth guards are highly recommended.</p> - + <h2><br> <a name="register" id="register"></a>Registration</h2> <p>• To register and pay for your clinic online, compelte the form below<br> • To register and pay for your clinic via mail, click here to download the registration form</p> diff --git a/test/test-pages/v8-blog/expected.html b/test/test-pages/v8-blog/expected.html index ba11321..76a479d 100644 --- a/test/test-pages/v8-blog/expected.html +++ b/test/test-pages/v8-blog/expected.html @@ -2,9 +2,7 @@ <p> Emscripten has always focused first and foremost on compiling to the Web and other JavaScript environments like Node.js. But as WebAssembly starts to be used <em>without</em> JavaScript, new use cases are appearing, and so we've been working on support for emitting <a href="https://github.com/emscripten-core/emscripten/wiki/WebAssembly-Standalone"><strong>standalone Wasm</strong></a> files from Emscripten, that do not depend on the Emscripten JS runtime! This post explains why that's interesting. </p> - <h2 id="using-standalone-mode-in-emscripten"> - Using standalone mode in Emscripten <a href="#using-standalone-mode-in-emscripten">#</a> - </h2> + <p> First, let's see what you can do with this new feature! Similar to <a href="https://hacks.mozilla.org/2018/01/shrinking-webassembly-and-javascript-code-sizes-in-emscripten/">this post</a> let's start with a "hello world" type program that exports a single function that adds two numbers: </p> diff --git a/test/test-pages/yahoo-2/expected.html b/test/test-pages/yahoo-2/expected.html index 64bca3d..92ef439 100644 --- a/test/test-pages/yahoo-2/expected.html +++ b/test/test-pages/yahoo-2/expected.html @@ -3,7 +3,7 @@ <div> <p><span>1 / 5</span></p> <div> - + <h2>In this photo dated Tuesday, Nov, 29, 2016 the Soyuz-FG rocket booster with the Progress MS-04 cargo ship is installed on a launch pad in Baikonur, Kazakhstan. The unmanned Russian cargo space ship Progress MS-04 broke up in the atmosphere over Siberia on Thursday Dec. 1, 2016, just minutes after the launch en route to the International Space Station due to an unspecified malfunction, the Russian space agency said.(Oleg Urusov/ Roscosmos Space Agency Press Service photo via AP)</h2> <p>In this photo dated Tuesday, Nov, 29, 2016 the Soyuz-FG rocket booster with the Progress MS-04 cargo ship is installed on a launch pad in Baikonur, Kazakhstan. The unmanned Russian cargo space ship Progress MS-04 broke up in the atmosphere over Siberia on Thursday Dec. 1, 2016, just minutes after the launch en route to the International Space Station due to an unspecified malfunction, the Russian space agency said.(Oleg Urusov/ Roscosmos Space Agency Press Service photo via AP)</p> </div></div> <div> |