From 3464e0409dfda479e00957a5b684389d949fb27a Mon Sep 17 00:00:00 2001 From: Andres Rey Date: Sat, 10 Dec 2016 14:54:28 +0000 Subject: Modified the expected files to match the readability.php style of result --- test/test-pages/002/expected.html | 115 +++++++++++++++--------------------- test/test-pages/ars-1/expected.html | 36 +++++------ 2 files changed, 63 insertions(+), 88 deletions(-) (limited to 'test') diff --git a/test/test-pages/002/expected.html b/test/test-pages/002/expected.html index f70b310..25820a5 100644 --- a/test/test-pages/002/expected.html +++ b/test/test-pages/002/expected.html @@ -18,19 +18,18 @@ extensible web movement.

As of this writing, the Fetch API is available in Firefox 39 (currently Nightly) and Chrome 42 (currently dev). Github has a Fetch polyfill.

- +

Feature detection

Fetch API support can be detected by checking for Headers,Request, Response or fetch on the window or worker scope.

- +

Simple fetching

The most useful, high-level part of the Fetch API is the fetch() function. In its simplest form it takes a URL and returns a promise that resolves to the response. The response is captured as a Response object.

-

- 
fetch("/data.json").then(function(res) {
+                        

-                                    
fetch("/data.json").then(function(res) {
   // res instanceof Response == true.
   if (res.ok) {
     res.json().then(function(data) {
@@ -43,10 +42,9 @@
   console.log("Fetch failed!", e);
 });

                                         

+

Submitting some parameters, it would look like this:

-

- 
fetch("http://www.example.org/submit.php", {
+            

-                        
fetch("http://www.example.org/submit.php", {
   method: "POST",
   headers: {
     "Content-Type": "application/x-www-form-urlencoded"
@@ -62,12 +60,12 @@
   alert("Error submitting form!");
 });

                             

+

The fetch() function’s arguments are the same as those passed to the

Request() constructor, so you may directly pass arbitrarily complex requests to fetch() as discussed below.

- +

Headers

Fetch introduces 3 interfaces. These are Headers, Request and @@ -78,39 +76,36 @@

supporting CORS rules and ensuring cookies aren’t readable by third parties.

The Headers interface is a simple multi-map of names to values:

-

- 
var content = "Hello World";
+            

-                        
var content = "Hello World";
 var reqHeaders = new Headers();
 reqHeaders.append("Content-Type", "text/plain"
 reqHeaders.append("Content-Length", content.length.toString());
 reqHeaders.append("X-Custom-Header", "ProcessThisImmediately");

                             

+

The same can be achieved by passing an array of arrays or a JS object literal

to the constructor:

-

- 
reqHeaders = new Headers({
+            

-                        
reqHeaders = new Headers({
   "Content-Type": "text/plain",
   "Content-Length": content.length.toString(),
   "X-Custom-Header": "ProcessThisImmediately",
 });

                             

+

The contents can be queried and retrieved:

-

- 
console.log(reqHeaders.has("Content-Type")); // true
+            

-                        
console.log(reqHeaders.has("Content-Type")); // true
 console.log(reqHeaders.has("Set-Cookie")); // false
 reqHeaders.set("Content-Type", "text/html");
 reqHeaders.append("X-Custom-Header", "AnotherValue");
-
+ 
 console.log(reqHeaders.get("Content-Length")); // 11
 console.log(reqHeaders.getAll("X-Custom-Header")); // ["ProcessThisImmediately", "AnotherValue"]
-
+ 
 reqHeaders.delete("X-Custom-Header");
 console.log(reqHeaders.getAll("X-Custom-Header")); // []

                             

+

Some of these operations are only useful in ServiceWorkers, but they provide

a much nicer API to Headers.

Since Headers can be sent in requests, or received in responses, and have @@ -135,16 +130,15 @@ console.log(<

All of the Headers methods throw TypeError if name is not a valid HTTP Header name. The mutation operations will throw TypeError if there is an immutable guard. Otherwise they fail silently. For example:

-

- 
var res = Response.error();
+            

-                        
var res = Response.error();
 try {
   res.headers.set("Origin", "http://mybank.com");
 } catch(e) {
   console.log("Cannot pretend to be a bank!");
 }

                             

-
+
+

Request

The Request interface defines a request to fetch a resource over HTTP. @@ -152,29 +146,26 @@ console.log(< a body, a request mode, credentials and cache hints.

The simplest Request is of course, just a URL, as you may do to GET a resource.

-

- 
var req = new Request("/index.html");
+        

-                    
var req = new Request("/index.html");
 console.log(req.method); // "GET"
 console.log(req.url); // "http://example.com/index.html"

                         

+

You may also pass a Request to the Request() constructor to create a copy.

(This is not the same as calling the clone() method, which is covered in

the “Reading bodies” section.).

-

- 
var copy = new Request(req);
+        

-                    
var copy = new Request(req);
 console.log(copy.method); // "GET"
 console.log(copy.url); // "http://example.com/index.html"

                         

+

Again, this form is probably only useful in ServiceWorkers.

The non-URL attributes of the Request can only be set by passing initial

values as a second argument to the constructor. This argument is a dictionary.

-

- 
var uploadReq = new Request("/uploadImage", {
+        

-                    
var uploadReq = new Request("/uploadImage", {
   method: "POST",
   headers: {
     "Content-Type": "image/png",
@@ -182,7 +173,7 @@ console.log(<
   body: "image data"
 });

                         

+

The Request’s mode is used to determine if cross-origin requests lead to valid responses, and which properties on the response are readable. Legal mode values are "same-origin", "no-cors" (default) @@ -191,15 +182,14 @@ console.log(< origin with this mode set, the result is simply an error. You could use this to ensure that

a request is always being made to your origin.

-

- 
var arbitraryUrl = document.getElementById("url-input").value;
+            

-                        
var arbitraryUrl = document.getElementById("url-input").value;
 fetch(arbitraryUrl, { mode: "same-origin" }).then(function(res) {
   console.log("Response succeeded?", res.ok);
 }, function(e) {
   console.log("Please enter a same-origin URL!");
 });

                             

+

The "no-cors" mode captures what the web platform does by default for scripts you import from CDNs, images hosted on other domains, and so on. First, it prevents the method from being anything other than “HEAD”, @@ -217,15 +207,14 @@ fetch(arbitraryUrlmost interesting photos today like this:

-

- 
var u = new URLSearchParams();
+            

-                        
var u = new URLSearchParams();
 u.append('method', 'flickr.interestingness.getList');
 u.append('api_key', '<insert api key here>');
 u.append('format', 'json');
 u.append('nojsoncallback', '1');
-
+ 
 var apiCall = fetch('https://api.flickr.com/services/rest?' + u);
-
+ 
 apiCall.then(function(response) {
   return response.json().then(function(json) {
     // photo is a list of photos.
@@ -237,14 +226,13 @@ apiCall.then(
   });
 });

                             

+

You may not read out the “Date” header since Flickr does not allow it via

Access-Control-Expose-Headers.

-

- 
response.headers.get("Date"); // null
+
- 
response.headers.get("Date"); // null

+

The credentials enumeration determines if cookies for the other domain are

sent to cross-origin requests. This is similar to XHR’s withCredentials @@ -262,7 +250,7 @@ apiCall.then( <img>tag in the controlled document, “worker” if it is an attempt to load a worker script, and so on. When used with the fetch() function, it is “fetch”.

- +

Response

Response instances are returned by calls to fetch(). @@ -296,14 +284,13 @@ apiCall.then( The

idiomatic way to return a Response to an intercepted request in ServiceWorkers is:

-

- 
addEventListener('fetch', function(event) {
+            

-                        
addEventListener('fetch', function(event) {
   event.respondWith(new Response("Response body", {
     headers: { "Content-Type" : "text/plain" }
   });
 });

                             

+

As you can see, Response has a two argument constructor, where both arguments are optional. The first argument is a body initializer, and the second is a dictionary to set the status, statusText and headers.

@@ -311,7 +298,7 @@ apiCall.then( response. Similarly, Response.redirect(url, status) returns a Response resulting in

a redirect to url.

- +

Dealing with bodies

Both Requests and Responses may contain body data. We’ve been glossing @@ -347,43 +334,40 @@ apiCall.then(

This is a significant improvement over XHR in terms of ease of use of non-text data!

Request bodies can be set by passing body parameters:

-

- 
var form = new FormData(document.getElementById('login-form'));
+            

-                        
var form = new FormData(document.getElementById('login-form'));
 fetch("/login", {
   method: "POST",
   body: form
 })

                             

+

Responses take the first argument as the body.

-

- 
var res = new Response(new File(["chunk", "chunk"], "archive.zip",
+                

-                            
var res = new Response(new File(["chunk", "chunk"], "archive.zip",
                        { type: "application/zip" }));

                                 

+

Both Request and Response (and by extension the fetch() function), will try to intelligently determine the content type. Request will also automatically set a “Content-Type” header if none is set in the dictionary.

- +

Streams and cloning

It is important to realise that Request and Response bodies can only be read once! Both interfaces have a boolean attribute bodyUsed to determine if it is safe to read or not.

-

- 
var res = new Response("one time use");
+                

-                            
var res = new Response("one time use");
 console.log(res.bodyUsed); // false
 res.text().then(function(v) {
   console.log(res.bodyUsed); // true
 });
 console.log(res.bodyUsed); // true
-
+ 
 res.text().catch(function(e) {
   console.log("Tried to read already consumed Response");
 });

                                 

+

This decision allows easing the transition to an eventual stream-based Fetch API. The intention is to let applications consume data as it arrives, allowing for JavaScript to deal with larger files like videos, and perform things @@ -397,24 +381,23 @@ res.text(clone() MUST be called before the body of the corresponding object has been used. That is, clone() first, read later.

-

- 
addEventListener('fetch', function(evt) {
+                

-                            
addEventListener('fetch', function(evt) {
   var sheep = new Response("Dolly");
   console.log(sheep.bodyUsed); // false
   var clone = sheep.clone();
   console.log(clone.bodyUsed); // false
-
+ 
   clone.text();
   console.log(sheep.bodyUsed); // false
   console.log(clone.bodyUsed); // true
-
+ 
   evt.respondWith(cache.add(sheep.clone()).then(function(e) {
     return sheep;
   });
 });

                                 

-
+
+

Future improvements

Along with the transition to streams, Fetch will eventually have the ability diff --git a/test/test-pages/ars-1/expected.html b/test/test-pages/ars-1/expected.html index 3e04fb3..324787f 100644 --- a/test/test-pages/ars-1/expected.html +++ b/test/test-pages/ars-1/expected.html @@ -1,14 +1,8 @@ -

-
-
-
-

A flaw in the wildly popular online game Minecraft makes it easy for just about anyone to crash the server hosting the game, according to a computer programmer who has released proof-of-concept code that exploits the vulnerability.

-

"I thought a lot before writing this post," Pakistan-based developer Ammar Askar wrote in a blog post published Thursday, 21 months, he said, after privately reporting the bug to Minecraft developer Mojang. "On the one hand I don't want to expose thousands of servers to a major vulnerability, yet on the other hand Mojang has failed to act on it."

-

The bug resides in the networking internals of the Minecraft protocol. It allows the contents of inventory slots to be exchanged, so that, among other things, items in players' hotbars are displayed automatically after logging in. Minecraft items can also store arbitrary metadata in a file format known as Named Binary Tag (NBT), which allows complex data structures to be kept in hierarchical nests. Askar has released proof-of-concept attack code he said exploits the vulnerability to crash any server hosting the game. Here's how it works.

-
-

The vulnerability stems from the fact that the client is allowed to send the server information about certain slots. This, coupled with the NBT format’s nesting allows us to craft a packet that is incredibly complex for the server to deserialize but trivial for us to generate.

-

In my case, I chose to create lists within lists, down to five levels. This is a json representation of what it looks like.

- 
rekt: {
+

+                            
A flaw in the wildly popular online game Minecraft makes it easy for just about anyone to crash the server hosting the game, according to a computer programmer who has released proof-of-concept code that exploits the vulnerability.
"I thought a lot before writing this post," Pakistan-based developer Ammar Askar wrote in a blog post published Thursday, 21 months, he said, after privately reporting the bug to Minecraft developer Mojang. "On the one hand I don't want to expose thousands of servers to a major vulnerability, yet on the other hand Mojang has failed to act on it."
The bug resides in the networking internals of the Minecraft protocol. It allows the contents of inventory slots to be exchanged, so that, among other things, items in players' hotbars are displayed automatically after logging in. Minecraft items can also store arbitrary metadata in a file format known as Named Binary Tag (NBT), which allows complex data structures to be kept in hierarchical nests. Askar has released proof-of-concept attack code he said exploits the vulnerability to crash any server hosting the game. Here's how it works.

+                            
The vulnerability stems from the fact that the client is allowed to send the server information about certain slots. This, coupled with the NBT format’s nesting allows us to craft a packet that is incredibly complex for the server to deserialize but trivial for us to generate.

+                            
In my case, I chose to create lists within lists, down to five levels. This is a json representation of what it looks like.

+                            
rekt: {
     list: [
         list: [
             list: [
@@ -33,14 +27,12 @@
         ...
     ]
     ...
-}

-            
The root of the object, rekt, contains 300 lists. Each list has a list with 10 sublists, and each of those sublists has 10 of their own, up until 5 levels of recursion. That’s a total of 10^5 * 300 = 30,000,000 lists.

-            
And this isn’t even the theoretical maximum for this attack. Just the nbt data for this payload is 26.6 megabytes. But luckily Minecraft implements a way to compress large packets, lucky us! zlib shrinks down our evil data to a mere 39 kilobytes.

-            
Note: in previous versions of Minecraft, there was no protocol wide compression for big packets. Previously, NBT was sent compressed with gzip and prefixed with a signed short of its length, which reduced our maximum payload size to 2^15 - 1. Now that the length is a varint capable of storing integers up to 2^28, our potential for attack has increased significantly.

-            
When the server will decompress our data, it’ll have 27 megs in a buffer somewhere in memory, but that isn’t the bit that’ll kill it. When it attempts to parse it into NBT, it’ll create java representations of the objects meaning suddenly, the sever is having to create several million java objects including ArrayLists. This runs the server out of memory and causes tremendous CPU load.

-            
This vulnerability exists on almost all previous and current Minecraft versions as of 1.8.3, the packets used as attack vectors are the 0x08: Block Placement Packet and 0x10: Creative Inventory Action.

-            
The fix for this vulnerability isn’t exactly that hard, the client should never really send a data structure as complex as NBT of arbitrary size and if it must, some form of recursion and size limits should be implemented.

-            
These were the fixes that I recommended to Mojang 2 years ago.

-        

-        
Ars is asking Mojang for comment and will update this post if company officials respond.

-    

\ No newline at end of file
+}

+

The root of the object, rekt, contains 300 lists. Each list has a list with 10 sublists, and each of those sublists has 10 of their own, up until 5 levels of recursion. That’s a total of 10^5 * 300 = 30,000,000 lists.

+

And this isn’t even the theoretical maximum for this attack. Just the nbt data for this payload is 26.6 megabytes. But luckily Minecraft implements a way to compress large packets, lucky us! zlib shrinks down our evil data to a mere 39 kilobytes.

+

Note: in previous versions of Minecraft, there was no protocol wide compression for big packets. Previously, NBT was sent compressed with gzip and prefixed with a signed short of its length, which reduced our maximum payload size to 2^15 - 1. Now that the length is a varint capable of storing integers up to 2^28, our potential for attack has increased significantly.

+

When the server will decompress our data, it’ll have 27 megs in a buffer somewhere in memory, but that isn’t the bit that’ll kill it. When it attempts to parse it into NBT, it’ll create java representations of the objects meaning suddenly, the sever is having to create several million java objects including ArrayLists. This runs the server out of memory and causes tremendous CPU load.

+

This vulnerability exists on almost all previous and current Minecraft versions as of 1.8.3, the packets used as attack vectors are the 0x08: Block Placement Packet and 0x10: Creative Inventory Action.

+

The fix for this vulnerability isn’t exactly that hard, the client should never really send a data structure as complex as NBT of arbitrary size and if it must, some form of recursion and size limits should be implemented.

+

These were the fixes that I recommended to Mojang 2 years ago.

+

Ars is asking Mojang for comment and will update this post if company officials respond.

\ No newline at end of file -- cgit v1.2.3