Handling Binary Data — Building a HTTP Server from scratch

On the last post of BTS: HTTP Server series.
I wrote a barebone HTTP server that can handle requests and respond appropriately.
I think I covered the basics, but that server is limited in what it can do.
It can only handle text-based Requests and Responses... That means no image or other media exchange.
And then, if the Request or the Response is larger than a KB, I'm out of luck. Again, not great for media...

This article is a transcript of a Youtube video I made.

Oh, hey there...

That's my challenge for today, refactor my server to handle arbitrarily sized Requests and avoid treating everything as
text...

If I want to be able to handle large requests, the first thing I can do is to read the stream in chunks, 1KB at a time
until there's nothing left to read.
Once I have all of my chunks, I can concatenate them together into one Typed Array. And voila, arbitrarly sized Request!

const concat = (...chunks) => {
  const zs = new Uint8Array(chunks.reduce((z, ys) => z + ys.byteLength, 0));
  chunks.reduce((i, xs) => zs.set(xs, i) || i + xs.byteLength, 0);
  return zs;
};

const chunks = [];
let n;
do {
    const xs = new Uint8Array(1024);
    n = await r.read(xs);
    chunks.push(xs.subarray(0, n));
} while (n === 1024);

const request = concat(...chunks);

The second challenge is to figure out how much of the data stream is the Request line and the Headers versus the body...
I want to avoid reading too far into the body, since it might be binary data.
I know that the body starts after the first empty line of the Request.
So I could technically, search for the first empty line and then I'll know that the rest is the body and only parse the first part.

So I wrote this function that will try to find a sequence within the array. First tries to find the first occurence of
a byte, and then I can just test the following bytes until I have a match.
In our case, I want to find a two CRLF sequences. So I try to find the first CR, then check if it is followed by LF, CR
and LF... And, I repeat this until I find the empty line.

export const findIndexOfSequence = (xs, ys) => {
  let i = xs.indexOf(ys[0]);
  let z = false;

  while (i >= 0 && i < xs.byteLength) {
    let j = 0;
    while (j < ys.byteLength) {
      if (xs[j + i] !== ys[j]) break;
      j++;
    }
    if (j === ys.byteLength) {
      z = true;
      break;
    }
    i++;
  }

  return z ? i : null;
};

🐙 You will find the code for this post here: https://github.com/i-y-land/HTTP/tree/episode/03

The problem with this approach is that I have to traverse the whole request, and it might end up that the request doesn't
have a body, and therefore I wasted my time.

Instead, I will read the bytes one line at a time, finding the nearest CRLF and parse them in order.
On the first line, I will extract the method and the path.
Whenever I find an empty line, I will assume the is body is next and stop.
For the remaining lines, I will parse them as header.

// https://github.com/i-y-land/HTTP/blob/episode/03/library/utilities.js#L208
export const readLine = (xs) => xs.subarray(0, xs.indexOf(LF) + 1);

export const decodeRequest = (xs) => {
  const headers = {};
  let body, method, path;
  const n = xs.byteLength;
  let i = 0;
  let seekedPassedHeader = false;
  while (i < n) {
    if (seekedPassedHeader) {
      body = xs.subarray(i, n);
      i = n;
      continue;
    }

    const ys = readLine(xs.subarray(i, n));

    if (i === 0) {
      if (!findIndexOfSequence(ys, encode(" HTTP/"))) break;
      [method, path] = decode(ys).split(" ");
    } else if (
      ys.byteLength === 2 &&
      ys[0] === CR &&
      ys[1] === LF &&
      xs[i] === CR &&
      xs[i + 1] === LF
    ) {
      seekedPassedHeader = true;
    } else if (ys.byteLength === 0) break;
    else {
      const [key, value] = decode(
        ys.subarray(0, ys.indexOf(CR) || ys.indexOf(LF)),
      ).split(/(?<=^[A-Za-z-]+)\s*:\s*/);
      headers[key.toLowerCase()] = value;
    }

    i += ys.byteLength;
  }

  return { body, headers, method, path };
};

On the other hand, the function to encode the Response is absurdly simpler, I can pretty much use the function I already made
and just encode the result. The biggest difference, is that I have to be aware that the body might not
be text and should be kept as a Typed Array. I can encode the header and then concat the result with the body.

// https://github.com/i-y-land/HTTP/blob/episode/03/library/utilities.js#L248
export const stringifyHeaders = (headers = {}) =>
  Object.entries(headers)
    .reduce(
      (hs, [key, value]) => `${hs}\r\n${normalizeHeaderKey(key)}: ${value}`,
      "",
    );

export const encodeResponse = (response) =>
  concat(
    encode(
      `HTTP/1.1 ${statusCodes[response.statusCode]}${
        stringifyHeaders(response.headers)
      }\r\n\r\n`,
    ),
    response.body || new Uint8Array(0),
  );

From there, I have enough to write a simple server using the serve function I've implemented previously.
I can decode the request... then encode the response.

...
serve(
  Deno.listen({ port }),
  (xs) => {
    const request = decodeRequest(xs);

    if (request.method === "GET" && request.path === "/") {
      return encodeResponse({ statusCode: 204 })
    }
  }
).catch((e) => console.error(e));

I could respond to every requests with a file. That is a good start to a static file server.

...
    if (request.method === "GET" && request.path === "/") {
      const file = Deno.readFile(`${Deno.cwd()}/image.png`); // read the file
      return encodeResponse({
        body: file,
        headers: {
          "content-length": file.byteLength,
          "content-type": "image/png"
        },
        statusCode: 200
      });
    }

I can start my server and open a browser to visualize the image.

With a bit more effort, I can serve any file withing a given directory.
I would attempt to access the file and cross-reference the MIME type from a currated list using the extension.
If the system can't find the file, I will return 404 Not Found.

const sourcePath =
    (await Deno.permissions.query({ name: "env", variable: "SOURCE_PATH" }))
            .state === "granted" && Deno.env.get("SOURCE_PATH") ||
    `${Deno.cwd()}/library/assets_test`;
...
    if (request.method === "GET") {
      try {
        const file = await Deno.readFile(sourcePath + request.path); // read the file
        return encodeResponse({
          body: file,
          headers: {
            "content-length": file.byteLength,
            ["content-type"]: mimeTypes[
              request.path.match(/(?<extension>\.[a-z0-9]+$)/)?.groups?.extension
                .toLowerCase()
              ].join(",") || "plain/text",
          },
          statusCode: 200
        });
      } catch (e) {
        if (e instanceof Deno.errors.NotFound) { // if the file is not found
          return encodeResponse({
            body: new Uint8Array(0),
            headers: {
              ["Content-Length"]: 0,
            },
            statusCode: 404,
          });
        }

        throw e;
      }
    }

With a broadly similar approach, I can receive any file.

const targetPath =
    (await Deno.permissions.query({ name: "env", variable: "TARGET_PATH" }))
            .state === "granted" && Deno.env.get("TARGET_PATH") ||
    `${Deno.cwd()}/`;
...
    if (request.method === "GET") { ... }
    else if (request.method === "POST") {
      await Deno.writeFile(targetPath + request.path, request.body); // write the file
      return encodeResponse({ statusCode: 204 });
    }

Now, you can guess if you look at the position of your scrollbar that things can't be that simple...

I see two problems with my current approach.
I have to load whole files into memory before I can offload it to the File System which that can become a bottle neck at
scale.
Another surprising issue is with file uploads...
When uploading a file, some clients, for example curl will make the request in two steps... The first request is
testing the terrain stating that it wants to upload a file of a certain type and length and requires that the server
replies with 100 continue before sending the file.
Because of this behaviour I need to retain access to the connection, the writable resource.
So I think I will have to refactor the serve function from accepting a function that takes a Typed Array as an
argument, to a function that takes the connection.
This could also be positive change that would facilitate implementing powerful middleware later on...

export const serve = async (listener, f) => {
  for await (const connection of listener) {
    await f(connection);
  }
};

There's two ways that my server can handle file uploads.
One possibility is that the client tries to to post the file directly,
I have the option to read the header and refuse the request if it's too large. The other possibility is that the
client expects me to reply first.
In both case I will read the first chunk and then start creating the file with the data processed. Then I want to
to read one chunk at a time from the connection and systematically write them to the file. This way, I never hold
more than 1KB in memory at a time... I do this until I can't read a whole 1KB, this tells me that the file has been
completely copied over.

export const copy = async (r, w) => {
  const xs = new Uint8Array(1024);
  let n;
  let i = 0;
  do {
    n = await r.read(xs);
    await w.write(xs.subarray(0, n));
    i += n;
  } while (n === 1024);

  return i;
};
...
    let xs = new Uint8Array(1024);
    const n = await Deno.read(r.rid, xs);
    const request = xs.subarray(0, n);
    const { fileName } = request.path.match(
      /.*?\/(?<fileName>(?:[^%]|%[0-9A-Fa-f]{2})+\.[A-Za-z0-9]+?)$/,
    )?.groups || {};

    ...

    const file = await Deno.open(`${targetPath}/${fileName}`, {
      create: true,
      write: true,
    });

    if (request.headers.expect === "100-continue") {
      // write the `100 Continue` response
      await Deno.write(connection.rid, encodeResponse({ statusCode: 100 }));

      const ys = new Uint8Array(1024);
      const n = await Deno.read(connection.rid, ys); // read the follow-up
      xs = ys.subarray(0, n);
    }

    const i = findIndexOfSequence(xs, CRLF); // find the beginning of the body

    if (i > 0) {
      await Deno.write(file.rid, xs.subarray(i + 4)); // write possible file chunk
      if (xs.byteLength === 1024) {
        await copy(connection, file); // copy subsequent chunks
      }
    }

    await connection.write(
      encodeResponse({ statusCode: 204 }), // terminate the exchange
    );
...

From there, I can rework the part that responds with a file.
Similarly to the two-step request for receiving a file, a client may opt to request the headers for a given file
with the HEAD method.
Because I want to support this feature, I can first get information from the requested file, then I can start writing
the headers and only if the request's method is GET -- not HEAD -- I will copy the file to the connection.

...
    try {
      const { size } = await Deno.stat(`${sourcePath}/${fileName}`);

      await connection.write(
        encodeResponse({
          headers: {
            ["Content-Type"]: mimeTypes[
              fileName.match(/(?<extension>\.[a-z0-9]+$)/)?.groups?.extension
                .toLowerCase()
              ].join(",") || "plain/text",
            ["Content-Length"]: size,
          },
          statusCode: 200,
        }),
      );

      if (request.method === "GET") {
        const file = await Deno.open(`${sourcePath}/${fileName}`);
        await copy(file, connection);
      }
    } catch (e) {
      if (e instanceof Deno.errors.NotFound) {
        Deno.write(
          connection.rid,
          encodeResponse({
            headers: {
              ["Content-Length"]: 0,
            },
            statusCode: 404,
          }),
        );
      }

      throw e;
    }
...

Wow. At this point I have to be either very confident with my programming skills or sadistic...
I need to implement a slew of integrations tests before going any further.
I created four static files for this purpose, a short text file, less than a KB, a longer text file, an image and
music...
For that purpose, I wrote a higher-order-function that will initialize the server before calling the test function.

// https://github.com/i-y-land/HTTP/blob/episode/03/library/integration_test.js#L6
const withServer = (port, f) =>
  async () => {
    const p = await Deno.run({ // initialize the server
      cmd: [
        "deno",
        "run",
        "--allow-all",
        `${Deno.cwd()}/cli.js`,
        String(port),
      ],
      env: { LOG_LEVEL: "ERROR", "NO_COLOR": "1" },
      stdout: "null",
    });

    await new Promise((resolve) => setTimeout(resolve, 1000)); // wait to be sure

    try {
      await f(p); // call the test function passing the process
    } finally {
      Deno.close(p.rid);
    }
  };

With that, I generate a bunch of tests to download and upload files; this ensures that my code is working as expected.

// https://github.com/i-y-land/HTTP/blob/episode/03/library/integration_test.js#L58
[...]
  .forEach(
    ({ headers = {}, method = "GET", path, title, f }) => {
      Deno.test(
        `Integration: ${title}`,
        withServer(
          8080,
          async () => {
            const response = await fetch(`http://localhost:8080${path}`, {
              headers,
              method,
            });
            await f(response);
          },
        ),
      );
    },
  );

When I got to that point, I realized that my serve function was starting to be very... long.
I knew I needed to refactor it into two functions receiveStaticFile and sendStaticFile.
But, because I need to be able to check the Request line to route to the right function, and I can only read the request
once...
I knew that I was in trouble.

I need something that can keep part of the data in memory while retaining access to the raw connection...

...
    if (method === "POST") {
      return receiveStaticFile(?, { targetPath });
    } else if (method === "GET" || method === "HEAD") {
      return sendStaticFile(?, { sourcePath });
    }
...

I could have decoded the request and shove the connection in there and call it a day...
But it didn't feel right aaaand I guess I love making my life harder.

const request = decodeRequest(connection);
request.connection = connection;

...
    if (method === "POST") {
      return receiveStaticFile(request, { targetPath });
    } else if (method === "GET" || method === "HEAD") {
      return sendStaticFile(request, { sourcePath });
    }
...

The solution I came up with was to write a buffer. It would hold in memory only a KB at a time, shifting the bytes
each time I read a new chunk. The advantage of that is I can move the cursor back to the beginning of the buffer
and read-back parts that I need.
Best of all, the buffer has the same methods as the connection; so the two could be used interchangeably.
I won't go into the details because it's a bit dry, but if you want to checkout the code, it's currently on Github.

// https://github.com/i-y-land/HTTP/blob/episode/03/library/utilities.js#L11
export const factorizeBuffer = (r, mk = 1024, ml = 1024) => { ... }

With this new toy I can read a chunk from the connection, route the request, move the cursor back to the beginning and
pass the buffer to the handler function like nothing happened.

The peek function specifically has a similar signature to read, the difference is that it will move the cursor
back, read a chunk from the buffer in memory and then finally move the cursor back again.

serve(
  Deno.listen({ port }),
  async (connection) => {
    const r = factorizeBuffer(connection);

    const xs = new Uint8Array(1024);
    const reader = r.getReader();
    await reader.peek(xs);
    const [method] = decode(readLine(xs)).split(" ");

    if (method !== "GET" && method !== "POST" && method !== "HEAD") {
      return connection.write(
        encodeResponse({ statusCode: 400 }),
      );
    }

    if (method === "POST") {
      return receiveStaticFile(r, { targetPath });
    } else {
      return sendStaticFile(r, { sourcePath });
    }
  }
)

To finish this, like a boss, I finalize the receiveStaticFile (https://github.com/i-y-land/HTTP/blob/episode/03/library/server.js#L15) and sendStaticFile (https://github.com/i-y-land/HTTP/blob/episode/03/library/server.js#L71) functions, taking care of all
the edge cases.
Finally, I run all the integration tests to confirm that I did a good job. And uuugh. Sleeeep.

This one turned out to be a lot more full of surprise than I was prepared for.
When I realized that some client send file in two-steps, it really threw a wrench to my plans...
But it turned out to an amazing learning opportunity.
I really hope that you are learning as much as I am.
On the bright side, this forced me to put together all the tools that I know I will need for the next post.
Next, I want to look into streaming in more details and build some middlewares, starting with a logger.
From there, I am sure that I can tackle building a nice little router which will wrap this up pretty nicely.

All of the code is available on Github, if you have a question do no hesitate to ask...
Oh speaking of that, I launched a Discord server, if you want to join.

🐙 You will find the code for this episode here: https://github.com/i-y-land/HTTP/tree/episode/03

💬 You can join the I-Y community on Discord: https://discord.gg/eQfhqybmSc

At any rate, if this article was useful to you, hit the like button, leave a comment to let me know or best of all,
follow if you haven't already!

Ok bye now...