The better way to access process.env variables

No cloud application lives alone, but is often depending on input and output connections to function. This require config about the upstream / downstream services we are interacting with.

I can't tell you how many times I've seen the following error in production logs:

Error: Cannot read property 'foo' undefined

These bugs may not be trivial to address. So what's really going on here? Why was this not caught in a unit or integration test?

The reason can often be found in missing runtime configuration when deploying your app to higher environments from e.g dev to uat to staging. Runtime variables need to be configured in CI/CD (depending on your setup) and/or in AWS Secrets Manager.

Consider below function uploading images to AWS S3:

export const putObject = async (
  bucket: string = process.env.S3Bucket,
  key: string,
  image: Buffer,
): Promise<string> => {
  const input: PutObjectCommandInput = {
    Bucket: bucket,
    Key: key,
    Body: image,
    ContentType: 'image/jpeg',
  };

  const command: PutObjectCommand = new PutObjectCommand(input);

  try {
    await s3Client.send(command);
  } catch (error) {
    const { name, message } = error;
    logger.error(`s3.PutObjectCommand ${name} ${message}`, {
      data: omit(input, 'Body'),
    });
    throw error;
  }
};

Let's focus on L2:

bucket: string = process.env.S3Bucket,

What happens when process.env.S3Bucket is missing from the current runtime env? The application breaks with an error from @AWS-SDK:

Error: No value provided for input HTTP label: Bucket.
    at resolvedPath (/Users/christian/Documents/GitHub/serverless-image-transformation/node_modules/@aws-sdk/smithy-client/dist-cjs/resolve-path.js:19:15)
    at serializeAws_restXmlPutObjectCommand (/Users/christian/Documents/GitHub/serverless-image-transformation/node_modules/@aws-sdk/client-s3/dist-cjs/protocols/Aws_restXml.js:2437:40)
    at <anonymous> (/Users/christian/Documents/GitHub/serverless-image-transformation/node_modules/@aws-sdk/middleware-serde/dist-cjs/serializerMiddleware.js:12:21)

Now imagine the person reading the error logs is a DevOps Engineer in another team, who lacks specific domain knowledge about your app. Now a ticket is being raised, pending investigation. Meanwhile your customers are having a bad user experience, which may result in financial loss.

You must log meaningful error messages so engineers without specific domain knowledge understands the root cause and can action it immediately to minimise service disruption

The solution to this pickle comes two fold:

Do not access process.env directly and do not assume any variables are defined
Access process.env variables using a helper function doing the proper checks and throwing meaningful configuration errors

So instead of the error Error: No value provided for input HTTP label: Bucket. I'd rather see this in the logs:

ConfigurationError: S3Bucket is required

This is a no-brainer and I reckon anyone with a degree in software development will know where to start looking.

Such a function could look like this:

import createError from 'http-errors';
import { Config } from '@/constants';

export const getConfig = (
  key: string,
  isRequired = true,
  fallbackValue?: string,
): string | undefined => {
  const value: string | undefined = process.env[String(key)];

  if (!value?.length) {
    const message = `Configuration error: '${key}' is required`;

    if (isRequired) {
      throw createError(500, message);
    }
    console.warn(
      `Configuration warning: Optional key '${key}' accessed, but not present in runtime config. Fallback value: ${fallbackValue}`,
    );
    return fallbackValue || undefined;
  }
  return value;
};

/utils/env.ts

Source:

The fix up

Let's fix up our S3 upload function to utilise the getConfig helper.

First off, let's stop using "magic strings" and statically type the config keys into a file and check it into the git repo:

export class Config {
  static readonly S3_BUCKET = 'S3_BUCKET';
  static readonly MY_OTHER_VAR = 'MY_OTHER_VAR';
}

config.ts

We're doing this for below reasons:

Avoiding typos in config key names e.g process.env.MY_COFNIG_VAR
Making app configuration a first class citizen
A config file also serves as documentation - e.g DevOps can also read the file and assist with configuration during deployments
When using static configuration keys we can iterate over them and check on app invocation if the configuration requirements have been satisfied (avoid running business processes with an incomplete config)
Refactoring configuration key names is a breeze, because you only have to change it in the config.ts file
Examining the git log you can track when the config keys were changed and who changed them
When working on another developer's codebase requiring a .env file on localhost but you have no idea what that required config might look like, you now have config.ts to provide clarity

Next, let's use the helper (in L5):

import { getConfig } from '@/utils/env';
import { Config } from '@/constants';

export const putObject = async (
  bucket: string = getConfig(Config.S3_BUCKET),
  key: string,
  image: Buffer,
): Promise<string> => {
  const input: PutObjectCommandInput = {
    Bucket: bucket,
    Key: key,
    Body: image,
    ContentType: 'image/jpeg',
  };

  const command: PutObjectCommand = new PutObjectCommand(input);

  try {
    await s3Client.send(command);
  } catch (error) {
    const { name, message } = error;
    logger.error(`s3.PutObjectCommand ${name} ${message}`, {
      data: omit(input, 'Body'),
    });
    throw error;
  }
};

This is much better. It's a clean approach using an inline helper and errors are handled uniformly at a central point, throwing meaningful messages relating to configuration errors - and not @AWS-SDK errors.

As an added bonus, the app will fail before invoking the S3 upload command which we know will fail due to missing config.

Yeah, yeah, but is is good enough?

There is one issue I see with above solution: The app only breaks when trying to access a specific missing runtime variable.

Imagine you had an app, that under limited circumstances would upload a file to S3. Let's imagine that only happened when a customer with an expired pre-saved credit logged into her account. That happens sometimes, but maybe not every day.

What if the S3_BUCKET config was missing? Now it appears as the app is running smoothly on a day-to-day basis, but eventually a breakage will occur.

What if, we could catch all of these shenanigans immediately? We could cook up a solution where the entire config was verified before invoking the main handler.

Given we know the config key names from config.ts we can just run a check in a Middy middleware function:

import type { Config } from '@/constants';
import { getConfig } from '@/utils/env';
import middy from '@middy/core';
import { APIGatewayProxyEvent, APIGatewayProxyResult } from 'aws-lambda';

export const verifyConfig = (
  config: Config,
): middy.MiddlewareObj<APIGatewayProxyEvent, APIGatewayProxyResult> => {
  const before = (): void => {
    Object.keys(config).forEach((key) => {
      if (Object.prototype.hasOwnProperty.call(config, key)) {
        getConfig(<Config>key);
      }
    });
  };

  return {
    before,
  };
};

/middlewares/check-config.ts

And implementing the middleware is simple enough:

import middy from '@middy/core';
import type { Config } from '@/constants';
import { verifyConfig } from '@/middleware/verify-config';

middy(handler)
    .use(verifyConfig(Config))

/middlewares/verify-config.ts

Now, should the app have any missing config - it will break immediately and no business related functions would have been invoked:

{
    "statusCode": 500,
    "message": "Configuration error: 'FILE_SIZE_POST_UPLOAD_LIMIT_MB' is required",
    "stack": "/Users/christian/Documents/GitHub/serverless-image-transformation/src/utils/env.ts:16\n      throw createError(500, message);\n            ^\n\nInternalServerError: Configuration error: 'FILE_SIZE_POST_UPLOAD_LIMIT_MB' is required\n    at getConfig (/Users/christian/Documents/GitHub/serverless-image-transformation/src/utils/env.ts:16:13)\n    at before (/Users/christian/Documents/GitHub/serverless-image-transformation/src/middleware/verify-config.ts:12:7)\n    at runMiddlewares (/Users/christian/Documents/GitHub/serverless-image-transformation/node_modules/@middy/core/index.js:118:27)\n    at runRequest (/Users/christian/Documents/GitHub/serverless-image-transformation/node_modules/@middy/core/index.js:78:9)\n    at async MessagePort.<anonymous> (file:///Users/christian/Documents/GitHub/serverless-image-transformation/node_modules/serverless-offline/src/lambda/handler-runner/worker-thread-runner/workerThreadHelper.js:24:14)"
}

An error that makes sense!

Now we're cooking with gas 😉

Happy coding.