dmno.dev sneak peek #1 ✨

dmno-config-schema-features.md

DMNO config engine example

👋 Hello there!

This is a little sneak peek of what we're working on over at DMNO / 👂 "domino"

Our first release will be an open source tool called the “dmno config engine” that lets you define a schema for all of the config (env vars) in your entire stack. If you've done any kind of development on a team or of larger systems, you've certainly struggled with managing secrets and configuration. It may not be your #1 pain point, but I guarantee it's something that comes up on every project, over and over again. And even when you've hacked together a system that you're relatively happy with, it involves a ton of custom glue to hold it all together and you can probably imagine something better.

We're hoping to solve that once and for all. We think this could become the default way that we all configure our javascript/typescript apps. We actually think this could just be the way we deal with config for all kinds of apps for all kinds of languages and systems, but we’ll settle for taking over the JS world first…

So what are the benefits of a system like this?

• single centralized system for all of your config for your entire stack with the ability to share config across services
• built-in documentation for your teammates (and your future self)
• type-safety and intellisense/auto-completion when using your config throughout your code
• validation of your config (fail fast and early) with helpful error messages instead of unexpected crashes
• reduce "secret sprawl" and the need to configure many environments on many platforms, and keep those values in sync
• use the same system for all of your configuration, secret and non-secret (while handling your secrets securely)
• understand (and visualize) the way all your configuration is connected (hello DAG!)
• securely sync secrets to various backends (ex: 1password, AWS secrets manager) or commit them securely (encrypted) into your repo
• understand the full shape of how your config changes in various environments, without needing access to the actual values

While this system is particularly useful in the context of a mono-repo or multiple related services, we think there is a ton of value using it even for a single isolated service. With minimal work, we want to give you a developer experience that is an order of magnitude better than the status quo - and we think we’re on the right track.

The code below is all a bit technical and it's really meant to be a showcase of all the various features and flexibility of how you can define a "config schema" with our tool. It's meant to show you the rough shape of how it works and what is possible, not to provide a real-world use-case or help you understand the why, (which you likely know first-hand already).

If you like what you see, have any comments or feedback, want to be a beta tester, or just want to chat, please send us a message! (theo at dmno dot dev)

And don’t forget to sign up for our mailing list at https://dmno.dev and we’ll keep you in the loop 💌

schema-features-example.mts

import {
  defineConfigSchema, DmnoBaseTypes, configPath, dmnoFormula,switchByNodeEnv,
  createDmnoDataType, injectPlugin, ValidationError, EncryptedFileStorePlugin
} from '@dmno/core';
import { OnePasswordDmnoPlugin } from '@dmno/1password-plugin';

// plugins can be used to create reusable functionality and can reference config items in their initialization
const encryptedSecrets = new EncryptedFileStorePlugin({ name: 'local-secrets', key: configPath('LOCAL_SECRETS_KEY') });

// pre-configured plugins can be auto-injected from those that were initialized in the workspace root
// just by type if there is only one instance, or with an aditional instance name if needed
const onePassSync = injectPlugin(OnePasswordDmnoPlugin);
// const onePassProdVault = injectPlugin('prod-vault', OnePasswordDmnoPlugin); // example with a name

export default defineConfigSchema({
  // each service can be explicitly named or will default to the name from its package.json
  name: 'api', 
  
  // explicitly set a parent service to nest them, otherwise everything is a child of the "root" workspace
  // this affects the dependency graph of services and it affects "picking" logic (see below)
  parent: 'group1',

  // config items can be "picked" from other services (in every service except the root)
  // while picking from an ancestor, you can pick from _all_ config items in that service
  // otherwise you can only pick items that have been marked as `expose: true`
  pick: [
    // you can specify the source service name and key(s) to pick
    {
      source: 'root',
      key: 'SINGLE_KEY',
    },

    // if source is omitted, it will fallback to the workspace root
    { key: 'OTHER_KEY_FROM_ROOT' },

    // shorthand to pick single key from root
    'SHORTHAND_PICK_FROM_ROOT',
    
    // you can pick multiple keys at once
    {
      source: 'other-service',
      key: ['MULTIPLE', 'KEYS'],
    },
    
    // you can pick by filtering keys with a function
    // (filters from all items or just exposed items depending if the source is an ancestor)
    {
      source: 'root',
      key: (key) => key.startsWith('DB_'),
    },

    // keys can be transformed, and you can use a static value if picking a single key
    {
      key: 'ORIGINAL_KEY',
      renameKey: 'NEW_KEY_NAME',
    },

    // or use a function if picking multiple
    {
      key: ['KEY1', 'KEY2'],
      renameKey: (k) => `PREFIX_${k}`,
    },

    // values can also be transformed with functions
    {
      key: 'GROUP1_THINGY',
      transformValue: (v) => v + 1,
    },
  ],
  
  // services also define more config items relevant to only themselves and to be picked by others
  schema: {

    // SETTING ITEM TYPE  -----------------------------------------------------------------
    // the default method, where a datatype is called as a function with some settings
    EXTENDS_TYPE_INITIALIZED: {
      extends: DmnoBaseTypes.number({ min: 0, max: 100 })
    },
    // you can use a type that has not been initialized if no settings are needed
    EXTENDS_TYPE_UNINITIALIZED: {
      extends: DmnoBaseTypes.number
    },
    // string/named format works for some basic types (string, number, boolean, etc) with no settings
    EXTENDS_STRING: {
      extends: 'number'
    }, 
    // passing nothing will try to infer the type from a static value or fallback to a string otherwise
    DEFAULTS_TO_NUMBER: { value: 42 },       // infers number
    DEFAULTS_TO_STRING: { value: 'cool' },   // infers string
    FALLBACK_TO_STRING_NO_INFO: { },         // assumes string
    FALLBACK_TO_STRING_UNABLE_TO_INFER: {    // assumes string
      value: onePassSync.item('secret-id-12345'),  
    },

    // an additional shorthand is provided for config items with no settings other than extends/type
    // (although not recommended because attaching additional metadata/info is helpful)
    SHORTHAND_TYPE_NAME: 'number',
    SHORTHAND_TYPE_UNINITIALIZED: DmnoBaseTypes.number,
    SHORTHAND_TYPE_INITIALIZED: DmnoBaseTypes.number({ min: 100 }),
    
    // and of course you can use custom types (see below), which can in turn extend other types
    USE_CUSTOM_TYPE: {
      extends: MyCustomPostgresConnectionUrlType, 
      // additional settings can be added/overridden as normal
      required: true,
    },


    // SETTING VALUES -----------------------------------------------------------------
    // config items can specify how to set their value within their schema
    // so you can set sensible defaults, or even set all possible values and sync secrets securely with various backends
    // overrides from .env file(s) and actual environment variables will also be applied
    // and then coercion/validation logic will be run on the resolved value

    // values can be set to a static value - useful for constants and settings that will be overridden by env vars
    STATIC_VAL: {
      value: 'static'
    },
    // or use a function that takes a ctx object that has other config item values available
    FN_VAL: {
      value: (ctx) => `prefix_${ctx.get('OTHER_ITEM')}`
    },
    // a simple formula DSL is provided which handles common cases without needing to write a function at all
    SET_BY_FORMULA2: {
      value: dmnoFormula('prefix_{{ OTHER_ITEM }}'),
    },
    // or synced with a secure backend using a plugin
    SECRET_EXAMPLE: {
      value: onePassSync.itemByReference("op://dev test/example/username"),
    },

    // or switched based on another value (often an env flag, but not always)
    // and a "value resolver" can always return another resolver, which lets you easily compose functionality 
    // NOTE - it's easy to author your own reusable resolvers to create whatever functionality you need
    SWITCHED_ENV_EXAMPLE: {
      value: switchByNodeEnv({
        _default: 'default-value',
        staging: (ctx) => `${ctx.get('NODE_ENV')}-value`,
        production: onePassSync.item("asdf1234zxcv6789"),
      }),
    },

    // COMPLEX TYPES (object, arrays, maps) //////////////////////////
    OBJECT_EXAMPLE: {
      extends: DmnoBaseTypes.object({
        CHILD1: { },
        CHILD2: { },
      }),
    },

    ARRAY_EXAMPLE: {
      extends: DmnoBaseTypes.array({
        itemSchema: {
          extends: 'number'
        },
        minLength: 2,
      }),
    },

    DICTIONARY_EXAMPLE: {
      extends: DmnoBaseTypes.dictionary({
        itemSchema: {
          extends: 'number'
        },
        validateKeys: (key) => key.length === 2,
      }),
    },

    // VALIDATIONS + COERCION /////////////////////////////////////////////////
    
    // most validation logic will likely be handled by helpers on reusable types
    // but sometimes you may need something more custom
    // it will run _after_ the type (extends) defined validation(s)
    VALIDATE_EXAMPLE: {
      extends: DmnoBaseTypes.string({ isLength: 128, startsWith: 'pk_' }),
      validate(val, ctx) {
        // validations can use the ctx to access other values
        if (ctx.get('NODE_ENV') === 'production') {
          if (!val.startsWith('pk_live_')) {
            // throw a ValidationError with a meaningful message
            throw new ValidationError('production key must start with "pk_live_"');
          }
        }
        return true;
      }
    },

    // async validations can be used if a validation needs to make async calls
    // NOTE - these will be triggered on-demand rather than run constantly like regular validations
    ASYNC_VALIDATE_EXAMPLE: {
      asyncValidate: async (val, ctx) => {
        // if the request succeeds, we know the value was ok
        await fetch(`https://example.com/api/items/${val}`);
        return true;
      }
    },


    // MORE SETTINGS //////////////////////////////////////////////
    KITCHEN_SINK: {
      // some basic info will help within the UI and be included in generated ts types
      // as well as help other devs understand what this env var is for :)
      summary: 'short label',
      description: 'longer description can go here',
      // mark an item as required so it will fail validation if empty
      required: true,
      // mark an item as secret so we know it must be handled sensitively!
      // for example, it will not be logged or injected into front-end builds
      secret: true,
      // understand when this value is used, which lets us parallelize run/deploy
      // and know when a missing item should be considered a critical problem or be ignored
      useAt: ['build', 'boot', 'deploy'],
      // mark an item as being "exposed" for picking by other services
      expose: true,
      // override name when importing/exporting into process.env
      importEnvKey: 'IMPORT_FROM_THIS_VAR',
      exportEnvKey: 'EXPORT_AS_THIS_VAR',
    },
  },
});

// our custom type system allows you to build your own reusable types
// or to take other plugin/community defined types and tweak them as necessary
// internally a chain of "extends" types is stored and settings are resolved by walking up the chain
const MyCustomPostgresConnectionUrlType = createDmnoDataType({
  // you can extend one of our base types or another custom type...
  extends: DmnoBaseTypes.url,

  // all normal config item settings are supported
  secret: true,

  // a few docs related settings made for reusable types (although they can still be set directly on items)
  // these will show up within the UI and generated types in various ways
  typeDescription: 'Postgres connection url',
  externalDocs: {
    description: 'explanation from prisma docs',
    url: 'https://www.prisma.io/dataguide/postgresql/short-guides/connection-uris#a-quick-overview'
  },
  ui: {
    // uses iconify names, see https://icones.js.org for options
    icon: 'akar-icons:postgresql-fill',
    color: '336791', // postgres brand color :)
  },

  // for validation/coercion, we walk up the chain and apply the functions from top to bottom
  // for example, given the following type chain:
  // - DmnoBaseTypes.string - makes sure the value is a string
  // - DmnoBaseTypes.url - makes sure that string looks like a URL
  // - PostgresConnectionUrlType - checks that url against some custom logic
  validate(val, ctx) {
    // TODO: check this url looks like a postgres connection url
  },
  // but you can alter the exection order, or disable the parent altogether
  runParentValidate: 'after', // set to `false` to disable running the parent's validate
});

example of the auto-generated types in action

example of validations failing