This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Technical Documentation

Database and application server technical documentation for developers of the software.

As with many other business management systems, the Muse Systems Business Management System (MuseBMS) is a large, enterprise class system trying to solve many different problems in business computing. In an ideal world each distinct business problem would be solved by a dedicated application built for purpose. Taking this siloed approach, however, would result in pushing the complexity out of individual applications into a still more complex set of integration layers which would have to harmonize data, timing, and enforce controls across all applications used by the business. And this is the crux of why large, complex, and often times unwieldy enterprise business systems are created: it is less complex and unwieldy to solve these problems together than separately. Ultimately all of the individual and seemingly independent functions of a business must work in harmony and coordination with each other for the business as a whole to be successful and maximally efficient. The software a business uses should facilitate the cross business unit flow of information and allow for coordination of business activities that wouldn’t otherwise be possible without the software.

None of this is to say that we shouldn’t attempt to bring as much order as possible to the problem of enterprise software development; if we are to create a workable and maintainable business management system our efforts in this regard are essential. Herein we’ll document the technical aspects of the MuseBMS, but also put forth a larger set of principles by which we’ll organize our application in an attempt to make the entire system as manageable as possible.

Foundational Technologies

The application is built using a few core, widely available open source technologies which include:

  • PostgreSQL

    MuseBMS principally stores application data using the PostgreSQL Object Relational Database Management System (ORDBMS). There are incidental uses of other kinds of data management mechanisms, but for the most part these are bundled modules in the application programming environment (Elixir/BEAM) and are used to facilitate systems operations more than core application features.

  • Elixir

    The application server is written using the Elixir programming language. Elixir is a language which runs on the Erlang BEAM Virtual Machine, “known for creating low-latency, distributed, and fault-tolerant systems.

  • Phoenix Framework

    The Phoenix Framework is used to create complex, interactive web based user interfaces and APIs using Elixir.

  • Hugo & The Docsy Theme for Hugo

    This documentation is created using the Hugo static web site generator and the Docsy theme for Hugo. Our Elixir API documentation is create using Elixir’s ExDoc library.

Naturally there are many other libraries and technologies involved in supporting more specific or niche roles, but the three listed above are foundational technologies used in the MuseBMS.

1 - High Level Architecture

This section describes the highest level concepts used to organize the software.

Foundational Ideas

The MuseBMS is built somewhat differently than most Elixir/Phoenix based applications. In the typical Elixir/Phoenix development model, the application is built as a monolith with the database schema, business logic, external APIs, and user interfaces all developed within a single Phoenix application. Naturally in the better examples of these kinds of applications there are boundaries and separations of concerns, but these are primarily matters of developer practice rather than strict, technically enforced boundaries. The MuseBMS does not follow this typical model.

The MuseBMS is structured as many small Elixir Projects which are dedicated to narrowly defined and specific scopes of functionality. In many ways, these small Elixir Projects look and behave like typical Elixir libraries. They are independently testable, they offer well defined APIs, and they could be reused in different business system scenarios. This model of development was inspired by Dave Thomas’s ideas and work.

In terms of trade-offs between the conventional Elixir/Phoenix model and the model we adopt here, our sense is that we gain:

  • Clearer, technically enforced boundaries between different features and functions including a strict isolation of internal logic.

  • Greater chance of building re-usable components for use in other projects we may undertake.

  • Increased safety during refactoring or maintenance related efforts.

  • Greater ease in swapping out feature/functionality implementations with new implementations.

However we buy those gains with the cost of:

  • Increased boiler-plate code and repetitive configuration to maintain since each component is an Elixir project.

  • Increased time identifying the correct feature/function boundaries.

  • Increased indirection as different layers of the system expose or re-expose lower level Component functionality.

  • Time lost explaining our unique project organization to Elixir developers otherwise well versed in more conventional Elixir development.

In the end, we obviously consider the gains of this unconventional approach greater than the costs, at least for the MuseBMS. For some costs such as increased boiler-plate code, there are automated solutions which can mitigate the negative impacts. While increased time and consideration of identifying correct boundaries is listed as a negative, in many ways the forcing of that practice also is a positive since it discourages deferring some more difficult questions which could be more costly to address later in the process.

The MuseBMS Component Model

The MuseBMS software is organized into three different levels of abstractions:

Components

These are Elixir Projects which are base level implementations of basic functionality out of which higher level parts of the application are constructed. Some Components are simpler (lower level Components) and some Components can be more complex (higher level Components), even depending on other lower level Components. In all cases the key is that all the features implemented by a single component are all closely related to a single idea at the core of the component: these are the building blocks out which our other levels of abstraction are built.

Typically, Components should be used to encapsulate logic or define services which can then be used by higher level abstractions. For example a Component may define a GenServer or a Supervision Tree, but the Component should not itself instantiate those pieces as running processes; consumers of the Component, typically Subsystems, should be responsible for managing those runtime concerns.

Subsystems

At this level of implementation we are combining many of our Components together into a complete implementation of what we’d call an “application” (as opposed to an Elixir Project) with the exception of user interfaces. The stitching together of Components into complete, and possibly complex, chains of business logic operations is our concern here. At this level we’re expecting to cross the boundaries of single functional ideas and instead are interested in building the most efficient end-to-end business operations.

As with Components, Subsystems are implemented as independent Elixir Projects including their own test suites, documentation, and well defined APIs. Unlike Components, Subsystems are only allowed to depend on Components; they should never depend on other Subsystems and in most cases Subsystems should not depend on third party libraries either as doing so would suggest that there is a missing Component. The expectation is there will be many fewer Subsystems created than Components.

Finally this is the level at which most runtime concerns will exist. A Subsystem will typically be responsible for starting and supervising any required services defined by its dependencies. Exceptions to this rule chiefly deal with web and API interfaces provided by the Phoenix framework and managed at the Platform level.

Platforms

This is the implementation layer in which we use the Phoenix Framework to deliver web interfaces and external APIs to users of the application in addition to hosting the runtime services of the various hosted applications. The Platform depends on our different application Subsystems to incorporate core business logic, but the platform provides the logic responsible to connecting the web interface/API to the business logic.

The Platform level is designed to host multiple distinct user applications. Currently there are two applications expected to be hosted on the platform: 1) a central application responsible for administration of the platform as a whole including the creation and maintenance of instances of the other applications available in the Platform and authentication; 2) the MuseBMS application itself. It is conceivable that other applications could also be supported alongside MuseBMS, such as specialized versions of the MuseBMS dedicated to specific industrial verticals.

There are substantial third party dependencies due to the nature of Phoenix Framework, but of our logic we should only be depending on the created Subsystems as dependencies; note there are a couple exceptions to this rule, for example the web form components are implemented as a Component level Elixir/OTP Application (MscmpSystForms) and that Component is directly depended on by the Platform.

It is expected that there will only be one Platform level Elixir Application, though that Application may have different release profiles depending on which Subsystems are to be released to the final user.

Database Handling and the Component Model

Another of our departures from typical Elixir/Phoenix related software development is that we do not use Ecto migrations for database deployment, but rather use our own database management Component (MscmpSystDb) for managing and deploying the database related code.

As it relates to our levels of abstraction, the Component level is where the database schemas for persistence and other required database support is defined. Migrations are built and deployed at the Subsystem level; all of the database sources of the Subsystem’s dependencies are incorporated into the migrations of the Subsystem and at runtime the Subsystem will ensure that any unapplied migrations required by that Subsystem are applied as needed.

Currently, the database Component’s migration builder expects all of the database source code to live in a dedicated database source tree. Each Component that defines database code will therefore not only have an Elixir Project defining its application code, but will also have a corresponding directory in the database source tree as well. The details of source code organization will be addressed elsewhere.

2 - Source Code Organization

A discussion on how the MuseBMS source code is organized.

Directory Structure

The MuseBMS source and documentation are managed using a single monorepo. The project is subdivided into various conceptually significant directories.

.
├── app_server
│   ├── components
│   │   ├── application
│   │   └── system
│   │       ├── mscmp_syst_authn
│   │       ├── mscmp_syst_db
│   │       ├── mscmp_syst_enums
│   │       ├── mscmp_syst_error
│   │       ├── mscmp_syst_forms
│   │       ├── mscmp_syst_instance
│   │       ├── mscmp_syst_limiter
│   │       ├── mscmp_syst_mcp_perms
│   │       ├── mscmp_syst_network
│   │       ├── mscmp_syst_options
│   │       ├── mscmp_syst_perms
│   │       ├── mscmp_syst_session
│   │       ├── mscmp_syst_settings
│   │       └── mscmp_syst_utils
│   ├── platform
│   │   └── msplatform
│   └── subsystems
│       ├── mssub_bms
│       └── mssub_mcp
├── database
│   ├── all
│   ├── components
│   │   ├── application
│   │   │   ├── mscmp_acc_calendar
│   │   │   ├── mscmp_brm_contact
│   │   │   ├── mscmp_brm_country
│   │   │   ├── mscmp_brm_entity
│   │   │   ├── mscmp_brm_entity_bank
│   │   │   ├── mscmp_brm_entity_facility
│   │   │   ├── mscmp_brm_entity_inventory
│   │   │   ├── mscmp_brm_entity_person
│   │   │   ├── mscmp_brm_entity_purch
│   │   │   ├── mscmp_brm_entity_selling
│   │   │   ├── mscmp_brm_entity_staff
│   │   │   ├── mscmp_brm_person
│   │   │   ├── mscmp_brm_person_contact
│   │   │   ├── mscmp_brm_place
│   │   │   └── mscmp_syst_interactions
│   │   └── system
│   │       ├── mscmp_syst_authn
│   │       ├── mscmp_syst_docnum
│   │       ├── mscmp_syst_enums
│   │       ├── mscmp_syst_formats
│   │       ├── mscmp_syst_instance
│   │       ├── mscmp_syst_mcp_perms
│   │       ├── mscmp_syst_perms
│   │       ├── mscmp_syst_session
│   │       └── mscmp_syst_settings
│   └── subsystems
│       ├── mssub_bms
│       └── mssub_mcp
├── documentation
│   ├── book
│   └── technical
│       ├── app_server
│       │   ├── mscmp_syst_authn
│       │   ├── mscmp_syst_db
│       │   ├── mscmp_syst_enums
│       │   ├── mscmp_syst_error
│       │   ├── mscmp_syst_forms
│       │   ├── mscmp_syst_instance
│       │   ├── mscmp_syst_limiter
│       │   ├── mscmp_syst_mcp_perms
│       │   ├── mscmp_syst_network
│       │   ├── mscmp_syst_options
│       │   ├── mscmp_syst_perms
│       │   ├── mscmp_syst_session
│       │   ├── mscmp_syst_settings
│       │   ├── mscmp_syst_utils
│       │   └── mssub_mcp
│       └── database
│           ├── mssub_bms
│           └── mssub_mcp
└── tools
    ├── build_tools
    └── database_scripts

Top Level Directory Description

Here we describe the general organization of the top level directories which provide the fundamental organization of the MuseBMS source code.

app_server

The directory which contains the Elixir Projects and related source code. These are organized into “MuseBMS Component Model” related directories.

  • components/application

    Component Elixir Projects which define business domain related logic. For example a product pricing related Component would be an Application Component.

  • components/system

    Component Elixir Projects which define system support related logic. These kinds of components will address subjects such as database management or authentication.

  • platform

    The Platform Elixir/Phoenix Project which defines web and external API interfaces (user interfaces collectively) as well as provides the logic to translate user interface interactions to the application business logic.

  • subsystems

    The Subsystem related Elixir Projects which encapsulate business logic into complete applications, sans user interfaces.

database

This directory contains database source files which define the schema, functions and procedures, and any seed data which will be built into migrations by the MscmpSystDb migrator. Sub-directory structure generally follows the “MuseBMS Component Model”, with some exception. Finally, these directories will also contain data and functionality which are used in support of application unit and integration tests as the tests require.

  • all

    Common or generally applicable database schema, functions/procedures, and extensions which aren’t specific to any one Component or Subsystem.

  • components/application

    For each Elixir Project Application Component that defines database schema or functions/procedures, a corresponding database/components/application directory containing database source code will exist.

  • components/system

    For each Elixir Project System Component that defines database schema or functions/procedures, a corresponding database/components/system directory containing database source code will exist.

  • subsystems

    The database related source for the subsystems concerns itself chiefly with testing related support and migration building; the working database schema files are generally defined at the Component level even though migration building is a Subsystem level concern.

documentation

Contains the source of all the MuseBMS documentation. Different documentation authoring or generation tools may be used as appropriate for the specific subject matter of the documentation.

  • book

    The primary technical and end user documentation of the application built using the Hugo static site generator in conjunction with the Docsy documentation theme for Hugo. The document you are currently reading is part of this documentation.

    Ultimately this documentation is published independently via the Muse Systems website and is also included in the MuseBMS software release to support all online documentation scenarios.

  • technical/app_server

    This documentation is primarily aimed at documenting the APIs of the various Elixir components used throughout the application, though there is a fair amount of conceptual documentation when those components depend on a specific conceptual understand of the application logic.

    This documentation is generated from the Elixir source code commentary using Elixir’s ExDoc.

    This documentation is also imported into the book to support technical documentation references from documentation written there.

  • technical/database

    Database documentation is generated directly from the database using the SchemaSpy tool. Aside from introspecting the database schema, substantial amounts of database related explanatory documentation is drawn from database defined comments which are applied to the various database objects. This means that substantial database documentation is drawn directly from the database source code files.

    While the Component level Elixir code is fully documented, the database documentation is currently only generated at the Subsystem level. This avoids certain redundancies in documentation since many Components have common dependencies which would be included in the documentation for each dependent Component. The Subsystem is a complete expression of an application and is therefore best suited to express the complete set of data relationships the application will depend on.

    This documentation is imported into the book to support technical documentation references from the documentation written there.

tools

The tools directory is intended to contain useful tools for building, testing, and to generally aid in the development of the system.

3 - System Component Listing

Here we begin documenting the system in detail, moving away from purely conceptual discussions to examining more concrete implementation details.

Overview

The listing below shows the currently existing Elixir components which make up the application server (app_server), their dependencies, and their relationship with each other.

Each listed Component includes links to its application API documentation. If the Component depends on the database, a link to the Component specific database documentation and ERD is also included. Note that the inclusion of the database documentation at the Component level is not meant to imply that each Component requires its own database, but rather to ensure that only the relevant database documentation is presented in the Component context.

Platform

Msplatform

Msplatform is an Elixir/Phoenix umbrella project which provides the runtime environment for its hosted user applications and defines the user interfaces (web/external API). The umbrella itself exists to support these application/Subsystem runtime boundaries including the conditional inclusion/exclusion of various Subsystems in any given client specific build/release. The apps of the Msplatform umbrella are:

Subsystems

Component Documentation

These Components are listed in “Lower Level Component” to “Higher Level Component” order. Lower Level Components offer more simple, base level functionality whereas Higher Level Components will offer more complex functionality closer to the final business logic. Often times Higher Level Components will depend on Lower Level Components.

  • MscmpSystAuthn

    (API Docs) / (Database Docs & ERD)

    API for the management of user authentication.

    This Component provides a global method of authentication for users wishing to use the system.

  • MscmpSystDb

    (API Docs)

    A database management library for developing and managing database-per-tenant oriented systems. To achieve this we wrap and extend the popular Ecto and EctoSql libraries with a specialized templated migration system and add additional, opinionated abstractions encapsulating the tenant model as it relates to development, data access, and runtime concerns.

  • MscmpSystEnums

    (API Docs) / (Database Docs & ERD)

    A framework for user configurable ’list of values’ type functionality.

  • MscmpSystError

    (API Docs)

    This module defines a nested structure for reporting errors in contexts where a result should be represented by an error result. By capturing lower level errors and reporting them in a standard way, various application errors, especially non-fatal errors, can be handled as appropriate and logged for later analysis.

    • First Party Dependencies

      (none)

    • Third Party Dependencies

      (none)

  • MscmpSystForms

    (API Docs)

    The MscmpSystForms module provides a standard methodology for authoring application user interface forms in support of business management system development.

  • MscmpSystInstance

    (API Docs) / (Database Docs & ERD)

    “Instances” are instances of running application environments. Instances are established to host the application for different purposes, such as for running the application for production, training, and testing purposes; or as a means to implement multi-tenancy where each tenant application environment is an Instance.

    Each Instance also requires supporting data in order to facilitate runtime actions, such as defining database roles with which to access database data. Such supporting information is also managed via this component.

  • MscmpSystLimiter

    (API Docs)

    This component limits the rate at which targeted services can be called by any one caller to a level which preserves the availability of resources to all users of the system, or makes brute force information gathering prohibitively time intensive to would be attackers of the system.

  • MscmpSystMcpPerms

    (API Docs) / (Database Docs & ERD)

    Implements MscmpSystPerms related functionality for the MssubMcp subsystem.

  • MscmpSystNetwork

    (API Docs)

    Simple IP address handling and convenience functionality.

  • MscmpSystOptions

    (API Docs)

    API for retrieving and working with option files stored in the application server file system.

  • MscmpSystPerms

    (API Docs) / (Database Docs & ERD)

    Provides a generalized foundation for user permission system implementations.

    The principle idea of this component is to organization permissions in a way that higher level components can introduce the concept of user and establish contexts of applicability while keeping a cohesion in permissioning capability. To this end this component provides the core concepts for use in any permissioning system using this ecosystem of components.

  • MscmpSystSession

    (API Docs) / (Database Docs & ERD)

    Session Management API & Runtime

  • MscmpSystSettings

    (API Docs) / (Database Docs & ERD)

    The Settings Service provides caching and management functions for user configurable options which govern how the application operates. Multiple Settings Service instances may be in operation depending on the needs of the application; for example, in the case of multi-tenancy, each tenant will have its own instance of the Setting Service running since each tenant’s needs of the application may unique.

  • MscmpSystUtils

    (API Docs)

    Common utility functions generally useful across components.

    • First Party Dependencies

      (none)

    • Third Party Dependencies

      (none)

4 - Elixir Project Structure

Each of the application server code of the various Components, Subsystems, and Platforms making up the MuseBMS is implemented as independent Elixir projects, each of which is organized along lines which support our development philosophy. While our Elixir projects largely conform to the standard patterns used by most Elixir projects, there are nonetheless departures from convention which are worthy of discussion.

Central Motivation

As has been said, the directory organization is based on the standard Elixir project organization as created by running mix new myapp. Our Components are also started this way and then modified to match our requirements.

A core development principle for our Elixir related projects includes the idea of strictly separating out API definitions from the business logic of the Component. It follows that this separation of interface from logic impacts our project structure. Within the lib directory which typically holds Elixir source files and possibly other directories, we allow for three directories:

  • api - for Elixir source files defining the public API of the Component

  • impl - for Elixir source files implementing business logic concerns of the Component.

  • runtime - for Elixir source files implementing Component runtime concerns such as defining GenServers.

Under our structure, there are no source files directly homed in the lib directory itself.

Directory Structure and Common Files

Below is an example of a typical MuseBMS Elixir project followed by a detailed description. Depending on the specific features and scope of the Component not all directories or common files displayed below will be present.

.
├── README.md
├── config
├── database_utils
│   └── reset_dev_database.psql
├── dev_support
│   └── dev_support.ex
├── lib
│   ├── api
│   │   ├── msdata
│   │   ├── msform
│   │   ├── types
│   │   ├── types.ex
│   │   └── <module_name>.ex
│   ├── impl
│   │   ├── msdata
│   │   └── msform
│   └── runtime
├── mix.exs
├── priv
│   ├── database
│   └── plts
└── test
    ├── support
    └── test_helper.exs

Top Level Entry Descriptions

What follows are the important details regarding each of the organizational elements at the top level, with additional details as relevant.

README.md

The README.md files serves the typical purpose of describing the Component’s purpose and basics about usage. An important extra role for the README.md is that it is incorporated into the project’s technical documentation as the opening text.

Given this important opening role, the README.md file should contain a solid overview as well as an introduction to the Component at the conceptual and definitional levels.

config

The config directory is the standard Elixir project directory containing compile time related configurations. We do not use the config directory or its files in any non-standard way and actively avoid having our Components depend on these configuration files. There are exceptions, such as when third party libraries require configuration using this method, but even then our usage is in keeping with Elixir standard practices.

database_utils

This directory contains psql scripts, specifically written for the needs of this component, which are useful during development and testing. Only Components which have a database requirement will have this directory.

  • reset_dev_database.psql

    This script is run at the end of mix test execution to reset the testing database back to a clean state. It is specific to the Component where it is defined, though this script will be similar across all components where found.

dev_support

The dev_support directory contains Elixir code which is use to aid development activities. For the most part, this involves building migrations and deploying them to a development database server. Other dependencies which require services to be started may also be started via code sourced from this directory.

  • dev_support.ex

    Typically this is the only file in this directory. While that’s not a hard and fast rule that future Components will follow, there are few needs for development runtime support and so a single module makes sense.

lib

This is the traditional home of Elixir application source code and this doesn’t change for this project. As previously discussed, where we depart from convention is in how the source code inside this directory is organized.

  • api

    All source files in this directory should be focused on defining and documenting the APIs used to interact with the Component. Business logic implementation should not, as a practice, be included in the API defining source files.

    Files in api will typically consist of defdelegate calls, struct definitions, and type definitions, all of which should be documented.

    Some of the favored practices just discussed are possible sources of performance degradation. The first choice is to follow the standard established here and assume performance penalties are inconsequential. If performance issues arise, then we break ti standard documented here with a focus on preserving the API, but moving implementation closer to the API defining functions. We don’t expect material performance impacts in all but a few edge cases.

    • msdata

      This is special sub-directory containing the definitions of Ecto.Schema structs which establish database table representations. In addition there will be defdelegate references to Ecto.Changeset processing functions. Components which do not define database schema will not have an api/msdata directory.

      The structs created in msdata are namespaced directly to the Msdata namespace, so for example a users table may be in a struct module named Msdata.Users. Naturally, this means that all Ecto Schemas in msdata must be uniquely named across all Components in the project.

    • msform

      This is special sub-directory containing the source files which define MscmpSystForms implementing modules. More information about the organization of the api/msforms directory can be found in the MscmpSystForms Developing Forms documentation.

      This directory will only exist for Components implementing MscmpSystForms based user interfaces.

    • types

      The types directory hosts source files which principally define public structs which are usable outside of the Component and which aren’t defined in a more dedicated part of the api directory hierarchy (e.g. msdata database related structs). Public struct source files will include the struct definition and defdelegate calls to impl or runtime hosted function implementations. This directory will not exist if there are no module-level type definitions.

    • types.ex

      This source file holds typespecs for simpler types which don’t require a full module to be created. Again, these types should be documented and considered useful outside of the Component. Typically this fill will list all its defined types in alphabetical order, though other organization is acceptable if there are enough defined types to make more topical organization useful.

    • <module name>.ex

      A source file containing the Component’s principle API definition and API documentation. This file carries the same name as the Elixir project and will front the majority of the Component’s functionality on offer to the outside world. In a standard Elixir project, a source file with roughly the same purpose and name would be found directly in lib; we simply move that file to api and more strictly define its purpose.

  • impl

    This directory provides a home for the application’s business logic implementation. The source files in this directory will contain the vast majority of code, though there should be no ExDoc related doc tags in these files aside from @moduledoc false. Comments describing feature/function intention are allowed and, indeed, welcomed.

    • msdata

      This is the implementation side of previously discussed api/msdata directory. If a data struct requires business logic implementation, such as the definition of Ecto.Changeset processing or protocol implementation, a sub-directory will be created here with the same name as the corresponding struct in api/msdata. Other, more general changeset validation or helper modules which implement data related logic may host their source files directly in the root of impl/msdata. Standard names for these sorts of files include the following:

      • impl/msdata/helpers.ex

        Helper functions which work across multiple structs’ logic.

      • impl/msdata/general_validators.ex

        Validation and Changeset processing functions which are reusable across multiple data defining structs in api/msdata.

      • impl/msdata/<data struct name>/validators.ex

        Source files hosting Changeset processing functions which are delegated to from struct modules defined in api/msdata.

      • impl/msdata/<data struct name>/protocol.ex

        Source files implementing various protocols for a given struct as needed.

    • msform

      This is the implementation oriented compliment to the api/msform directory. For more complete documentation on the use of this directory please see the MscmpSystForms Developing Forms documentation.

  • runtime

    Source files which define and implement runtime services, such as GenServers. This directory will only exist if the Component defines such runtime services. Note that code in runtime source files is limited to the runtime concerns. Business logic implementations continue to be hosted in the impl directory structure and referenced from the runtime service as necessary.

mix.exs

This is the typical Elixir mix.exs defining project file. There are some organizational differences between our approach and the typical generated Elixir mix.exs, but there are no differences that go further than trivial stylist differences.

priv

This is the standard Elixir priv directory, though there are a few special sub-directories to be aware of.

  • priv/database

    In Component level Elixir projects, this directory may be visible and hosts database migrations built during the testing process. When tests are not actually running, the directory will exist but will be empty as the testing cleanup processes will typically delete this database related testing artifacts.

    At the Subsystem level the priv/database directory will host sub-directories named after their corresponding database type name (see the MscmpSystDb documentation for more about database types). The database migrations built for the database type will be hosted by the appropriate sub-directory. Note that these database migration files are persistent and drive the release as opposed to the transient nature of Component level migrations.

  • priv/plts

    This directory hosts generated PLT files for Dialyzer analysis during development and the Continuous Integration process. There is some historical precedence for doing handling PLTs and we’re just following this community convention.

    PLT files should not be part of releases and so this directory should not be a factor outside of development and testing processes.

test

This is the standard Elixir project testing directory. We follow standard conventions for the most part in regard to testing. We do define two different kinds of testing:

  • Unit Testing

    Unit tests are aimed at testing non-private impl and runtime related code and functions at a granular level. We are not confirming the public API in unit testing, but rather the internal implementations for correct results and typing.

  • Integration Testing

    Integration testing works to test the public API of the Component in a fashion which mimics the end-to-end business process as much as possible, including the Component’s interaction with any dependencies.

These two testing types will not be run together and will typically use different testing database seed data (if database interactions are part of the component’s functionality.)

Sub-directories and files to the test directory include:

  • support

    For testing support related files. It is typical for this directory to host implementations of ExUnit.CaseTemplate and logic for building, migrating, and seeding test databases, runtime services defined by dependencies and making good on other testing pre- conditions.

  • test_helper.exs

    This is the standard Elixir/ExUnit test_helper.exs file, typically modified to help in supporting testing related database operations and post-testing cleanup.

5 - Trigger Naming Conventions

Database trigger and trigger function naming conventions not only serve the purpose of identifying specific triggers easily, but also in ordering trigger execution at runtime.

Trigger Function Names

Trigger function names follow a convention which 1) establishes that the function is specifically a trigger function; 2) at what transaction time the trigger runs; 3) under which operations the trigger runs; and 4) a name relevant to the purpose of the trigger.

The form of the name is: trig_<b/a/i>_<i/u/d>_<purpose>

  1. trig_: the indication that this is a trigger returning function which should not be used outside of trigger contexts.

  2. <b/a/i>: an indicator of what phase of the transaction the trigger is expected to be fired. The available options are b for BEFORE triggers, a for AFTER triggers, and i for INSTEAD OF triggers. If the trigger function is designed to be called in more than one transaction phase, multiple indicators may be placed next to each other.

  3. <i/u/d>: establishes which operations are supported by the trigger function. The available options are i for INSERT, ‘u’ for UPDATE, and d for DELETE. If a trigger function is designed to support more than a single operation, the identifier of the supported operations may be placed next to each other.

  4. <purpose>: a descriptive name for the trigger function.

As an example, consider a trigger function designed to verify that a record is ‘consistent’ prior to allowing an inserted or updated record to be visible to the database. Such a trigger may be named something like:

  • trig_a_iu_validate_consistency

This would indicate that the trigger is expected to be an AFTER trigger which can be run during both INSERT and UPDATE operations.

Trigger Names

Trigger names follow a similar naming convention to the trigger function naming conventions discussed above. In fact, trigger names should be closely based on the trigger function name the trigger calls. The first difference in naming is that an execution order setting prefix is added prior to the trig_ opening. The next difference is in the way the previously defined indicators in the name are used. The <b/a/i> and <i/u/d> indicators present in trigger function names indicate which runtime conditions the trigger function supports; these same indicators are used in trigger names as well, but in trigger names the indicators specify the specific firing conditions of the trigger and so may differ slightly from the supported indicators for the trigger function name.

Using our example trigger function name above, lets suppose that we want to create two separate triggers which have independent conditions for firing, but both call the same trigger function; perhaps the INSERT call fires the trigger function unconditionally, but the UPDATE call only fires if a specific WHEN condition is true. The two trigger names could be:

  • a50_trig_a_i_validate_consisteny for the INSERT trigger
  • a50_trig_a_u_validate_consistency for the UPDATE trigger.

Note that the i and u indicators are separate in the trigger names, but are next to each other in the trigger function name.

Trigger Name Prefix Conventions

Triggers are executed by the database in trigger name order according to the database collation. To facilitate the organization of our code and to allow the execution ordering to proceed in a deterministic way, we adopt the following trigger name prefixes.

  • a00_ - a99_ : These are early validations designed to stop later, possibly more expensive processing later in the trigger sequences. We shouldn’t be changing or setting data in this class of function, though we might look up data in other tables.

  • b00_ - b99_ : Data manipulation functions. These are the functions we want updating or setting values that are appropriate for the limits of database maintained business logic (data integrity/ consistency focused).

  • c00_ - c99_ : Late validation functions. These are the functions which validate data after the data manipulation phase. In reality most data validation should probably be here since the “b” class triggers can take a starting invalid record and make it valid.

  • z00_ - z99_ : Utility/Auxiliary processing functions. This class of trigger processes records/data/etc that aren’t directly related to the business data in the record. For example, the diagnostic columns such as last updated or updated by or functions that record data changes to an audit table are the kinds of functions which should be in the “z” class. These functions should not make business validation judgements or take transformative actions on business data. These functions are really operational in nature and work on records which, from a business data perspective, are in their final form.

6 - Build Processes

Overview of building various pieces of our application including application releases and documentation.

There are several project artifacts which are subject to a build process prior to their being useful in the project. Chiefly these are the Msplatform application and various documentation pieces.

6.1 - Msplatform Build Process / Linux

Instructions for building the Msplatform Elixir application as a release. This process targets Linux x86_64 systems.

When building Msplatform we assume that we are including ERTS as part of the release.

We largely follow the standard Phoenix “Deploying with Releases” documentation to build a release, modified somewhat to support both the umbrella project structure and our own needs. this process looks like:

mkdir msplatform_linux_release
cd msplatform_linux_release
git clone https://github.com/MuseSystems/musebms.git --branch versions
cd musebms/app_server/platform/msplatform
mix deps.get --only prod
MIX_ENV=prod mix compile
cd apps/msapp_mcp_web
MIX_ENV=prod mix assets.deploy
MIX_ENV=prod mix phx.gen.release
cd ../../
MIX_ENV=prod mix release
cd _build/prod/rel
tar -cvjSf ../../../../../../../msplatform_release.tar.bz2 msplatform
cd ../../../../../../../

7 - Development Testing Standards

As development proceeds, the need to test code under a variety of conditions exists. This document establishes certain minimum standards for development related testing.

Testing Concerns

Currently there are two broad testing concerns which mirror the highest level structure of our application.

Application Server Testing

This is testing of the Elixir application server code. While this testing concern is not meant to be an explicit testing of database code (schema, constraints, store procedures, etc.) the database code will nonetheless be exercised by application.

Application server testing current is focused on using the Elixir ExUnit test suit with workflows prescribed in this document to attain our application development testing goals.

Database Testing

Database testing is focused on testing the database explicitly outside of the context of the application related code.

When direct database testing is incorporated into the test suite, we’ll like base such testing on PgTap.

Testing Types

Each of our testing areas of concern are subject to three kinds of testing.

Unit Testing

Unit testing targets the intended to confirm the functioning of any internal/private API functions defined by a Component. Unit tests have the following qualities:

  • Testing Target

    Internal, private APIs

  • Ordering

    Random & asynchronous

  • Data

    Expects the database to be sufficiently seeded such that any private API function can execute normally without interfering with other, possibly concurrently running tests.

    Ideally, the seeded test data is randomly generated, though this is not commonly done at present.

Integration Testing

The focus of Integration Testing is twofold: 1) test that public API functions are working correctly; and 2) establish that the supported business process can be operated successfully and completely using the public API. Traits of integration testing are:

  • Testing Target

    Public APIs & end-to-end business processing adequacy

  • Ordering

    Business processed ordered & synchronous

  • Data

    Pre-testing database seeding is as minimal as possible with later tests relying on data created during earlier tests.

    Ideally, test data is generated randomly though this is not consistently done at present.

Documentation Testing

Documentation Testing, where supported by the test suites, is intended to ensure that any documented code examples are representations of working code. The qualities of documentation testing are:

  • Testing Target

    Public API documented examples

  • Ordering

    Random & asynchronous

  • Data

    Expects the database to be sufficiently seeded such that any public API function can execute normally without interfering with other, possibly concurrently running tests.

    The seeded data must be well defined and correspond directly to the data seen in the documented code examples.

Testing an the Component Model

The Component Model establishes a hierarchy of application concerns which influences the expectations of our testing regime.

Component Level Testing

Each Component is responsible for defining its own unit, integration, and documentation testing suite. Passing the complete test suite is a pre-requisite for inclusion in any release oriented code branch of the larger application.

If a Component has dependencies, it may assume that those dependencies have defined and passed their own test suites. If they require data seeding in order to support testing of the current Component, the typical process should be to simply load the appropriate testing data from the dependency’s test suite as a prerequisite to loading the current Component’s seed data. This does mean that across Components some effort for cross-Component compatibility of data seeding should be considered.

Subsystem Level Testing

Subsystems are mostly aggregates of Components and their testing will typically reflect this. While unit tests and documentation tests will exist for Subsystems, these testing concerns will not differ from Component level testing; indeed, just as Components will assume their dependencies are independently tested prior to being available as a dependency, the dependencies of the Subsystem are similarly treated.

Integration testing for Subsystems, however, is much more important and extensive. The reason for this is that Subsystems must ensure that their combinations of large numbers of dependencies (Components) must all work well enough together to form a coherent application. Therefore, end-to-end business process testing is key to ensuring application level degrees of interoperability. This is the full expression of the business logic of the application and testing that expression is critical.

Seeding data for Subsystem integration testing is usually restricted to the data seeded as part of a new system implementation, with no specifically testing related data being seeded. All testing data is created using the Public APIs of the application with the data from earlier tests providing inputs to later tests.

Unit and documentation test data, insofar as it is required, is typically seeded from the unit and documentation test seed data from each of the dependencies prior to the tests being run; naturally if there are Subsystem specific requirements for seeding test data the Subsystem’s test suite can define and load any additional seed data as needed.

Platform Level Testing

Testing the Platform related code is focused on Integration testing similar to the testing focus of Subsystems. The key difference is that the integration focus is less concerned with testing end-to-end business logic as it is with integration testing of the system’s external “user interfaces”; user interfaces as used here is broadly interpreted as any interface to external users. Examples of user interfaces are “web application”, “External, Public API”, or any other sanctioned means by which an external user may interact with the running Elixir applications.

The data sources and seed data concerns of the Subsystem level of testing are broadly applicable to Platform Level Testing.