Technical Documentation

• 1: High Level Architecture
• 2: Source Code Organization
• 3: System Component Listing
• 4: Elixir Project Structure
• 5: Trigger Naming Conventions
• 6: Build Processes
• 6.1: Msplatform Build Process / Linux
• 7: Development Testing Standards

1 - High Level Architecture

Foundational Ideas

The MuseBMS is built somewhat differently than most Elixir/Phoenix based applications. In the typical Elixir/Phoenix development model, the application is built as a monolith with the database schema, business logic, external APIs, and user interfaces all developed within a single Phoenix application. Naturally in the better examples of these kinds of applications there are boundaries and separations of concerns, but these are primarily matters of developer practice rather than strict, technically enforced boundaries. The MuseBMS does not follow this typical model.

The MuseBMS is structured as many small Elixir Projects which are dedicated to narrowly defined and specific scopes of functionality. In many ways, these small Elixir Projects look and behave like typical Elixir libraries. They are independently testable, they offer well defined APIs, and they could be reused in different business system scenarios. This model of development was inspired by Dave Thomas’s ideas and work.

In terms of trade-offs between the conventional Elixir/Phoenix model and the model we adopt here, our sense is that we gain:

• Clearer, technically enforced boundaries between different features and functions including a strict isolation of internal logic.

• Greater chance of building re-usable components for use in other projects we may undertake.

• Increased safety during refactoring or maintenance related efforts.

• Greater ease in swapping out feature/functionality implementations with new implementations.

However we buy those gains with the cost of:

• Increased boiler-plate code and repetitive configuration to maintain since each component is an Elixir project.

• Increased time identifying the correct feature/function boundaries.

• Increased indirection as different layers of the system expose or re-expose lower level Component functionality.

• Time lost explaining our unique project organization to Elixir developers otherwise well versed in more conventional Elixir development.

In the end, we obviously consider the gains of this unconventional approach greater than the costs, at least for the MuseBMS. For some costs such as increased boiler-plate code, there are automated solutions which can mitigate the negative impacts. While increased time and consideration of identifying correct boundaries is listed as a negative, in many ways the forcing of that practice also is a positive since it discourages deferring some more difficult questions which could be more costly to address later in the process.

The MuseBMS Component Model

The MuseBMS software is organized into three different levels of abstractions:

Components

These are Elixir Projects which are base level implementations of basic functionality out of which higher level parts of the application are constructed. Some Components are simpler (lower level Components) and some Components can be more complex (higher level Components), even depending on other lower level Components. In all cases the key is that all the features implemented by a single component are all closely related to a single idea at the core of the component: these are the building blocks out which our other levels of abstraction are built.

Typically, Components should be used to encapsulate logic or define services which can then be used by higher level abstractions. For example a Component may define a GenServer or a Supervision Tree, but the Component should not itself instantiate those pieces as running processes; consumers of the Component, typically Subsystems, should be responsible for managing those runtime concerns.

Subsystems

At this level of implementation we are combining many of our Components together into a complete implementation of what we’d call an “application” (as opposed to an Elixir Project) with the exception of user interfaces. The stitching together of Components into complete, and possibly complex, chains of business logic operations is our concern here. At this level we’re expecting to cross the boundaries of single functional ideas and instead are interested in building the most efficient end-to-end business operations.

As with Components, Subsystems are implemented as independent Elixir Projects including their own test suites, documentation, and well defined APIs. Unlike Components, Subsystems are only allowed to depend on Components; they should never depend on other Subsystems and in most cases Subsystems should not depend on third party libraries either as doing so would suggest that there is a missing Component. The expectation is there will be many fewer Subsystems created than Components.

Finally this is the level at which most runtime concerns will exist. A Subsystem will typically be responsible for starting and supervising any required services defined by its dependencies. Exceptions to this rule chiefly deal with web and API interfaces provided by the Phoenix framework and managed at the Platform level.

Platforms

This is the implementation layer in which we use the Phoenix Framework to deliver web interfaces and external APIs to users of the application in addition to hosting the runtime services of the various hosted applications. The Platform depends on our different application Subsystems to incorporate core business logic, but the platform provides the logic responsible to connecting the web interface/API to the business logic.

The Platform level is designed to host multiple distinct user applications. Currently there are two applications expected to be hosted on the platform: 1) a central application responsible for administration of the platform as a whole including the creation and maintenance of instances of the other applications available in the Platform and authentication; 2) the MuseBMS application itself. It is conceivable that other applications could also be supported alongside MuseBMS, such as specialized versions of the MuseBMS dedicated to specific industrial verticals.

There are substantial third party dependencies due to the nature of Phoenix Framework, but of our logic we should only be depending on the created Subsystems as dependencies; note there are a couple exceptions to this rule, for example the web form components are implemented as a Component level Elixir/OTP Application (MscmpSystForms) and that Component is directly depended on by the Platform.

It is expected that there will only be one Platform level Elixir Application, though that Application may have different release profiles depending on which Subsystems are to be released to the final user.

Database Handling and the Component Model

Another of our departures from typical Elixir/Phoenix related software development is that we do not use Ecto migrations for database deployment, but rather use our own database management Component (MscmpSystDb) for managing and deploying the database related code.

As it relates to our levels of abstraction, the Component level is where the database schemas for persistence and other required database support is defined. Migrations are built and deployed at the Subsystem level; all of the database sources of the Subsystem’s dependencies are incorporated into the migrations of the Subsystem and at runtime the Subsystem will ensure that any unapplied migrations required by that Subsystem are applied as needed.

Currently, the database Component’s migration builder expects all of the database source code to live in a dedicated database source tree. Each Component that defines database code will therefore not only have an Elixir Project defining its application code, but will also have a corresponding directory in the database source tree as well. The details of source code organization will be addressed elsewhere.

2 - Source Code Organization

Directory Structure

The MuseBMS source and documentation are managed using a single monorepo. The project is subdivided into various conceptually significant directories.

.
├── app_server
│   ├── components
│   │   ├── application
│   │   └── system
│   │       ├── mscmp_syst_authn
│   │       ├── mscmp_syst_db
│   │       ├── mscmp_syst_enums
│   │       ├── mscmp_syst_error
│   │       ├── mscmp_syst_forms
│   │       ├── mscmp_syst_instance
│   │       ├── mscmp_syst_limiter
│   │       ├── mscmp_syst_mcp_perms
│   │       ├── mscmp_syst_network
│   │       ├── mscmp_syst_options
│   │       ├── mscmp_syst_perms
│   │       ├── mscmp_syst_session
│   │       ├── mscmp_syst_settings
│   │       └── mscmp_syst_utils
│   ├── platform
│   │   └── msplatform
│   └── subsystems
│       ├── mssub_bms
│       └── mssub_mcp
├── database
│   ├── all
│   ├── components
│   │   ├── application
│   │   │   ├── mscmp_acc_calendar
│   │   │   ├── mscmp_brm_contact
│   │   │   ├── mscmp_brm_country
│   │   │   ├── mscmp_brm_entity
│   │   │   ├── mscmp_brm_entity_bank
│   │   │   ├── mscmp_brm_entity_facility
│   │   │   ├── mscmp_brm_entity_inventory
│   │   │   ├── mscmp_brm_entity_person
│   │   │   ├── mscmp_brm_entity_purch
│   │   │   ├── mscmp_brm_entity_selling
│   │   │   ├── mscmp_brm_entity_staff
│   │   │   ├── mscmp_brm_person
│   │   │   ├── mscmp_brm_person_contact
│   │   │   ├── mscmp_brm_place
│   │   │   └── mscmp_syst_interactions
│   │   └── system
│   │       ├── mscmp_syst_authn
│   │       ├── mscmp_syst_docnum
│   │       ├── mscmp_syst_enums
│   │       ├── mscmp_syst_formats
│   │       ├── mscmp_syst_instance
│   │       ├── mscmp_syst_mcp_perms
│   │       ├── mscmp_syst_perms
│   │       ├── mscmp_syst_session
│   │       └── mscmp_syst_settings
│   └── subsystems
│       ├── mssub_bms
│       └── mssub_mcp
├── documentation
│   ├── book
│   └── technical
│       ├── app_server
│       │   ├── mscmp_syst_authn
│       │   ├── mscmp_syst_db
│       │   ├── mscmp_syst_enums
│       │   ├── mscmp_syst_error
│       │   ├── mscmp_syst_forms
│       │   ├── mscmp_syst_instance
│       │   ├── mscmp_syst_limiter
│       │   ├── mscmp_syst_mcp_perms
│       │   ├── mscmp_syst_network
│       │   ├── mscmp_syst_options
│       │   ├── mscmp_syst_perms
│       │   ├── mscmp_syst_session
│       │   ├── mscmp_syst_settings
│       │   ├── mscmp_syst_utils
│       │   └── mssub_mcp
│       └── database
│           ├── mssub_bms
│           └── mssub_mcp
└── tools
    ├── build_tools
    └── database_scripts

Top Level Directory Description

Here we describe the general organization of the top level directories which provide the fundamental organization of the MuseBMS source code.

app_server

The directory which contains the Elixir Projects and related source code. These are organized into “MuseBMS Component Model” related directories.

• components/application

  Component Elixir Projects which define business domain related logic. For example a product pricing related Component would be an Application Component.

• components/system

  Component Elixir Projects which define system support related logic. These kinds of components will address subjects such as database management or authentication.

• platform

  The Platform Elixir/Phoenix Project which defines web and external API interfaces (user interfaces collectively) as well as provides the logic to translate user interface interactions to the application business logic.

• subsystems

  The Subsystem related Elixir Projects which encapsulate business logic into complete applications, sans user interfaces.

database

This directory contains database source files which define the schema, functions and procedures, and any seed data which will be built into migrations by the MscmpSystDb migrator. Sub-directory structure generally follows the “MuseBMS Component Model”, with some exception. Finally, these directories will also contain data and functionality which are used in support of application unit and integration tests as the tests require.

• all

  Common or generally applicable database schema, functions/procedures, and extensions which aren’t specific to any one Component or Subsystem.

• components/application

  For each Elixir Project Application Component that defines database schema or functions/procedures, a corresponding database/components/application directory containing database source code will exist.

• components/system

  For each Elixir Project System Component that defines database schema or functions/procedures, a corresponding database/components/system directory containing database source code will exist.

• subsystems

  The database related source for the subsystems concerns itself chiefly with testing related support and migration building; the working database schema files are generally defined at the Component level even though migration building is a Subsystem level concern.

documentation

Contains the source of all the MuseBMS documentation. Different documentation authoring or generation tools may be used as appropriate for the specific subject matter of the documentation.

• book

  The primary technical and end user documentation of the application built using the Hugo static site generator in conjunction with the Docsy documentation theme for Hugo. The document you are currently reading is part of this documentation.

  Ultimately this documentation is published independently via the Muse Systems website and is also included in the MuseBMS software release to support all online documentation scenarios.

• technical/app_server

  This documentation is primarily aimed at documenting the APIs of the various Elixir components used throughout the application, though there is a fair amount of conceptual documentation when those components depend on a specific conceptual understand of the application logic.

  This documentation is generated from the Elixir source code commentary using Elixir’s ExDoc.

  This documentation is also imported into the book to support technical documentation references from documentation written there.

• technical/database

  Database documentation is generated directly from the database using the SchemaSpy tool. Aside from introspecting the database schema, substantial amounts of database related explanatory documentation is drawn from database defined comments which are applied to the various database objects. This means that substantial database documentation is drawn directly from the database source code files.

  While the Component level Elixir code is fully documented, the database documentation is currently only generated at the Subsystem level. This avoids certain redundancies in documentation since many Components have common dependencies which would be included in the documentation for each dependent Component. The Subsystem is a complete expression of an application and is therefore best suited to express the complete set of data relationships the application will depend on.

  This documentation is imported into the book to support technical documentation references from the documentation written there.

tools

The tools directory is intended to contain useful tools for building, testing, and to generally aid in the development of the system.

3 - System Component Listing

Overview

The listing below shows the currently existing Elixir components which make up the application server (app_server), their dependencies, and their relationship with each other.

Each listed Component includes links to its application API documentation. If the Component depends on the database, a link to the Component specific database documentation and ERD is also included. Note that the inclusion of the database documentation at the Component level is not meant to imply that each Component requires its own database, but rather to ensure that only the relevant database documentation is presented in the Component context.

Platform

Msplatform

Msplatform is an Elixir/Phoenix umbrella project which provides the runtime environment for its hosted user applications and defines the user interfaces (web/external API). The umbrella itself exists to support these application/Subsystem runtime boundaries including the conditional inclusion/exclusion of various Subsystems in any given client specific build/release. The apps of the Msplatform umbrella are:

• MsappMcp

  ^{(API Docs)}

  Provides the MCP “controller” level logic which responds to user input from the view layer and invokes logic from the MssubMcp Subsystem. MsappMcp also defines the abstract configuration of the various user interface forms according to form development standards set by the MscmpSystForms Component.

  • First Party Dependencies

    mscmp_syst_error, mscmp_syst_forms, mscmp_syst_network, mscmp_syst_options, mscmp_syst_perms, mssub_mcp

  • Third Party Dependencies

    finch, phoenix_pubsub, swoosh

• MsappMcpWeb

  ^{(API Docs)}

  Defines the Phoenix user interface “view” layer used by users of the Master Control Program (MCP) application. In addition, operational issues such as request routing and other typical Phoenix web related functions are handled here as well. This component invokes MsappMcp functionality in order to carry out user wishes.

  • First Party Dependencies

    mscmp_syst_forms, msapp_mcp

  • Third Party Dependencies

    gettext, jason, phoenix, phoenix_ecto, phoenix_html, phoenix_live_view, phoenix_live_dashboard, plug_cowboy telemetry_metrics, telemetry_poller,

Subsystems

• MssubBms

  ^{(API Docs) / (Database Docs & ERD)}

  A business logic subsystem implementing the Muse Systems Business Management Systems. An Msplatform application delivering a complete small to mid-sized business management solution.

  • First Party Dependencies

    mscmp_syst_enums, mscmp_syst_error, mscmp_syst_settings, mscmp_syst_utils

  • Third Party Dependencies

    (none)

• MssubMcp

  ^{(API Docs) / (Database Docs & ERD)}

  The “Master Control Program” subsystem which provides global user authentication and environment/tenant administration for all applications running on the MuseBMS platform.

  • First Party Dependencies

    mscmp_syst_authn, mscmp_syst_enums, mscmp_syst_error, mscmp_syst_instance, mscmp_syst_mcp_perms, mscmp_syst_options, mscmp_syst_perms, mscmp_syst_settings, mscmp_syst_session, mscmp_syst_utils

  • Third Party Dependencies

    (none)

Component Documentation

These Components are listed in “Lower Level Component” to “Higher Level Component” order. Lower Level Components offer more simple, base level functionality whereas Higher Level Components will offer more complex functionality closer to the final business logic. Often times Higher Level Components will depend on Lower Level Components.

• MscmpSystAuthn

  ^{(API Docs) / (Database Docs & ERD)}

  API for the management of user authentication.

  This Component provides a global method of authentication for users wishing to use the system.

  • First Party Dependencies

    mscmp_syst_db, mscmp_syst_enums, mscmp_syst_error, mscmp_syst_instance, mscmp_syst_limiter, mscmp_syst_network, mscmp_syst_options, mscmp_syst_utils

  • Third Party Dependencies

    argon2_elixir, jason, nimble_totp, pathex, timex

• MscmpSystDb

  ^{(API Docs)}

  A database management library for developing and managing database-per-tenant oriented systems. To achieve this we wrap and extend the popular Ecto and EctoSql libraries with a specialized templated migration system and add additional, opinionated abstractions encapsulating the tenant model as it relates to development, data access, and runtime concerns.

  • First Party Dependencies

    mscmp_syst_error, mscmp_syst_network, mscmp_syst_utils

  • Third Party Dependencies

    ecto, ecto_sql, jason, postgrex, toml

• MscmpSystEnums

  ^{(API Docs) / (Database Docs & ERD)}

  A framework for user configurable ’list of values’ type functionality.

  • First Party Dependencies

    mscmp_syst_db, mscmp_syst_error, mscmp_syst_utils

  • Third Party Dependencies

    (none)

• MscmpSystError

  ^{(API Docs)}

  This module defines a nested structure for reporting errors in contexts where a result should be represented by an error result. By capturing lower level errors and reporting them in a standard way, various application errors, especially non-fatal errors, can be handled as appropriate and logged for later analysis.

  • First Party Dependencies

    (none)

  • Third Party Dependencies

    (none)

• MscmpSystForms

  ^{(API Docs)}

  The MscmpSystForms module provides a standard methodology for authoring application user interface forms in support of business management system development.

  • First Party Dependencies

    mscmp_syst_error, mscmp_syst_perms, mscmp_syst_utils

  • Third Party Dependencies

    gettext, phoenix, phoenix_ecto, phoenix_live_view

• MscmpSystInstance

  ^{(API Docs) / (Database Docs & ERD)}

  “Instances” are instances of running application environments. Instances are established to host the application for different purposes, such as for running the application for production, training, and testing purposes; or as a means to implement multi-tenancy where each tenant application environment is an Instance.

  Each Instance also requires supporting data in order to facilitate runtime actions, such as defining database roles with which to access database data. Such supporting information is also managed via this component.

  • First Party Dependencies

    mscmp_syst_db, mscmp_syst_enums, mscmp_syst_error, mscmp_syst_options, mscmp_syst_utils

  • Third Party Dependencies

    (none)

• MscmpSystLimiter

  ^{(API Docs)}

  This component limits the rate at which targeted services can be called by any one caller to a level which preserves the availability of resources to all users of the system, or makes brute force information gathering prohibitively time intensive to would be attackers of the system.

  • First Party Dependencies

    mscmp_syst_error, mscmp_syst_utils

  • Third Party Dependencies

    hammer

• MscmpSystMcpPerms

  ^{(API Docs) / (Database Docs & ERD)}

  Implements MscmpSystPerms related functionality for the MssubMcp subsystem.

  • First Party Dependencies

    mscmp_syst_authn, mscmp_syst_db, mscmp_syst_error, mscmp_syst_perms, mscmp_syst_utils

  • Third Party Dependencies

    (none)

• MscmpSystNetwork

  ^{(API Docs)}

  Simple IP address handling and convenience functionality.

  • First Party Dependencies

    mscmp_syst_error

  • Third Party Dependencies

    jason

• MscmpSystOptions

  ^{(API Docs)}

  API for retrieving and working with option files stored in the application server file system.

  • First Party Dependencies

    mscmp_syst_db, mscmp_syst_error

  • Third Party Dependencies

    toml

• MscmpSystPerms

  ^{(API Docs) / (Database Docs & ERD)}

  Provides a generalized foundation for user permission system implementations.

  The principle idea of this component is to organization permissions in a way that higher level components can introduce the concept of user and establish contexts of applicability while keeping a cohesion in permissioning capability. To this end this component provides the core concepts for use in any permissioning system using this ecosystem of components.

  • First Party Dependencies

    mscmp_syst_db, mscmp_syst_error, mscmp_syst_utils

  • Third Party Dependencies

    (none)

• MscmpSystSession

  ^{(API Docs) / (Database Docs & ERD)}

  Session Management API & Runtime

  • First Party Dependencies

    mscmp_syst_db, mscmp_syst_error, mscmp_syst_utils

  • Third Party Dependencies

    (none)

• MscmpSystSettings

  ^{(API Docs) / (Database Docs & ERD)}

  The Settings Service provides caching and management functions for user configurable options which govern how the application operates. Multiple Settings Service instances may be in operation depending on the needs of the application; for example, in the case of multi-tenancy, each tenant will have its own instance of the Setting Service running since each tenant’s needs of the application may unique.

  • First Party Dependencies

    mscmp_syst_db, mscmp_syst_error, mscmp_syst_utils

  • Third Party Dependencies

    (none)

• MscmpSystUtils

  ^{(API Docs)}

  Common utility functions generally useful across components.

  • First Party Dependencies

    (none)

  • Third Party Dependencies

    (none)

4 - Elixir Project Structure

Central Motivation

As has been said, the directory organization is based on the standard Elixir project organization as created by running mix new myapp. Our Components are also started this way and then modified to match our requirements.

A core development principle for our Elixir related projects includes the idea of strictly separating out API definitions from the business logic of the Component. It follows that this separation of interface from logic impacts our project structure. Within the lib directory which typically holds Elixir source files and possibly other directories, we allow for three directories:

• api - for Elixir source files defining the public API of the Component

• impl - for Elixir source files implementing business logic concerns of the Component.

• runtime - for Elixir source files implementing Component runtime concerns such as defining GenServers.

Under our structure, there are no source files directly homed in the lib directory itself.

Directory Structure and Common Files

Below is an example of a typical MuseBMS Elixir project followed by a detailed description. Depending on the specific features and scope of the Component not all directories or common files displayed below will be present.

.
├── README.md
├── config
├── database_utils
│   └── reset_dev_database.psql
├── dev_support
│   └── dev_support.ex
├── lib
│   ├── api
│   │   ├── msdata
│   │   ├── msform
│   │   ├── types
│   │   ├── types.ex
│   │   └── <module_name>.ex
│   ├── impl
│   │   ├── msdata
│   │   └── msform
│   └── runtime
├── mix.exs
├── priv
│   ├── database
│   └── plts
└── test
    ├── support
    └── test_helper.exs

Top Level Entry Descriptions

What follows are the important details regarding each of the organizational elements at the top level, with additional details as relevant.

README.md

The README.md files serves the typical purpose of describing the Component’s purpose and basics about usage. An important extra role for the README.md is that it is incorporated into the project’s technical documentation as the opening text.

Given this important opening role, the README.md file should contain a solid overview as well as an introduction to the Component at the conceptual and definitional levels.

config

The config directory is the standard Elixir project directory containing compile time related configurations. We do not use the config directory or its files in any non-standard way and actively avoid having our Components depend on these configuration files. There are exceptions, such as when third party libraries require configuration using this method, but even then our usage is in keeping with Elixir standard practices.

database_utils

This directory contains psql scripts, specifically written for the needs of this component, which are useful during development and testing. Only Components which have a database requirement will have this directory.

• reset_dev_database.psql

  This script is run at the end of mix test execution to reset the testing database back to a clean state. It is specific to the Component where it is defined, though this script will be similar across all components where found.

dev_support

The dev_support directory contains Elixir code which is use to aid development activities. For the most part, this involves building migrations and deploying them to a development database server. Other dependencies which require services to be started may also be started via code sourced from this directory.

• dev_support.ex

  Typically this is the only file in this directory. While that’s not a hard and fast rule that future Components will follow, there are few needs for development runtime support and so a single module makes sense.

lib

This is the traditional home of Elixir application source code and this doesn’t change for this project. As previously discussed, where we depart from convention is in how the source code inside this directory is organized.

• api

  All source files in this directory should be focused on defining and documenting the APIs used to interact with the Component. Business logic implementation should not, as a practice, be included in the API defining source files.

  Files in api will typically consist of defdelegate calls, struct definitions, and type definitions, all of which should be documented.

  Some of the favored practices just discussed are possible sources of performance degradation. The first choice is to follow the standard established here and assume performance penalties are inconsequential. If performance issues arise, then we break ti standard documented here with a focus on preserving the API, but moving implementation closer to the API defining functions. We don’t expect material performance impacts in all but a few edge cases.

  • msdata

    This is special sub-directory containing the definitions of Ecto.Schema structs which establish database table representations. In addition there will be defdelegate references to Ecto.Changeset processing functions. Components which do not define database schema will not have an api/msdata directory.

    The structs created in msdata are namespaced directly to the Msdata namespace, so for example a users table may be in a struct module named Msdata.Users. Naturally, this means that all Ecto Schemas in msdata must be uniquely named across all Components in the project.

  • msform

    This is special sub-directory containing the source files which define MscmpSystForms implementing modules. More information about the organization of the api/msforms directory can be found in the MscmpSystForms Developing Forms documentation.

    This directory will only exist for Components implementing MscmpSystForms based user interfaces.

  • types

    The types directory hosts source files which principally define public structs which are usable outside of the Component and which aren’t defined in a more dedicated part of the api directory hierarchy (e.g. msdata database related structs). Public struct source files will include the struct definition and defdelegate calls to impl or runtime hosted function implementations. This directory will not exist if there are no module-level type definitions.

  • types.ex

    This source file holds typespecs for simpler types which don’t require a full module to be created. Again, these types should be documented and considered useful outside of the Component. Typically this fill will list all its defined types in alphabetical order, though other organization is acceptable if there are enough defined types to make more topical organization useful.

  • <module name>.ex

    A source file containing the Component’s principle API definition and API documentation. This file carries the same name as the Elixir project and will front the majority of the Component’s functionality on offer to the outside world. In a standard Elixir project, a source file with roughly the same purpose and name would be found directly in lib; we simply move that file to api and more strictly define its purpose.

• impl

  This directory provides a home for the application’s business logic implementation. The source files in this directory will contain the vast majority of code, though there should be no ExDoc related doc tags in these files aside from @moduledoc false. Comments describing feature/function intention are allowed and, indeed, welcomed.

  • msdata

    This is the implementation side of previously discussed api/msdata directory. If a data struct requires business logic implementation, such as the definition of Ecto.Changeset processing or protocol implementation, a sub-directory will be created here with the same name as the corresponding struct in api/msdata. Other, more general changeset validation or helper modules which implement data related logic may host their source files directly in the root of impl/msdata. Standard names for these sorts of files include the following:

    • impl/msdata/helpers.ex

      Helper functions which work across multiple structs’ logic.

    • impl/msdata/general_validators.ex

      Validation and Changeset processing functions which are reusable across multiple data defining structs in api/msdata.

    • impl/msdata/<data struct name>/validators.ex

      Source files hosting Changeset processing functions which are delegated to from struct modules defined in api/msdata.

    • impl/msdata/<data struct name>/protocol.ex

      Source files implementing various protocols for a given struct as needed.

  • msform

    This is the implementation oriented compliment to the api/msform directory. For more complete documentation on the use of this directory please see the MscmpSystForms Developing Forms documentation.

• runtime

  Source files which define and implement runtime services, such as GenServers. This directory will only exist if the Component defines such runtime services. Note that code in runtime source files is limited to the runtime concerns. Business logic implementations continue to be hosted in the impl directory structure and referenced from the runtime service as necessary.

mix.exs

This is the typical Elixir mix.exs defining project file. There are some organizational differences between our approach and the typical generated Elixir mix.exs, but there are no differences that go further than trivial stylist differences.

priv

This is the standard Elixir priv directory, though there are a few special sub-directories to be aware of.

• priv/database

  In Component level Elixir projects, this directory may be visible and hosts database migrations built during the testing process. When tests are not actually running, the directory will exist but will be empty as the testing cleanup processes will typically delete this database related testing artifacts.

  At the Subsystem level the priv/database directory will host sub-directories named after their corresponding database type name (see the MscmpSystDb documentation for more about database types). The database migrations built for the database type will be hosted by the appropriate sub-directory. Note that these database migration files are persistent and drive the release as opposed to the transient nature of Component level migrations.

• priv/plts

  This directory hosts generated PLT files for Dialyzer analysis during development and the Continuous Integration process. There is some historical precedence for doing handling PLTs and we’re just following this community convention.

  PLT files should not be part of releases and so this directory should not be a factor outside of development and testing processes.

test

This is the standard Elixir project testing directory. We follow standard conventions for the most part in regard to testing. We do define two different kinds of testing:

• Unit Testing

  Unit tests are aimed at testing non-private impl and runtime related code and functions at a granular level. We are not confirming the public API in unit testing, but rather the internal implementations for correct results and typing.

• Integration Testing

  Integration testing works to test the public API of the Component in a fashion which mimics the end-to-end business process as much as possible, including the Component’s interaction with any dependencies.

These two testing types will not be run together and will typically use different testing database seed data (if database interactions are part of the component’s functionality.)

Sub-directories and files to the test directory include:

• support

  For testing support related files. It is typical for this directory to host implementations of ExUnit.CaseTemplate and logic for building, migrating, and seeding test databases, runtime services defined by dependencies and making good on other testing pre- conditions.

• test_helper.exs

  This is the standard Elixir/ExUnit test_helper.exs file, typically modified to help in supporting testing related database operations and post-testing cleanup.

5 - Trigger Naming Conventions

Trigger Function Names

Trigger function names follow a convention which 1) establishes that the function is specifically a trigger function; 2) at what transaction time the trigger runs; 3) under which operations the trigger runs; and 4) a name relevant to the purpose of the trigger.

The form of the name is: trig_<b/a/i>_<i/u/d>_<purpose>

1. trig_: the indication that this is a trigger returning function which should not be used outside of trigger contexts.

2. <b/a/i>: an indicator of what phase of the transaction the trigger is expected to be fired. The available options are b for BEFORE triggers, a for AFTER triggers, and i for INSTEAD OF triggers. If the trigger function is designed to be called in more than one transaction phase, multiple indicators may be placed next to each other.

3. <i/u/d>: establishes which operations are supported by the trigger function. The available options are i for INSERT, ‘u’ for UPDATE, and d for DELETE. If a trigger function is designed to support more than a single operation, the identifier of the supported operations may be placed next to each other.

4. <purpose>: a descriptive name for the trigger function.

As an example, consider a trigger function designed to verify that a record is ‘consistent’ prior to allowing an inserted or updated record to be visible to the database. Such a trigger may be named something like:

• trig_a_iu_validate_consistency

This would indicate that the trigger is expected to be an AFTER trigger which can be run during both INSERT and UPDATE operations.

Trigger Names

Trigger names follow a similar naming convention to the trigger function naming conventions discussed above. In fact, trigger names should be closely based on the trigger function name the trigger calls. The first difference in naming is that an execution order setting prefix is added prior to the trig_ opening. The next difference is in the way the previously defined indicators in the name are used. The <b/a/i> and <i/u/d> indicators present in trigger function names indicate which runtime conditions the trigger function supports; these same indicators are used in trigger names as well, but in trigger names the indicators specify the specific firing conditions of the trigger and so may differ slightly from the supported indicators for the trigger function name.

Using our example trigger function name above, lets suppose that we want to create two separate triggers which have independent conditions for firing, but both call the same trigger function; perhaps the INSERT call fires the trigger function unconditionally, but the UPDATE call only fires if a specific WHEN condition is true. The two trigger names could be:

• a50_trig_a_i_validate_consisteny for the INSERT trigger
• a50_trig_a_u_validate_consistency for the UPDATE trigger.

Note that the i and u indicators are separate in the trigger names, but are next to each other in the trigger function name.

Trigger Name Prefix Conventions

Triggers are executed by the database in trigger name order according to the database collation. To facilitate the organization of our code and to allow the execution ordering to proceed in a deterministic way, we adopt the following trigger name prefixes.

• a00_ - a99_ : These are early validations designed to stop later, possibly more expensive processing later in the trigger sequences. We shouldn’t be changing or setting data in this class of function, though we might look up data in other tables.

• b00_ - b99_ : Data manipulation functions. These are the functions we want updating or setting values that are appropriate for the limits of database maintained business logic (data integrity/ consistency focused).

• c00_ - c99_ : Late validation functions. These are the functions which validate data after the data manipulation phase. In reality most data validation should probably be here since the “b” class triggers can take a starting invalid record and make it valid.

• z00_ - z99_ : Utility/Auxiliary processing functions. This class of trigger processes records/data/etc that aren’t directly related to the business data in the record. For example, the diagnostic columns such as last updated or updated by or functions that record data changes to an audit table are the kinds of functions which should be in the “z” class. These functions should not make business validation judgements or take transformative actions on business data. These functions are really operational in nature and work on records which, from a business data perspective, are in their final form.

6 - Build Processes

There are several project artifacts which are subject to a build process prior to their being useful in the project. Chiefly these are the Msplatform application and various documentation pieces.

6.1 - Msplatform Build Process / Linux

When building Msplatform we assume that we are including ERTS as part of the release.

We largely follow the standard Phoenix “Deploying with Releases” documentation to build a release, modified somewhat to support both the umbrella project structure and our own needs. this process looks like:

mkdir msplatform_linux_release
cd msplatform_linux_release
git clone https://github.com/MuseSystems/musebms.git --branch versions
cd musebms/app_server/platform/msplatform
mix deps.get --only prod
MIX_ENV=prod mix compile
cd apps/msapp_mcp_web
MIX_ENV=prod mix assets.deploy
MIX_ENV=prod mix phx.gen.release
cd ../../
MIX_ENV=prod mix release
cd _build/prod/rel
tar -cvjSf ../../../../../../../msplatform_release.tar.bz2 msplatform
cd ../../../../../../../

7 - Development Testing Standards

Testing Concerns

Currently there are two broad testing concerns which mirror the highest level structure of our application.

Application Server Testing

This is testing of the Elixir application server code. While this testing concern is not meant to be an explicit testing of database code (schema, constraints, store procedures, etc.) the database code will nonetheless be exercised by application.

Application server testing current is focused on using the Elixir ExUnit test suit with workflows prescribed in this document to attain our application development testing goals.

Database Testing

Database testing is focused on testing the database explicitly outside of the context of the application related code.

When direct database testing is incorporated into the test suite, we’ll like base such testing on PgTap.

Testing Types

Each of our testing areas of concern are subject to three kinds of testing.

Unit Testing

Unit testing targets the intended to confirm the functioning of any internal/private API functions defined by a Component. Unit tests have the following qualities:

• Testing Target

  Internal, private APIs

• Ordering

  Random & asynchronous

• Data

  Expects the database to be sufficiently seeded such that any private API function can execute normally without interfering with other, possibly concurrently running tests.

  Ideally, the seeded test data is randomly generated, though this is not commonly done at present.

Integration Testing

The focus of Integration Testing is twofold: 1) test that public API functions are working correctly; and 2) establish that the supported business process can be operated successfully and completely using the public API. Traits of integration testing are:

• Testing Target

  Public APIs & end-to-end business processing adequacy

• Ordering

  Business processed ordered & synchronous

• Data

  Pre-testing database seeding is as minimal as possible with later tests relying on data created during earlier tests.

  Ideally, test data is generated randomly though this is not consistently done at present.

Documentation Testing

Documentation Testing, where supported by the test suites, is intended to ensure that any documented code examples are representations of working code. The qualities of documentation testing are:

• Testing Target

  Public API documented examples

• Ordering

  Random & asynchronous

• Data

  Expects the database to be sufficiently seeded such that any public API function can execute normally without interfering with other, possibly concurrently running tests.

  The seeded data must be well defined and correspond directly to the data seen in the documented code examples.

Testing an the Component Model

The Component Model establishes a hierarchy of application concerns which influences the expectations of our testing regime.

Component Level Testing

Each Component is responsible for defining its own unit, integration, and documentation testing suite. Passing the complete test suite is a pre-requisite for inclusion in any release oriented code branch of the larger application.

If a Component has dependencies, it may assume that those dependencies have defined and passed their own test suites. If they require data seeding in order to support testing of the current Component, the typical process should be to simply load the appropriate testing data from the dependency’s test suite as a prerequisite to loading the current Component’s seed data. This does mean that across Components some effort for cross-Component compatibility of data seeding should be considered.

Subsystem Level Testing

Subsystems are mostly aggregates of Components and their testing will typically reflect this. While unit tests and documentation tests will exist for Subsystems, these testing concerns will not differ from Component level testing; indeed, just as Components will assume their dependencies are independently tested prior to being available as a dependency, the dependencies of the Subsystem are similarly treated.

Integration testing for Subsystems, however, is much more important and extensive. The reason for this is that Subsystems must ensure that their combinations of large numbers of dependencies (Components) must all work well enough together to form a coherent application. Therefore, end-to-end business process testing is key to ensuring application level degrees of interoperability. This is the full expression of the business logic of the application and testing that expression is critical.

Seeding data for Subsystem integration testing is usually restricted to the data seeded as part of a new system implementation, with no specifically testing related data being seeded. All testing data is created using the Public APIs of the application with the data from earlier tests providing inputs to later tests.

Unit and documentation test data, insofar as it is required, is typically seeded from the unit and documentation test seed data from each of the dependencies prior to the tests being run; naturally if there are Subsystem specific requirements for seeding test data the Subsystem’s test suite can define and load any additional seed data as needed.

Platform Level Testing

Testing the Platform related code is focused on Integration testing similar to the testing focus of Subsystems. The key difference is that the integration focus is less concerned with testing end-to-end business logic as it is with integration testing of the system’s external “user interfaces”; user interfaces as used here is broadly interpreted as any interface to external users. Examples of user interfaces are “web application”, “External, Public API”, or any other sanctioned means by which an external user may interact with the running Elixir applications.

The data sources and seed data concerns of the Subsystem level of testing are broadly applicable to Platform Level Testing.