Open Standards: Finding solutions together

Standards are everywhere, from HTML used to structure this text to the layout of your keyboard. There is a standard for almost anything. They can emerge from legislation (like seat belt requirements), standards bodies such as IETF or ISO, through market dominance (e.g. the S3 API) or from collaboration between affected parties.

I use the term "open standard" in the terms of open-source, an open standard is freely accessible and can be adopted, implemented and extended by anyone. Open standards are voluntary agreements between implementers on a common solution to a shared problem.

You can design a protocol, but you cannot design a standard.

A standard only emerges when affected stakeholders find an agreement on how to solve a shared problem. The same applies to extensions.  If your extension is not adopted by other implementers, it's just another protocol and likely adds to fragmentation.

Establishing and evolving standards is therefore as much a social challenge as a technical one. It requires finding and continuously updating an agreement among diverse parties.

Failing to do so might lead to a situation as in the following XKCD sketch by Randall Munroe:

Why would you do that?

Why invest that effort instead of simply building your own protocol? Isn't it easier and faster to roll your own solution?

The hardest part of developing something new is often not *how* something should work, but to understand *what* is actually needed. Collaborating on a shared solution exposes you to many more perspectives on the problem. This helps avoid developing yourself into a corner that may later require an expensive rewrite.

In many cases, a shared standard is what makes a feature possible in the first place. If your service needs communicate with others, it is far easier to implement one shared protocol than to implement and maintain individual integrations for each and every service your users might encounter.

The benefits of a standard grow with adoption. As more services integrate it, the value for users increases, without increasing your own implementation effort. A growing ecosystem may also allow you to build upon existing open source implementations instead of starting from scratch.

How would you get started?

The goal of OpenGeoMesh, for example, is to enable collaboration across as many organizations and software tools as possible. Designing a protocol is therefore not enough, a broadly accepted standard must be established.

Before negotiating an agreement, you need to identify potential users and implementers. They are the experts on their workflows, constraints, and infrastructure. Research existing solutions to make sure you are not reinventing the wheel. 

Standards themselves benefit from standards. Agreement is easier when building on something already established and widely used. Leveraging existing standards improves compatibility with existing infrastructure, makes the solution more intuitive for users and developers, and reduces implementation effort thanks to available open source libraries.

Discussing with stakeholders what they already use proved to be a valuable way to identify potential candidates. Options differ in popularity, complexity, implementation effort and long-term maintainability.

An important step is to clarify the scope of the new standard. Which aspects should be covered by the new standard? Which should be delegated to existing standards? What should be left for future iterations or explicitly omitted? This is a delicate balance between complexity, usability and interoperability. Doing one thing well reduces complexity, but omitting important features may lead to incompatible implementations.

Requirements evolve. To prevent a standard from stagnating, you must define how changes and extensions are proposed and adopted. You cannot mandate implementers to follow updates, they must be convinced of their value. Without evolution, interoperability degrades or the standard is eventually replaced.

Clear governance processes are therefore essential. How can changes be brought up for discussion? How is a decision on a change made? Should the standard align with an established standardization body like IETF or should it be community governed?

The outcomes of these discussions must be condensed into a concise specification containing all necessary details to build interoperable implementations. This document also serves as shared reference point and helps uncover unresolved questions and disagreements.

A formal voting can checkpoint agreement, but the real test of a standard is adoption among implementers and users.

Experiences from working on OpenGeoMesh

The idea for OpenGeoMesh was born during a forest fire in the mountains. Firefighters and mountain rescue teams had to collaborate in challenging terrain to control the fire and to ensure the safety of the crews. Due to missing interfaces between their situation-management software, situation maps had to be drawn on paper, or exchanged via screenshots and photos via WhatsApp.

I'm part of a team of volunteer software developers within bavarian mountain rescue, so I began discussing potential solutions with other developers. These conversations quickly expanded to other emergency services and vendors of situation-management software. Soon I found myself talking about OpenGeoMesh to anybody willing to listen, which led to unexpected interest from e.g. municipal services seeking secure ways to share maps of sensitive infrastructure with contractors at construction sites.

In hindsight, establishing shared discussion channels early on might have simplified coordination instead of engaging stakeholders individually. Two central requirements emerged, strict access controls due to the sensitive nature of the exchanged data and decentralization to support continued operation during internet outages.

We evaluated federated protocols such as ActivityPub and OpenCloudMesh. P2P protocols were also considered, but their integration into existing systems, especially regarding authentication, was deemed too complex.

We finally decided to base our work on OpenCloudMesh. It's purpose aligns closely with OpenGeoMesh and it explicitly supports the combination with other transport protocols. Its specification is tightly focused on the discovery of sharing servers and the exchange of access metadata only, allowing us to implement a working prototype quickly. Some ambiguities in the draft specification were clarified by participating directly in the IETF working group.

To exchange geospatial data, a structured and flexible API is required. The most established standard in this domain is WFS by the Open Geospatial Consortium. While this standard is already in use by some vendors of situation-management software, OpenGeoMesh will be based on its successor OGC API - Features. It is supported by most open source GIS servers, so the added maintenance effort is acceptable. OGC API - Features is expected to replace WFS in the long term and enables modern API practices without mixing XML and JSON payloads.

The standard is organized in modular building blocks. Selecting a small mandatory subset allowed us to limit the initial implementation effort, while keeping room for future extensions.

To support emergency services, situation maps need to clearly depict the current state of rescue operations. In Germany there are strict guidelines on how certain events have to be drawn using a fixed set of icons and other map objects. To represent these in GeoJSON we adopted the Situation Data Exchange Format (SDXF), which integrates seamlessly with OGC API - Features.

Throughout the process, discussions where continuously supported by prototype implementations to explore the various options and validate their technical feasibility. This resulted in a set of reusable libraries which simplify adoption of OpenGeoMesh. Meanwhile, the first vendors are already working on independent implementations, a promising sign that OpenGeoMesh is gaining traction.