XC-Messages File Schema

XC-Messages File Schema#

FILE & WIRE SCHEMA and generated LANGUAGE BINDINGS

for SYSTEMS | GEOMETRY | DATA | GRAPHICS

I. Introduction#

Messages (aka XC-Messages) is an object-oriented schema for engineers and scientists that efficiently enables numerical computing data to be seamlessly shared across platforms and languages.

Generated interface code delivers high-performance, elegant, and flexible messaging across science and engineering applications. Encoding/decoding is similar to XML/JSON but faster and denser. The machine-generated library contains utilities to flatten and reconstruct object-oriented and vectorized data structures encountered in numerical simulation setup, expression, and results (e.g., systems engineering, CFD, FEA, EDA, geometry processing).

Standard Protocol Buffer (Protobuf) definitions offer compact layouts for numerical computing storage and transmission. The schema is passed into the protoc compiler to generate high-performance binary-encoded accessors. Xplicit Computing applications are built atop these bindings; end-users can apply the same standards for custom integration.

The overarching CAE concept: each system manages its own parameters, members, and references (including child subsystems), encapsulated in *.xcs or *.json files (optionally referencing native/external files).

Users can use all or part of the provided messages. Tighter conformance enables high-performance integration across CAE applications.

Benefits:

Binary file-and-wire formats for storage/transmission
Universal accessors and serialization utilities for most OS/languages
Native compatibility across XCOMPUTE and user-defined applications
Efficient parallel read/write with forward/reverse compatibility
Free and open standard built on agnostic infrastructure

Limitations:

Protobuf 32-bit indexing → 2^31 (~2 billion) element limit per system/geometry
Messages >64 MB require special handling

1. Offsets can extend to `2^64`, but this feature is not yet mature in the working spec. 2. See [Section IV-7: Loading large messages](#sec_iv_7).

Proto Definitions#

Four *.proto files are central to the schema:

concept.proto → domain setup, models, parameters, associations (*.xcs/*.json)
vector.proto → numeric arrays using packed arena allocation (*.xco)
spatial.proto → topology of elements and regions (*.xcg)
meta.proto → metadata and user-graphics (*.xcm)

Message classes are available for each file. In C++, they reside under Messages:: after including *.pb.h headers. Other languages organize interfaces differently. Refer to *.proto definitions for schema details; access patterns are built directly from these assignments. Standardized get/set functors access members encoded in protocol buffer streams.

There are ~18 useful messages relating to different concepts. Users don’t need to modify definitions, but *.proto files serve as handy references. vector.proto is well-suited for storing/transmitting vectorized data and provides a good entry point.

II. Adding Messages to Your Project#

Download the latest definitions: Download or Clone from Github

The protoc compiler provides bindings for: C++, C#, Java, JavaScript, Objective-C, PHP, Python, Ruby

Dart and Go are available as official plugins. Third-party bindings exist for 30+ languages (see Section VI).

Import relevant files as headers/libraries. Statically-compiled languages (C++, Obj-C, C#) require linking to libxcmessages.a. Google Protobuf 3 Libraries may also be required.

Verify compiler installation:

protoc --version

III. Serializing and Parsing Messages#

To save/transmit, associative data structures must be serialized into contiguous memory/storage formats. C++ examples using std streams:

// Assuming a Messages::Vector64 msg ...
std::ofstream outfile(path);
msg.SerializeToOstream(&outfile);
outfile.close();

Serialized data transmits over file-and-wire, then deserializes on the receiving end:

std::ifstream infile(path);
Messages::Vector64 msg;
msg.ParseFromIStream(&infile);
infile.close();

Refer to Protobuf3 Tutorials for comprehensive cross-language patterns.

IV. C++ Examples#

Explore examples/hello_vector for save/load demonstrations. Compiles out-of-box with build.sh, save.sh, load.sh using gcc/clang. Ensure Google Protobuf 3 Libraries and schema library are linked in CMakeLists.txt:

target_link_libraries(some_app /path/to/libprotobuf.a)
target_link_libraries(some_app /path/to/libxcmessages.a)

Or compile directly:

c++ save_msg.cpp -L/path/to/libprotobuf.a -L/path/to/libxcmessages.a

1. Assigning Messages#

Include headers:

#include "concept.pb.h"
#include "spatial.pb.h"
#include "vector.pb.h"

a. Fill repeated fields (one-by-one)

Messages::Vector64 msg;
msg.set_name("Position");
msg.set_components(3);
msg.add_values(pos.x);
msg.add_values(pos.y);
msg.add_values(pos.z);

b. Fill repeated fields (serial loop)

for (auto val : other)
    msg.add_values(some_function(val));

c. Fill repeated fields (parallel loop)

auto N = other.size();
auto& values = *msg.mutable_values();
values.resize(N);
#pragma omp parallel for
for (auto n=0; n<N; n++)
    values[n] = some_function(other[n]);

2. Saving Messages to File#

std::ofstream outfile(path);
msg.SerializeToOstream(&outfile);
outfile.close();

3. Loading Messages from File#

#include "vector.pb.h"
Messages::Vector64 msg;
std::ifstream infile(path);
msg.ParseFromIStream(&infile);
infile.close();

b. For messages >64MB, see Section IV-7.

4. Accessing Messages#

a. Manual entry access

auto& vec_name = msg.name();
auto C = msg.components();
auto& rev = msg.revision();
std::vector<double> output;
if (msg.values_size() > 1 && rev.major_rev() < 42) {
    output.push_back(msg.values(0));
    output.push_back(msg.values(1));
}

b. Serial loop

std::vector<double> output;
for (auto value : msg.values())
    output.push_back(some_function(value));

c. Parallel loop

auto N = msg.values_size();
std::vector<double> output(N);
#pragma omp parallel for
for (auto n=0; n<N; n++)
    output[n] = some_function(msg.values(n));

5. Embedding Messages#

Complex structures require mutable_ getters returning pointers to underlying objects.

a. Pointer assignment

auto rev = msg.mutable_revision();
rev->set_major_revision(4);
rev->set_minor_revision(13);

b. Reference assignment (cleaner)

auto& rev = *msg.mutable_revision();
rev.set_major_revision(4);
rev.set_minor_revision(13);

Note: Omitting & in auto& creates a stack copy; assigned values won’t affect the original message.

6. Copying Messages#

a. msg = other_msg; b. msg.CopyFrom(other_msg); c. Low-level contiguous copy (IEEE-754 compliant):

msg.mutable_values()->Resize(other.size());
memcpy(msg.mutable_values()->data(), other.data(), other.size()*sizeof(double));

7. Loading Large Messages with `Google::Protobuf::CodedFileStream`#

Bypasses Protobuf 3’s 64MB security limit. Limits can be increased to int32 max.

#include <google/protobuf/io/coded_stream.h>
#include <google/protobuf/io/zero_copy_stream_impl.h>
Messages::Vector64 msg;
unsigned int IObyteLimit{ 1 << 31 };

int fd = open(path.c_str(), O_RDONLY);
FileInputStream raw(fd);
CodedInputStream inputStream(&raw);

auto limit = inputStream.PushLimit(-1);
inputStream.SetTotalBytesLimit(IObyteLimit >> 1, IObyteLimit);

if (msg.ParseFromCodedStream(&inputStream)) {
    inputStream.PopLimit(limit);
    close(fd);
} 
else {
    // handle error
}

V. Python Examples#

Python 3 bindings are generated by protoc, producing concept_pb2.py, spatial_pb2.py, vector_pb2.py, meta_pb2.py. See Section VI for build instructions.

Set path via environment or sys.path:

PYTHONPATH=/path/to/pythonbindings
# or
import sys
sys.path.insert(1, "/path/to/pythonbindings")

1. Assigning Messages#

import vector_pb2 as vector
msg = vector.Vector64()
msg.name = "Position"
msg.components = 3
msg.values.append(pos.x)
msg.values.append(pos.y)
msg.values.append(pos.z)

b. Serial loop

for val in other:
    msg.values.append(val)
# or
msg.values[:] = other

2. Saving Messages#

serial = msg.SerializeToString()
with open(path, "wb") as sout:
    sout.write(serial)

3. Loading Messages#

import vector_pb2 as vector
msg = vector.Vector64()
with open(path, "rb") as sin:
    msg.ParseFromString(sin.read())

4. Accessing Messages#

vec_name = msg.name
C = msg.components
output = []
if len(msg.values) > 1 and rev.major_rev < 42:
    output.append(msg.values[0])
    output.append(msg.values[1])

b. Serial loop

output = []
for i in range(len(msg.values)):
    output.append(msg.values[i])
# or
output = msg.values[:]

5. Embedding Messages#

Python uses call-by-object-reference. Scalar assignments are call-by-value; object assignments are call-by-reference.

rev = msg.revision
rev.major_rev = 4
rev.minor_rev = 13
print(msg)
print(rev)

6. Copying Messages#

Object assignment creates references. Use deepcopy to separate:

from copy import deepcopy
msg1 = msg
msg2 = deepcopy(msg1)
msg2.revision.minor_rev = msg1.revision.minor_rev + 1

VI. Building Custom Bindings#

protoc supports C++, C#, Java, JavaScript, Objective-C, PHP, Python, Ruby. Plugins exist for Dart, Go, Kotlin, and 30+ third-party languages.

Reuse is encouraged without altering existing definitions. Enumerate custom extensions >100 to avoid conflicts.

Verify installation:

which protoc
protoc --version

Generate bindings:

mkdir -p cpp python java javascript ruby objc csharp
protoc --cpp_out=cpp --csharp_out=csharp --objc_out=objc --ruby_out=ruby \
    --python_out=python --java_out=java --js_out=javascript \
    vector.proto system.proto spatial.proto meta.proto

JavaScript requires additional options:

--js_out=javascript,import_style=commonjs,binary:.

VII. License and Fair Use#

SPDX-License-Identifier: BSD-3-Clause
OSI-License-URL: https://opensource.org/licenses/BSD-3-Clause

The four proto files, generated library/bindings, and README are provided under the BSD 3-Clause License. Free for personal, academic, commercial, and research use. No warranty implied unless stated in a separate agreement. The README must remain alongside *.proto definitions.

Custom extensions are permitted for private use but should not be distributed publicly to preserve compatibility. Share proposed public schema updates with maintainers at info@xplicitcomputing.com.

XC-Messages File Schema

Contents

XC-Messages File Schema#

I. Introduction#

Proto Definitions#

II. Adding Messages to Your Project#

III. Serializing and Parsing Messages#

IV. C++ Examples#

1. Assigning Messages#

2. Saving Messages to File#

3. Loading Messages from File#

4. Accessing Messages#

5. Embedding Messages#

6. Copying Messages#

7. Loading Large Messages with Google::Protobuf::CodedFileStream#

V. Python Examples#

1. Assigning Messages#

2. Saving Messages#

3. Loading Messages#

4. Accessing Messages#

5. Embedding Messages#

6. Copying Messages#

VI. Building Custom Bindings#

VII. License and Fair Use#

7. Loading Large Messages with `Google::Protobuf::CodedFileStream`#