Byblos

Byblos SDK
Created by				Alexander Nicholi
Initial release				unreleased
Written in				ANSI C, Make, POSIX sh
OSes				Darwin/XNU, GNU/Linux
ISAs				Apple silicon, x86-64
Licence				ASL 1.1

Byblos is a software development kit developed for Sirius DOS, A* and Unix-based servers, with distinction for MILOTIX and Linux, BSD and Darwin (pan-Unix) support. It is divided into two versions: Byblos 1, a provisional SDK that uses existing compiler toolchains and host infrastructure of Unix-like operating systems, and Byblos 2, the proper SDK that provides its own C compiler, assembler, and the first C* compiler.

Byblos 1

Byblos 1 is the provisional Byblos SDK, architected around Intel and Apple Silicon macOS as well as x86-64 GNU/Linux. Fortunately, all of our compiler toolchains (with the exception of OpenWatcom detailed later) have a relatively straightforward installation process on all of the supported host operating systems.

Byblos 1 should support compiling assembly, ANSI C and binpacked assets on all target platforms. Additionally, it supports host-only just-in-time compilation and running of Vala programs, enabling the creation of developer utilities with graphical interfaces. Vala only depends on the system toolchain and the Vala package of the native package manager.

Platform support

Byblos 1 is multi-platform. While it endeavours to target all systems of relevance to Xion Megatrends, only three host platforms are supported: GNU/Linux on Intel 64, Darwin/XNU on Intel 64 and Darwin/XNU on Apple silicon. Since Byblos 1 is a provisional SDK that leans on existing platform provisions for brevity, it additionally supports Vala for easily developing performant and optionally GUI-enabled applications for helping development.

Supported targets

i	Target	Full name	ISA	Host	ASM	C	bin	Notes
	agbhb	GBA homebrew	ARMv4T	✖	✔	✔	✔	Scratch-building GBA ROMs.
	agbsp	GBA sourcepage	ARMv4T	✖	✔	✔	✔	Source-patching existing GBA ROMs.
	astar	Anodyne A*	unknown	✖	✖	✖	✖	Too early to specify.
	darwin86	Intel Mac	x86-64 v1	✔	–	✔	–
	darwinm1	Apple Silicon Mac	ARMv8	✔	–	✔	–	Only host that can target everything.
	freebsd	FreeBSD	x86-64 v1	✖	✔	✔	–
	ibmpc	Real mode IBM-PC	Intel i286	✖	✔	✔	✔	Not specific to any DOS conventions.
	illumos	illumos/OpenIndiana	SPARC64 V9	✖	✔	✔	–
	linux32	GNU/Linux 32-bit	Intel i686	✖	✔	✔	–
	linux64	GNU/Linux 64-bit	x86-64 v1	✔	✔	✔	–
	milotix	MILOTIX	x86-64 v3	✖	✔	✔	–
	openbsd	OpenBSD	x86-64 v1	✖	✔	✔	–
	pcdos	Real mode PC-DOS	Intel i286	✖	✔	✔	✔
	win311	Windows 3.11	Intel i286	✖	✔	✔	✔
	win95	Windows 95, 98 & Me	Intel i386	✖	✔	✔	–
	winnt32	Windows NT 4.0 32-bit	Intel i386	✖	✔	✔	–
	winnt64	Windows NT 5.1 64-bit	x86-64 v1	✖	✔	✔	–

Notes

• binpacking is a highly portable method of compiling assets statically into binaries
• For assembly, uncertainty exists about toolchain support on macOS
• For binpacking, it may eventually work but needs further research

Compile time support matrix

Host	Target
	agbhb	agbsp	astar	darwin86	darwinm1	freebsd
linux64	devkitARM	devkitARM	unknown	none	none	LLVM
darwin86	devkitARM	devkitARM	unknown	hb:LLVM	hb:LLVM	hb:LLVM
darwinm1	devkitARM	devkitARM	unknown	hb:LLVM	hb:LLVM	hb:LLVM
Host	Target
	ibmpc	illumos	linux32	linux64	milotix	openbsd
linux64	OpenWatcom	LLVM	LLVM	LLVM	unknown	LLVM
darwin86	OpenWatcom	hb:LLVM	hb:LLVM	hb:LLVM	unknown	hb:LLVM
darwinm1	OpenWatcom	hb:LLVM	hb:LLVM	hb:LLVM	unknown	hb:LLVM
Host	Target
	pcdos	win311	win95	winnt32	winnt64
linux64	OpenWatcom	OpenWatcom	MinGW	MinGW	MinGW
darwin86	OpenWatcom	OpenWatcom	MinGW	MinGW	MinGW
darwinm1	OpenWatcom	OpenWatcom	MinGW	MinGW	MinGW

Runtime support matrix

Host	Target
	agbhb	agbsp	astar	darwin86	darwinm1	freebsd
linux64	mGBA	mGBA	unknown	none	none	QEMU/u
darwin86	mGBA	mGBA	unknown	native	none	QEMU/u
darwinm1	mGBA	mGBA	unknown	Rosetta 2	native	QEMU/u
Host	Target
	ibmpc	illumos	linux32	linux64	milotix	openbsd
linux64	Bochs	QEMU/h	native-ish	native	unknown	QEMU/u
darwin86	Bochs	QEMU/h	QEMU/u	QEMU/u	unknown	QEMU/u
darwinm1	Bochs	QEMU/h	QEMU/u	QEMU/u	unknown	QEMU/u
Host	Target
	pcdos	win311	win95	winnt32	winnt64
linux64	DOSBox	DOSBox	Wine	Wine	Wine
darwin86	DOSBox	DOSBox	Wine	Wine	Wine
darwinm1	DOSBox	DOSBox	Wine	Wine	Wine

Tool list

Tool name	Description
devkitARM	devkitPro's 32-bit ARM GCC toolchain.
hb:LLVM	LLVM installed via Homebrew on macOS.
LLVM	LLVM installed on a Unix-like.
OpenWatcom	Open source continuation of the Watcom C compiler for 16-bit DOS systems.
MinGW	Modified GCC for targeting Microsoft Windows.
mGBA	Accurate GBA emulator.
QEMU/u	QEMU virtualisation framework, in the low-friction userland virtualisation mode.
QEMU/h	QEMU virtualisation framework, using a full-fat VM.
Bochs	Accurate IBM-PC emulator.
Wine	Translation layer for Windows programs to run on other operating systems.
Rosetta 2	Apple's transition system for running Intel Mac programs on Apple Silicon.
DOSBox	Accurate MS-DOS emulator with a focus on gaming.

Programs list

i	Utility
	Hinterlib	✔	✖	✖	✔	✔	✔	✔	✔	✔	✔	✔	✔	✔	✔	✔	✔	✔
	Inbound	✖	✖	✖	✔	✔	✔	✖	✔	✔	✔	✔	✔	✖	✖	✖	✖	✖
	Outbound	✖	✖	✖	✔	✔	✔	✖	✔	✔	✔	✔	✔	✖	✖	✖	✖	✖
	Rebound	✖	✖	✖	✖	✖	✔	✖	✖	✖	✔	✔	✔	✖	✖	✖	✖	✖
	Earthbound	✖	✖	✖	✔	✔	✔	✖	✔	✔	✔	✔	✔	✖	✔	✔	✔	✔
	Forerunner	✖	✖	✖	✖	✖	✔	✖	✖	✖	✔	✔	✔	✖	✖	✖	✖	✖
	Precursor	✖	✖	✖	✔	✔	✔	✖	✔	✔	✔	✔	✔	✖	✖	✖	✖	✖
	Simbel	✖	✖	✖	✔	✔	✔	✖	✔	✔	✔	✔	✔	✖	✖	✖	✖	✖
	Oración	✖	✖	✖	✔	✔	✔	✔	✔	✔	✔	✔	✔	✔	✔	✖	✖	✖
	FCC	✖	✖	✖	✔	✔	✔	✔	✔	✔	✔	✔	✔	✔	✔	✖	✖	✖
	Quindle	✖	✖	✔	✖	✖	✖	✔	✖	✖	✖	✖	✖	✔	✖	✖	✖	✖
	Gauntlet	✖	✖	✖	✔	✔	✔	✔	✔	✔	✔	✔	✔	✔	✖	✖	✖	✖
	Senusret's	✖	✖	✔	✖	✖	✖	✔	✖	✖	✖	✖	✖	✔	✖	✖	✖	✖
	Sirius C*	✖	✖	✖	✔	✔	✔	✔	✔	✔	✔	✔	✔	✔	✔	✖	✖	✖
	NatPython	✖	✖	✖	✔	✔	✔	✖	✔	✔	✔	✔	✔	✖	✖	✖	✖	✖
	SablMap	✖	✖	✖	✔	✔	✖	✖	✖	✖	✔	✖	✖	✖	✖	✖	✖	✖
	CheckASCII	✖	✖	✖	✔	✔	✖	✖	✖	✖	✔	✖	✖	✖	✖	✖	✖	✖
	Checkline	✖	✖	✖	✔	✔	✖	✖	✖	✖	✔	✖	✖	✖	✖	✖	✖	✖
	Flow72	✖	✖	✖	✔	✔	✖	✖	✖	✖	✔	✖	✖	✖	✖	✖	✖	✖

Getting the toolchains

Byblos 1 depends on devkitARM, MinGW, LLVM and OpenWatcom to compile code for the various targets it supports. All but OpenWatcom can be installed in binary form using package managers.

Follow devkitPro's wiki page to install devkitARM on macOS or GNU/Linux with respect to your distro. Once you have {dkp-,}pacman, run {dkp-,}pacman -S devkit-env devkitARM devkitARM-gdb gba-tools general-tools grit to obtain all necessary tools and scripts for Byblos' agbhb and agbsp platforms.

MinGW and LLVM can be installed in the typical way with your package manager. Do not confuse LLVM with Xcode's clang on macOS! Byblos on macOS uses LLVM as installed with Homebrew. The system toolchain is only used to bootstrap OpenWatcom and compile Vala code for developers.

Getting OpenWatcom

Unfortunately the OpenWatcom 2 project does not provide distro packages or macOS binaries, so it must be built from source. Arch Linux users have the benefit of an AUR package that mostly automates this, and the pathing dictated by our step-by-step process here should be consistent with that script so there are no quirks in usage.

1. Run xcode-select --install
2. Follow brew.sh#install
3. Run brew install dosbox-x
4. Manually choose the latest release from the repository
   • Select the source code .tar.gz in particular
5. untar the source into a $dir of your choice
6. Set environment variable OWTOOLS=CLANG
7. Set environment variable OWDOSBOX=dosbox-x
8. Run export OWTOOLS OWDOSBOX
9. Run cd $dir
10. Run ./build.sh
11. Run ./build.sh rel
12. Run sudo mkdir /opt/watcom
13. Run sudo cp -a rel/. /opt/watcom/
14. Append the following lines to your .profile (or your equivalent to .profile):
    • WATCOM=/opt/watcom
    • EDDAT=$WATCOM/eddat
    • PATH="$PATH:$WATCOM/armo64"
    • export WATCOM EDDAT PATH
15. Run source .profile (or your equivalent to .profile)
16. Verify this all worked by running wcl386 -h

Installing the SDK

With the toolchains installed, setting up the rest of the Byblos SDK is a relatively anodyne affair. In general, all utilities honour the value of $BYBLOS as provenance of the SDK root. If it is empty or undefined, the value defaults to /opt/byblos. In any case, the $BYBLOS root has the following hierarchy:

$BYBLOS/ – SDK root

$BYBLOS/conf/ – configuration settings

$BYBLOS/include/ – header files (.h) and assembly headers (.inc)

$BYBLOS/man/ – manuals

$BYBLOS/sbin/ – script binaries (architecture independent)

$BYBLOS/share/ – shared resources and miscellanea

$BYBLOS/slib/ – script dependencies/libraries (architecture independent)

$BYBLOS/system/ – private folder for internal SDK business

$BYBLOS/target/$t/ – parent folder for target-specific sub-hierarchies ($t is the target ident)

$BYBLOS/target/$t/bin/ – runnable binaries

$BYBLOS/target/$t/lib/ – static libraries

$BYBLOS/target/$t/share/ – target-specific miscellanea (e.g. associated production materials, emulator settings, etc.)

Byblos 1 entirely avoids pre-compiling native programs, instead leaning on the externally-provided toolchains to run C code and Vala code in utilities as necessary. So, it lacks a host bin/ folder or associated lib/ folder, which are replaced by sbin/ and slib/ respectively. Hashbangs are used to achieve program invocation.

The SDK operates on the premise that the working development directory can be anywhere. SDK-specific helper scripts have provenance in $BYBLOS and those helper scripts should be written to correctly guess the necessary host toolchains and their locations or error out quickly on failure to find them.

Byblos 2

Byblos 2 is a purpose-built SDK that includes dedicated stand-alone tools for compiling code and assets for all supported targets. Initially it will run on the same hosts supported by Byblos 1, with wider support to coincide with the porting of Vala to liberate it from conventional host support structures. Its core is comprised of the following programs:

• Oración, an assembler and desymboliser
  • Intel 8086, i286, i486, i686, x86-64 v1-v4, ARMv4T, ARMv5TE, ARMv6, ARMv7, ARMv8, SPARC64 V9, Motorola 68000, MOS 6502, MOS 65816, and Itanium/IA-64 are the supported architectures
• FCC, an expressly non-optimising ANSI C compiler
  • Uses Oración as its backend
• Sirius C*, a provisional C* compiler
  • Derived from FCC, also uses Oración as its backend
• Inbound, a software build system
  • Derived from Make, uses a cut-down in-house derivative
• Outbound, a software packaging system
• NatPython, a scripting interpreter
  • Hard fork of Python 3.3
• Simbel, a documentation authoring and generation toolset
• Precursor, a hub-and-spoke version control system
• Forerunner, a source code management system built on top of Precursor
• Gauntlet, a public key cryptosystem for signing and verifying code
• Quindle, a graphical editor program for authoring code
• Cigarbochs, an emulator for debugging and testing code
  • Simulates x86, ARM, 68000, 65(02/816) and any bespoke A* ISAs
• Various helper utilities for sanitising and analysing code

Additionally, Byblos 2 integrates support for Vala using the default system compiler toolchain and libraries. This makes for an accessible and painless path to quickly writing GUI and CLI programs alike for use by developers to help author large, complex programs.