.Dd December 29, 2015
.Dt DEVELOPMENT 7
.Os
.Sh NAME
.Nm development
.Nd operating system development instructions
.Sh SYNOPSIS
.Pa /src
.Sh DESCRIPTION
Releases come with the system source code in
.Pa /src
as a
.Xr git 1
repository.
It can be modified, compiled and installed on the current system.
The source code is built with a
.Xr make 1
build system.
The source code can be located in any location, if so, simply substitute
.Pa /src
with the real location.
These instructions only apply to building the operating system from within
itself, those building it from another operating system needs to follow
.Xr cross-development 7
instead.
.Pp
If you are building a new version of the operation system where build tools have
been added or changed, you first need to install the new tools.
This is not needed when building the matching release.
To do so, run as root:
.Bd -literal
    cd /src
    make distclean            # fully clean build directory
    make install-build-tools  # install new tools
    make clean-build-tools    # clean for real build below
.Ed
.Pp
To build the base operating system and upgrade the current
system, run as root:
.Bd -literal
    cd /src
    make            # build new operating system in /src/sysroot
    make sysmerge   # upgrade current operating system with /src/sysroot
.Ed
.Pp
The build system creates a minimal root filesystem structure in the
.Pa /src/sysroot
and builds each operating system component in turn, installing them into the
sysroot.
The ports in the
.Pa /src/ports
are built automatically by downloading the upstream releases and applying the
appropriate patches.
The result is a system image that can be turned into working system by adding
important configuration such as
.Xr passwd 5 .
.Pp
The
.Sy sysmerge
make target ensures a system is built in
.Pa /src/sysroot
and then runs the
.Xr sysmerge 8
program which installs the new system files onto the existing system.
It updates the system manifest as well all ports installed in the sysroot.
The
.Xr initrd 7
is automatically regenerated using
.Xr update-initrd 8 .
The bootloader, if enabled in
.Xr upgrade.conf 5 ,
is reinstalled and configured as necessary.
The new user-space is running on completion, though existing processes will be
running the old programs.
A reboot is needed to run the new kernel.
If the ABI changed and the current kernel isn't able to run the new programs,
then the upgrade is delayed and will be automatically completed on the next
boot.
The
.Sy sysmerge-wait
make target forces waiting until the next boot.
.Ss Root Makefile
The
.Pa /src/Makefile
handles the high level build of the operating system.
The important targets are:
.Bl -tag -width "12345678"
.It Sy all
(default) Build each component in turn and install them into the sysroot.
.It Sy available-ports
Search for newer available versions of ports.
.It Sy build-tools
Make all build tools.
.It Sy clean
Clean the component directories and the port source code.
.Sy ( clean-core , clean-ports )
.It Sy clean-build-tools
Clean the directories of all build tools.
.It Sy clean-cross-compiler
Clean the directories for the cross-compiler.
.It Sy clean-cross-toolchain
Clean the directories for the build tools and cross-compiler.
.Sy ( clean-build-tools , clean-cross-compiler )
.It Sy distclean
Run every clean target such that the source code is ready for distribution.
.Sy ( clean-builds , clean-core , clean-mirror , clean-release , clean-repository , clean-sysroot , distclean-ports, clean-cross-compiler )
.It Sy distclean-ports
Remove the port source code extractions.
.It Sy extract-ports
Extract the upstream release for each
.Xr port 5
in
.Pa /src/ports
and apply the appropriate patches.
.It Sy install-build-tools
Install all build tools after making them.
.It Sy install-cross-compiler
Install the cross-compiler after making it.
.It Sy install-cross-toolchain
Install the build tools and cross-compiler after making them.
.Sy ( install-build-tools , install-cross-compiler )
.It Sy iso
Create a release iso in the
.Pa /src/builds
directory after making
.Sy all .
.It Sy mirror
Download the upstream release for each
.Xr port 5
in
.Pa /src/ports
from
.Ev SORTIX_PORTS_MIRROR
into the
.Pa /src/mirror
directory.
The operating system can be built without network connectivity once the mirror
is populated.
The local mirror has the same structure as the remote mirror and can be used as
a remote mirror.
.It Sy mostlyclean
Clean everything except binary packages and the mirror of upstream releases.
.Sy ( clean-builds , clean-ports , clean-release , clean-sysroot , distclean-ports, clean-cross-compiler )
.It Sy presubmit
Verify the port configuration
.Sy ( verify-ports ) ,
the coding style is followed
.Sy ( verify-coding-style ) ,
the manual pages does not have lints
.Sy ( verify-manual ) ,
the build tools compile
.Sy ( verify-build-tools ) ,
that everything compiles without warnings on all architectures
.Sy ( verify-build ) ,
and the system headers works in all supported configurations
.Sy ( verify-headers ) .
.It Sy release
Make
.Sy iso
and construct release directory tree in
.Pa /src/release
suitable for online publishing.
.It Sy sortix.iso
Make
.Sy iso
and place it in the current directory as
.Pa sortix.iso .
.It Sy sysmerge
Upgrade the current operating system using the sysroot after making the
.Sy all
target.
.It Sy sysmerge-full
Like
.Sy sysmerge
but do a full operating system upgrade that uninstalls ports not present in the
sysroot using
.Fl \-full .
.It Sy sysmerge-full-wait
The combination of
.Sy sysmerge-full
and
.Sy sysmerge-full-wait .
.It Sy sysmerge-wait
Like
.Sy sysmerge
but delay the upgrade until the next boot using
.Fl \-wait .
.It Sy sysroot-base-headers
Create the sysroot and install only the headers of the standard library and
kernel into it.
This is useful when bootstrapping the runtime libraries of the compiler that
need to know about libc prior to building libc.
.It Sy upgrade-ports
Search for newer available versions of ports and update the
.Sy VERSION
variable in the
.Xr port 5
and switch it into development mode.
.El
.Pp
The important environment variables influencing the Makefile are:
.Bl -tag -width "12345678"
.It Ev BUILD
The platform of the current operating system.
This defaults to the current machine and operating system.
.It Ev HOST
Specifies platform on which the compiled code will run.
This defaults to the current machine and operating system.
This is used when cross-compiling the operating system.
When cross-compiling the operating system, it must be set to one of
.Sy i686-sortix
and
.Sy x86_64-sortix .
This must be unset when building the build tools as they run on the current
operating system.
The compiler tools are prefixed with this variable if it does not match
.Ev BUILD.
.It Ev OPTLEVEL
Specifies compiler optimization options that gets added to
.Ev CFLAGS
and
.Ev CXXFLAGS .
.It Ev PACKAGES
Specifies which ports to build, or all ports if unset.
Suffixing a port with a
.Sq "!"
includes its mandatory dependencies, and suffixing it with
.Sq !!
includes the optional dependencies as well.
The sets of ports defined in
.Pa /src/build-aux/ports.conf
can be specified as well.
For instance
.Sy PACKAGES=minimal!
builds all the minimal ports with no optional dependencies.
.It Ev SORTIX_INCLUDE_SOURCE
Specifies whether the source code is included in the sysroot.
This must be one of
.Sy no , yes
or
.Sy git
and defaults to
.Sy git
if
.Xr git 1
is installed and
.Sy yes
otherwise.
.It Ev SORTIX_ISO_COMPRESSION
Specifies the compression algorithm used in iso files.
This must be one of
.Sy none , gzip
or
.Sy xz
and defaults to
.Sy xz .
.It Ev SORTIX_PORTS_MIRROR
Upstream releases of ports are downloaded from this mirror, defaulting to the
official mirror, and falling back to the upstream release site if the mirror
failed.
.El
.Ss Components
The operating systems components, such as libc and the kernel, each have their
own directory by that name.
It contains a makefile that can build and install that component.
This allows building and installing only that component onto the current
operating system.
.Pp
For instance, to build and install libc, run as root:
.Bd -literal
    cd /src/libc
    make
    make install
.Ed
.Pp
Note the individual makefiles only install the new system files and leak any
files that don't exist anymore; and they also don't run any upgrade hooks to
migrate the current system.
This mechanism isn't supported unless you are building the same source code as
the current operating system.
The global
.Sy sysmerge
makefile targets should be used instead as the supported mechanism for operating
system upgrades.
.Pp
System libraries are statically linked and you will have to relink programs with
the new library for changes to take effect.
Building the whole operating system from the root makefile ensures components
are built in the right order such that all programs use fresh libraries.
The root makefile invokes component makefiles with
.Ev SYSROOT
set to
.Pa /src/sysroot
to force the compiler to locate files there.
Likewise when installing, it sets
.Ev DESTDIR
to
.Pa /src/sysroot
to make it install files there.
.Ss Directories
In addition to the directories for each operating system component, there are
these special directories:
.Bl -tag -width "12345678"
.It Pa /src/builds
The build artifacts produced when building the operating system.
.It Pa /src/mirror
The upstream releases of ports are downloaded on the first use and cached inside
this mirror directory.
.It Pa /src/ports
Each subdirectory contains a
.Xr port 5
which is automatically built along with the operating system per the
.Ev PACKAGES
environment variable.
.It Pa /src/release
The
.Sy release
root makefile target creates this directory and populates it with a directory
structure suitable for online publishing of a release.
.It Pa /src/repository
This directory stores the binary packages and is created when they are built.
This works as a cache so ports don't have to be rebuilt every time the operating
system is.
Packages are also copied from here rather than the sysroot when making releases.
.It Pa /src/sysroot
This directory is made when building the operating system and the freshly made
files are installed here.
The build system uses this as the system root which forces the compiler to look
here for headers and libraries.
This ensures a clean bootstrap where files from the current operating system do
not leak into the new system.
.It Pa /src/sysroot-overlay
If this directory exists, it is added to the initrd of the produced iso and can
contain additional system files.
.El
.Ss Build Tools
Some components are used to build the source code and must match the versions in
the source code being built.
These are currently:
.Pp
.Bl -bullet -compact
.It
carray
.It
kblayout-compiler
.It
mkinitrd
.It
sf
.It
tix
.El
.Pp
If the currently installed versions of those tools are older than the ones in
the source code, you must update them.
The
.Sy clean-build-tools
root makefile target cleans the applicable directories, the
.Sy build-tools
root makefile target builds them from the source code, and the
.Sy install-build-tools
root makefile target installs the new version.
You must clean the compiled files from the source code afterwards because the
compiled tools are intended to run on the current system, and have not been
built properly using
.Pa /src/sysroot .
.Ss Ports
Each
.Xr port 5
in the
.Pa /src/ports
directory will get built automatically when and installed into the sysroot when
building the whole operating system.
Installable binary packages are created in the
.Pa /src/repository/$HOST
directory using
.Xr tix-port 8
and can be installed with
.Xr tix-install 8 .
If an existing binary package with the right version exists in the repository,
it is used instead of the building the port again.
.Pp
The ports system workflow is described in
.Xr porting 7 .
.Ss Patches
The source code is managed as a
.Xr git 1
repository and you can make your own changes and commit them.
A good approach is to set up your own local development branch and work there:
.Bd -literal
    git checkout -b local
    git add utils/hello.c
    git commit -m 'Add hello(1).'
.Ed
.Pp
The
.Sy presubmit
makefile target can be used to verify your work needs some of the development
conventions.
.Pp
Prepare a set of patches suitable for upstream submission and submit a merge
request to the upstream project.
.Pp
If your installation does not have network connectivity, you will need to
submit the changes from another system.
If you are dual booting and have another operating system with network
connectivity, you can boot into the other operating system and mount the
appropriate filesystem from there.
If you have a serial line, you can produce a set of .patch files containing
your changes with
.Bd -literal
    git format-patch master..local
.Ed
.Pp
and then transfer them over the serial connection as described in
.Xr serial-transfer 7 .
.Ss Releases
CD-ROM release of the operating system can be built with the
.Sy iso
root makefile target.
This will build the whole operating system, if not done already, and produce a
bootable iso for the current architecture in the
.Pa /src/builds
directory.
The
.Sy sortix.iso
root makefile target will do the above and place a
.Pa sortix.iso
file in the current directory.
.Pp
The
.Sy release
root makefile target will run the
.Sy iso
target and prepare a
.Pa /src/release
directory with a directory structure and miscellaneous files suitable for a
formal online release.
.Ss Following Development
The
.Xr following-development 7
manual page documents what needs to be done to stay updated with the latest
developments.
You will need to read the new version of that document whenever you update the
source code.
.Sh SEE ALSO
.Xr git 1 ,
.Xr make 1 ,
.Xr port 5 ,
.Xr cross-development 7 ,
.Xr following-development 7 ,
.Xr installation 7 ,
.Xr porting 7 ,
.Xr serial-transfer 7 ,
.Xr upgrade 7 ,
.Xr sysinstall 8 ,
.Xr sysmerge 8 ,
.Xr update-initrd 8