isthmus introduction eid architecture adding recipes misc plan

2017-11-16 03:48

i486:
lyd 1rootfs 1eid 7

arm:eid 7/isthmus/recipes/x264 snapshot-20170126-2245-stable/isthmus/recipes/x264 snapshot-20170126-2245-stable

commit 307df1f006974f7fde2ff87053bc2a0354fedac0
Author: Øyvind Kolås 
Date:   Thu Nov 16 03:46:02 2017 +0100

    eid: install package containing bin with name of symlink

commit 8a20289f253478bbf32450f94984fe68ab57eda3
Author: Øyvind Kolås 
Date:   Tue Oct 31 15:57:22 2017 +0100

    x264: add ncurses as build-dep

commit cbc0bcfa093667e9a565cfcb39559df16f161569
Author: Øyvind Kolås 
Date:   Tue Oct 31 00:01:20 2017 +0100

    rc.disk: strip out .0 from computed size

plan

broken things:

better containment than chroots wanted; proot breaks things
when upgrading a package that changes the soname of a library - dependent packages are not rebuilt automatically
recipies fail to add due to deps failing, : mostly due to failing / incomplete but usable packages : failing but complete packages returning non-zero exit from build_one!
luajit depends on libgcc which isn't found when just luajit gets installed
git doesnt work with qemu (the same binaries worked on actual arm) : can this be neon instructions?
if updates have been done in chroot, updates outside seem broken - probably need to revisit the hard-link updating
icu needs g++ with namespace support
figlet installs files in /usr/local

reproducable builds

i486 on a 32bit / 64bit host, builds seem to be bit-output repeatable, but builds are not repeatable between 32bit and 64bit build hosts. offsets are shifted depending on whether it was generate on 32 or 64bit.
"gcc varies offset allocations on compiles, so unrepeatable"(http://eid.cool/stage/gcc-unrepeatable-build.png)
old timestamp makes very verbose unpacking
make an abstraction that permits switching between proot, bubblewrap, chroot abstractions

missing features:

compacting for master rather than slaves, maybe only doing log compacting?
config: build only from source / fetch binaries only
config: (auto)compacting the log
generating -dev packages, containing the .a .h and perhaps more files
ensure that bootstrapping isthmus from within isthmus is possible
use IPFS as part of CDN
auto install curl if needing to fetch from https (which is more and more common)

improvements:

make package builds use temporary work folder, and install deps as asked, thus ensuring a clean environment; and getting closer to reproducible builds.
make git checkouts store the hash in place of tarball-hash in build-log
add switches to get build output straight to console,
add switches to keep workfiles around in /tmp/isthmus.$USER
display name of file being fetched before curl/wget progress, and how many files will be processed this aids usability in constrained-bandwidth and limited resource environs : this already done for updates and source downloads
depend less on $SUDO in file system building tasks - maybe it is possible to have all eid activities for managing a system happen without root access? : use systemd-nspawn, bubblewrap or unshare to contain build without sudo or fakeroot
upgrade to newer "long term support" kernel

document - soft and hard depenencies - the internal builds are more important than cross builds, packages should rebuild cleanly from within jail, including core set of gcc/binutils/make - MAKE_FLAGS - quirks - distcc configuration of debian/arch/redhat hosts

desirable recipes:

ipfs systemd wayland snabbswitch luvit clang