Venkateshprasad Bhat and Joris Kraak presented Shipping Julia Packages with System Images at JuliaCon 2023. Bhat discussed their innovative use of Julia system images, highlighting how system images have transformed the way that JuliaHub develops, ships, and maintains software.
These system images have created a streamlined development and deployment process that ensures consistent performance across different environments.
Reliability in Scalability
One of the standout advantages of system images is their reliability in scalability. The presentation highlighted that even when dealing with a substantial number of packages – upwards of 100 to 200 – the build process remains robust.
While it takes some time to build a large system image, the resulting system image is readily shippable. This scalability lends itself to quicker deployment and wider usage.
Consistency and Reproducibility
System images pave the way for an environment that remains consistent throughout the team. When team members share the same system image, this eliminates discrepancies caused by varying software setups.
Additionally, results are reproducible, making it easier to diagnose and fix errors. This feature is particularly valuable when multiple developers are working on the same feature simultaneously.
Private Packages and Code Obfuscation
For those dealing with proprietary or sensitive code, system images offer an elegant solution. Bhat explained that users can include private packages and even employ code obfuscation techniques within the image in these cases.
This means that the source code does not need to be shipped. This enhances security while also maintaining functionality.
How to Build Relocatable System Images
Building relocatable system images is a crucial aspect for ensuring seamless deployment across different machines. The approach included the following steps:
- Depot and CPU Target Alignment: Align the source depot path with that of the target machine. This alignment minimizes startup errors and improves startup times.
- Package Selection: Carefully select the CPU target packages to include in the system image, covering both primary dependencies and packages that users are likely to employ. This ensures a more robust and versatile system image.
- Pre-Compiling: To ensure that functions are pre-compiled, include unit tests or custom scripts in the pre-compile phase. This step optimizes startup times and overall performance.
- Preference Management: Leverage preferences for customizing behavior during pre-compilation, ensuring that critical functionalities are optimized and ready to use.
- Avoid Hard-Coded Paths: While it's best to avoid hard-coded file paths, if necessary, opt for relocatable folders or artifacts using RelocatableFolders.jl to allow for greater flexibility and compatibility.
- Integrating Libraries: If certain libraries cannot be installed on the target machine, explore JLL packages. These self-contained libraries can be added to the library path, simplifying the deployment process.
The system images with packages baked in, provide users a feature-rich environment for usage and building off of these features. It was noted that while developers are tinkering with the package and developing core functionalities, recently introduced package images could provide flexibility with speed. This highlights JuliaHub's commitment to continuous improvement and adaptability to different development needs.
By sharing their experience at JuliaCon, the presenters have shown how robust Julia’s ecosystem is in productizing the packages. As the Julia ecosystem evolves, there is much anticipation for the future possibilities.
View the full presentation here.
