One of the core needs software development teams have is the ability to work in parallel on building and updating a piece of software. Teams of scientists, machine learning practitioners, and R&D engineers also have the same needs. And as the complexity of projects grows, so too does the need to manage workflows and changes to the core software with teams across departments. With this in mind, we’ve just made a major upgrade to the projects feature on JuliaHub.
A project in JuliaHub is a collection of resources such as directories, folders, files and datasets and allows you to invite others into your project workspace and collaborate on that set of resources. Previously, the projects feature allowed teams to import a repository from a git-provider such as GitHub, GitLab, or Bitbucket and make those files available to anyone on a particular JuliaHub instance. This was sufficient, but really lacked a lot of core features that would help teams manage collective changes to those files.
The major update to how projects now works includes team access controls, a web interface that lets you create your directory structure directly in JuliaHub itself, and allows you to upload files and datasets without having to launch the JuliaIDE. This new projects paradigm now works as a focal point to how you interact with JuliaHub overall. When you create a new project, you then assign team members and groups to that project. This essentially means that only the users within the project can access any files, datasets, and folders that are part of that same project. This is a way to provide a clear permissioning mechanism on JuliaHub. So, how does this all actually work?
How does the JuliaHub Projects feature work?
In the top menu, under the Compute tab, there is a projects dropdown link. Select it and you go to an area that allows you to create a project or view projects created by others in your JuliaHub instance. Each project is essentially a container for a bunch of resources like directories or collections of datasets. Each top-level resource on the project (what we call a directory) also has separate access controls so you can set the permissions to be granular even inside the project itself.
Additionally, when it comes to editing files, the paradigm is that there is one shared/global workspace that holds the files and folders across the project, but for each individual who wants to edit those files, they get their own workspace which is a local working copy of the project files. In your local workspace, you open your preferred IDE to edit your working copy and then you hit a publish button to synchronize your changes back to the shared workspace of the project. We use git behind the scenes for this functionality which allows for rollback and version control.
How do Team Access Controls work in JuliaHub Projects?
JuliaHub now has the ability to populate groups using your organization’s identity provider. This then allows members of those groups to be assigned access to a project. If you do not have groups available via an identity provider, you can add your JuliaHub users one at a time to any project. Each group or user can be given a role - which is a specific set of access controls within the project. Each project has a default set of permissions such as Viewer, Editor, Owner and each group or user can have a different set of permissions as well.
In fact, you can even set permission on top-level resources. For example, you can have datasets that not everyone in the project can see. Since access control is resource based, different people can be given permissions to those resources differently within the project. These new features make privacy easy.
Building a Directory and Folder Structure
Currently, when you start a new project, you can create two kinds of top-level folder structures. The first are dataset directories that can store large datasets. The second kind are general directories for everything else: code, files, pdfs, text documents, images. You can manually upload files to JuliaHub or launch the JuliaIDE and drag and drop files or create new files and folders. We don’t have templating at this time, but we are planning on adding this in the future - a way to clone an existing project so you can re-use the directory and folder structure again in a new project.
The best part of these new features are that you can now work collaboratively with other team members on the same shared/global project. For example, if Mark wants to edit files in the project, he can get a clone of the project (a local workspace), make his changes, and then when ready, commit those changes back to the shared workspace of the project. Mindy can work in parallel and commit her changes too. Changes by any one team member in one application can be saved to the shared workspace and then be available to other team members.
This functionality is git-based and this ensures that upstream changes never discounts anything for good - your admin can always rollback a commit and go back to a previous iteration of your files and folders. This also means that JuliaHub is recording every change and can thus, trace events to every artifact (dataset, folder, file, package).
Traceability
This traceability will be available in a Dashboard that provides administrators a way to track the lineage of artifacts within your project.
For example, you can select any dataset and see the time and date that dataSet was changed, the event that took place, and the user that made the changes. This format is more human-readable than raw audit log information and has filter mechanisms that allow you to download only the data you need in csv format.
Projects and Team Management features are launching this week on paid subscriptions, but we do plan on rolling out a version for our free version in the next month or two.
If you’d like to learn more about how these features work, you can refer to the documentation or watch the collaboration webinar that shows a bit more of these features in action.
Want to check out JuliaHub? Start for free with our standard tier.
Interested in taking Projects for a spin? Contact Sales to learn more.