1 unstable release
|0.1.0||Nov 20, 2021|
#146 in macOS and iOS APIs
Apple Open Source Downloader
This repository defines a Rust crate and CLI program to automate the downloading of Apple's open source code from https://opensource.apple.com/.
The primary goal of this project is to enable more intuitive usage and inspection of Apple's open source code. Using this tool you can:
- Convert the history of an Apple component (like the
xnucore OS primitives) to a Git repository and easily view differences between releases.
- Convert the history of a set of open source components (such as everything comprising macOS) to a Git repository and easily view differences between releases.
- Query for all available open source components and releases.
The canonical home for this project is https://github.com/indygreg/apple-opensource-downloader. Please report issues or submit enhancements there.
# From crates.io $ cargo install apple-opensource-downloader # From Git $ cargo install --git https://github.com/indygreg/apple-opensource-downloader.git --branch main
apple-opensource-downloader CLI is provided. It defines sub-commands to
perform various actions. Run
apple-opensource-downloader help to see the help.
Download a Single Component to a Git Repository
component-to-git sub-command will download all versions of a named
software component (see the component list at
https://opensource.apple.com/tarballs/) and write their contents as Git commits
to a Git repository. This allows you to see differences between the versions of
$ apple-opensource-downloader component-to-git --no-bare xnu aos/xnu <wait for this to finish> $ cd aos/xnu $ git log commit c2011455c3d75195791bd20d189abae4917c8c81 (HEAD -> main, tag: 7195.141.2) Author: Apple Open Source <email@example.com> Date: Fri Jan 1 00:00:00 2021 +0000 xnu 7195.141.2 Downloaded from https://opensource.apple.com/tarballs/xnu/xnu-7195.141.2.tar.gz commit e76ea20b5519ae2eaf2b74698bd2331141e028fa (tag: 7195.121.3) Author: Apple Open Source <firstname.lastname@example.org> Date: Fri Jan 1 00:00:00 2021 +0000 xnu 7195.121.3 Downloaded from https://opensource.apple.com/tarballs/xnu/xnu-7195.121.3.tar.gz ...
The Git trees and commit objects should be deterministic provided that the version of this software is identical and the Apple-hosted source archives don't change. i.e. different machines should produce Git commits with the same commit IDs.
Download all Components to Git Repositories
components-to-gits sub-command will download each available component and
write each to separate Git repositories. It is equivalent to running
component-to-git for every named component.
Download An Apple Software Release to a Git Repository
release-to-git command can be used to download all components in a logical
Apple software release (such as macOS or iOS) to a Git repository. This enables
you to view the history and changes of open source components between releases.
$ apple-opensource-downloader release-to-git --no-bare macos aos/macOS $ cd aos/macOS $ git log
The HTML parsing isn't the most robust and may not scrape all available software.
If Apple changes the HTML on opensource.apple.com, it will break this tool.
If Apple imposes throttling on their servers, it will likely break this tool.
Created Git repositories don't use packfiles and their performance may be
git gc after Git repo creation to optimize the Git repositories.
We don't support incrementally updating Git repositories. Git repositories have their history recreated from scratch on every invocation. This is obviously inefficient.
Various advertised URLs on opensource.apple.com result in an HTTP 404. These are sometimes ignored by this tool.
When importing software releases (such as macOS), the components from one release
to the next may vary. e.g. SQLite could be there in release A, gone in release B,
and reappear in release C. This may make
git diff output non-representative.
Git commits have a hard-coded date that has no basis in reality.
The naming and layout of Apple's components can at times be confusing and inconsistent. We don't yet make an effort to reconcile this.
The content downloaded by this tool may be governed by license and usage restrictions defined outside this tool. Check for usage restrictions posted at https://opensource.apple.com/ and within the downloaded content.
i.e. if you redistribute the downloaded content, Apple may take an issue with that.