Fix docker build slowness and disk bloat #35

Merged
telackey merged 1 commits from dboreham/use-overlay2-docker-driver into main 2023-04-21 21:49:05 +00:00
Owner

Mount an anonymous volume at /var/lib/docker. This has the side effect of causing the docker daemon inside the executor container to use overlay2 for its storage driver. Without this docker uses vfs for its storage driver. The impact of this is significant on the container build process because docker relies on the storage driver to implement a layer delta storage scheme. Without that feature in the storage driver, a fall back implementation where each layer is a complete snapshot of the entire image contents is used. This means for example that when building a 1G image with 10 layers, you end up with 10G of data, copied 10 times. But if overlay2 is used, with its capability to store only image differences, the same workload results in approximately 1G of storage and almost no copying.

Mount an anonymous volume at /var/lib/docker. This has the side effect of causing the docker daemon inside the executor container to use overlay2 for its storage driver. Without this docker uses vfs for its storage driver. The impact of this is significant on the container build process because docker relies on the storage driver to implement a layer delta storage scheme. Without that feature in the storage driver, a fall back implementation where each layer is a complete snapshot of the entire image contents is used. This means for example that when building a 1G image with 10 layers, you end up with 10G of data, copied 10 times. But if overlay2 is used, with its capability to store only image differences, the same workload results in approximately 1G of storage and almost no copying.
telackey reviewed 2023-04-21 19:09:33 +00:00
Sign in to join this conversation.
No reviewers
No Milestone
No project
No Assignees
2 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: cerc-io/hosting#35
No description provided.