cerc-io/lighthouse
16
0
Fork 0
Code Issues 1 Pull Requests Packages Projects Releases Wiki Activity
456b313665
lighthouse/common
History
Paul Hauner 456b313665 Tune GNU malloc (#2299) 
## Issue Addressed

NA

## Proposed Changes

Modify the configuration of [GNU malloc](https://www.gnu.org/software/libc/manual/html_node/The-GNU-Allocator.html) to reduce memory footprint.

- Set `M_ARENA_MAX` to 4.
    - This reduces memory fragmentation at the cost of contention between threads.
- Set `M_MMAP_THRESHOLD` to 2mb
    - This means that any allocation >= 2mb is allocated via an anonymous mmap, instead of on the heap/arena. This reduces memory fragmentation since we don't need to keep growing the heap to find big contiguous slabs of free memory.
- ~~Run `malloc_trim` every 60 seconds.~~
    - ~~This shaves unused memory from the top of the heap, preventing the heap from constantly growing.~~
    - Removed, see: https://github.com/sigp/lighthouse/pull/2299#issuecomment-825322646

*Note: this only provides memory savings on the Linux (glibc) platform.*
    
## Additional Info

I'm going to close #2288 in favor of this for the following reasons:

- I've managed to get the memory footprint *smaller* here than with jemalloc.
- This PR seems to be less of a dramatic change than bringing in the jemalloc dep.
- The changes in this PR are strictly runtime changes, so we can create CLI flags which disable them completely. Since this change is wide-reaching and complex, it's nice to have an easy "escape hatch" if there are undesired consequences.

## TODO

- [x] Allow configuration via CLI flags
- [x] Test on Mac
- [x] Test on RasPi.
- [x] Determine if GNU malloc is present?
    - I'm not quite sure how to detect for glibc.. This issue suggests we can't really: https://github.com/rust-lang/rust/issues/33244
- [x] Make a clear argument regarding the affect of this on CPU utilization.
- [x] Test with higher `M_ARENA_MAX` values.
- [x] Test with longer trim intervals
- [x] Add some stats about memory savings
- [x] Remove `malloc_trim` calls & code
2021-05-28 05:59:45 +00:00
..
account_utils Enable Compatibility with Windows (#2333) 2021-05-19 23:05:16 +00:00
clap_utils Tune GNU malloc (#2299) 2021-05-28 05:59:45 +00:00
compare_fields Check for unused deps in CI (#1262) 2020-06-14 10:59:50 +10:00
compare_fields_derive Update external deps (#1711) 2020-10-05 08:22:19 +00:00
deposit_contract Allow specifying alternative url for deposit_contract (#2295) 2021-04-16 06:47:34 +00:00
directory fix default network handling (#2029) 2020-12-08 05:41:10 +00:00
eth2 Monitoring service api (#2251) 2021-05-26 05:58:41 +00:00
eth2_config Add prater testnet config (#2260) 2021-03-17 00:47:06 +00:00
eth2_interop_keypairs Fix new clippy lints (#2036) 2020-12-03 01:10:26 +00:00
eth2_network_config v1.2.1 (#2263) 2021-03-18 04:20:46 +00:00
eth2_wallet_manager Use OS file locks in validator client (#1958) 2020-11-26 11:25:46 +00:00
fallback Fallback nodes for eth1 access (#1918) 2020-11-27 08:37:44 +00:00
filesystem Enable Compatibility with Windows (#2333) 2021-05-19 23:05:16 +00:00
hashset_delay Update to tokio 1.1 (#2172) 2021-02-10 23:29:49 +00:00
lighthouse_metrics Monitoring service api (#2251) 2021-05-26 05:58:41 +00:00
lighthouse_version v1.3.0 (#2310) 2021-04-13 22:46:34 +00:00
lockfile Enable Compatibility with Windows (#2333) 2021-05-19 23:05:16 +00:00
logging Update external deps (#1711) 2020-10-05 08:22:19 +00:00
lru_cache Keep track of failed head chains and prevent re-lookups (#1534) 2020-08-18 03:54:09 +00:00
malloc_utils Tune GNU malloc (#2299) 2021-05-28 05:59:45 +00:00
monitoring_api Monitoring service api (#2251) 2021-05-26 05:58:41 +00:00
remote_signer_consumer Enable Compatibility with Windows (#2333) 2021-05-19 23:05:16 +00:00
rest_types Update to tokio 1.1 (#2172) 2021-02-10 23:29:49 +00:00
sensitive_url Add SensitiveUrl to redact user secrets from endpoints (#2326) 2021-05-04 01:59:51 +00:00
slot_clock Add specific log and metric for delayed blocks (#2308) 2021-04-13 02:16:59 +00:00
task_executor Add testing for beacon node and validator client CLI flags (#2311) 2021-05-06 00:36:22 +00:00
test_random_derive Update external deps (#1711) 2020-10-05 08:22:19 +00:00
validator_dir Enable Compatibility with Windows (#2333) 2021-05-19 23:05:16 +00:00
warp_utils Monitoring service api (#2251) 2021-05-26 05:58:41 +00:00
README.md Directory Restructure (#1163) 2020-05-18 21:24:23 +10:00

README.md

eth2

Common crates containing eth2-specific logic.