456b313665
## Issue Addressed NA ## Proposed Changes Modify the configuration of [GNU malloc](https://www.gnu.org/software/libc/manual/html_node/The-GNU-Allocator.html) to reduce memory footprint. - Set `M_ARENA_MAX` to 4. - This reduces memory fragmentation at the cost of contention between threads. - Set `M_MMAP_THRESHOLD` to 2mb - This means that any allocation >= 2mb is allocated via an anonymous mmap, instead of on the heap/arena. This reduces memory fragmentation since we don't need to keep growing the heap to find big contiguous slabs of free memory. - ~~Run `malloc_trim` every 60 seconds.~~ - ~~This shaves unused memory from the top of the heap, preventing the heap from constantly growing.~~ - Removed, see: https://github.com/sigp/lighthouse/pull/2299#issuecomment-825322646 *Note: this only provides memory savings on the Linux (glibc) platform.* ## Additional Info I'm going to close #2288 in favor of this for the following reasons: - I've managed to get the memory footprint *smaller* here than with jemalloc. - This PR seems to be less of a dramatic change than bringing in the jemalloc dep. - The changes in this PR are strictly runtime changes, so we can create CLI flags which disable them completely. Since this change is wide-reaching and complex, it's nice to have an easy "escape hatch" if there are undesired consequences. ## TODO - [x] Allow configuration via CLI flags - [x] Test on Mac - [x] Test on RasPi. - [x] Determine if GNU malloc is present? - I'm not quite sure how to detect for glibc.. This issue suggests we can't really: https://github.com/rust-lang/rust/issues/33244 - [x] Make a clear argument regarding the affect of this on CPU utilization. - [x] Test with higher `M_ARENA_MAX` values. - [x] Test with longer trim intervals - [x] Add some stats about memory savings - [x] Remove `malloc_trim` calls & code
51 lines
1.5 KiB
Rust
51 lines
1.5 KiB
Rust
use beacon_chain::test_utils::EphemeralHarnessType;
|
|
use environment::null_logger;
|
|
use http_metrics::Config;
|
|
use reqwest::StatusCode;
|
|
use std::net::Ipv4Addr;
|
|
use std::sync::Arc;
|
|
use tokio::sync::oneshot;
|
|
use types::MainnetEthSpec;
|
|
|
|
type Context = http_metrics::Context<EphemeralHarnessType<MainnetEthSpec>>;
|
|
|
|
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
|
async fn returns_200_ok() {
|
|
async {
|
|
let log = null_logger().unwrap();
|
|
|
|
let context = Arc::new(Context {
|
|
config: Config {
|
|
enabled: true,
|
|
listen_addr: Ipv4Addr::new(127, 0, 0, 1),
|
|
listen_port: 0,
|
|
allow_origin: None,
|
|
allocator_metrics_enabled: true,
|
|
},
|
|
chain: None,
|
|
db_path: None,
|
|
freezer_db_path: None,
|
|
log,
|
|
});
|
|
|
|
let ctx = context.clone();
|
|
let (_shutdown_tx, shutdown_rx) = oneshot::channel::<()>();
|
|
let server_shutdown = async {
|
|
// It's not really interesting why this triggered, just that it happened.
|
|
let _ = shutdown_rx.await;
|
|
};
|
|
let (listening_socket, server) = http_metrics::serve(ctx, server_shutdown).unwrap();
|
|
|
|
tokio::spawn(async { server.await });
|
|
|
|
let url = format!(
|
|
"http://{}:{}/metrics",
|
|
listening_socket.ip(),
|
|
listening_socket.port()
|
|
);
|
|
|
|
assert_eq!(reqwest::get(&url).await.unwrap().status(), StatusCode::OK);
|
|
}
|
|
.await
|
|
}
|