lighthouse/common/clap_utils/src/lib.rs

118 lines
4.1 KiB
Rust
Raw Normal View History

//! A helper library for parsing values from `clap::ArgMatches`.
use clap::ArgMatches;
use eth2_network_config::Eth2NetworkConfig;
use ssz::Decode;
use std::path::PathBuf;
use std::str::FromStr;
Tune GNU malloc (#2299) ## Issue Addressed NA ## Proposed Changes Modify the configuration of [GNU malloc](https://www.gnu.org/software/libc/manual/html_node/The-GNU-Allocator.html) to reduce memory footprint. - Set `M_ARENA_MAX` to 4. - This reduces memory fragmentation at the cost of contention between threads. - Set `M_MMAP_THRESHOLD` to 2mb - This means that any allocation >= 2mb is allocated via an anonymous mmap, instead of on the heap/arena. This reduces memory fragmentation since we don't need to keep growing the heap to find big contiguous slabs of free memory. - ~~Run `malloc_trim` every 60 seconds.~~ - ~~This shaves unused memory from the top of the heap, preventing the heap from constantly growing.~~ - Removed, see: https://github.com/sigp/lighthouse/pull/2299#issuecomment-825322646 *Note: this only provides memory savings on the Linux (glibc) platform.* ## Additional Info I'm going to close #2288 in favor of this for the following reasons: - I've managed to get the memory footprint *smaller* here than with jemalloc. - This PR seems to be less of a dramatic change than bringing in the jemalloc dep. - The changes in this PR are strictly runtime changes, so we can create CLI flags which disable them completely. Since this change is wide-reaching and complex, it's nice to have an easy "escape hatch" if there are undesired consequences. ## TODO - [x] Allow configuration via CLI flags - [x] Test on Mac - [x] Test on RasPi. - [x] Determine if GNU malloc is present? - I'm not quite sure how to detect for glibc.. This issue suggests we can't really: https://github.com/rust-lang/rust/issues/33244 - [x] Make a clear argument regarding the affect of this on CPU utilization. - [x] Test with higher `M_ARENA_MAX` values. - [x] Test with longer trim intervals - [x] Add some stats about memory savings - [x] Remove `malloc_trim` calls & code
2021-05-28 05:59:45 +00:00
pub mod flags;
pub const BAD_TESTNET_DIR_MESSAGE: &str = "The hard-coded testnet directory was invalid. \
This happens when Lighthouse is migrating between spec versions \
or when there is no default public network to connect to. \
During these times you must specify a --testnet-dir.";
/// Attempts to load the testnet dir at the path if `name` is in `matches`, returning an error if
/// the path cannot be found or the testnet dir is invalid.
pub fn parse_testnet_dir(
matches: &ArgMatches,
name: &'static str,
) -> Result<Option<Eth2NetworkConfig>, String> {
let path = parse_required::<PathBuf>(matches, name)?;
Eth2NetworkConfig::load(path.clone())
.map_err(|e| format!("Unable to open testnet dir at {:?}: {}", path, e))
.map(Some)
}
/// Attempts to load a hardcoded network config if `name` is in `matches`, returning an error if
/// the name is not a valid network name.
pub fn parse_hardcoded_network(
matches: &ArgMatches,
name: &str,
) -> Result<Option<Eth2NetworkConfig>, String> {
let network_name = parse_required::<String>(matches, name)?;
Eth2NetworkConfig::constant(network_name.as_str())
}
/// If `name` is in `matches`, parses the value as a path. Otherwise, attempts to find the user's
/// home directory and appends `default` to it.
pub fn parse_path_with_default_in_home_dir(
matches: &ArgMatches,
name: &'static str,
default: PathBuf,
) -> Result<PathBuf, String> {
matches
.value_of(name)
.map(|dir| {
dir.parse::<PathBuf>()
.map_err(|e| format!("Unable to parse {}: {}", name, e))
})
.unwrap_or_else(|| {
dirs::home_dir()
.map(|home| home.join(default))
.ok_or_else(|| format!("Unable to locate home directory. Try specifying {}", name))
})
}
/// Returns the value of `name` or an error if it is not in `matches` or does not parse
/// successfully using `std::string::FromStr`.
pub fn parse_required<T>(matches: &ArgMatches, name: &str) -> Result<T, String>
where
T: FromStr,
<T as FromStr>::Err: std::fmt::Display,
{
parse_optional(matches, name)?.ok_or_else(|| format!("{} not specified", name))
}
/// Returns the value of `name` (if present) or an error if it does not parse successfully using
/// `std::string::FromStr`.
pub fn parse_optional<T>(matches: &ArgMatches, name: &str) -> Result<Option<T>, String>
where
T: FromStr,
<T as FromStr>::Err: std::fmt::Display,
{
matches
.value_of(name)
.map(|val| {
val.parse()
.map_err(|e| format!("Unable to parse {}: {}", name, e))
})
.transpose()
}
/// Returns the value of `name` or an error if it is not in `matches` or does not parse
/// successfully using `ssz::Decode`.
///
/// Expects the value of `name` to be 0x-prefixed ASCII-hex.
pub fn parse_ssz_required<T: Decode>(
matches: &ArgMatches,
name: &'static str,
) -> Result<T, String> {
parse_ssz_optional(matches, name)?.ok_or_else(|| format!("{} not specified", name))
}
/// Returns the value of `name` (if present) or an error if it does not parse successfully using
/// `ssz::Decode`.
///
/// Expects the value of `name` (if any) to be 0x-prefixed ASCII-hex.
pub fn parse_ssz_optional<T: Decode>(
matches: &ArgMatches,
name: &'static str,
) -> Result<Option<T>, String> {
matches
.value_of(name)
.map(|val| {
if let Some(stripped) = val.strip_prefix("0x") {
let vec = hex::decode(stripped)
.map_err(|e| format!("Unable to parse {} as hex: {:?}", name, e))?;
T::from_ssz_bytes(&vec)
.map_err(|e| format!("Unable to parse {} as SSZ: {:?}", name, e))
} else {
Err(format!("Unable to parse {}, must have 0x prefix", name))
}
})
.transpose()
}