You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
vec or hashmap seem to still hold heap memory after lifetime #73307
use std::thread;use std::time::Duration;use std::collections::HashMap;fnmain(){//let value = read_file("./1k.conf");let value = "test content".to_string();println!("begin");
thread::sleep(Duration::from_secs(20));test_vec_not_clear(&value);wait_for_free(10);test_map_not_clear(&value);wait_for_free(10);wait_for_free(40);test_map_not_clear(&value);wait_for_free(10);println!("end");loop{
thread::sleep(Duration::from_secs(20));}}fnwait_for_free(count:i32){for i in0..count {println!("wait count: {}", i);
thread::sleep(Duration::from_secs(1));}}fntest_vec_not_clear(value:&str){println!("to test vec not clear");
thread::sleep(Duration::from_secs(5));letmut vec = Vec::new();for _i in0..1000000{
vec.push(value.to_string());}}fntest_map_not_clear(value:&str){println!("to test map not clear");
thread::sleep(Duration::from_secs(5));letmut map = HashMap::new();for i in0..1000000{
map.insert(i.to_string(), value.to_string());}}
then use valgrind to start: valgrind --tool=massif --detailed-freq=1 --massif-out-file=massif.out.test.%p ./test_mem
I expected to see this happen:
memory will down to slow usage after line:24
28M -> 300M -> 28M -> 300M -> 28M -> 300M -> 28M
Instead, this happened:
the memory usage is :
28M -> 300M -> 85M -> 300M -> 137M -> 300M -> 137M
the massif report:
Constantly allocating and deallocating memory is a costly operation. In order to save on performance I'm guessing Rust chooses to reuse some of it. I don't really see an issue here, this behavior seems completely normal. What are you really trying to do ?
Hi. I'm not sure it is correct behavior.
Try to change HashMap to BtreeMap or vec. Result: BtreeMap and Vec will like you expected (return memory to system)
I want to know...how to control it...and why it happening?
For example I'm going to load bigdata to hashMap and never use it...My process will never back this memory to System?
Hi I have a similar issue with my project.
Now I'm testing the hashmap performances.
use std::{collections::HashMap, thread, time::Duration};fnfunc(){letmut map:HashMap<String,Vec<u8>> = HashMap::new();for i in0..100_000{let key = format!("{}", i);
map.insert(key,vec![0;50_000]);}}fnmain(){func();println!("LOOP");loop{
thread::sleep(Duration::from_secs(5));}}
Running this the memory occupied from the map is not released to the system, so it takes about 5 GB of RAM.
Instead if I use an HashMap<i32, Vec> after the executing of func the memory is released and the consumption of RAM drop from 5GB to about zero.
Can you show me a way to understand what this behavior depends on?
Thanks a lot!
Reacted by Guillermo Céspedes Tabárez and Hu Jiang
use std::{collections::HashMap, thread, time::Duration};fnfunc(){letmut map:HashMap<i32,Vec<u8>> = HashMap::new();for num in0..100_000{let key = format!("{}_test", num);// This line causes HashMap<i32, Vec<u8>> not to release memory.. Why ?let _key:&'staticstr = Box::leak(key.into_boxed_str());
map.insert(num,vec![0;50_000]);}}fnmain(){func();println!("LOOP");loop{
thread::sleep(Duration::from_secs(5));}}
Reacted by Guillermo Céspedes Tabárez, rodrimati1992 and Mateusz Mikuła
Memory is released but I did it with little knowledge of memory management and this doesn't seem performant.
use std::{collections::HashMap, thread, time::Duration};#[derive(Debug,PartialEq)]enumExample{Vec(Vec<u8>),Test(u32),None}fnto_static_str<S:AsRef<str>>(str:S) -> &'staticstr{let boxed = str.as_ref().to_owned().into_boxed_str();// Use Box::into_raw we can reclaim ownership with from_rawunsafe{&*Box::into_raw(boxed)}}unsafefnfree_static_str(str:&'staticstr){// Take back ownership using unsafe codelet boxed = Box::from_raw(str as*conststras*mutstr);// Not necessary since it would get dropped anyway at the end of the function,// but it makes the intent more explicit
std::mem::drop(boxed);}fntest_hashmap_static_str(i:i32){letmut map:HashMap<&'staticstr,Example> = HashMap::new();letmut vec_keys_hashmap = Vec::new();println!("load memory");for num in0..100_000{let str_num = format!("{}_test_{}", num, i);let key_hashmap = to_static_str(str_num);
map.insert(key_hashmap,Example::Vec(vec![0;100_000]));
vec_keys_hashmap.push(key_hashmap);}
thread::sleep(Duration::from_secs(5));println!("clear memory");// map.clear();// map.shrink_to_fit();// drop(map);for key_hashmap in vec_keys_hashmap {unsafe{free_static_str(key_hashmap);}}}fnmain(){for i in0..5{test_hashmap_static_str(i);
thread::sleep(Duration::from_secs(5));}println!("END");loop{
thread::sleep(Duration::from_secs(5));}}
I am writing a microservice that uses HashMap with keys static str and I cannot let the memory increase with use incoming new requests to server, I would like to control it.
I encountered the same problem with the latest 1.64 version, put the 8GB vec into the hashmap, and after calling clear, use the standard allocator, the memory will not be recycled, and the effect will be better after replacing it with jemalloc, but there are also some that cannot be recycled.
We have four different people reporting similar but different things. Going through them one by one.
@Right2Left: when I profile with Massif I see three peaks, with memory use dropping to almost zero after each one. The peak for the test_vec_not_clear call is smaller because it's just a Vec plus the one million copies of value. The peak for the test_map_not_clear calls are larger because it's a HashMap (which uses more memory than the same-sized Vec) plus the one million copies of value plus the one million i strings. Everything here looks as expected, with the data structures being dropped at the end of each containing function.
@willygroup: profiling with Massif and top I see that Rust deallocates the HashMap, but the RSS remains high afterwards. This must be due to the system allocator choosing to hold on to the memory. Further allocations and deallocations after func are likely to cause the system allocator to eventually return that memory to the OS, though the exact behaviour will depend on which platform you are on. The system allocator is likely to behave reasonably for real-world code, even though it behaves surprisingly on test programs like this that do a small number of gigantic allocations.
@dertin: Box::leak deliberately leaks memory. You probably don't want to use it except in very unusual circumstances. And the unsafe code in your second example looks dangerous. I recommend not doing that kind of thing.
@hechangqin: you haven't given enough details to say for sure, but note that HashMap::clear will remove all entries from a hashmap but hold onto the memory for reuse. Doing HashMap::shrink_to_fit afterwards will free the memory. As above, whether or not the system allocator chooses to then release that freed memory back to the OS is unpredictable.
@nnethercote Thanks for your explanation. After I call HashMap::clear, I have called HashMap::shrink_to_fit, but the phenomenon is that the memory is not recycled. But after replacing the standard allocator with jemalloc, it got better, but there are still some parts that are not recycled.
In fact, we need to cache about 8GB of data in the HashMap, and then we need to refresh the cache every night. When we use HashMap::clear and HashMap::shrink_to_fit to clear the HashMap, and load the latest 8GB data to cache again, and observe through top that the actual usage of the rust process has exceeded 10GB, an out-of-memory error will occur.
Do you have any good suggestions for my application scenario?
@hechangqin@nnethercote
really has issues, I use hashbrown and test to insert hash table for two days, then clean it, and found the memory increase to 4G from 1G, then app crash; want to know why hash table free consume large mount of memory?
I conducted tests on both Mac OS and Windows, only HashMap::with_capacity(1000000) can perfectly avoid the generation of memory fragments. On the other hand, HashMap:: new() will definitely generate memory fragments when triggering expansion
use std::collections::HashMap;fnmain(){
std::thread::sleep(std::time::Duration::from_secs(10));for i in0..20{println!("{}",i);// test_hashmap(HashMap::new());test_hashmap(HashMap::with_capacity(1000000));}println!("all done,now wait");
std::thread::sleep(std::time::Duration::from_secs(3600));}fntest_hashmap(muth:HashMap<i64,i64>){for v in0..1000000{
h.insert(v, v);}}
Activity
Right2Left commentedon Jun 13, 2020
the last snapshot is:
#-----------
snapshot=80
#-----------
time=33575861397
mem_heap_B=104436498
mem_heap_extra_B=2822926
mem_stacks_B=0
heap_tree=detailed
n2: 104436498 (heap allocation functions) malloc/new/new[], --alloc-fns, etc.
n2: 104435061 0x1126AB: alloc::alloc::alloc (alloc.rs:84)
n1: 102760464 0x117E9F: hashbrown::raw::RawTable::new_uninitialized (mod.rs:383)
n1: 102760464 0x1182A3: hashbrown::raw::RawTable::try_with_capacity (mod.rs:407)
n1: 102760464 0x118DE5: hashbrown::raw::RawTable::resize (mod.rs:788)
n1: 102760464 0x116E4C: hashbrown::raw::RawTable::reserve_rehash (mod.rs:669)
n1: 102760464 0x119466: hashbrown::raw::RawTable::reserve (mod.rs:626)
n1: 102760464 0x118BEF: hashbrown::raw::RawTable::insert (mod.rs:841)
n1: 102760464 0x11585E: hashbrown::map::HashMap<K,V,S>::insert (map.rs:855)
n1: 102760464 0x119FC8: std::collections::hash::map::HashMap<K,V,S>::insert (map.rs:817)
n2: 102760464 0x119DA9: test_mem::test_map_not_clear (main.rs:52)
n1: 102760464 0x11988E: test_mem::main (main.rs:21)
n1: 102760464 0x11C2BF: std::rt::lang_start::{{closure}} (rt.rs:67)
n1: 102760464 0x125CE2: {{closure}} (rt.rs:52)
n1: 102760464 0x125CE2: std::panicking::try::do_call (panicking.rs:292)
n1: 102760464 0x127AA9: __rust_maybe_catch_panic (lib.rs:78)
n1: 102760464 0x1266DF: try<i32,closure-0> (panicking.rs:270)
n1: 102760464 0x1266DF: catch_unwind<closure-0,i32> (panic.rs:394)
n1: 102760464 0x1266DF: std::rt::lang_start_internal (rt.rs:51)
n1: 102760464 0x11C298: std::rt::lang_start (rt.rs:67)
n0: 102760464 0x119E49: main (in /home/test/hzx/test_mem/target/debug/test_mem)
n0: 0 in 1 place, below massif's threshold (1.00%)
n2: 1674597 0x112541: <alloc::alloc::Global as core::alloc::Alloc>::alloc (alloc.rs:172)
n1: 1123344 0x113BE6: alloc::raw_vec::RawVec<T,A>::allocate_in (raw_vec.rs:98)
n1: 1123344 0x113A54: alloc::raw_vec::RawVec::with_capacity (raw_vec.rs:167)
n1: 1123344 0x112E82: alloc::vec::Vec::with_capacity (vec.rs:362)
n1: 1123344 0x115FFE: alloc::slice::hack::to_vec (slice.rs:158)
n1: 1123344 0x10D253: alloc::slice::<impl [T]>::to_vec (slice.rs:394)
n1: 1123344 0x10D283: alloc::slice::<impl alloc::borrow::ToOwned for [T]>::to_owned (slice.rs:727)
n1: 1123344 0x1196C8: alloc::str::::to_owned (str.rs:207)
n1: 1123344 0x1164C3: <alloc::string::String as core::convert::From<&str>>::from (string.rs:2231)
n2: 1123344 0x116133: ::to_string (string.rs:2191)
n2: 1123332 0x119D7D: test_mem::test_map_not_clear (main.rs:52)
n1: 1123332 0x11988E: test_mem::main (main.rs:21)
n1: 1123332 0x11C2BF: std::rt::lang_start::{{closure}} (rt.rs:67)
n1: 1123332 0x125CE2: {{closure}} (rt.rs:52)
n1: 1123332 0x125CE2: std::panicking::try::do_call (panicking.rs:292)
n1: 1123332 0x127AA9: __rust_maybe_catch_panic (lib.rs:78)
n1: 1123332 0x1266DF: try<i32,closure-0> (panicking.rs:270)
n1: 1123332 0x1266DF: catch_unwind<closure-0,i32> (panic.rs:394)
n1: 1123332 0x1266DF: std::rt::lang_start_internal (rt.rs:51)
n1: 1123332 0x11C298: std::rt::lang_start (rt.rs:67)
n0: 1123332 0x119E49: main (in /home/test/hzx/test_mem/target/debug/test_mem)
n0: 0 in 1 place, below massif's threshold (1.00%)
n0: 12 in 2 places, all below massif's threshold (1.00%)
n0: 551253 in 2 places, all below massif's threshold (1.00%)
n0: 1437 in 19 places, all below massif's threshold (1.00%)
crlf0710 commentedon Jun 14, 2020
EDIT: Sorry i read the issue wrong.
Right2Left commentedon Jun 15, 2020
I had try use:
map.clear();
map.shrink_to_fit();
drop(map);
but not effect.
tesuji commentedon Jun 15, 2020
The allocator doesn't returns memory to the OS but reuse them for later allocations.
Right2Left commentedon Jun 15, 2020
i think so, but I have no idea about how much will reuse? What is the relationship with code? Is there any option to returns memory directly?
CohenArthur commentedon Jun 15, 2020
Constantly allocating and deallocating memory is a costly operation. In order to save on performance I'm guessing Rust chooses to reuse some of it. I don't really see an issue here, this behavior seems completely normal. What are you really trying to do ?
sfackler commentedon Jun 16, 2020
Rust is not choosing to do anything here - the system allocator is.
DAigory commentedon Nov 26, 2021
Hi. I'm not sure it is correct behavior.
Try to change HashMap to BtreeMap or vec. Result: BtreeMap and Vec will like you expected (return memory to system)
I want to know...how to control it...and why it happening?
For example I'm going to load bigdata to hashMap and never use it...My process will never back this memory to System?
willygroup commentedon Apr 21, 2022
Hi I have a similar issue with my project.
Now I'm testing the hashmap performances.
Running this the memory occupied from the map is not released to the system, so it takes about 5 GB of RAM.
Instead if I use an HashMap<i32, Vec> after the executing of
functhe memory is released and the consumption of RAM drop from 5GB to about zero.Can you show me a way to understand what this behavior depends on?
Thanks a lot!
dertin commentedon Jul 14, 2022
I have the same problem I use an HashMap<&'static str, MyEnum>
Is there any news about the cause and how to avoid it?
Thanks!
dertin commentedon Jul 14, 2022
dertin commentedon Jul 15, 2022
Memory is released but I did it with little knowledge of memory management and this doesn't seem performant.
I am writing a microservice that uses HashMap with keys static str and I cannot let the memory increase with use incoming new requests to server, I would like to control it.
hechangqin commentedon Sep 30, 2022
I encountered the same problem with the latest 1.64 version, put the 8GB vec into the hashmap, and after calling clear, use the standard allocator, the memory will not be recycled, and the effect will be better after replacing it with jemalloc, but there are also some that cannot be recycled.
nnethercote commentedon Oct 5, 2022
We have four different people reporting similar but different things. Going through them one by one.
@Right2Left: when I profile with Massif I see three peaks, with memory use dropping to almost zero after each one. The peak for the
test_vec_not_clearcall is smaller because it's just aVecplus the one million copies ofvalue. The peak for thetest_map_not_clearcalls are larger because it's aHashMap(which uses more memory than the same-sizedVec) plus the one million copies ofvalueplus the one millionistrings. Everything here looks as expected, with the data structures being dropped at the end of each containing function.@willygroup: profiling with Massif and
topI see that Rust deallocates theHashMap, but the RSS remains high afterwards. This must be due to the system allocator choosing to hold on to the memory. Further allocations and deallocations afterfuncare likely to cause the system allocator to eventually return that memory to the OS, though the exact behaviour will depend on which platform you are on. The system allocator is likely to behave reasonably for real-world code, even though it behaves surprisingly on test programs like this that do a small number of gigantic allocations.@dertin:
Box::leakdeliberately leaks memory. You probably don't want to use it except in very unusual circumstances. And the unsafe code in your second example looks dangerous. I recommend not doing that kind of thing.@hechangqin: you haven't given enough details to say for sure, but note that
HashMap::clearwill remove all entries from a hashmap but hold onto the memory for reuse. DoingHashMap::shrink_to_fitafterwards will free the memory. As above, whether or not the system allocator chooses to then release that freed memory back to the OS is unpredictable.nnethercote commentedon Oct 5, 2022
As far as I can tell Rust is doing the right thing here in all cases, even though the system allocator's behaviour may be surprising in some cases.
hechangqin commentedon Oct 9, 2022
@nnethercote Thanks for your explanation. After I call
HashMap::clear, I have calledHashMap::shrink_to_fit, but the phenomenon is that the memory is not recycled. But after replacing the standard allocator with jemalloc, it got better, but there are still some parts that are not recycled.In fact, we need to cache about 8GB of data in the
HashMap, and then we need to refresh the cache every night. When we useHashMap::clearandHashMap::shrink_to_fitto clear theHashMap, and load the latest 8GB data to cache again, and observe through top that the actual usage of the rust process has exceeded 10GB, an out-of-memory error will occur.Do you have any good suggestions for my application scenario?
skydig commentedon Jun 26, 2023
@hechangqin @nnethercote
really has issues, I use hashbrown and test to insert hash table for two days, then clean it, and found the memory increase to 4G from 1G, then app crash; want to know why hash table free consume large mount of memory?
zhuxiujia commentedon Dec 22, 2023
HashMap::with_capacity(1000000)can perfectly avoid the generation of memory fragments. On the other hand,HashMap:: new()will definitely generate memory fragments when triggering expansionxkikeg commentedon Aug 8, 2024
I also ran into this issue.
ASAN_OPTIONS=detect_leaks=1,symbolize=1 RUSTFLAGS=-Zsanitizer=address cargo +nightly bench -Zbuild-std --target x86_64-unknown-linux-gnu --profile=test
This gave me tons of hashmap memory leak that was originally acquired in https://github.com/rust-lang/hashbrown/blob/v0.14.5/src/raw/mod.rs#L1788