Repository navigation
Endless Indexing #106
Description
I awaited for indexing complete, got a message about compacting finished, the folder with electrs now uses 800 gb, but i still keep getting messages about indexing, the http server is unavailable. I restarted electrs 16 hours ago (after compaction finished and i started get indexing messages), but it seems electrs is not synced.
Reacted by Carlos Vieira and SpontaneousOverthrow
Activity
murbot commentedon Sep 27, 2024
@Krokochik, did you get anywhere with this? I'm in exactly the same boat, and it's so slow. The system io, cpu and memory are all very low (close to 0), so there don't appear to be any bottlenecks.
Krokochik commentedon Sep 27, 2024
Sadly, the only thing I came up is to use electrs instead of esplora 'cause esplora was indexing over 3 weeks and even didn't complete for that period. Please let me know if you get smth
murbot commentedon Sep 27, 2024
Ok, thanks for letting me know. I've exhausted all my resources at this point, so I think I'll abort and switch to romanz/electrs as well. What a shame.
shesek commentedon Sep 29, 2024
There are two indexing stages, one for adding transactions and for one indexing history, each followed by rocksdb compaction.
The logs should show "adding transactions from N blocks", then compaction, then "indexing history from N blocks", then another final compaction.
It seems that you finished the first stage but not the second before restarting, and that the "indexing" messages are normal and just means that it did not finish indexing yet.
shesek commentedon Sep 29, 2024
(As an aside, for personal use you probably should enable
--lightmode, which reduces storage requirements and makes the first stage significantly faster)Krokochik commentedon Sep 29, 2024
Yes, "indexing" messages are completely normal, but what is not normal is that the second indexing stage lasts over 3 weeks on powerfuly machine with ssd. About
--lightmode: as you say, the first index stage completed so I have no need to accelerate it. I also need esplora not for personal, but public useKrokochik commentedon Sep 29, 2024
Also as you can see CPU usage time is significantly lower than executing time, so I'm confident this is an issue
Krokochik commentedon Sep 29, 2024
Also now I switched to original romanz/electrs and that indexing finished in about 9 hours
shesek commentedon Sep 29, 2024
Looking at your log messages it appears that its indexing nearly 600 blocks every 10 minutes, or nearly 1 block per second, which seems reasonable. Did it slow down significantly when it reached more recent (and heavier) blocks?
It's difficult to tell why without more information. One possibility is that it doesn't saturate CPU due to a bottleneck with your SSD.
romanz/electrs has different design trade-offs and indeed uses a much more lightweight index. If it’s meeting your needs, it might be a good idea to stick with it.
Krokochik commentedon Sep 29, 2024
As I could remember, it was indexing 1 blk*.dat file every 10 mins, and it seems to be absurd since bitcoin has 4523 block files, i.e. indexing should take approximately 45230min=31day. The ssd has speed about 7400mb/s. Ideally I need esplora index designed for public use, but I can't wait so much
Krokochik commentedon Sep 30, 2024
Now I checked and yeah, on the 2nd stage it is indexing 1 blk*.dat file per ~10 minutes (on another machine now). This issue is critical
pokrovskyy commentedon Oct 27, 2024
Same experience. Currently at history sync, its beyond slow. Currently projecting around 1 week sync. SSD is high-speed datacenter-grade, CPU barely touched, RAM at 10%. I would assume this should maybe get parallelized somehow because currently it does not seem hit any viable bottlenecks. Will keep it running for a week and see where it gets me. Would settle for original esplora but this one has some useful endpoints I need + HTTP API, so would still give it a try! But indexing should be significantly improved IMO
Krokochik commentedon Oct 27, 2024
I'd say you have no chance to make index just in a week, but anyway please give us know what happen
pokrovskyy commentedon Oct 27, 2024
I have re-done the math this morning and timed .blk file processing average (on the
historyphase). So far I'm getting 75s per one .blk file. All .blk files are about the same size, so I suppose it does not matter that average Bitcoin block gets bigger over time (just less blocks per .blk file). With 4k files left should be in fact 3.5 days. I'll try to not forget and update you here how it goes.On bottleneck tracking:
historyRocksDB storeIDK what's the deal here, but it sure seems it can use some optimization, it seems the bottleneck is somewhere in the logic and not in the resources
pokrovskyy commentedon Oct 28, 2024
Well, although indexing continues at projected pace (at 30% now), I will be aborting the sync. Ended up switching to https://github.com/cculianu/Fulcrum - took around 6 hrs to fully sync (160GB final index) and it works blazing fast (comparing to original https://github.com/romanz/electrs). Also put together a simple Nodejs HTTP API wrapper as a replacement for Explora's HTTP API, so no reason to continue with Esplora's backend sync (for my particular use-case)
Krokochik commentedon Oct 28, 2024
I wonder if esplora maintainers are going to fix this
Merge pull request Blockstream#106 from mempool/junderw/fuck-c
RooSoft commentedon Nov 3, 2025
Came here while the node is parsing about two .dat files per minute... Low CPU usage, low memory footprint, low disk usage...
Seems to run 20x faster than most comments here seem to suggest, so I will let it do its thing.
Still, no sign of physical bottleneck while working so slow feels a bit weird.