You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hi, I'm facing a memory issue when trying to generate a xlsx file with >600 columns and ~10k rows. The memory usage grows up to 3GB aprox and it's making the dyno be restarted by heroku.
I'm wondering if there is an known way to generate it without making a heavy use of memory.
The way I'm doing it can be represented by this script:
Reacted by Lonre Wang, Arkadii Butermanov, Thomas Wickham, Matthias Bourg, Johannes Treitz, Alberto Pat, Matheus Santana, maudite, Kiko Castro, Chrıs Seelus and 11 more
RubyZip isn't the issue I think, axlsx is.
One of the problems is that axlsx creates an enormous string which is then passes to RubyZip. That's very inefficient..
It is possible it won't help in your situation though.. the fact is that atm axlsx allocates way too much objects and causes the heap to grow very big.. I'm trying to reduce the number of allocated objects..
We have a similar issue. We're building sheets with hundreds of thousands of rows and as many as 100 tabs. axlsx uses 5.3GB to build a 111MB xlsx file.
I would be happy to test some changes on this, but your branch is no longer available. Any further updates on this?
@jeremywadsack I've merged most of the changes into the master branch, but it's still using a lot of memory. I hope to have some time to look at this issue again in the coming weeks
The problem with these memory issues we face using AXLSX via the to_spreadsheet gem is, that the memory won't be free after xslx creation. This means we have to manually restart our worker dyno for now and also limit the maximum number of rows per Excel sheet.
@Paxa Nice work. Depending on the answer of the maintainer of the to_spreadsheet gem (allows a Rails app to render Excel files using the existing slim/haml/erb/etc views) I'll see if fast_excel could be used as an alternative to AXLSX.
Well, yes, @jurriaan said this in 2014: "the fact is that atm axlsx allocates way too much objects and causes the heap to grow very big."
I think there are two approaches here, neither is likely to be "easy":
Significantly reduce the size of the Cell objects.
Flush rows to disk as they are created.
The problem with the first approach is that we might generate a large excel file with 20+ tabs and each of them with five million rows (I know Excel doesn't support that many rows, but the file format does). I think you'd need to reduce the size of the Cell object to under 50 bytes to make a significant difference.
The problem with the second approach is that it means the developer would have to write rows in sequential order. That's a change to the interface, so would need to be "opt-in".
We've been very happy with axlsx but the memory usage really started to hit us. This may not apply to everyone, but if all you need is simple XLSX formatted output, I'd look into rubyXL. Here's a benchmark showing that its retained memory is constant, while memory retained by axlsx grows linearly with the number of rows/columns.
Reacted by Daisuke Tsuji, Mihail Toderasco, Dimitris Giannakos, Chrıs Seelus, Luciano Santobuono and Vinicius Cubas Brand
Activity
michaeldauria commentedon Jan 15, 2014
We're experiencing the same issue, perhaps RubyZip could be replaced with https://bitbucket.org/winebarrel/zip-ruby/wiki/Home ?
jurriaan commentedon Jan 15, 2014
RubyZip isn't the issue I think, axlsx is.
One of the problems is that axlsx creates an enormous string which is then passes to RubyZip. That's very inefficient..
jurriaan commentedon Jan 15, 2014
Tried to fix it @ https://github.com/jurriaan/axlsx/tree/fix_enormous_string Please report if this helps (it should reduce usage to 1.5 GB or so)
jurriaan commentedon Jan 17, 2014
@guilleva @michaeldauria Did this change work for you? It should be possible to reduce the memory usage even further, but it's a beginning :)
guilleva commentedon Jan 17, 2014
Hi jurriaan, thanks for looking into this, I haven't been able to test it, I really sorry, I will do my best to do it today and I will let you know.
Thanks again
jurriaan commentedon Jan 17, 2014
Just pushed some new commits, the example now uses ~700 MB on my system..
michaeldauria commentedon Jan 22, 2014
I will try this out today and report back
jurriaan commentedon Jan 22, 2014
It is possible it won't help in your situation though.. the fact is that atm axlsx allocates way too much objects and causes the heap to grow very big.. I'm trying to reduce the number of allocated objects..
michaeldauria commentedon Jan 28, 2014
This did not end up helping too much, memory seemed more stable, but we still ran out anyway.
jeremywadsack commentedon May 29, 2014
We have a similar issue. We're building sheets with hundreds of thousands of rows and as many as 100 tabs. axlsx uses 5.3GB to build a 111MB xlsx file.
I would be happy to test some changes on this, but your branch is no longer available. Any further updates on this?
jurriaan commentedon Jun 19, 2014
@jeremywadsack I've merged most of the changes into the master branch, but it's still using a lot of memory. I hope to have some time to look at this issue again in the coming weeks
maxle commentedon Sep 18, 2014
Hi. Any updates on this?
srpouyet commentedon Feb 3, 2015
Our Delayed Job workers aren't processing jobs (out of memory), probably because of this issue. Any updates on the status? Cheers!
pallymore commentedon Apr 27, 2015
+1 on this. We are having the same issue with huge datasets :/
jeremywadsack commentedon Apr 27, 2015
We just ended up spinning up more hardware to deal with this for the time being. :/
13 remaining items
jeremywadsack commentedon Oct 27, 2016
#352 is a similar request to append rows to an existing file.
Paxa commentedon Apr 23, 2017
I've been struggling with same issues, when generating > 100k rows our servers started to use swap and become very slow.
Solved by making own excel library https://github.com/paxa/fast_excel, it works 3-4 times faster and consume 5-20 MB ram regardless of data size
cseelus commentedon Nov 28, 2017
The problem with these memory issues we face using AXLSX via the to_spreadsheet gem is, that the memory won't be free after xslx creation. This means we have to manually restart our worker dyno for now and also limit the maximum number of rows per Excel sheet.
@Paxa Nice work. Depending on the answer of the maintainer of the to_spreadsheet gem (allows a Rails app to render Excel files using the existing slim/haml/erb/etc views) I'll see if fast_excel could be used as an alternative to AXLSX.
crystianleao commentedon Mar 22, 2018
So, @cseelus, i'm having issues with this lib's memory usage too. What did you do? @Paxa 's lib or to_spreadsheet?
Thank you!
cseelus commentedon Mar 25, 2018
For now we limit the number of rows per xslx file to mitigate the memory problems using the to_spreadsheet gem.
barttenbrinke commentedon Apr 4, 2018
According to my basic memoryprofile skills, the main issue is here: https://github.com/randym/axlsx/blob/master/lib/axlsx/workbook/worksheet/row.rb#L156 This is never released, expanding tot up to 3GB in my testcase.
simi commentedon Apr 5, 2018
@barttenbrinke can you share your code to reproduce and measure this problem?
comictvn commentedon Apr 13, 2018
anyone already fixed the issue ?
barttenbrinke commentedon Apr 13, 2018
@simi https://github.com/SamSaffron/memory_profiler @comictvn This wil not be an easy fix, seeing that row.rb has not been touched in > year, and the fact that this ticket is from 2014, I would not get your hopes up.
jeremywadsack commentedon Apr 13, 2018
Well, yes, @jurriaan said this in 2014: "the fact is that atm axlsx allocates way too much objects and causes the heap to grow very big."
I think there are two approaches here, neither is likely to be "easy":
Cellobjects.The problem with the first approach is that we might generate a large excel file with 20+ tabs and each of them with five million rows (I know Excel doesn't support that many rows, but the file format does). I think you'd need to reduce the size of the
Cellobject to under 50 bytes to make a significant difference.The problem with the second approach is that it means the developer would have to write rows in sequential order. That's a change to the interface, so would need to be "opt-in".
iamtheschmitzer commentedon Aug 3, 2018
👍 for solution 2
mkhoa1412 commentedon Aug 21, 2018
+1, this is a huge issue for me. I'd be happy to hear about possible workarounds.
janklimo commentedon Sep 25, 2018
We've been very happy with
axlsxbut the memory usage really started to hit us. This may not apply to everyone, but if all you need is simple XLSX formatted output, I'd look into rubyXL. Here's a benchmark showing that its retained memory is constant, while memory retained byaxlsxgrows linearly with the number of rows/columns.gdimitris commentedon May 3, 2019
@randym any update on this ?
wangDvp commentedon Dec 16, 2025
I use the fast_excel instead of this gem,
this is my local benchmark test result
`
`