Skip to content

working with big spss files #79

Open

Description

@AtanasAtanasovIpsos

First I want to say this library is great!
We have some raw SPSS files that are extremely large(about 6GB with 1.3 million of vars). SPSS itself can work with those. Pyreadstat however cannot handle it even with the option of reading the metadata only. While there is still plenty of RAM available left in the system (the usage of Python is about 1.5 GB) and there is 64 GB ram on the machine. The stacktrace is as follows:

File "C:\Users\thomas\Downloads\Ipsos\Carlsberg\build_column_overview.py", line 32, in <module>
    df, meta = pyreadstat.read_sav(os.path.join(path, file), metadataonly=True)
  File "pyreadstat\pyreadstat.pyx", line 325, in pyreadstat.pyreadstat.read_sav
  File "pyreadstat\_readstat_parser.pyx", line 945, in pyreadstat._readstat_parser.run_conversion
  File "pyreadstat\_readstat_parser.pyx", line 784, in pyreadstat._readstat_parser.run_readstat_parser
  File "pyreadstat\_readstat_parser.pyx", line 714, in pyreadstat._readstat_parser.check_exit_status
ReadstatError: Unable to allocate memory

This happens both on Windows10 64 bit and Linux64bit with python(64 bit)=3.8 and pyreadstat=1.0.2

Now I understand that spss is probably not the best file-format for this data, but unfortunately, that is what we have.

Activity

  1. ofajardo commented on Sep 21, 2020

    @ofajardo
    Collaborator

    thanks for the report. Would you be able to produce some python code that using pyreadstat.write_sav, produces a large sample file that raises the error on your end? This is to be able to reproduce the issue but without the need of you transferring the file (but just the code to produce the file)

  2. AtanasAtanasovIpsos commented on Sep 21, 2020

    @AtanasAtanasovIpsos
    Author

    Thanks for the reply. I will try playing with the write_sav and will see if I can produce such a file.

  3. AtanasAtanasovIpsos commented on Sep 21, 2020

    @AtanasAtanasovIpsos
    Author

    Here is an example of a code that will generate about 84.6MB of a file that cannot be read back due to the same error.

    import random
    import pandas as pd
    import numpy as np
    import pyreadstat
    
    N=1300000
    DataSet = pd.DataFrame(np.random.randn(1, N),columns=['A'+str(x) for x in range(1,N+1)])
    pyreadstat.write_sav(DataSet,'DataFile.sav')
    #%%
    df, meta = pyreadstat.read_sav('DataFile.sav', metadataonly=True)
    
  4. evanmiller commented on Sep 21, 2020

    @evanmiller

    Hi, ReadStat restricts individual memory allocations to 16 MB - this is to prevent denial of service type scenarios with mal-formed data. With 1.3 million variables in your file you are likely hitting that limit with the column metadata.

    Some options are 1) Increasing the limit 2) Adding an option to specify the limit and 3) Removing the limit altogether.

  5. AtanasAtanasovIpsos commented on Sep 21, 2020

    @AtanasAtanasovIpsos
    Author

    The second option would be the best one for me. Or something like:

    df, meta = pyreadstat.read_sav('DataFile.sav', metadataonly=True,safetylimits=False)
    
  6. ofajardo commented on Sep 21, 2020

    @ofajardo
    Collaborator

    That's a good suggestion.

    Given the experience with pyreadr I think 1 is nit good because there will be always somebody with a larger file that will hit the new limit. I personally think removing it would be better, as it was done in pyreadr. That will be less confusing for the users, as they don't need to be aware if the extra flag to inactivate the limit.

  7. AtanasAtanasovIpsos commented on Sep 21, 2020

    @AtanasAtanasovIpsos
    Author

    you are right about the bigger files. now that I think more about it, removing the limit seems also good solution :)

  8. ofajardo commented on Dec 3, 2020

    @ofajardo
    Collaborator

    hi @evanmiller, is this something coming in Readstat version 1.1.5, or not yet? (just for clarity)

  9. evanmiller commented on Dec 3, 2020

    @evanmiller

    @ofajardo No solution yet

  10. ofajardo commented on Dec 3, 2020

    @ofajardo
    Collaborator

    @evanmiller Ok thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions