Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Performance regression for dumping/loading DataFrames #20605

Closed
2 tasks done
datenzauberai opened this issue Jan 7, 2025 · 0 comments · Fixed by #20641
Closed
2 tasks done

Performance regression for dumping/loading DataFrames #20605

datenzauberai opened this issue Jan 7, 2025 · 0 comments · Fixed by #20641
Assignees
@nameexhaustion
Labels
A-serde Area: seralization and deserialization accepted Ready for implementation bug Something isn't working performance Performance issues or improvements python Related to Python Polars regression Issue introduced by a new release

Comments

@datenzauberai
Copy link
Contributor

datenzauberai commented Jan 7, 2025 •
edited
Loading

Checks

  • I have checked that this issue has not already been reported.
  • I have confirmed this bug exists on the latest version of Polars.

Reproducible example

import polars as pl
import numpy as np
import pickle

row_count = 4_000_000
col_count = 16
df = pl.DataFrame({
    f"col_{i}": pl.Series(np.random.randint(low=-2**32+1, high=2**32-1, size=row_count))
    for i in range(col_count)
})
eager_file_name = f"eager_{pl.__version__}.pkl"
lazy_file_name = f"lazy_{pl.__version__}.pkl"

with open(eager_file_name, "wb") as f:
    pickle.dump(df, f)

with open(lazy_file_name, "wb") as f:
    pickle.dump(df.lazy(), f)

with open(arrow_file_name, "wb") as f:
    pickle.dump(df.to_arrow(), f)

df = pickle.load(open(eager_file_name, "rb"))
df = pickle.load(open(lazy_file_name, "rb"))
df = pl.DataFrame(pickle.load(open(arrow_file_name, "rb")))

Log output

No response

Issue description

With the updated serialization code in 18.0.0 using IPC instead of Serde (#20266) it looks like the performance for dumping and loading DataFrame and LazyFrame is now on par, but significantly worse compared to 1.17.1 for the given data (4M rows, 16 cols, random integers).

Possible workaround: convert the DataFrame to PyArrow (pyarrow.lib.Table) before dump and convert back to DataFrame after load.

┌─────────┬───────────────────┬──────────────────┬────────────────────┬───────────────────┬───────────────────┬──────────────────┬────────────────────┬───────────────────┐
│ version ┆ dump in s (eager) ┆ dump in s (lazy) ┆ dump in MB (eager) ┆ dump in MB (lazy) ┆ load in s (eager) ┆ load in s (lazy) ┆ total in s (eager) ┆ total in s (lazy) │
╞═════════╪═══════════════════╪══════════════════╪════════════════════╪═══════════════════╪═══════════════════╪══════════════════╪════════════════════╪═══════════════════╡
│ 1.17.1  ┆ 4.98              ┆ 4.25             ┆ 489                ┆ 306               ┆ 1.28              ┆ 7.49             ┆ 6.26               ┆ 11.74             │
│ 1.18.0  ┆ 15.0              ┆ 16.2             ┆ 314                ┆ 314               ┆ 4.45              ┆ 4.82             ┆ 19.45              ┆ 21.02             │
│ 1.19.0  ┆ 16.3              ┆ 16.4             ┆ 314                ┆ 314               ┆ 4.36              ┆ 4.1              ┆ 20.66              ┆ 20.5              │
└─────────┴───────────────────┴──────────────────┴────────────────────┴───────────────────┴───────────────────┴──────────────────┴────────────────────┴───────────────────┘

Expected behavior

Fast pickle load times are important, because a lot of python caching frameworks pickle the data. Would be desirable to have performance comparable with PyArrow.

Installed versions

--------Version info---------
Polars:              1.19.0
Index type:          UInt32
Platform:            macOS-14.7.1-arm64-arm-64bit
Python:              3.10.13 (main, Nov 15 2023, 16:29:06) [Clang 15.0.0 (clang-1500.0.40.1)]
LTS CPU:             False

----Optional dependencies----
adbc_driver_manager  <not installed>
altair               <not installed>
azure.identity       <not installed>
boto3                <not installed>
cloudpickle          <not installed>
connectorx           <not installed>
deltalake            <not installed>
fastexcel            <not installed>
fsspec               <not installed>
gevent               <not installed>
google.auth          <not installed>
great_tables         <not installed>
matplotlib           <not installed>
nest_asyncio         1.6.0
numpy                2.1.3
openpyxl             <not installed>
pandas               <not installed>
pyarrow              18.1.0
pydantic             <not installed>
pyiceberg            <not installed>
sqlalchemy           <not installed>
torch                <not installed>
xlsx2csv             <not installed>
xlsxwriter           <not installed>
@datenzauberai datenzauberai added bug Something isn't working needs triage Awaiting prioritization by a maintainer python Related to Python Polars labels Jan 7, 2025
@nameexhaustion nameexhaustion added performance Performance issues or improvements regression Issue introduced by a new release accepted Ready for implementation A-serde Area: seralization and deserialization and removed needs triage Awaiting prioritization by a maintainer labels Jan 8, 2025
@nameexhaustion nameexhaustion self-assigned this Jan 8, 2025
@github-project-automation github-project-automation bot moved this to Ready in Backlog Jan 8, 2025
@nameexhaustion nameexhaustion mentioned this issue Jan 9, 2025
Merged
@github-project-automation github-project-automation bot moved this from Ready to Done in Backlog Jan 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nameexhaustion nameexhaustion

Labels
A-serde Area: seralization and deserialization accepted Ready for implementation bug Something isn't working performance Performance issues or improvements python Related to Python Polars regression Issue introduced by a new release
Projects
Backlog
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: Fix performance regression for DataFrame serialization/pickling nameexhaustion/polars
2 participants
@datenzauberai @nameexhaustion