Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rust writer panics on empty record batches #2253

Closed
aersam opened this issue Mar 6, 2024 · 1 comment · Fixed by #2254
Closed

Rust writer panics on empty record batches #2253

aersam opened this issue Mar 6, 2024 · 1 comment · Fixed by #2254
Labels
bug Something isn't working

Comments

@aersam
Copy link
Contributor

aersam commented Mar 6, 2024 •
edited
Loading

Environment

Delta-rs version: 0.16

Binding: python

Bug

What happened:

Calling write_deltalake caused this failure:

thread '<unnamed>' panicked at crates\core\src\operations\write.rs:293:54:
index out of bounds: the len is 0 but the index is 0
stack backtrace:
   0:     0x7ffd8f077542 - BrotliDecoderVersion
   1:     0x7ffd8f09fa4d - BrotliDecoderVersion
   2:     0x7ffd8f072e91 - BrotliDecoderVersion
   3:     0x7ffd8f07736a - BrotliDecoderVersion
   4:     0x7ffd8f079819 - BrotliDecoderVersion
   5:     0x7ffd8f0794d5 - BrotliDecoderVersion
   6:     0x7ffd8f079d44 - BrotliDecoderVersion
   7:     0x7ffd8f079c19 - BrotliDecoderVersion
   8:     0x7ffd8f077e49 - BrotliDecoderVersion
   9:     0x7ffd8f0798e2 - BrotliDecoderVersion
  10:     0x7ffd8f1e95e7 - BrotliDecoderVersion
  11:     0x7ffd8f1e97f4 - BrotliDecoderVersion
  12:     0x7ffd8c7bc20c - PyInit__internal
  13:     0x7ffd8c03cfbc - BrotliDecoderSetParameter
  14:     0x7ffd8c13a99e - BrotliDecoderSetParameter
  15:     0x7ffd8c2f1273 - BrotliDecoderSetParameter
  16:     0x7ffd8c2f38f2 - BrotliDecoderSetParameter
  17:     0x7ffd8c2aa1ae - BrotliDecoderSetParameter
  18:     0x7ffd8c2f29b1 - BrotliDecoderSetParameter
  19:     0x7ffdf7cdf43c - PyObject_MakeTpCall
  20:     0x7ffdf7caf128 - PyObject_Vectorcall
  21:     0x7ffdf7cb08d4 - PyEval_EvalFrameDefault
  22:     0x7ffdf7cd548f - PyObject_GetBuffer
  23:     0x7ffdf7dd2ed9 - PyLong_AsUnsignedLongMask
  24:     0x7ffdf7cb3f54 - PyEval_EvalFrameDefault
  25:     0x7ffdf7cd548f - PyObject_GetBuffer
  26:     0x7ffdf7dd2ed9 - PyLong_AsUnsignedLongMask
  27:     0x7ffdf7cb3f54 - PyEval_EvalFrameDefault
  28:     0x7ffdf7cd548f - PyObject_GetBuffer
  29:     0x7ffdf7dd2ed9 - PyLong_AsUnsignedLongMask
  30:     0x7ffe4c8958af - PyInit__asyncio
  31:     0x7ffe4c895723 - PyInit__asyncio
  32:     0x7ffdf7cdfac8 - PyObject_MakeTpCall
  33:     0x7ffdf7eb459c - PyContext_NewHamtForTests
  34:     0x7ffdf7eb484d - PyContext_NewHamtForTests
  35:     0x7ffdf7cccbc0 - PyUnicode_RichCompare
  36:     0x7ffdf7d3038f - PyObject_Call
  37:     0x7ffdf7d30104 - PyList_AsTuple
  38:     0x7ffdf7cb52ba - PyEval_EvalFrameDefault
  39:     0x7ffdf7cd8334 - PyFunction_Vectorcall
  40:     0x7ffdf7d30009 - PyList_AsTuple
  41:     0x7ffdf7cb52ba - PyEval_EvalFrameDefault
  42:     0x7ffdf7cd8334 - PyFunction_Vectorcall
  43:     0x7ffdf7d30098 - PyList_AsTuple
  44:     0x7ffdf7cb52ba - PyEval_EvalFrameDefault
  45:     0x7ffdf7cd8334 - PyFunction_Vectorcall
  46:     0x7ffdf7d2c20e - PyObject_FastCallDictTstate
  47:     0x7ffdf7d2b897 - PyObject_Call_Prepend
  48:     0x7ffdf7d2b7c4 - PyEval_EvalCode
  49:     0x7ffdf7cdfac8 - PyObject_MakeTpCall
  50:     0x7ffdf7caf128 - PyObject_Vectorcall
  51:     0x7ffdf7cb08d4 - PyEval_EvalFrameDefault
  52:     0x7ffdf7cd8334 - PyFunction_Vectorcall
  53:     0x7ffdf7d30098 - PyList_AsTuple
  54:     0x7ffdf7cb52ba - PyEval_EvalFrameDefault
  55:     0x7ffdf7cd8334 - PyFunction_Vectorcall
  56:     0x7ffdf7d2c20e - PyObject_FastCallDictTstate
  57:     0x7ffdf7d2b897 - PyObject_Call_Prepend
  58:     0x7ffdf7d2b7c4 - PyEval_EvalCode
  59:     0x7ffdf7d300d1 - PyList_AsTuple
  60:     0x7ffdf7cb52ba - PyEval_EvalFrameDefault
  61:     0x7ffdf7cd8334 - PyFunction_Vectorcall
  62:     0x7ffdf7d30098 - PyList_AsTuple
  63:     0x7ffdf7cb52ba - PyEval_EvalFrameDefault
  64:     0x7ffdf7cd8334 - PyFunction_Vectorcall
  65:     0x7ffdf7d30098 - PyList_AsTuple
  66:     0x7ffdf7cb52ba - PyEval_EvalFrameDefault
  67:     0x7ffdf7cd8334 - PyFunction_Vectorcall
  68:     0x7ffdf7d2c20e - PyObject_FastCallDictTstate
  69:     0x7ffdf7d2b897 - PyObject_Call_Prepend
  70:     0x7ffdf7d2b7c4 - PyEval_EvalCode
  71:     0x7ffdf7cdfac8 - PyObject_MakeTpCall
  72:     0x7ffdf7caf128 - PyObject_Vectorcall
  73:     0x7ffdf7cb08d4 - PyEval_EvalFrameDefault
  74:     0x7ffdf7cd8334 - PyFunction_Vectorcall
  75:     0x7ffdf7d30098 - PyList_AsTuple
  76:     0x7ffdf7cb52ba - PyEval_EvalFrameDefault
  77:     0x7ffdf7cd8334 - PyFunction_Vectorcall
  78:     0x7ffdf7d2c20e - PyObject_FastCallDictTstate
  79:     0x7ffdf7d2b897 - PyObject_Call_Prepend
  80:     0x7ffdf7d2b7c4 - PyEval_EvalCode
  81:     0x7ffdf7cdfac8 - PyObject_MakeTpCall
  82:     0x7ffdf7caf128 - PyObject_Vectorcall
  83:     0x7ffdf7cb08d4 - PyEval_EvalFrameDefault
  84:     0x7ffdf7cd8334 - PyFunction_Vectorcall
  85:     0x7ffdf7d30098 - PyList_AsTuple
  86:     0x7ffdf7cb52ba - PyEval_EvalFrameDefault
  87:     0x7ffdf7cd8334 - PyFunction_Vectorcall
  88:     0x7ffdf7d2c20e - PyObject_FastCallDictTstate
  89:     0x7ffdf7d2b897 - PyObject_Call_Prepend
  90:     0x7ffdf7d2b7c4 - PyEval_EvalCode
  91:     0x7ffdf7cdfac8 - PyObject_MakeTpCall
  92:     0x7ffdf7caf128 - PyObject_Vectorcall
  93:     0x7ffdf7cb08d4 - PyEval_EvalFrameDefault
  94:     0x7ffdf7d2bbd3 - PyMapping_Check
  95:     0x7ffdf7d2b453 - PyEval_EvalCode
  96:     0x7ffdf7d2a290 - PyEval_GetBuiltins
  97:     0x7ffdf7d2a160 - PyEval_GetBuiltins
  98:     0x7ffdf7cccbc0 - PyUnicode_RichCompare
  99:     0x7ffdf7caf52b - PyObject_Vectorcall
 100:     0x7ffdf7cb08d4 - PyEval_EvalFrameDefault
 101:     0x7ffdf7cd8334 - PyFunction_Vectorcall
 102:     0x7ffdf7d3038f - PyObject_Call
 103:     0x7ffdf7d63f82 - Py_Exit
 104:     0x7ffdf7d643df - Py_RunMain
 105:     0x7ffdf7d642bd - Py_RunMain
 106:     0x7ffdf7cee76d - Py_Main
 107:     0x7ff7932c1230 - <unknown>
 108:     0x7ffe66e77344 - BaseThreadInitThunk
 109:     0x7ffe686e26b1 - RtlUserThreadStart

What you expected to happen:
Should return a good exception or do nothing

I'll write a PR, it's trivial to fix
@aersam aersam added the bug Something isn't working label Mar 6, 2024
aersam added a commit to bmsuisse/delta-rs that referenced this issue Mar 6, 2024
@aersam
Fixes delta-io#2253
0bafd27
@aersam aersam mentioned this issue Mar 6, 2024
Merged
@aersam
Copy link
Contributor Author

aersam commented Mar 6, 2024

Actually I'd prefer not to throw at all, if you try to append an empty record batch reader. It's what the pyarrow engine is also doing

rtyler pushed a commit that referenced this issue Mar 6, 2024
@aersam @rtyler
Fixes #2253
154f9b5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: fixes panic on empty write bmsuisse/delta-rs
1 participant
@aersam