Ah, my favorite type of error: sporadic!
A day days we started getting failures when using fetch_pandas_all() against Snowflake. The query was a basic ‘SELECT *’ against an analytical data set. Here is the trace:
Traceback (most recent call last):
File "/Users/david/path/test/main.py", line 31, in <module>
df = cs.fetch_pandas_all()
File "/Users/david/path/.scriptenv/lib/python3.8/site-packages/snowflake/connector/cursor.py", line 978, in fetch_pandas_all
return self._result_set._fetch_pandas_all(**kwargs)
File "/Users/david/path/.scriptenv/lib/python3.8/site-packages/snowflake/connector/result_set.py", line 185, in _fetch_pandas_all
dataframes = list(self._fetch_pandas_batches())
File "/Users/david/path/.scriptenv/lib/python3.8/site-packages/snowflake/connector/result_set.py", line 181, in _fetch_pandas_batches
return self._create_iter(iter_unit=IterUnit.TABLE_UNIT, structure="pandas")
File "/Users/david/path/.scriptenv/lib/python3.8/site-packages/snowflake/connector/result_set.py", line 223, in _create_iter
first_batch_iter = self.batches[0].create_iter(**kwargs)
File "/Users/david/path/.scriptenv/lib/python3.8/site-packages/snowflake/connector/result_batch.py", line 717, in create_iter
return self._get_pandas_iter(connection=connection, **kwargs)
File "/Users/david/path/.scriptenv/lib/python3.8/site-packages/snowflake/connector/result_batch.py", line 699, in _get_pandas_iter
dataframe = self.to_pandas(connection=connection, **kwargs)
File "/Users/david/path/.scriptenv/lib/python3.8/site-packages/snowflake/connector/result_batch.py", line 691, in to_pandas
table = self.to_arrow(connection=connection)
File "/Users/david/path/.scriptenv/lib/python3.8/site-packages/snowflake/connector/result_batch.py", line 684, in to_arrow
return self._create_empty_table()
File "/Users/david/path/.scriptenv/lib/python3.8/site-packages/snowflake/connector/result_batch.py", line 674, in _create_empty_table
fields = [
File "/Users/david/path/.scriptenv/lib/python3.8/site-packages/snowflake/connector/result_batch.py", line 675, in <listcomp>
field(s.name, FIELD_TYPE_TO_PA_TYPE[s.type_code]) for s in self.schema
IndexError: list index out of range
This error happens… sometimes. Restarting the Python kernel may fix it for a run and adding a limit statement to the select (larger than the number of rows) would sometimes fix it.
The culprit… the Geography columns (points and lines) that we recently added to the set. I’m still not sure why it would sometimes work, but removing all of those columns got everything back to reliably working.
Leave a Reply