AsyncArrowWriter API to get the total size of a written parquet file

[alamb]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
We are writing Parquet files with AsyncArrowWriter and then reading them back in with DataFusions's PartitionedFile which requires the file length (as part of ObjectMeta)

It is not clear how we could get the actual length from the AsyncArrowWriter

Before you make th call to AsyncArrowWriter::close the "written bytes" will not reflect the file footer size

However, looking at the API, AsyncArrowWriter::close doesn't seem to return the overall file size (it only returns the FileMetadata which does not have the overall file size

Describe the solution you'd like

I would like to be able to get the total file size from the writer

Here is a test that shows the usecas (in https://github.com/apache/arrow-rs/blob/7f2d9ac14b1b5b846feb130f5cbdfd64e6616cb9/parquet/src/arrow/async_writer/mod.rs#L389-L388):

    #[tokio::test]
    async fn test_async_writer_bytes_written() {
        let col = Arc::new(Int64Array::from_iter_values([1, 2, 3])) as ArrayRef;
        let to_write = RecordBatch::try_from_iter([("col", col)]).unwrap();

        let temp = tempfile::tempfile().unwrap();

        let file = tokio::fs::File::from_std(temp.try_clone().unwrap());
        let mut writer = AsyncArrowWriter::try_new(file.try_clone().await.unwrap(), to_write.schema(), None).unwrap();
        writer.write(&to_write).await.unwrap();
        // Note this is smaller than the actual size of the file
        // as it doesn't include the metadata
        let reported = writer.bytes_written();
        let metadata = writer.close().await.unwrap();

        // HOW do I get the total size of the file written?
        let actual = file.metadata().await.unwrap().len() as usize;

        assert_eq!(reported, actual);
    }

Describe alternatives you've considered

Additional context

[tustvold]

Perhaps we could provide methods to return the inner writer, similar to what we do for the sync ArrowWriter - https://docs.rs/parquet/latest/parquet/arrow/arrow_writer/struct.ArrowWriter.html#method.into_inner

[alamb]

Perhaps we could provide methods to return the inner writer, similar to what we do for the sync ArrowWriter - https://docs.rs/parquet/latest/parquet/arrow/arrow_writer/struct.ArrowWriter.html#method.into_inner

That is a good idea -- -- I think we would have to add a finish method too. So in that case the the example would look something like

...
        writer.write(&to_write).await.unwrap();
        let metadata = writer.finish().await?;
        let inner_writer = writer.into_inner();
        // get size from inner_writer
...

👍

[etseidl]

If we add finish, how about just:

        writer.write(&to_write).await.unwrap();
        let metadata = writer.finish().await?;
        // this should now include the metadata
        let reported = writer.bytes_written();