To trigger automatic clean-ups in Apache Spark, you can use the "spark

How can you trigger automatic clean-ups in Spark to handle accumulated metadata?

Question Posted / Mukesh Kumar Gangwar

1 Answers
469 Views
I also Faced
E-Mail Answers

Answer Posted / Mukesh Kumar Gangwar

To trigger automatic clean-ups in Apache Spark, you can use the "spark.sql.execution.arrow.enableHadoopMetadataCleaner" configuration property. This property enables a metadata cleaner that cleans up Hadoop Distributed File System (HDFS) blocks when they are no longer needed. By default, it is set to false.

Is This Answer Correct ?

0 Yes

0 No

Post New Answer View All Answers

Please Help Members By Posting Answers For Below Questions

What is the latest version of spark?

288

What is meant by Transformation? Give some examples.

328

List the advantage of Parquet file in Apache Spark?

474

Explain how RDDs work with Scala in Spark

355