Load files in appending in databricks

Shambhu Rai 1,411

Hi Expert,

I have 2 files in blob storage .

file1.csv, file2.csv

how will i load it in a view using appending order using *.csv extension using temp view

file_list = dbutils.fs.ls("/mnt/mount/test/file")

1 answer

Subashri Vasudevan 11,206 Reputation points

2023-11-29T09:53:05.2066667+00:00
Hi,

You can try the below code - (I am assuming, the files are under test/file folder)

final_files_list=[] file_list = dbutils.fs.ls("/mnt/mount/test/file") for file in file_list: final_files_list.append(file.name) df = spark.read.load(path=final_files_list,format="csv", sep=",", inferSchema="true", header="true") df.createOrReplaceTempView("TestFiles")

We are creating a pyspark data frame from the given csv files, and convert that to a temp view called TestFiles.

Please try and let us know for questions.

Thanks
Please sign in to rate this answer.
Shambhu Rai 1,411 Reputation points

2023-11-29T10:14:01.0166667+00:00

getting error: path does not found or exist

Shambhu Rai 1,411 Reputation points

2023-11-29T10:24:01.1566667+00:00

i want to load file data not path

Subashri Vasudevan 11,206 Reputation points

2023-11-29T10:48:46.1166667+00:00

Can you post your full code? Have you mounted your blob storage on data lake already?

Shambhu Rai 1,411 Reputation points

2023-11-29T11:58:43.7266667+00:00

i need date filter on the blob storage file before add into it.. your query is working directly on blob storage path

from pyspark.sql.functions import current_timestamp from pyspark.sql.functions import * from pyspark.sql.types import * from datetime import * import pandas as pd from pyspark.sql.types import IntegerType,StringType,LongType,StructField,StructType schema= StructType([ StructField("path", StringType()), StructField("name", StringType()), StructField("size", IntegerType()), StructField("moditime", LongType()), ]) ============================= file_list = dbutils.fs.ls("dbfs:/mnt/mount/test/test") display(file_list) ============================ file_list = dbutils.fs.ls("dbfs:/mnt/mount/test/test") display(file_list) ============================== df=df.select('path','name','size',(df.moditime/1000).cast('timestamp').alias('Updatedtime')) ======================== df=df.select('path','name','size','Updatedtime').filter(df['Updatedtime']>= current_timestamp()).show()

Shambhu Rai 1,411 Reputation points

2023-11-30T12:32:49.06+00:00

suggestion pls
Sign in to comment

Use comments to ask for clarification, additional information, or improvements to the question.

Share via

Load files in appending in databricks

1 answer

Your answer