SparkException: Job aborted due to stage failure: Task not serializable: java.io.NotSerializableException:

Question

Hi,

I am running this code but this is throwing this error:

SparkException: Job aborted due to stage failure: Task not serializable: java.io.NotSerializableException:

Answer 1

@Anonymous ,

The above error can be triggered when you initialize a variable on the driver (master), but then try to use it on one of the workers. In that case, Spark Streaming will try to serialize the object to send it over to the worker, and fail if the object is not serializable.

For more details, refer “Job aborted due to stage failure: Task not serializable:”.

Hope this helps. Do let us know if you any further queries.

----------------------------------------------------------------------------------------

Do click on "Accept Answer" and Upvote on the post that helps you, this can be beneficial to other community members.

Answer 2

Hi
I was able to resolve this issue by adding attribute to the class for each class UID number will be different.

@SerialVersionUID(114L)
class Functions() extends Serializable {

Regards
Rajaniesh