Build up the maps that represent stage-job relationships
Build up the maps that represent stage-job relationships
ID of the job
IDs of the associated stages
Close log file, and clean the stage relationship in stageIdToJobId
Close log file, and clean the stage relationship in stageIdToJobId
ID of the job
Create a folder for log files, the folder's name is the creation time of jobLogger
Create a folder for log files, the folder's name is the creation time of jobLogger
Create a log file for one job
Create a log file for one job
ID of the job
Fail to create log file
Write info into log file
Write info into log file
ID of the job
Info to be recorded
Controls whether to record time stamp before the info, default is true
Called when the application ends
Called when the application ends
Called when the application starts
Called when the application starts
Called when a new block manager has joined
Called when a new block manager has joined
Called when an existing block manager has been removed
Called when an existing block manager has been removed
Called when the driver receives a block update info.
Called when the driver receives a block update info.
Called when environment properties have been updated
Called when environment properties have been updated
Called when the driver registers a new executor.
Called when the driver registers a new executor.
Called when the driver receives task metrics from an executor in a heartbeat.
Called when the driver receives task metrics from an executor in a heartbeat.
Called when the driver removes an executor.
Called when the driver removes an executor.
When job ends, recording job completion status and close log file
When job ends, recording job completion status and close log file
Job end event
When job starts, record job property and stage graph
When job starts, record job property and stage graph
Job start event
When stage is completed, record stage completion status
When stage is completed, record stage completion status
Stage completed event
When stage is submitted, record stage submit info
When stage is submitted, record stage submit info
Stage submitted event
When task ends, record task completion status and metrics
When task ends, record task completion status and metrics
Task end event
Called when a task begins remotely fetching its result (will not be called for tasks that do not need to fetch the result remotely).
Called when a task begins remotely fetching its result (will not be called for tasks that do not need to fetch the result remotely).
Called when a task starts
Called when a task starts
Called when an RDD is manually unpersisted by the application
Called when an RDD is manually unpersisted by the application
Record job properties into job log file
Record job properties into job log file
ID of the job
Properties of the job
Record task metrics into job log files, including execution info and shuffle metrics
Record task metrics into job log files, including execution info and shuffle metrics
Stage ID of the task
Status info of the task
Task description info
Task running metrics
Write info into log file
Write info into log file
ID of the stage
Info to be recorded
Controls whether to record time stamp before the info, default is true
:: DeveloperApi :: A logger class to record runtime information for jobs in Spark. This class outputs one log file for each Spark job, containing tasks start/stop and shuffle information. JobLogger is a subclass of SparkListener, use addSparkListener to add JobLogger to a SparkContext after the SparkContext is created. Note that each JobLogger only works for one SparkContext
NOTE: The functionality of this class is heavily stripped down to accommodate for a general refactor of the SparkListener interface. In its place, the EventLoggingListener is introduced to log application information as SparkListenerEvents. To enable this functionality, set spark.eventLog.enabled to true.
(Since version 1.0.0) Log application information by setting spark.eventLog.enabled.