Spark Command: /usr/lib/jvm/java-21-openjdk-amd64/bin/java -cp /opt/spark-4.1.1-bin-hadoop3/conf/:/opt/spark-4.1.1-bin-hadoop3/jars/slf4j-api-2.0.17.jar:/opt/spark-4.1.1-bin-hadoop3/jars/* -Xmx1g -XX:+IgnoreUnrecognizedVMOptions --add-modules=jdk.incubator.vector --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang.invoke=ALL-UNNAMED --add-opens=java.base/java.lang.reflect=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.util.concurrent=ALL-UNNAMED --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED --add-opens=java.base/jdk.internal.ref=ALL-UNNAMED --add-opens=java.base/sun.nio.ch=ALL-UNNAMED --add-opens=java.base/sun.nio.cs=ALL-UNNAMED --add-opens=java.base/sun.security.action=ALL-UNNAMED --add-opens=java.base/sun.util.calendar=ALL-UNNAMED --add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED -Djdk.reflect.useDirectMethodHandle=false -Dio.netty.tryReflectionSetAccessible=true -Dio.netty.allocator.type=pooled -Dio.netty.handler.ssl.defaultEndpointVerificationAlgorithm=NONE --enable-native-access=ALL-UNNAMED org.apache.spark.deploy.worker.Worker --webui-port 8081 spark://dis-compute-1-of-3:7077
========================================
WARNING: Using incubator modules: jdk.incubator.vector
Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
26/01/19 22:08:29 INFO Worker: Started daemon with process name: 141329@dis-compute-1-of-3
26/01/19 22:08:29 INFO SignalUtils: Registering signal handler for TERM
26/01/19 22:08:29 INFO SignalUtils: Registering signal handler for HUP
26/01/19 22:08:29 INFO SignalUtils: Registering signal handler for INT
26/01/19 22:08:30 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
26/01/19 22:08:30 INFO SecurityManager: Changing view acls to: hadoop
26/01/19 22:08:30 INFO SecurityManager: Changing modify acls to: hadoop
26/01/19 22:08:30 INFO SecurityManager: Changing view acls groups to: hadoop
26/01/19 22:08:30 INFO SecurityManager: Changing modify acls groups to: hadoop
26/01/19 22:08:30 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: hadoop groups with view permissions: EMPTY; users with modify permissions: hadoop; groups with modify permissions: EMPTY; RPC SSL disabled
26/01/19 22:08:30 INFO Utils: Successfully started service 'sparkWorker' on port 40431.
26/01/19 22:08:30 INFO Worker: Worker decommissioning not enabled.
26/01/19 22:08:30 INFO Worker: Starting Spark worker 10.2.224.20:40431 with 2 cores, 4.8 GiB RAM
26/01/19 22:08:30 INFO Worker: Running Spark version 4.1.1
26/01/19 22:08:30 INFO Worker: Spark home: /opt/spark-4.1.1-bin-hadoop3
26/01/19 22:08:30 INFO ResourceUtils: ==============================================================
26/01/19 22:08:30 INFO ResourceUtils: No custom resources configured for spark.worker.
26/01/19 22:08:30 INFO ResourceUtils: ==============================================================
26/01/19 22:08:30 INFO JettyUtils: Start Jetty 0.0.0.0:8081 for WorkerUI
26/01/19 22:08:30 INFO Utils: Successfully started service 'WorkerUI' on port 8081.
26/01/19 22:08:30 INFO WorkerWebUI: Bound WorkerWebUI to 0.0.0.0, and started at http://dis-compute-1-of-3.js2local:8081
26/01/19 22:08:30 INFO Worker: Connecting to master dis-compute-1-of-3:7077...
26/01/19 22:08:30 INFO TransportClientFactory: Successfully created connection to dis-compute-1-of-3/10.2.224.20:7077 after 32 ms (0 ms spent in bootstraps)
26/01/19 22:08:30 INFO Worker: Successfully registered with master spark://dis-compute-1-of-3.js2local:7077
26/01/19 22:08:30 INFO Worker: Worker cleanup enabled; old application directories will be deleted in: /opt/spark-4.1.1-bin-hadoop3/work
26/01/19 22:08:49 INFO Worker: Asked to launch executor app-20260119220849-0002/0 for HelloHosts
26/01/19 22:08:49 INFO Worker: Asked to launch executor app-20260119220849-0002/1 for HelloHosts
26/01/19 22:08:49 INFO SecurityManager: Changing view acls to: hadoop
26/01/19 22:08:49 INFO SecurityManager: Changing modify acls to: hadoop
26/01/19 22:08:49 INFO SecurityManager: Changing view acls groups to: hadoop
26/01/19 22:08:49 INFO SecurityManager: Changing modify acls groups to: hadoop
26/01/19 22:08:49 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: hadoop groups with view permissions: EMPTY; users with modify permissions: hadoop; groups with modify permissions: EMPTY; RPC SSL disabled
26/01/19 22:08:49 INFO SecurityManager: Changing view acls to: hadoop
26/01/19 22:08:49 INFO SecurityManager: Changing modify acls to: hadoop
26/01/19 22:08:49 INFO SecurityManager: Changing view acls groups to: hadoop
26/01/19 22:08:49 INFO SecurityManager: Changing modify acls groups to: hadoop
26/01/19 22:08:49 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: hadoop groups with view permissions: EMPTY; users with modify permissions: hadoop; groups with modify permissions: EMPTY; RPC SSL disabled
26/01/19 22:08:49 INFO ExecutorRunner: Launch command: "/usr/lib/jvm/java-21-openjdk-amd64/bin/java" "-cp" "/opt/spark-4.1.1-bin-hadoop3/conf/:/opt/spark-4.1.1-bin-hadoop3/jars/slf4j-api-2.0.17.jar:/opt/spark-4.1.1-bin-hadoop3/jars/*" "-Xmx1024M" "-Dspark.driver.port=46445" "-Djava.net.preferIPv6Addresses=false" "-XX:+IgnoreUnrecognizedVMOptions" "--add-modules=jdk.incubator.vector" "--add-opens=java.base/java.lang=ALL-UNNAMED" "--add-opens=java.base/java.lang.invoke=ALL-UNNAMED" "--add-opens=java.base/java.lang.reflect=ALL-UNNAMED" "--add-opens=java.base/java.io=ALL-UNNAMED" "--add-opens=java.base/java.net=ALL-UNNAMED" "--add-opens=java.base/java.nio=ALL-UNNAMED" "--add-opens=java.base/java.util=ALL-UNNAMED" "--add-opens=java.base/java.util.concurrent=ALL-UNNAMED" "--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED" "--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED" "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED" "--add-opens=java.base/sun.nio.cs=ALL-UNNAMED" "--add-opens=java.base/sun.security.action=ALL-UNNAMED" "--add-opens=java.base/sun.util.calendar=ALL-UNNAMED" "--add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED" "-Djdk.reflect.useDirectMethodHandle=false" "-Dio.netty.tryReflectionSetAccessible=true" "-Dio.netty.allocator.type=pooled" "-Dio.netty.handler.ssl.defaultEndpointVerificationAlgorithm=NONE" "--enable-native-access=ALL-UNNAMED" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler@dis-compute-1-of-3.js2local:46445" "--executor-id" "1" "--hostname" "10.2.224.20" "--cores" "1" "--app-id" "app-20260119220849-0002" "--worker-url" "spark://Worker@10.2.224.20:40431" "--resourceProfileId" "0"
26/01/19 22:08:49 INFO ExecutorRunner: Launch command: "/usr/lib/jvm/java-21-openjdk-amd64/bin/java" "-cp" "/opt/spark-4.1.1-bin-hadoop3/conf/:/opt/spark-4.1.1-bin-hadoop3/jars/slf4j-api-2.0.17.jar:/opt/spark-4.1.1-bin-hadoop3/jars/*" "-Xmx1024M" "-Dspark.driver.port=46445" "-Djava.net.preferIPv6Addresses=false" "-XX:+IgnoreUnrecognizedVMOptions" "--add-modules=jdk.incubator.vector" "--add-opens=java.base/java.lang=ALL-UNNAMED" "--add-opens=java.base/java.lang.invoke=ALL-UNNAMED" "--add-opens=java.base/java.lang.reflect=ALL-UNNAMED" "--add-opens=java.base/java.io=ALL-UNNAMED" "--add-opens=java.base/java.net=ALL-UNNAMED" "--add-opens=java.base/java.nio=ALL-UNNAMED" "--add-opens=java.base/java.util=ALL-UNNAMED" "--add-opens=java.base/java.util.concurrent=ALL-UNNAMED" "--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED" "--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED" "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED" "--add-opens=java.base/sun.nio.cs=ALL-UNNAMED" "--add-opens=java.base/sun.security.action=ALL-UNNAMED" "--add-opens=java.base/sun.util.calendar=ALL-UNNAMED" "--add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED" "-Djdk.reflect.useDirectMethodHandle=false" "-Dio.netty.tryReflectionSetAccessible=true" "-Dio.netty.allocator.type=pooled" "-Dio.netty.handler.ssl.defaultEndpointVerificationAlgorithm=NONE" "--enable-native-access=ALL-UNNAMED" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler@dis-compute-1-of-3.js2local:46445" "--executor-id" "0" "--hostname" "10.2.224.20" "--cores" "1" "--app-id" "app-20260119220849-0002" "--worker-url" "spark://Worker@10.2.224.20:40431" "--resourceProfileId" "0"
26/01/19 22:09:00 INFO Worker: Asked to kill executor app-20260119220849-0002/0
26/01/19 22:09:00 INFO Worker: Asked to kill executor app-20260119220849-0002/1
26/01/19 22:09:00 INFO ExecutorRunner: Runner thread for executor app-20260119220849-0002/1 interrupted
26/01/19 22:09:00 INFO ExecutorRunner: Killing process!
26/01/19 22:09:00 INFO ExecutorRunner: Runner thread for executor app-20260119220849-0002/0 interrupted
26/01/19 22:09:00 INFO ExecutorRunner: Killing process!
26/01/19 22:09:00 INFO Worker: Executor app-20260119220849-0002/0 finished with state KILLED message Runner thread for executor app-20260119220849-0002/0 interrupted exitStatus 143
26/01/19 22:09:00 INFO ExternalShuffleBlockResolver: Clean up non-shuffle and non-RDD files associated with the finished executor 0
26/01/19 22:09:00 INFO ExternalShuffleBlockResolver: Executor is not registered (appId=app-20260119220849-0002, execId=0)
26/01/19 22:09:01 INFO Worker: Executor app-20260119220849-0002/1 finished with state KILLED message Runner thread for executor app-20260119220849-0002/1 interrupted exitStatus 143
26/01/19 22:09:01 INFO ExternalShuffleBlockResolver: Clean up non-shuffle and non-RDD files associated with the finished executor 1
26/01/19 22:09:01 INFO ExternalShuffleBlockResolver: Executor is not registered (appId=app-20260119220849-0002, execId=1)
26/01/19 22:09:01 INFO ExternalShuffleBlockResolver: Application app-20260119220849-0002 removed, cleanupLocalDirs = true
26/01/19 22:09:01 INFO Worker: Cleaning up local directories for application app-20260119220849-0002
26/01/19 22:10:03 INFO Worker: Asked to launch executor app-20260119221003-0003/0 for HelloHosts
26/01/19 22:10:03 INFO SecurityManager: Changing view acls to: hadoop
26/01/19 22:10:03 INFO SecurityManager: Changing modify acls to: hadoop
26/01/19 22:10:03 INFO SecurityManager: Changing view acls groups to: hadoop
26/01/19 22:10:03 INFO SecurityManager: Changing modify acls groups to: hadoop
26/01/19 22:10:03 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: hadoop groups with view permissions: EMPTY; users with modify permissions: hadoop; groups with modify permissions: EMPTY; RPC SSL disabled
26/01/19 22:10:03 INFO ExecutorRunner: Launch command: "/usr/lib/jvm/java-21-openjdk-amd64/bin/java" "-cp" "/opt/spark-4.1.1-bin-hadoop3/conf/:/opt/spark-4.1.1-bin-hadoop3/jars/slf4j-api-2.0.17.jar:/opt/spark-4.1.1-bin-hadoop3/jars/*" "-Xmx1024M" "-Dspark.driver.port=38025" "-Djava.net.preferIPv6Addresses=false" "-XX:+IgnoreUnrecognizedVMOptions" "--add-modules=jdk.incubator.vector" "--add-opens=java.base/java.lang=ALL-UNNAMED" "--add-opens=java.base/java.lang.invoke=ALL-UNNAMED" "--add-opens=java.base/java.lang.reflect=ALL-UNNAMED" "--add-opens=java.base/java.io=ALL-UNNAMED" "--add-opens=java.base/java.net=ALL-UNNAMED" "--add-opens=java.base/java.nio=ALL-UNNAMED" "--add-opens=java.base/java.util=ALL-UNNAMED" "--add-opens=java.base/java.util.concurrent=ALL-UNNAMED" "--add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED" "--add-opens=java.base/jdk.internal.ref=ALL-UNNAMED" "--add-opens=java.base/sun.nio.ch=ALL-UNNAMED" "--add-opens=java.base/sun.nio.cs=ALL-UNNAMED" "--add-opens=java.base/sun.security.action=ALL-UNNAMED" "--add-opens=java.base/sun.util.calendar=ALL-UNNAMED" "--add-opens=java.security.jgss/sun.security.krb5=ALL-UNNAMED" "-Djdk.reflect.useDirectMethodHandle=false" "-Dio.netty.tryReflectionSetAccessible=true" "-Dio.netty.allocator.type=pooled" "-Dio.netty.handler.ssl.defaultEndpointVerificationAlgorithm=NONE" "--enable-native-access=ALL-UNNAMED" "org.apache.spark.executor.CoarseGrainedExecutorBackend" "--driver-url" "spark://CoarseGrainedScheduler@dis-compute-1-of-3.js2local:38025" "--executor-id" "0" "--hostname" "10.2.224.20" "--cores" "2" "--app-id" "app-20260119221003-0003" "--worker-url" "spark://Worker@10.2.224.20:40431" "--resourceProfileId" "0"
26/01/19 22:10:11 INFO Worker: Asked to kill executor app-20260119221003-0003/0
26/01/19 22:10:11 INFO ExecutorRunner: Runner thread for executor app-20260119221003-0003/0 interrupted
26/01/19 22:10:11 INFO ExecutorRunner: Killing process!
26/01/19 22:10:11 INFO Worker: Executor app-20260119221003-0003/0 finished with state KILLED message Runner thread for executor app-20260119221003-0003/0 interrupted exitStatus 143
26/01/19 22:10:11 INFO ExternalShuffleBlockResolver: Clean up non-shuffle and non-RDD files associated with the finished executor 0
26/01/19 22:10:11 INFO ExternalShuffleBlockResolver: Executor is not registered (appId=app-20260119221003-0003, execId=0)
26/01/19 22:10:11 INFO Worker: Cleaning up local directories for application app-20260119221003-0003
26/01/19 22:10:11 INFO ExternalShuffleBlockResolver: Application app-20260119221003-0003 removed, cleanupLocalDirs = true