What happens to an interrupted job?

Question

I have a background in economics and auctions and understand the spot VM conceptually on that level, but I don't understand the practical implementation of it. In particular: If I am running an job on a spot VM and it is interrupted because my reservation price is too low, will it automatically continue once I am in the money again or will I need to somehow restart it? If I need to restart it, can you refer me to some script that will do this automatically, or must I restart it manually?

Answer 1

When you are running a Spot VM and if the reservation price goes down, Then there is no way to restart the job.

Generally these spot instances will be managed by an orchestrator which manages the jobs which will be deployed on those VM's.

So its the job of the orchestrator to reschedule or restart or wait till the orchestrator gets a VM to deploy.

An example of that orchestrator is Azure batch.

Lets say if you want to convert a hd video into many different resolution videos. In this scenario, converting those videos dont need to happen in realtime. We can wait for the vms prices to reduce to the acceptable value for the workload and then use those vms. We can use azure batch to accomplish that jobs. Lets see how we can do that.

You can have the converter application in the spot VM's image. Then run that image with azure batch.

Then you can create multiple jobs in the batch to cover those videos. Jobs will wait until the vms are available. We can also make the jobs to reschedule into other vms if the vm gets terminated.

We can also restart the jobs. Overall we can manager the jobs lifecycle.