Tentacle reports a `System.IO.DirectoryNotFoundException` because it could not find a part of the path

[sburmanoctopus]

Team

• I've assigned a team label to this issue

What happened?

Tentacle sometimes reports a System.IO.DirectoryNotFoundException because it could not find a part of the path.

Reproduction

This was noticed while monitoring for RPC retry errors. It would only happen rarely, but it was noticed.

Error and Stacktrace

Server exception: 
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.IO.DirectoryNotFoundException: Could not find a part of the path.
   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.File.InternalMove(String sourceFileName, String destFileName, Boolean checkHost)
   at Halibut.Transport.Protocol.TemporaryFileStream.SaveTo(String filePath)
   at System.Collections.Generic.List`1.ForEach(Action`1 action)
   at Octopus.Tentacle.Scripts.ScriptWorkspaceFactory.PrepareWorkspace(ScriptTicket ticket, String scriptBody, Dictionary`2 scripts, ScriptIsolationLevel isolationLevel, TimeSpan scriptMutexAcquireTimeout, String scriptMutexName, String[] scriptArguments, List`1 files) in C:\\BuildAgent\\work\\639265b01610d682\\source\\Octopus.Tentacle\\Scripts\\ScriptWorkspaceFactory.cs:line 70
   at Octopus.Tentacle.Services.Scripts.ScriptServiceV2.StartScript(StartScriptCommandV2 command) in C:\\BuildAgent\\work\\639265b01610d682\\source\\Octopus.Tentacle\\Services\\Scripts\\ScriptServiceV2.cs:line 63
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Halibut.ServiceModel.ServiceInvoker.Invoke(RequestMessage requestMessage)
   at Halibut.Transport.Protocol.MessageExchangeProtocol.InvokeAndWrapAnyExceptions(RequestMessage request, Func`2 incomingRequestProcessor)
   at Halibut.ServiceModel.HalibutProxyWithAsync.ThrowExceptionFromReceivedError(ServerError error, ILog logger)
   at Halibut.ServiceModel.HalibutProxyWithAsync.MakeRpcCall(MethodInfo asyncMethod, Object[] args)
   at Halibut.ServiceModel.HalibutProxyWithAsync.InvokeAsyncT[T](MethodInfo asyncMethod, Object[] args)
   at System.Reflection.AsyncDispatchProxyGenerator.InvokeAsync[T](Object[] args)
   at Octopus.Tentacle.Client.Decorators.HalibutExceptionAsyncScriptServiceV2Decorator.StartScriptAsync(StartScriptCommandV2 command, HalibutProxyRequestOptions halibutProxyRequestOptions)
   at Octopus.Tentacle.Client.Scripts.ScriptExecutionOrchestrator.<>c__DisplayClass14_1.<<ExecuteScript>b__2>d.MoveNext()
--- End of stack trace from previous location ---
   at Octopus.Tentacle.Client.Utils.AsyncHalibutFeatureExtensionMethods.WhenEnabled[T](AsyncHalibutFeatureWithResult`1 asyncHalibutFeatureWithResult, Func`1 action)
   at Octopus.Tentacle.Client.Scripts.ScriptExecutionOrchestrator.<>c__DisplayClass14_0.<<ExecuteScript>g__StartScriptAction|0>d.MoveNext()
--- End of stack trace from previous location ---
   at Octopus.Tentacle.Client.Execution.RpcCallExecutor.<>c__DisplayClass7_1`1.<<ExecuteWithRetries>b__3>d.MoveNext()
--- End of stack trace from previous location ---
   at Octopus.Tentacle.Client.Execution.RpcCallExecutor.<>c__DisplayClass7_0`1.<<ExecuteWithRetries>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Octopus.Tentacle.Client.Retries.RpcCallRetryHandler.<>c__DisplayClass11_0`1.<<ExecuteWithRetries>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Octopus.Tentacle.Client.Retries.RpcCallRetryHandler.<>c__DisplayClass10_0`1.<<ExecuteWithRetries>g__ExecuteAction|2>d.MoveNext()
--- End of stack trace from previous location ---
   at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext)
   at Polly.AsyncPolicy.ExecuteAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
   at Octopus.Tentacle.Client.Retries.RpcCallRetryHandler.ExecuteWithRetries[T](Func`2 action, OnRetyAction onRetryAction, OnTimeoutAction onTimeoutAction, CancellationToken cancellationToken)
   at Octopus.Tentacle.Client.Retries.RpcCallRetryHandler.ExecuteWithRetries[T](Func`2 action, OnRetyAction onRetryAction, OnTimeoutAction onTimeoutAction, Boolean abandonActionOnCancellation, TimeSpan abandonAfter, CancellationToken cancellationToken)
   at Octopus.Tentacle.Client.Execution.RpcCallExecutor.ExecuteWithRetries[T](RpcCall rpcCall, Func`2 action, ILog logger, Boolean abandonActionOnCancellation, ClientOperationMetricsBuilder clientOperationMetricsBuilder, CancellationToken cancellationToken)
   at Octopus.Tentacle.Client.Scripts.ScriptExecutionOrchestrator.ExecuteScript(CancellationToken scriptExecutionCancellationToken)
   at Octopus.Tentacle.Client.TentacleClient.ExecuteScript(StartScriptCommandV2 startScriptCommand, Action`1 onScriptStatusResponseReceived, Func`2 onScriptCompleted, ILog logger, CancellationToken scriptExecutionCancellationToken)
   at Octopus.Server.Orchestration.Targets.Tentacles.TentacleRemoteEndpointFacadeV2.ExecuteCommand(StartScriptCommand startScriptCommand, ITaskLog taskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/Targets/Tentacles/TentacleRemoteEndpointFacadeV2.cs:line 71
   at Octopus.Server.Orchestration.Targets.Common.RemoteEndpointFacadeCancellationTokenDecorator.ExecuteCommand(StartScriptCommand command, ITaskLog taskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/Targets/Common/RemoteEndpointFacadeCancellationTokenDecorator.cs:line 36
   at Octopus.Server.Orchestration.Targets.Tentacles.Observability.ErrorLoggingRemoteEndpointFacadeDecorator.ExecuteCommand(StartScriptCommand command, ITaskLog taskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/Targets/Tentacles/Observability/ErrorLoggingRemoteEndpointFacadeDecorator.cs:line 75
   at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionExecution.Immediate.ExecutionTargets.TentacleExecutionTarget.Execute(ScriptCollection bootstrapperScripts, IReadOnlyList`1 bootstrapperArguments, IReadOnlyList`1 files, Nullable`1 forceIsolationLevel, Boolean raw, ITaskLog taskLog, String isolationMutexName, CancellationToken cancellationToken, Nullable`1 isolationMutexTimeout) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/ActionExecution/Immediate/ExecutionTargets/TentacleExecutionTarget.cs:line 70
   at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionExecution.Immediate.ImmediateExecutor.RunOnTarget(ScriptCollection finalBootstrapperScripts, String platform, IBootstrapperScriptGenerator bootstrapperScriptGenerator, IReadOnlyList`1 bootstrapperArguments, IReadOnlyList`1 allFiles, Nullable`1 isolation, Nullable`1 isolationMutexTimeout, String isolationMutexName, CalamariFlavour flavour, IReadOnlyList`1 deploymentTools, ExecutionTargetDetails executionTargetDetails, CalamariPlatformConstraint calamariPlatformConstraint, ITaskLog taskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/ActionExecution/Immediate/ImmediateExecutor.cs:line 292
   at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionExecution.Immediate.ImmediateExecutor.ExecuteCalamari(CalamariFlavour calamariFlavour, String calamariCommand, IReadOnlyList`1 calamariArguments, IReadOnlyList`1 files, IReadOnlyList`1 deploymentTools, VariableCollection extraVariables, TargetManifest targetManifest, CalamariPlatformConstraint calamariPlatformConstraint, Nullable`1 isolationMutexTimeout, String isolationMutexName, ITaskLog taskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/ActionExecution/Immediate/ImmediateExecutor.cs:line 224
   at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionExecution.CommandBuilders.CalamariCommandBuilder.Execute(ITaskLog taskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/ActionExecution/CommandBuilders/CalamariCommandBuilder.cs:line 162
   at Octopus.Server.Orchestration.ServerTasks.Deploy.Steps.Acquire.PackageAcquisitionExecutionHandlers.<>c__DisplayClass7_0.<<ApplyDelta>b__0>d.MoveNext() in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/Steps/Acquire/PackageAcquisitionExecutionHandlers.cs:line 136
--- End of stack trace from previous location ---
   at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.AdHocActionDispatcher.InvokeActionHandler(Machine target, ActionHandlerInvocation actionHandler, ActionAndTargetScopedVariables actionAndTargetScopedVariables, IExecutor executor, TargetManifest targetManifest, ITaskLog taskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/ActionDispatch/AdHocActionDispatcher.cs:line 238
   at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.AdHocActionDispatcher.ExecuteOnDeploymentTarget(DeploymentTarget deploymentTarget, ActionHandlerInvocation actionHandler, TargetManifest targetManifest, ITaskLog taskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/ActionDispatch/AdHocActionDispatcher.cs:line 195
   at Octopus.Server.Orchestration.ServerTasks.Deploy.ActionDispatch.AdHocActionDispatcher.Dispatch(Machine machine, ActionHandlerInvocation actionHandler, ITaskLog taskLog, CancellationToken cancellationToken, VariableCollection variables) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/ActionDispatch/AdHocActionDispatcher.cs:line 80
   at Octopus.Server.Orchestration.ServerTasks.Deploy.Steps.Acquire.PackageAcquisitionExecutionHandlers.Invoke(ActionHandlerInvocation invocation, ITaskLog taskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/Steps/Acquire/PackageAcquisitionExecutionHandlers.cs:line 150
   at Octopus.Server.Orchestration.ServerTasks.Deploy.Steps.Acquire.PackageAcquisitionExecutionHandlers.ApplyDelta(StoredPackage nearestLocalPackage, Stream packageDeltaFile, String fullPathOnRemoteMachine, String deltaFileName, String newFileName, ITaskLog taskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/Steps/Acquire/PackageAcquisitionExecutionHandlers.cs:line 135
   at Octopus.Server.Orchestration.Targets.Common.PackageStagingMediator.UploadAndApplyDelta(StoredPackage packageToPush, StoredPackage nearestLocalPackage, String fullPathOnRemoteMachine, PackageAcquisitionExecutionHandlers execs, ITaskLog taskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/Targets/Common/PackageStagingMediator.cs:line 324
   at Octopus.Server.Orchestration.Targets.Common.PackageStagingMediator.TryPushPackageAsDelta(IPackageFeedWithLocalCache packageFeed, StoredPackage packageToPush, IEnumerable`1 existingRemotePackages, PackageAcquisitionExecutionHandlers execs, ITaskLog taskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/Targets/Common/PackageStagingMediator.cs:line 262
   at Octopus.Server.Orchestration.Targets.Common.PackageStagingMediator.PushPackageWithDeltaCompression(StoredPackage packageToPush, PackageIdentity packageIdentity, IDeploymentPackageDownloader deploymentPackageDownloader, PackageAcquisitionExecutionHandlers execs, ITaskLog taskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/Targets/Common/PackageStagingMediator.cs:line 200
   at Octopus.Server.Orchestration.Targets.Common.PackageStagingMediator.PushPackage(DependencyIdentity identity, StoredPackage packageToPush, IDeploymentPackageDownloader deploymentPackageDownloader, DeltaCompressionOptions deltaCompressionOptions, PackageCacheOptions packageCacheOptions, PackageAcquisitionExecutionHandlers execs, ITaskLog taskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/Targets/Common/PackageStagingMediator.cs:line 145
   at Octopus.Server.Orchestration.Targets.Common.PackageStagingMediator.DownloadPackageAndPushToAgent(DependencyIdentity dependencyIdentity, IDeploymentPackageDownloader deploymentPackageDownloader, DeltaCompressionOptions deltaCompressionOptions, PackageCacheOptions packageCacheOptions, PackageAcquisitionExecutionHandlers execs, ITaskLog stepTaskLog, ITaskLog deploymentTargetTaskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/Targets/Common/PackageStagingMediator.cs:line 99
   at Octopus.Server.Orchestration.ServerTasks.Deploy.Steps.Acquire.AcquireMachinePackageTask.StagePackage(PackageAcquisitionExecutionHandlers packageAcquisitionHandlers, ITaskLog stepTaskLog, ITaskLog deploymentTargetTaskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/Steps/Acquire/AcquireMachinePackageTask.cs:line 227
   at Octopus.Server.Orchestration.ServerTasks.Deploy.Steps.Acquire.AcquireMachinePackageTask.<>c__DisplayClass27_0.<<Acquire>b__0>d.MoveNext() in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/Steps/Acquire/AcquireMachinePackageTask.cs:line 134
--- End of stack trace from previous location ---
   at Octopus.Core.Threading.SingleResultProcess`1.Perform(Func`2 workerTask, Action logAction, CancellationToken cancellationToken) in ./source/Octopus.Core/Threading/SingleResultProcess.cs:line 45
   at Octopus.Core.Threading.SingleWorkerDoesWorkCoordinator`1.DoWork(String key, Func`2 workerTask, Action logAction, CancellationToken cancellationToken) in ./source/Octopus.Core/Threading/SingleWorkerDoesWorkCoordinator.cs:line 43
   at Octopus.Server.Orchestration.ServerTasks.Deploy.Steps.Acquire.AcquireMachinePackageTask.Acquire(AcquiredPackageMap acquiredPackageMap, ITaskLog stepTaskLog, ITaskLog deploymentTargetTaskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/Steps/Acquire/AcquireMachinePackageTask.cs:line 132
   at Octopus.Server.Orchestration.ServerTasks.Deploy.Steps.Acquire.AcquireMachinePackageTask.Acquire(AcquiredPackageMap acquiredPackageMap, ITaskLog stepTaskLog, ITaskLog deploymentTargetTaskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/Steps/Acquire/AcquireMachinePackageTask.cs:line 156
   at Octopus.Server.Orchestration.ServerTasks.Deploy.Steps.Acquire.AcquireMachinePackageTask.Acquire(AcquiredPackageMap acquiredPackageMap, ITaskLog stepTaskLog, ITaskLog deploymentTargetTaskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/Steps/Acquire/AcquireMachinePackageTask.cs:line 183
   at Octopus.Server.Orchestration.ServerTasks.Deploy.PlannedStepControllers.AcquirePackagesStepController.<>c__DisplayClass4_3`3.<<Execute>b__5>d.MoveNext() in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/PlannedStepControllers/AcquirePackagesStepController.cs:line 78
--- End of stack trace from previous location ---
   at Octopus.Server.Orchestration.ServerTasks.Deploy.Guidance.ExecuteWithoutGuidance(Func`2 callback, String actionName, Boolean actionIsRequiredToRun, ITaskLog taskLog, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/Guidance.cs:line 149
   at Octopus.Server.Orchestration.ServerTasks.Deploy.Guidance.Execute(Func`2 callback, String actionName, Boolean actionIsRequiredToRun, ITaskLog taskLog, Action callbackOnExclude, CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/Guidance.cs:line 79
   at Octopus.Server.Orchestration.ServerTasks.Deploy.PlannedStepControllers.AcquirePackagesStepController.<>c__DisplayClass4_3`3.<<Execute>b__4>d.MoveNext() in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/PlannedStepControllers/AcquirePackagesStepController.cs:line 85
--- End of stack trace from previous location ---
   at Octopus.Server.Orchestration.ServerTasks.Deploy.TransientErrorDetectionExecutor.Execute(Func`2 action, ExecutionPlan plan, ITaskLog taskLog, CancellationToken cancellationToken, DeploymentTarget deploymentTarget) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/TransientErrorDetectionExecutor.cs:line 50
   at Octopus.Server.Orchestration.ServerTasks.Deploy.TransientErrorDetectionExecutor.Execute(Func`2 action, ExecutionPlan plan, ITaskLog taskLog, CancellationToken cancellationToken, DeploymentTarget deploymentTarget) in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/TransientErrorDetectionExecutor.cs:line 50
   at Octopus.Server.Orchestration.ServerTasks.Deploy.PlannedStepControllers.AcquirePackagesStepController.<>c__DisplayClass4_2`3.<<Execute>b__3>d.MoveNext() in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/PlannedStepControllers/AcquirePackagesStepController.cs:line 89
--- End of stack trace from previous location ---
   at Octopus.Server.Infrastructure.Orchestration.UnitsOfWork.UnitOfWorkExecutor.<>c__DisplayClass6_0`3.<<Execute>b__0>d.MoveNext() in ./source/Octopus.Server/Infrastructure/Orchestration/UnitsOfWork/UnitOfWorkExecutor.cs:line 99
--- End of stack trace from previous location ---
   at Octopus.Core.Infrastructure.UnitsOfWork.UnitOfWorkExtensionMethods.Do(IUnitOfWork unitOfWork, Func`1 action, CancellationToken cancellationToken, String name) in ./source/Octopus.Core/Infrastructure/UnitsOfWork/UnitOfWorkExtensionMethods.cs:line 58
   at Octopus.Core.Infrastructure.UnitsOfWork.UnitOfWorkExtensionMethods.Do(IUnitOfWork unitOfWork, Func`1 action, CancellationToken cancellationToken, String name) in ./source/Octopus.Core/Infrastructure/UnitsOfWork/UnitOfWorkExtensionMethods.cs:line 58
   at Octopus.Server.Infrastructure.Orchestration.UnitsOfWork.UnitOfWorkExecutor.Execute[T1,T2,T3](Func`5 action, CancellationToken cancellationToken, String name) in ./source/Octopus.Server/Infrastructure/Orchestration/UnitsOfWork/UnitOfWorkExecutor.cs:line 102
   at Octopus.Server.Infrastructure.Orchestration.UnitsOfWork.UnitOfWorkExecutor.Execute[T1,T2,T3](Func`5 action, CancellationToken cancellationToken, String name) in ./source/Octopus.Server/Infrastructure/Orchestration/UnitsOfWork/UnitOfWorkExecutor.cs:line 102
   at Octopus.Server.Orchestration.ServerTasks.Deploy.PlannedStepControllers.AcquirePackagesStepController.<>c__DisplayClass4_1`3.<<Execute>b__2>d.MoveNext() in ./source/Octopus.Server/Orchestration/ServerTasks/Deploy/PlannedStepControllers/AcquirePackagesStepController.cs:line 92
--- End of stack trace from previous location ---
   at Octopus.Server.Orchestration.ServerTasks.ParallelWorkOrderAsync`1.<>c__DisplayClass7_1.<<Execute>b__3>d.MoveNext() in ./source/Octopus.Server/Orchestration/ServerTasks/ParallelWorkOrderAsync.cs:line 61
--- End of stack trace from previous location ---
   at Octopus.Server.Orchestration.ServerTasks.OctoThreadClosureAsync`1.Execute(CancellationToken cancellationToken) in ./source/Octopus.Server/Orchestration/ServerTasks/OctoThreadClosureAsync.cs:line 61

More Information

No response

Workaround

No response