In this article I describe my problem with Azure Pipelines self hosted agent and workspace permissions denied when building golang application. I was creating a common pipeline for building Golang application when I got stuck because of a problem with removing files in workspace.

The problem was occurred after one job finished and another started. In logs was information:

Starting: Checkout mateoops/linkoln@feature/dockerize
==============================================================================
Task         : Get sources
Description  : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
Version      : 1.0.0
Author       : Microsoft
Help         : [More Information](https://go.microsoft.com/fwlink/?LinkId=798199)
==============================================================================
Syncing repository: mateoops/linkoln (GitHub)
git version
git version 2.43.5
git config --get remote.origin.url
git clean -ffdx
warning: failed to remove pkg/mod/github.com/bytedance/sonic/[email protected]/funcdata.go: Permission denied
...
warning: failed to remove pkg/mod/gopkg.in/[email protected]/go.mod: Permission denied
warning: failed to remove pkg/mod/gopkg.in/[email protected]/limit_test.go: Permission denied
warning: failed to remove pkg/mod/gopkg.in/[email protected]/node_test.go: Permission denied
warning: failed to remove pkg/mod/gopkg.in/[email protected]/parserc.go: Permission denied
warning: failed to remove pkg/mod/gopkg.in/[email protected]/readerc.go: Permission denied
warning: failed to remove pkg/mod/gopkg.in/[email protected]/resolve.go: Permission denied
warning: failed to remove pkg/mod/gopkg.in/[email protected]/scannerc.go: Permission denied
warning: failed to remove pkg/mod/gopkg.in/[email protected]/sorter.go: Permission denied
warning: failed to remove pkg/mod/gopkg.in/[email protected]/suite_test.go: Permission denied
warning: failed to remove pkg/mod/gopkg.in/[email protected]/writerc.go: Permission denied
warning: failed to remove pkg/mod/gopkg.in/[email protected]/yaml.go: Permission denied
warning: failed to remove pkg/mod/gopkg.in/[email protected]/yamlh.go: Permission denied
warning: failed to remove pkg/mod/gopkg.in/[email protected]/yamlprivateh.go: Permission denied
##[warning]Unable to run "git clean -ffdx" and "git reset --hard HEAD" successfully, delete source folder instead.
##[error]One or several exceptions have been occurred.
##[error]System.UnauthorizedAccessException: Access to the path '/opt/az-agent/_work/1/s/pkg/mod/google.golang.org/[email protected]/cmd/protoc-gen-go/testdata/imports/test_b_1/m2.pb.go' is denied.
 ---> System.IO.IOException: Permission denied
   --- End of inner exception stack trace ---
   at System.IO.FileSystem.DeleteFile(String fullPath)
   at System.IO.FileInfo.Delete()
   at Microsoft.VisualStudio.Services.Agent.Util.IOUtil.<>c__DisplayClass12_0.<DeleteDirectory>b__0(FileSystemInfo item) in /mnt/vss/_work/1/s/src/Agent.Sdk/Util/IOUtil.cs:line 156
   at System.Linq.Parallel.ForAllOperator`1.ForAllEnumerator`1.MoveNext(TInput& currentElement, Int32& currentKey)
   at System.Linq.Parallel.ForAllSpoolingTask`2.SpoolingWork()
   at System.Linq.Parallel.SpoolingTaskBase.Work()
   at System.Linq.Parallel.QueryTask.BaseWork(Object unused)
   at System.Linq.Parallel.QueryTask.RunTaskSynchronously(Object o)
   at System.Threading.Tasks.Task.InnerInvoke()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)

The error indicated that the Azure Pipelines task is attempting to clean the Git repository (git clean -ffdx) or reset it (git reset –hard HEAD) but is failing due to permission issues on some files and directories. This is causing the pipeline task to fail.

After investigate I I have established that:

  • no other process was blocking files
  • files had correctly permissions and was created by my agent user
  • the same problem occurred when I tried remove file from shell failed to remove pkg/mod/gopkg.in/[email protected]/yamlprivateh.go: Permission denied

I found the bug - owner hadn’t write permission set on module directory. When I fixed it “by hand” - files was able to delete. After more investigation I found cause of the problem. In the pipeline I set GOPATH as follow: GOPATH: '$(system.defaultWorkingDirectory)'

This was mistake. GOPATH shouldn’t be set into workspace directory. I change GOPATH to /opt/gopath directory which resolve the problem. Now all the packages are shared in this directory and no problems occurs.

Why this problem occurred?

I copied parts of pipeline from another project from Github. There are hundreds of Github projects with set GOPATH to workspace. BUT it doesn’t mistake when you use hosted runners which is not persistent. On a self hosted agent GOPATH must be set out of workspace directory.