java.nio.file.AccessDeniedExceptions for ORT Versions 31+ on MS Windows

[MNesche]

Describe the bug

Any ORT Version starting with 31.0.0, running on MS Windows 10, result in a java.nio.file.AccessDeniedException when using the scanner.

To Reproduce

Steps to reproduce the behavior:

1. First do 'ort --info scan -i \analyzer-result.yml -o '
2. See error

Expected behavior

Scan will finish successfully.

Console / log output

16:51:19.367 [main] INFO  org.ossreviewtoolkit.utils.common.ProcessCapture - Running 'scancode.bat --version' in 'C:\temp\tools_ort\ort-31.0.0\bin'...
Exception in thread "main" java.nio.file.FileSystemException: Failed to delete one or more files. See suppressed exceptions for details.
        at kotlin.io.path.PathsKt__PathRecursiveFunctionsKt.deleteRecursively(PathRecursiveFunctions.kt:313)
        at org.ossreviewtoolkit.utils.common.ExtensionsKt.safeDeleteRecursively(Extensions.kt:127)
        at org.ossreviewtoolkit.utils.common.ExtensionsKt.safeDeleteRecursively$default(Extensions.kt:124)
        at org.ossreviewtoolkit.downloader.DefaultWorkingTreeCache.shutdown(WorkingTreeCache.kt:100)
        at org.ossreviewtoolkit.plugins.commands.scanner.ScannerCommand$runScanners$5.invokeSuspend(ScannerCommand.kt:240)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:104)
        at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:277)
        at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:95)
        at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:69)
        at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
        at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:48)
        at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
        at org.ossreviewtoolkit.plugins.commands.scanner.ScannerCommand.runScanners(ScannerCommand.kt:240)
        at org.ossreviewtoolkit.plugins.commands.scanner.ScannerCommand.run(ScannerCommand.kt:142)
        at com.github.ajalt.clikt.parsers.Parser.finalizeAndRun(Parser.kt:348)
        at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:218)
        at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:245)
        at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:42)
        at com.github.ajalt.clikt.core.CliktCommand.parse(CliktCommand.kt:457)
        at com.github.ajalt.clikt.core.CliktCommand.parse$default(CliktCommand.kt:454)
        at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:474)
        at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:481)
        at org.ossreviewtoolkit.cli.OrtMainKt.main(OrtMain.kt:85)
        Suppressed: java.nio.file.AccessDeniedException: C:\Users\user~1\AppData\Local\Temp\ort-DefaultWorkingTreeCache15475176999660290213\.git\objects\pack\pack-999f753d47b525ce870505aa892cdf180a99ff5d.idx
                at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:89)
                at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
                at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
                at java.base/sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:273)
                at java.base/sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:109)
                at java.base/java.nio.file.Files.deleteIfExists(Files.java:1191)
                at kotlin.io.path.PathsKt__PathRecursiveFunctionsKt.insecureHandleEntry$PathsKt__PathRecursiveFunctionsKt(PathRecursiveFunctions.kt:455)
                at kotlin.io.path.PathsKt__PathRecursiveFunctionsKt.insecureEnterDirectory$PathsKt__PathRecursiveFunctionsKt(PathRecursiveFunctions.kt:466)
                at kotlin.io.path.PathsKt__PathRecursiveFunctionsKt.insecureHandleEntry$PathsKt__PathRecursiveFunctionsKt(PathRecursiveFunctions.kt:447)
                at kotlin.io.path.PathsKt__PathRecursiveFunctionsKt.insecureEnterDirectory$PathsKt__PathRecursiveFunctionsKt(PathRecursiveFunctions.kt:466)
                at kotlin.io.path.PathsKt__PathRecursiveFunctionsKt.insecureHandleEntry$PathsKt__PathRecursiveFunctionsKt(PathRecursiveFunctions.kt:447)
                at kotlin.io.path.PathsKt__PathRecursiveFunctionsKt.insecureEnterDirectory$PathsKt__PathRecursiveFunctionsKt(PathRecursiveFunctions.kt:466)
                at kotlin.io.path.PathsKt__PathRecursiveFunctionsKt.insecureHandleEntry$PathsKt__PathRecursiveFunctionsKt(PathRecursiveFunctions.kt:447)
                at kotlin.io.path.PathsKt__PathRecursiveFunctionsKt.insecureEnterDirectory$PathsKt__PathRecursiveFunctionsKt(PathRecursiveFunctions.kt:466)
                at kotlin.io.path.PathsKt__PathRecursiveFunctionsKt.insecureHandleEntry$PathsKt__PathRecursiveFunctionsKt(PathRecursiveFunctions.kt:447)
                at kotlin.io.path.PathsKt__PathRecursiveFunctionsKt.deleteRecursivelyImpl$PathsKt__PathRecursiveFunctionsKt(PathRecursiveFunctions.kt:369)
                at kotlin.io.path.PathsKt__PathRecursiveFunctionsKt.deleteRecursively(PathRecursiveFunctions.kt:310)
                ... 23 more
        Suppressed: java.nio.file.AccessDeniedException: C:\Users\user~1\AppData\Local\Temp\ort-DefaultWorkingTreeCache15475176999660290213\.git\objects\pack\pack-999f753d47b525ce870505aa892cdf180a99ff5d.pack
                at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:89)
                at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
                at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
                at java.base/sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:273)
                at java.base/sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:109)
                at java.base/java.nio.file.Files.deleteIfExists(Files.java:1191)
                at kotlin.io.path.PathsKt__PathRecursiveFunctionsKt.insecureHandleEntry$PathsKt__PathRecursiveFunctionsKt(PathRecursiveFunctions.kt:455)
                at kotlin.io.path.PathsKt__PathRecursiveFunctionsKt.insecureEnterDirectory$PathsKt__PathRecursiveFunctionsKt(PathRecursiveFunctions.kt:466)
                at kotlin.io.path.PathsKt__PathRecursiveFunctionsKt.insecureHandleEntry$PathsKt__PathRecursiveFunctionsKt(PathRecursiveFunctions.kt:447)
                at kotlin.io.path.PathsKt__PathRecursiveFunctionsKt.insecureEnterDirectory$PathsKt__PathRecursiveFunctionsKt(PathRecursiveFunctions.kt:466)
                at kotlin.io.path.PathsKt__PathRecursiveFunctionsKt.insecureHandleEntry$PathsKt__PathRecursiveFunctionsKt(PathRecursiveFunctions.kt:447)
                at kotlin.io.path.PathsKt__PathRecursiveFunctionsKt.insecureEnterDirectory$PathsKt__PathRecursiveFunctionsKt(PathRecursiveFunctions.kt:466)
                at kotlin.io.path.PathsKt__PathRecursiveFunctionsKt.insecureHandleEntry$PathsKt__PathRecursiveFunctionsKt(PathRecursiveFunctions.kt:447)
                at kotlin.io.path.PathsKt__PathRecursiveFunctionsKt.insecureEnterDirectory$PathsKt__PathRecursiveFunctionsKt(PathRecursiveFunctions.kt:466)
                at kotlin.io.path.PathsKt__PathRecursiveFunctionsKt.insecureHandleEntry$PathsKt__PathRecursiveFunctionsKt(PathRecursiveFunctions.kt:447)
                at kotlin.io.path.PathsKt__PathRecursiveFunctionsKt.deleteRecursivelyImpl$PathsKt__PathRecursiveFunctionsKt(PathRecursiveFunctions.kt:369)
                at kotlin.io.path.PathsKt__PathRecursiveFunctionsKt.deleteRecursively(PathRecursiveFunctions.kt:310)
                ... 23 more

Environment

Output of the ort requirements -l commands command:

C:\temp\tools_ort\ort-31.0.0\bin>ort requirements -l commands
 ______________________________
/        \_______   \__    ___/ The OSS Review Toolkit, version 31.0.0,
|    |   | |       _/ |    |    built with JDK 11.0.24+8, running under Java 21.0.3.
|    |   | |    |   \ |    |    Executing 'requirements' as 'user' on Windows 10
\________/ |____|___/ |____|    with 24 CPUs and a maximum of 8116 MiB of memory.

Environment variables:
ORT_CONFIG_DIR = C:\temp\wspProdukte\ort-config
ORT_DATA_DIR = C:\Users\user\.ort
USERPROFILE = C:\Users\user
OS = Windows_NT
COMSPEC = C:\Windows\system32\cmd.exe
JAVA_HOME = C:\Dev\jdk-21

Looking for ORT configuration in the following file:
        C:\temp\wspProdukte\ort-config\config.yml

Scanners:
        - Askalono: Requires 'askalono.exe' in no specific version. Tool not found.
        - BoyterLc: Requires 'lc.exe' in no specific version. Tool not found.
        - Licensee: Requires 'licensee.bat' in no specific version. Tool not found.
        * ScanCode: Requires 'scancode.bat' in version >=3.0.0. Found version 32.1.0.

PackageManagers:
        - Bazel: Requires 'bazel' in version >=7.0.0. Tool not found.
        - Bower: Requires 'bower.cmd' in version >=1.8.8. Tool not found.
        - Cargo: Requires 'cargo' in no specific version. Tool not found.
        - CocoaPods: Requires 'pod.bat' in version >=1.11.0. Tool not found.
        - Composer: Requires 'composer.bat' in version >=1.5.0. Tool not found.
        - Conan: Requires 'conan' in version >=1.44.0 and <2.0.0. Tool not found.
        - GoMod: Requires 'go' in version >=1.21.1. Tool not found.
        * Npm: Requires 'npm.cmd' in version >=6.0.0 and <11.0.0. Found version 10.2.3.
        - NuGetInspector: Requires 'nuget-inspector' in no specific version. Tool not found.
        - Pipenv: Requires 'pipenv' in version >=2018.10.9. Tool not found.
        - Pnpm: Requires 'pnpm.cmd' in version >=5.0.0 and <10.0.0. Tool not found.
        - Poetry: Requires 'poetry' in no specific version. Tool not found.
        - Pub: Requires 'dart.bat' in version >=2.10.0. Tool not found.
        - PythonInspector: Requires 'python-inspector' in version >=0.9.2. Tool not found.
        - Sbt: Requires 'sbt.bat' in version >=0.13.0. Tool not found.
        - Stack: Requires 'stack' in version >=2.1.1. Tool not found.
        - SwiftPm: Requires 'swift.exe' in no specific version. Tool not found.
        - Yarn: Requires 'yarn.cmd' in version >=1.3.0 and <1.23.0. Tool not found.

VersionControlSystems:
        * Git: Requires 'git' in version >=2.29.0. Found version 2.46.2.windows.1.
        - GitRepo: Requires 'repo' in no specific version. Tool not found.
        - Mercurial: Requires 'hg' in no specific version. Tool not found.

Prefix legend:
        - The tool was not found in the PATH environment.
        + The tool was found in the PATH environment, but not in the required version.
        * The tool was found in the PATH environment in the required version.

ScanCode license texts found in 'C:\temp\tools_ort\scancode-toolkit\venv\Lib\site-packages\licensedcode\data\licenses'.

Not all tools requirements were satisfied:
        ! Some tools were not found at all.

Or manually specify:

• ORT version: [e.g. 22.1.0]
• Java version: [e.g. 17]
• OS: [e.g. Linux]

And specify (relevant parts of) your ORT configuration (config.yml):

no config related parts

Additional context

According to sschuberth, the bug might be caused by the changes in commit:
a477ded

[sschuberth]

@randygeyer, as you seem to be running into this as well, would you be able to assist with debugging this? Like finding out what's probably locking the .git directory on Windows so that deletion fails?

I'm not in favor of just reverting a477ded as that would reintroduce #8987.

[randygeyer]

@sschuberth I'll look into it as I have reproduced this in my own unrelated code that also uses jgit and java.nio.file.Path.deleteRecursively(), which I work around using java.nio.file.Path.toFile().deleteRecursively() currently. I see that work-around will reintroduce #8987.

However, I won't have cycles for this for a week or two, and this is a critical blocker for me. How can I work around this in the meantime? Is there an alternative WorkingTreeCache that can be configured?

[sschuberth]

How can I work around this in the meantime?

Please give #9348 a try.

Is there an alternative WorkingTreeCache that can be configured?

No.

[sschuberth]

Note to myself: With DefaultPackageProvenanceResolverFunTest we already have a test that fails on Windows due to this issue.

[sschuberth]

So the problem is that files within .git have the read-only attribute set on Windows. I'll work on a fix.

[randygeyer]

Interesting that File.deleteRecursively() ignores that.

Any estimate on a release date for either the .recover workaround or the fix?

[sschuberth]

Any estimate on a release date for either the .recover workaround or the fix?

We do weekly releases on Thursdays. I'm confident that a fix makes it into it.